Frontier

1. The idea

Frontier is a machine learning system that treats project planning as a prediction problem. Given a repository's commit history, file activity patterns, and codebase state, it predicts the highest-leverage next task using a 4-stage LLM pipeline inspired by Self-Guided Self-Play. A trained logistic regression model re-ranks candidates using 1545-dimensional feature vectors (1536-dim text embeddings + 9 engineered features), and an evaluation framework scores predictions against actual subsequent commit diffs to measure whether trajectory-aware planning outperforms static goal decomposition.

2. The pipeline

The system is a 4-stage inference pipeline where each stage operates on structured output from the previous one. The Historian produces a latent project state representation. The Conjecturer generates candidate tasks conditioned on that state. The Guide scores candidates across 7 dimensions using a rubric analogous to the Guide reward in SGS. The Planner selects the optimal task via a blended ranking of LLM scores and trained model predictions.

3. Historian

The Historian ingests raw repository signals — up to 50 commits with file-level diffs, README content, open issues, pull requests, recursive file tree, branch metadata, and source code from the top-edited files — and produces a structured latent state. Pure computation extracts file activity features (edit counts, directory-level aggregation, hotspot detection at 3+ edits), and TODO/FIXME patterns via regex. The LLM then maps this feature-rich context into a typed schema: project summary, recent trajectory, completed capabilities, active workstreams, gaps, blockers, inferred frontier, detected tech stack, and a momentum classification (stalled/slow/steady/active/rapid).

4. Conjecturer

The Conjecturer generates 8–12 candidate tasks conditioned on the Historian's inferred frontier. Each task includes a 2–4 entry evidence chain — a structured provenance trace linking the recommendation to specific commits, file patterns, README discrepancies, or implementation gaps. The Conjecturer also receives a self-play reward signal: past tasks the user completed (positive examples) and skipped (negative examples) appear as in-context few-shot demonstrations, biasing generation toward actionable task types without gradient updates.

↓ generates

5. Guide

The Guide scores each candidate on 7 dimensions (0–5 each, total 0–35), applying penalties for vague, disconnected, or oversized tasks. This maps directly to R_guide in SGS: quality pressure that prevents degenerate candidate generation. The Guide also receives the previous run's accuracy score and adjusts scoring weights accordingly — low accuracy shifts priority toward trajectoryFit and taskClarity. After scoring, if a trained logistic regression model exists, candidates are re-ranked via blended score: 70% Guide + 30% model prediction scaled to [0, 35].

6. Planner

The Planner selects the optimal task from the top 5 re-ranked candidates and generates a concrete execution plan in three time variants. It also produces a counterfactual — what a context-free planner would recommend — and explains why that recommendation is suboptimal given the current project state. This counterfactual is central to the evaluation hypothesis: trajectory-aware planning should consistently outperform static goal decomposition.

1. 30-minute version. Set up webhook route handler with Stripe signature verification. Test with Stripe CLI.
2. 60-minute version. Add payment confirmation logic, update order status in database, handle idempotency keys.
3. 90-minute version. Add error handling, retry logic for failed webhooks, logging, and a basic admin view for payment events.

7. The SGS paper

Frontier is inspired by Scaling Self-Play with Self-Guidance (Bailey, Wen, Dong, Hashimoto, Ma · Stanford · 2026). The paper addresses a fundamental problem in self-play: when a model generates problems for itself, it learns to game the reward by producing artificially complex or degenerate problems. SGS solves this by adding a third role.

The Solver and Conjecturer improve together through self-play. The Guide acts as quality control — it scores synthetic problems for relevance and clarity. Without the Guide, the Conjecturer degenerates: by iteration 100, over 80% of generated problems have overly complex disjunctive conclusions. The Guide prevents this collapse by zeroing the reward for degenerate outputs.

Reward function

My interpretation

I saw the 3-role architecture as a natural fit for project planning. A software project has a state (what's been built), a frontier (what comes next), and candidate actions (possible tasks). The Historian reconstructs state, the Conjecturer generates candidates conditioned on the frontier, and the Guide scores them to filter out noise. The developer is the Solver — they execute the recommended task, and their subsequent commits become the reward signal for the next round.

Where I diverge: SGS uses REINFORCE to update the Conjecturer's weights. I use in-context learning instead — past completed and skipped tasks appear as few-shot demonstrations in the Conjecturer and Guide prompts. The model weights never change, but the system's effective behavior does. A separate logistic regression model, trained on task embeddings and outcomes, also re-ranks Guide scores. The honest framing: Frontier is an SGS-inspired system that uses embeddings-based learning with a prompt-based approximation of the self-play feedback loop.

8. Evaluation framework

The central question is not whether Frontier can recommend tasks, but whether trajectory-aware planning is actually better than static planning. To answer this, I built an evaluation framework that runs both planners independently on the same repository and goal, then compares them side by side.

After both planners produce recommendations, an LLM generates a diff analysis identifying which additional signals changed the output. The framework supports two evaluation modes: human preference voting (baseline better / tie / frontier better) and automated outcome scoring. Outcome evaluation fetches subsequent commit diffs via the GitHub API — not commit messages, but actual code patches — and uses GPT-4o-mini to score each prediction 0–100 against the substance of what was actually built. This produces a quantitative signal for the core hypothesis: does trajectory information improve prediction accuracy?

9. Embeddings and learning

Every recommended task is embedded as a 1536-dimensional vector using OpenAI's text-embedding-3-small model and stored in SQLite as a float32 blob alongside its outcome label (completed, started, or skipped). On subsequent analyses, cosine similarity retrieval finds the k nearest past tasks, and the system constructs a learning signal: which task types the user tends to complete vs skip, overall accuracy trend, and similar-task outcomes. This signal feeds into both the LLM prompts (as in-context demonstrations) and the trained model.

The logistic regression model trains via SGD on binary cross-entropy loss with L2 regularization (λ=0.001) over 50 epochs, requiring a minimum of 5 labeled samples. The input is a 1545-dimensional vector: 1536-dim task embedding concatenated with 7 Guide scoring dimensions, normalized estimated time, and a task-type index. Labels are binary: completed/started = 1, skipped = 0. The model's P(completion) prediction is blended with the Guide's score at a 30/70 ratio, producing a final ranking that gradually adapts to each user's behavioral patterns.

Why it matters

The core research question is whether development trajectory contains enough signal to outperform static planning. The evaluation framework generates quantitative evidence: prediction scores, human preference rates, and accuracy trends over time. Each comparison adds a labeled data point. The system is designed so that running more evaluations directly answers the hypothesis rather than requiring separate analysis.

The broader pattern — reconstruct state from sequential decisions, generate candidates conditioned on the frontier, score with a quality filter, learn from outcomes via embeddings and a trained model — generalizes beyond software. Any domain with a trajectory of decisions and measurable outcomes can use the same architecture to move from static goal decomposition to trajectory-aware prediction.