Stop Asking "Should We Fine-Tune?"
"The agent isn't good enough — should we fine-tune it?" is the wrong first question. Training is expensive, slow to iterate, and painful to roll back. The right first question is diagnostic: why is it not good enough? Different failure causes point to wildly different — and differently priced — fixes. Teams that skip the diagnosis run RL pipelines on problems a one-line prompt change would have solved.
The Decision Tree
Work down this list in order. Each branch is cheap to evaluate with a small labeled set of your own tasks.
- The knowledge or tools just aren't there. The correct answer exists in your docs, or the right tool call is obviously available, and the agent misses it anyway. This is a context problem: fix retrieval, tool descriptions, or system prompt organization. Training cannot reliably teach your changing facts.
- Success on well-specified tasks is near zero. If the model can't do the task even once with a perfect prompt, the base capability is missing. Fine-tuning won't create it; you need a different (or larger) model, or task decomposition. Check pass@k first — if it is near zero on clear inputs, stop here.
- Capability exists, but format is unstable. The right content, wrong shape: JSON that half-parses, sections in the wrong order, tone that drifts. This is the classic supervised fine-tuning (SFT) case — demonstration data teaches "always answer like this" cheaply and reliably. SFT is memory: it clones patterns you show it.
- Multiple answers are valid, but some are better — measurably. There is a scoreable difference between acceptable and excellent outputs, and you can express it as a reward. Now reinforcement learning enters: RL generalizes beyond the demonstrations, finding behaviors nobody explicitly showed. It amplifies whatever signal you give it — including flaws in the signal.
SFT Is Memory, RL Is Generalization
The most useful one-line model of post-training: SFT clones, RL explores. SFT is imitation — show the model 500 perfect trajectories and it absorbs the pattern, including your quirks and your data's blind spots. RL is trial against a reward — the model discovers routes to higher scores nobody demonstrated, for better and worse.
Two corollaries save real money:
- Data and environments matter more than algorithms. A mediocre algorithm on great data beats a state-of-the-art algorithm on sloppy data. Half the work of a successful RL run is building the environment and reward, not tuning the optimizer.
- Reward design is threat modeling. The model optimizes the reward, not your intention. Reward the final state and it may shortcut the process; reward the process steps and it may ritualize them. The robust pattern is reward the outcome, constrain the process — with verifiable checks, and validation penalties for skipping them.
The Cheap Fixes Come First — and They're underrated
Before any training run, in rough order of cost: rewrite the system prompt (days), add or fix tools (days), add few-shot examples (days), few-shot → retrieval of similar solved cases (weeks), then SFT (weeks-months), then RL (months). Post-hoc, the training decision should be re-validated against the same eval suite that motivated it: if the score didn't move, the diagnosis was wrong, not the learning rate.
And an honest budget note: SFT data synthesis — turning your best real trajectories into training examples — is itself a pipeline (collection, filtering, formatting, decontamination). Budget for it as a project, not a script.
The Decision Tree in Practice: A Diagnostic Session
Here is how the tree resolves for the most common complaint — "the agent is dumb, let's fine-tune." Run these checks in order; stop at the first fix that moves the metric:
- Context check (hours). Give the agent the exact document an expert would use. If pass@k jumps, the problem was missing context — build retrieval, not training data. This resolves roughly half of "dumb agent" cases on real projects.
- Model ceiling check (an afternoon). Run the same task on the strongest available model with a well-specified prompt. pass@k near zero means the model class cannot do the task reliably at all — no amount of fine-tuning fixes a missing capability; change the task decomposition or wait for a better base model.
- Format check (a day). The answers are right but the parser chokes: valid reasoning, broken JSON. This is the textbook SFT case — small dataset (500–2 000 examples), cheap, fast, done in a week. Do not skip it just because it feels unsophisticated.
- Preference check (weeks). Outputs pass automated checks, humans still prefer a different style, and you can grade the difference on a scale. Now RL (usually DPO-class first, RLHF if you have a real reward model) is justified — and the earlier diagnostic steps gave you a clean baseline to measure the gain against.
Budget heuristic from production experience: if the fix is not worth at least two weeks of an engineer's time to validate, it is not worth a training run. Prompt and retrieval fixes ship in days; a training run that cannot beat a two-day prompt fix was not worth running.
The Four Diagnoses at a Glance
| Diagnosis | Time to validate | Fix | Typical dataset / cost |
|---|---|---|---|
| Missing context | Hours | Retrieval pipeline | Docs + embeddings; no training |
| Missing capability | An afternoon | Stronger model / task decomposition | None; wait or redesign |
| Unstable format | A week | SFT | 500–2 000 examples |
| Scoreable preference | Weeks | DPO first, then RLHF | Preference pairs + reward model |
Key Takeaways
- Diagnose before training: missing context, missing capability, unstable format, or scoreable preference — four different problems, four different fixes.
- pass@k near zero on well-specified tasks means wrong model, not insufficient training.
- Unstable format → SFT. Measurable quality gradient with a clean reward → RL.
- SFT clones (including your flaws); RL generalizes (including your reward hacks).
- Data and environments beat algorithms. Design the reward like a threat model.
