The scarcest resource in language-model training right now is human judgment. RLHF, the technique behind ChatGPT, needs crowds of annotators ranking candidate answers so a reward model can learn what "better" means. That pipeline is slow, expensive, and effectively gated: a lab can afford it; a student or a small company cannot. Which raises a question almost too cheap to be legal: if a strong model can already produce and judge good answers, why does a human need to be in the loop at all?
Together with a Stanford student, I spent the past few weeks testing exactly that. We trained our own model, a 6B-parameter GPT-J, through the complete RLHF pipeline (supervised fine-tuning, reward model, PPO), with every human touchpoint replaced by a stronger model. ChatGPT stands in for the crowd. The target skill is long-form question answering with citations.
Training recipe
We kept the design deliberately faithful to the human-annotated original, so that anything we learned would transfer. The pipeline, end to end:
Synthetic data. Starting from ~10k open-source Q&A prompts:
- Generate diversity. For each question, ChatGPT produced two additional answers under different prompts, one direct and one written with citations, giving three candidates per question alongside the original.
- Replace the crowd with a judge. The same teacher model ranked all three candidates, using a prompt that mirrors the ranking task human annotators perform in standard RLHF, justification included.
- Distill judgment into data. Parsing the rankings gave pairwise preference labels; after filtering, 131K comparison pairs.
Data collection took six days on a standard API key — four generating answers, two generating rankings, exponential backoff doing the patience — all for the price of the API calls. The same labels from crowdworkers would take weeks and a real budget.
Fine-tuning. We trained the student through the standard three stages: SFT on the 10k Q&A pairs, a reward model on the 131K comparisons, PPO on the same prompts, all on a fork of CarperAI's open-source trlX. Each stage took about three hours on 8×A100s. That's roughly nine GPU-hours of training on top of a weekend of API calls, for a recipe that normally implies an annotation contract.
How good is the student?
This is a proof of concept, so the evaluation is qualitative: side-by-side comparisons against the student's own teacher on live queries, with retrieved web snippets provided as context. Three results stand out.
It learned grounded citation. Given sources, the student summarizes them faithfully and places citations at the right claims, which is the behavior we trained for. The teacher answers the same questions fluently but from memory, with nothing to check.
It can beat its teacher on freshness. Because the student answers from retrieved context rather than memorized weights, it correctly reported the Silicon Valley Bank collapse, news from this week, with a citation on each claim. Its teacher, whose knowledge ends in 2021, cannot. The student knows things the model that trained it doesn't, and that inversion delights me.
Form arrives before substance. There is a negative result worth stating plainly. The student reproduces the bulleted structure of its training answers beautifully but is weaker at organizing content semantically: it learned what a good answer looks like before it learned what one is. My guess is that this gap is a signature of synthetic supervision, because style transfers faster than judgment.
Known limitations
A single teacher explores a narrow band of styles; many use cases would need far more prompt variation than our two. More fundamentally, a student inherits its teacher's flaws wholesale. Biases, blind spots, and any poisoning an adversary slips into the teacher's outputs flow silently downstream into the student's weights. Synthetic data is a supply chain, and supply chains have provenance problems. And a qualitative evaluation like ours shows that a behavior exists; it doesn't measure how reliably.
Assets
The pipeline is open: the synthetic-data generation code (answer generation, ranking prompts, backoff handling) and our fork of trlX used for the RLHF stages.
What this changes
The finding is narrow but real: model-generated supervision taught a small model a genuine skill, for six days of API calls and nine GPU-hours. That puts the full RLHF recipe within reach of a student project, because that is what this was. The obvious next questions are scale questions — bigger students, more varied teachers, quantitative evals. If the recipe holds up there, annotation stops being the gate on who gets to fine-tune.
* As we were writing this up, Stanford released Alpaca, which fine-tunes LLaMA on 52K instructions distilled from a stronger model. The same bet, arrived at independently. Alpaca stops at supervised fine-tuning; our project pushes the idea through the full RLHF loop, judge included. The idea is in the air.