Chapter 6 · 2022
Why doesn’t it just do what I asked?
You ask ChatGPT to make a shopping list from a recipe. A next-word machine trained on the internet might keep going with “bake at 350°.” That’s a likely continuation. It is not what you asked.
Making it bigger did not make it follow you.
The leftover from last chapter is a mismatch of jobs. “Guess the next word on the web” is not the same as “help this person, stay truthful, don’t be nasty.”
Show it the kind of answer you wanted. Then, when two answers appear, mark the better one.
Here is how they turned a guesser into a helper.
People write prompts (and real users send some too). Other people write the kind of reply they actually wanted. The model practices imitating those replies.
not “what a webpage would say next” · what a helper would say
Now the model writes several answers to the same prompt. A person picks which one is better. Do this a lot, and you get a second model whose only job is: “which of these would a person prefer?”
The helper is trained again, this time to produce answers the ranking model would score highly — while staying close enough to the copied-examples model that it doesn’t wander off into nonsense.
The homework changed from “sound like the internet” to “sound like the answer a person would pick.”
Copying good replies is supervised fine-tuning. Learning from the ranked pairs is reinforcement learning from human feedback. The resulting models are InstructGPT.
A much smaller helper beat the giant guesser.
On the kinds of prompts people actually send, people preferred answers from a far smaller InstructGPT over answers from the giant GPT-3. The helper was also a bit more truthful and a bit less toxic, without falling apart on ordinary language tests.
It still made simple mistakes. The paper does not claim a finished assistant. It claims a promising direction: fine-tune with human feedback.
Giant next-word
Likely continuation. Not necessarily the thing you asked for.
Shown, then ranked
A smaller helper people preferred — because the homework matched the job.
That’s why this still matters.
When a chatbot tries to follow your instructions instead of rambling like a webpage, this is the idea underneath: people demonstrated, people ranked, the machine learned to prefer.
Neighborhoods. Looking around. Next-word practice. Both-ways understanding. Scale. Then a human “that’s better.” That’s the path from locker-number words to the AI you already use.