Chapter 5 · 2020
What if a few examples in the chat were enough?
You already do this in ChatGPT: paste a couple of samples, then say “now do it for this one.” This paper asked whether a giant next-word guesser could pick up a new pattern that way — without another training run.
The leftover problem was the extra hat.
GPT and BERT got good by practicing on lots of text, then fine-tuning. Powerful. Also heavy: new labels, new training, a new specialist for every job.
What if the examples live in the prompt — and the model just keeps guessing the next word?
Here is the new way to give homework.
Still a looking-around machine that writes left to right. Still trained to guess what comes next, on a huge sweep of internet text.
The paper’s bet: as the guesser grows, it gets smoother at picking up a pattern from the words you just typed — not only from a separate fine-tuning pass.
A related study found that loss on the next-word game falls in a smooth, predictable way as you scale the model, the data, and the compute together. Bigger helps — if you feed it enough.
Zero examples: just the instruction. One example: a single sample, then the real item. A few examples: a tiny cheat sheet, then “now you go.” The weights do not update. The model simply continues the text.
the cheat sheet is the homework · no extra training pass
The prompt became the classroom. The giant guesser became the student who can catch on mid-lesson.
Researchers called this in-context learning, and the three flavors zero-shot, one-shot, and few-shot. The model is GPT-3.
On many tasks, a few examples were surprisingly enough.
With a few samples in the prompt — and no extra training — the giant model often came close to specialist systems that had been fine-tuned. Translation, question answering, made-up on-the-spot tasks. It also wrote news articles that people had a hard time telling from real ones.
It still failed plenty of things. It copied biases from its internet diet. Scale helped. Scale was not magic.
Fine-tune a specialist
New labels. New training. A new copy per job.
Show a few examples
Keep one giant guesser. Put the lesson in the prompt.
That’s why this still matters.
This is why you can invent a task in the chat box and sometimes get a decent try on the first reply. The classroom moved into the prompt.