Chapter 3 · 2018
What if we just keep guessing the next word?
You already watch ChatGPT finish your sentence. This chapter is the research idea underneath that habit: practice by always guessing what comes next — then, later, specialize.
Labeled answers were expensive. Unlabeled books were everywhere.
Teaching a machine a specific job — “is this review positive?” — usually meant people tagging thousands of examples. Meanwhile, the world was full of ordinary text with no tags at all.
Hide the next word. Guess it. Repeat — on a huge pile of books.
Here is how the practice actually works.
Use the Transformer from last chapter — the one where words can see other words. But while guessing the next word, it may only look backward. The future isn’t written yet, so peeking ahead would be cheating.
Read a stretch of story. Hide the upcoming word. Guess it. Then reveal it and move one step forward. No one has to label “positive” or “negative.” The next word is the label.
left side in · next word out
After the book practice, give the same machine a small stack of real task examples — a question, a review, a sentence to judge. Tweak it so it can do that job. The architecture barely changes. The heavy lifting already happened while guessing the next word.
First get generally good at language. Then put on a small job-specific hat.
Researchers called the book practice unsupervised pre-training and the job hat fine-tuning. The model is a Transformer that only looks left while it writes — the ancestor people now nickname GPT.
They tried it on a spread of language jobs.
After the two-stage practice, the same general model beat specialist systems on nine of the twelve jobs they studied. A version of the same machine that skipped the book-reading warmup did much worse. The books were doing real work.
Job-only training
Start from scratch on the tagged examples you can afford.
Books, then a hat
Absorb language from unlabeled text, then adapt with a little labeled practice.
That’s why this still matters.
“Guess the next word on a lot of text, then specialize” is still the skeleton of how large chat models are grown.