Chapter 4 · 2018

Why not look at the words on both sides?

In “the ___ of the river,” you use river as much as the. A next-word guesser never gets to do that during practice, because “river” is still in the future.

The leftover problem was a one-way window.

GPT’s practice was: look left, guess right. For jobs like “what does this sentence mean?” or “answer a question about this paragraph,” the useful clue might sit on either side of the important word.

The bank → of the river

writing practice · the right side is off-limits

Hide a word in the middle. Let every other word look both ways to fill it in.

A row of word-stones with a hole in the middle; a detective looks left and right to guess the missing word. — The blank is in the middle. Left and right are both fair game.

Here is how they made “both ways” a fair game.

1 · Cover some words

Take a real sentence. Hide a word (sometimes a few). The machine’s job is to guess the missing piece.

the ? of the river

clues on the left and the right

2 · Look both ways on purpose

Same looking-around machine as chapter 2 — but now every word may attend to every other word, including the ones after the blank. That is the difference from GPT’s one-way window.

3 · A second little game with two sentences

Sometimes they also show two stretches of text and ask: does the second one really follow the first? That nudges the model to think about pairs, not just single sentences — handy later for questions and answers packed together.

4 · Then a tiny extra piece for a real job

After that practice, they add one extra output layer and fine-tune on a specific task — question answering, sentence judgment, and so on. The same pretrained body, a small new hat.

Understanding a finished sentence is a different sport from writing the next word — so they changed the homework.

They called the model BERT — bidirectional encoder representations from Transformers. The hide-a-word game is a masked language model. The pair game is next-sentence prediction.

It set new records on a pile of understanding tests.

Question answering, language inference, and a bundle of other sentence jobs jumped. The paper’s claim is simple: deep both-ways practice, then a small fine-tune, beats one-way pre-training on those understanding benchmarks.

One-way practice

Great for writing the next word. Blind to clues on the right.

Both-ways practice

The blank sits in the middle. The whole sentence may help fill it.

That’s why this still matters.

Search, autocomplete, and “understand this paragraph” tools still lean on the idea: if the text is already there, let every word see every other word.