Chapter 2 · 2017
How does AI figure out which words matter?
You type a sentence into ChatGPT. Somehow it notices that in “the bank of the river,” bank is dirt, not money. The neighborhood map is not enough. It has to look at these words.
For a long time, machines still walked a narrow trail.
Translation programs looked at one word. Then the next. Then the next. Each step had to wait for the last. Faraway words were hard to keep.
must wait · faraway words fade
What if every word could look at every other word — all at once?
no waiting line · everyone can see everyone
The machine didn’t need more memory. It needed a better way to look.
Watch one word do the whole trick.
Take “the bank of the river.” The word bank glances at everyone else in this sentence — not at its old house on the map.
Bank checks the other words: the, of, the, river.
Not equally. River is useful right now. Little words like the barely matter. Money-bank neighbors are not even in the room.
thicker = more useful in this sentence
Bank mixes the useful bits into what it knows next. After that mix, “bank” in this sentence means the dirt beside water.
Researchers called that looking-around trick attention. They built a whole translator out of it — no waiting line underneath — and named the machine the Transformer.
One glance can get blurry.
If you average everyone together, the useful word can drown. So they let the model take several looks at once, from different angles, then stitch the notes together.
Look A
Might watch nearby grammar — who sits next to whom.
Look B
Might watch meaning — river, not money.
Look C
Might watch a farther partner — a word at the other end of the sentence.
The paper doesn’t assign those jobs by name. It just found that several looks at once worked better than one.
They called the several-looks idea multi-head attention.
Looking at everyone at once forgets the line.
If every word can see every other word immediately, “the cat sat” could be shuffled into “sat the cat.” So they stamp each word with where it sits.
same letters · different place in line
That stamp got a name: positional encoding.
They tried it on translation.
Old way
Walk the line. Wait. Faraway words fade. Slow to train.
Looking around
Better translations than the old machines — even when those old machines teamed up. And it learned much faster.
The writer still has to say the new sentence one word at a time. Looking around didn’t remove that last wait.
That’s why this still matters.
This was a 2017 paper. The looking-around machine it introduced became the foundation for the chatbots people use now.
Not because it was a bigger brain. Because the words finally got a way to see each other.