Word2Vec — Worked Example

1

Start with a sentence

The model slides across plain text, taking each word in turn as the input word.

The · cat · sat · on · the · mat

2

Pick the input word and a window

Take sat as input. The window is not fixed: the model draws a random R between 1 and C, then uses R words of history and R of future as the labels to predict. Small R happens often, so nearby words get seen more.

The cat sat on the

Drawn here with R = 2, giving R × 2 = 4 predictions

From the paper: For each training word we select randomly a number R in range 1 to C, then use R words from history and R words from the future as correct labels, requiring R x 2 word classifications (§3.2, p.5)

3

Every word starts as noise

Each word owns one vector, initially meaningless. The best-scoring Skip-gram run in the paper uses 300 numbers per word.

            sat  → [0.02, -0.11, 0.08, …]   (300 dimensions)

            cat  → [-0.05, 0.19, 0.03, …]

From the paper: Skip-gram with vector dimensionality 300 trained on 783M words reached 50.0 semantic and 55.9 syntactic accuracy (§4.3, p.8)

4

Predict the neighbours

Using sat's vector, the model scores the vocabulary and should rank its real neighbours highly. Wrong guesses get penalised, and both vectors shift a little.

sat → predict → cat on

Cost per word is C × (D + D × log2(V)) — driven by the window, not the vocabulary

5

Repeat across the corpus

Every word takes its turn as input, across a Google News corpus of about 6B tokens with the vocabulary capped at the 1 million most frequent words. Words sharing contexts drift together.

cat, dog, kitten cluster · car sits far away

From the paper: We used a Google News corpus containing about 6B tokens, and restricted the vocabulary size to 1 million most frequent words (§4.2, p.6)

6

Score it with an analogy

This is the paper's own test, not a folk example. To find the word that is to small what biggest is to big, do the arithmetic and then search the space for the nearest vector by cosine distance.

            X = vector("biggest") − vector("big") + vector("small")

            nearest(X) → "smallest"

From the paper: We compute vector X = vector(biggest) − vector(big) + vector(small), then search the vector space for the word closest to X measured by cosine distance (§4, p.5)

7

The payoff

The same trick carries semantic relationships, not just grammatical ones. Subtract two vectors to define a relationship, then add it to a third word.

Worth knowing: the famous King − Man + Woman ≈ Queen line comes from earlier work the paper cites on p.2, not from Word2Vec itself. Word2Vec's contribution is making these operations accurate enough to be useful.

From the paper: The relationship is defined by subtracting two word vectors and adding the result to another word, so that Paris − France + Italy = Rome (§5, p.9)