Section Deep Dives
Word2Vec — one section at a time, plain language
The problem with word IDs
Most NLP systems treat each word as an atomic unit — just an index in a vocabulary. Index 3 and index 7 are simply different integers, so the model has no way to know that two words might mean similar things.
That simplicity has real virtues — it is robust, and simple models on huge data often beat complex models on small data. But it hits a ceiling: N-grams can already be trained on virtually all available text, and there is nowhere further to go.
From the paper: Current systems treat words as atomic units — there is no notion of similarity between words, as these are represented as indices in a vocabulary (§1, p.1)
Standing on NNLM's shoulders
Neural network language models already learned word vectors — but jointly with a full statistical language model, using a linear projection layer plus a non-linear hidden layer. A later architecture split the job in two: learn the vectors first, then train the language model. The vectors could be learned without ever building the full NNLM.
The scale problem was the opening. Nobody had trained these architectures past a few hundred million words, at dimensionality between 50 and 100. This paper goes after billions of words and a million-word vocabulary.
From the paper: None of the previously proposed architectures has been successfully trained on more than a few hundred of millions of words, with a modest dimensionality of the word vectors between 50 and 100 (§1.1, p.2)
The complexity budget
Every model in the paper is measured by one formula. Training cost is proportional to epochs × training words × cost per word. The first two are fixed by your data and patience, so the only lever is Q — the work done per word.
O = E × T × Q
E = epochs (typically 3–50) · T = training words (up to one billion) · Q = cost per word
In the older NNLM, most of Q is spent between the projection layer and the non-linear hidden layer. Word2Vec's whole strategy is to delete that hidden layer, collapsing Q to little more than a lookup and a softmax.
From the paper: For all the following models, the training complexity is proportional to O = E × T × Q, where E is number of the training epochs, T is the number of the words in the training set (§2, p.3)
Context predicts the middle word
Context → centre. Remove the non-linear hidden layer, and share the projection layer across every word so all context words land in the same place — their vectors get averaged. Word order stops mattering, which is why it is called bag-of-words.
4 history + 4 future words in, one middle word out
The best configuration uses four history words and four future words as input, and is trained to classify the current middle word correctly. Using future words is unusual for a language model — but this is not a language model, so nothing forbids it.
From the paper: Best performance came from a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current middle word (§3.1, p.4)
Middle word predicts context
Centre → context. Run CBOW backwards: take the current word as input and predict the words within a range before and after it. Widening the range improves the vectors but costs more compute, so distant words are sampled less often.
Q = C × (D + D × log2(V))
C is the maximum distance considered. For each training word the model picks a random R between 1 and C, then uses R words of history and R of future as labels — so cost scales with the window, not the vocabulary. The experiments use C = 10.
From the paper: If we choose C = 5, for each training word we select randomly a number R in range 1 to C, then use R words from history and R words from the future as correct labels; in the experiments we use C = 10 (§3.2, p.5)
Cheaper models, better vectors
Training used a Google News corpus of about 6B tokens, with the vocabulary capped at the 1 million most frequent words. Quality is scored on a test set of semantic and syntactic analogy questions.
Both new models beat NNLM on accuracy while costing far less: NNLM needed 14 days across 180 CPU cores, CBOW needed 2 days across 140. The two architectures also split the win — Skip-gram is much stronger on semantics, CBOW on syntax.
From the paper: On the 6B set, NNLM reached 50.8 total accuracy in 14 x 180 days x CPU cores, CBOW reached 63.7 in 2 x 140, and Skip-gram reached 65.6 in 2.5 x 125 (§4.4, p.8)
Why this changed NLP
The finding is that very simple architectures produce high-quality vectors, which means the compute saved can be spent on more data instead. The authors project training on corpora of one trillion words as a realistic target.
vector(King) − vector(Man) + vector(Woman) ≈ Queen
Shown in earlier work (§1.2, p.2) — this paper's goal is to make it far more accurate
Afterwards the authors released multi-threaded C++ code running at billions of words per hour, plus more than 1.4 million named-entity vectors trained on over 100 billion words. That release, more than the paper itself, is what put embeddings into everyone's pipeline.
From the paper: We published single-machine multi-threaded C++ code, and more than 1.4 million vectors that represent named entities, trained on more than 100 billion words (§7, p.11)