01. Embedding Basics — Word2Vec, SGNS, and Gravity#

All About Embedding · Part 1: Word2Vec, SGNS, and Gravity · Part 2: Implementation, Relations, and Meaning Axes · Part 3: Culture and Semantic Change · Part 4: Research Companion

© 2026 Byunghwee Lee. Created and edited in collaboration with OpenAI Codex. Prohibited from copying, reprinting, distributing, and using without prior consent.

This lecture note follows a small corpus through “center–context events → query–key learning → negative sampling → shifted PMI → exponential gravity law”.

When you first encounter Word2Vec, you see the results. “cat” and “dog” are close, and “car” and “truck” are close. This is surprising, but it leaves you with important questions.

Who told the two words that they are “similar”?

No one told them. The model only received sentences and the event of “appearing together in the surrounding context”. This note traces how these sparse events become a map of meaning. Finally, you will be able to read the same learning again in the following equation.

$$ \text{observed interaction} \approx \text{source mass} \times \text{destination mass} \times \exp(\text{latent attraction}) $$

This equation is the point where Word2Vec and the exponential gravity law meet.

After reading this note#

• You can explain that Word2Vec learns “(center, context)” events, not just individual words.
• You can explain why each word needs both a query and a key vector, and calculate the gradient each receives by hand.
• You understand that negative sampling is not just a simple speed-up technique, but also defines a comparison baseline (noise baseline).
• You can explain why words with similar contexts have similar vectors.
• You can read the inner product of SGNS as log-odds between data and noise, and interpret the exponent as the attraction of the gravity law.
• You can distinguish the training score from cosine similarity and explain the roles of the input and output embeddings.

Table of Contents#

1. Ⅰ. Starting Point — Learning Signals from Corpus — 1–5
2. Ⅱ. Learning Dynamics of SGNS — 6–13
3. Ⅲ. From PMI to Gravity — 14–21
4. Ⅳ. How to Read the Learned Space — 22–25

Continue with Part 2: Implementing Word2Vec — From Data to Meaning Axes, which begins a new six-chapter sequence.

A Brief Preview#

flowchart TD
    A[Sentences] --> B[Generate positive pairs<br/>with a window]
    B --> C[Generate negative pairs<br/>from noise]
    C --> D[query · key]
    D --> E[Real/fake<br/>binary classification]
    E --> F[Accumulate gradients<br/>in shared vectors]
    F --> G[Similar contexts<br/>→ similar directions]
    G --> H[Semantic structure]
    H --> L[Continue in Part 2:<br/>relations and meaning axes]
    E --> I[Optimal score<br/>= log data/noise]
    I --> J[Exponentiate]
    J --> K[exponential<br/>gravity law]

In this flow, each arrow represents a problem solved by the previous step. In particular, “negative sampling → noise distribution → gravity law” is not a disjointed appendix, but a continuous question.

To say “Is this interaction strong?”, you must first define “compared to what?”.

Ⅰ. Starting Point — Learning Signals from Corpus#

↑ Table of Contents

1. The “Corpus Village” to Follow#

This note will not frequently change the examples. Let’s follow this very small Corpus Village until the end.

the cat likes pet food
the dog likes pet food
the cat enjoys warm home
the dog enjoys warm home

the car follows road traffic
the truck follows road traffic
the car carries heavy load
the truck carries heavy load

We can already see two neighborhoods.

However, the model does not know that cat is an animal, nor that car is a vehicle. The model will only see the following events:

(cat, pet)    actually observed
(cat, home)   actually observed
(car, road)   actually observed
(cat, road)   unobserved or generated as noise

The subsequent positive pair is (cat, pet), and the negative pair to compare is (cat, road).

30-Second Key Takeaways#

Skip-gram with Negative Sampling (SGNS) assigns each word two types of vectors.

• input embedding: used when the word serves as the center: query vector $q_w$
• output embedding: used when answering with a candidate: key vector $k_c$

If cat asks and pet is considered a candidate answer, their scores are:

$$ s(\text{cat},\text{pet}) = q_{\text{cat}}^\top k_{\text{pet}} $$

This score is calculated as:

$$ (\text{cat},\text{pet})\text{ is a real pair} \quad\Rightarrow\quad q_{\text{cat}}^\top k_{\text{pet}}\uparrow $$

$$ (\text{cat},\text{road})\text{ is a negative pair} \quad\Rightarrow\quad q_{\text{cat}}^\top k_{\text{road}}\downarrow $$

So far, we have “attraction and repulsion”. However, the really important thing is that many pairs share the same vector. For example, cat and dog both need to have high scores on pet, food, and home, so it is more efficient for their queries to take similar directions. Semantic similarity is not given as the correct answer, but rather as a result of repeatedly solving the same prediction problem.

In this document, “query-key” is a modern interpretation of the term. In the original Word2Vec literature and implementations, it is usually referred to as “input/output embedding”. Although the query/key in the Transformer is similar in shape, the vectors in Word2Vec are static parameters attached to the word ID, and are generated differently depending on the context.

2. Skip-gram learns “encounters”, not sentences#

Suppose the window size is 2 in “the cat likes pet food.” If the center word is cat, then the context within two positions is:

the, likes, pet

Therefore, the positive pair is:

(cat, the)
(cat, likes)
(cat, pet)

By moving the center by one position in the same sentence and repeating it across the entire corpus, we get a large multiset of (center, context) pairs. If a pair appears 10 times, it is not just one learning event, but 10 learning events. The number of repetitions is the signal.

raw sentences
    ↓ sliding window
(center, context, count)
    ↓
sparse word-context relation matrix

This completes the first causal relationship.

Sentences do not directly provide meaning. The window defines the “relationship”, and the repeated relationship is the learning signal.

When the window size is small, it captures more local grammatical relationships, while when it’s large, it captures more words related to the same topic. Therefore, the embedding is a product of the corpus, not just the corpus, but also the “observation device” of corpus + window.

Practice: How the window affects relationships#

Consider the sentence “the dog enjoys warm home”. If “enjoys” is the center and the window size is 1 and 2, what contexts would be created? How would the relationships learned differ in the two settings?

3. Why does Word2Vec have two vectors?#

Let $V$ be the vocabulary size and $d$ be the embedding dimension.

There are two embedding matrices.

$$ Q \in \mathbb{R}^{V\times d} $$

$$ K \in \mathbb{R}^{V\times d} $$

Q[w] is the vector used when the word $w$ appears as the center.

$$ q_w = Q[w] $$

K[c] is the vector used when the word $c$ appears as the context.

$$ k_c = K[c] $$

Therefore, the score in Word2Vec is

$$ s(w,c)=q_w^\top k_c $$

.

This is very similar to the attention mechanism in Transformer, where

$$ q_i^\top k_j $$

is calculated.

The difference is that in Transformer, the query/key are dynamically generated based on the input, while in Word2Vec, there is a static query/key vector that is learned for each token in the vocabulary.

4. Word2Vec from a Query-Key Perspective#

It is very easy to understand Word2Vec if we consider it as follows:

“Given a center word $w$, how compatible is the context candidate $c$?”

The center word acts as a query.

$$ q_w $$

The context candidate acts as a key.

$$ k_c $$

compatibility:

$$ q_w^\top k_c $$

In the corpus village,

center = cat

we ask this question to the candidate key,

pet
home
road
traffic

let’s say it is

If the learning is successful,

$$ q_{\text{cat}}^\top k_{\text{pet}} $$

$$ q_{\text{cat}}^\top k_{\text{home}} $$

and

$$ q_{\text{cat}}^\top k_{\text{road}} $$

are large,

then Word2Vec is a learned retrieval system that searches for a context key that is suitable for the query “cat”.

5. The initial vectors have no meaning#

Initially, the embeddings are randomly initialized.

q_cat = [ 0.01, -0.03,  0.02 ]
k_dog = [-0.02,  0.04,  0.01]

For example:

Initially, there is no semantic structure at all.

As the training process repeats the observed co-occurrence, a certain direction emerges.

cat → likes, pet, food, enjoys, warm, home
dog → likes, pet, food, enjoys, warm, home

The key is that if the same relationships are repeated, “cat” and “dog” should have high dot products with similar context keys.

Therefore, the two query vectors naturally align in a similar direction.

In other words, similarity is not directly enforced, but rather

it is indirectly created because it is necessary to predict similar contexts.

Ⅱ. Learning Dynamics of SGNS#

↑ Table of Contents

6. Full Softmax Skip-gram#

The original Skip-gram model has the following probability model:

The probability of the context $c$ given the center word $w$:

$$ P(c\mid w) = \frac{\exp(q_w^\top k_c)} {\sum_{c'=1}^{V}\exp(q_w^\top k_{c'})} $$

In other words, all vocabulary candidates and scores are calculated.

The log likelihood is

$$ \log P(c\mid w) = q_w^\top k_c - \log \sum_{c'}\exp(q_w^\top k_{c'}) $$

.

The problem is that if the vocabulary is 100,000, we need to calculate 100,000 scores for each positive pair.

Therefore, it is expensive.

7. The idea of Negative Sampling#

Negative Sampling changes the problem.

Existing Problem:

Which context among all words is the correct answer?

Negative Sampling:

Is this (center, context) pair a genuine pair from real data, or a fabricated pair from noise?

In other words, it converts multiclass classification into binary classification.

positive example:

(cat, pet) -> y = 1

negative example:

(cat, road) -> y = 0
(cat, traffic) -> y = 0
(cat, load) -> y = 0

score:

$$ s=q_w^\top k_c $$

probability:

$$ P(D=1\mid w,c)=\sigma(q_w^\top k_c) $$

Here,

$$ \sigma(x)=\frac{1}{1+e^{-x}} $$

is.

Here, “negative” is not the correct answer defined as “words that have no semantic relationship in the world.” It is simply a contrast set drawn from a predefined noise distribution. It is possible that a real positive is also drawn as a negative. In large corpora, probabilistic learning signals generally work, and positive tokens can be rejected depending on the implementation.

8. SGNS objective#

Given a positive pair $(w, c)$, draw the negative contexts

$$ c_1^-,\dots,c_{n_{\mathrm{neg}}}^-. $$

The objective to maximize is:

$$ \log\sigma(q_w^\top k_c) + \sum_{i=1}^{n_{\mathrm{neg}}} \log\sigma(-q_w^\top k_{c_i^-}) $$

In terms of loss, it is:

$$ L = -\log\sigma(q_w^\top k_c) - \sum_i \log\sigma(-q_w^\top k_{n_i}) $$

9. What exactly does the gradient from a positive pair do?#

Score for positive pair:

$$ s=q_w^\top k_c $$

loss:

$$ L_+=-\log\sigma(s) $$

Taking the derivative, we get

$$ \frac{\partial L_+}{\partial s} = \sigma(s)-1 $$

and

$$ \frac{\partial s}{\partial q_w}=k_c $$

therefore

$$ \frac{\partial L_+}{\partial q_w} = (\sigma(s)-1)k_c $$

gradient descent results in

$$ q_w \leftarrow q_w - \eta(\sigma(s)-1)k_c $$

i.e.

$$ q_w \leftarrow q_w + \eta(1-\sigma(s))k_c $$

this.

Similarly,

$$ k_c \leftarrow k_c + \eta(1-\sigma(s))q_w $$

this holds.

Therefore, the two vectors in a positive pair move in opposite directions.

10. What does the gradient do for negative pairs?#

For a negative sample $n$

$$ L_-=-\log\sigma(-q_w^\top k_n) $$

we should.

$$ s_n=q_w^\top k_n $$

if

$$ \frac{\partial L_-}{\partial s_n} = \sigma(s_n) $$

therefore

$$ \frac{\partial L_-}{\partial q_w} = \sigma(s_n)k_n $$

gradient descent:

$$ q_w \leftarrow q_w - \eta\sigma(s_n)k_n $$

and

$$ k_n \leftarrow k_n - \eta\sigma(s_n)q_w $$

that is, a negative pair moves in a direction that reduces the dot product of the two vectors.

11. Important Intuition: Gradients are not always the same size#

in a positive pair,

$$ 1-\sigma(s) $$

is the gradient strength.

if $s$ is already very large,

$$ \sigma(s)\approx1 $$

then there is almost no update.

that is, if the model already

cat → pet

knows it well, it does not make large changes.

Conversely, if it is positive but $s \ll 0$,

$$ 1-\sigma(s)\approx1 $$

then it is strongly pulled.

The same is true for negative pairs.

If the dot product is already very low in the negative case,

$$ \sigma(s)\approx0 $$

then there is almost no update.

Ultimately, SGNS modifies based on the pair that is being poorly evaluated.

12. A Typical Training Step#

For example,

center = cat
positive context = pet

negative contexts:
road
traffic
load

let’s say.

forward:

q = Q["cat"]

k_pos = K["pet"]

k_neg = [
    K["road"],
    K["traffic"],
    K["load"]
]

positive score:

s_pos = dot(q, k_pos)

negative scores:

s_neg = k_neg @ q

loss:

loss_pos = -log(sigmoid(s_pos))
loss_neg = -sum(log(sigmoid(-s_neg)))

loss = loss_pos + loss_neg

after backpropagation,

q_cat      → toward k_pet
k_pet      → toward q_cat

q_cat      → away from k_road
q_cat      → away from k_traffic
q_cat      → away from k_load

moves.

Let’s look at a single negative example for illustration. Let’s assume the learning rate is $\eta = 0.1$.

$$ q_{cat}=[0.4,-0.2],\qquad k_{pet}=[0.1,0.5],\qquad k_{road}=[-0.3,0.2] $$

Before updating, the scores and magnitudes are as follows:

When combining the two gradients for the same forward state, we obtain approximately

$$ q_{cat}^{new} \approx [0.4,-0.2] +0.1(0.515)[0.1,0.5] -0.1(0.460)[-0.3,0.2] =[0.419,-0.183] $$

something like this. Focus on the direction, not the absolute values of the numbers. Add the key directional component of pet and subtract the key directional component of road. Simultaneously, $k_{pet}$ and $k_{road}$ are also adjusted to match their opposite roles. Repeat this process millions of times, but each time, the same parameters are exposed to the requirements of multiple relationships.

A positive pair does not necessarily need a positive score. Showing $n_{\mathrm{neg}}$ negative pairs shifts the optimal baseline by $n_{\mathrm{neg}}$. What matters is whether a relevant pair scores higher than an unrelated pair and whether the score reflects the data-to-noise ratio. The $-\log n_{\mathrm{neg}}$ term makes this precise.

A quick exercise: Which pair will move the most?#

The model incorrectly evaluates the positive pair as $s_+=-3$ and the negative pair as $s_-=3$. What are the respective scalar strengths of the $1-\sigma(s_+)$ and $\sigma(s_-)$? What happens if the signs of the scores are reversed?

13. How do vectors create meaning?#

This is the most important part.

For example, suppose we have two words.

cat
dog

They share the following contexts in the corpus:

pet
food
likes
home
warm
enjoys

Therefore, during learning,

$$ q_{cat}^\top k_{pet}\uparrow $$

$$ q_{cat}^\top k_{home}\uparrow $$

and

$$ q_{dog}^\top k_{pet}\uparrow $$

$$ q_{dog}^\top k_{home}\uparrow $$

we get

Therefore, the query vectors $q_{cat}$ and $q_{dog}$ should have similar projections for multiple keys.

In other words,

$$ QK^\top $$

the two rows of the matrix become similar.

This means that the two rows of $QK^\top$ become similar. This is a natural compression result that occurs when $K$ can represent the relationship structure and the low-dimensional optimization is well-suited, but it is not an equation that is guaranteed in all cases. If you use this intuition, you get

$$ q_{cat}\approx q_{dog} $$

something like this.

Conversely, cat and car have different sets of keys that require high scores. cat is biased towards pet/home, while car is biased towards road/load. This shared context creates a common pressure that builds two neighborhoods in the corpus village.

Ⅲ. From PMI to Gravity#

↑ Table of Contents

14. Word2Vec factors a huge relationship matrix#

Collect every word-context score into a matrix $S$, whose $(w,c)$ entry is

$$ S_{wc}=s_{wc}=q_w^\top k_c. $$

If the word query vectors form the rows of $Q$ and the context key vectors form the rows of $K$, then

$$ Q\in\mathbb{R}^{V\times d}, \qquad K\in\mathbb{R}^{V\times d}, $$

and the complete score matrix is

$$ S=QK^\top. $$

Word2Vec does not construct $S$ as a free $V\times V$ parameter table. Instead, it represents the table through the much smaller matrices $Q$ and $K$, usually with $d\ll V$. Because both factors have only $d$ columns,

$$ \operatorname{rank}(S)\le d. $$

Thus, the score matrix is a low-rank representation of the full word-context relationship structure. Throughout this chapter, uppercase $K$ denotes the context key matrix; we will use $n_{\mathrm{neg}}$ for the number of negative samples per positive pair.

15. Connecting SGNS and PMI: From one pair to the matrix#

Levy & Goldberg’s important interpretation is that SGNS implicitly performs shifted PMI matrix factorization. We will derive the shift before returning to the matrix-level constraint.

Step 1: Write the objective for one pair#

Let $n_{\mathrm{neg}}$ be the number of negative samples drawn for each observed positive pair. For a fixed word-context pair $(w,c)$, suppose temporarily that its scalar score $s_{wc}$ can be optimized independently. Its expected contribution to the SGNS objective is

$$ \ell_{wc} = P_{\text{data}}(w,c)\log\sigma(s_{wc}) + n_{\mathrm{neg}}P_{\text{data}}(w)P_n(c)\log\sigma(-s_{wc}). $$

The factor $n_{\mathrm{neg}}$ appears because SGNS creates $n_{\mathrm{neg}}$ noise pairs for every positive pair.

Step 2: Solve for the pairwise optimum#

Setting the derivative with respect to $s_{wc}$ to zero gives

$$ P_{\text{data}}(w,c)\sigma(-s_{wc}) = n_{\mathrm{neg}}P_{\text{data}}(w)P_n(c)\sigma(s_{wc}). $$

Using $\sigma(s)/\sigma(-s)=e^s$, we obtain

$$ e^{s_{wc}^*} = \frac{P_{\text{data}}(w,c)} {n_{\mathrm{neg}}P_{\text{data}}(w)P_n(c)}, $$

and therefore

$$ \boxed{ s_{wc}^* = \log\frac{P_{\text{data}}(c\mid w)}{P_n(c)} -\log n_{\mathrm{neg}} }. $$

The $-\log n_{\mathrm{neg}}$ term is therefore not an extra assumption. It is the log-odds correction produced by comparing each positive pair with $n_{\mathrm{neg}}$ negative pairs.

Step 3: Recover shifted PMI as a special case#

If the noise distribution equals the actual context unigram distribution,

$$ P_n(c)=P(c), $$

then

$$ PMI(w,c) = \log\frac{P(w,c)}{P(w)P(c)} = \log\frac{P(c\mid w)}{P(c)}. $$

Substituting this into the pairwise optimum gives

$$ \boxed{ s_{wc}^*=PMI(w,c)-\log n_{\mathrm{neg}} }. $$

This clean shifted-PMI formula is exact only for unigram noise. With the practical choice $P_n(c)\propto P(c)^{3/4}$, an additional context-frequency correction remains, as we will derive below.

Step 4: Return to the embedding constraint#

The derivation above treats every $s_{wc}$ as an independent scalar. Actual embeddings cannot choose the scores independently because Section 13 showed that the entire matrix must have the form

$$ S=QK^\top, \qquad \operatorname{rank}(S)\le d. $$

If the matrix of pairwise optima $S^*$ has rank greater than $d$, no $d$-dimensional $Q$ and $K$ can reproduce all of its entries exactly. SGNS therefore learns the factors jointly and finds the best low-rank compromise under its weighted logistic objective.

The dot product is thus more than a generic similarity score. At the pairwise optimum, it represents the log ratio between the observed conditional context frequency and the noise baseline, shifted by the number of negative samples.

16. Using PMI to count#

Let $N(w,c)$ be the observed count of the word-context pair $(w,c)$. Its marginal counts are

$$ N(w)=\sum_c N(w,c) $$

$$ N(c)=\sum_w N(w,c) $$

Let $N$ be the total number of pairs. Then

$$ P(w,c)=\frac{N(w,c)}{N} $$

$$ P(w)=\frac{N(w)}{N} $$

$$ P(c)=\frac{N(c)}{N} $$

Therefore,

$$ PMI(w,c) = \log \frac{N(w,c)N} {N(w)N(c)}. $$

Under the unigram-noise assumption from Section 14, the idealized SGNS score is therefore

$$ q_w^\top k_c \approx \log \frac{N(w,c)N} {N(w)N(c)} - \log n_{\mathrm{neg}}. $$

17. Noise distribution: “Compared to what?”#

The key point is that the noise distribution $P_n(c)$ defines the baseline against which the model evaluates the data.

• Uniform noise: Assume that all words will appear equally often.
• Unigram noise: Assume that words that appear frequently in the corpus will also appear frequently in the noise.
• 3/4-smoothed unigram: Show frequent words frequently, but do not overwhelm them with the raw frequency.

The choice widely used in Word2Vec is

$$ P_n(c) = \frac{f(c)^{3/4}} {\sum_u f(u)^{3/4}} $$

To see how the $3/4$ power softens the head of the distribution, consider these hypothetical frequencies.

The ratio of the:pet is 100:1 in raw unigram, but after 3/4 smoothing, it is approximately 31.6:1. This reduces the dominance of frequent words like the while also preventing rare words from being over-represented.

The 3/4 value is not just a mathematical constraint, but also a practical choice that has worked well in terms of balancing calculation efficiency and quality. The noise that is more appropriate may depend on the corpus and the purpose.

Now, let’s ask the key question in the corpus village.

• The word “the” appears frequently around “cat.” However, since it is common, this is not surprising.
• The word “pet” appears less frequently overall than “the,” but it appears more frequently when “cat” is present.
• The word “road” is a normal word in itself, but it appears less frequently in the context of “cat.”

Therefore, we need to compare the observed frequency with the baseline frequency, rather than the raw count.

$$ \frac{P_{\text{data}}(c\mid w)}{P_n(c)} $$

This is the transition from negative sampling to the gravity law.

18. The inner product of SGNS is the data-vs-noise log-odds#

Positive pairs come from the actual pair distribution.

$$ (w,c)\sim P_{\text{data}}(w,c) $$

If we create $n_{\mathrm{neg}}$ negative pairs for each positive pair, then the noise pairs come in the following way:

$$ w\sim P_{\text{data}}(w), \qquad c\sim P_n(c) $$

Therefore, the relative frequency of the same $(w,c)$ in both worlds is:

The log-odds of the optimal Bayes classifier is:

$$ \begin{aligned} \log\frac{P(D=1\mid w,c)}{P(D=0\mid w,c)} &= \log\frac{P_{\text{data}}(w,c)} {n_{\mathrm{neg}} P_{\text{data}}(w)P_n(c)}\\ &= \log\frac{P_{\text{data}}(c\mid w)} {n_{\mathrm{neg}} P_n(c)}. \end{aligned} $$

SGNS attempts to represent this log-odds as a single inner product.

$$ \boxed{ q_w^\top k_c \approx \log\frac{P_{\text{data}}(c\mid w)}{n_{\mathrm{neg}} P_n(c)} } $$

It is important to read this equation aloud.

The larger $q_{cat}^\top k_{pet}$ is, the more often “pet” appears near “cat” in the observed data than the noise baseline would predict. If “road” is frequently drawn from the noise distribution but rarely observed near “cat,” then $q_{cat}^\top k_{road}$ will be smaller.

Strictly speaking, negative sampling is not exactly the same as the standard NCE, which aims to directly recover the normalized language model probability. However, the “noise-contrastive view” of “learning the log density ratio by classifying data and noise” is accurate and useful for understanding the above equation.

Remember these two points.

1. If each $s_{wc}$ can be treated as an independent parameter, then the optimal equation can be matched for each pair.
2. In reality, there is a rank-$d$ constraint $s_{wc} = q_w^\top k_c$, so all pairs share a limited vector and approximate the entire log-odds matrix.

It is precisely this constraint that leads to structural compression, or embedding, rather than simple memorization.

19. Taking the logarithm results in the exponential gravity law#

Now, we don’t need to artificially introduce a new model. We can simply exponentiate the equation we just obtained.

$$ q_w^\top k_c \approx \log\frac{P_{\text{data}}(w,c)} {n_{\mathrm{neg}} P_{\text{data}}(w)P_n(c)} $$

Taking the exponential of both sides and simplifying, we get:

$$ \boxed{ P_{\text{data}}(w,c) \approx n_{\mathrm{neg}}\, P_{\text{data}}(w) P_n(c) \exp(q_w^\top k_c) } $$

We can absorb the global constant and the normalization difference into $C$, which allows us to use the same notation for count or flow.

$$ F_{wc} \approx C\,m_w m_c\exp(q_w^\top k_c), \qquad m_w=P_{\text{data}}(w),\quad m_c=P_n(c) $$

These three components represent distinct roles.

In other words, the embedding does not simply memorize the raw co-occurrence, but compresses the residual attraction that cannot be explained by the basic activity levels of the two endpoints into a low-dimensional space.

Unigram noise creates the most familiar gravity model#

If $P_n(c) = P_{\text{data}}(c)$, then

$$ P_{\text{data}}(w,c) \propto P(w)P(c)\exp(q_w^\top k_c) $$

we have

$$ q_w^\top k_c \approx PMI(w,c)-\log n_{\mathrm{neg}} $$

where the inner product represents the unigram probability of each word, which plays the role of two masses.

With $3/4$ noise, the destination mass changes#

If the practical noise distribution is

$$ P_n(c)=\frac{P(c)^{3/4}}{Z_{3/4}}, \qquad Z_{3/4}=\sum_u P(u)^{3/4}, $$

then $Z_{3/4}$ is the normalization factor that ensures $\sum_c P_n(c)=1$. Under this noise distribution,

$$ P_{\text{data}}(w,c) \propto P(w)P(c)^{3/4}\exp(q_w^\top k_c) $$

The same content can be represented by a score.

$$ q_w^\top k_c \approx PMI(w,c)-\log n_{\mathrm{neg}} +\frac14\log P(c)+\log Z_{3/4}. $$

Therefore, the statement “SGNS always learns $PMI-\log n_{\mathrm{neg}}$” is incomplete. It is exact in the special case of unigram noise; with $3/4$ noise, a context-frequency correction term remains.

Practice: Converting log-odds to flow#

Let $P(w) = 0.1$, $P_n(c) = 0.02$, $n_{\mathrm{neg}} = 5$, and $q_w^\top k_c = \log 3$. What is the gravity term that predicts $P(w, c)$ for pairwise optimum? What does the “3” represent here?

20. Why is it called “gravity”?#

The classic gravity model for spatial interaction is

$$ F_{ij} = G\frac{m_i m_j}{r_{ij}^{\gamma}} $$

like this. The more cities there are, the more the movement increases, and the farther apart they are, the less the movement decreases. Taking the logarithm gives

$$ \log F_{ij} = \log G+\log m_i+\log m_j-\gamma\log r_{ij} $$

This is a structure similar to the one obtained in Word2Vec.

$$ \log F_{wc} \approx \log C+\log m_w+\log m_c+q_w^\top k_c $$

In particular, the general exponential gravity model

$$ F_{ij}\propto m_i m_j e^{-\beta d_{ij}} $$

compared to SGNS, the affinity $q_i^\top k_j$ learned in place of $-\beta d_{ij}$ is used.

$$ -\beta d_{ij} \quad\longleftrightarrow\quad q_i^\top k_j $$

However, the dot product itself should not be called a physical distance. Word2Vec does not receive distance as input. The data learns which latent attraction best explains the flow. Therefore, a more accurate representation is as follows:

The traditional gravity model assumes that observed distances explain the interaction. SGNS discovers latent affinity that explains the interaction in the query-key geometry.

21. The end of the corpus village: From frequency to meaning#

Now, let’s look at the three contexts surrounding “cat” again.

If the difference in scores between the two contexts is exponentiated, $n_{\mathrm{neg}}$ disappears, making it more intuitive.

$$ \exp\left( q_{cat}^\top k_{pet} -q_{cat}^\top k_{road} \right) \approx \frac{P(pet\mid cat)/P_n(pet)} {P(road\mid cat)/P_n(road)}. $$

The left side represents the relative attraction in the embedding space, and the right side represents the relative surprise in the corpus. Learning adjusts $Q$ and $K$ to make these two worlds compatible.

And “dog” also has a similar relative surprise profile when considering almost the same contexts.

$$ q_{cat}^\top K^\top \approx q_{dog}^\top K^\top $$

Therefore, within the low-dimensional compression, $q_{cat}$ and $q_{dog}$ find similar directions. On the other hand, “car” and “truck” create profiles that resemble each other on the side of “road/traffic/load”. Finally, the two villages seen by humans also appear in the vector space.

co-occurrence count
    ↓ remove the portion explained by endpoint frequency using a noise baseline
data / noise ratio
    ↓ log
query · key
    ↓ joint low-dimensional approximation
semantic structure
    ↓ exp
mass × mass × latent attraction

This is the one-liner story of this note.

Ⅳ. Reading the Learned Space#

↑ Table of Contents

22. Dot product is not distance#

Important point to note here.

The score of Word2Vec is

$$ q_i^\top k_j $$

not

$$ -\|q_i-k_j\| $$

trivial.

However,

$$ \|q-k\|^2 = \|q\|^2+\|k\|^2-2q^\top k $$

if the norm is somewhat similar, then a larger dot product results in a smaller Euclidean distance.

Similarly, cosine similarity is

$$ \cos(q,k) = \frac{q^\top k} {\|q\|\|k\|} $$

defined as.

Therefore, while the learning process directly adjusts the dot product, it also gives meaning to the cosine geometry.

23. Why is only the input embedding used during inference?#

During training,

Q = input embeddings
K = output embeddings

there are two matrices.

However, in downstream similarity tasks, we often use

Q[word]

or methods like

or

(Q[word] + K[word]) / 2

to combine them.

This is because Q represents the representation of a word when it appears as the center, and it often aligns well with the semantic embedding that we typically want.

However, theoretically, Q and K have different roles.

24. Asymmetry between Query Vector and Key Vector#

In Word2Vec, $q_w$ and $k_w$ are not the same parameters.

For example, the gradient received when using the word “bank” as a center and when using it as context can be different.

Therefore, generally,

$$ q_w \neq k_w $$

is true.

This also relates to directed graph embedding.

For example,

$$ i\rightarrow j $$

In the case of a directed relation, it is natural to keep the source embedding and target embedding separate.

Word2Vec is essentially a directed prediction problem.

center → context

25. Comparison with Transformer attention#

Word2Vec:

$$ q_w^\top k_c $$

Transformer:

$$ q_i^\top k_j $$

Both are compatibility scores.

However, there is a difference.

Word2Vec#

word id
↓
embedding lookup
↓
q_w / k_c
↓
dot product

The vectors are static.

Transformer#

token representation h_i
↓
W_Q h_i
↓
q_i

token representation h_j
↓
W_K h_j
↓
k_j

The vectors are context-dependent.

In other words, Transformer can be seen as a much more dynamic extension of Word2Vec’s query-key matching idea.

Continue to Part 2: Implementing Word2Vec — From Data to Meaning Axes.