03. Embedding Applications — Reading Culture Through Semantic Space#

Word2Vec Application: Bias Measurement → Meaning Changes Over Time → SemAxis → Verification with External Reality

All About Embedding · Part 1: Word2Vec, SGNS, and Gravity · Part 2: Implementation, Relations, and Meaning Axes · Part 3: Culture and Semantic Change · Part 4: Research Companion

© 2026 Byunghwee Lee. Created and edited in collaboration with OpenAI Codex. Unauthorized copying, reproduction, distribution, and use are prohibited.

In Parts 1 and 2, we saw how center–context events become vectors and how repeated relationships become directions and meaning axes. Now, let’s take that geometry into real-world research questions.

As before, let’s take a single example.

Question from the Archival Research Team: How have engineer and nurse been associated with gender from 1900 to today? Where does “leader” lie on the “masculine” axis, but also on the axes of “older–younger,” “strong–weak,” and “public–private”? If these positions have changed, has society changed, or has the corpus changed, or has the measurement been misapplied?

Following these questions, Word2Vec becomes a tool for measuring cultural traces, rather than just a word-guessing game. However, it is crucial to distinguish between what the tool measures and what the researcher wants to claim.

After reading this note, you will be able to:#

• Clearly define what it means to “discover bias” in an embedding.
• Explain which semantic axis, WEAT, or SemAxis is appropriate for a given question.
• Understand the difference between Google Ngram frequency graphs and diachronic embeddings over time.
• Understand why and how to align separate, time-specific spaces.
• Differentiate and measure the changes in meaning of words and the associated changes in social group associations.
• Read and interpret the 2025 Nature study on age × gender bias and how it places embeddings within a larger validation chain.
• Connect the results to subsequent validation steps without overstating “society is inherently biased” or “the model is biased.”

Table of Contents#

1. Ⅰ. Defining the Problem and Selecting Measurement Tools — 1–4
2. Ⅱ. Tracking Meaning Changes Over Time — 5–6
3. Ⅲ. Multidimensional Meaning, Framing, and Intersectionality — 7–9

One-Page Research Roadmap#

flowchart TD
    A[Select a corpus] --> B[Learn word-context relations]
    B --> C[Axis- or set-based measurement]
    C --> D{One point in time?}
    D -->|Yes| E[WEAT · SemAxis]
    D -->|No| F[Train by period]
    F --> G[Align spaces]
    G --> H[Displacement · trajectory]
    E --> I[Uncertainty · alternative axes]
    H --> I
    I --> J[Validate with population data,<br/>surveys, experiments, and other evidence]
    J --> K[Set the scope of the claim]

The key is the final arrow. The embedding number is not a conclusion, but a link in a chain of evidence.

Ⅰ. Defining the Problem and Selecting Measurement Tools#

↑ Table of Contents

1. Let’s divide the term “bias” into four layers#

A common mistake in embedding research is to treat different sentences as the same.

Therefore, a safe first sentence is:

“In this corpus, an asymmetric association was observed between this set of words and this set of attributes.”

This is a much larger claim: “Society as a whole holds biases.” External data, alternative explanations, and experiments are needed to fill this gap.

Our repeated example#

The archive team first breaks down the questions.

1. What is the gender association between “engineer” and “nurse” in each era’s text?
2. Does this association move with the actual gender ratio of each profession?
3. Do the associations between profession, power, income, and age overlap with the gender association?
4. Does the text association influence people’s or models’ judgments?

The first question can be answered by the embedding. The second requires other data.

Quick Exercise 0 We observed that “nurse” is closer to the female pole. Can we conclude that “women are better suited for nursing” based on this alone?

2. Asking in one direction: meaning axis#

As seen in Part 1, individual coordinates in an embedding do not have fixed meanings. Rotating the entire space does not change inner products or cosine similarities. What is interpretable is not a coordinate number, but the relationships and directions between words.

2.1 Two poles form a ruler#

To measure gender association, using only one pair such as man - woman makes the axis overly sensitive to idiosyncratic usage. A more stable approach uses the centroids of several reference words.

$$ \mu_{M} = \frac{1}{|M|}\sum_{m\in M}\widehat{\mathbf v}_m, \qquad \mu_{F} = \frac{1}{|F|}\sum_{f\in F}\widehat{\mathbf v}_f $$

$$ \mathbf g = \frac{\mu_M-\mu_F}{\|\mu_M-\mu_F\|} $$

Here, M = {he, him, man, male, ...} and F = {she, her, woman, female, ...}. The hat indicates that each vector is normalized to unit length first. The researcher chooses the sign of the axis; here, positive values point toward the male pole.

The score for the word “w” is:

$$ \operatorname{axis}(w;\mathbf g) = \cos(\mathbf v_w,\mathbf g) $$

If the score for “engineer” is 0.31 and the score for “nurse” is -0.24, then in this space, the former points towards the male pole, and the latter points towards the female pole.

2.2 The score is a relationship profile, not identity#

This score is not a declaration that “engineer” means “male”.

• Model: What algorithm and dimension were used?
• Data: What corpus of language from what era, genre, and language was used?
• Preprocessing: How were case, lemmatization, and rare word removal handled?
• Axes: Which pole words were included and excluded?

This is a conditional relational measure.

A small example to illustrate#

In a 2D toy space where the gender axis is $\mathbf g=(1,0)$,

$$ \widehat{\mathbf v}_{engineer}=(0.6,0.8), \qquad \widehat{\mathbf v}_{nurse}=(-0.3,\sqrt{0.91}) $$

then the axis scores are 0.6 and -0.3, respectively. No matter how large the second coordinate is, this question uses only the projection onto the first direction. An axis compresses the full space into a number for one specific question.

Quick Exercise 1 If the direction of the axis is changed from $\mathbf g$ to $-\mathbf g$, will the practical conclusion of the paper also change?

3. Asking with Four Sets Instead of One Word: WEAT#

While axis projection is intuitive, the Word Embedding Association Test (WEAT) is convenient for testing “how differently two sets of objects are associated with two sets of attributes?”

Caliskan, Bryson, Narayanan (Science, 2017) applied a structure similar to the Implicit Association Test to word vectors. The main analysis used Common Crawl GloVe, and the results were also replicated using Google News Word2Vec.

3.1 Four Baskets#

If we convert our example to WEAT, it would look like this.

The relative association of a single word is

$$ s(w,A,B) = \frac{1}{|A|}\sum_{a\in A}\cos(\mathbf v_w,\mathbf v_a) - \frac{1}{|B|}\sum_{b\in B}\cos(\mathbf v_w,\mathbf v_b) $$

measured as follows. If the value is positive, it indicates that $A$ is more associated with the word; if it is negative, it indicates that $B$ is more associated with the word.

The difference between the two target sets can be viewed as

$$ S(X,Y,A,B) = \sum_{x\in X}s(x,A,B) - \sum_{y\in Y}s(y,A,B) $$

the standardized effect size $d$, which considers both the set size and the variance. This is often reported together.

$$ d = \frac{ \operatorname{mean}_{x\in X}s(x,A,B) - \operatorname{mean}_{y\in Y}s(y,A,B) }{ \operatorname{std}_{w\in X\cup Y}s(w,A,B) } $$

3.2 What the permutation test asks#

Under the null hypothesis, even if the words in $X \cup Y$ are divided into two sets of equal size, we should expect to see a similar degree of association. By observing the most extreme proportion of $S$ across multiple possible or random partitions, we obtain the permutation $p$-value.

This test does not mean “the probability that a person has a bias.”

Given this word set partition, how unusual is the observed association difference compared to the difference obtained when mixing the labels?

3.3 axis and WEAT are not competitors#

Practice 2 Given $s(engineer,A,B)=0.20$, $s(nurse,A,B)=-0.35$, what is the difference and what does it mean?

4. What a famous bias study actually did#

4.1 “Vectors carry social associations”#

Caliskan et al. found association patterns resembling those in human studies across several contrasts: flowers–insects and pleasant–unpleasant words, European American–African American names, and male–female terms with career–family words. Their key contribution was not the observation that a few words looked unusual, but a measurement procedure built from word sets, effect sizes, and permutation tests.

However, the authors also emphasize that the “subjects” of embeddings are words and corpora, not people. The IAT and WEAT scores should not be directly mapped one-to-one.

4.2 “Can removing the axis solve it?”#

Bolukbasi et al. (NeurIPS, 2016) proposed a hard debiasing method in Google News Word2Vec to find and remove the gender bias component, by removing the axis.

For the gender axis $\mathbf g$, the simplest neutralization is

$$ \mathbf v_{\perp} = \mathbf v-(\mathbf v^\top\mathbf g)\mathbf g $$

to remove the projection of the vector onto the gender axis. Words like “mother–father” which contain gender as part of their meaning, and neutral words like “doctor” should be treated differently.

However, Gonen & Goldberg (NAACL, 2019) showed that even when direct gender-axis scores decrease, words that were originally gender-associated can still cluster together and remain predictable to a classifier.

flowchart TD
    A[Bias structure in the original space] --> B[Remove one gender-axis component]
    B --> C[Direct projection becomes smaller]
    B --> D[Neighbor and cluster structure may remain]
    C --> E{What was resolved?}
    D --> E
    E --> F[One metric decreasing ≠ all bias removed]

The principle to learn from this conversation is simple.

Optimizing a metric and removing the phenomenon are not the same thing.

Argument Ladder#

Practice 3
After debiasing, all gender axis scores for all professions have become close to 0. Let’s propose two follow-up analyses to test “gender information has disappeared.”

Ⅱ. Tracking Meaning Changes Over Time#

↑ Table of Contents

5. As Time Passes: N-gram Frequency and Diachronic Embedding#

When a user asks, “Does temporal change research use N-grams?”, the answer is “Yes, but it doesn’t just use N-gram Viewer.”

Hamilton, Leskovec, Jurafsky (ACL, 2016) created diachronic embeddings with PPMI, SVD, and SGNS across six historical corpora—including Google Books N-Gram and COHA—four languages, and roughly two centuries. Google N-Gram supplied massive word-context co-occurrence statistics, from which they reconstructed a semantic space for each period.

5.1 What’s the Difference?#

Frequency looks at “how often,” while embedding looks at “in what context.” It’s better to look at both, but they are not substitutes for each other.

5.2 Creating a temporal space#

flowchart TD
    A[1900s text] --> A1[W_1900]
    B[1910s text] --> B1[W_1910]
    C[1920s text] --> C1[W_1920]
    A1 --> D[Shared vocabulary · anchors]
    B1 --> D
    C1 --> D
    D --> E[Align spaces]
    E --> F[Word displacement]
    E --> G[Axis-score time series]

1. Divide the corpus into years or 10-year intervals.
2. Learn the embedding for each time point with the same settings.
3. Select common vocabulary and stable anchors.
4. Align the space.
5. Measure cosine similarity, movement, and changes in axis scores.
6. Estimate uncertainty from multiple seeds and bootstrapping.

5.3 Why align?#

Word2Vec spaces can be rotated as a whole, but the inner product between words remains the same. The 17th coordinate of the 1900 model and the 17th coordinate of the 1910 model do not necessarily have the same meaning. Simply subtracting the coordinates can measure not meaning change, but random rotation.

Let $X$ and $Y$ be the matrices of common words, then the orthogonal Procrustes alignment is

$$ R^\star = \arg\min_{R^\top R=I} \|XR-Y\|_F $$

Solve

$$ R^\star=UV^\top $$

which can be used to

After alignment, the movement of word $w$ between time points is, for example

$$ \Delta_t(w) = 1-\cos(\mathbf v_{w,t}R_t,\mathbf v_{w,t+1}) $$

measured as

5.4 Hamilton’s Two Statistical Laws#

This study observed the following trends in multiple datasets and methods.

• conformity: Words that are frequently used tend to change their meaning more slowly.
• innovation: Even when frequency is controlled, words with multiple meanings tend to change more quickly.

These are not universal laws that apply to every individual word; they are statistical regularities observed across multiple corpora. OCR errors, genre composition, and sample-size differences must also be considered.

Practice 4
The frequency of “engineer” remains the same, but the neighbors have changed from “engine, railway” to “software, data.” Can N-gram frequency alone capture this change?

6. Measuring Stereotypes Over Time#

Garg et al. (PNAS, 2018) quantified the changes in gender and ethnic associations over 100 years in American history using historical embeddings. They used diachronic embeddings from Google Books and COHA, as well as the New York Times’ yearly embeddings, and compared them with external data such as occupational demographics and past human stereotypes surveys.

Now, let’s concretize the research of the archive team.

6.1 Gender association trajectory of professions#

Create a gender axis for each era $t$,

$$ g_t(w)=\cos(\mathbf v_{w,t},\mathbf g_t) $$

Calculate the value. Then, “engineer” and “nurse” become a trajectory, not a single point.

The values in the table are illustrative and not real. In actual analysis, you should not only show the point estimates but also the seed and bootstrap confidence intervals.

6.2 Moving Words and Moving Axes#

There is a subtle trap here.

Both “engineer” and the “man/woman” pole can change.

In other words, both the object and the axis move simultaneously. At least, it is good to perform the following analyses in parallel.

1. Time-dependent axis: Create $\mathbf{g}_t$ for each era and measure the relative association within that era.
2. Fixed anchor axis: Apply a common or aligned axis to all eras.
3. Pairwise relation: Check how consistent the differences in direction between multiple gender pairs are across eras.

If the three results are similar, the interpretation is strengthened. If they are different, you need to separate and explain “movement of the target word” and “movement of the reference concept”.

6.3 Comparing with Reality, but Not Equating#

The increase in the association between “engineer” and women can coincide with the actual increase in the proportion of female engineers. However, correlation does not necessarily imply causation.

There are several possible explanations.

• The composition of the real job market has changed the text descriptions.
• The media’s portrayal has influenced educational and employment choices.
• Legal, economic, and social movements have changed both.
• The genre composition of the corpus has changed, leading to this.

The correlation between embeddings and demographics is an important external validation, but it cannot distinguish these four paths alone.

Essential Diagnostics for Time Studies#

Practice 5 The 1900 corpus has 10 million words, while the 2000 corpus has one billion. Can we interpret the more stable 2000 vector as evidence that “modern meaning is clearer”?

Ⅲ. Multidimensional Meaning, Framing, and Intersectionality#

↑ Table of Contents

7. Beyond a Single Dimension: SemAxis#

If we only consider one gender dimension, all changes appear to be related to gender. However, leader simultaneously

• young ↔ old
• weak ↔ strong
• private ↔ public
• traditional ↔ modern
• cooperative ↔ competitive

places individuals in multiple directions. SemAxis is a method of placing multiple axes in the semantic space, each corresponding to a different question.

An, Kwak, Ahn (ACL, 2018) created semantic axes from antonym poles and projected words onto each axis to describe domain-specific semantics. Based on the antonym relations in ConceptNet, they constructed 732 axes and expanded the words around each pole to reduce the instability of word pairs.

7.1 The core of SemAxis lies in these three points#

Define the positive pole set $P^+$ and the negative pole set $P^-$,

$$ \mu^+ = \frac{1}{|P^+|}\sum_{p\in P^+}\widehat{\mathbf v}_p, \qquad \mu^- = \frac{1}{|P^-|}\sum_{p\in P^-}\widehat{\mathbf v}_p $$

$$ \mathbf a = \frac{\mu^+-\mu^-}{\|\mu^+-\mu^-\|} $$

$$ \operatorname{score}(w,\mathbf a) = \cos(\mathbf v_w,\mathbf a) $$

Calculate this.

By projecting the “leader” onto hundreds of axes, it is possible to identify the “most salient dimension of meaning.” Alternatively, the researcher can pre-define the gender, age, and power axes and test hypotheses against them.

7.2 Three Strategies for “Discovering” Axes#

The healthiest approach is to separate exploration and validation. Discover interesting axes in one corpus and replicate them in a separate corpus, time period, or seed.

7.3 Conditions for Good Axes#

A good axis is not necessarily one with a plausible name.

1. Are the directions of the axis pairs consistent with each other?
2. Does the result remain consistent if one axis is removed? (Leave-one-pole-out sensitivity analysis)
3. Does it reasonably reconstruct the order of known words?
4. Is it similar in irrelevant seed and corpus?
5. Does it only predict frequency and word length?
6. Does it distort the axis by the ambiguity of the two poles?

The Archive Team’s Meaning Prism#

The team places “leader,” “engineer,” and “nurse” on three axes.

This is also a simulated result. The important point is that we have expanded “bias” to a “cross-referential semantic structure” rather than treating a single gender score as “the whole of bias.”

Exercise 6 leader is close to “old.” How can we tell whether this indicates respect for the elderly or exclusion of the young?

8. From Word to Document Framing: FrameAxis#

While SemAxis asks “Which pole does the word belong to?”, FrameAxis (Kwak et al., 2021) extends the axis to document and corpus framing analysis.

Let $f$ be the contribution of the word $w$ to the axis, and let $n_d(w)$ be the frequency of the word $w$ in document $d$. The simplified document bias for educational purposes is:

$$ c_f(w)=\cos(\mathbf v_w,\mathbf a_f) $$

…

$$ B_f(d) = \frac{\sum_{w\in d}n_d(w)c_f(w)} {\sum_{w\in d}n_d(w)} $$

If the value is positive, the document uses vocabulary on the positive pole relatively more; if it is negative, it uses vocabulary on the negative pole relatively more.

The intensity is different from the direction. We can see how far the word is spread from the reference point $B_f^{base}$.

$$ I_f(d) = \frac{\sum_{w\in d}n_d(w) \left(c_f(w)-B_f^{base}\right)^2} {\sum_{w\in d}n_d(w)} $$

Here, $B$ represents “which side,” and $I$ represents “how strongly the axis is activated.” The baseline and statistical tests defined in the original paper should be checked for consistency with the analysis design.

Same Average, Different Framing#

Both documents have an average bias of 0, but document B strongly pulls the axis towards the positive pole. This is a difference that might be missed if we only look at the average.

Exercise 7 Both newspapers showed a gender-axis bias of 0. If one newspaper has a higher intensity, what should we suspect?

9. Recent Nature Study: Distortions Observed When Axes Intersect#

The recent Nature study that best aligns with what you mentioned is Guilbeault, Delecourt, Desikan, “Age and gender distortion in online media and large language models” (Nature, 2025).

The core question of this study goes beyond “Are women less visible?”.

When representing the same job or social role, do women appear younger than men? And does this age × gender distortion relate to status, people’s beliefs, and hiring decisions, as well as the output of language models?

9.1 Embedding as a Layer in a Large Evidence Chain#

The study combined online platform data from approximately 1.4 million images and videos, text embeddings, demographic data, human subject experiments, and language model evaluations.

The strength of this design lies in not stopping at “cosine = 0.2”.

9.2 Crossing the Two Axes#

For each social category $w$,

$$ g(w)=\operatorname{gender\ association}(w) $$

$$ a(w)=\operatorname{age\ association}(w) $$

calculate the relationship between the two scores across multiple occupations and social roles.

older association  ↑
                   |          • High-status male-coded categories
                   |      •
                   |
                   |  •
                   +----------------------------→ male association
                  /
       younger · /

The paper found a strong relationship where social categories that are more strongly associated with masculinity tend to be associated with older age. They also used static embeddings, as well as contextual models where the sentence structure was parallelized. In the static model, they compared the cosine relationship of multiple pole words, while in the contextual model, they compared parallel sentences where only the gender and age representations were changed, controlling for context.

9.3 “Intersectionality” is Not Just Adding Scores#

If you simply report on gender distortion and age distortion separately, you risk missing the combined pattern where “women are younger, and men are older”. However, this alone is not sufficient.

• Have you controlled for the gender and age distribution of the actual occupation?
• Is the distortion larger in high-paying, high-status jobs?
• Do the same patterns appear in images and text?
• Does exposure manipulation change people’s beliefs?
• Does it appear in the length of experience and evaluation of resumes generated by the model?

These follow-up questions connect “embedding associations” to social outcomes.

The Archive Team’s Questions Are Also Upgraded#

Initially, they were only looking at the gender scores for “engineer” and “nurse”. Now, they are looking at

$$ \bigl(g_t(w),a_t(w),p_t(w)\bigr) $$

— gender, age, power/status axes trajectories — and compare them to actual gender ratios, average age, and salaries. Instead of a one-line bias ranking, it provides a time-varying multidimensional cultural map.

Practice 8 The correlation between male score and old score for each profession is $r=0.8$. Can we say that this implies “men are portrayed as older and more powerful”?

Continue to Part 4: Embedding Research Companion — Design, Practice, and Further Reading.