04. Embedding Research Companion — Design, Practice, and Further Reading#

Research Companion: Study Design → Implementation → Diagnostics → Practice → Further Reading

All About Embedding · Part 1: Word2Vec, SGNS, and Gravity · Part 2: Implementation, Relations, and Meaning Axes · Part 3: Culture and Semantic Change · Part 4: Research Companion

© 2026 Byunghwee Lee. Created and edited in collaboration with OpenAI Codex. Unauthorized copying, reproduction, distribution, and use are prohibited.

This companion continues Part 3 by turning embedding-based cultural analysis into a concrete research workflow. It is organized as an appendix-like lab: specify a study, implement the core measurements, inspect the results, practice the reasoning, and use the references to go further.

After reading this note, you will be able to:#

• Translate a broad cultural question into a measurable embedding study.
• Implement axes, WEAT, temporal alignment, and document framing in a compact workflow.
• Choose figures and robustness checks that match the claim.
• Audit an embedding paper using a concise checklist.
• Identify useful extensions and further readings for a new project.

Table of Contents#

1. Ⅰ. Designing and Implementing the Study — 1–3
2. Ⅱ. Practice, Check, and Synthesis — 4–7
3. Ⅲ. Further Reading — References and Appendix

Ⅰ. Designing and Implementing the Study#

↑ Table of Contents

1. Designing and Implementing a Single Study from Start to Finish#

Now, let’s write a project proposal for the records team as if it were a real research proposal.

1.1 Research Questions#

Bad questions:

“How do people in society perceive engineers?”

Measurable questions:

“What are the changes in the cosine projection of ‘engineer’ on the predefined gender axis in the SGNS space of the English text corpus from 1900–2000, and are these changes correlated with the female ratio in American professions?”

This sentence includes data, time, model, axes, metrics, and external validation.

1.2 Analysis Specifications#

These numbers are examples. Adjust them to fit the actual corpus size and reproducibility.

1.3 Analysis Procedure#

1. First, fix the hypothesis and poles. Don’t just look for the results that support your hypothesis.
2. Examine the frequency and original samples for each era.
3. Train era embeddings with the same hyperparameters and multiple seeds.
4. Align the space using a common anchor.
5. Verify the internal consistency and leave-one-pole-out stability of the axes.
6. Calculate scores, neighbors, and displacement for each word, along with uncertainty intervals.
7. Perform robustness analysis by swapping corpus, window, pole, anchor, and seed.
8. Compare with independent data such as population statistics, surveys, and experiments.
9. Only claim up to the level that is observable.

1.4 When Results Aren’t Pretty, That’s Real Research#

Results like these also have meaning.

• Google Books shows changes, but COHA doesn’t → Genre and sample composition hypothesis
• Axis score changes, but neighbors remain similar → Possible movement of the axis itself
• Average changes, but smaller than seed variance → Difficult to call this a change
• Text embedding changes, but not correlated with Census → Separation of expression and composition
• Controlling for occupational gender eliminates the effect → Possible reflection of real-world composition
• Remains even after control → Further exploration of optional descriptions and stereotypes

Instead of asking “Did we get the desired conclusion?”, it’s better to ask “What alternative explanations were ruled out?”

2. Minimal Implementation: Axes, WEAT, Alignment, and Document Framing#

The functions below form the framework of the analysis. For a full research-level analysis, you will need to handle OOV (out-of-vocabulary) words, multiple seeds, bootstrapping, permutation, and multiple comparison corrections.

import numpy as np


def unit(x, eps=1e-12):
    x = np.asarray(x, dtype=float)
    return x / max(np.linalg.norm(x), eps)


def cosine(x, y):
    return float(unit(x) @ unit(y))


def centroid_axis(E, positive_words, negative_words):
    # Normalize each vector first to reduce word-specific norm effects.
    pos = np.mean([unit(E[w]) for w in positive_words], axis=0)
    neg = np.mean([unit(E[w]) for w in negative_words], axis=0)
    return unit(pos - neg)


def axis_score(E, word, axis):
    return cosine(E[word], axis)


def relative_association(E, word, A, B):
    sim_a = np.mean([cosine(E[word], E[a]) for a in A])
    sim_b = np.mean([cosine(E[word], E[b]) for b in B])
    return float(sim_a - sim_b)


def weat_effect_size(E, X, Y, A, B):
    sx = np.array([relative_association(E, w, A, B) for w in X])
    sy = np.array([relative_association(E, w, A, B) for w in Y])
    pooled = np.concatenate([sx, sy])
    return float((sx.mean() - sy.mean()) / pooled.std(ddof=1))


def orthogonal_align(X, Y):
    # Two matrices whose rows are shared anchor words. Align X @ R to Y.
    U, _, Vt = np.linalg.svd(X.T @ Y, full_matrices=False)
    return U @ Vt


def semantic_displacement(v_old, v_new, R):
    return 1.0 - cosine(v_old @ R, v_new)


def document_frame_score(tokens, E, axis):
    values = [axis_score(E, w, axis) for w in tokens if w in E]
    return float(np.mean(values)) if values else np.nan

Core of Permutation Test#

def weat_statistic(E, X, Y, A, B):
    return (
        sum(relative_association(E, w, A, B) for w in X)
        - sum(relative_association(E, w, A, B) for w in Y)
    )


def permutation_p_value(E, X, Y, A, B, repeats=10000, seed=0):
    rng = np.random.default_rng(seed)
    words = np.array(list(X) + list(Y), dtype=object)
    observed = weat_statistic(E, X, Y, A, B)
    extreme = 0

    for _ in range(repeats):
        shuffled = rng.permutation(words)
        Xp = shuffled[:len(X)]
        Yp = shuffled[len(X):]
        if weat_statistic(E, Xp, Yp, A, B) >= observed:
            extreme += 1

    # Add 1 to avoid reporting p=0 in a randomization test.
    return (extreme + 1) / (repeats + 1)

In practice, if the null hypothesis does not have a predefined direction, you should use a two-tailed test to compare the absolute values. For small sets where a precise permutation is possible, you can enumerate all possible divisions.

3. Good Graphs to Add After Implementation#

With one table and four graphs, you can convey most of the story.

Figure 1. Meaning Map at a Single Point#

• x-axis: gender score
• y-axis: age score
• Points: professions
• Point size: corpus frequency
• Color: actual female ratio or job title

Question to ask: Are male-dominated professions also portrayed as older and more powerful?

Figure 2. Trajectory over Time#

• x-axis: decade
• y-axis: axis score
• Lines: ‘engineer’, ‘nurse’, ‘leader’
• Bands: 95% confidence intervals for multiple seeds or bootstrapping
• Secondary panel: Actual composition from Census

Question to ask: Do the changes in text embeddings precede, follow, or are unrelated to changes in the real composition?

Figure 3. Neighbor Changes Table#

Adding the actual word neighbors next to the trajectory makes it much more intuitive to understand “why the score changed.”

Figure 4. Robustness Heatmap#

• Rows: pole set, corpus, window, alignment anchor
• Columns: key words, era
• Cells: direction and magnitude of effect

Reading Questions: Does the conclusion depend solely on a single model selection?

Figure 5. bias–intensity plane#

• X-axis: FrameAxis bias
• Y-axis: FrameAxis intensity
• Points: articles, newspapers, political parties, era

The documents with the same average direction but different framing strength are separated.

Ⅱ. Practice, Check, and Synthesis#

↑ Table of Contents

4. Practice Problems — Small Lab#

A. Concepts and Claims#

1. Classifying into Four Layers#

Classify the following conclusions into the four layers of the 0th section.

1. scientist is closer to the male pole.
2. The proportion of men among actual scientists is higher.
3. Participants rated identical resumes with male names more highly.
4. Respondents associated science with men more quickly.

2. Rewriting “The model has bias”#

WEAT noted this. Rewrite it into a more accurate sentence.

B. Calculation#

3. Axis projection#

Given the unit vector $\mathbf a = (0.8, 0.6)$ and the unit word vector $\mathbf v = (0.6, 0.8)$, find the dot product.

4. Sign of the WEAT effect#

The average relative association of $X$ is 0.3, $Y$ is -0.1, and the combined standard deviation is 0.2. Find the effect size $d$ and interpret it.

5. Semantic displacement#

After sorting, the cosine between the two time periods for the same word is 0.92. What is the value of $1-\cos$?

C. Research Design#

6. N-gram or Embedding?#

What should be used for the questions “When did the word AI become popular?” and “When did AI become closer to creativity than automation?”

7. Moving Axes#

The gender axis has changed significantly between the 1920s and the 2020s. How can we strengthen the comparison of occupational scores across these periods?

8. Genre confound#

In one period, the proportion of romance novels increased significantly, and the female score for nurse also increased suddenly. What re-analysis is needed?

9. SemAxis Multiple Comparisons#

When searching for a leader among 732 axes, 37 were found to be significant ($p<0.05$). Why can’t we simply call these 37 “discoveries”?

10. Debiasing Evaluation#

Design evaluation tables before and after removing gender axes.

11. Evidence Chain for Nature Research#

Given the observation that female figures appear younger in the image, what alternative explanations can be provided by adding text embedding, demographics, and human experiments?

12. My One-Sentence Research Plan#

Please fill in the blanks.

“Using the [embedding] learned from the [perspective] of the [corpus], I will measure the changes in [word groups] corresponding to the [specified axis], and compare them with [external data].”

5. Checklist for Reading Papers#

Data#

• What is the era, language, genre, and platform of the corpus?
• Who are the missing and overrepresented groups?
• Have you processed frequency, OCR, duplicate documents, and named entity issues?

Vectors#

• Is it static or contextual?
• What representation are you using? input, output, average, layer?
• Have you replicated it with multiple seeds and models?
• How have you aligned it in time and space using anchors and methods?

Measurement#

• Have you pre-defined the pole and target words?
• Does one word dominate the results?
• Are the axes, WEAT, and neighborhood structures consistent?
• Have you handled multiple comparisons according to the number of exploration axes?

Interpretation#

• Have you distinguished between corpus association, human stereotypes, reality construction, and downstream harm?
• Are you using correlation as causation?
• Are you validating it with external data and actual context?
• Does debiasing only hide measurement values?

6. Summarizing Research Connections in a Table#

7. The Final Scene: Are Vectors Mirrors or Maps?#

The records team finally obtained “engineer” and “nurse” as two lines. But a good researcher doesn’t just shout “society has changed” when seeing these lines.

They ask:

• Are these lines created by word frequency or corpus genre?
• Is the gender itself moving?
• Do they remain in other poles, seeds, and models?
• Are they moving with actual occupational composition?
• Are they connected to people’s beliefs and judgments?
• Does reducing the measurement value also reduce actual harm?

Across Parts 1 and 2, Word2Vec compressed co-occurring encounters into a latent space and turned recurring relations into interpretable directions. Part 3 placed axes in that space, overlaid them with time, and compared them with external data.

Therefore, the best one-sentence statement for using embeddings in cultural research is:

Embeddings are not simply mirrors reflecting the truth of society, but rather, maps created by specific data and measurement designs. Good research involves discovering patterns on these maps and continuously validating them with other maps and real-world indicators.

Ⅲ. Further Reading#

↑ Table of Contents

References and Further Reading#

1. Bolukbasi, T. et al. (2016). “Man is to Computer Programmer as Woman is to Homemaker?” Debiasing Word Embeddings. NeurIPS.
2. Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016). “Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.” ACL.
3. Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). “Semantics derived automatically from language corpora contain human-like biases.” Science, 356, 183–186.
4. Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). “Word embeddings quantify 100 years of gender and ethnic stereotypes.” PNAS, 115, E3635–E3644. Research code
5. An, J., Kwak, H., & Ahn, Y.-Y. (2018). “SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment.” ACL.
6. Kozlowski, A. C., Taddy, M., & Evans, J. A. (2019). “The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings.” American Sociological Review.
7. Gonen, H., & Goldberg, Y. (2019). “Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them.” NAACL.
8. Kwak, H., An, J., Jing, E., & Ahn, Y.-Y. (2021). “FrameAxis: characterizing microframe bias and intensity with word embedding.” PeerJ Computer Science.
9. Guilbeault, D., Delecourt, S., & Desikan, B. S. (2025). “Age and gender distortion in online media and large language models.” Nature, 646, 1129–1137.

Next, it is beneficial to study how to control sentence patterns in contextual embeddings, separate the sense of polysemous words across time, and connect the axes to actual downstream audits.

Appendix. Recent Interesting Embedding Applications#

The following studies use embeddings not as simple classification inputs, but as a space to measure distances, axes, trajectories, and concept structures. I have briefly summarized recent papers suitable for presenting at a research seminar, focusing on the ideas and points to consider.

1. The Meaning Space of Language and Climate#

Fu et al. (2026), Semantic similarity across languages reflects neurocognitive dimensions shaped by climate, Nature Communications.

• Idea: Project 53 languages’ word embeddings into 13 semantic dimensions (e.g., sensation, motor, emotion, social) and compare the commonalities and differences between languages.
• Validation Chain: Connect the embedding analysis with 8 language speakers’ behavioral ratings, a colexification network for 2,681 languages, and exploratory fMRI data.
• Interesting Point: Test a bold hypothesis that language-specific semantic differences are related to climatic differences, not just language families and cultures.
• Points to Consider: The results are correlations, not causation. The climate, geography, culture, and language family are strongly intertwined, and the results may be sensitive to translation and corpus composition.

2. Creating a Concept Space with LLM’s Selection Behavior#

Du et al. (2025), Human-like object concept representations emerge naturally in multimodal large language models, Nature Machine Intelligence.

• Idea: Learn a 66-dimensional concept embedding by training on approximately 4.7 million “choose one from three” judgments performed by 1,854 objects using LLMs and a multimodal LLM.
• Interesting Point: Reconstruct a new meaning space from the model’s observed selection behavior, rather than directly reading the hidden vectors.
• External Comparison: Compare the range and dimensionality of the learned space with fMRI representations of human judgments and visual stimuli.
• Points to Consider: The similarity in geometry between the model and humans does not necessarily mean that they form concepts in the same way. There is also the possibility that a common categorical structure or learning material creates the alignment.

3. Contextually Changing Meaning Axes#

Zeng, Jin, and Voigt (2024), Adaptive Axes: A Pipeline for In-domain Social Stereotype Analysis, EMNLP.

• Idea: Mask the target group name and embed only the surrounding context, separating the associations stored in the target token from the framing in the current document.
• Extension: Instead of using only fixed WordNet antonym axes, create and refine new meaning axes specific to domains such as science, health, and art using LLMs.
• Interesting Point: This addresses the problem that the same group can be described differently depending on the domain, using contextual embeddings and adaptive axes.
• Points to Consider: If the LLM creates axes, the model’s learning data and cultural assumptions will enter the measurement tool. It is necessary to verify the validity of the axes with alternative axes and human evaluations.

4. Users’ “Information Displacement” After Fact-Checking#

Kim et al. (2025), Differential impact from individual versus collective misinformation tagging on the diversity of Twitter (X) information engagement and mobility, Nature Communications.

• Idea: Define content diversity and information mobility based on the average embedding of a user’s past tweets, centered around their usual interests, and the degree to which a new tweet deviates from that center.
• Research Question: Compare the different effects that personal fact-checking and community-verified “Community Notes” have on subsequent information seeking.
• Interesting Point: Instead of using the embedding distance as an explanatory figure, make it an actual behavioral outcome variable and incorporate it into interrupted time series and delayed-feedback analysis.
• Caution: As observational data, it is difficult to completely eliminate selection bias and unobserved events. The novelty in the embedding is not necessarily the same as diversity of perspectives or changes in attitudes.

5. Reading Researcher Mobility Paths as Sentences#

Murray et al. (2023), Unsupervised embedding of trajectories captures the latent structure of scientific migration, PNAS.

• Idea: Train Word2Vec by changing institution·region = word, researcher's affiliation mobility path = sentence.
• Theoretical Connection: Demonstrate that SGNS is connected to the gravity model of mobility, and learn functional distance based on language, culture, and prestige, which cannot be explained solely by geographical distance.
• Interesting Point: Not only do we apply natural language algorithms to other sequential data, but we also provide a mathematical correspondence with existing mobility theories.
• Caution: The basic model assumes symmetry in the flow of mobility. The affiliation information in the paper cannot clearly distinguish between long-term migration, short-term visits, and simultaneous affiliation.

Criteria to Choose at Seminars#

The common question that underlies these studies is:

What to use as a point, what to make into neighboring observations, and what real-world data to use to verify the distance and direction of that space?