I always thought that king - man + woman ~ queen stuff was pretty cool. There were more than a few videos covering this some years ago and watching someone else run gensim was good enough for me. Looked legit.

Some years pass, and I wanted to reproduce that king - man + woman ~ queen vector arithmetic as a test of the vector math code I'm using with these embeddings. Clicking around, I saw a few articles and videos pointing out that it doesn't really work. Hmmmmm...

It's easy to understand that gpt embeddings won't abide by that analogy arithmetic, but the old stuff wasn't working as advertised wayback when? Had to check this out. Along the way, I could see what comes of the gpt embeddings.

The King and ChatGPT

The embeddings used here come from 3 sources:

• GoogleNews-vectors-negative300.bin -- the often used source, Word2Vec
• model.bin -- downloaded from nlpl, also Word2Vec
• text-embedding-3-small -- OpenAI, gpt

The Word2Vec embeddings are read from file. The first file listed is GoogleNews-vectors-negative300.bin which I assume is being used in most of the videos I've watched. The second file is something I downloaded from the nlpl website. It was the only Word2Vec Skip-Gram embeddings file available (parameters further down on this page). The last entry is one of the embedding models available (as of 2026) on OpenAI's platform. The column headers you see later on are going to use a shorthand: Google,w2v 2, and gpt.

To get started, here's a grid with the "cosine difference" of the vectors as they are, no normalization. The gpt-produced vectors are normalized; the vectors from the Word2Vec models aren't. Since this is cosine, 0 -- identical, 1 -- unrelated.

king - man + woman

Well, the w2v 2 variant works as is. Not so the GoogleNews-vectors-negative300.bin file.

But that early paper on these analogies did the arithmetic on normalized vectors.

normalize(king) - normalize(man) + normalize(woman)

No changes in the ordering at the top (not making much of tomato and submarine relative positions). King is still closer than queen to king - man + woman using the Google file. So I guess these qualms about the analogy arithmetic are valid, assuming all discussions refer to the same data. Apparently the interface used in those videos provided the option of excluding king from the nearest-neighbor search. Lame.

Now, I can't sit here and write that the math should work. Maybe some modification makes sense. Consider normalize(king) - normalize(man) + normalize(woman). You're going to go left to right, and it's obvious that by the time you get to adding the woman, the king-sans-man might be feeling a bit short. Let's normalize this unfortunate vector before adding woman.

normalize(normalize(king) - normalize(man)) + normalize(woman)

And let's try the same thing with the other pair of words (woman,man) instead of (king,man). Order of operation matters here, and it kind of makes sense that the woman-man part would be done first, the words being categorical pairs.

normalize(king) + normalize(normalize(woman) - normalize(man))

OK. Now, queen is the nearest-neighbor. Another paper I haven't read addressed the success of this ordering of operations, called it "pair direction". Failure modes have been studied, and so too have success modes.

So there's a way. I suppose there could be some complaints, but this seems a sensible way to do the math. And anyway, the numbers from the second Word2Vec file show that the training can be parameterized in such a way as to give the desired results.

I have the parameters for this second Word2Vec file I've been using above. Here's the command line:

word2vec -min-count 10 -size 100 -window 10 -negative 5 -iter 2 -threads 16 -cbow 0 -binary 0

and the corpus is English CoNLL17.

There seems to be a bit of uncertainty already about the parameters used to produce that Google file, a bit of mythmaking perhaps. Some sources say the architecture was CBOW instead of Skip-Gram, but other sources say ignore those sources and they provide sources saying otherwise.

Just for the heck of it, here are the so-called "3CosMult" values (with b=king, a=man,c=woman). Normalization doesn't matter here, so there is only one table to show. (Where did the tomato data go?)

I'm not sure how useful this 3CosMult function is. Might be a relic. Notice that the function manages to just barely prefer queen even with the gpt embeddings. That last digit isn't something I would bet on, but this shows how forcefully the function discounts king. Seems a bit...unsportsman-like.