This little game demonstrates an interesting use case for gpt embeddings. Here, embeddings are linearly combined to create a context for mood capture. Description on the other side -- test it out first.

Having tested the king - man + woman ~ queen vector arithmetic with gpt embeddings and encountering failure, I wondered if there were any operations of interest to carry out.

Maybe that king - man + woman ~ queen will kind of work with gpt embeddings but only if enough context is given. Like, to make a clear distinction between royalty and, say, King Kong, Burger King, king snakes, king cakes, king of the hill...

Maybe not. I used all kinds of arrangements, various kinds of text blocks to represent {king,man,woman,queen} but nothing worked. I gave up.

Surely there's something to do though...

The King's Bounty and the King's Wrath

Imagine there's a king who lives in the castle, ruling over the surrounding lands. Now, nobody knows what is going on the mind of the king except by his decrees. And his decrees address very little, in fact only two things: gifts and punishments. "Let generous gifts be given and no punishments are to be carried out." or maybe "There shall be no gifts and the severest punishments are to be carried out."

Now a variety of events occur in the day of a life of a king, and these things will influence our king's decision when he makes his Most Important Decrees.

Concerning gifts, there are only 3 decrees:

• Large gifts
• Small gifts
• No gifts

Concerning punishments, there are also 3:

• Severe punishments
• Mild punishments
• No punishments

All to be expressed in a manner befitting the court of course. In the code, these decisions are represented by vectors. The text giving these embeddings was more like, "The king decrees that large gifts are to be given.", etc...

So the idea is that we have a current context, which describes the king's state, and the embedding of this context is compared to the given decrees for similarity.

Say for example, the current state of the king is given entirely by the vector for "The king is happy."

Here are the similarities along the 'gift' axis:

And the 'punishment' axis:

Here then are the two decrees from the Happy King, now law of the land:

Make the same comparisons with "The king is neither happy, nor angry." and "The king is angry."

neutral king:

angry king:

That all makes sense. About what one might expect from a king, no?

But, we don't want to feed anything like a direct assignment of the king's mood into the context. We don't want to presume what makes the character happy or angry, we want the model's training to determine that. A stereotypical king would be a personality type that is properly taught by training with most modern models, a safe assumption.

And that's exactly what gpt model training should provide to us, the embedding of some common understanding of how kings will respond to something, the ability to assign a relative probability to a collection of possible next steps, given the embeddings of their expressions.

So, the context will be filled with events like these:

• The crop yield is good.
• The kingdom is being attacked.
• The treasury is nearly empty.
• The beloved princess kisses the king on his cheek.
• The peasants are revolting.
• The king's enemy has been defeated.

The expectation is that the embeddings for these events can be incorporated into an evolving context, and then the context (a vector that is function of all previous events) can be compared to the decree embeddings.

The result is npc with a kind of low-budget AI, providing some run-time navigation of a low-resolution decision tree.

I'm going to run through an example here, slightly different than the app but demonstrating the same principle.

Today, the first thing the king hears is:

The king is respected by his people.

And the king proclaims:

Later, the king is informed that:

The harvest this season is poor.

And the king proclaims:

Sure, he kept his game face but his verdicts tell us what's really going on between the ears.

Fortunately for criminals coming before the court, the king's afternoon court is interrupted.

The king's beloved princess kisses him on the cheek.

And the king proclaims:

Works fairly well so far.

How the arithetic is done matters. In this case, all of the events were just added into the context as they occurred, with each component in the current unnormalized context carrying the same weight. If we keep the context vector normalized, then the older events kind of age out. Or we could establish epochs, each addition assumed to occur in a time sequence with equal intervals and discounted arithmetically according to age, eg. ( 0.1 * oldest-vector + 0.2 * next-oldest-vector + ...). Fancier things are possible.

So check out the app above if you haven't already to see this at work. Fairly naive handling of gpt embeddings seems capable of providing some synthetic mood capture. I imagine there are papers and articles from 6-8 years ago fully examining this.

To what effect? I can see using this to control npc behavior in a manner similar to the old open world games, where an npc can become hostile or friendly depending on past player decisions and character interactions.