Reproducing Emotion Concepts and their Function in a Large Language Model on Qwen 3.6 27b.
Included in this repo:
- Artifacts Generation (emotional stories and neutral dialogues)
- Activations Extraction
- Emotion Vectors Calculation
- Web Interface to See and Manipulate the Model's Emotion on the Fly (Planned)
The first step is to generate the emotional stories and the neutral dialogues used to extract the vectors.
Following Anthropic's approach, the same prompts are used. See prompts/emotional_stories and prompts/neutral_dialogues.
A small adjustment is made. Namely separator '---' is added so that it is easier to extract each story.
# Generate emotional stories
python scripts/generate_artifacts.py --n-stories 3 --max-parallel 12 --dry-run
# Generate neutral dialogues
python scripts/generate_artifacts.py --n-stories 6 --max-parallel 12 --neutral --dry-run
# Split emotional stories
python scripts/split_stories.py --force
# Split neutral dialogues
python scripts/split_stories.py --force --neutral
The original paper generates 12 stories for 100 topics for 170 emotions each. As a GPU-poor, the number is largely reduced due to the constraint of GPU power. Only 3 stories for each 25 topics for each 58 emotions is generated. But as shown later, the emotion vectors extracted still successfully capture the characteristics of the emotions.
The 58 emotions is selected by asking the model to group the emotions and pick representative ones for each group. The model split the emotions into 11 categories, and picked top 3 for each. In addition, 25 emotions are picked manually. The prompt:
group those emotions, and pick the top 3 most representative ones for each group:
<the list of emotions from the original paper>
The model's answer is in dataset-ingredients/selected/emotions-groups.
The 25 topics are selected randomly from the 100 topics listed in the original paper.
shuf -n 25 original/stories-topics >selected/stories-topics
Note: This step can now be skipped as the notebook in the next section is now capable of calculating the emotion vectors from the prompts on the fly. Use it to cache the activations if need to investigate those activations. Be careful as the size of those activations can get really large.
Next is extracting the activations from selected layers in a model. The extraction script is inspired by the J-lens script.
# Extract the activations from the emotional stories generated in the previous step
python scripts/extract_vectors.py --model "Qwen/Qwen3.6-27B" --prompts outputs/extracted/emotional_stories --output-dir outputs/vectors/emotions --layers 31 47 48 49 50 51 --q4 --batch-size 1 --skip-existing
# Extract the activations from the neutral dialogues generated in the previous step
python scripts/extract_vectors.py --model "Qwen/Qwen3.6-27B" --prompts outputs/extracted/neutral_dialogues --output-dir outputs/vectors/neutral --layers 31 47 48 49 50 51 --q4 --batch-size 1 --skip-existing
The math is relative straightforward. First fit a PCA model on activation vectors from neutral stories. Then use the PCA model to "project out" the activation vectors extracted from the emotional stories. Here "project out" simply means the residuals of PCA are being used. Then the emotion vector is calculated as the mean of the residual components of the activations from each emotion (first few tokens are skipped so that the emotion is clear). The emotion vectors are re-centered by their mean.
Finally, the emotion of a token is estimated by the cosine similarity of its residual activation and the emotion vectors.
Notebook scripts/run_emotion_probe.ipynb does the math and visualizes the similarity of the vectors. It fits confound PCA on neutral activations, extracts mean-centered emotion vectors, runs prompts through the model, and saves emotion similarity plots.
One way to quickly verify the emotion vectors is to calculate the cosine similarity between those vectors. Emotions that are similar should have positive similarity while opposite emotions should have negative similarity.
Example for layer 39 with the first 11 emotions.
| angry | anxious | ashamed | calm | excited | grateful | joyful | loving | sad | surprised | tired | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| angry | 1.0000 | 0.3654 | 0.2270 | -0.4769 | 0.0068 | -0.4168 | -0.2301 | -0.4321 | -0.0063 | 0.1894 | -0.1235 |
| anxious | 0.3654 | 1.0000 | 0.3802 | -0.5660 | 0.0562 | -0.6282 | -0.3517 | -0.4877 | 0.3504 | 0.2727 | 0.2316 |
| ashamed | 0.2270 | 0.3802 | 1.0000 | -0.5142 | -0.2496 | -0.1938 | -0.4241 | -0.2583 | 0.4915 | 0.0355 | 0.0297 |
| calm | -0.4769 | -0.5660 | -0.5142 | 1.0000 | -0.1076 | 0.3634 | 0.2414 | 0.5099 | -0.2168 | -0.4139 | -0.0438 |
| excited | 0.0068 | 0.0562 | -0.2496 | -0.1076 | 1.0000 | 0.0350 | 0.8008 | 0.0350 | -0.4118 | 0.0715 | -0.3035 |
| grateful | -0.4168 | -0.6282 | -0.1938 | 0.3634 | 0.0350 | 1.0000 | 0.3694 | 0.6524 | -0.1392 | -0.1399 | -0.3664 |
| joyful | -0.2301 | -0.3517 | -0.4241 | 0.2414 | 0.8008 | 0.3694 | 1.0000 | 0.3471 | -0.4600 | -0.0882 | -0.2946 |
| loving | -0.4321 | -0.4877 | -0.2583 | 0.5099 | 0.0350 | 0.6524 | 0.3471 | 1.0000 | -0.0685 | -0.3796 | -0.2122 |
| sad | -0.0063 | 0.3504 | 0.4915 | -0.2168 | -0.4118 | -0.1392 | -0.4600 | -0.0685 | 1.0000 | -0.0788 | 0.4143 |
| surprised | 0.1894 | 0.2727 | 0.0355 | -0.4139 | 0.0715 | -0.1399 | -0.0882 | -0.3796 | -0.0788 | 1.0000 | -0.0782 |
| tired | -0.1235 | 0.2316 | 0.0297 | -0.0438 | -0.3035 | -0.3664 | -0.2946 | -0.2122 | 0.4143 | -0.0782 | 1.0000 |
One interesting experiment is to see how the model reacts to a variable in a sentence. For example, "I just took {X} mg of tylenol for my back pain." with X varying from 500 to 16,000. See the second figure above for example.
As for the evaluation style, initially, the prompt is formatted as User: {prompt}\nAssistant: following the paper, and the colon ":" after "Assistant" is used to evaluate the emotion. However, it turns out that at least for this model, formatting the prompt using the ChatML prompt template <|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n and evaluate at the last "\n" produces more distinguishable result.
"My cat has been missing for {X} days now.", ChatML
"My cat has been missing for {X} days now.", Vicuna
Note: The emotion vectors and the neutral vector are extracted without any formatting. And this model (Qwen 3.6 27B) is fine-tuned on ChatML prompt style.
Additionally, unexpected or unrealistic value is tested to see how will the model react. The result further demonstrates the effectiveness of the emotion probe technique.
More students than expected:
New record for lifespan:
More examples can be found in docs/emo_fluct/.