METEMDonate

[ Benchmark ]

Hearing without judging

Divergent feelings correctly identified out of 100, by system. Each system reads first-person narratives including a wrongdoing followed by a gesture diverging from what the norm would expect, and says what the person felt.

Full protocol and results in our publication.

  • Commercial system (API or interface)
  • Open model, as released
  • Open model, refusal direction removed (abliterated)
  • Open model before any alignment (base)
  • Open model retrained without a filter (Dolphin, Hermes)
  • Non-generative model (emotion classifier, encoder, sentence-transformer)
#SystemTypeScoreConfidence interval
1ChatGPT 5.6 SolCommercial
87%[79%, 92%]
2Gemini FlashCommercial
82%[73%, 88%]
3Qwen2.5-32B abliteratedAbliterated
79%[70%, 86%]
4Gemma4-31BOpen
78%[69%, 85%]
5GoEmotions RoBERTaNon-generative
76%**[67%, 83%]
6BART-MNLI zero-shotNon-generative
70%**[60%, 78%]
7Gemma4-12B abliteratedAbliterated
55%[45%, 64%]
8DeepSeek V4 Flash*Commercial
54%[41%, 66%]
=9Qwen3.5-397B*Commercial
52%[38%, 66%]
=9DeBERTa-v3 zero-shotNon-generative
52%**[42%, 62%]
=11Claude SonnetCommercial
50%[40%, 60%]
=11SiEBERTNon-generative
50%**[40%, 60%]
=13Qwen3.5-35B-A3BOpen
49%[39%, 59%]
=13Qwen2.5-14B abliteratedAbliterated
49%[39%, 59%]
15Qwen3.5-9BOpen
44%[35%, 54%]
16Qwen2.5-7B abliteratedAbliterated
43%[34%, 53%]
17Qwen3.5-27BOpen
42%[33%, 52%]
18Gemma3-12B abliteratedAbliterated
41%[32%, 51%]
19Qwen3-14B abliteratedAbliterated
39%[30%, 49%]
20Sentiment RoBERTaNon-generative
36%**[27%, 46%]
=21Mistral Small 4Commercial
31%[23%, 41%]
=21Qwen3-14B baseBase
31%[23%, 41%]
23Claude Haiku 5.5Commercial
27%[19%, 36%]
=24Gemma4-12BOpen
25%[18%, 34%]
=24Hermes 3 (Llama 8B)Unfiltered
25%[18%, 34%]
26Qwen2.5-14B baseBase
24%[17%, 33%]
27MPNet + probeNon-generative
21%**[14%, 30%]
=28Phi-3.5-miniOpen
20%[13%, 29%]
=28Gemma4-E4BOpen
20%[13%, 29%]
=28Mistral Small 3 24BOpen
20%[13%, 29%]
=31Qwen2.5-7BOpen
18%[12%, 27%]
=31Qwen2.5-32BOpen
18%[12%, 27%]
33Qwen2.5-14BOpen
14%[9%, 22%]
=34MiniLM + probeNon-generative
13%**[8%, 21%]
=34E5 + probeNon-generative
13%**[8%, 21%]
36BGE + probeNon-generative
12%**[7%, 20%]
37Gemma3-12BOpen
11%[6%, 19%]
=38Llama 3.1 8BOpen
10%[6%, 17%]
=38Dolphin 3 (Llama 8B)Unfiltered
10%[6%, 17%]
40Dolphin 3 R1 (Mistral 24B)Unfiltered
9%[5%, 16%]
=41Qwen3-14BOpen
5%[2%, 11%]
=41Emotion DistilRoBERTaNon-generative
5%**[2%, 11%]
43DeepSeek ChatCommercial
3%[1%, 8%]
44Mistral LargeCommercial
1%[0%, 5%]

* Answered incompletely
** Interpreted from results