MultivationBench: AI Models Still Fail at Understanding Why People Act — and the Numbers Prove It

What MultivationBench actually tests

We have benchmarks for math, coding, and medical knowledge. But human motivation — why someone reaches for a phone at a dinner table, why a character walks away from a conversation — has been largely left unevaluated by the AI industry. A team from HKUST and Amazon aims to change that with MultivationBench, published on arXiv on July 29.

The benchmark contains 1,000 visual narratives drawn from three datasets — SSID (social-media-style stories), StoryReasoning, and MovieBench (cinematic scenes). For each story, the team identified 4,023 visually grounded character behaviors and then asked four questions per behavior: two about which Maslow-level need the behavior serves (theoretical definition + practical motivation), and two about the corresponding Reiss desire. That yields 16,092 evaluation instances across eight models.

The critical challenge isn't just recognizing what's in a single image. It's revising your interpretation as new evidence emerges. In one example from the paper, a character named Debbie reaches for her phone during a formal dinner. Early in the story, this looks like information-seeking — a "Cognitive" need. But later images reveal family photos on the phone and Debbie's social isolation at the table. The correct motivation shifts to "Love and Belonging." Models that latch onto the first plausible explanation fail the longer context.

The psychological grounding that makes this different

Most AI social-reasoning benchmarks use ad-hoc labels — "helpful," "aggressive," "cooperative." MultivationBench is grounded in two established psychological frameworks:

• Maslow's Expanded Hierarchy of Needs (8 levels): Physiological, Safety, Love and Belonging, Esteem, Cognitive, Aesthetic, Self-Actualization, and Transcendence. This captures the broad level of need driving a behavior.
• Reiss's 16 Basic Desires: Physical Exercise, Eating, Order, Saving, Tranquility, Romance, Family, Acceptance, Social Contact, Independence, Vengeance, Honor, Power, Status, Curiosity, and Idealism. This captures the specific desire content — a finer-grained lens on motivation.

Each behavior gets a multi-label annotation — because real actions often serve more than one motivation simultaneously. Three human annotators (graduate students from English-speaking universities) provided ground truth, with Fleiss' κ agreement scores ranging from 74.9% to 81.0% across the four task types.

What the numbers say: models vs. humans

The paper tests eight MLLMs across three modalities (full multimodal, text-only, image-only) and three story-length buckets (Short: 2–5 images, Medium: 6–9, Long: 10+). Here's the overall performance table I've compiled from the paper's results:

Two things jump out. First, the Exact Match gap between the best model and humans is enormous — Phi-4-multimodal hits 39.2%, while humans score 70.6% on Maslow Definition and 78.6% on Maslow Practical Motivation. That's a 31–39 percentage point gap. Second, story-level consistency essentially does not exist. Even Phi-4-multimodal, the best performer on EM, manages to answer every question within a single story correctly just 0.8% of the time. For context: that's 8 out of 1,000 stories.

From the perspective of someone who runs AI in production, this is sobering. We run our AI Arena benchmark rig on an RTX 5060 Ti 16 GB, testing both cloud and local models for real-world throughput and quality. What MultivationBench measures — the ability to track evolving intent across time — is precisely what you need for any AI system that interacts with humans over more than a single turn: customer support chatbots, AI assistants in healthcare, educational tools. The models we test locally, like the Phi family, show exactly this pattern: they can nail individual exchanges but lose the thread when the conversation extends.

Where models fail: over-interpretation and surface-level cues

The paper's error analysis reveals a consistent pattern: models over-interpret rather than under-interpret. Across most high-capacity models, false positives (hallucinating motivations that aren't there) outnumber false negatives (missing valid motivations) by roughly 3-to-1. Nemotron-Nano-12B, the worst offender, produced over 31,000 false positives compared to about 7,000 false negatives. The behaviour is familiar to anyone who's seen an LLM confidently explain a character's "inner conflict" that doesn't exist — the model accumulates earlier cues and, rather than discarding them as irrelevant after new evidence, keeps piling them onto its interpretation.

Label-wise performance tells a similar story. Models handle concrete, single-cue motivations (Eating: ~60% F1, Safety: ~55%, Romance: ~50%) decently well. You see food, you guess hunger. Fair enough. But performance collapses on motivations that require accumulated context across the narrative: Self-Actualization (~20%), Transcendence (~20%), Status (~25%), Independence (~25%). These are things you can't infer from a single frame — they require understanding a character's goals, values, and social position over time.

Why visual input matters — and what the ablation tells us

One of the paper's most instructive findings comes from the modality ablation. When you strip away images and feed models only text descriptions of the same stories, F1 scores hold up surprisingly well, but Exact Match accuracy collapses. Gemini-3-Flash, for instance, retains a decent text-only F1 (53.7% multimodal → 49.2% text-only) but its EM accuracy drops from 35.7% to 13.6%. In plain English: the text gives you enough to guess the general motivational ballpark, but you need the images to nail the exact interpretation. Visual evidence provides disambiguating detail — a facial expression, a gesture, the spatial relationship between characters — that text narratives alone can't capture.

This has practical implications. For developers building multimodal AI applications, the finding suggests that if you need precision in social understanding, you cannot skip visual input. Text-only pipelines might get you partial coverage, but the error rate on exact interpretation will be 2–3× higher. For applications where misunderstanding motivation carries risk — mental health chatbots, automated HR tools, content moderation — that gap matters.

The EU angle: social reasoning AI and regulation

Here in Europe, this research lands at an interesting moment. The EU AI Act classifies AI systems by risk level, and a key factor in that classification is whether the system makes inferences about personal characteristics, emotions, or psychological states. MultivationBench effectively measures exactly this capability — and finds it severely lacking in current models. That creates a regulatory paradox: the systems most likely to be deployed for high-stakes social reasoning (in hiring, education, law enforcement) are simultaneously the least competent at it.

Under the AI Act's risk framework, systems used for emotion recognition in workplaces or education fall under high-risk classification, requiring conformity assessments, human oversight, and transparency obligations. Yet the benchmark shows that even the best multimodal models can't reliably distinguish between a character acting out of "Esteem" versus "Status" — categories that have real legal consequences in discrimination cases.

There's also a GDPR dimension. Article 22 of the GDPR gives EU citizens the right not to be subject to solely automated decisions producing legal effects. If an AI system's motivation inference influences a decision (say, flagging an employee as "disengaged" based on behavior analysis in workplace monitoring), the data subject has a right to human intervention and an explanation. Given that the best model gets motivation wrong 60%+ of the time on an exact-match basis, those explanations would frequently be incorrect.

For European companies building or deploying AI: if your product makes inferences about why users or employees behave a certain way, assume the model gets the exact motivation wrong most of the time. Plan your system design accordingly — human review gates, confidence thresholds, and transparency about the fallibility of these inferences are not optional additions; they're likely regulatory requirements.

What this means for the AI industry

MultivationBench exposes a specific failure mode that deserves more attention: the gap between instance-level accuracy and narrative-level consistency. Every model tested can occasionally guess a character's motivation from a single scene. But none can maintain a coherent understanding across a full story. This isn't just a benchmark limitation — it mirrors what we see in production AI systems that handle multi-turn conversations or long-form content. The model is right sentence by sentence but wrong about the overall picture.

The paper also points toward the next frontier: active motivation reasoning. Current benchmarks, including this one, treat the model as a passive observer. A genuinely capable social AI would need to do more — anticipate how motivations evolve, update earlier inferences after new evidence, and simulate how interventions might alter behavior. That's closer to what you'd need for an AI agent operating in the real world, not just classifying pre-recorded stories.

For developers, the practical takeaway is clear: if your application involves tracking user intent, emotional state, or behavioral motivation across time, expect current models to fail at narrative consistency. Build fallback mechanisms. Test with long-context scenarios, not just single-turn interactions. And if you're in the EU, factor in the regulatory dimension — the AI Act is watching what these systems claim to do versus what they actually can do.