Why Classic Benchmarks Fall Short
Until recently, the capabilities of large language models were measured primarily using tests like MMLU or HumanEval. However, these evaluate isolated knowledge or short snippets of code. With the advent of agentic AI — systems that independently plan, use tools, and interact with the real world — these metrics ceased to be sufficient.
Agent benchmarks bring a fundamental shift: instead of questions and answers, they test models in long, multi-step scenarios. They must understand the assignment, search for information, run code, click on the web, or communicate with the user. And what is key — the result is verified automatically, not subjectively.
"Agent benchmark scores are, however, heavily dependent on test setup," warns an analyst from the MarkTechPost portal. The model, prompt, access to tools, number of attempts, and even evaluator version can significantly change the results. No number should therefore be read in isolation.
1. SWE-bench Verified — When AI Fixes Real Code
The first and most watched benchmark tests whether an agent can solve real-world problems from GitHub. SWE-bench Verified contains 500 verified tasks from twelve popular Python repositories. The agent must produce a working patch — not just describe the solution, but actual code that passes unit tests.
When the benchmark launched in 2023, the Claude 2 model managed only 1.96% of tasks. Results from the turn of 2025 and 2026 show that top models are moving above 80% on SWE-bench Verified — exact values vary, however, depending on the agent framework and tool setup. Generally speaking, closed models lead over open-source ones, and the model itself is only one part of the success.
For Czech developers and companies, this benchmark is a key indicator: a model that performs well on SWE-bench can significantly speed up code maintenance, refactoring, or bug fixes in internal systems.
2. GAIA — An Assistant That Doesn't Cheat
GAIA tests general assistant capabilities: multi-step reasoning, web browsing, tool use, and basic multimodal understanding. Tasks are simply formulated, but solving them requires a chain of non-trivial operations — exactly what a real digital assistant encounters.
GAIA is widely used in research and maintains an active leaderboard on Hugging Face. Its design resists shortcuts: the agent cannot "guess" its way to the result. For teams developing universal assistants, GAIA is one of the most reliable sources of feedback.
3. WebArena — Autonomy on the Real Web
WebArena creates functional websites across four domains: e-commerce, social forums, developer tools, and content management. The agent receives natural language instructions and must carry them out exclusively through a live browser. The benchmark contains 812 long tasks. The original best GPT-4 agent achieved only 14.41% success rate, while the human baseline is 78.24%.
By the beginning of 2025, the situation improved: the specialized IBM CUGA system reached 61.7% and the OpenAI Computer-Using Agent reported 58.1%. This progress reflects stronger planning, specialized action modules, and state tracking. WebArena is today the standard for testing true web autonomy, not just scripted automation.
4. τ-bench — When Reliability Fails
While many benchmarks evaluate one-time success, τ-bench (Tau-bench) tests something far more practical: consistency. It simulates multi-turn conversations between a user and an agent equipped with domain APIs and rules. At the same time, it evaluates three things — the ability to obtain information, rule compliance, and repeatability of results.
The results are alarming: even top agents like GPT-4o successfully handle less than 50% of tasks. And their consistency is even worse — the pass^8 metric in the retail domain drops below 25%. This means that an agent that manages a task once cannot reliably repeat it eight times in a row. For real-world deployment in call centers or customer support, where millions of interactions take place, this unreliability is critical.
5. ARC-AGI-2 and ARC-AGI-3 — Measuring True Intelligence
ARC-AGI, created by François Chollet, tests fluid intelligence: the ability to generalize to completely new visual tasks that resist memorization. ARC-AGI-1 was practically saturated by 2025 — top models reached over 90%. ARC-AGI-2, released in March 2025, closed these gaps.
The ARC Prize 2025 competition on Kaggle attracted 1,455 teams. The best competitive score reached 24% using the specialized NVIDIA NVARC system. Among commercial models, the situation is rapidly evolving: GPT-5.2 reached 52.9%, Claude Opus 4.6 68.8%, and Gemini 3.1 Pro after the February 2026 version 77.1% — more than double its predecessor.
The biggest challenge is ARC-AGI-3, launched in March 2026. It is an interactive video game where the agent must explore a new environment, infer goals, and plan actions without explicit instructions. The technical report states: humans handle 100% of environments, while top AI systems in March 2026 achieve below 1%. Four major labs — Anthropic, Google DeepMind, OpenAI, and xAI — have included ARC-AGI in their official model cards.
6. OSWorld — Controlling a Real Computer
OSWorld offers 369 tasks across web and desktop applications, the file system, and cross-application workflows on Ubuntu, Windows, and macOS. The agent must interact through an actual graphical interface — keyboard and mouse, not pure APIs.
At the time of publication at NeurIPS 2024, humans handled over 72% of tasks, while the best model achieved only 12.24%. Since then, the benchmark has been upgraded to OSWorld-Verified, which fixes hundreds of reported issues and improves evaluation reliability. Multimodal demands — the combination of visual perception, operating system knowledge, and planning — make OSWorld substantially more challenging than purely text-based benchmarks.
7. AgentBench — Diagnostics Across Worlds
Last but not least, AgentBench tests breadth. It evaluates models in eight different environments: OS interaction, database queries, knowledge graph navigation, card games, puzzles, household task planning, web shopping, and web browsing. Instead of depth in one domain, it examines how well a model adapts across completely different tasks.
AgentBench is best for comparing architectures and revealing where skill transfer stops working. A model that excels on SWE-bench, for example, may completely fail in a database environment. This cross-domain overview has no parallel in this group of seven.
What This Means for the Czech Republic and Europe
For Czech companies and developers, these benchmarks are a practical compass when selecting models for automation. While closed models generally lead, open-source alternatives are rapidly catching up especially in narrow domains — and these may be more affordable and better customizable for Czech businesses.
From a European perspective, there is also a regulatory dimension: the AI Act requires high-impact systems to be transparent and verifiable. Agent benchmarks provide exactly such an objective metric. For Czech development teams that want to deploy agent systems in production, it is crucial to understand that no single test tells the whole truth — and that reliability is often a bigger problem than peak performance in one attempt.
None of the benchmarks mentioned are localized into Czech, but their methodology and some open-source implementations can be freely used by Czech researchers and companies for their own evaluations. Combined with Czech datasets, local variants of tests relevant to the Czech language and market could emerge.
Why do agent benchmarks often report different scores for the same model?
The result depends on the so-called scaffold — that is, the prompt, available tools, number of attempts, evaluator version, and other settings. Two teams may test the same model and arrive at significantly different numbers. Therefore, it is important to read not only the score but also the test conditions.
Can a Czech company run these benchmarks itself?
Yes. Most benchmarks, including GAIA, WebArena, OSWorld, and AgentBench, are open-source and available on GitHub. However, they require technical knowledge of Python and often access to powerful GPUs. For companies without their own infrastructure, it may be more practical to follow public leaderboards or use third-party services.
Why is ARC-AGI-3 so difficult when models achieve high scores elsewhere?
ARC-AGI-3 tests interactive adaptation in an unknown environment without explicit instructions. While other benchmarks evaluate the execution of a known task, ARC-AGI-3 measures the ability to independently discover rules and goals. This is a fundamentally different challenge for current models — and that is why it is considered the purest measure of generalization.