The end of the popularity contest: How Arena is now measuring factual accuracy

Arena.ai, formerly known as LMSYS Chatbot Arena, has been the most cited independent leaderboard for large language models for years. Its huge advantage is the "blind testing" method — users compare two anonymous responses and vote for the better one without knowing which model generated it. It is precisely thanks to this approach that Arena has become a kind of public judge of AI quality.

But now the platform is adding a fundamental innovation: factual accuracy. Each model response is automatically broken down into so-called atomic verifiable claims — the smallest possible statements whose truthfulness can be independently verified. Search agents then assign a calibrated probability of truthfulness to each claim. In practice, this means that a text response contains about five verifiable claims on average, while a web search response has around ten.

The entire system is built on a massive dataset: over 2 million labeled claims, sourced from more than 130,000 text battles and 40,000 battles in the search category. Between 76 and 88 percent of all battles contain at least one verifiable claim, and across models, the rate of truthful claims ranges between 87 and 89 percent.

The results of these "factual battles" are then combined with traditional user votes using the statistical Bradley–Terry model. The default weight of factual accuracy is set at 25 percent — the rest is made up of human preferences. Yet even this relatively small weight was enough to dramatically shuffle the model rankings.

Who won and who tanked

The biggest climber is GPT-5.5 by OpenAI, which jumped 13 positions to claim seventh place. This jump suggests that the model was systematically undervalued in purely preference-based voting — users may have perceived it as less entertaining or less eloquent, but its claims held up much better under factual verification than it appeared.

Conversely, Meta Muse Spark, which was in seventh place on the preference leaderboard, plummeted 13 positions to twentieth place. This is particularly striking — a model that users consistently preferred in direct comparison turned out to be significantly less reliable when it came to verifying its claims. In other words: Muse Spark excels at persuading, but doesn't always stand on solid facts.

Not even Claude Fable 5 by Anthropic, the previous leader of the overall text leaderboard, emerged from the change unscathed — it moved to second place. OpenAI models generally maintained or improved their positions as the weight of factual accuracy increased, which speaks to more consistent fact-handling across their product line.

Why it matters — and not just for researchers

This change has far-reaching implications. Above all, it shows that human judgment is no longer enough. As model responses become increasingly sophisticated, fluid, and confident, human evaluators find it harder and harder to recognize factual errors hidden in perfectly sounding text. In doing so, Arena is essentially acknowledging what many in the field suspected: we need machines to check machines.

For companies building their products on these models, this is critical information. Imagine a bank using AI for risk analysis, or a hospital deploying a model to support diagnostics. In these scenarios, the fluency of the response is irrelevant — what matters is whether the model tells the truth. And right now, thanks to Arena, they have a public, transparent, data-driven tool to compare the factual reliability of individual models.

The dataset of two million labeled claims also covers topics from regulated industries such as finance, healthcare, and legal services — precisely the areas where factual errors carry direct and often serious consequences.

What it means for Czech companies and users

For Czech developers and companies integrating AI models into their applications, the new leaderboard offers concrete guidance. If you're building a chatbot for customer support in Czech, you'll probably appreciate a model that communicates convincingly. But if you're developing a tool for analyzing legal documents or financial statements, you primarily need factual accuracy — and the new leaderboard helps you pick the right model.

All the major models mentioned on the leaderboard — GPT-5.5, Claude, and Gemini — support Czech at a very good level. GPT-5.5 is available through ChatGPT (including the free tier with limits), Claude Fable 5 via Claude.ai (with a Claude Pro subscription at approximately 500 CZK per month), and Gemini models through Google AI. Differences in the quality of Czech responses do exist among these models, but the new factual leaderboard adds an entirely new dimension of comparison — not how nicely the model speaks Czech, but how truthful its answers are.

The Bradley–Terry model: the math that decides

For more technically inclined readers, let us add that Arena uses the Bradley–Terry model to combine preference votes and factual scores — a statistical method originally developed for pairwise comparisons in psychometrics and sports analytics. In the context of AI leaderboards, this model estimates the "strength" of each model based on the results of all pairwise battles. Now, the dimension of factual accuracy has also entered the calculation, making it the first truly multidimensional evaluation system in the history of AI benchmarking.