Been rerunning a few public evals with paraphrased prompts. Nothing semantic, just phrasing and answer-format instructions. The leaderboard order flips for models within a couple of points of each other. Standard deviation across prompt templates is often larger than the gap between ranked neighbours.
It feels like we keep reporting single-number scores that are really samples from
It feels like we keep reporting single-number scores that are really samples from