Raw rasters instead of tidy mosaics
Most land-change evaluations hand a model preprocessed imagery: already scaled, already masked, already reprojected. EarthRecord, posted to arXiv on 6 November 2026 as 2611.12934, does the opposite. The agent gets the archive as it comes from Landsat TM, ETM+ and OLI, MODIS, VIIRS and Sentinel-2, and has to do cloud masking, radiometric scaling, reprojection and cross-sensor harmonisation itself.
That order of operations matters. If the agent skips a scaling step in year three of a series, the answer breaks two steps later and the traceback is buried in a raster stack. Haoran Zhang, Huizhen Sun, Hanbin Wen, Jie Leng, Jiaxin Zhu and Zhaohua Sun built the thing across AIEyes Technologies, HKUST (Guangzhou), GDUT, Anhua Ocean Intelligent Equipment and the University of Hong Kong. Code sits on GitHub, data and questions on Hugging Face.
The site list and the question set
Fifty-six sites. Forty-eight core areas, each 10 to 30 km per side, spread over six continents and all five Köppen climate groups. Eight are famous, seven are stable, thirty-three are obscure on purpose, so a model cannot coast on prior knowledge of the Amazon or the Aral Sea. Eight administrative regions are added on top.
The coverage runs from 1984 to 2026, with a median of 1,101 Landsat scenes per core area and 109 questions that span sensor generations. Across 238 questions in ten analysis tasks, 153 are answerable by computing over the record, 73 are pure visual recognition, and 12 cannot be answered from the archive at all. Those 12 are a calibration test: does the agent say the evidence is missing, or does it invent a plausible number?
Two hundred and twenty-six of the 238 questions carry a score. The other twelve measure restraint, not accuracy.
Where the models land
Scoring uses pass^3, meaning the same answer has to come out of three separate runs inside a harness that allows up to 3,600 seconds and 300 messages per run. On the 226 scored tasks, four systems occupy a single band.
| System | pass^3 on 226 tasks |
|---|---|
| Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Sol, GPT-6.1 Sol | 81.0% – 85.4% |
| Majority-class baseline | 23.9% |
| GPT-6 Luna | 18.6% |
| Qwen 3.7 Flash | below the baseline |
The spread between the top of that band and GPT-6 Luna is 66.8 percentage points. A cheap model that lands 5.3 points under the majority-class baseline is not just weaker, it is answering in a way that a constant guess would beat. Mapping from imagery, not question interpretation, is named in the paper as the main bottleneck.
Instrument pairs and what happens when one sensor lies
The interesting part of the methodology is the stress test. The authors built 22 instrument pairs, plus 8 placebo pairs as a control. In an instrument pair, the data from a single satellite is altered while the ground truth stays untouched. A model that genuinely cross-checks sensors should notice the disagreement and hold its original answer.
Roughly half of the correct answers followed the alteration instead. The agent read the raw pixels it was given, treated them as fact, and dropped the cross-validation step it had been asked to perform. That is a failure mode worth naming precisely: the models are not bad at satellite imagery in general, they are bad at doubting a single source when a second one is sitting in the same directory.
The European half of the archive
One of the four sensor families in EarthRecord is ESA's Sentinel-2, part of the Copernicus programme. Its data is free and open, which means a team in Prague or Lisbon can reproduce a large slice of this benchmark without a licence fee or a data-sharing agreement. Access runs through the Copernicus Data Space Ecosystem.
There is a useful contrast here. Europe already publishes ready-made Copernicus Land Monitoring Service high-resolution layers built on exactly this imagery. Those layers are the preprocessed shortcut EarthRecord refuses to give the agent. So the benchmark measures something the existing European products do not: whether an autonomous pipeline can reconstruct a defensible answer from the raw archive, with its working visible.
One regulatory detail is worth flagging. Since 2 August 2026, Article 50 of the EU AI Act requires machine-readable labelling of AI-generated output. A land-change report produced by an agent pipeline and published by a European authority or consultancy falls inside that scope. The 12 insufficient-evidence questions and the instrument pairs both test whether the agent can show where an answer came from.
What it takes to run it
The evaluation executes in Docker and needs 4 CPUs and 24 GB of RAM per concurrent run. No GPU requirement is listed. A machine with 32 GB of system memory runs one evaluation at a time, 64 GB gives you two. On a rig like our AI Arena setup with an RTX 5060 Ti 16 GB, the graphics card would sit idle for the full 3,600-second ceiling; the constraint is system RAM, and the tool-calling loop that drives the agent.
That design makes the benchmark model-agnostic in a practical sense. Any agent that can call a shell, write Python and open an image can be pointed at it, cloud or local.
What the paper does not settle
The archive snapshot stops on 30 September 2026, so any event after that date is outside the questions. The score band of 81.0% to 85.4% covers four systems, which means the benchmark currently separates tiers of models rather than individual frontier releases. And the instrument-pair result is reported as a rough proportion, not a per-model breakdown of which systems resisted the alteration and which did not.
Do I need a GPU to evaluate a model on EarthRecord?
No. The published execution requirement is 4 CPUs and 24 GB of RAM per concurrent run inside Docker. The agent drives Python, a shell and an image viewer, so the wall-clock limit of 3,600 seconds per run is met with CPU and memory, not VRAM.
Can open-weight models be tested on it?
The paper already reports Qwen 3.7 Flash, which finished under the majority-class baseline of 23.9%. Any model with working tool calls can be wired into the harness, so Mistral Large 4 or Meta's Muse Spark Open are candidates for a local reproduction, though no results for them appear in the paper.
Why does the dataset include eight "famous" sites if the point is avoiding prior knowledge?
They act as a control. The authors separate 8 famous, 7 stable and 33 obscure core areas, and run a closed-book setting without imagery to measure how much a model already knows. The gap between the closed-book score and the full score on obscure sites is what shows whether the agent actually read the pixels.
