Long-context evals may just be testing retrieval

Been reproducing a few long-context benchmarks on smaller open models and the numbers look strange. Once I control for where the answer sits in the prompt, most reported gains shrink a lot. It reads more like local retrieval than reasoning across the whole window. Has anyone else tried ablating needle position and distractor density? Also interested in contamination checks that actually work on these sets, since half the tasks feel familiar. Happy to share scripts and raw runs if people want to compare notes.