Reproducing long-context needle tests is harder than it looks

I spent a week trying to replicate needle-in-a-haystack results from a recent paper and kept landing 10-15 points below their reported numbers. Turned out the haystack construction and where the needle gets inserted mattered more than I expected, and my tokenizer chunking differed slightly. Once I matched their exact prompt template and depth sampling, the gap mostly closed. It makes me wonder how many long-context claims in the wild hinge on unreported formatting choices. Has anyone else tried reproducing these evals? Curious whether fixed prompt suites would help, or whether that just invites contamination.