I've been testing a couple of the 200k-context models on real document sets, and the drop-off past roughly 80k tokens is steeper than the marketing implies. Needle-in-a-haystack scores look great, but ask for a synthesis across six long files and it starts missing details that sit right in the middle of the window. Curious whether others see the same, or if it's prompt structure on my end. Also wondering if the cheaper tiers are quietly truncating before the stated limit. Anyone have a benchmark that actually reflects multi-document work rather than single-needle recall?
Long context claims vs. what I actually get
Replies
No comments yet — be the first to share your thoughts.