Inference caching at scale: are we over-engineering?

We've been deploying a few large models behind an API, and I keep going back and forth on prompt caching strategies. On one hand, semantic caching can cut costs dramatically for repetitive workloads. On the other, the added complexity of invalidation and cache misses is starting to hurt our p99 latency. We're also seeing GPU memory pressure from keeping hot prefixes resident. Curious how others are handling this in production. Are you relying on framework-level caching or building your own? And at what point does the infrastructure overhead outweigh the inference savings? I'd love to hear some real-world metrics before we sink more time into this.