Running LLMs in production: real cost surprises?

We've been serving a fine-tuned 7B model for a few months now, and I'm still surprised where the money actually goes. GPU rental is obvious, but the hidden memory overhead for KV caches and the cost of duplicated model instances across regions really add up. We're using vLLM with continuous batching, which helped, but we're starting to think a lot more about cold starts and autoscaling for spiky traffic. Curious how others are handling this in production. Are you seeing more value in quantization, speculative decoding, or just sticking with cheaper hardware and accepting higher latency? Also, anyone actually moved away from pure managed APIs due to cost? Let's compare notes on what worked and what didn't.