Inference cost optimization strategies for production LLMs

We are burning through our inference budget faster than expected now that our LLM features are in production. We already use continuous batching and FP16, but costs are still high. I am curious what others are doing in practice: are you seeing real gains from quantization like INT8 or FP8, or is speculative decoding more impactful? Also, how are you balancing cost against p95 latency for user-facing features? We are considering moving some traffic to smaller distilled models with a fallback to the larger model for harder requests. Would love to hear what infrastructure choices made the biggest dent in your monthly bill without killing reliability.