GPU idle time is quietly eating our inference budget

We run a mid-sized inference cluster and recently instrumented GPU utilization per model endpoint. The results were rough: average SM occupancy sat around 34%, with long idle gaps between batches at low-traffic hours. Nobody owned this metric, so it went unnoticed for months. We're now weighing a few options: request coalescing with a short queueing window, moving cold models to CPU or spot instances, and consolidating replicas behind a smarter router. Curious what others have tried. Has anyone measured real dollar savings from batching versus just over-provisioning for latency headroom? Also interested in whether autoscaling on queue depth rather than GPU utilization worked better in practice.