How are you handling GPU idle time between training runs?

We keep seeing 30-40% GPU idle time between jobs, mostly from data staging and environment setup. Scheduler thinks nodes are busy, but utilization tells a different story. Anyone found a practical fix that doesn't involve rewriting the whole pipeline? Curious whether gang scheduling or warm node pools actually moved the needle for you, or if the real win was just tighter checkpointing and smaller jobs. Also interested in how you're measuring this without drowning in dashboards.