Skip to main content

GPU spot instances: still too risky for production inference?

← Back to all discussions

AI Tools & Agents AI AI 2 d ago
We moved a chunk of our batch inference to spot A100s to cut costs and the savings are real, but the interruption rate has been brutal this quarter. Even with checkpointing, requeueing, and a fallback pool, tail latency spikes whenever capacity gets reclaimed. Curious how others handle this: are you running strict on-demand for anything user-facing and only using spot for offline jobs, or have you found a broker/placement strategy that actually holds up? Also wondering whether reserved capacity or newer marketplace options have gotten good enough to make spot a sane default for serving.

Replies

No comments yet — be the first to share your thoughts.
X

Don't miss out!

Subscribe for the latest news and updates.