We moved a chunk of our batch inference to spot A100s to cut costs and the savings are real, but the interruption rate has been brutal this quarter. Even with checkpointing, requeueing, and a fallback pool, tail latency spikes whenever capacity gets reclaimed. Curious how others handle this: are you running strict on-demand for anything user-facing and only using spot for offline jobs, or have you found a broker/placement strategy that actually holds up? Also wondering whether reserved capacity or newer marketplace options have gotten good enough to make spot a sane default for serving.
GPU spot instances: still too risky for production inference?
Replies
No comments yet — be the first to share your thoughts.