We build a recovery tool, so it may be surprising to read this from us: most image generation workloads on SaladCloud do not need one.

Inference has nothing to recover

A ComfyUI or image generation API turns a request into an image and keeps no progress between requests. If a node disappears, the only thing lost is the request it was working on. Nothing on that node needs to be restored.

The better pattern is the one SaladCloud is built around:

  • Run more than one replica. The Container Gateway only sends traffic to instances that are ready, so a lost node simply stops receiving requests.
  • Use a readiness probe that reflects real readiness, so traffic does not arrive before your model is loaded.
  • Put jobs in a queue when they take longer than a single HTTP request. SaladCloud's job queue retries a job on another node if one fails.
  • Retry from the client for the rare request that was in flight when a node left.

One practical detail from our own testing: if your server sits behind the Container Gateway, make sure it listens on ::, not only 0.0.0.0. The gateway connects over IPv6, and a server bound only to IPv4 answers 503 through the gateway. We wrote up that and other recipe lessons here.

Where a recovery tool earns its place

The picture changes when a single job holds hours of progress on one node: LoRA or model fine-tuning, long batch renders, or anything that builds up state as it runs. Losing that node loses the work since the last checkpoint, and a replica does not help because the progress lived on the node that left.

That is the case SpotWarp is for. It copies the container's /workspace to your own machine while the job runs and, when the node is lost, restores it onto the replacement node and runs your resume command. Your job still has to save checkpoints; SpotWarp moves them and restarts the job. We explain the options in more detail in this guide.

Quick test

Ask one question: if this node disappeared right now, would I lose anything besides the request in progress? If the answer is no, use replicas and a queue. If the answer is hours of training, protect the checkpoints.

Sources: SaladCloud troubleshooting guide, SaladCloud long-running tasks guide.