SaladCloud is cheap because it runs on consumer GPUs that can disconnect at any moment. For inference that is mostly fine. For a fine-tuning run that takes many hours, the real cost of a lost node is everything your job did since the last time it saved its state.
What SaladCloud itself recommends
SaladCloud's own guidance for long-running jobs is clear: save your progress regularly to cloud storage, and resume from the last saved state when the job starts again on a new node. It describes the trade-off honestly. Saving often costs upload bandwidth, saving rarely costs more lost work. Its documentation also notes that most nodes can run continuously for over 10 hours, and that failure risk rises for jobs that run longer than a day.
For jobs over 12 hours, SaladCloud points to Kelpie, its job tool with built-in data sync to S3-compatible storage and no limit on node reallocations. If you already write checkpoints to R2 or S3 and run your job through Kelpie or a queue, you have a solid setup and may not need anything else.
The part nobody can do for you
Every option below, ours included, depends on one thing: your training script must save checkpoints and be able to resume from one. No tool can recover progress that was never written to disk. If your script only saves at the end, fix that first. Most frameworks make it simple; Hugging Face's Trainer, for example, can resume optimizer and scheduler state from a checkpoint folder.
Your options, side by side
| Approach | What it needs | Trade-off |
|---|---|---|
| Save to R2/S3 yourself | Upload and download code in your job, a storage bucket | Most flexible; you write and maintain the sync and resume logic |
| Kelpie (SaladCloud) | Kelpie set up with your storage | Built for long jobs on SaladCloud; tied to its job model |
| SpotWarp | A machine of yours that stays on, SSH access to the container | No bucket and no sync code; copies go through your own connection |
| Do nothing | Nothing | Fine for short or stateless jobs; a lost node means starting over |
Where SpotWarp fits
SpotWarp is a small client that runs on your own machine. It checks your SaladCloud container group every 5 seconds and copies the container's /workspace to your machine in the background, skipping files written in the last 10 seconds so it does not copy a half-written checkpoint. When the node is lost, it waits for the replacement node in the same container group, restores the workspace, and runs the resume command you gave it. Your SaladCloud API key stays on your machine.
By default it keeps a second replica running as a warm standby, so a replacement node is already up. In one live test on October 21, 2026, with a 5 MB checkpoint and the standby already running, the workspace was restored and the resume command started 17.1 seconds after we forced the node away through the API, and the file hash matched. That was a single run. Without a standby, recovery has to wait for SaladCloud to allocate a new node, which takes minutes.
What it costs and what it does not do
- The warm standby is a second running replica, so it roughly doubles the container cost while it runs. You can turn it off with
--no-standby. - Your machine has to stay on, and checkpoints travel over your own internet connection. Very large checkpoints will take a while.
- On Windows, backups copy the whole workspace each cycle; Linux and macOS copy only what changed.
- It does not write checkpoints for you, and there is no alerting yet if a recovery fails.
The honest recommendation
If you are comfortable with cloud storage, SaladCloud's own pattern of checkpoints in R2 or S3, with Kelpie for very long jobs, is a good answer. SpotWarp is for people who would rather not set up a bucket and sync code and want the restore and restart handled for them. Read the source and decide.
Sources: SaladCloud long-running tasks guide, and our own test on October 21, 2026.