SpotWarp Documentation for SaladCloud

Introduction

SpotWarp is a lightweight, 100% local Python autopilot daemon that runs continuous background backups of your SaladCloud container workspace to your own machine the whole time the container runs — so losing a node never costs you your saved training progress, not just downtime.

SpotWarp supports SaladCloud only (Same-Cloud recovery, Salad to Salad). It checks your container group every 5 seconds and, by default, scales it to 2 replicas so a warm standby is already running (turn this off with --no-standby). When your active node is lost, SpotWarp restores your already-backed-up workspace onto the replacement and runs your --resume-cmd. In our live test on 2026-10-21 this took 17.1 seconds with the standby ready; without a standby (--no-standby), SaladCloud has to allocate a new node first, which takes minutes.

Your SaladCloud Setup

SpotWarp does not rent, resell, or hold any GPU inventory — we never broker compute. You run your containers directly on SaladCloud, and SpotWarp is the local daemon that watches and protects them.

1. Create your container group in the SaladCloud portal or API. Our ready-made recipe is in the GitHub repository (ghcr.io/enplabs/spotwarp:pytorch2.1, PyTorch 2.1 / CUDA 12.1).
2. Generate a SaladCloud API key and note your organization name and project name (the default project is default).
3. Make sure SSH access to your container works from your machine. SpotWarp uses SSH to copy files and run your resume command; the default key is ~/.ssh/id_ed25519, or pass --ssh-key.

Installation

Install SpotWarp on your local workstation or orchestrator. Python is completely optional — our standalone C-binaries run with zero external dependencies.

Option A: Universal 1-Liner (Recommended)

Copy# Linux / macOS
curl -fsSL https://gpu-action.com/install.sh | bash

# Windows PowerShell
irm https://gpu-action.com/install.ps1 | iex

Option B: Standalone C-Binaries (Direct Download)

Pre-compiled standalone binaries with embedded C-engine, published on GitHub Releases with SHA256 checksums you can verify:

• Linux x86_64: spotwarp-linux-x86_64 (13MB, glibc 2.31+)
• macOS: spotwarp-darwin-arm64 (6MB, Apple Silicon)
• Windows: spotwarp-windows-x64.zip (6MB) — the same released Windows binary, served from this site as a ZIP because browsers and antivirus commonly block a bare .exe download. Extract it and you get spotwarp.exe, byte-identical to spotwarp-windows-x64.exe on the release page.

Option C: PyPI Package (For Docker / Python Workflows)

Copypip install --upgrade spotwarp

Windows Backup Transport

SpotWarp uses rsync for incremental workspace backups when it is available (Linux/macOS). On Windows, it uses scp (bundled with Windows' native OpenSSH client), which copies the whole workspace each cycle, so large workspaces back up more slowly there.

Quick Setup Wizard

Run the one-time interactive setup wizard to save your settings locally to ~/.spotwarp/config.json:

Copyspotwarp init

The wizard will prompt you for your SpotWarp License Key (press Enter for the built-in 14-day evaluation mode), your SaladCloud API key, organization name, project name, an optional container group name, the local backup directory, and whether to keep a warm standby replica. Once configured, you never need to export environment variables or pass keys via CLI again.

Alternative: Manual Environment Variables

If you prefer not to use a local config file (e.g. inside ephemeral CI/CD pipelines), you can still export keys via standard environment variables:

Copy# Linux / macOS
export SALAD_API_KEY="your_salad_api_key"
export SALAD_ORG="your_org_name"

# Windows PowerShell
$env:SALAD_API_KEY="your_salad_api_key"
$env:SALAD_ORG="your_org_name"

Start Guard Daemon

Once configured, start protecting your workloads with a single command:

Option A: Live Console Mode

Copyspotwarp start

Option B: 24/7 Background Daemon Mode

Run SpotWarp in detached background mode so you can safely close your terminal window:

Copyspotwarp start -d

Daemon Management Commands

Copyspotwarp status   # Check if background guard is alive
spotwarp stop     # Gracefully stop the background guard
spotwarp config   # View your current local configuration

Custom Backup Location

By default, workspace backups are written to ./backups/<container_group_name>/. To use a different drive, pass --backup-dir:

Copyspotwarp start --backup-dir "D:\SpotwarpBackups"

Stateful Auto-Resume

You can configure the daemon to automatically resume PyTorch/TensorFlow training scripts on the replacement node pointing to your synced checkpoints:

Copyspotwarp start --license-key YOUR_LICENSE_KEY --resume-cmd "python /workspace/train.py --resume"

When your active node is lost, SpotWarp waits for a running replacement in the same container group, checks that SSH is ready, restores your backed-up /workspace files, and executes the --resume-cmd in the background over SSH with nohup. If a replacement is stuck downloading the image, SpotWarp asks SaladCloud to reallocate it.

Security & Sandbox

License checks and status heartbeats are sent to gpu-action.com over HTTPS.

The CLI daemon runs strictly in user space on your local computer. It does not require root/sudo access, and can run inside an isolated container. Your SaladCloud API key is read locally and used only to call the SaladCloud API from your machine.

Troubleshooting

"License Invalid or Expired" or "License server returned HTTP 401"

Your --license-key value doesn't match an active license. This is almost always a copy-paste issue — a prefix got dropped, or two keys got concatenated together. Copy the full key directly from your confirmation email and paste it in one piece. Still stuck? Email [email protected] and we'll look it up.

"Protected Containers: 0"

The guard has not found a running SaladCloud instance yet. Check that your container group is in the running state, that SALAD_API_KEY and SALAD_ORG (or the values saved by spotwarp init) are correct, and that the project name matches (the default is default). If you use environment variables, they must be set in the same terminal window you ran spotwarp start in.

Backups are not appearing

A freshly started container can take a minute or so after it shows as running before its SSH access is ready. SpotWarp retries every 30 seconds and does not print a console message for a single failed sync. If nothing appears after several minutes, check that you can reach the container yourself with ssh -p <port> root@<host> using the same key, and that /workspace/ exists inside the container.

Where are my backups?

Locally, in .\backups\<container_group_name>\ relative to wherever you ran spotwarp start from — unless you set --backup-dir or SPOTWARP_BACKUP_DIR (see Start Guard Daemon), in which case it's there instead. Nothing is ever uploaded off your machine.

Still stuck?

Email [email protected] with your license key and the console output around the issue — we read every message.