> ## Documentation Index > Fetch the complete documentation index at: https://docs.aireserve.com/llms.txt > Use this file to discover all available pages before exploring further. # Multi-node clusters > Launch 2–8 interconnected GPU nodes as one resource for distributed training and large-model inference. A cluster is `nodes` identical machines of `gpusPerNode` GPUs each, placed in one location on a private high-bandwidth interconnect, and managed as a single resource: one quote, one launch, one price, one terminate. Use it when a model does not fit one machine (DeepSeek-V3 or Kimi-K2 class) or for distributed training. The normalized contract guarantees private node connectivity, not a specific fabric such as InfiniBand, RDMA, or NVLink. It does not expose enough provider identity to verify one after placement. If a fabric is a hard requirement, deploy through the provider passthrough from the outset and verify it there. ## What every node gets | On every node | Value | | - | - | | `MASTER_ADDR` / `PRIMARY_ADDR` | Address of the rank-0 node on the interconnect | | `MASTER_PORT` / `PRIMARY_PORT` | Rendezvous port; every port is open between nodes | | `NODE_RANK` | This node's global rank; `0` is the primary | | `NCCL_SOCKET_IFNAME` | `ens1`, so NCCL uses the interconnect, not the public interface | | Image, env, ports, SSH keys, start command | Identical on every node | | Shared volume | One persistent volume mounted at the same path on every node | Set `MALLOC_ARENA_MAX=1` in `env` for tensor-parallel workloads. The provider-agnostic cluster contract does not currently expose `/dev/shm` sizing; workloads that require a larger shared-memory mount need a provider-native deployment path. ## Shapes Self-serve clusters are 2 to 8 nodes of 1 to 8 GPUs each (16 to 64 GPUs at full width). Larger clusters are arranged through sales. `gpusPerNode` cannot exceed what the accelerator offers per machine. `orchestrator` picks the topology: | `orchestrator` | What you get | | - | - | | `none` | Bare nodes with the variables above; bring your own launcher (default) | | `slurm` | A managed Slurm controller on rank 0 and compute nodes elsewhere | | `ray` | A managed Ray head on rank 0 and workers elsewhere | ## Quote, then launch **1. Quote the shape** ```bash theme={"dark"} aireserve gpu clusters quote \ --accelerator H200_141GB \ --nodes 2 \ --gpus-per-node 8 \ --persistent-gb 1000 \ --max-hourly-price 100 ``` The quote prices the whole cluster: every node's GPUs, every root disk, and the shared volume. Inter-node networking is included in the rate. `availability` is the stock signal for one node of this shape; whether every node can be placed together is confirmed only at launch. **2. Launch from the quote** ```bash theme={"dark"} aireserve gpu clusters launch --quote-id \ --env-file ./train.env \ --ssh-key-id \ --wait ``` Workload settings (`env`, `ports`, SSH keys, `dockerStartCmd`) can be added at launch without changing the price. Repeating a priced field with a different value answers `409 price_changed`. **3. Connect** ```bash theme={"dark"} aireserve gpu clusters connections aireserve gpu clusters logs --node 1 ``` `connections` lists the primary node's SSH, every node's SSH and its address on the interconnect. Logs stream per node; rank 0 by default. Direct REST: ```http theme={"dark"} POST /v1/gpu/clusters/quote Content-Type: application/json { "accelerator": "H200_141GB", "nodes": 2, "gpusPerNode": 8, "storage": { "persistentGb": 1000 }, "maxHourlyPrice": "100" } ``` ```http theme={"dark"} POST /v1/gpu/clusters Idempotency-Key: launch-kimi-2026-10-13 Content-Type: application/json { "quoteId": "", "env": { "HF_TOKEN": "..." }, "sshKeyIds": [""] } ``` ## Storage A cluster mounts at most one persistent volume, shared by every node at the same path. Create it with the cluster (`storage`) so it lands in the cluster's location, or mount an existing one (`volumeId`), which pins placement to where that volume lives. The volume survives terminate and bills until you delete it; you cannot delete or grow it while a cluster is attached. Weights for a 600 GB to 1 TB model take 35 to 70 minutes to download at typical rates. Put them on the shared volume once and relaunch against it. ## Lifecycle | Status | Meaning | | - | - | | `provisioning` | The provider is placing nodes | | `running` | Every node verified running on the expected hardware; billing starts here (`readyAt`) | | `degraded` | Some nodes left `running` after the cluster was admitted. Billing continues; recover or terminate | | `failed` | The cluster never came up (see `lastError`); nothing is billed | | `terminated` | Every node released; only the shared volume keeps billing | `running` is infrastructure readiness. It does not probe a model server, distributed rendezvous, or application health endpoint. Make the shared command rank-aware and run your own readiness check before admitting traffic. There is no stop. A cluster bills from `readyAt` until you terminate it: ```bash theme={"dark"} aireserve gpu clusters terminate --wait ``` Terminate is always accepted, even mid-provision. If payment grace expires on an unpaid hour, the cluster is terminated (not stopped); the shared volume is kept. ## Recover a degraded cluster A cluster that loses a node after it was admitted goes `degraded` and keeps billing at the locked rate until you act. By default AI Reserve leaves it for you. Opt into automatic recovery per cluster: ```bash theme={"dark"} aireserve gpu clusters recovery --mode restart --max-attempts 3 --cooldown-minutes 5 ``` On the next health pass every node that is not running is brought back one by one through the provider's own restart or start, on the same hardware, with the same interconnect address. Healthy nodes, the shared volume and the rate are untouched, so a distributed job that tolerates a node restart can resume without a relaunch. Attempts are bounded per unhealthy episode exactly as for instances, the `recovery` object on the cluster reports the episode, and `nodes[].status` shows each member. Make one attempt now with: ```bash theme={"dark"} aireserve gpu clusters recover --wait ``` A node the provider cannot bring back is skipped and the attempt resolves on the remaining nodes; once every attempt is spent the cluster stays `degraded` with `lastError: recovery_exhausted` and terminate is the way out. ## Pricing ``` usdPerHour = gpusPerNode × nodes × perGpuUsdPerHour + diskGbPerNode × nodes × rootDiskRate ``` Per-GPU pricing uses the provider's cluster rate, which differs from the single-machine rate for the same accelerator. The whole cluster accrues hourly while any node runs, at the rate locked in the quote. ## Errors | When | Error | What to do | | - | - | - | | At quote | `409 no_capacity` with `reason` and the shape | Try fewer GPUs per node, another region, or a higher ceiling | | At launch | `409 price_changed` | Request a fresh quote | | At launch | `403 quota_exceeded` | Nodes count as machines and every GPU counts; raise org limits | | After accept | operation `failed`, `provider_rejected:no_capacity` | Every node could not be placed together after retries; nothing billed. Re-quote | | After accept | operation `failed`, `provider_rejected:validation` | The image or request was refused; fix and relaunch | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.