Skip to main content
This playbook serves moonshotai/Kimi-K3 with SGLang on one 8×B300 node. That is the smallest topology whose unified low-latency and balanced paths are marked verified by the SGLang maintainers.
Vendor-verified · AI Reserve qualification pending. SGLang’s verified B300 result used revision v0.5.18 @ 71de97b2, but its public NVIDIA image reference is still floating. Resolve lmsysorg/sglang:latest to the digest you qualify and store that digest in the deployment record before production.

Recipe contract

Kimi K3 always reasons. It also requires the complete prior assistant message— including reasoning_content and tool_calls—on every multi-turn or tool follow-up. Dropping those fields changes the conversation contract.

0. Set up your shell

Every command on this page is copy-paste. Paste this block once per terminal and replace only the API key; every later step reads these variables.
Two keys, two jobs. AIRESERVE_API_KEY authenticates you to AI Reserve and creates the volume, quote, and instance. SGLANG_API_KEY is a password you invent for the SGLang server you are about to start: step 4 passes it as SGLang’s --api-key, so the public HTTP port rejects anyone who does not send it. No vendor issues it and AI Reserve never sees it.
The Python and TypeScript tabs read the same three variables from the environment. The shell tabs need curl and jq; the Python tabs need requests (python -m pip install requests).

Check stock first

A quote is free and creates nothing. Run one before creating storage, because the volume in step 1 is placed next to this hardware and fails the same way when none is in stock:
A response with availability and price means you can continue. An error with code: no_capacity means this shape is not in stock in that region right now: try another region here and in steps 1 and 2, or wait, and read no_capacity reasons.

1. Create the weights volume

Available now. The accelerator hint places the volume in a data center where 8×B300 has stock today, so the instance in step 4 can mount it. If step 0 reported no stock, this call fails the same way. Storage continues billing after the instance is terminated.
Wait for the operation to finish before you quote compute. Retrying the same idempotency key never creates a second volume.

2. Quote 8×B300

Available now. A quote does not reserve scarce B300 capacity. Launch it before its 120-second expiry.

3. Pin and download the model

Manual today. The serving process can download into HF_HOME, but a separate preload makes startup and rollback predictable. The verification command requires the current hf CLI from huggingface_hub 1.18 or later, not the deprecated huggingface-cli:
The canonical artifact is already quantized. Do not silently substitute the NVIDIA NVFP4 or Ascend W4A8 conversions; each needs its own image, topology, quality suite, and acceptance record.

4. Launch SGLang

Available now, runtime qualification pending. This is the conservative 65K B300 profile published by SGLang. It keeps the vision path enabled, uses TP8, and avoids speculative decoding until target-only quality and latency are known. The API key protects the public HTTP port.
Wait for the launch operation the same way as in step 1 (wait_for and waitFor are already defined there):
A running instance is not a ready model. SGLang still has to load about 1.56 TB of weights (downloading them first unless step 3 staged them) and shard them across eight GPUs, which takes tens of minutes. Step 5 waits for /v1/models before the first request. To watch progress:
From the logs, record the reported model load, static-memory allocation, maximum token pool, and admitted-request cap. Do not raise --mem-fraction-static or context until that evidence is captured.

5. Verify text, vision, and thinking history

Retrieve the port-30000 URL from the instance’s connections response and set it as ENDPOINT.
Wait until the model is loaded. The first successful /v1/models is the readiness signal; until then the port returns connection errors or 5xx:

Vision smoke test

IMAGE_URL is any image your server can fetch; a public sample is set here so the test runs as written. Exercise the vision encoder separately from text:
For the next turn, append assistant_message exactly as returned. Verify all three reasoning levels, streaming, tool schemas, malformed-tool recovery, and a real image request. Do not publish self-hosted video support: the current open checkpoint metadata and serving processor establish text and image input only.

6. Qualify the operating point

SGLang’s B300 figures used 8,192 input and 1,024 output tokens, 64 warm-up requests, and cache flushes. Reproduce that baseline, then replace synthetic traffic with your traces:
Measure text and image traffic separately. Sweep request length, image count, concurrency, and arrival rate; report p50/p95/p99 TTFT, TPOT, queue time, throughput, failures, HBM, the KDA state-pool admitted-request cap, KV capacity, and cache hit rate. Only after the target-only baseline passes should you test DSpark. Never set SGLANG_SIMULATE_ACC_LEN in a production or accuracy run: upstream used it for synthetic throughput projection, not measured acceptance. Compare DSpark and target-only serving with identical prompts and retain tool/vision quality tests.
A single instance is not a production service. AI Reserve raw compute does not yet manage replicas, a health-aware load balancer, draining, autoscaling, durable logs, or GPU metrics. Add those controls externally and retain a previously qualified image digest for rollback.

7. Clean up

Primary sources