The short version
Provider failover
Fallback chains. Name the models you accept, in order, on the request (models or
fallbacks) or set a default chain per model in the portal under Inference → Router. When
the primary fails on an upstream error (provider 5xx, rate limit, timeout, connection error)
after the gateway’s own retries, the same request is sent to the next model in your list.
The response carries x-aireserve-served-model whenever something other than the primary
answered. Details, limits and edge cases: Fallback models.
Platform provider failover. Several models are served by more than one provider with
identical weights, for example the Claude models on Anthropic and on Amazon Bedrock. Once
failover is on for your organization, a failure at the primary provider rolls the same
request to the sibling leg automatically: same model, same request, nothing to configure per
model and nothing for your code to do. Your ledger records both the model you asked for and
the leg that served, so billing and reporting follow what actually ran.
Who turns it on. Failover is off for a new organization, so a provider’s real error
reaches you instead of a serve on a model nobody chose. Your administrators switch it on in
the portal: configuring any fallback chain under Inference → Router enables both layers for
the organization (provider failover for the models we have paired across providers, plus
the chains you wrote for specific models), and a single request opts in on its own by
carrying models.
To keep a chained model on the same weights first, list its sibling as the first rung
(claude-sonnet-4-6-bedrock ahead of a different model, for example). We can also set a
chain up for you; ask in your shared Slack channel.
Health breaker. Provider legs share one health record across the whole gateway fleet.
Three slow failures (hangs or 5xx) on a leg inside a rolling two-minute window trip it open:
new requests from organizations with a chain are routed to the next healthy rung instead of
each one rediscovering the failure and paying the full timeout. A leg closes again only after
consecutive successful synthetic probes, never on a timer, so a provider that is still sick
does not get traffic back. Rate limits (429) deliberately do not trip the breaker; they are
handled per request by retries and the chain walk, which costs milliseconds rather than a
leg.
What never fails over. Errors that are yours to fix (a malformed request, an unknown
model, a budget or spend cap you have exhausted) are returned honestly with a
stable error code and are never retried on another model.
Requests on your own provider key (BYOK) never fall back to a
different provider.
Traffic splitting and routing control
Serving pools. For a model served on more than one provider with identical weights, we can split an organization’s traffic across those providers by weight, for example 70/30 between Anthropic and Bedrock, including a minimum-prompt-size rule so only long prompts take a given leg. A canary compares each leg with the source leg over consecutive five-minute windows on p95 latency, cache-hit ratio, rate-limit rate and upstream error rate, and trips a degraded leg out of the split automatically; its share redistributes to the healthy legs and a human re-arms it after the cause is understood. Ask us to set a pool up for your organization. Routing variants and pins.:nitro and :floor suffixes ask for the fastest or the
cheapest provider of a model per request, and an organization can pin a model to one provider
for data-residency or contractual reasons. See
Provider routing variants.
Disclosure. Whenever a request is answered by something other than exactly what you asked
for, the response says so: x-aireserve-served-model names the model that answered and
x-aireserve-served-provider the provider, and the usage ledger keeps the requested and
served identities side by side on every row. No header means the request was served as
requested.
Transport recovery
- Stale connections. A connection to the inference engine or a provider that was closed on the far side while idle is retried once at connect time, before anything is sent.
- Streams that die before the first byte. A streamed request whose provider connection drops before any content has been written is re-sent once, automatically. Nothing has reached you and nothing has been billed, so the retry is invisible.
- First-token and silence budgets. A stream that produces no first token inside its budget is cut with a 504 instead of hanging your client, and the leg that hung is counted against its health. Once content is flowing, only silence can end a stream, never a wall clock, so long generations are safe.
- Deadlines. The request budget is minted at the edge and travels with the request to every hop, so a slow upstream is given up on at the budget, not after a chain of nested timeouts.
- Retry-After. Every 503 the gateway answers for itself carries
Retry-After, and the SDKs’ default retry policies honor it. See Retry policy.
The platform
Change safety
- Staging first, every time. Every merge deploys to a full staging environment (same services, same configuration shape) and runs the smoke suite before anything reaches production.
- Approved, pinned production deploys. Production deploys are dispatched explicitly, reviewer-approved and pinned to a commit. A configuration guard refuses a deploy that would silently change a strict serving flag.
- Post-deploy error canary. After traffic cuts over, a canary compares each serving service’s error rate against the window before the deploy and raises an alarm on a regression.
- Rollback is one command. Previous revisions are retained; a rollback is a traffic switch to the last known-good revision, done by a person within minutes.
- Smoke test on a real path. The suite exercises the production path customers use (gateway, proxy, engine, providers, voice) rather than an internal shortcut.
Monitoring and alerting
- External uptime checks from Google’s global checker fleet probe the API and the portal every 60 seconds over the same DNS, certificate and load-balancer path customers use.
- A platform-health sweep runs every 5 minutes with detectors for success rate, latency and timeouts per model and provider, provider-origin and router-origin rate limiting, settlement lag, traffic flat-lines and per-customer latency; each fires once per crossing, with a cooldown, and posts a recovery line when it clears.
- Provider status pages are aggregated at aireserve.com/status so you can see at a glance whether an incident is upstream.
- Alerts reach the team in Slack and by email at any hour; billing and provider-account canaries run every 15 minutes so a dead provider key is found before you find it.
What we measure
Figures for the 30 days to October 14, 2026, measured at the API edge (every request toapi.aireserve.com, all customers, all endpoints):
We publish these numbers rather than a service-level agreement. No uptime guarantee or
service credits are in force today; an SLA follows once cross-region failover is in place and
has been exercised. If you need the same figures for your own traffic, your usage analytics
carry status and latency per request, and your account team can pull them for a review.
Questions, or a questionnaire to fill in? Write to security@aireserve.com
or ask in your shared Slack channel.