Routing and failover

One model name, several providers behind it. Split traffic by weight, and fail over when one breaks.

Routing is set per model. Open Models & Providers → Models, open the model, and go to its Routing tab. Only project admins and account admins can change it.

Weights split traffic

Each row is a destination: a provider and the model id it serves. Its Weight is its share of traffic.

• Weights split requests across destinations. With 70/30, about 70% go to the first and 30% to the second.
• Weights always add up to 100. Change one, and the others adjust to keep the total.
• A new model starts at 100/0: everything goes to the first destination.
• A 0% destination takes no traffic while another is healthy. It is still a backup.

The destination with the highest weight is marked Primary. The others are marked Failover, in order of weight.

Choose Save routing to apply a change.

When a destination fails

A request that fails on its destination is retried on the next one by weight. Retries happen only before any part of the response reaches the caller.

The destination	Retried on the next one?
Could not be reached	Yes
Answered with a 5xx error, or "overloaded"	Yes
Rate-limited the request (429)	Yes, and the destination is skipped for a while (below)
Is being skipped after repeated failures	Yes. Nothing was sent to it.
Did not start answering within 60 seconds	No. The caller gets 504. The provider may still be working on it, and billing it.
Answered with another 4xx error	No. That is a problem with the request or the setup.

• Two attempts in all. A request tries its destination and, at most, one more.
• Large requests are not retried. A request body over 8 MiB goes to one destination only.
• A caller's own provider key never moves to another provider. A request that brought its own key is retried only on the same provider.
• When every attempt fails, the caller gets the last provider's real answer — its status, headers and body — not a made-up error. A 429 with Retry-After therefore still tells the client how long to wait.

Failing destinations are skipped briefly

• After a 429, the destination is skipped for as long as the provider's Retry-After says, or 10 seconds if it says nothing, and never longer than 60 seconds. It is skipped for that one model only; the provider's other models keep routing.
• When an account runs out of credit — Anthropic's spend limit, or OpenAI's quota — every model on that provider is skipped for 60 seconds.
• After repeated errors, the gateway stops sending to a provider for a short time. Calls that would have gone there fail over. When no destination is left, the caller gets 503 upstream_unavailable with a Retry-After.

Each gateway replica decides this on its own, so with more than one replica a provider can take a few more requests before every replica skips it.

Session affinity

Pin a session to one provider keeps each caller's requests on the same destination, so the provider's prompt cache stays warm. Without it, a conversation can land on another provider and resend its cached context at full input price.

It is on by default. The pin is soft: a session still moves when its provider fails, rate-limits it, or when you change the weights. When the provider recovers, the session moves back.

Remove a model

Remove alias on the Models list, or on a provider's model table, stops the gateway serving that name.

• Callers still sending it start failing.
• Every grant on it is removed with it.
• Removal is permanent. The name is released and can be published again later, with no grants.

See what routing did

Open the model's Activity tab for its recent calls — who called, which provider served it, and how many tokens it used. The Overview tab shows Requests (24h), Added latency (p50) and Failover rate.