Retries and fallback order | Respan Docs

Retries and Fallbacks

Retries and fallbacks solve different parts of a failed gateway request:

A retry repeats the current model/provider route after a delay.
A fallback advances to the next configured model after the current route exhausts its attempts.

The gateway returns as soon as one attempt succeeds. If every eligible attempt fails, it returns the terminal error; configuring fallback models improves resilience but does not guarantee success.

Execution Order

For a request that reaches an upstream provider, Respan applies reliability controls in this order:

Preflight checks validate authentication, the request, and applicable limits. A request rejected here never reaches a model, so provider retries and model fallback do not run.
Route selection resolves the primary model, provider or load-balancing route, credentials, and ordered fallback models.
Current-route attempts call the selected route up to num_retries total attempts when retries are enabled. The delay begins at retry_after, increases exponentially, includes jitter, and is capped at 60 seconds.
Fallback advance moves to the next eligible model only after the current route exhausts its attempts and the failure is not marked fail-fast.
Return stops on the first success. If no route succeeds, Respan returns the last terminal error with per-model failure context where available.

Despite the field name, num_retries is the maximum number of attempts for the current route, including the first attempt. For example, num_retries: 3 allows at most three calls to that route. Setting retry_enabled: false makes one attempt per eligible route before fallback is considered.

Reliability Matrix

Outcome or failure stage	Retry current route?	Advance to a fallback model?	Result
Preflight authentication, validation, or limit rejection	No provider call	No	Return the rejection immediately
Current route succeeds	No further attempt	No	Return the successful response
Retryable provider or network failure	Yes, up to `num_retries` total attempts	After attempts are exhausted	Try the next eligible model
Current route exhausts attempts with a non-fail-fast error	No attempts remain on that route	Yes	Continue through `fallback_models` in order
Deterministic or unsafe-to-repeat failure marked fail-fast, such as an oversized context, an invalid customer-owned credential, or a model-side read timeout	No further retry	No further model fallback	Return the terminal error
Every eligible route fails	No	No routes remain	Return the terminal gateway error

Do not use retries to mask invalid input or limits. Fix request validation, credentials, context size, and spend/rate-limit configuration at their source. Repeating the same deterministic failure adds latency and can consume provider quota.

Configure Retries

Via UI

OpenAI Python SDK

Standard API

Go to the Retries page, enable retries, set the maximum attempts, and set the initial delay.

Supported Retry Parameters

Troubleshooting

The request failed without calling a fallback. Check whether authentication, validation, limits, context size, or a customer-owned credential caused a preflight or fail-fast rejection.
Latency is higher than expected. Each fallback model has its own attempt budget. Reduce num_retries, shorten the initial delay, or shorten the fallback chain.
A fallback uses the wrong credentials. Credentials are resolved per model. Review provider keys and per-model credential overrides.
You need distribution rather than failure recovery. Use load balancing; it selects a route before the retry/fallback sequence.