## What is Respan’s AI Gateway?

An AI gateway is a single API endpoint between your application and multiple LLM providers. Respan’s routes to 1000+ models across every major provider (OpenAI, Anthropic, Google, Bedrock, Azure, and more). Point any LLM SDK at Respan and you get automatic tracing, prompt management, fallbacks, load balancing, and caching, without changing how you call the model.

**Considerations:**

- May not suit products with strict latency requirements ( **50 to 150ms** added).
- May not be ideal for those who don’t want to integrate a third-party service into the core of their application.

* * *

## Setup

### 1. Get your Respan API key

Create an account on [Respan](https://platform.respan.ai/), then generate a key on the [API keys page](https://platform.respan.ai/platform/api/api-keys).

**Environment management:** create separate API keys for test and production instead of toggling an `env` parameter. Cleaner separation, better security.

* * *

### 2. Choose how Gateway requests are funded

Every Gateway request uses your Respan API key. For access to the upstream model, choose the funding path available for the model you want to call:

- **Use Respan credits.** Add credits and configure balance controls on the [Billing page](https://platform.respan.ai/platform/api/billing). See [Billing and credits](/content/docs/documentation/admin/credits-billing/index.html).
- **Bring your own provider key.** Connect credentials from your own provider account on the [Providers page](https://platform.respan.ai/platform/api/providers). See [Provider keys](/content/docs/documentation/admin/llm-provider-keys/index.html) and the [OpenAI provider guide](/content/docs/integrations/gateway/model-providers/openai/index.html).

* * *

### 3. Point your app at the Respan endpoint

Route any LLM SDK to `https://api.respan.ai/api/` using your Respan API key. Pick the path that fits your workflow.

###### AI (CLI)

###### Manually

The fastest way. The Respan CLI hands off to your coding agent (Claude Code, Cursor, Codex, and others), which detects your project’s LLM provider and rewrites your existing calls to point at the gateway.

```

```

It also creates a Respan API key for you and saves it to `.env` if one isn’t already set.

* * *

### 4. Use prompts (Optional)

Manage prompt templates centrally instead of hardcoding them. Create and version a prompt in Respan, then reference it by `prompt_id` in your gateway calls to ship new versions without changing code.

[1](/content/docs/documentation/features/gateway/gateway-quickstart#create-a-prompt/index.html)

### Create a prompt

Go to the [Prompts page](https://platform.respan.ai/platform/prompts) and create a new prompt. Write your system message and user template with `{{variables}}`:

```
System: You are a helpful assistant that speaks {{language}}.
User: {{user_message}}
```

Save and deploy. Copy the **prompt ID**.

[2](/content/docs/documentation/features/gateway/gateway-quickstart#use-the-prompt/index.html)

### Use the prompt

Call the gateway with your prompt ID. Respan injects the prompt and fills in the variables:

```

```

For versioning, deployment, and testing see [Prompt management](/content/docs/documentation/features/prompt-management/prompt-management-quickstart/index.html).

* * *

## What the gateway provides

You’re routing traffic through the gateway. Here’s what else it gives you.

- **[One endpoint, 1000+ models](/content/docs/documentation/features/gateway/custom-models/index.html)** A single OpenAI-compatible endpoint routes to every major provider: OpenAI, Anthropic, Google, Bedrock, Azure, and more.
- **[A platform, not just a proxy](/content/docs/documentation/features/gateway/routing-and-passthrough/index.html)** Observability, evals, prompt management, monitors, and spend limits share the gateway’s data plane, with _no second SDK or vendor_.
- **[Reliability](/content/docs/documentation/features/gateway/advanced/index.html)** Ordered model fallback, load balancing across deployments and providers, and configurable auto-retries with backoff.
- **[Custom attributes](/content/docs/documentation/features/gateway/respan-params/index.html)** Tag requests with `customer_identifier` and `metadata`, then break down latency, errors, and eval scores by them.
- **[Caching](/content/docs/documentation/features/gateway/caching/index.html)** Response caching with configurable TTL and per-customer scoping.
- **[Observability](/content/docs/documentation/features/observe/view-debug-traces/index.html)** Every call is traced end-to-end: model attempted, fallback fired, cache hit/miss, tokens, latency.
- **[Evals & prompt management](/content/docs/documentation/features/prompt-management/prompt-management-quickstart/index.html)** Run online and offline eval pipelines against production traffic, and manage versioned prompts referenced by ID.
- **[Limits & monitors](/content/docs/documentation/features/gateway/limits/index.html)** Soft and hard caps on requests or tokens, scoped per key, model, or customer, with Slack, email, or webhook alerts.

## Examples

###### Streaming

Stream tokens as they’re generated by passing `stream=True`.

```

```

###### Function calling

Let the model call your functions by passing a `tools` array.

```

```

###### Enable thinking

Have supported models return their reasoning before the final answer by passing a `thinking` config.

```

```

Choose models that support thinking like `gpt-5`, `claude-sonnet-4-5-20250929`. See [Log content types](/content/docs/documentation/features/tracing/log-content-types#thinking-blocks/index.html) for details on the response structure.

###### Upload PDF

Send PDFs as `file` content blocks for the model to read.

```

```

###### Upload image

Pass images using `image_url` content blocks or via prompt variables.

```

```