10 Best LLM Gateways in 2026

Dylan Cable September 3, 2026

Ten gateways compared on what they actually cost to run at volume, not on model counts. Real pricing, real caching behavior, and where each one stops being the right call.

The best LLM gateway is the layer between your app and every model provider, turning routing, fallback, caching, and cost control into one endpoint instead of provider-specific code spread across your stack. Past month six of production, that layer stops being optional.

Provider outages happen, cost guardrails matter, and switching models without redeploying is the difference between a 30-minute decision and a 30-day project. This guide compares the 10 best LLM gateways (also called AI gateways) in 2026 on models, pricing, caching, and compliance, with a clear pick for each use case.

The quick verdict: the best LLM gateway for each use case

• Respan if you want an LLM engineering platform that unifies an AI gateway, observability, evaluations, and prompt management in one place.
• OpenRouter if you want a large model catalog behind a single OpenAI-compatible endpoint.
• TrueFoundry if you want a gateway bundled with deployment, governance, and MCP.
• LiteLLM if you want an open-source gateway to self-host.
• Portkey if you want a governance-focused gateway, now part of Palo Alto Networks.
• Cloudflare AI Gateway if you already run on Cloudflare and want edge routing.
• Kong AI Gateway if you already operate a Kong API mesh and want AI controls in it.
• Helicone if you already run it, though it is now in maintenance mode under Mintlify.
• Vercel AI Gateway if you build AI apps on Vercel.
• Braintrust if evaluation is your primary need and routing is secondary.

LLM gateway comparison (2026)

What we looked for (and how we judged)

Before the list, a quick distinction, because people use three words for overlapping things. A proxy forwards requests to providers and adds logging. A router picks the best model or provider per request based on cost, latency, or content. A gateway is the full package: proxy plus router plus caching plus cost controls plus guardrails plus observability hooks. Most tools below are full gateways. A few lean toward proxy or router territory, and we say so where it matters.

We weighted the criteria by how much they affect a production deployment, not how good they look on a feature grid.

• Models supported (must-have): count, and how fast new models land.
• OpenAI-compatible drop-in (must-have): most teams already have OpenAI-format code.
• Caching (high-value): exact-match, semantic, or both.
• Rate limiting and budgets (must-have): per user, per feature, per dollar.
• Cost guardrails and analytics (high-value): alerting on spikes, attribution by feature.
• Observability integration (high-value): traces and eval scores attached to calls.
• Self-host (depends on use case): data residency and compliance requirements.

The 10 best LLM gateways in 2026

1. Respan

Best for: Teams that want gateway, observability, evals, and prompt management on one platform.

The take: Most gateways below route well. The structural question is what your stack looks like six months later. A gateway, plus a separate observability tool, plus a separate eval tool, plus a separate prompt manager is four products, four invoices, and four integrations that have to agree with each other. Respan runs all four on one data plane, so a fallback you set in the gateway, a spend limit you cross, and the trace that explains a slow call all live in one dashboard. The reusable idea is simple: route, observe, and evaluate every LLM call in one place. Instead of reading logs after the fact, you run observability in production, know when production shifts, and act before it spreads.

That consolidation is not theoretical at scale. Retell AI runs more than a million LLM calls an hour through Respan without sampling, and when a provider's streaming endpoint slows down, an on-call engineer flips that model to a fallback in the gateway and watches the retry rate drop on the same dashboard that surfaced the problem. No log triangulation across services.

Strengths:

• Full visibility: per-call traces show exactly what your agents did, so you find the cause in a fraction of the usual debugging time
• One platform: gateway, observability, evals, and monitoring in one place, so you debug production without stitching separate tools together
• Reliability: automatic failover keeps you online when a model or provider fails, with ordered fallback, load balancing, and retries
• Cost clarity: spend broken down by monitors and metadata, so you know where every dollar goes
• Accurate quality measurement: evals that show what is working, what is not, and why
• Actionable metrics: every metric links back to the call that produced it, so investigations do not dead-end
• Enterprise-ready compliance: SOC 2, HIPAA, GDPR, and ISO 27001

Trade-offs:

• More than you need if you only want bare routing, since Respan is a full platform
• Less battle-tested at the ten-year-incumbent scale of Cloudflare

Pricing: Free tier with generous limits. Pro and Enterprise tiers above it.

You can route, observe, and evaluate your LLM calls on Respan's AI gateway, and the free tier is enough to run real production traffic through it before you commit.

2. OpenRouter

Best for: The widest model catalog and the simplest integration.

The take: OpenRouter's value is breadth. It aggregates 500+ models from 60+ providers, including obscure and experimental ones, behind one OpenAI-compatible endpoint. Integration is about as easy as it gets: change your base URL and you are routing. The tradeoff with tools like OpenRouter shows up on the invoice at scale, where the fees become a recurring line item.

Strengths:

• One of the largest model catalogs in the market
• Simple OpenAI-compatible drop-in
• Provider fallback supported
• Strong developer community

Trade-offs:

• A 5.5% fee on credit purchases (5.0% on crypto) that compounds at high volume
• Observability is limited to activity logs
• No semantic caching and no prompt management
• No self-host option

Pricing: Pay-as-you-go at pass-through provider rates, plus the 5.5% credit fee. Free models available with rate limits.

3. TrueFoundry

Best for: An enterprise gateway bundled with deployment, governance, and MCP.

The take: TrueFoundry is an enterprise AI platform whose gateway routes across 1,000+ models with intelligent routing, fallbacks, and load balancing. The weight is on enterprise controls: virtual keys, budgets, and cost caps; built-in observability and governance; SSO, RBAC, and audit logs; and an integrated MCP gateway for agent and tool workloads. It also offers self-hosted and VPC deployment for data-residency needs. The pitch is one vendor for the gateway plus the broader production-AI stack.

Strengths:

• 1,000+ models with intelligent routing, fallbacks, and load balancing
• Virtual keys, budgets, and cost controls
• Built-in observability and governance
• Enterprise security: SSO, RBAC, audit logs
• Integrated MCP gateway
• Self-hosted, VPC, and air-gapped deployment options

Trade-offs:

• A broader platform than a standalone gateway, which is more than you need if you only want routing
• Enterprise-oriented onboarding and pricing that lean on sales rather than self-serve
• Smaller gateway-specific community than OpenRouter or LiteLLM

Pricing: Free to start; then $499/month.

4. LiteLLM

Best for: An open-source gateway you self-host and fully control.

The take: LiteLLM is the open-source gateway that standardizes 100+ providers behind one OpenAI-compatible interface, usable as a Python SDK or a proxy server. It is popular exactly where a managed cloud gateway is not an option, and it gives you full control at the cost of running the infrastructure yourself.

Security note (March 2026): A supply-chain attack compromised specific LiteLLM PyPI packages via a poisoned GitHub Action, with a credential-stealing payload. The malicious versions were quarantined quickly and the maintainers shipped a hardened, isolated CI/CD pipeline. Pin to the latest patched release (v1.83.0 or later) and rotate any secrets that touched the affected builds.

Strengths:

• Open source, MIT-licensed, self-hostable
• Broad model support, 100+ providers
• Routing strategies including latency, cost, and least-busy
• Budgets and rate limits per user, team, or key
• Free when self-hosted

Trade-offs:

• Self-hosting is real operational work
• Observability beyond basic logging depends on integrations
• Advanced features like JWT auth and audit logs sit behind the enterprise tier
• No first-party managed cloud without third-party hosting

Pricing: Open source and free to self-host. Enterprise pricing is custom.

5. Portkey

Best for: A governance-heavy gateway, now inside Palo Alto Networks.

The take: Portkey is the gateway built for strict governance: audit logs, RBAC, guardrails, and policy enforcement across a very broad model catalog. As of May 29, 2026, Palo Alto Networks completed its acquisition of Portkey, and Portkey now serves as the core AI Gateway inside the Prisma AIRS security platform. If you are evaluating it fresh, evaluate it as part of that platform direction rather than as an independent startup.

Strengths:

• Enterprise governance: audit logs, RBAC, guardrails, PII controls
• 1,600+ models across multiple modalities
• Provider fallback, semantic caching, and cost guardrails
• SOC 2, HIPAA, GDPR, and ISO 27001 compliance

Trade-offs:

• Now tied to the Prisma AIRS platform roadmap post-acquisition
• Pricing gets complex at the upper tiers, with key features gated to Enterprise
• Feature density is more than teams that only want routing will use

Pricing: Free tier; paid plans from $49/month, with custom enterprise pricing.

6. Cloudflare AI Gateway

Best for: Teams already on Cloudflare who want edge routing.

The take: Cloudflare's gateway runs on their edge network, so requests route close to the user, and it plugs tightly into the rest of the Cloudflare stack like Workers and R2. If you already live on Cloudflare, it is the path of least resistance. Outside that ecosystem, the case is weaker.

Strengths:

• Edge-routed for low latency
• Mature caching, plus analytics, rate limiting, retries, and model fallback
• Tight integration with the Cloudflare stack
• Generous free tier

Trade-offs:

• Smaller model catalog than OpenRouter or Respan
• Most useful inside the Cloudflare ecosystem
• Lighter observability than dedicated tools
• No self-host; Cloudflare-managed only

Pricing: Pay-per-request with a generous free tier. Includes a 5% fee applied to all credits purchased through Unified Billing.

7. Kong AI Gateway

Best for: Teams already running a Kong API mesh who want AI controls inside it.

The take: Kong AI Gateway extends Kong's established API-management platform into LLM traffic, with an AI Proxy that standardizes calls to OpenAI, Anthropic, Bedrock, Vertex, Azure, Mistral, and Cohere. In April 2026 it added an Agent Gateway (in AI Gateway 3.14) to govern LLM, MCP, and agent-to-agent traffic. It is a strong fit when Kong already runs your API layer, because AI controls slot into the same control plane. It is heavier to justify when AI routing is the only reason you would adopt it, since the platform was built for general API management first.

Strengths:

• AI Proxy across major providers through Kong's control plane
• Governance depth: PII sanitization, content-safety guardrails, audit logging, token-based rate limiting
• Agent Gateway for MCP and agent-to-agent workloads
• Self-hosted or managed via Konnect

Trade-offs:

• Heaviest to operate if you are not already a Kong shop
• Pricing follows API-management metrics rather than token or model units, which can bite high-volume AI workloads
• Semantic caching and routing land on the enterprise tier

Pricing: Free trial, then $500/month per control plane.

8. Helicone

Status update (March 2026): Mintlify acquired Helicone. Cloud services stay live in maintenance mode, with security patches, bug fixes, and new-model support continuing, but active feature development has ended. Treat it as a sunset product for fresh selection, and plan a migration path if you already run it. We keep a current rundown of the best Helicone alternatives after the Mintlify acquisition.

Best for (historically): A lightweight cost gateway with a proxy install.

The take: Helicone paired a Rust-based router with built-in observability and a one-URL install, which was a genuinely good design. The maintenance-mode status is the deciding factor now. The product keeps working, but a proxy in your critical path that is no longer actively developed is a risk to weigh, especially as provider APIs keep moving.

Strengths:

• Open-source self-host still available
• Low routing overhead, with health-aware load balancing and cross-provider caching
• Observability built into the routing layer

Trade-offs:

• Maintenance mode: no new features, only patches and new-model support
• Observability is coupled to Helicone's own platform
• Proxy-in-critical-path risk without active development behind it

Pricing: Open-source self-host is free. Cloud plans from $79/month and team plans from $799/month.

9. Vercel AI Gateway

Best for: Teams building AI apps natively on Vercel.

The take: Vercel's gateway is a routing layer tied to Vercel's platform, giving one endpoint for 100+ models with OpenAI and Anthropic SDK compatibility and pass-through provider pricing. If you deploy on Vercel, it is a convenient first-party option. Off Vercel, it is awkward, and it is a routing convenience layer more than a full control plane.

Strengths:

• First-party for Vercel-deployed apps, with tight AI SDK integration
• No markup on provider token rates, plus BYOK
• Automatic model fallback on provider outages

Trade-offs:

• Cloud-managed only and tightly coupled to Vercel
• No conditional routing, A/B testing, or traffic splitting
• Observability limited to usage and billing
• Semantic caching requires manual setup with a separate Redis instance

Pricing: $5/month gateway credit per account, then pay-as-you-go at provider rates. Platform compute is billed separately.

10. Braintrust

Best for: Teams whose primary problem is evaluation, with routing secondary.

The take: Braintrust is an evaluation platform with an integrated gateway bolted to the front. The gateway routes across OpenAI, Anthropic, Google, and others, is SDK-compatible, and logs every request, which makes it a fair option when you want tracing and caching in the routing layer. It is built to serve Braintrust's eval workflows, not to be a standalone routing product, so weigh it as an eval tool first.

Strengths:

• Logs and traces every request without extra instrumentation
• Encrypted response caching with configurable TTL
• SDK-compatible across major providers
• SOC 2 Type II and HIPAA, with cloud and hybrid options

Trade-offs:

• Designed around Braintrust's eval workflows rather than routing
• No conditional routing or native A/B testing at the gateway
• Largely closed-source, with self-host on enterprise only

Pricing: Gateway core is free on all plans. Pro is $249/month; enterprise is custom.

How to choose the right LLM gateway

The right choice is less about which tool tops a list and more about which constraints matter most for your stack. Work through these before committing.

• Deployment model - Decide managed versus self-host first. Managed removes the ops burden; self-host earns its keep only when data residency or compliance rules block a cloud service.
• Model coverage - Confirm the providers and models you actually use are supported, and how fast new ones get added. Coverage of your stack matters more than raw catalog size.
• Reliability - Look for automatic provider fallback, retries, and load balancing. A gateway without failover is only half a gateway.
• Scope beyond routing - Decide whether you want routing alone or a platform that also traces calls, tracks cost, and runs evals. Separate tools mean more integrations to keep in sync.
• Cost model - Understand how it charges: pass-through provider rates, a platform fee, per request, or per log. Fees that look small compound at high volume.
• Governance and compliance - If you handle regulated data, confirm SOC 2, HIPAA, or GDPR, plus access controls, audit logs, and guardrails.
• Ecosystem fit - A gateway that aligns with infrastructure you already run is often the path of least resistance, as long as it does not lock you in.

Weight the factors that are expensive to reverse, like deployment model and compliance, over the ones you can tune later, like caching strategy or which models you route to.

Common mistakes when adopting a gateway

Most gateway problems in production trace back to a handful of decisions made early and never revisited. These are the ones that cost teams the most.

1. Skipping a gateway "for now." You will need one within six months, and migrating production traffic later is the painful version of this decision.
2. Not configuring fallback. A gateway without provider fallback is half a gateway. Set it on day one.
3. Enabling semantic cache by default. A wrong cache hit ships a stale answer. Start with exact-match and validate semantic before turning it on.
4. No per-feature cost guardrails. The first runaway agent can drain a monthly budget in hours. Attribute and cap by feature.

None of these are hard to avoid, but all of them are hard to unwind once traffic is flowing. Set fallback, guardrails, and caching policy before you route production load, not after the first incident. The mistakes specific to the routing decision itself, including where routing rules should live, are covered in the best LLM routers comparison.

FAQ

What is the best LLM gateway?

For teams that want one platform instead of four, Respan is the best LLM gateway: it runs the gateway alongside observability, evals, and prompt management on a single data plane, so you debug production without stitching separate tools together.

There are others that are popular as well: OpenRouter leads on catalog breadth, LiteLLM on open-source self-host, Portkey and TrueFoundry on enterprise governance, and Cloudflare and Vercel inside their own ecosystems.

What's the difference between an LLM gateway and an AI gateway?

In practice the terms are used interchangeably. "AI gateway" is the slightly broader umbrella, and some vendors, including Cloudflare and Vercel, brand their products that way. "LLM gateway" points specifically at routing and governing large-language-model traffic, which is what this list covers.

Do gateways add latency?

A well-designed gateway adds a small amount of overhead per call, and with caching enabled it often reduces median latency, because cache hits return in single-digit milliseconds. The real number depends on the gateway's architecture and how far it sits from your providers, so test it against your own traffic.

Should I self-host or use a managed gateway?

Managed is right for most teams, because it removes the ops burden on a component holding every provider key you own. Self-host when an air-gapped environment or a contractual infrastructure requirement rules cloud out, and LiteLLM and Kong both support it. Most teams asking this question are really asking about data handling rather than where the container runs, though, and that gets answered by compliance posture instead: Respan carries SOC 2, HIPAA under a BAA, GDPR, and ISO 27001, with PII masking and the option to omit request and response bodies from logs.

Can I switch gateways later?

Usually, yes. If the gateway is OpenAI-compatible, which most are, switching is a base-URL change. Lock-in risk is highest with proprietary SDKs and lowest with OpenAI-compatible interfaces.

Which has the largest model catalog?

Respans reaches 1,000+ models across every major provider through one endpoint, which covers what production teams actually call. OpenRouter covers 500+ across 80+ providers. Though, it's worth keeping in mind that raw count can be a weak way to choose, because what decides the question is how fast new models land and whether the ones already in your stack are supported.

Recent changes in the LLM gateway landscape (2026)

The category consolidated fast this year, which is most of the reason a 2025-era list is already stale.

• Portkey joined Palo Alto Networks. The acquisition closed May 29, 2026, and Portkey is now the core AI Gateway inside Prisma AIRS. Evaluate it as a platform component, not an independent tool.
• Helicone entered maintenance mode under Mintlify. Acquired in March 2026, still live, but no new features. Fresh deployments should weigh the sunset risk, and existing users should plan a migration.
• Kong shipped an Agent Gateway. Added in April 2026 (AI Gateway 3.14) to govern LLM, MCP, and agent-to-agent traffic, part of a broader move to treat the gateway as the control point for agentic workloads.
• LiteLLM hardened its pipeline after a supply-chain incident. A March 2026 compromise of specific PyPI packages pushed the project to an isolated CI/CD pipeline. Pin to a patched release and rotate exposed secrets.

The through-line is consolidation: standalone tools are being absorbed into larger platforms, which raises the value of picking a gateway that already owns the surrounding workflow rather than one you will re-evaluate at the next acquisition.

That is what Respan is built for. Run gateway, observability, evals, and prompt management on one platform with Respan's AI gateway.