LLM Gateway

LLM Gateway is an open-source AI gateway that routes one OpenAI-compatible endpoint to hundreds of models across many providers — with automatic provider selection, fallback, usage analytics, and cost tracking. Use the hosted gateway at api.llmgateway.io or point the adapter at your own self-hosted deployment.

Installation

shell

npm install @tanstack/ai-llmgateway

Basic Usage

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("gpt-5.6-terra"),
  messages: [{ role: "user", content: "Hello!" }],
});

llmGatewayText reads your API key from the LLM_GATEWAY_API_KEY environment variable. Use createLLMGatewayText to pass it explicitly.

Configuration

import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText(
  "gpt-5.6-terra",
  process.env.LLM_GATEWAY_API_KEY!,
  {
    baseURL: "https://api.llmgateway.io/v1", // Optional — set for self-hosted deployments
  },
);

LLM Gateway is open source and self-hostable; point baseURL at your own deployment to keep the same adapter surface.

Available Models

Any model listed at llmgateway.io/models works — pass its id as the model name. A bare model id lets the gateway route to the best available provider; prefix it with provider/ to pin routing to a specific provider:

plaintext

model: "gpt-5.6-terra"          // gateway picks the provider
model: "claude-sonnet-5"        // gateway picks the provider
model: "moonshot/kimi-k3"       // always routed to Moonshot
model: "fireworks/kimi-k3"      // always routed to Fireworks

A curated set of flagship models (see LLMGATEWAY_CHAT_MODELS) additionally carries per-model type metadata — input modalities and provider options — with editor autocomplete. Uncurated ids still work and fall back to text-only input with the generic options.

Example: Chat Completion

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
  });

  return toServerSentEventsResponse(stream);
}

Example: With Tools

import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}

Model Options

The gateway accepts the standard Chat Completions parameters and forwards them to the routed provider (parameters a provider doesn't support are stripped server-side). Sampling parameters live in modelOptions:

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
    reasoning_effort: "high",
  },
});

reasoning_effort accepts the extended scale none / minimal / low / medium / high / xhigh / max in addition to OpenAI's standard tiers — which tiers a model honors depends on the model and provider it is routed to (see the model's page on llmgateway.io/models).

Reasoning models stream their thinking as reasoning_content deltas, which the adapter surfaces as AG-UI REASONING_* events.

Summarization

import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  stream: false,
});

console.log(result.summary);

Environment Variables

Set your API key in environment variables:

shell

LLM_GATEWAY_API_KEY=llmgtwy_your-api-key

Get an API key from the LLM Gateway dashboard.

Installation#

Basic Usage#

Configuration#

Available Models#

Example: Chat Completion#

Example: With Tools#

Model Options#

Summarization#

Environment Variables#