Dynamic Routes

Dynamic routes let you move routing logic out of your application code and into the gateway. Instead of hardcoding a model, you define a named decision graph — conditions on the request, classifier verdicts, percentage-based traffic splits, and model targets — and invoke it by putting dynamic/<name> in the model field of a chat request. Dynamic routes are resolved on /v1/chat/completions and the endpoints that proxy through it (/v1/responses, /v1/messages, and the AI SDK gateway endpoints under /v*/ai) — not on embeddings, images, audio, video, rerank, OCR, or moderations:

curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "x-user-tier: paid" \
  -d '{
    "model": "dynamic/support",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The graph is evaluated on every request. Once it resolves to a model, the request flows through the same smart routing as any other request: weighted provider scoring, sticky sessions, and automatic cross-provider fallback all apply to the resolved target.

The route graph

A route is a JSON document with an entry node id and a list of nodes. Evaluation starts at entry and follows branches until it reaches a model node (route the request) or an end node (reject the request with a 400).

{
	"entry": "tier",
	"nodes": [
		{
			"id": "tier",
			"type": "conditional",
			"conditions": [
				{
					"field": { "source": "header", "path": "x-user-tier" },
					"op": "eq",
					"value": "paid",
					"next": "premium"
				}
			],
			"else": "split"
		},
		{ "id": "premium", "type": "model", "model": "claude-sonnet-4-6" },
		{
			"id": "split",
			"type": "percentage",
			"splits": [
				{ "weight": 80, "next": "stable" },
				{ "weight": 20, "next": "experiment" }
			]
		},
		{
			"id": "stable",
			"type": "model",
			"model": "gpt-5-nano",
			"providers": ["openai"]
		},
		{ "id": "experiment", "type": "model", "model": "gemini-2.5-flash" }
	]
}

Node ids may contain letters, digits, hyphens, and underscores.

Node types

conditional

Evaluates its conditions top to bottom; the first match wins and the request follows that condition's next. When nothing matches, the request follows else.

Each condition reads one field from the request:

Supported operators:

classifier

Rates the request with a classifier and branches on one of its answers. Set kind to the classifier to use (currently jev, TypeSafe's decision model) and on to the answer to branch on:

{
	"id": "rate",
	"type": "classifier",
	"kind": "jev",
	"on": "difficulty",
	"cases": [
		{ "value": "high", "next": "frontier" },
		{ "value": "low", "next": "cheap" }
	],
	"else": "balanced"
}

The node adds one short round trip before the upstream call, and only routes that contain a classifier node pay for it. Each rating is billed at the catalogue rate for jev-1.13.0 — roughly $0.0001 per call — and recorded as its own log entry against the same project and API key as the request that triggered it; see what the classifier costs. A sticky session is rated once and the rest of the conversation reuses that verdict, so a multi-turn chat is not re-rated per turn.

else covers both outcomes you need it to: no case matched, and no verdict was produced. The classifier is fail-open — an outage, a timeout, a missing credential, or a compliance policy that blocks its provider all take the else branch rather than failing the route. Give else a sensible default model rather than an end node.

The verdict is recorded on the request log under routingMetadata.dynamicRoute.classifier, so you can see which branch was taken and why.

percentage

Splits traffic across branches by relative weight. The draw is deterministic per session: the split key is the request's session id (x-session-id and the other sticky session routing signals), so a conversation keeps its assignment across requests instead of flip-flopping between experiment arms. Requests without a session id get an independent draw per request.

Weights are relative — { 80, 20 } and { 4, 1 } produce the same split.

model

Terminates evaluation and routes the request to a catalog model (see the models page for available ids). Optional providers restricts routing to those providers in preference order for initial selection and, on non-sticky requests, cross-provider fallback. Eligibility checks and hybrid-mode key preference apply first. A healthy session pin takes precedence; when a pin becomes unavailable, the next request chooses its replacement using the order. Sticky requests still skip cross-provider retries. Scores choose among regions of the same provider, and explicit provider order disables exploration. When providers is omitted, every eligible provider serving the model is a candidate and weighted smart routing picks the best one.

A model node can also target one of your organization's custom catalog models as <customProviderName>/<modelName>. Such a node must not set providers (the custom model already fixes its provider), and the referenced custom model must be active in your organization when the route is saved or published.

end

Terminates evaluation and rejects the request with a 400. Useful as an explicit deny branch — for example, refusing traffic that doesn't carry a required header.

Validation

Graphs are validated when you save a draft and again when you publish:

• every referenced node must exist and be reachable from entry
• cycles are rejected — evaluation is deterministic per request, so a revisited node would loop forever
• model ids and provider ids must exist in the catalog, and each listed provider must actually serve the model
• operator/value mismatches (e.g. gt with a non-numeric value) are rejected
• a classifier node's case values must be valid for its on answer, and must not repeat

An invalid graph can never be saved or published, and publishing re-validates against the live model catalog so a stale draft can't resurrect a removed model.

Versions and rollback

Edits always go to the route's draft. Publishing snapshots the draft as an immutable, numbered version and points the route at it; the published version is what serves traffic. Rolling back re-points the route at any previous version instantly — no data migration, no re-deploy.

A route only serves requests when it is enabled and has a published version; otherwise requests using it are rejected with a 404.

Observability

Each request served through a dynamic route records the route name, published version, and the node path the evaluation took in its routing metadata, alongside the usual provider scoring details — visible in the activity log detail view.

The project's Analytics page compares each route's spend with sending every request to the priciest model in the route; see routing savings.

Reliability

Published routes are cached in the gateway and carry a stale fallback: when the database is temporarily unreachable, recently used routes keep resolving from cache, the same way API keys and project settings do.