In a traditional gateway, routing is a forwarding problem. A request arrives, the gateway evaluates policy, picks a backend cluster, and sends the request there. The backend is usually a deterministic service. If it is healthy, the request succeeds. If it fails, the gateway retries, fails over, or returns an error.

LLM systems break that mental model.

The backend is no longer just a service. It is a probabilistic reasoning system. It may need tools. It may need memory. It may need retrieval. It may hallucinate. It may require a verifier. It may run on a GPU pool whose KV cache is warm for one session and cold for another. It may be cheap for short requests and expensive for long-context ones. It may be allowed for one tenant and forbidden for another.

A request is no longer only traffic. It is semantic work.

Routing is the moment where semantic intent becomes infrastructure placement.

That is the starting point of vLLM SR.

The project is not trying to build a prettier model selector. The deeper idea is to turn every LLM request into an explainable capability path: which evidence was observed, which derived facts were produced, which policy matched, which selection or collaboration algorithm ran, which plugins were attached, which backend and hardware path was used, and what trace was left behind.

I want to walk through that design as a system, not as a feature list. We will keep one running example in view, then open the layers one by one: signals, projections, decisions, selection algorithms, looper algorithms, router memory and learning, plugins, Envoy integration, model design, native bindings, hardware support, observability, and evaluation.

A Request Walks Into the Router

Imagine an enterprise AI gateway receives this request:

Analyze this customer contract, compare it with our internal policy, and draft a response. It may contain personal information. Use tools if needed.

If we look only at surface text, this is a contract-analysis request. If we look only at token count, it may be a long-context request. If we look at risk, it touches PII, internal policy, tool access, and factual correctness. If we look at infrastructure, it depends on tenant permissions, pool health, cache warmth, context-window capacity, and model cost.

A naive model picker asks one question:

Which model should answer this prompt?

vLLM SR asks a better question:

What must the system know before it is allowed to spend intelligence?

The split is intentional. Each layer owns a different kind of reasoning.

The core design choice is separation of concerns. If a request goes to a frontier model with RAG, tool filtering, and a response verifier, the answer should not be “because the router said so.” The router should be able to show the chain: evidence, projection, policy, algorithm, learning decision, plugins, mutations, backend path, and replay id.

The Full Request Lifecycle

For the contract request, the router first extracts request context: body, headers, tenant identity, session id, conversation id, tool state, prior response markers, and any existing routing metadata.

It then runs only the signals referenced by the active recipe. Domain, PII, context length, authorization, knowledge-base relevance, fact-check need, tool availability, and jailbreak or prompt-injection signals can run independently. The point is not to run every detector on every request; the point is to gather the evidence that the configured policy actually needs.

Projections then compose that evidence into stable route facts: privacy_sensitive, legal_contract_path, long_context, needs_internal_rag, and verification_required. The decision engine matches those facts against route policy. A matched decision exposes candidate models and, depending on its algorithm, either a single-model selection path or a looper collaboration path. Router Learning may then adjust the proposed model or hold the current model if switching would break session continuity. Finally, plugins attach route-scoped behavior before the gateway executes the main traffic path.

The model name is only the visible end of the route. The deeper value is that the system can explain why this was not a cheap direct-answer path. Privacy, internal knowledge, tool exposure, factuality, and context pressure are different concerns. vLLM SR gives each concern a place in the architecture.

The Recipe Is the System Contract

The next design step is easy to miss: vLLM SR is not primarily configured as a collection of ad hoc callbacks. It is configured as a recipe. That recipe is the contract between application intent, routing policy, backend inventory, plugin behavior, learning boundaries, and gateway mutation.

In production, routing policy tends to spread. One part lands in application code, one part in gateway config, one part in prompt templates, one part in a model registry, one part in a dashboard, and one part in a notebook used by the evaluation team. Once that happens, nobody can answer a simple operational question: “Why did this request take this path?”

vLLM SR’s v0.3-style contract tries to keep that answer in one durable shape.

That is the operational difference between a router that happens to make good choices and a router that can be trusted. A recipe is diffable. It can be validated. It can be replayed against stored traffic. It can be promoted across environments. It can be patched by offline learning without letting the hot path rewrite production policy.

The rule is conservative on purpose: runtime intelligence may propose better choices, but the recipe defines the legal search space. That is what keeps “adaptive routing” from becoming hidden policy mutation.

Research as Design Pressure

The paper shelf behind vLLM SR is not a citation dump. Each research direction creates pressure on one part of the router. Read the table as a map from problem to architecture.

The pressure from the research shelf is clear: a router cannot be one classifier. If it were, every new paper would become another special case. In vLLM SR, a paper can become a signal, projection, decision primitive, selection algorithm, looper algorithm, plugin, learning signal, model-card field, or pool feedback loop.

Signals: Evidence Before Judgment

Signals are deliberately humble. A signal answers one factual question. It does not choose the model. It does not select a plugin. It does not decide the route.

For the contract example, the router may need to know whether the request is legal or enterprise-related, whether it contains PII, whether the user is authorized for premium or private models, whether it needs internal knowledge, whether tools are available, whether the context is too large for a short-context backend, and whether the response needs verification.

The maintained runtime surface currently exposes eighteen base signal families: authz, complexity, context, conversation, domain, embedding, fact_check, jailbreak, keyword, language, modality, pii, preference, reask, structure, kb, user_feedback, and event. projection is also a rule condition type, but it is not a raw sensor. It is the decision-visible output of the projection layer.

The runtime behavior is as important as the list of names.

The payoff is evidence reuse. A pii signal can influence privacy policy, cache policy, provider selection, and audit policy. A conversation signal can influence router protection, tool filtering, and replay. A context signal can influence long-context pools, compression, and cost-aware selection. If signals directly selected models, that reuse would disappear.

Projections: The Coordination Layer

Signals are often messy. Some are booleans. Some are classifier probabilities. Some are similarity scores. Some are raw metrics such as token count or KB relevance. Production policy should not need to manually combine all of those raw values every time.

Projections turn evidence into stable routing facts.

One subtle rule keeps projections clean: decisions read projection outputs, not every intermediate score name. A partition or weighted score can be rich internally, but the policy surface should see facts such as legal_contract_path, risk_high, or long_context_lane.

That is why the projection layer can grow without making decisions unreadable. A risk score might combine PII, jailbreak, tenant tier, KB source, and response verifier need. The decision should not have to carry that formula inline. It should match a named derived fact and leave the math in a traceable coordination layer.

Projections keep the rest of the system readable. Signals remain fine-grained. Decisions remain policy-oriented. Algorithms remain responsible for selection or collaboration. The coordination work lives in between, where it can be traced.

Decisions: Policy Needs a Shape

Once projections produce stable facts, decisions express route policy.

The decision engine is intentionally closer to a boolean circuit than a hidden Python function. Routing policy should be inspectable, diffable, auditable, sortable, and eventually compilable across infrastructure layers.

For the contract request, the policy can be simplified as:

A route is not just an endpoint. It is a policy-approved capability bundle: candidate models, algorithm, plugins, gateway mutations, retention behavior, fallback, and diagnostics.

In the current contract, a decision is a route object, not only a rule tree.

Decision selection is intentionally deterministic. If matched decisions use tiers, lower tier values win first, then confidence, priority, and name. Without tiers, strategy=confidence ranks by confidence before priority; the default strategy ranks priority before confidence. Even fallback is explicit: an empty AND can be a catch-all route, but it has zero confidence so it does not outrank real evidence-backed decisions.

Retention emits make policy visible beyond the immediate model choice. A decision can express drop to skip semantic-cache writes, ttl_turns to bound cache lifetime, keep_current_model to protect session continuity, or prefer_prefix_retention to tell the serving pool that KV/prefix reuse matters.

That shift from classification-style routing to signal-decision routing is not cosmetic. Classification is useful for demos. Decision architecture is what production policy needs.

Selection Algorithms: Choosing One Model After Policy

Many router designs start with the selector: embeddings, MLPs, bandits, or an LLM-as-router that picks a model directly. That easily turns one algorithm into a sink for every concern: semantic fit, cost, latency, safety, authorization, session continuity, and provider policy.

vLLM SR keeps the order stricter. The decision matches first. Then a selection algorithm chooses one model from the policy-approved candidate set.

The boundary is strict: selection algorithms choose one model from modelRefs. They do not run a panel, coordinate a workflow, or hold a session model. Multi-model collaboration belongs to Looper. Session and conversation stability belongs to Router Learning.

Looper Algorithms: Selecting a Model Collaboration Path

Looper is important enough to discuss separately because it changes what the router is selecting.

A selection algorithm chooses one model. A Looper algorithm chooses a model collaboration path: a bounded execution pattern involving escalation, fan-out, panel judgment, multi-round reasoning, or micro-agent workflows. This is usually the path for scaling model capability without exposing a new application protocol. The client may still call one logical model name, but the router executes a structured collaboration behind that name.

These are not just “more algorithms.” They are the router’s answer to capability scaling.

Confidence keeps the cost curve low by starting small and escalating only when confidence is insufficient. It can use average log probability, margin, a hybrid score, self-verification, or an AutoMix-style entailment verifier. The question is not “small or large model?” It is “is the current answer good enough to stop?”

Ratings uses concurrency as a controlled resource. Instead of one winner, several candidates participate, bounded by max_concurrent, and the router aggregates successful responses. This is useful when operators want ensemble behavior or live comparison without letting fan-out become unbounded.

Fusion is the clean panel pattern. It sends the request to analysis models, asks a judge to identify consensus, contradictions, partial coverage, and blind spots, and then synthesizes a final answer. The important design point is that Fusion policy lives under the matched decision. vllm-sr/auto can decide whether Fusion is warranted; vllm-sr/fusion narrows matching to Fusion-capable decisions instead of silently falling back to ordinary single-model routing.

ReMoM is the multi-round version of the same philosophy. It uses a breadth schedule such as [3, 2] or [32, 4], distributes calls across model candidates, compacts intermediate responses when needed, and synthesizes the final answer. This is useful when the value comes from exploration over multiple reasoning paths rather than one panel pass.

Router Flow turns the route into a bounded micro-agent workflow. A static flow can define roles such as thinker, worker, verifier, and final synthesizer. A dynamic flow can ask a planner model to produce a plan, but worker execution remains constrained to the decision’s modelRefs. Tool calls preserve the OpenAI-compatible contract while the router stores enough workflow state to resume the correct worker after tool results return.

The Looper layer is the bridge from routing to model collaboration. It lets the router scale capability through multiple models while keeping policy, traces, cost boundaries, and public API shape explicit.

Router Memory and Learning: Adaptation Is Not an Algorithm

Session-aware and learning-related behavior should not be hidden inside decision.algorithm. In the clean vLLM SR design, this belongs to Router Learning.

The distinction matters. A decision says what is allowed. A selection or looper algorithm produces a base result. Router Learning then asks whether the system should adapt that result from runtime experience, and whether switching is safe in the current session or conversation.

Learning can improve a route inside policy. It must not become a second, invisible policy system.

The runtime order is fixed:

The public learning concepts are intentionally small:

Adaptation’s day-0 strategy is routing_sampling. It scores candidates from local experience: quality seed, good-fit outcomes, underpowered outcomes, overprovisioned outcomes, failures, latency evidence, cache reuse, effective input cost, and reliability. The default candidate set is decision, which means adaptation may only choose among the matched decision’s modelRefs. Broader scopes such as tier and global are more powerful, but they need stronger guards.

Protection is the session-aware half. It has a preflight guard and a switch guard. Preflight suppresses stochastic sampling during tool loops, protocol-sensitive continuations, or routine continuation steps. The switch guard decides whether to hold the current model, allow the proposal, or perform a bounded rescue switch. The simplified rule is:

switch if proposal_gain >= switch_margin + stability_weight * switch_cost

The switch cost can include cache warmth, handoff cost, tool-loop state, provider state, turn count, and switch history. Session-aware routing is therefore not sticky sessions. It is controlled continuity. The router keeps a model when switching is unsafe or not worth it, and it can switch again at idle boundaries, decision drift, or rescue conditions.

Sensitive routes can bypass learning entirely:

routing:
  decisions:
    - name: local_privacy_policy
      modelRefs:
        - model: local-private-model
      adaptations:
        mode: bypass

That boundary is the contract. Learning can improve choices inside recipe policy. It cannot silently rewrite the recipe, add a new privacy exception, change a decision priority, or mutate modelRefs on the request path. Offline recipe learning can propose those changes as reviewable artifacts, but live routing remains governed by the recipe.

Plugins: Behavior Belongs to the Route

After a route is selected, the request still may not be ready for the model. It may need cache, memory, RAG, tool filtering, request parameter caps, prompt mutation, response verification, fast policy response, image generation, or replay.

These behaviors should not be global decoration. Privacy routes may need to bypass cache. High-risk factual routes may require verification. Agentic routes may need tool boundaries. Low-risk summarization may need only replay. Plugins are therefore route-scoped.

Some plugins are worth reading as miniature subsystems.

A route is better understood as an execution contract. The model is one part of it. The route also carries constraints, tools, knowledge, verification, memory, and evidence.

The Hot Path: Header, Body, Route, Response

The conceptual architecture only becomes convincing when it touches the request path. In vLLM SR, the hot path is shaped around a simple constraint: the router must see enough context to make a semantic decision, but it should avoid turning every request into an expensive full parse and full detector run.

The route lifecycle inside the gateway path looks like this.

Protocol compatibility is part of the same story. The router should not hide provider differences behind a vague facade. It should translate them explicitly at the boundary: OpenAI-compatible chat, Responses-style calls, Anthropic-style provider routing, direct looper model slugs, and backend-specific provider profiles all become inputs to one routing engine. Applications keep a familiar API shape, while the infrastructure keeps the differences visible enough to debug.

The ExtProc path matters because it gives semantic policy a boundary-native shape. vLLM SR can receive enough request and response context to make semantic decisions while still returning control to the real gateway data plane.

Envoy Integration: Put Semantic Policy at the Boundary

The Envoy integration shows the intended boundary clearly.

Envoy owns the data plane: TLS, clusters, endpoint health, timeouts, retries, load balancing, and filter chains. vLLM SR should not rebuild those capabilities. It should act as the semantic policy plane: receive request context through External Processing, evaluate the route, and return header/body mutations.

The adoption advantage is large. Applications do not need a new private API just to benefit from semantic routing. They can keep calling familiar model APIs while infrastructure maps logical model names to small models, frontier models, RAG paths, verifier paths, Looper collaboration paths, workflows, or private hardware lanes.

Protocol work such as SIRP and multi-provider inference API matters for the same reason. Semantic routing should strengthen the control plane without forcing every application team into a custom gateway dialect.

Model Design: Model Name Is the Wrong Primitive

The router cannot make production-grade decisions from model names alone.

gpt-4, qwen3-32b, claude-opus, or local-private identifies an endpoint or alias. It does not describe reasoning ability, coding strength, tool behavior, vision support, context window, latency distribution, cost, hardware path, privacy boundary, or observed failure modes.

Users can ask for auto. The system cannot treat auto as a magic endpoint. Internally, it must expand into model cards, candidate sets, route policy, Looper eligibility, learning boundaries, and execution paths.

Bindings: Fast ML Without Turning the Router Into a Model Server

vLLM SR’s control plane is written around Go because gateway integration, configuration, request mutation, response processing, Envoy ExtProc, and Kubernetes-style infrastructure fit Go well. But routing also has ML hot paths: embeddings, classification, modality detection, LoRA classification, and MLP selectors.

The binding layer keeps those concerns separated.

The rule is not “everything must be native.” The rule is capability must be explicit. If a deployment lacks a native classifier, the router should know. If ONNX supports only part of the contract, policy should not pretend otherwise. If backend lifecycle needs reset boundaries, the router should expose that instead of hiding it. Intel/OpenVINO-oriented paths can be valuable deployment options, but they belong in the hardware/runtime discussion unless they are part of the same advertised native capability contract.

Hardware Is a Routing Variable

Hardware support is often described as a compatibility matrix. For semantic routing, it is more than that.

Different hardware paths imply different latency, cost, memory behavior, kernel availability, quantization support, context-window economics, privacy boundary, and energy curve. A router that ignores hardware is only doing half the job.

The acceleration story is not one-size-fits-all. CUDA gives the broadest default serving path. ROCm/AITER makes AMD pools a serious production option rather than a compatibility afterthought. Intel XPU and OpenVINO-style paths matter for enterprises that already own Intel-heavy infrastructure or need CPU/XPU portability. CPU and local paths remain important because privacy, availability, and cost sometimes beat raw throughput.

WRP is the right mental model here. Workload is semantic. Pool is physical. Router translates between them.

A future router should know when CUDA is overloaded, when ROCm is cost-effective, when XPU is sufficient, when CPU/local is preferable for privacy, and when preserving KV cache is worth more than reselecting a theoretically better model.

Observability: The Router Must Leave Evidence Behind

If the system makes semantic decisions, operators need to see those decisions.

Replay is not only for dashboards. It is the second control plane. The first control plane makes the live decision; replay preserves enough evidence to debug that decision, compare it against alternatives, attach outcomes, and produce offline recipe patches.

If the router records what it saw, what it decided, what happened, and how users, agents, verifiers, or evals responded, the next generation of selectors, thresholds, learning strategies, and recipes can improve.

Without traces, the router is another opaque model. With traces, it becomes a system component.

Evaluation: The Router Is a Frontier

The wrong way to evaluate a router is to count features. A router with many knobs can still be bad. The right question is whether it improves the frontier between quality, cost, latency, safety, privacy, reliability, and hardware efficiency.

Router evaluation, fleet simulation, replay traces, RouterArena-style comparison, Looper evals, and offline recipe learning matter because semantic routing is infrastructure. It has to be measured like infrastructure.

The Part I Care About Most

The most important design belief in vLLM SR is not any single signal, plugin, model, algorithm, or hardware platform. It is the separation of concerns.

Signals observe. Projections coordinate. Decisions express policy. Selection algorithms choose one model. Looper algorithms scale capability through model collaboration. Router Learning adapts and protects within recipe boundaries. Plugins mutate behavior. Bindings accelerate hot-path ML. Envoy integration places semantic policy at the traffic boundary. Model cards connect logical names to real capabilities. Hardware metadata connects semantic constraints to physical execution. Observability makes every decision accountable.

That separation is what lets the router evolve without turning every new idea into a fork of the hot path.

A new jailbreak detector can become a signal. A new risk formula can become a projection. A new compliance rule can become a decision. A new selector can become a selection algorithm. A new panel or workflow primitive can become a Looper algorithm. A new verifier can become a plugin. A new accelerator lane can become model-card metadata. A new benchmark can become an evaluation loop. A new fleet model can feed pool-aware routing. A new learning strategy can propose better candidates without rewriting policy.

That is why I like the phrase Intelligence Control Plane. Not because the router is always intelligent by itself, but because it gives the system a place to allocate intelligence deliberately.

The first stage of AI infrastructure made intelligence callable.

The next stage has to make intelligence allocatable, explainable, and optimizable.

Calling a model was the first abstraction. Allocating intelligence is the next one.

That is the philosophy of vLLM SR.

Source Trail

This article is based on the vLLM Semantic Router codebase, website research archive, vLLM project blog posts, my earlier essays on LLM routing, and infrastructure references around Envoy, vLLM, ROCm, Intel XPU, and Gateway API inference routing.

That work mattered because it made AI production systems possible.

But it also started to converge. Once every AI gateway begins to look like the same recipe, the next question is no longer how to forward requests more reliably. The question becomes what “routing” should mean when the backend is no longer a deterministic service, but a model that can reason, act, call tools, fail in subtle ways, and change behavior through feedback.

Back then, I described the second half this way:

In the first half, routing transports requests. In the second half, routing builds collective intelligence.

I still believe that, but after another half year of watching enterprise AI adoption, open source inference, model releases, agentic coding, and infrastructure teams trying to control real token bills, I would say it more directly.

The second half of routing is not only the move from endpoint selection to behavior control. It is about intelligence resource allocation: deciding how much intelligence each piece of semantic work deserves, where that intelligence should run, and what cost, latency, privacy exposure, hardware capacity, and energy footprint the system is allowed to spend.

That is why semantic routing now feels less like an API gateway problem and more like energy infrastructure.

1. The Bill Arrives Before the Architecture

Every computing era begins by making a scarce resource feel abundant.

The industrial age made mechanical force portable. The cloud era made compute elastic. The AI era has made something stranger callable: intelligence, exposed through APIs, metered by tokens, and inserted into millions of workflows.

The first visible effect is acceleration. A frontier model makes a product feel smarter. An agentic coding tool changes how developers work. A support assistant compresses repetitive labor. The new resource looks flexible, almost liquid.

Then the bill arrives, and the architecture becomes visible.

Salesforce: Token Spend Becomes Architecture

In May 2026, Business Insider reported that Salesforce CEO Marc Benioff said the company could spend around $300 million this year on Anthropic tokens. The number is eye-catching, but the architecture question behind it matters more.

Benioff pointed toward an intermediary layer that decides which inputs deserve a frontier model and which can be handled by smaller models. That is the moment model choice stops feeling like a developer preference and starts becoming infrastructure strategy.

Microsoft: Tool Choice Becomes Control

Around the same time, The Verge reported that Microsoft was preparing to wind down many Claude Code licenses internally and move more developers toward GitHub Copilot CLI. The point is not a verdict on Claude Code; the report described it as popular inside Microsoft. The point is that popularity did not settle the architecture.

At enterprise scale, an AI coding tool is also a security boundary, a developer workflow, a cost center, and a platform surface. That is not per-request semantic routing, but it is the same discipline: raw capability is only one variable; control of the decision layer starts to matter just as much.

Red Hat: Hybrid AI Needs Routing

Red Hat framed a related tension as the “agentic paradox”: frontier models are often the fastest path to agentic adoption, but routing every agentic workload to them becomes difficult to sustain at enterprise scale because of cost, latency, confidentiality, sovereignty, and control.

NVIDIA: The Supply Side Gets Faster

There is also a supply-side version of the same pressure. NVIDIA recently described AI data centers as token factories, arguing that AI efficiency and revenue scale through better performance per watt. That is exactly what frontier hardware and inference engine teams are trying to do: squeeze more useful tokens out of every watt of power and every dollar of infrastructure.

These stories are not identical. One is about token bills. One is about the cost and governability of AI tool surfaces. One is about hybrid AI. One is about performance per watt. But together they show the same transition:

AI is moving from capability discovery to resource discipline.

The supply side will keep improving tokens per watt per dollar. The demand side now needs to decide which semantic work deserves those tokens in the first place. A faster token factory still wastes power if every workload is sent through the most expensive lane.

Frontier models remain the fastest way to discover new capabilities, but they are not, by themselves, the architecture of scale.

2. Tokens Are Not the Unit

The industry likes tokens because tokens are easy to count. They are convenient for billing, quotas, and dashboards. They turn something messy, language, into something finance and infrastructure teams can track.

But tokens are a billing unit, not a value unit.

The same token count can pass through a small local model, a specialized open model, a frontier closed model, a reasoning model, or a model running on an edge device. The bill, latency, energy footprint, privacy risk, and intelligence density are all different.

Counting tokens is like counting kilowatt-hours without asking what the electricity powered. One kilowatt-hour used to keep a hospital operating and one kilowatt-hour wasted by an idle machine share the same unit, not the same value. AI systems have the same problem.

One token may be cheap autocomplete. Another may be the deciding step in a high-risk reasoning chain. One request may be a low-stakes formatting task. Another may require retrieval, verification, policy checks, and a stronger model because the failure cost is high.

So the real question is not:

How many tokens did we use?

The real question is:

Did we spend the right kind of intelligence in the right place?

That is a different kind of accounting. It asks the system to understand the semantic workload before spending compute. Is the task simple or ambiguous? Private or public? Reversible or dangerous? Latency-sensitive or quality-sensitive? Cheap enough to answer locally, or important enough to escalate?

This is where token economics becomes system design.

3. Frontier-by-Default Is a Temporary Architecture

The simplest AI architecture is always:

Send everything to the strongest model.

It is easy to build. It is easy to explain. It gives good demos. It minimizes engineering effort in the short term because the model absorbs complexity that the system does not know how to model yet.

In the early stage of adoption, that is rational.

When a team is trying to prove AI works, frontier-by-default is often the fastest path. It reduces product risk. It gives users a better first experience. It lets teams focus on workflow and adoption instead of building a routing layer too early.

But architecture that works for discovery often breaks at scale.

When every workflow becomes AI-assisted, when agents run continuously, when coding tools expand across thousands of developers, when customer support, sales, analytics, compliance, and internal operations all begin to consume model calls, the question changes.

The company no longer asks:

Can AI do this?

It asks:

Can we afford this, control this, govern this, and scale this?

That is when frontier-by-default starts to look less like an architecture and more like a subsidy paid by the early phase of adoption.

The future is not one frontier model serving every request. The future is a heterogeneous fabric of intelligence:

• frontier closed models for high-value reasoning and hard escalation paths
• open-weight models for controllable, private, and cost-efficient workloads
• small models for classification, extraction, intent detection, routing, and frequent low-risk tasks
• domain models for specialized knowledge and enterprise context
• verifiers for checking claims, actions, policies, and citations
• retrieval systems and memory layers for grounding
• edge models for local privacy, low latency, and near-zero marginal cost
• different generations of hardware that still need to be used efficiently

This is not anti-frontier. It is what mature infrastructure does: preserve the most capable resource for the places where it actually changes the outcome.

Electric grids do not route every load through the most expensive power source. Networks do not send every packet through the same path. Cloud platforms do not run every job on the largest instance type.

Mature systems differentiate demand, and AI systems will have to do the same.

4. Hybrid AI Is Not Just a Deployment Topology

If frontier-by-default is the temporary architecture, hybrid AI is what comes after.

But hybrid AI is not a slide with one cloud model, one private model, one open model, and one edge model. That is inventory. Architecture starts when the system can decide what each piece of work actually deserves.

The architecture question is sharper:

Who decides where semantic work should go?

A hybrid system without routing is just more endpoints. It may lower cost in obvious cases, but it cannot reason about the tradeoff between quality, latency, privacy, risk, and energy. The missing piece is a control layer that turns workload signals into capability paths.

A path may be:

• answer directly with a small model
• retrieve first, then answer with a mid-size model
• classify risk, then apply a stricter policy
• ask a clarification question before spending expensive reasoning
• run a verifier before returning the answer
• escalate to a frontier model only when uncertainty or failure cost justifies it
• keep the task on-device because privacy matters more than marginal quality

This is why “the right model for every request” is a useful slogan but not the whole idea. The real unit of architecture is not always a model; it is a capability path.

5. Semantic Routing as Energy Infrastructure

At small scale, model choice feels like a quality decision. At infrastructure scale, it becomes an allocation problem, which is why I keep coming back to energy.

Energy here is not a decorative metaphor. I do not mean semantic routing is the power grid; I mean intelligence is now a metered, energy-consuming resource, and the system needs a way to decide when spending it is justified.

The wrong lesson is “use cheaper models.” Cheap-by-default is the mirror image of frontier-by-default. Both avoid judgment.

The real point is to budget intelligence. A small model may be enough for routine work. An open model may be right when control and cost matter. A frontier model is still worth it when reasoning depth or failure cost justifies the bill. Sometimes a verifier beats another generation step. Sometimes retrieval beats reasoning. Sometimes asking one clarifying question is the most efficient path.

In the old internet, routing moved packets. In AI systems, the traffic is semantic work: intent, uncertainty, context, memory, privacy risk, tool use, and action. Once traffic carries meaning, routing stops being thin forwarding and becomes allocation.

For me, this is the first-principles reason semantic routing matters: it treats intelligence as a resource to schedule, not a magic endpoint to call.

The industry is already attacking the supply side. Hardware teams and inference engine teams are doing exactly what they should do: maximize tokens per watt per dollar.

Better accelerators, better kernels, better batching, better KV-cache systems, better speculative decoding, better serving engines, and better cluster scheduling all push in the same direction: more useful tokens from the same power envelope.

But supply-side efficiency is only half the problem. The other half is demand-side allocation.

If every semantic workload can consume the most expensive path, the system is powerful but wasteful. If every workload is forced onto the cheapest path, the system is efficient but dumb. The hard problem is knowing when each path is worth it.

That is what turns hybrid AI from a deployment pattern into an energy architecture.

6. The Layer We Are Missing

But an energy architecture cannot stop at the slogan level. If semantic routing is going to decide how intelligence is spent, the decision layer has to become something engineers can inspect, policy teams can constrain, and infrastructure teams can improve.

That is the missing layer: not another model endpoint, but the system primitives that make routing decisions expressible, observable, measurable, and compatible with the rest of the serving stack. I think it needs at least six pieces.

1. Workload signals.

The system has to understand more than prompt length and tenant ID. It needs signals for intent, domain, difficulty, uncertainty, privacy, safety, tool requirements, expected output type, failure cost, and historical behavior. Without workload signals, routing collapses back into static rules and cannot make intelligent allocation decisions.

2. Routing memory.

A router should not repeat the same decision blindly. It should remember which paths worked, which failed, which users or tasks require stricter handling, and which policies caused friction. Routing without memory cannot improve; it can only re-run yesterday’s assumptions.

3. Policy languages.

As routing becomes more powerful, organizations need a way to express constraints: what can run where, which data can leave the boundary, when a verifier is mandatory, when escalation is allowed, and how cost-quality tradeoffs should be handled. Natural language prompts are not enough for infrastructure policy.

4. Evaluation.

If a router claims to be intelligent, it has to be measurable. Accuracy alone is not enough. We need cost-quality frontiers, task completion rates, latency distributions, privacy guarantees, safety outcomes, stability across multi-turn sessions, and evidence that the system improves through feedback. Otherwise “routing” becomes another hidden optimization with no accountable frontier.

5. System co-design.

I do not think this layer can be designed as an isolated policy box. The useful architecture is a co-designed loop: Workload -> Router -> Inference Pool. Workload signals shape routing decisions; routing decisions change batching, caching, queueing, model placement, KV-cache pressure, and hardware utilization; pool state should flow back into routing. A router that ignores the pool is incomplete, and an inference pool that ignores the workload is blind.

6. Transparent interfaces.

If this layer becomes important, it should be mostly invisible to applications and users. Product teams should not have to adopt a new serving surface just to benefit from better routing. OpenAI-compatible APIs, existing gateway protocols, and normal chat, completions, embeddings, and tool-calling semantics should keep working. The intelligence of the router should show up in better placement, policy, observability, and cost-quality behavior, not in a new API tax on the user.

That is where open source matters: not because every company will run the same router, or because one project should own the entire space, but because shared infrastructure needs shared language and shared contracts. It needs common workloads, common metrics, common failure cases, common policy concepts, and compatibility expectations, so routing can become infrastructure rather than a private product trick.

7. Why We Built vLLM Semantic Router

This is the problem space that led us to build vLLM Semantic Router.

The project started from a simple frustration: most routing layers could tell whether a backend was alive, busy, or cheap, but they knew very little about the work itself.

Early AI gateways made LLMs callable and manageable. That foundation is necessary. But once a system has multiple models, policies, risk levels, and deployment boundaries, routing has to answer a harder question:

Given this workload, this policy, this budget, this model pool, and this risk profile, what capability path should the system choose?

This is the part of Red Hat’s Technically Speaking conversation between Red Hat CTO Chris Wright and Steve Watt that matters to this argument. Their discussion around vLLM Semantic Router is not framed as another model-routing demo. It is about where enterprise AI needs control: which workloads can stay private, which policies decide the route, and how a broader accelerator landscape changes the economics underneath.

vLLM Semantic Router is one attempt to make that question concrete in open source. It is not the final answer, and it should not be. The field is still early: the right abstractions are being discovered, evaluation is immature, enterprises have different policies, models are changing quickly, hardware is changing more slowly, and edge, cloud, private deployment, and frontier APIs are evolving at different speeds.

That mismatch is exactly why routing matters. Models will keep improving, and hardware will keep improving, but mixed fleets do not automatically become intelligent systems. Someone still has to decide which request should touch which model, which data should cross which boundary, which accelerator should be used, and when the system should spend more reasoning instead of returning a cheaper answer.

There will be old GPUs, new GPUs, edge devices, private clusters, cloud APIs, open models, closed models, specialized models, verifiers, memory systems, and agentic tools all coexisting. The question is not whether one of them wins everything. The question is how to use all of them well.

The future of AI infrastructure is not one model. It is a system that knows how much intelligence each task deserves, and can explain the boundary it drew.

8. Before Power Becomes Tokens

My bet is simple: the next important AI infrastructure layer will sit before the token is generated. It will decide whether a piece of work deserves frontier reasoning, open-model inference, local compute, retrieval, verification, or no compute yet.

That layer is semantic routing.

In the cloud era, resource allocation was mostly deterministic. You asked for a machine, a container, or a node with a known shape: 1C / 2G, a memory limit, a CPU quota, a namespace, a cgroup. The boundary was explicit. The system could isolate it, meter it, schedule it, and enforce it.

AI changes the allocation primitive. The request is often a prompt, but the prompt carries more than text. It carries intent, uncertainty, context, risk, privacy, expected quality, and failure cost. In the cloud era, users asked for resources by declaring a shape. In the AI era, they increasingly ask for resources by describing a goal. The boundary is no longer only CPU and memory. The boundary is semantic.

This is why semantic routing is closer to virtualization than to a load balancer. Cloud virtualization turned physical machines into isolated, secure, controllable pools of compute. Applications expressed requirements; the infrastructure handled placement, limits, isolation, and enforcement.

AI needs the same abstraction for intelligence. A user should not have to reason directly about the frontier model, the open model, the local model, the verifier, or the accelerator generation underneath. Those are the substrate: intelligence and energy. The interface should be preference and objective: quality target, latency budget, privacy boundary, cost ceiling, safety policy, sovereignty constraint, and risk tolerance.

Every token begins as something physical: watts, hardware time, cooling, memory bandwidth, model capacity, and operational risk. Hardware and inference engines will keep making that conversion more efficient. But efficiency is incomplete if the system still spends its best intelligence on the wrong work.

This is the kind of control plane I think AI needs: one that reads workload signals, applies policy, respects privacy, understands the model and hardware pool, and decides how much intelligence a request is allowed to consume. In older systems, cgroups and namespaces made compute boundaries enforceable. AI needs the same kind of enforcement around intelligence: not only what can run, but how much reasoning, where, and under which constraints.

Frontier models remain essential. They expand what builders can imagine and show the rest of the stack what it must learn to support. But mature infrastructure is not defined by sending every request to the strongest resource. It is defined by knowing when that resource changes the outcome.

The first phase of AI was about access to intelligence. The next phase is about allocation.

Six months ago, I wrote that the second half of LLM routing would build collective intelligence. I would put it more sharply now:

The next stage of routing is where AI systems learn to budget intelligence.

Not just dollars. Watts, latency, privacy exposure, hardware scarcity, failure risk, and human trust.

That is why semantic routing is energy infrastructure: it defines the boundary of AI resource allocation before power becomes tokens.

Over the past few years, “LLM routing” has been treated as traditional networking problem in gateway engineering for the AI era: reliably send requests to a model, stream tokens steadily, and keep latency, cost, and availability within an SLA. It worked — and it worked too well. So well that it’s getting platformized and standardized into a reusable industrial recipe.

So what’s suddenly different?

In one sentence: models are exploding in number; the traditional AI-gateway recipe now works — and it generalizes — so AI gateways are multiplying. Not just for a single provider, but across providers; not just for one deployment style, but across the whole chain — from cloud APIs to self-hosted inference clusters.

It’s getting hard to say “I built an AI gateway,” because you quickly realize what you built is almost the same thing everyone else built.

When a general recipe solves more and more problems, incremental methods get steamrolled by standardized pipelines. The game shifts from solving problems to defining problems. Evaluation becomes more important than adding more features in the same road.

I think the second half of LLM routing starts at the same moment: when innovation gets squeezed out, we should rethink what “routing” even means in the AI era.

The first half

To understand the first half, look at its winners.

In the world of LLM routing, what’s the most “useful,” most “shippable,” and most “sellable to enterprises”? The winners are a complete Traditional AI Gateway / Inference Gateway capability set — productized faster than anything else:

• Like a traditional API gateway: unify backend LLM services, handle auth, token quotas, rate limiting, circuit breaking & retries, failover, canary releases, A/B tests, and auditing.
• Like an inference engine: load-awareness, queues & priority, prefix caching, P/D disaggregation, and scheduling across endpoints.

These solved the first-half core problem: making LLMs behave like a manageable backend service.

They share one thing: routing decisions are mostly based on system state (health, load, price, SLA, tenant policy). Models are endpoints; requests are packets.

Sure, you can “pick a model.” But you don’t have to answer “why,” and you don’t have to own whether the task actually got done.

That was perfectly reasonable in the first half: when a system is just getting started, fast productization and reliability are the value.

The recipe

So what force kicks off the second half?

A brutal-but-true structural force: homogenization.

If you study multiple AI gateways seriously, you reach a very unromantic conclusion: they look extremely similar. Innovation space gets compressed into “plugin lists” and “product packaging.” It’s not that people aren’t smart — it’s that the first-half objective function naturally converges to the same engineering solution.

You almost always see the same recipe:

• A single unified entry point (usually an OpenAI-style API) that hides provider differences.
• Token-based quota & cost governance (otherwise finance can’t close the books).
• The reliability trio: retries, fallback, circuit breaking, plus canary/AB.
• Observability & audit: usage, cost, errors, latency, tenant attribution.
• Inference scheduling & optimization: queues, priorities, load-awareness, prefix caching and reuse.

That’s why it’s called a “recipe”: it’s general, replicable, and industrializable.

Once the recipe is established, the industry enters a familiar phase — the recipe crushes incremental methods.

So the question becomes sharp:

If the first-half recipe already works and gets increasingly standardized, what are we even playing in the second half? Just keep adding items to the plugin menu?

I don’t think so. Because the nature of what sits behind the routing layer has changed.

In the traditional API world, backend services are closer to deterministic systems: you call an API, it either succeeds or fails, and the semantics are relatively stable. Gateway traffic routing makes perfect sense.

But in the LLM world, backend services are generative — and increasingly agent-like: they reason, act, call tools, and change the external world. The “backend” is no longer a passive service. It’s an entity that makes decisions, makes mistakes, and can self-reinforce.

If you still treat routing as pure forwarding, you’re effectively trying to govern a “thinking, acting” system with the abstraction of a transparent proxy.

So the second-half recipe must upgrade:

Make decisions more deliberative — and let the system become more reliable through feedback from the environment.

Routing needs Reasoning + Acting

The core of ReAct isn’t “write a longer chain-of-thought.” It’s interleaving reasoning and actions: reasoning helps planning, error correction, and policy updates; actions connect the system to the environment to obtain observations — reducing hallucinations and improving interpretability.

Bring this into routing and you get a direct conclusion: routing shouldn’t just make decisions based on static signals. Routing should also think first, act, then decide. It needs intelligence.

The “actions” can be concrete: retrieve first, run a verifier first, have an SLM clarify intent first, run safety checks on high-risk spans first, do a lightweight probe (e.g., use a cheap model to test uncertainty), and only then decide whether to escalate to a stronger model.

In other words, second-half routing is not just a traffic router.

It is an agent brain doing decision-time compute.

Routing needs search, backtracking, and self-evaluation

Tree of Thoughts (ToT) says: standard LM inference is a left-to-right greedy process at the token level, but complex tasks require exploration, foresight, and backtracking. So ToT treats “thinking” as searchable intermediate states: expand multiple paths, self-evaluate, and roll back when needed.

Bring that into routing and you end up redefining what “a route” even is: routing isn’t an if-else or a one-shot classification. It’s an explorable “strategy tree” — different paths correspond to different capability bundles (model / reasoning mode / tools / verification / caching / audit).

For the same request, you might expand multiple candidate paths:

• Path A: SLM direct answer (fast, cheap, risky)
• Path B: mid model + retrieval (steady, higher cost)
• Path C: strong reasoning model + verifier (most reliable, most expensive)
• Path D: clarify first, then answer (push uncertainty to the user)

The ToT lesson is: don’t pretend you can one-shot it. Allow the routing layer to do bounded search, using self-evals and external observations to choose next steps — and backtrack when necessary. Routing shifts from “splitting traffic” to “planning.”

Routing needs feedback and reflexion

Reflexion goes one step further: it decouples “getting stronger from mistakes” from weight updates. You don’t necessarily need to fine-tune. The system can turn task feedback into language-based reflection, write it into episodic memory, and make future decisions better.

Bring that into routing and you get the most critical step of the second half: the routing system must be reflexive. Otherwise it will keep repeating the same class of errors.

And feedback sources can be painfully real: positive/negative user feedback, whether the task was resolved in one pass, tool-call failures, factuality checks, KV-cache hit rates, whether a policy caused complaints or compliance incidents… These aren’t “training data,” but they are the most valuable online signals a system can get.

So the second-half recipe becomes clearer:

• The first half: “unified entry + token governance + reliability + pluginization + observability.”
• The second half must add: “semantic signals + think-then-decide + tree search + feedback/reflexion.”

That’s the bridge: homogenization isn’t bad — it means the infrastructure layer is getting solved. Real incremental innovation must move to a new abstraction layer:

move reasoning / acting / searching / reflexion up into the routing layer.

There’s an even deeper driver: collective intelligence

There’s also a more “anthropological” driver:

Human intelligence has never been just solo brilliance — it’s a composition of collective intelligence.

Teams have a measurable “collective intelligence factor” that explains performance across tasks, beyond just the maximum or average individual IQ.

MIT’s Center for Collective Intelligence even frames its core question as: how can people and machines be connected so they can “collectively act more intelligently,” i.e., build “superminds.”

Bring that lens into LLM systems and you get a natural analogy:

LLMs are moving from “one big model fighting alone” to “a collective system of multiple models + tools + verifiers + memory.”

Routing should be the organizing mechanism of that collective intelligence.

If you treat SLMs, LLMs, reasoning models, verifiers, retrievers, and tool executors as different “experts,” then the second-half routing task is no longer “pick an endpoint.” It becomes the classic collective-intelligence problem: diversity, independence, decentralization, and aggregation — letting the system collaborate without groupthink, and aggregating results into a useful delivery for the user.

That’s why I say second-half routing is more like a “brain.” It’s not just dispatching traffic — it’s organizing an intelligent group.

The second half

The core change of the second half can be summarized in one sentence:

the object of routing shifts from endpoints to behavior (system behavior).

You’re no longer just choosing “which model.” You’re choosing “how the system should behave”:

• Should we enable reasoning? At what intensity?
• Should we clarify first, then answer? Retrieve first, then reason?
• Should we run a verifier? Force citations? Enable stricter audit policies?
• If semantics drift across turns, should the strategy migrate accordingly?
• Is the failure cost high? Should we be conservative or aggressive?

If first-half routing is traffic control, second-half routing is closer to an Intelligence Control Plane: decisions driven by signals, constrained by policy, uncertainty reduced via search and actions, and continuously evolved through feedback and reflexion.

And this forces you to redo three things: Evaluation, Stateful, Learning. Because the moment you optimize for “best system behavior,” you’ll realize single-turn accuracy does not represent real-world utility.

Thesis: System Intelligence

The next stage is system intelligence: pushing intelligence up to the system level.

Under a mixture-of-models (SLM + LLM) architecture across edge + cloud, users shouldn’t need to care which model got called — the experience should feel more like “Auto”: the system chooses the capability path.

There are three directions. But they’re not three feature cards — they are three parts of a closed loop: decision, collaboration, governance.

1) Semantic Routing: signal-driven decision

Capture higher-value signals — the semantic signals we used to ignore.

• In the first half, signals mostly came from system state: load, health, cost.
• In the second half, signals must come from semantics and task state: domain, intent, uncertainty, risk level, factuality cues, user-feedback trajectories, hidden attack graphs inside conversations, etc.

More importantly: these signals shouldn’t just be used to “stamp a label.” They should feed into a Think-then-Decide loop: reason about what we’re missing and what we’re uncertain about, act to fill the gaps, then make the final decision (a routing-native version of ReAct).

2) Cross-LLM Intelligence: make the experts collaborate

If you view the system as collective intelligence, Cross-LLM is not “pick one” — it’s “divide work + aggregate.”

Different models vary wildly across tasks. Different verifiers, retrievers, and tool executors each excel at different subtasks. The second half is about enabling them to collaborate with minimal coordination overhead, while preserving independence to avoid error amplification through resonance (the classic failure mode of group systems).

• ToT gives a direct playbook: treat “what to do next” as a tree search, choose paths using self-evaluation and external feedback, backtrack when needed.
• Reflexion provides another axis: write failures into reusable reflective memory as a prior for future aggregation and scheduling — the system becomes more and more like a real team as it runs.

3) Guardrails: governance is not an add-on — it’s the “team norm” of collective intelligence

Once you build a collective system, the role of guardrails changes:

It’s not just blocking dangerous content. It’s closer to organizational: permissions, audits, norms — who can do what; which scenarios must escalate to stricter paths; which actions require double checks (verifiers); which information must cite sources; which failures require mandatory postmortems written into reflective memory.

In multi-turn conversations, attackers can decompose harmful intent into multiple harmless-looking prompts, gradually steering toward the target through a dialogue graph. Second-half guardrails must have session-level state and graph-level evidence, otherwise single-turn scanning is trivially bypassed.

Where it lives

System intelligence can live in three places — more like three product lines:

• In-engine: closer to inference engine. Good for multi-LoRA auto-selection, cross-instance semantic caching/stateful communication, and low-latency built-in guardrails.
• Above-LLM: a cross inference engine / external-provider control plane. Good for multi-model collaboration, memory management, policy search, and reflexion loops. It looks like an agent, but I prefer to call it the “brain above LLMs”: organizing collective intelligence.
• On-edge: a local-first product line. Put a lightweight intelligent router + SLM on-device to handle privacy-sensitive and high-frequency requests with near-zero marginal cost. When uncertainty, task difficulty, or policy requires more capability, the router escalates to remote LLM providers.

Routing becomes not only a control plane for cloud models, but also a boundary manager between local and remote intelligence — optimizing quality/cost and data exposure, connectivity, and energy.

Evaluation Matters

As models multiply and routing gets smarter, we can almost predict what happens next: more and more LLM routers —

open-source, commercial, cloud-native, bundled with inference engines.

The hard question is: how do we evaluate an intelligent router?

For traditional gateways, evaluation is straightforward: latency, throughput, SLA, error rate, cost.

But second-half routing doesn’t just pick an endpoint; it picks behavior: when to reason, when to retrieve, when to clarify, when to refuse, when to escalate to heavier paths.

So evaluation must upgrade:

What dimensions should define “good system intelligence”? Task completion rate? Multi-turn efficiency? Stability and consistency? Risk and compliance? The Pareto frontier of cost vs quality? Or whether the system becomes more reliable under long-term feedback?

Interestingly, we’re already seeing early “evaluation systems” emerge — for example, RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers: a public platform/leaderboard that puts routers under the same dataset, difficulty tiers, and metrics — and supports automated updates.

At minimum, it signals one thing: routing intelligence must be measurable. Otherwise it’s a castle in the air — and the narrative will quickly shift from “what features do you have” to “how good are you under the same constraints.”

This will take real effort and real resources.

In the End

Looking back at the first-half game:

• We turned LLMs into callable, governable, scalable services.
• We converged on unified entry, token governance, reliability, and inference scheduling.
• Homogenization arrived; the recipe started crushing incremental innovation.

The second-half game should be:

• Admit the backend has changed: it’s uncertain, and it can lie.
• Move reasoning + acting into routing decisions: think first, act, then decide.
• Move search and backtracking into routing search: deliberate over a strategy tree, not one-shot classification.
• Move feedback and reflexion into the routing loop: without changing weights, become more reliable through memory and reflection.
• Upgrade from solo intelligence to collective intelligence: make multiple models, tools, and verifiers collaborate like a team — and routing is the team’s organizing mechanism.

• In the first half, routing transports requests.
• In the second half, routing builds collective intelligence.

Welcome to the second half of LLM routing!

Acknowledgements

This blog post is inspired by The Second Half by Shunyu Yao.