The Philosophy of vLLM SR

The easiest way to misunderstand vLLM Semantic Router is to start from the word router.

In a traditional gateway, routing is a forwarding problem. A request arrives, the gateway evaluates policy, picks a backend cluster, and sends the request there. The backend is usually a deterministic service. If it is healthy, the request succeeds. If it fails, the gateway retries, fails over, or returns an error.

LLM systems break that mental model.

The backend is no longer just a service. It is a probabilistic reasoning system. It may need tools. It may need memory. It may need retrieval. It may hallucinate. It may require a verifier. It may run on a GPU pool whose KV cache is warm for one session and cold for another. It may be cheap for short requests and expensive for long-context ones. It may be allowed for one tenant and forbidden for another.

A request is no longer only traffic. It is semantic work.

Routing is the moment where semantic intent becomes infrastructure placement.

That is the starting point of vLLM SR.

The project is not trying to build a prettier model selector. The deeper idea is to turn every LLM request into an explainable capability path: which evidence was observed, which derived facts were produced, which policy matched, which selection or collaboration algorithm ran, which plugins were attached, which backend and hardware path was used, and what trace was left behind.

I want to walk through that design as a system, not as a feature list. We will keep one running example in view, then open the layers one by one: signals, projections, decisions, selection algorithms, looper algorithms, router memory and learning, plugins, Envoy integration, model design, native bindings, hardware support, observability, and evaluation.

A Request Walks Into the Router

Imagine an enterprise AI gateway receives this request:

Analyze this customer contract, compare it with our internal policy, and draft a response. It may contain personal information. Use tools if needed.

If we look only at surface text, this is a contract-analysis request. If we look only at token count, it may be a long-context request. If we look at risk, it touches PII, internal policy, tool access, and factual correctness. If we look at infrastructure, it depends on tenant permissions, pool health, cache warmth, context-window capacity, and model cost.

A naive model picker asks one question:

Which model should answer this prompt?

vLLM SR asks a better question:

What must the system know before it is allowed to spend intelligence?

The split is intentional. Each layer owns a different kind of reasoning.

The core design choice is separation of concerns. If a request goes to a frontier model with RAG, tool filtering, and a response verifier, the answer should not be “because the router said so.” The router should be able to show the chain: evidence, projection, policy, algorithm, learning decision, plugins, mutations, backend path, and replay id.

The Full Request Lifecycle

For the contract request, the router first extracts request context: body, headers, tenant identity, session id, conversation id, tool state, prior response markers, and any existing routing metadata.

It then runs only the signals referenced by the active recipe. Domain, PII, context length, authorization, knowledge-base relevance, fact-check need, tool availability, and jailbreak or prompt-injection signals can run independently. The point is not to run every detector on every request; the point is to gather the evidence that the configured policy actually needs.

Projections then compose that evidence into stable route facts: privacy_sensitive, legal_contract_path, long_context, needs_internal_rag, and verification_required. The decision engine matches those facts against route policy. A matched decision exposes candidate models and, depending on its algorithm, either a single-model selection path or a looper collaboration path. Router Learning may then adjust the proposed model or hold the current model if switching would break session continuity. Finally, plugins attach route-scoped behavior before the gateway executes the main traffic path.

The model name is only the visible end of the route. The deeper value is that the system can explain why this was not a cheap direct-answer path. Privacy, internal knowledge, tool exposure, factuality, and context pressure are different concerns. vLLM SR gives each concern a place in the architecture.

The Recipe Is the System Contract

The next design step is easy to miss: vLLM SR is not primarily configured as a collection of ad hoc callbacks. It is configured as a recipe. That recipe is the contract between application intent, routing policy, backend inventory, plugin behavior, learning boundaries, and gateway mutation.

In production, routing policy tends to spread. One part lands in application code, one part in gateway config, one part in prompt templates, one part in a model registry, one part in a dashboard, and one part in a notebook used by the evaluation team. Once that happens, nobody can answer a simple operational question: “Why did this request take this path?”

vLLM SR’s v0.3-style contract tries to keep that answer in one durable shape.

That is the operational difference between a router that happens to make good choices and a router that can be trusted. A recipe is diffable. It can be validated. It can be replayed against stored traffic. It can be promoted across environments. It can be patched by offline learning without letting the hot path rewrite production policy.

The rule is conservative on purpose: runtime intelligence may propose better choices, but the recipe defines the legal search space. That is what keeps “adaptive routing” from becoming hidden policy mutation.

Research as Design Pressure

The paper shelf behind vLLM SR is not a citation dump. Each research direction creates pressure on one part of the router. Read the table as a map from problem to architecture.

The pressure from the research shelf is clear: a router cannot be one classifier. If it were, every new paper would become another special case. In vLLM SR, a paper can become a signal, projection, decision primitive, selection algorithm, looper algorithm, plugin, learning signal, model-card field, or pool feedback loop.

Signals: Evidence Before Judgment

Signals are deliberately humble. A signal answers one factual question. It does not choose the model. It does not select a plugin. It does not decide the route.

For the contract example, the router may need to know whether the request is legal or enterprise-related, whether it contains PII, whether the user is authorized for premium or private models, whether it needs internal knowledge, whether tools are available, whether the context is too large for a short-context backend, and whether the response needs verification.

The maintained runtime surface currently exposes eighteen base signal families: authz, complexity, context, conversation, domain, embedding, fact_check, jailbreak, keyword, language, modality, pii, preference, reask, structure, kb, user_feedback, and event. projection is also a rule condition type, but it is not a raw sensor. It is the decision-visible output of the projection layer.

The runtime behavior is as important as the list of names.

The payoff is evidence reuse. A pii signal can influence privacy policy, cache policy, provider selection, and audit policy. A conversation signal can influence router protection, tool filtering, and replay. A context signal can influence long-context pools, compression, and cost-aware selection. If signals directly selected models, that reuse would disappear.

Projections: The Coordination Layer

Signals are often messy. Some are booleans. Some are classifier probabilities. Some are similarity scores. Some are raw metrics such as token count or KB relevance. Production policy should not need to manually combine all of those raw values every time.

Projections turn evidence into stable routing facts.

One subtle rule keeps projections clean: decisions read projection outputs, not every intermediate score name. A partition or weighted score can be rich internally, but the policy surface should see facts such as legal_contract_path, risk_high, or long_context_lane.

That is why the projection layer can grow without making decisions unreadable. A risk score might combine PII, jailbreak, tenant tier, KB source, and response verifier need. The decision should not have to carry that formula inline. It should match a named derived fact and leave the math in a traceable coordination layer.

Projections keep the rest of the system readable. Signals remain fine-grained. Decisions remain policy-oriented. Algorithms remain responsible for selection or collaboration. The coordination work lives in between, where it can be traced.

Decisions: Policy Needs a Shape

Once projections produce stable facts, decisions express route policy.

The decision engine is intentionally closer to a boolean circuit than a hidden Python function. Routing policy should be inspectable, diffable, auditable, sortable, and eventually compilable across infrastructure layers.

For the contract request, the policy can be simplified as:

A route is not just an endpoint. It is a policy-approved capability bundle: candidate models, algorithm, plugins, gateway mutations, retention behavior, fallback, and diagnostics.

In the current contract, a decision is a route object, not only a rule tree.

Decision selection is intentionally deterministic. If matched decisions use tiers, lower tier values win first, then confidence, priority, and name. Without tiers, strategy=confidence ranks by confidence before priority; the default strategy ranks priority before confidence. Even fallback is explicit: an empty AND can be a catch-all route, but it has zero confidence so it does not outrank real evidence-backed decisions.

Retention emits make policy visible beyond the immediate model choice. A decision can express drop to skip semantic-cache writes, ttl_turns to bound cache lifetime, keep_current_model to protect session continuity, or prefer_prefix_retention to tell the serving pool that KV/prefix reuse matters.

That shift from classification-style routing to signal-decision routing is not cosmetic. Classification is useful for demos. Decision architecture is what production policy needs.

Selection Algorithms: Choosing One Model After Policy

Many router designs start with the selector: embeddings, MLPs, bandits, or an LLM-as-router that picks a model directly. That easily turns one algorithm into a sink for every concern: semantic fit, cost, latency, safety, authorization, session continuity, and provider policy.

vLLM SR keeps the order stricter. The decision matches first. Then a selection algorithm chooses one model from the policy-approved candidate set.

The boundary is strict: selection algorithms choose one model from modelRefs. They do not run a panel, coordinate a workflow, or hold a session model. Multi-model collaboration belongs to Looper. Session and conversation stability belongs to Router Learning.

Looper Algorithms: Selecting a Model Collaboration Path

Looper is important enough to discuss separately because it changes what the router is selecting.

A selection algorithm chooses one model. A Looper algorithm chooses a model collaboration path: a bounded execution pattern involving escalation, fan-out, panel judgment, multi-round reasoning, or micro-agent workflows. This is usually the path for scaling model capability without exposing a new application protocol. The client may still call one logical model name, but the router executes a structured collaboration behind that name.

These are not just “more algorithms.” They are the router’s answer to capability scaling.

Confidence keeps the cost curve low by starting small and escalating only when confidence is insufficient. It can use average log probability, margin, a hybrid score, self-verification, or an AutoMix-style entailment verifier. The question is not “small or large model?” It is “is the current answer good enough to stop?”

Ratings uses concurrency as a controlled resource. Instead of one winner, several candidates participate, bounded by max_concurrent, and the router aggregates successful responses. This is useful when operators want ensemble behavior or live comparison without letting fan-out become unbounded.

Fusion is the clean panel pattern. It sends the request to analysis models, asks a judge to identify consensus, contradictions, partial coverage, and blind spots, and then synthesizes a final answer. The important design point is that Fusion policy lives under the matched decision. vllm-sr/auto can decide whether Fusion is warranted; vllm-sr/fusion narrows matching to Fusion-capable decisions instead of silently falling back to ordinary single-model routing.

ReMoM is the multi-round version of the same philosophy. It uses a breadth schedule such as [3, 2] or [32, 4], distributes calls across model candidates, compacts intermediate responses when needed, and synthesizes the final answer. This is useful when the value comes from exploration over multiple reasoning paths rather than one panel pass.

Router Flow turns the route into a bounded micro-agent workflow. A static flow can define roles such as thinker, worker, verifier, and final synthesizer. A dynamic flow can ask a planner model to produce a plan, but worker execution remains constrained to the decision’s modelRefs. Tool calls preserve the OpenAI-compatible contract while the router stores enough workflow state to resume the correct worker after tool results return.

The Looper layer is the bridge from routing to model collaboration. It lets the router scale capability through multiple models while keeping policy, traces, cost boundaries, and public API shape explicit.

Router Memory and Learning: Adaptation Is Not an Algorithm

Session-aware and learning-related behavior should not be hidden inside decision.algorithm. In the clean vLLM SR design, this belongs to Router Learning.

The distinction matters. A decision says what is allowed. A selection or looper algorithm produces a base result. Router Learning then asks whether the system should adapt that result from runtime experience, and whether switching is safe in the current session or conversation.

Learning can improve a route inside policy. It must not become a second, invisible policy system.

The runtime order is fixed:

The public learning concepts are intentionally small:

Adaptation’s day-0 strategy is routing_sampling. It scores candidates from local experience: quality seed, good-fit outcomes, underpowered outcomes, overprovisioned outcomes, failures, latency evidence, cache reuse, effective input cost, and reliability. The default candidate set is decision, which means adaptation may only choose among the matched decision’s modelRefs. Broader scopes such as tier and global are more powerful, but they need stronger guards.

Protection is the session-aware half. It has a preflight guard and a switch guard. Preflight suppresses stochastic sampling during tool loops, protocol-sensitive continuations, or routine continuation steps. The switch guard decides whether to hold the current model, allow the proposal, or perform a bounded rescue switch. The simplified rule is:

switch if proposal_gain >= switch_margin + stability_weight * switch_cost

The switch cost can include cache warmth, handoff cost, tool-loop state, provider state, turn count, and switch history. Session-aware routing is therefore not sticky sessions. It is controlled continuity. The router keeps a model when switching is unsafe or not worth it, and it can switch again at idle boundaries, decision drift, or rescue conditions.

Sensitive routes can bypass learning entirely:

routing:
  decisions:
    - name: local_privacy_policy
      modelRefs:
        - model: local-private-model
      adaptations:
        mode: bypass

That boundary is the contract. Learning can improve choices inside recipe policy. It cannot silently rewrite the recipe, add a new privacy exception, change a decision priority, or mutate modelRefs on the request path. Offline recipe learning can propose those changes as reviewable artifacts, but live routing remains governed by the recipe.

Plugins: Behavior Belongs to the Route

After a route is selected, the request still may not be ready for the model. It may need cache, memory, RAG, tool filtering, request parameter caps, prompt mutation, response verification, fast policy response, image generation, or replay.

These behaviors should not be global decoration. Privacy routes may need to bypass cache. High-risk factual routes may require verification. Agentic routes may need tool boundaries. Low-risk summarization may need only replay. Plugins are therefore route-scoped.

Some plugins are worth reading as miniature subsystems.

A route is better understood as an execution contract. The model is one part of it. The route also carries constraints, tools, knowledge, verification, memory, and evidence.

The Hot Path: Header, Body, Route, Response

The conceptual architecture only becomes convincing when it touches the request path. In vLLM SR, the hot path is shaped around a simple constraint: the router must see enough context to make a semantic decision, but it should avoid turning every request into an expensive full parse and full detector run.

The route lifecycle inside the gateway path looks like this.

Protocol compatibility is part of the same story. The router should not hide provider differences behind a vague facade. It should translate them explicitly at the boundary: OpenAI-compatible chat, Responses-style calls, Anthropic-style provider routing, direct looper model slugs, and backend-specific provider profiles all become inputs to one routing engine. Applications keep a familiar API shape, while the infrastructure keeps the differences visible enough to debug.

The ExtProc path matters because it gives semantic policy a boundary-native shape. vLLM SR can receive enough request and response context to make semantic decisions while still returning control to the real gateway data plane.

Envoy Integration: Put Semantic Policy at the Boundary

The Envoy integration shows the intended boundary clearly.

Envoy owns the data plane: TLS, clusters, endpoint health, timeouts, retries, load balancing, and filter chains. vLLM SR should not rebuild those capabilities. It should act as the semantic policy plane: receive request context through External Processing, evaluate the route, and return header/body mutations.

The adoption advantage is large. Applications do not need a new private API just to benefit from semantic routing. They can keep calling familiar model APIs while infrastructure maps logical model names to small models, frontier models, RAG paths, verifier paths, Looper collaboration paths, workflows, or private hardware lanes.

Protocol work such as SIRP and multi-provider inference API matters for the same reason. Semantic routing should strengthen the control plane without forcing every application team into a custom gateway dialect.

Model Design: Model Name Is the Wrong Primitive

The router cannot make production-grade decisions from model names alone.

gpt-4, qwen3-32b, claude-opus, or local-private identifies an endpoint or alias. It does not describe reasoning ability, coding strength, tool behavior, vision support, context window, latency distribution, cost, hardware path, privacy boundary, or observed failure modes.

Users can ask for auto. The system cannot treat auto as a magic endpoint. Internally, it must expand into model cards, candidate sets, route policy, Looper eligibility, learning boundaries, and execution paths.

Bindings: Fast ML Without Turning the Router Into a Model Server

vLLM SR’s control plane is written around Go because gateway integration, configuration, request mutation, response processing, Envoy ExtProc, and Kubernetes-style infrastructure fit Go well. But routing also has ML hot paths: embeddings, classification, modality detection, LoRA classification, and MLP selectors.

The binding layer keeps those concerns separated.

The rule is not “everything must be native.” The rule is capability must be explicit. If a deployment lacks a native classifier, the router should know. If ONNX supports only part of the contract, policy should not pretend otherwise. If backend lifecycle needs reset boundaries, the router should expose that instead of hiding it. Intel/OpenVINO-oriented paths can be valuable deployment options, but they belong in the hardware/runtime discussion unless they are part of the same advertised native capability contract.

Hardware Is a Routing Variable

Hardware support is often described as a compatibility matrix. For semantic routing, it is more than that.

Different hardware paths imply different latency, cost, memory behavior, kernel availability, quantization support, context-window economics, privacy boundary, and energy curve. A router that ignores hardware is only doing half the job.

The acceleration story is not one-size-fits-all. CUDA gives the broadest default serving path. ROCm/AITER makes AMD pools a serious production option rather than a compatibility afterthought. Intel XPU and OpenVINO-style paths matter for enterprises that already own Intel-heavy infrastructure or need CPU/XPU portability. CPU and local paths remain important because privacy, availability, and cost sometimes beat raw throughput.

WRP is the right mental model here. Workload is semantic. Pool is physical. Router translates between them.

A future router should know when CUDA is overloaded, when ROCm is cost-effective, when XPU is sufficient, when CPU/local is preferable for privacy, and when preserving KV cache is worth more than reselecting a theoretically better model.

Observability: The Router Must Leave Evidence Behind

If the system makes semantic decisions, operators need to see those decisions.

Replay is not only for dashboards. It is the second control plane. The first control plane makes the live decision; replay preserves enough evidence to debug that decision, compare it against alternatives, attach outcomes, and produce offline recipe patches.

If the router records what it saw, what it decided, what happened, and how users, agents, verifiers, or evals responded, the next generation of selectors, thresholds, learning strategies, and recipes can improve.

Without traces, the router is another opaque model. With traces, it becomes a system component.

Evaluation: The Router Is a Frontier

The wrong way to evaluate a router is to count features. A router with many knobs can still be bad. The right question is whether it improves the frontier between quality, cost, latency, safety, privacy, reliability, and hardware efficiency.

Router evaluation, fleet simulation, replay traces, RouterArena-style comparison, Looper evals, and offline recipe learning matter because semantic routing is infrastructure. It has to be measured like infrastructure.

The Part I Care About Most

The most important design belief in vLLM SR is not any single signal, plugin, model, algorithm, or hardware platform. It is the separation of concerns.

Signals observe. Projections coordinate. Decisions express policy. Selection algorithms choose one model. Looper algorithms scale capability through model collaboration. Router Learning adapts and protects within recipe boundaries. Plugins mutate behavior. Bindings accelerate hot-path ML. Envoy integration places semantic policy at the traffic boundary. Model cards connect logical names to real capabilities. Hardware metadata connects semantic constraints to physical execution. Observability makes every decision accountable.

That separation is what lets the router evolve without turning every new idea into a fork of the hot path.

A new jailbreak detector can become a signal. A new risk formula can become a projection. A new compliance rule can become a decision. A new selector can become a selection algorithm. A new panel or workflow primitive can become a Looper algorithm. A new verifier can become a plugin. A new accelerator lane can become model-card metadata. A new benchmark can become an evaluation loop. A new fleet model can feed pool-aware routing. A new learning strategy can propose better candidates without rewriting policy.

That is why I like the phrase Intelligence Control Plane. Not because the router is always intelligent by itself, but because it gives the system a place to allocate intelligence deliberately.

The first stage of AI infrastructure made intelligence callable.

The next stage has to make intelligence allocatable, explainable, and optimizable.

Calling a model was the first abstraction. Allocating intelligence is the next one.

That is the philosophy of vLLM SR.

Source Trail

This article is based on the vLLM Semantic Router codebase, website research archive, vLLM project blog posts, my earlier essays on LLM routing, and infrastructure references around Envoy, vLLM, ROCm, Intel XPU, and Gateway API inference routing.