The Semantic Routing Moment

tldr: Semantic routing is becoming the next important layer of AI infrastructure. vLLM Semantic Router was built for this moment.

Late last year, in The Second Half of LLM Routing, I argued that the next phase of LLM routing would not be another gateway. It would be a semantic decision layer that understands the request and decides how an AI system should behave.

That direction arrived faster than I expected. Across coding products, clouds, gateways, local AI, and multi-model systems, very different teams are converging on the same need: an intelligent decision layer above a changing model pool.

The Category Arrived

They are at different stages—shipped products, previews, and experimental infrastructure—but they are solving the same problem.

Cursor and Kilo put routing in the product; Databricks put it in the gateway.

DigitalOcean offers it as a managed service; Switchyard exposes it as open infrastructure.

LiteLLM and Lemonade make the layer deployable across hosted, local, and hybrid environments.

Fugu and Fusion show where this leads. The router may choose not only which model should answer, but how several models should work together. The products will change. What matters is that they appeared at once: semantic routing is now a product expectation and an enterprise infrastructure decision.

Why We Built Early

FrugalGPT, Hybrid LLM, and RouteLLM had already shown that model choice could be learned and optimized. In industry, however, the 2025 race was about larger models, faster accelerators, and more efficient serving.

What I saw behind that race was fragmentation:

• Models: large and small, open and closed, general and specialized, local and remote.
• Devices: CPUs, GPUs, NPUs, and accelerator generations with different economics.
• Locations: edge, cloud, data center, and enterprise on-prem.
• Preferences: cost first, accuracy first, speed first, privacy first, or a deliberate balance.

Progress was improving every component while making the whole system harder to compose.

How do we make fragmented models, devices, locations, and preferences behave like one coherent system?

That question became vLLM-SR. Serving makes intelligence cheaper; routing decides which intelligence a task deserves, where it runs, and under what budget and policy—the demand-side thesis I later described in Semantic Routing as Energy Infrastructure.

We began vLLM-SR in early 2025. Its public launch through vLLM that September was only Day 0. Each release since has moved the boundary of what the router had to own.

The architecture evolved with that boundary:

Signals → Projections → Decisions → Plugins → Algorithms → Model Pools

We never treated a release as the destination. Each version took on a harder, less explored problem, and each solution became the foundation for what we built next.

The Themis cycle brought 350+ commits and 80+ contributor identities; by July 2026, the project had reached 150+ contributors.

Three Bets

1. Users Should Choose the Preference, Not the Model

Semantic routing is important, but I do not think Auto is the right product abstraction. A black-box Auto button removes friction, but it also removes agency: the user is asked to trust a choice without knowing whose objective the system optimized.

The alternative is not to hand model selection back to the user. The model catalog is already too large, and it will keep growing across open and proprietary models. Even experts cannot choose the best model for every request.

Most people do not know which model they want—and they should not have to. They know what the task in front of them needs: speed, accuracy, cost, or a balance among them. That preference is a much simpler choice, and an important one to keep in the user’s hands.

The user chooses the preference. The system chooses how to realize it.

Building on our Mixture-of-Models direction, we are working toward a small, preference-aligned model family:

These models do not expand the catalog. They simplify it: each one is aligned with a clear outcome and uses the same interface as a single model. An accuracy-first request would look like this:

curl -sS http://127.0.0.1:8899/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-sr/mom-v1-ultra",
    "messages": [
      {
        "role": "user",
        "content": "Design a rollback-safe database migration plan."
      }
    ]
  }'

Behind each model, the system would choose from the underlying pool—selecting, cascading, verifying, or coordinating models—while keeping privacy, residency, authorization, and safety as hard constraints. The pool can keep evolving without pushing that complexity back to the user.

2. There Is No Best Routing Algorithm

Different constraints need different methods. Data residency needs hard rules; task intent may need a classifier; uncertain quality may favor a learned selector or cascade.

There is no single winner because there is no single routing objective.

The router therefore has to be programmable, not built around one champion algorithm. Products and enterprises should be able to adapt routing to different user preferences and constraints by composing signals, policies, model pools, and algorithms. Our white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalizes these as separate parts. In vLLM-SR, the algorithm belongs to a decision, allowing each preference to use the method that fits it without forcing tenants to rewrite policy.

3. Routing Alone Is Not Enough

A semantically perfect choice can still be wrong for the live system: the selected model may be overloaded, its cache may be cold, or another pool may already hold the session state.

Our vision paper, The Workload-Router-Pool Architecture for LLM Inference Optimization, treats three variables as one system. Workload describes the semantic task and its service objective. Router chooses the policy and capability path. Pool exposes the models, accelerators, caches, placements, and serving state that can realize it.

Workload is semantic. Pool is physical. The router translates between them.

Pool feedback changes which route is actually efficient. A modest selector with the right placement and cache state may outperform a sophisticated one that sees only the prompt.

Where It Runs

Thank You

I am especially grateful to the partners who worked alongside me during the early stage of vLLM Semantic Router: Huamin Chen, Chen Wang, and Yue Zhu.

As the project grew, Bowei He, Yankai Chen, Fuyuan Lyu, Steve Liu, Andy Luo, and Haichen Zhang helped expand its technical and research direction.

Many others kept the project moving: FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, Qiping Pan, and many more.

I am grateful to friends and collaborators across MBZUAI, McGill University, Mila, Rice University, vLLM, vLLM Production Stack, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, Tongji University, Intuit, Delta Electronics, Z.ai, Yokogawa, Kong, agentgateway, and the wider open-source community. AMD GPU resources and ROCm software supported training, research, end-to-end testing, and the playground; collaborations with Hugging Face Candle and Hugging Face Sentence Transformers helped move the router-model runtime forward.

Last year, semantic routing was still an argument. Today, it is an ecosystem.

We built vLLM-SR to make fragmented models, hardware, locations, and preferences usable without taking control away from users. We were early. Now the important part is what we build together.