vLLM Semantic Router
Signal-driven decision routing for mixture-of-modality deployments.
INTELLIGENT ROUTING FOR LLM SYSTEMS
It`s just a good thinking game 🎲
XUNZHUO
I study intelligent routing and system intelligence for LLMs.
My current work connects semantic routing with the Workload-Router-Pool architecture: using workload signals, policy loops, and coordinated serving pools to optimize quality, cost, latency, privacy, and safety together.
Latest log: The Second Half of LLM Routing
Offerings
V.2
-
01:00
SEMANTIC ROUTING
Signal-driven orchestration for model, tool, and policy selection.
- Reasoning-aware routing strategies
- Latency, cost, and quality controls
- Decision loops that optimize system behavior
-
02:35
OPEN SOURCE SYSTEMS
Pragmatic infrastructure work across gateways, inference stacks, and control planes.
- Gateway and Envoy ecosystem leadership
- Inference platform design and operations
- Portable patterns for production AI systems
-
03:20
RESEARCH AND STANDARDS
Writing, evaluation, and community work that reframes routing as a systems problem.
- Research on routing, search, and reasoning
- Community roles across CNCF and Kubernetes
- Standards work for AI gateway architecture
Selected
Works
Envoy Gateway
Manages Envoy Proxy as a standalone or Kubernetes-based application gateway.
Envoy AI Gateway
Manages unified access to generative AI services built on Envoy Gateway.
vLLM AIBrix
Cost-efficient and pluggable infrastructure components for GenAI inference.
Higress
AI gateway and AI-native API gateway.
Istio
Connects, secures, controls, and observes services.
Kiali
Observability console for Istio with service mesh.
Research
Highlights
Recent papers on routing, systems, and inference optimization.
A selection from recent work on semantic routing, agent behavior, and AI infrastructure. More papers are collected on Works.
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
When to Reason: Semantic Router for vLLM
Conflict-Free Policy Languages for Probabilistic ML Predicates: A Framework and Case Study with the Semantic Router DSL
FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
Adaptive Vision-Language Model Routing for Computer Use Agents
Community
Roles
Kubernetes AI Gateway WorkGroup
Leading the community effort to define standards for AI Gateway in the Kubernetes ecosystem.
CNCF Ambassador
Representing and promoting Cloud Native Computing Foundation projects and values globally.
Linux Foundation APAC Open Source Evangelist
Advocating for open source adoption and best practices across the Asia-Pacific region.
KubeCon Program Committee
Reviewing and selecting talks for one of the largest cloud-native conferences.