vLLM Semantic Router
System-level intelligent router for mixture-of-models.
INTELLIGENT ROUTING FOR LLM SYSTEMS
It`s just a good thinking game 🎲
XUNZHUO
I study intelligent routing and system intelligence for LLMs.
My current work is about moving routing beyond model selection and toward system behavior: semantic signals, coordinated model paths, and decision loops that optimize quality, cost, and reliability together.
Latest log: The Second Half of LLM Routing
Offerings
V.2
-
01:00
SEMANTIC ROUTING
Signal-driven orchestration for model, tool, and policy selection.
- Reasoning-aware routing strategies
- Latency, cost, and quality controls
- Decision loops that optimize system behavior
-
02:35
OPEN SOURCE SYSTEMS
Pragmatic infrastructure work across gateways, inference stacks, and control planes.
- Gateway and Envoy ecosystem leadership
- Inference platform design and operations
- Portable patterns for production AI systems
-
03:20
RESEARCH AND STANDARDS
Writing, evaluation, and community work that reframes routing as a systems problem.
- Research on routing, search, and reasoning
- Community roles across CNCF and Kubernetes
- Standards work for AI gateway architecture
Selected
Works
Envoy Gateway
Manages Envoy Proxy as a standalone or Kubernetes-based application gateway.
Envoy AI Gateway
Manages unified access to generative AI services built on Envoy Gateway.
vLLM AIBrix
Cost-efficient and pluggable infrastructure components for GenAI inference.
Istio
Connects, secures, controls, and observes services.
Kiali
Observability console for Istio with service mesh.
Research
Archive
Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents
Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
Adaptive Vision-Language Model Routing for Computer Use Agents
98x Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
When to Reason: Semantic Router for vLLM
Category-Aware Semantic Caching for Heterogeneous LLM Workloads
Community
Roles
Kubernetes AI Gateway WorkGroup
Leading the community effort to define standards for AI Gateway in the Kubernetes ecosystem.
CNCF Ambassador
Representing and promoting Cloud Native Computing Foundation projects and values globally.
Linux Foundation APAC Open Source Evangelist
Advocating for open source adoption and best practices across the Asia-Pacific region.
KubeCon Program Committee
Reviewing and selecting talks for one of the largest cloud-native conferences.