Papers on intelligent routing, decision models, agents, and inference systems.
Peer-reviewed at NeurIPS and SIGIR.
4 papersSignal-driven routing, policy languages, caching, and the decision layer for Mixture-of-Models.
9 papersIntroduces vLLM Semantic Router as a signal-driven routing framework for mixture-of-modality deployments, composing heterogeneous signals into deployment-specific policies across cost, privacy, latency, and safety.
Synthesizes recent routing, fleet, multimodal, and governance work into the Workload-Router-Pool architecture, framing routing as one layer in a broader inference optimization stack.
Routes prompts by reasoning requirements so reasoning is only invoked when it pays off, improving accuracy while cutting token usage and latency versus always-on reasoning.
Shows how probabilistic ML predicates in policy languages can silently co-fire on the same query and adds conflict detection plus a softmax prevention mechanism in the Semantic Router DSL.
Extends the Semantic Router DSL from stateless per-request routing to multi-step agent workflows, compiling verified decision artifacts across orchestration, Kubernetes, and protocol layers.
Presents OATS, an offline embedding-refinement method that improves semantic-router tool ranking under single-digit millisecond CPU budgets without adding serving-time model inference.
Combines flash attention, prompt compression, and near-streaming body processing to cut routing latency from seconds to tens of milliseconds on lightweight shared serving hardware.
Shows that conversational memory and retrieval-grounded routing let a lightweight 8B model recover most of a much larger model's performance on persistent user-specific queries while dramatically reducing cost.
Proposes category-aware semantic caching with category-specific similarity thresholds, TTLs, and quotas, using a hybrid split between in-memory HNSW retrieval and external document storage.
Pool routing, capacity planning, and energy as first-order inference objectives.
4 papersShows that tokens per watt scale inversely with the serving context window: on identical H100s, tok/W varies by nearly 40× from 2K to 128K context, and context-length routing combined with newer GPUs improves fleet energy efficiency by up to 5.76×.
Proposes token-budget-aware pool routing that estimates each request's token budget online and dispatches it to short- or long-context serving pools, reducing GPU cost while improving stability for LLM inference.
Derives the minimum-cost two-pool LLM fleet directly from workload distributions and P99 TTFT targets, then maps the optimal boundary to a deployable compress-and-route strategy.
Introduces a queueing-theory-grounded fleet planner and discrete-event simulator for sizing multi-pool LLM GPU fleets against P99 TTFT targets without requiring up-front hardware profiling.
Long-horizon and personal agents, computer-use reliability and security, and verified retrieval.
5 papersSplits long-horizon agent decisions into strategic, tactical, and operational scales, with a learned scheduler that decides which scale to invoke and how many reasoning tokens to spend, cutting inference tokens by 38–57% against flat ReAct baselines.
Presents a real-time verification layer for long-document RAG systems, handling contexts up to 32K tokens while balancing latency with grounding coverage for interactive production deployments.
Introduces a personal-model-first agent architecture where personal AI grows a correctable understanding of Identity, World, Pulse, and Journey through user-paced curiosity and reflection after each turn.
Formalizes the visual confused deputy as a security failure mode for computer-use agents and proposes a dual-channel guardrail for validating click targets and action reasoning before execution.
Proposes Adaptive VLM Routing to estimate step difficulty in computer-use agents and route each action to the cheapest model that can still satisfy a target reliability threshold.
Papers, open models, and benchmarks, co-authored in the open.
MBZUAI
McGill University
UC Berkeley
University of Chicago
New York University
UMass Boston
UIC
National Taiwan University
IBM Research
KR Labs