Routing
A serving system that decides which model, hardware tier, or execution path should handle a request so operators can balance cost, latency, safety, and specialization.
Opening summary
Routing means choosing where a request should go before the expensive work begins. A serving stack can send one request to a cheaper path, another to a faster path, and a third to a specialist path, all while presenting one product surface to the user.
At a glance
Released
June 2026
System type
System Routing
Related models
Related modules
What It Is
Routing is a serving-time control decision. The system inspects a request, applies simple policy or classification rules, and chooses which model, hardware tier, or execution path should answer it.Where It Sits
It sits near the entry of the inference stack, between the incoming request and the concrete serving path that will execute the model. Some routing happens before model execution starts, while other routing can hand work from one serving path to another after a first pass reveals more about the request.How It Works
A runtime usually begins with lightweight request features such as prompt length, account tier, safety risk, tool requirements, or whether the task looks general or specialized. A router then maps those signals onto an available path, such as a small fast model, a larger premium model, a specialist model, or stricter filtering and review stages.Routing System Flow
Request arrivesRead routing signalsChoose serving pathRun chosen backendRequest arrives to Read routing signalsRead routing signals to Choose serving pathChoose serving path to Run chosen backend
Request and weight flow