What is a routing policy?

Definition

An AI routing policy is the set of conditions and priorities used by a model router. It can map request types to model pools, apply capability requirements, prefer lower cost or latency and define fallback behavior when a model is unavailable.

Policies should be configurable, versioned and evaluated against the organization's own workload. Feedback from quality, cost, latency, caching and user preferences can support careful updates over time.

ELI5

A routing policy is the set of rules an AI system uses to choose which model should handle a request. The rules can consider task type, required capabilities, quality, cost, speed, privacy, and what to do when the preferred model is unavailable.

For example, simple classification requests may go to a fast inexpensive model while complex coding tasks go to a stronger one. The team can test and version the policy so a change in routing does not silently reduce quality.

What does an AI routing policy contain?

It can contain task classifications, eligible model pools, rankings, cost and latency limits, preferences and failover rules.

How often should a model routing policy change?

It should change when validated workload evidence shows better choices or when model capability, pricing or availability changes.

  1. Archana Kamath and Tyler Gillam beside the words Route Each Request
Definition card for routing policy: What is a routing policy?

An AI routing policy defines how a system chooses, ranks and fails over between models for incoming requests.