Manuja frames an LLM gatewayA large language model gateway is an intermediary that standardizes, routes and monitors application calls to one or more language-model providers. as the shared layer between applications and model providers, responsible for routing, authentication, fallback, rate limits, observability and governance. The design problem is not maximizing one metric, but choosing explicit tradeoffs among availability, latency, safety and cost for each route.
Naive retries and provider fallbacks can multiply both tail latency and spend. He recommends bounded sequential fallback for ordinary traffic, parallel calls only when the latency benefit justifies double cost, and circuit breakersA circuit breaker is a reliability control that temporarily blocks calls to a failing dependency after errors cross a defined threshold. with cooldowns so a failing provider is removed from the request path instead of repeatedly rediscovered.
Fallback providers also need enough reserved capacity, normalized tool schemas and explicit handling for token limits and stop reasons. Streaming responses sacrifice midstream recovery, while reasoning and router models create variable latency that is better controlled with fixed reasoning levels, route-specific deadlines and selective tail hedging near the P90 boundary.
GuardrailsA guardrail is a technical or procedural control that limits unsafe, disallowed, deceptive, or otherwise unwanted AI behavior. need their own time budgets and fail-open or fail-closed decisions. Manuja combines those controls with per-route API keys, bounded queues, load shedding and traffic priorities, then argues that gateway traffic should be decentralized to avoid a new single point of failure while policy, audit and governance remain centralized.
Watch on YouTube
