Manuja frames an LLM gateway as the shared layer between applications and model providers, responsible for routing, authentication, fallback, rate limits, observability and governance. The design problem is not maximizing one metric, but choosing explicit tradeoffs among availability, latency, safety and cost for each route.
Naive retries and provider fallbacks can multiply both tail latency and spend. He recommends bounded sequential fallback for ordinary traffic, parallel calls only when the latency benefit justifies double cost, and circuit breakers with cooldowns so a failing provider is removed from the request path instead of repeatedly rediscovered.
Fallback providers also need enough reserved capacity, normalized tool schemas and explicit handling for token limits and stop reasons. Streaming responses sacrifice midstream recovery, while reasoning and router models create variable latency that is better controlled with fixed reasoning levels, route-specific deadlines and selective tail hedging near the P90 boundary.
Guardrails need their own time budgets and fail-open or fail-closed decisions. Manuja combines those controls with per-route API keys, bounded queues, load shedding and traffic priorities, then argues that gateway traffic should be decentralized to avoid a new single point of failure while policy, audit and governance remain centralized.
Watch on YouTube
