A large language model gateway, commonly called an LLM gateway, sits between an application and model providers. It can provide a common API, authentication, routing, logging, rate limits, retries and provider failover for model calls.
A gateway can measure individual calls but may not understand the full behavior of an agent run. Loops, growing context, tool output and subagents can increase total spending inside the application, so run-level controls may be needed above the gateway.
ELI5
A large language model gateway is a managed doorway between an application and several AI model services. The application sends requests to the gateway, which chooses or reaches the right provider and applies shared rules.
For example, a coding tool could use one compatible gateway to connect a user's key to different model providers. The gateway can simplify switching, but it must keep keys private and clearly report where each request went.
What does an LLM gateway control?
It can control authentication, provider routing, rate limits, retries, logging and policy checks around model API calls.
Why might an LLM gateway miss runaway agent spending?
It sees model calls but may lack the run-level context needed to understand loops, subagents and expanding prompts.
