What is a cold start?

Definition

A cold start occurs when a service needs to prepare an execution environment rather than use one that is already ready. The work can include starting a process, loading application code, preparing dependencies and loading a model into memory.

This initialization contributes to latency but is distinct from the model's generation time. The delay varies by environment and workload. A later request may be faster when it can reuse an already prepared environment, although that reuse is not always available.

ELI5

A cold start is like opening a workshop for the first job of the day. Equipment must be prepared before the actual work begins.

For example, a hosted AI service may need to load a model before answering its first request. A second request can be quicker if the same ready environment is still available.

Is cold-start time the same as inference time?

No. It is preparation before or around execution. Inference is the computation that uses the model to produce an output.

Does every request experience a cold start?

Not necessarily. A request can reuse a ready environment, depending on the service's lifecycle and scheduling.

  1. Portrait of Dean Qui帽anola on a black background with the blue and white headline "LIVE GPU ITERATION".
  2. Thorsten Hans beside the blue and white headline EDGE-NATIVE AGENTS on a black background.
    Thorsten Hans Builds Edge-Native MCP Servers
    AI Engineer22m 37s1 VIEW
  3. Sonam Pankaj beside the headline Agents That Learn.
Definition card for cold start: What is a cold start?

A cold start is the initialization work required before an inactive or newly created execution environment can handle a request.