A cold start occurs when a service needs to prepare an execution environment rather than use one that is already ready. The work can include starting a process, loading application code, preparing dependencies and loading a model into memory.
This initialization contributes to latency but is distinct from the model's generation time. The delay varies by environment and workload. A later request may be faster when it can reuse an already prepared environment, although that reuse is not always available.
ELI5
A cold start is like opening a workshop for the first job of the day. Equipment must be prepared before the actual work begins.
For example, a hosted AI service may need to load a model before answering its first request. A second request can be quicker if the same ready environment is still available.
Is cold-start time the same as inference time?
No. It is preparation before or around execution. Inference is the computation that uses the model to produce an output.
Does every request experience a cold start?
Not necessarily. A request can reuse a ready environment, depending on the service's lifecycle and scheduling.



