Daniel Svonava defines a small model as one that fits on a single, widely available GPU. Rather than treat one model as a general answer to every task, he suggests evaluating specialized models for each part of a workflow. That turns one model endpoint into a fleet, with new demands on deployment and GPU utilization.
A top-down router can make poor placement decisions when many short requests arrive and worker queues change quickly. The proposed design puts requests into a shared queue so workers can pull work and form batches themselves. A lightweight gateway attaches metadata, while larger payloads can travel through a side channel instead of clogging the queue.
The design adds machine-local coordination when several GPUs share a host, and separates the worker from multiple model runtimes through a local socket. The speaker reports substantially higher cluster throughput from centralized queuing, while stressing that useful benchmarks locate the point where extra requested throughput begins to raise latency. His examples cover embeddings and task-specific generation; the closing section touches on packing several small models onto one GPU and pre-tuning model configurations.
Watch on YouTube
