Rachel Lee Nabors explains how local models can reduce dependency on hosted inference while keeping user data on device. They recommend prototyping a feature with a capable model, defining acceptable outputs and testing smaller candidates against those criteria.
Using summaries for the Mima social application, Nabors compares several model candidates with a hosted Claude baseline. In their tests Llama 3.2 offered a useful balance of accuracy and latency for the particular workload, rather than assuming the most popular model would be best.
They compare prompt variations, describe improvements from few-shot examples and simple post-processing, and inspect cases where an automated judge may be too strict. They recommend ongoing regression evaluations when prompts or models change.
Watch on YouTube
