Schmid distinguishes the agents developers use from the agents they build for customers. A developer can notice when a coding agent misses a skill and manually redirect it, but end users do not know which internal capability should trigger. Production systems therefore need evidence that the right skill is selected from ordinary user language and that it actually improves the final result.
Skills can encode temporary capabilities or durable preferences. Capability skills teach a model something it cannot yet perform consistently and may become unnecessary as foundation models improve. Preference skills capture organization-specific workflows, style, and domain knowledge. Both need regression tests, while evals reveal when a capability skill can safely be retired without losing performance.
A useful evaluation begins with representative prompts, expected outcomes, isolated workspaces, and several trials because agent behavior is nondeterministic. Deterministic scripts and regular expressions are often enough for files, commands, or structured outputs. More complex traces can use an LLM judge with a clear rubric, but the measurement should focus on task success rather than forcing one exact reasoning path.
Google DeepMind keeps tests beside internal skills and runs them whenever a skill changes. Schmid recommends including negative cases, testing multiple agent harnesses, keeping skills concise, and comparing runs with and without the skill. Ablation shows whether a skill adds value, while retained evals continue to detect regressions after that skill is removed.
Watch on YouTube
