Schmid distinguishes the agents developers use from the agents they build for customers. A developer can notice when a coding agent misses a skill and manually redirect it, but end users do not know which internal capability should trigger. Production systems therefore need evidence that the right skill is selected from ordinary user language and that it actually improves the final result.
SkillsAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work. can encode temporary capabilities or durable preferences. Capability skills teach a model something it cannot yet perform consistently and may become unnecessary as foundation models improve. Preference skills capture organization-specific workflows, style, and domain knowledge. Both need regression testsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked., while evals reveal when a capability skill can safely be retired without losing performance.
A useful evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. begins with representative promptsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it., expected outcomes, isolated workspaces, and several trials because agent behavior is nondeterministic. Deterministic scripts and regular expressions are often enough for files, commands, or structured outputs. More complex traces can use an LLM judge with a clear rubric, but the measurement should focus on task success rather than forcing one exact reasoning path.
Google DeepMind keeps tests beside internal skills and runs them whenever a skill changes. Schmid recommends including negative cases, testing multiple agent harnesses, keeping skills concise, and comparing runs with and without the skill. AblationAn ablation study compares a system with and without a selected component to examine that component's contribution. shows whether a skill adds value, while retained evals continue to detect regressions after that skill is removed.
Watch on YouTube
