Anthropic researcher Lance Martin argues that old prompt instructions can accumulate contradictions or unnecessary work as models change. He also explains why stable prompt prefixes matter for cachingPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. in multi-turn applications.
Lance Martin describes designing evaluations with clear criteria, useful headroom and low varianceEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. He discusses iterative optimization, warns that improvements can overfit an evaluation set, and suggests comparing training and held-out results.
The talk and audience questions cover model effort, human involvement in evaluation design, and separating recoverable session stateDurable workflow execution preserves progress so a multi-step process can resume after interruptions using recorded completion state. from agent execution. Martin presents cost savings from internal examples as illustrative results rather than universal guarantees.
Watch on YouTube
