Ameya Bhatawdekar explains why replacing an AI model often requires more than a model swap: the surrounding application may encode workarounds for limitations that newer models no longer have. Reworking that architecture exposes different failure modes, so an evaluation suite built for the previous design can provide incomplete coverage.
Ameya Bhatawdekar uses a notional site-reliability agent with read and write tools to trace this progression. A single model call mainly needs answer-quality checks; retrieval chains also need checks on parsing and retrieved context. Workflow graphs introduce routing, node contracts and retry behavior, while flexible reasoning-and-action loops require attention to the full execution path.
Ameya Bhatawdekar emphasizes repeated evaluation runs when the same input can produce different valid trajectories. A system succeeding occasionally demonstrates something different from a system succeeding consistently. Memory, code-execution sandboxes, tool integrations and reusable skills then add further components that older answer-focused tests may overlook.
Ameya Bhatawdekar recommends continually drawing evaluation cases from production behavior, covering both familiar failures and new kinds of mistakes that existing checks do not anticipate. The goal is to preserve working behavior while gaining new capabilities, with evaluation discipline supporting both small iterations and major architectural changes.
Watch on YouTube
