Databricks technical lead Sandipan Bhaumik outlines a framework for deploying AI systems, covering evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria., observabilityAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it., data foundations, multi-agent orchestrationA multi-agent system contains multiple AI agents that interact, coordinate, divide work, or influence one another while pursuing tasks. and governanceGovernance sets responsibilities, rules and oversight for decisions about how AI systems are used.. He argues that teams should define business success and measurement before choosing models.
A retail banking chatbot case study illustrates how evaluation sets and traces can help teams diagnose stale information and operational failures. Bhaumik also describes incident response and prompt change management, alongside examples from Databricks tools.
Watch on YouTube
