The AI Automators compares a small locally fine-tunedFine-tuning adapts a pretrained model by updating its parameters on examples chosen for a particular task or subject. Laya decision model with cloud-hosted Jev on choosing the next action from 30 customer-support toolsA decision model selects or scores possible answers for a defined decision.. The experiment separates training, validation, calibration and test dataEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. rather than judging the model on conversations used to train it.
With basic state, the presenter reports 82.2 percent accuracy for fine-tuned Laya and 67.3 percent for Jev on 1,000 test chats. Richer contextContext engineering designs how relevant instructions, evidence, state and tool information are assembled for a model's task. changes the result: Jev reaches 87.4 percent and Laya 86.1 percent. These are results from one dataset and configuration, not evidence that either model wins across all support systems.
The second part examines whether model confidence is trustworthy enough to permit automatic action. Temperature calibration reduces confident errorsConfidence calibration measures whether an AI system's stated certainty matches how often its predictions or assumptions are actually correct. at a 90 percent gate but increases human escalations, demonstrating a practical tradeoff between mistakes and staff workload. Course promotions and community appeals are omitted.
Watch on YouTube




