Laya and Jev: Fine-Tuning, Context and Calibration

The AI Automators14m 48s
Daniel Walsh against a black background beside the blue and white headline “SMALL MODEL REAL DECISIONS”.
1 VIEW
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    The AI Automators compares a small locally fine-tunedFine-tuning adapts a pretrained model by updating its parameters on examples chosen for a particular task or subject. Laya decision model with cloud-hosted Jev on choosing the next action from 30 customer-support toolsA decision model selects or scores possible answers for a defined decision.. The experiment separates training, validation, calibration and test dataEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. rather than judging the model on conversations used to train it.

    With basic state, the presenter reports 82.2 percent accuracy for fine-tuned Laya and 67.3 percent for Jev on 1,000 test chats. Richer contextContext engineering designs how relevant instructions, evidence, state and tool information are assembled for a model's task. changes the result: Jev reaches 87.4 percent and Laya 86.1 percent. These are results from one dataset and configuration, not evidence that either model wins across all support systems.

    The second part examines whether model confidence is trustworthy enough to permit automatic action. Temperature calibration reduces confident errorsConfidence calibration measures whether an AI system's stated certainty matches how often its predictions or assumptions are actually correct. at a 90 percent gate but increases human escalations, demonstrating a practical tradeoff between mistakes and staff workload. Course promotions and community appeals are omitted.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows70
    • Coding & tools20
    • Models & learning100
    • Infrastructure & compute30
    • Research & evaluation90
    • Safety & security20
    • Business & productivity65
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Daniel Walsh against a black background beside the blue and white headline “SMALL MODEL REAL DECISIONS”. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 5 October 2026 and duration 14m 48s.

    Fine-tuning helps a small local decision model compete with Jev, but richer context and confidence calibration change both accuracy and the workload passed to humans.