AI Evals: Judge Calibration, Policy and CI Gates

AI Engineer59m 49s
Tejas Kumar against a black background beside the blue and white headline “CALIBRATE AI JUDGES”.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Tejas Kumar distinguishes evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. before release from protections applied by an agent harnessAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. at runtime. He introduces test cases, expected behavior, judges and aggregation, then discusses position, verbosity, self-preference and sycophancy as reasons an automated judgeA large language model as a judge is an evaluation method in which a language model scores, compares or critiques another system's output using stated criteria. can give misleading results.

    The live coding example starts with a refund-policy scenario, moves beyond a brittle text check and compares judge decisions with a labeled datasetHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences.. Kumar adds policy context, changes model when necessary and builds a CI agreement gate. He closes with retrieval-backed policy and fresh production examples; his numerical thresholds are choices for the demonstration rather than universal reliability guarantees.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows60
    • Coding & tools85
    • Models & learning20
    • Infrastructure & compute10
    • Research & evaluation100
    • Safety & security30
    • Business & productivity10
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Tejas Kumar against a black background beside the blue and white headline “CALIBRATE AI JUDGES”. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 5 October 2026 and duration 59m 49s.

    Tejas Kumar builds a policy-grounded LLM judge and an agreement-based CI gate, showing why biased judges, missing context and stale data can undermine apparently passing evaluations.