Pamela Fox walks through a Python workshop for comparing models on concrete application tasks. Letter counting, retrieval-grounded answers and image questions expose different failure modes, while prompt changes sometimes help and sometimes only make an unsupported answer easier to recognize. The demonstrations are scenario-specific observations, not a universal model ranking.
Pamela Fox builds from individual tool calls to calculator loops and a travel-planning agent. She examines correct arguments, parallel execution, number of calls, budget checks and whether the agent completes the requested final step. The examples distinguish a correct final answer from an efficient, complete process, and show why the framework and harness matter alongside the model.
Pamela Fox then moves from manual inspection to automated checks, LLM judges, specification-derived evaluations with Assert and prompt optimization with DSPy. She stresses human calibration of judges, access to tool traces and task-specific rubrics. Slow, costly evaluations need a deliberate cadence; changing the judge should not require rerunning unchanged application outputs. The workshop closes with structured citations and local-model options, emphasizing flexibility rather than assuming that a model swap is merely a configuration change.
Watch on YouTube
