Xander Song introduces Harbor as a framework for evaluating agentsAgent evaluation tests whether an AI agent completes tasks correctly, consistently, and within its required boundaries. against realistic tasks in fresh sandboxed environmentsAn AI agent sandbox is an isolated execution environment that limits which files, processes, networks, credentials, and external systems an agent can access.. Instructions, environment data and verifier scripts define each task; repeated independent trials account for stochastic behaviour. Oracle solutions let teams check their setup before spending tokens on an agent, while traces help explain what happened.
He shows how the Phoenix team packages its single-node application, a seeded SQLite database, skills and interfaces together to test coding agents using a CLI or MCP server and its own server-side agent. The example compares a prompt change that encourages read-only SQL for aggregate questions instead of repeatedly paginating through REST or GraphQL. Song reports fewer tool calls while answer rewards remain similar in that experiment.
The discussion separates outcome-based regression testsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked. from tests that deliberately inspect execution tracesAn AI execution trace is a structured record of the steps, tool calls, state changes, and outputs produced during an AI workflow. to optimize tool selectionAgent tool use lets an agent call bounded external capabilities so it can retrieve information or change state beyond generating a response.. Song also explains limitations: distributed applications are harder to package, realistic evaluation data takes work, skill-specific assertions can be brittle, and a live natural-language filter demonstration needs a manual correction. The reported experiment is a team example rather than a general benchmark result.
Watch on YouTube
