Ray Fernando responds to a viewer’s question about the state of AI coding and traces the shift from chat-based workflows to tool-using agents. Ray Fernando says that a single benchmarkA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. captures only a narrow slice of an agent’s abilities.
Ray Fernando proposes testing how a model delegates to other agents, incorporates new information and coordinates long-running work. Ray Fernando describes these ideas as his own perspective and ongoing independent work.
Ray Fernando argues that practical testsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. should consider the tasks and outcomes that matter to a particular business. Ray Fernando illustrates his view with the prospect of a coordinating agent managing other tools and agentsA multi-agent system contains multiple AI agents that interact, coordinate, divide work, or influence one another while pursuing tasks..
Watch on YouTube
