Sean Cai distinguishes authentic professional trajectories from manufactured training tasks. Finished spreadsheets and records omit the decisions, tool use, recovery and changing constraints that produced them, so collecting more examples is not automatically the same as improving training signal.
He evaluates domains through decomposition, expert agreement and access to verified real examples. Finance-task illustrations show how similar aggregate scores can conceal opposite arithmetic and methodological weaknesses. A benchmarkA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. also measures its scaffoldAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. and grading conditions, and vendors selling both tests and training material can create misleading incentives.
Cai describes opportunities in enterprise model routingAI model routing sends each request to a model chosen for that request's complexity, cost, speed, privacy, or other requirements., repeated post-training and maintained reinforcement-learning environments. Robotics provides a counterexample because the right data modality remains unsettled. Market estimates, vendor examples and historical comparisons are the speaker's analysis rather than independently verified forecasts.
Watch on YouTube




