Hong warns that raw token consumptionAI token efficiency measures how effectively a model or workflow turns consumed input and output tokens into useful results. and internal usage leaderboards can reward activity without proving that AI has created customer value. He proposes trusted throughput instead: the rate at which a team turns AI-assisted workAI-assisted software development uses AI to support planning, coding, testing, review, debugging, documentation, and maintenance while people retain responsibility for the software. into reviewed, validated, maintainable and safely deployed outcomes.
The measurement should combine objective signals with human judgmentEvaluation is the systematic process of testing and judging an AI system against defined tasks, evidence, and success criteria.. Merged pull requests, customer-facing improvements and cost trends can show useful movement, while engineering review and downstream outcomes reveal whether the work deserves trust. The same framework also makes it possible to compare adoption by team or workflow without treating every generated token as progress.
As code generation becomes abundant, the bottleneck moves into review, continuous integration and organizational confidence. Hong recommends investing in AI-assisted first-pass reviewAI-assisted code review uses AI to inspect software changes for defects, risks, standards, missing tests, and opportunities for improvement while human reviewers retain responsibility., stronger test infrastructure and explicit controls such as budgets, quotas, anomaly detection and loop-step limitsA safeguard is a measure designed to prevent, limit, detect, or recover from unwanted behavior or harm in an AI system.. Prompt caching and context pruning can also reduce waste without weakening the work itself.
The practical goal is not maximum AI usage. It is a learning loop in which teams observe where AI succeeds, capture repeatable patterns and feed those lessons back into engineering practice. Hiring and company promotion at the end of the talk are omitted.
Watch on YouTube
