Nicholas Arcolano examines Jellyfish data on coding assistants and engineering workflows. He describes a shift in model use following a change in tool defaults and reports no clear change in merged pull request throughput in that comparison.
Using a curated dataset of about 50,000 engineers across 447 companies, Nicholas Arcolano compares tool choice, skill use, token consumption, organizational context and individual differences. He reports associations rather than proving that these factors independently cause the measured outcomes.
The talk uses merged pull requests as a throughput measureAn AI evaluation metric is a defined measurement used to quantify a particular aspect of an AI system's performance., and Nicholas Arcolano acknowledges that the analysisEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. shown does not include a quality metric. He argues that leaders should consider workflow and team context alongside model choiceModel selection chooses the AI model whose capability, quality, cost, speed, safety, and operating constraints best fit a task..
Watch on YouTube
