Dan Bjornn presents a customer messaging application built at Lease End to answer questions, arrange calls and connect customers with the sales team. An initial retrieval-based workflow struggled with conversational nuance, so the team fine-tuned a modelFine-tuning adapts a pretrained model by updating its parameters on examples chosen for a particular task or subject. to classify messages into six intent categoriesA classifier assigns an input to one or more defined categories based on rules or learned patterns.. The narrow task, high message volume and hopes of lower latency and greater vendor flexibility made fine-tuning appear attractive. The application reportedly contributed to $12 million in revenue at a 50-fold return within a year, but commercial success concealed mounting maintenance costs.
Dan Bjornn illustrates the failure modes with messages that confirm a future appointment or simply offer a greeting, which the application misinterprets as requests for an immediate call. Repairing these mistakes involved gathering and sometimes synthesizing examples, validating and labeling them, retraining, and checking for regressions. Fine-tuning itself could take about an hour, while the complete repair cycle typically took a week. The team consequently triaged problems by frequency and customer impact rather than correcting every issue as soon as it appeared.
Dan Bjornn calls the resulting rigidity a calcification tax: changes to model versions, providers and application architecture became harderVendor lock-in occurs when switching away from a provider becomes difficult or costly because systems depend on its proprietary interfaces, data, features, contracts, or operations. because the training data and surrounding workflow were tightly coupled. Inspired by how Claude Code changes behavior through skills and context, the team rebuilt the messaging workflow as skills, tools and supporting resourcesAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work. within an agent framework. Fixes could then be made in prompts or skill files, evaluated against a curated setEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria., and deployed by uploading Markdown files to storage. He reports that this reduced the repair cycle to less than an hour.
Dan Bjornn reports improved accuracy and lower total operating cost after the rebuild, even though stronger models increased the API cost per message. The talk argues for comparing maintenance effort, migration flexibility and customer impact alongside inference price and latency. It does not provide detailed accuracy measurements or independently audited financial results. Privacy, data control and offline operation are acknowledged as situations in which fine-tuning may remain worth considering, but the speaker urges teams to account for the continuing cost of keeping the system adaptable.
Watch on YouTube
