Dmitry Petrov argues that agent habits suited to code and business tables become expensive when applied to large collections of video, sensor readings and other binary data. A modest number of files can expand into many nested detections, frames and relationships, making scattered metadata files awkward to query.
His DataChain demonstration uses Python and Pydantic schemas to connect stored files, extraction functions and queryable metadata. An execution engine handles parallel work, while checkpoints and incremental updates preserve completed computation after failures or the arrival of additional files.
Materialised datasets and a shared knowledge base make later questions cheaper to answer. Petrov emphasises recording why a dataset was created, its source code, schema, dependencies and lineage so another agent or teammate can reuse it. The point is a data-specific harness around the model, not simply choosing a stronger model.
Watch on YouTube
