The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

AI Engineer17m 31s
Marah Abdin in blue and Robert McHardy in off-white gesture toward each other beside the blue-and-white headline “SCALE WITHOUT CORRUPTION” on black.
1 VIEW
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Marah Abdin describes synthetic dataSynthetic data is information generated to represent examples or situations rather than collected directly from the events it describes. as a complement to organic material, exposing implicit reasoning, planning and structure rather than merely replacing real text. Rephrasing and specialized code and STEM pipelines reduce repetition, while staged generation makes difficult tasks manageable for the teacher model.

    She presents composable pipelines with seeds, metadata, generators, filters and validators. Hive adds configurable agents, orchestrationAgent orchestration coordinates AI agents, tools, people, tasks, state, and control flow so a larger workflow reaches a verified outcome. and supervision. Robert McHardy explains why data qualityTraining data is the collection of examples and signals used to adjust an AI model's parameters so it learns useful patterns. and training implementation must be treated together, using replica-weight hashes to detect silent corruption from a defective GPU.

    A BF16 accumulation bottleneck is corrected with FP32, while an FP8 kernel race condition reveals a blind spot in ordinary replica checks. The speakers carry revised data and numerical safeguards into larger Laguna models. Reported base-model evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. are preliminary, not final post-training performance or independent reproduction.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows25
    • Coding & tools25
    • Models & learning100
    • Infrastructure & compute75
    • Research & evaluation75
    • Safety & security0
    • Business & productivity0
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Marah Abdin in blue and Robert McHardy in off-white gesture toward each other beside the blue-and-white headline “SCALE WITHOUT CORRUPTION” on black. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 26 July 2026 and duration 17m 31s.

    Marah Abdin and Robert McHardy explain why scaling language models requires diverse synthetic data and a training system that detects hardware faults, precision problems and kernel corruption.