Weight Folding, CUDA Streams, and an AI Inference Bug - Filip Makraduli

AI Engineer17m 18s
Filip Makraduli in a blue shirt rests his hand on his chin beside the white and blue headline THE INFERENCE BUG on black.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Filip Makraduli presents an approach to reducing RMSNorm overhead in transformer inferenceInference is the use of a trained model to produce an output from supplied input.. Although normalization accounts for a small share of arithmetic, repeated kernel launches, memory movement and waiting can consume noticeable wall-clock timePerformance profiling measures where a program spends time or computational resources so optimization can target observed bottlenecks. during decoding.

    The approach folds a normalization gain into a following matrix weight ahead of execution, defers a scalar division so vector and matrix work can overlap, and can remove a redundant normalization in suitable architectures. The speaker distinguishes the simple weight transformation from the lower-level GPU kernel workA graphics processing unit kernel is a function compiled to run many parallel operations on a graphics processor or related accelerator. needed for parallel execution.

    In a CUDA implementation, generated text began repeating earlier output because two streams joined without an explicit wait. Short tests did not reveal the race, while longer generation exposed a one-step lagEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. Waiting for both operations before post-scaling fixed the stale read.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows0
    • Coding & tools100
    • Models & learning75
    • Infrastructure & compute100
    • Research & evaluation75
    • Safety & security0
    • Business & productivity0
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Filip Makraduli in a blue shirt rests his hand on his chin beside the white and blue headline THE INFERENCE BUG on black. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 19 September 2026 and duration 17m 18s.

    Filip Makraduli explains how weight folding and overlapping GPU work can reduce RMSNorm overhead, then traces a stale-output bug to missing CUDA stream synchronization.