Max Ryabinin explains why long contexts matter for agents and video models, and why attention computation and memory consumptionAI compute infrastructure is the hardware, facilities, networks, power, cooling, storage, and software used to train and run AI models. make long-sequence training difficult.
Max Ryabinin walks through sharding, Ulysses context parallelism, activation checkpointing, CPU offloading and sequence chunking. He describes an additional tiled attention approach that reuses smaller buffers, reporting results at up to five million tokens while acknowledging throughput and memory tradeoffs.
Watch on YouTube
