Károly Zsolnai-Fehér presents Qwen 3.8 Flash Next as a mixture-of-experts modelA mixture-of-experts model contains multiple specialized subnetworks and activates a selected subset for each input instead of using every parameter every time. designed to use only a small portion of its parameters for each token. That structure makes it attractive for machines with ample memory but comparatively limited memory bandwidth, while early local tests show practical generation speeds.
Károly Zsolnai-Fehér highlights Qwen Sparse AttentionSparse attention reduces transformer computation by allowing each token to attend to a selected subset of other tokens instead of comparing with every token., which groups tokens into small blocksBlock-sparse attention divides a token sequence into blocks and computes attention for only selected blocks, making long-context processing more efficient than dense full attention. before searching for the most relevant context. This avoids the quadratic growth of full attentionQuadratic attention complexity is the tendency of full self-attention work and attention-map size to grow roughly with the square of the input sequence length. and can make increasingly long conversations or document collectionsA long-context AI model can process substantially more tokens in one request, allowing it to work across long documents, conversations, or codebases. cheaper to process.
Károly Zsolnai-Fehér also explains gated residual branches that protect information from being repeatedly overwritten across layers, plus an engram embedding layer that retrieves common short token combinations cheaply. Together, these mechanisms help the model compete with much larger open-weight systems while remaining more efficient to run.
Watch on YouTube
