A dense AI model uses the same connected parameter set for every token processed through a layer. This makes its computational path straightforward, but a large dense model may require substantial memory bandwidth and arithmetic work for every generated token.
Dense models contrast with mixture-of-experts models, which route a token through selected expert blocks. Parameter count alone therefore does not provide a complete comparison of memory needs, active compute or inference speed across the two designs.
ELI5
A dense model uses its full connected set of model parameters when it processes each token. The path is consistent, unlike a mixture-of-experts model that selects only some expert blocks for each token.
For example, a dense model with 20 billion parameters may need to move and use that large weight set for every generated token. This can make hardware needs easier to understand, but memory bandwidth and calculation costs grow with the model's active size.
How is a dense model different from a mixture-of-experts model?
A dense model activates its full parameter path for each token, while a mixture-of-experts model selects only some expert components for a given token.
Does a dense model use every stored parameter for every token?
In the usual architecture-level sense, its layers use the complete dense path, unlike sparse expert routing that activates only selected parameter blocks.
