The speaker explains how mixture-of-experts models route tokens through selected componentsA mixture-of-experts model contains multiple specialized subnetworks and activates a selected subset for each input instead of using every parameter every time., then describes using calibration data to assess which experts to prune. The talk distinguishes pruning from quantization and stresses the hardware and evaluation tradeoffs.
The speaker demonstrates compressed models on local hardwareOn-device inference runs a trained model on the user's device rather than relying entirely on a remote model service., including assistant and design workflows, and discusses how the selection of calibration data can influence results. Claims about costs and performance are presented as the speaker’s estimates and experience.
In audience questions, the speaker discusses router behavior, evaluation after pruning, using already quantized models and sharing calibration observations. The proposed future compression levels are speculative.
Watch on YouTube




