What is mixed precision?

Definition

Less sensitive parts can use smaller number formats to reduce storage and computation. Parts where rounding would hurt the result can use more precise formats.

The choice depends on the workload and hardware. A comparison should check quality as well as the speed and memory improvements.

ELI5

For example, a model might store most weights with less detail while keeping a few sensitive sections at higher precision.

This can let it fit on a smaller device while preserving accuracy where rounding would otherwise cause problems.

What does precision refer to?

It describes how finely a number format represents values, which affects the rounding incurred when a value is stored or processed.

Will every device run mixed precision faster?

Performance depends on which numeric formats the hardware can process efficiently and how much conversion is needed.

  1. Chris Alexiuk, Daniel Han, Asma Beevi in a light headscarf, and Merve Noyan beside the white-and-blue headline “AI AT THE EDGE” on black.
  2. A flat compression graphic beside the words Qwen Quants Run Faster
Definition card for mixed precision: What is mixed precision?

Mixed precision uses multiple numeric precisions within a computation or model.