A neural audio codec learns how to map an audio waveform into a compact latent representation and reconstruct sound from that representation. The compressed form may use discrete tokens arranged across one or more codebooks.
Generative speech systems can predict codec tokens instead of producing every waveform sample directly. This makes audio compatible with sequence models, but multiple codebooks may require additional prediction steps and can increase generation complexity.
Quality depends on the codec's compression rate, training data and reconstruction objective. A compact representation saves computation or bandwidth, but excessive compression can remove detail or introduce audible artifacts.
ELI5
A neural audio codec is like a learned shorthand for sound. It turns a long waveform into a smaller set of symbols and later uses those symbols to rebuild the audio.
For example, a speech generator can predict the shorthand symbols for a sentence and let the codec turn them into sound. If the shorthand is too small, some voice detail may be lost.
