What is pretraining?

Definition

Pretraining exposes a model to a chosen distribution of data and learning objectives. The resulting parameters can provide a starting point for many tasks, with the scope of those capabilities depending on the training design.

Later use may involve prompting, in-context examples or additional training. Pretraining does not eliminate the need to evaluate the actual task, especially when its data or requirements differ from those represented during training.

ELI5

Pretraining is the earlier learning that prepares a model for later use. It can build useful patterns that the model then applies or adapts to more specific tasks.

For example, a model can train on many generated prediction problems before receiving a new table. That earlier practice may help it respond, but the new table still needs its own performance checks.

Is pretraining the same as inference?

No. Pretraining learns model parameters, while inference uses a trained model to produce an output.

Does pretraining guarantee performance on every later task?

No. Generalization depends on the model, training distribution and requirements of the later task.

  1. Frank Hutter in blue and Tim Scarfe wearing glasses and navy gesture beside the blue-and-white headline “TABULAR AI” on black.
  2. Nate B Jones beside the words Look Beyond the Model Race
Definition card for pretraining: What is pretraining?

Pretraining is an initial training stage that develops reusable AI model capabilities before later adaptation or use on particular tasks.