What is vision transformer?

Definition

An image can be divided into patches that are converted into numerical representations and combined with positional information. Attention mechanisms learn relationships among these representations for a visual task.

Intermediate features are learned numerical structures rather than necessarily recognizable image parts. Their usefulness depends on training data, architecture and evaluation; an intuitive patch description does not fully explain the model's internal computation.

ELI5

Imagine dividing a picture into small tiles and considering how the tiles relate to understand the whole scene.

For example, a model can connect information from patches covering an object's shape and surroundings rather than treating each patch as an independent picture.

Are the tokens necessarily words?

No. In a vision transformer, tokens can represent visual patches or other image-related information.

Are its intermediate features always easy to interpret?

No. Learned numerical representations need not correspond to simple human-readable image features.

  1. Richard Diehl Martinez, Ben Affleck and Pierce Freeman beside the blue and white headline AFFLECK EXPLAINS AI on a black background.
Definition card for vision transformer: What is vision transformer?

A vision transformer applies transformer-style processing to visual data, commonly by representing image patches as tokens.