Károly Zsolnai-Fehér describes research using models to translate internal activations into text and back, comparing the reconstructed activations with the originals. The approach aims to make aspects of model behavior easier to inspectInterpretability is the ability to understand how a model represents information or produces its behavior..
The video discusses examples involving rhyme planning, an incorrect calculator result and awareness of evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. These are research demonstrations rather than proof that every interpretation is accurate.
Zsolnai-Fehér stresses that the method is noisy, requires careful choices about network layersA neural network is a computational model built from connected units whose adjustable parameters transform inputs into outputs. Training changes those parameters so the network can learn useful patterns. and can be costly. He cautions against describing it as a perfect mind reader.
Watch on YouTube
