NeurIPS 2021: Cockpit: A Practical Debugging Tool for Training Deep Neural Networks
Cockpit: A Practical Debugging Tool for Training Deep Neural Networks
Frank Schneider, Felix Dangel, and Philipp Hennig
Advances in Neural Information Processing Systems (NeurIPS) 2021
► Paper: https://arxiv.org/abs/2102.06604
► Cockpit Code: https://github.com/f-dangel/cockpit
► Try it: pip install cockpit-for-pytorch
When engineers train deep learning models, they are very much “flying blind”. Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network’s training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a collection of instruments that enable a closer look into the inner workings of a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PyTorch, is a valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets.
► Find out more about our research at https://uni-tuebingen.de/en/134428.