Seminars & Colloquia Calendar
Statistical physics of learning in neural networks: the importance of data structure
Marc Mézard - Laboratoire de Physique de l’Ecole Normale Supérieure, PSL University, Paris
Location: zoom
Date & time: Wednesday, 12 May 2021 at 10:45AM - 11:45AM
Marc Mézard - Laboratoire de Physique de l’Ecole Normale Supérieure, PSL University, Paris
Wednesday, May 12, 10:45AM
“Statistical physics of learning in neural networks: the importance of data structure”
The highly structured character of data used in training deep networks is a crucial ingredient of their performance. Yet theoretical work has largely overlooked this structure. Modelling structured data, analyzing the learning and the generalization of deep network trained on this data, are major challenges. This talk will describe several recent developments in this direction.
We shall introduce a generative model for structured datasets, the hidden manifold model, in which high-dimensional inputs lie on a lower-dimensional folded manifold, as in real datasets. The analytic study of learning with such data ensembles is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. This can also be extended to data drawn from pre-trained generative models. The Gaussian equivalence, which can be proven in some cases, allows to apply statistical physics methods that accurately describe the learning dynamics and the phase diagram.
