Connectivity of Solutions in the Spherical Negative Perceptron - Brandon Annesi - Young Seminar SIFS

Connectivity of Solutions in the Spherical Negative Perceptron
Brandon Annesi, Università Bocconi
Abstract: Empirical studies on the landscape of neural networks have shown that low-energy configurations are often found in complex connected structures, where zero-energy paths between pairs of distant solutions can be constructed. In this talk I will review some of these results, and introduce the spherical negative perceptron, a prototypical non-convex neural network model framed as a continuous constraint satisfaction problem. Thanks to a new analytical method for computing energy barriers in the simplex with vertex configurations sampled from the equilibrium, I will show that in the over-parameterized regime the solution manifold displays simple connectivity properties. There exists a large geodesically convex component that is attractive for a wide range of optimization dynamics. Inside this region a subset of atypical high-margin solutions that are geodesically connected with most other solutions can be identified, giving rise to a star-shaped geometry. I will then characterize the organization of the connected space of solutions and show numerical evidence of a transition, at larger constraint densities, where the aforementioned simple geodesic connectivity breaks down.