Loss landscape: SGD can have a better view than GD
Author(s)
Poggio, Tomaso; Cooper, Yaim![Thumbnail](/bitstream/handle/1721.1/126041/CBMM-Memo-107.pdf.jpg?sequence=5&isAllowed=y)
DownloadCBMM-Memo-107.pdf (1.033Mb)
Metadata
Show full item recordAbstract
Consider a loss function L = ni=1 l2i with li = f(xi) − yi, where f(x) is a deep feedforward network with R layers, no bias terms and scalar output. Assume the network is overparametrized that is, d >> n, where d is the number of parameters and n is the number of data points. The networks are assumed to interpolate the training data (e.g. the minimum of L is zero). If GD converges, it will converge to a critical point of L, namely a solution of ni=1 li∇li = 0. There are two kinds of critical points - those for which each term of the above sum vanishes individually, and those for which the expression only vanishes when all the terms are summed. The main claim in this note is that while GD can converge to both types of critical points, SGD can only converge to the first kind, which include all global minima.
Date issued
2020-07-01Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;107