From Associative Memories to Deep Networks
Author(s)
Poggio, Tomaso
DownloadCBMM-Memo-114.pdf (1.011Mb)
Metadata
Show full item recordAbstract
About fifty years ago, holography was proposed as a model of associative memory. Associative memories with similar properties were soon after implemented as simple networks of threshold neurons by Willshaw and Longuet-Higgins. In these pages I will show that today’s deep nets are an incremental improvement of the original associative networks. Thinking about deep learning in terms of associative networks provides a more realistic and sober perspective on the promises of deep learning and on its role in eventually understanding human intelligence. As a bonus, this discussion also uncovers connections with several interesting topics in applied math: random features, random projections, neural ensembles, randomized kernels, memory and generalization, vector quantization and hierarchical vector quantization, random vectors and orthogonal basis, NTK and radial kernels.
Date issued
2021-01-12Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;114