dc.contributor.author | Liao, Qianli | |
dc.contributor.author | Kawaguchi, Kenji | |
dc.contributor.author | Poggio, Tomaso | |
dc.date.accessioned | 2016-10-21T15:42:40Z | |
dc.date.available | 2016-10-21T15:42:40Z | |
dc.date.issued | 2016-10-19 | |
dc.identifier.uri | http://hdl.handle.net/1721.1/104906 | |
dc.description.abstract | We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations. | en_US |
dc.description.sponsorship | This work was supported by the Center for Brains, Minds and Machines
(CBMM), funded by NSF STC award CCF - 1231216. | en_US |
dc.language.iso | en_US | en_US |
dc.publisher | Center for Brains, Minds and Machines (CBMM), arXiv | en_US |
dc.relation.ispartofseries | CBMM Memo Series;057 | |
dc.rights | Attribution-NonCommercial-ShareAlike 3.0 United States | * |
dc.rights.uri | http://creativecommons.org/licenses/by-nc-sa/3.0/us/ | * |
dc.subject | Batch Normalization (BN) | en_US |
dc.subject | recurrent learning | en_US |
dc.subject | Lp Normalization | en_US |
dc.title | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning | en_US |
dc.type | Technical Report | en_US |
dc.type | Working Paper | en_US |
dc.type | Other | en_US |
dc.identifier.citation | arXiv:1610.06160v1 | en_US |