Reply to Huszár: The elastic weight consolidation penalty is empirically valid
2018; National Academy of Sciences; Volume: 115; Issue: 11 Linguagem: Inglês
10.1073/pnas.1800157115
ISSN1091-6490
AutoresJames Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska‐Barwińska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell,
Tópico(s)Machine Learning in Materials Science
ResumoIn our recent work on elastic weight consolidation (EWC) (1) we show that forgetting in neural networks can be alleviated by using a quadratic penalty whose derivation was inspired by Bayesian evidence accumulation. In his letter (2), Dr. Huszar provides an alternative form for this penalty by following the standard work on expectation propagation using the Laplace approximation (3). He correctly argues that in cases when more than two tasks are undertaken the two forms of the penalty are different. Dr. Huszar also shows that for a toy linear regression problem his expression appears to be better. We would like to thank Dr. Huszar for pointing out … [↵][1]1To whom correspondence should be addressed. Email: kirkpatrick@google.com. [1]: #xref-corresp-1-1
Referência(s)