Photo by hajjidirir from unsplash
In online convex optimization, adaptive algorithms, which can utilize the second-order information of the loss function’s (sub)gradient, have shown improvements over standard gradient methods. This paper presents a framework Follow… Click to show full abstract
In online convex optimization, adaptive algorithms, which can utilize the second-order information of the loss function’s (sub)gradient, have shown improvements over standard gradient methods. This paper presents a framework Follow the Bregman Divergence Leader that unifies various existing adaptive algorithms from which new insights are revealed. Under the proposed framework, two simple adaptive online algorithms with improvable performance guarantee are derived. Furthermore, a general equation derived from a matrix analysis generalizes the adaptive learning to nonlinear case with kernel trick.
Journal Title: IEEE Transactions on Neural Networks and Learning Systems
Year Published: 2017
Link to full text (if available)
Share on Social Media: Sign Up to like & get
recommendations!