LAUSR

For large-scale machine learning tasks, distributing data in multiple clients, and using distributed optimization algorithms with a parameter server can accelerate the training process. The federated average algorithm has been widely used for distributed optimization via training local models in parallel and aggregating local models in a server to obtain the global model. To further improve the performance of the federated average algorithm, a novel federated learning algorithm have been proposed in this article by embedding a lazy strategy in the distributed Adam-type algorithm. In the proposed algorithm, the learning rate is adjusted adaptively in local update and lazy update strategy is applied on the second-order momentum of clients to make the learning rate identical. The convergence of the proposed algorithm is provided for both convex and nonconvex loss functions. Experiments have been conducted on MNIST digit recognition data set and CIFAR-10 data set. Experimental results show that the proposed algorithm can significantly reduce the communication overhead, thereby reduce the training time by 60% for CIFAR-10 data set, and the proposed algorithm achieve better performance than the federated average algorithm and its momentum version.