Title: Federated Learning Can Find Friends That Are Advantageous and Help with Low-Resource Machine Translation
Abstract: In Federated Learning (FL), the distributed nature and heterogeneity of client data present both opportunities and challenges. While collaboration among clients can significantly enhance the learning process, not all collaborations are beneficial; some may even be detrimental.In this talk, I will discuss our novel algorithm that assigns adaptive aggregation weights to clients participating in FL training, identifying those with data distributions most conducive to a specific learning objective.As I will explain during the talk, the proposed aggregation method converges no worse than the method that aggregates only the updates received from clients with the same data distribution.Furthermore, empirical evaluations consistently reveal that collaborations guided by the proposed algorithm outperform traditional FL approaches.Moreover, in the second part of my talk, I will explain how the proposed approach can be adjusted to the training of LLMs on Low-Resource languages
Présentation :
Speaker: Eduard Gorbunov is a research scientist at MBZUAI (where he worked before as a postdoc), hosted by Samuel Horváth and Martin Takáč. Previously, he worked as a junior researcher at MIPT and as a research consultant (remote postdoc) at Mila in the group of Gauthier Gidel. He obtained his PhD degree at MIPT, Phystech School of Applied Mathematics and Informatics, where he worked under the supervision of professors Alexander Gasnikov and Peter Richtárik.Eduard’s research interests include Stochastic Optimization and its applications to Machine Learning, Distributed Optimization, Derivative-Free Optimization, and Variational Inequalities.
Bibliography: N. Tupitsa, S. Horváth, M. Takáč, E. Gorbunov. Federated Learning Can Find Friends That Are Advantageous, arXiv preprint, https://arxiv.org/abs/2402.05050V. Moskvoretskii, N. Tupitsa, C. Biemann, S. Horváth, E. Gorbunov, I. Nikishina. Low-Resource Machine Translation through the Lens of Personalized Federated Learning, EMNLP 2024 (Findings), https://arxiv.org/abs/2406.12564
