Research

The project proposal gathers groups with expertise in Scheduling, Checkpointing Techniques, Tensor Train Decomposition, Deep Neural Networks, that are shared between the Russian and French sides. Based on this shared expertise, the goal is to concentrate on Training for Deep Neural Networks, with a focus on memory issues. Indeed, Tensor Train Decomposition can be used to save memory when storing the weights of the network, while Checkpointing strategies can be used to control the memory needs required by the storage of activations during the training phase. There is a strong interest in considering both approaches for High Performance DNN training since both can be efficiently combined together with parallel DNN training strategies such as Model Parallelism (for Checkpointing) and Data Parallelism (for TT decompositions). Our work program directly follows the main objectives mentioned above, where the main goal is to increase the size and depth of the learning networks and to accelerate the training process.