Federated Distillation with Dataset Distillation Using Model Trajectory Matching

Abstract

Federated Learning (FL) is a machine learning technique that trains the global model on the server by leveraging the knowledge of clients without sharing the private data. In federated learning, knowledge distillation, widely known as federated distillation, is utilized for model combination to address data-heterogeneous problems arising from the non-IID distribution of private data across clients. However, existing federated distillation methods require additional training data or models to distill the knowledge of local clients. Previous federated distillation methods utilize public data, which has a different distribution from private data, or artificial data created by a generator adversarially trained on local models. However, we supposed that artificial data created by generator cannot precisely represent the semantic information of private data. To address this issue, we propose a new federated distillation framework using dataset distillation with model trajectory matching. To create effective training artificial data, we train the artificial data by comparing local model parameters before and after learning from private data. We prove that using artificial data generated by dataset distillation with model trajectory matching shows better result than generator. Additionally, we take verification experiment to compare how much the artificial data generated by generator and dataset distillation contains semantic information of private data.