Abstract
Distributed federated learning (DFL) enables mobile devices to fine-tune a large language model (LLM) collaboratively by directly sharing local updated parameters on a device-to-device (D2D) network through wireless connections. However, the local fine-tuning and updated parameters aggregation impose a significant burden on devices with limited computational and communication resources. Furthermore, aggregating parameters from heterogeneous datasets can lead to catastrophic knowledge interference, thereby degrading the overall performance of LLM. To address these issues, we formulate an optimization problem whose goal is to maximize LLM's overall performance under resource constraint. To address this problem, we employ a sparse-LoRA method that reduces both computational and communication overhead by decomposing parameters into low-rank and sparse activated matrices. To further enhance the DFL performance, we impose a parameter-orthogonalization constraint on the LoRA matrix on each device to mitigate knowledge interference. Finally, we analyze the relationship between D2D topology and DFL performance, and propose a cluster-based topology design for efficient parameter aggregation. Simulation results demonstrate that the proposed approach effectively reduces communication resources consumption by 55% and enhances DFL performance by 22% compared to baselines.