Abstract
Multimodal Federated Learning (FL) targets the intersection of two promising research directions in IoT scenarios: leveraging complementary multimodal information to enhance downstream inference performance and conducting distributed training with privacy protection. However, the majority of existing works primarily focus on applying different FL methods in a straightforward manner after the multimodal feature fusion stage without fundamentally disentangling the multimodal FL across both the feature space and the sample space. There still exists an important tradeoff between the computationally demanding nature of multimodal information and the limited computing resources in IoT systems. To tackle this challenge, we propose a Hybrid Federated learning algorithm tailored for Multimodal IoT systems (HFM). HFM utilizes vertical FL to distribute computing resources across the feature space and horizontal FL to distribute computing resources across the sample space. In this paper, we theoretically prove that the convergence of HFM depends on the frequency of vertical FL communication and horizontal FL communication, as well as the number of vertical partitions and horizontal partitions. Furthermore, we empirically demonstrate that HFM outperforms three types of baselines based on two public multimodal datasets, thereby making it practical for multimodal IoT systems that require rapid and accurate downstream inference tasks, such as classification, prediction, etc.