Abstract
To improve communication efficiency and handle data heterogeneity challenges in federated learning (FL), fine-tuning the pre-trained large models rather than training neural networks from scratch has received increasing attention in recent years, especially under cross-domain settings. However, such a cross-domain federated fine-tuning scenario opens up a broader attack surface for new threats, especially backdoors, posing significant security risks. Existing backdoor attacks mainly focus on label shift scenarios and use explicit triggers, which lack transferability and effectiveness in cross-domain settings, thereby exhibiting significant weaknesses. In this paper, we propose Silent Penetrator, an innovative penetration scheme tailored for cross-domain federated fine-tuning, which exploits a feature shift-induced backdoor to elicit specific symptoms in the trusted private data of targeted victims. In Silent Penetrator, the attacker can obtain a high-quality poisoned dataset by leveraging the available domain information as the text prompts for Stable Diffusion, and inject a domain-sensitive backdoor that can be unconsciously triggered by unmodified private data of the victims. To achieve stronger and more persistent penetration, we thoroughly explore the adversary's configurable space and enhance our backdoor injection utilizing contrastive-enhanced boundary deviation and cross-domain predictive confrontation. Extensive experiments on three cross-domain datasets and four state-of-the-art federated fine-tuning frameworks validate the effectiveness of Silent Penetrator in successfully compromising target clients. Furthermore, our backdoor enhancement strategy improves the penetration accuracy by over 10% in most scenarios and significantly enhances the durability of the penetration compared to four state-of-the-art backdoor enhancement techniques.