Federated Prompt Tuning with Selective Prompt Synchronization for Privacy-Preserving NLP
Keywords:
Federated learning, prompt tuning, privacy, selective synchronization, NLP, large language modelsAbstract
The rapid advancement of large language models has transformed natural language processing but simultaneously intensified concerns around data privacy and sovereignty. Federated learning provides a compelling framework for collaborative model training without exposing raw user data, yet deploying these massive models across decentralized clients remains impractical due to enormous communication costs. Prompt tuning, which freezes the pre-trained model and learns only a small set of continuous embeddings, has emerged as a parameter-efficient alternative with strong downstream performance. When prompt tuning is integrated into federated settings, however, all prompt parameters are typically synchronized across clients, which still incurs nontrivial communication overhead and creates new privacy vulnerabilities through the shared representation space. This paper proposes a federated prompt tuning architecture augmented with selective prompt synchronization, in which only a carefully chosen subset of prompt tokens is exchanged between local clients and the aggregation server. Drawing on selective prompt insertion concepts, the system learns to identify and transmit the most informative prompt dimensions while suppressing redundant or privacy-sensitive information. We present a comprehensive system-level analysis that examines the structural trade-offs among communication efficiency, model accuracy, privacy protection, fairness, and governance. The discussion extends to integration with differential privacy, robustness against adversarial clients, sustainability, and policy implications in cross-silo and cross-device deployments. This work offers a blueprint for building secure, efficient, and responsible federated natural language processing infrastructures.
References
1. McMahan, B., Moore, E., Ramage, D., Hampson, S., & Aguera y Arcas, B. (2017). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 1273-1282.
2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
3. Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045-3059.
4. Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 4582-4597.
5. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. Proceedings of the 36th International Conference on Machine Learning, 2790-2799.
6. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., ... & Zhao, S. (2021). Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2), 1-210.
7. Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., ... & van Overveldt, T. (2019). Towards federated learning at scale: System design. Proceedings of the 2nd SysML Conference.
8. Yang, Y., Zhang, H., Wang, S., & Sun, M. (2023). Federated prompt tuning for language models. Findings of the Association for Computational Linguistics: ACL 2023, 2044-2056.
9. Sun, T., He, J., Qiu, X., & Huang, X. (2022). Black-box tuning for language-model-as-a-service. Proceedings of the 39th International Conference on Machine Learning, 20864-20885.
10. Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2023). Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 5, 429-450.
11. Reddi, S. J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konecny, J., Kumar, S., & McMahan, H. B. (2021). Adaptive federated optimization. Proceedings of the 9th International Conference on Learning Representations.
12. Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016). Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 308-318.
13. Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4), 211-407.
14. Wang, Z., Dai, Z., Poczos, B., & Carbonell, J. (2021). Characterizing and avoiding negative transfer in federated transfer learning with pre-trained models. Proceedings of the 2021 IEEE International Conference on Big Data, 1132-1141.
15. Zhu, W., & Tan, M. (2023, December). SPT: Learning to selectively insert prompts for better prompt tuning. In Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 11862-11878).
16. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730-27744.
17. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.
18. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. Proceedings of the 11th International Conference on Learning Representations.
19. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623.
20. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6), 1-35.
21. Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., ... & Bonawitz, K. (2018). Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604.
22. Zhang, D. Y., Kou, Z., & Wang, D. (2021). Fairness in federated learning: A survey. ACM Computing Surveys, 54(11s), 1-38.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
