Communication-Efficient Distributed Training for Large Graph Neural Networks
Keywords:
graph neural networks, distributed training, communication efficiency, graph partitioning, federated learning, GPU kernels, system architecture, fairness, sustainabilityAbstract
Large graph neural networks have become central to representation learning over relational, molecular, social, and infrastructural data, yet their training remains constrained by irregular memory access, neighborhood explosion, and expensive inter-device communication. This paper examines communication-efficient distributed training for large graph neural networks from a system-level perspective. It argues that the central challenge is not merely reducing message volume but reconciling graph topology, partitioning strategy, synchronization policy, hardware heterogeneity, and governance requirements within a single training infrastructure. The analysis first reviews architectural foundations and scaling limits, including depth, sampling, and variance reduction. It then examines communication bottlenecks created by feature exchange, gradient synchronization, and remote neighborhood aggregation. The paper evaluates partitioning, caching, compression, sparsification, asynchronous coordination, and decentralized optimization as complementary design levers. It also considers hardware and kernel co-design, especially the role of specialized GPU kernels and memory hierarchies in determining practical efficiency. Beyond performance, the discussion addresses robustness, fairness, privacy, and policy implications, because distributed graph training often operates across organizational boundaries and sensitive relational data. The paper concludes that sustainable large-scale graph learning requires adaptive communication schedules, topology-aware resource governance, and reproducibility standards that treat communication efficiency as a socio-technical property rather than a purely computational metric.
References
1. Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations (pp. 1-14). OpenReview.
2. Hamilton, W. L., Ying, R., & Leskovec, J. (2017). Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (Vol. 30, pp. 1024-1034). Curran Associates.
3. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations (pp. 1-12). OpenReview.
4. Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61-80.
5. Li, G., Müller, M., Thabet, A., & Ghanem, B. (2019). DeepGCNs: Can GCNs go as deep as CNNs? In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9267-9276). IEEE.
6. Chen, J., Zhu, J., & Song, L. (2018). Stochastic training of graph convolutional networks with variance reduction. In Proceedings of the 35th International Conference on Machine Learning (pp. 941-949). PMLR.
7. Chiang, W. L., Liu, X., Si, S., Li, Y., Bengio, S., & Hsieh, C. J. (2019). Cluster-GCN: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 257-266). ACM.
8. Zeng, H., Zhou, H., Srivastava, A., Kannan, R., & Prasanna, V. (2020). GraphSAINT: Graph sampling based inductive learning method. In Proceedings of the 8th International Conference on Learning Representations (pp. 1-19). OpenReview.
9. Fey, M., & Lenssen, J. E. (2019). Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds (pp. 1-9). OpenReview.
10. Wang, M., Zheng, D., Ye, Z., Gan, Q., Li, M., Song, X., ... & Zhang, Z. (2019). Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR Workshop on Representation Learning on Graphs and Manifolds (pp. 1-9). OpenReview.
11. Zhu, R., Zhao, K., Yang, H., Lin, W., Zhou, C., Ai, B., ... & Li, J. (2019). AliGraph: A comprehensive graph neural network platform. Proceedings of the VLDB Endowment, 12(12), 2094-2105.
12. Peng, H., Xie, X., Shivdikar, K., Hasan, M. A., Zhao, J., Huang, S., ... & Ding, C. (2024, April). Maxk-gnn: Extremely fast gpu kernel design for accelerating graph neural networks training. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (pp. 683-698).
13. Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., ... & Chintala, S. (2020). PyTorch Distributed: Experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, 13(12), 3005-3018.
14. Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnovic, M. (2017). QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (Vol. 30, pp. 1709-1720). Curran Associates.
15. Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2018). Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the 6th International Conference on Learning Representations (pp. 1-14). OpenReview.
16. Lian, X., Zhang, C., Zhang, H., Hsieh, C. J., Zhang, W., & Liu, J. (2017). Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5330-5340). Curran Associates.
17. Assran, M., Loizou, N., Ballas, N., & Rabbat, M. (2019). Stochastic gradient push for distributed deep learning. In Proceedings of the 36th International Conference on Machine Learning (pp. 344-353). PMLR.
18. McMahan, B., Moore, E., Ramage, D., Hampson, S., & Agüera y Arcas, B. (2017). Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (pp. 1273-1282). PMLR.
19. Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., ... & Roselander, J. (2019). Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1, 374-388.
20. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., ... & Zhao, S. (2021). Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2), 1-210.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
