Joint Optimization of Optical Network Latency and GPU Communication Efficiency in Distributed AI Training
Keywords:
optical networks; distributed AI training; GPU communication; latency optimization; collective communication; data center architecture; sustainability; governanceAbstract
Distributed artificial intelligence training has become a foundational workload for modern computing infrastructure, yet its performance is increasingly constrained not only by accelerator capacity but also by the latency and efficiency of communication across optical networks and graphics processing unit interconnects. This paper examines the joint optimization of optical network latency and graphics processing unit communication efficiency in distributed artificial intelligence training from a system-level perspective. It argues that latency and communication efficiency cannot be treated as independent engineering variables because choices in optical switching, topology, synchronization, collective communication, congestion control, and workload placement mutually shape one another. The analysis considers architectural trade-offs among optical circuit switching, wavelength switching, packet-based transport, and hybrid electrical-optical fabrics, with attention to clock synchronization, hollow core fiber, in-network aggregation, and collective communication scheduling. It further explores governance, fairness, sustainability, and policy implications, including energy proportionality, carbon accounting, equitable access to advanced network infrastructure, and resilience against optical and synchronization failures. Through cross-domain comparisons with high-performance computing, cloud data centers, and telecommunications, the paper identifies structural tensions between low-latency specialization and general-purpose flexibility. The conclusion emphasizes that joint optimization requires cross-layer telemetry, adaptive control, transparent benchmarking, and institutional coordination. Without such integrated approaches, distributed training systems risk over-investing in raw bandwidth while underutilizing accelerators, exacerbating energy use, and reproducing infrastructural inequities.
References
1. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q., & Ng, A. Y. (2012). Large scale distributed deep networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems (Vol. 25, pp. 1223–1231). Curran Associates.
2. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., ... Zheng, X. (2016). TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (pp. 265–283). USENIX Association.
3. Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019). PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (pp. 1–15). ACM. https://doi.org/10.1145/3341301.3359646
4. Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–16). IEEE. https://doi.org/10.1109/SC41405.2020.00024
5. Sergeev, A., & Del Balso, M. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv. https://arxiv.org/abs/1802.05799
6. Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., & Su, B.-Y. (2014). Scaling distributed machine learning with the parameter server. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (pp. 583–598). USENIX Association.
7. Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnovic, M. (2017). QSGD: Communication-efficient SGD via gradient quantization and encoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30, pp. 1709–1720). Curran Associates.
8. Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2018). Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations. OpenReview.
9. Hoefler, T., & Belli, R. (2015). Scientific benchmarking of parallel computing systems: Twelve ways to tell the masses when reporting performance results. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–12). ACM. https://doi.org/10.1145/2807591.2807644
10. Kachris, C., & Tomkos, I. (2012). A survey on optical interconnects for data centers. IEEE Communications Surveys & Tutorials, 14(4), 1021–1036. https://doi.org/10.1109/SURV.2011.122111.00069
11. Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H. H., Subramanya, V., Fainman, Y., Papen, G., & Vahdat, A. (2010). Helios: A hybrid electrical/optical switch architecture for modular data centers. In Proceedings of the ACM SIGCOMM 2010 Conference (pp. 339–350). ACM. https://doi.org/10.1145/1851182.1851223
12. Mittal, R., Lam, V. T., Dukkipati, N., Blem, E., Wassel, H., Ghobadi, M., Vahdat, A., Wang, Y., Wetherall, D., & Zats, D. (2015). TIMELY: RTT-based congestion control for the datacenter. ACM SIGCOMM Computer Communication Review, 45(4), 537–550. https://doi.org/10.1145/2829988.2787510
13. Cheng, Q., Bahadori, M., Glick, M., Rumley, S., & Bergman, K. (2018). Recent advances in optical technologies for data centers: A review. Optica, 5(11), 1354–1370. https://doi.org/10.1364/OPTICA.5.001354
14. Zhou, Zichuan, et al. "Low-latency wavelength-switched clock-synchronized intra-data center interconnects enabled by hollow core fiber." Optics Express 31.15 (2023): 24739-24748.
15. Luo, L., Nelson, J., Ceze, L., Phanishayee, A., & Krishnamurthy, A. (2020). Parameter hub: A rack-scale parameter server for distributed deep neural network training. In Proceedings of the ACM Symposium on Cloud Computing (pp. 41–54). ACM. https://doi.org/10.1145/3419111.3421280
16. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645–3650). ACL. https://doi.org/10.18653/v1/P19-1355
17. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021). Carbon emissions and large neural network training. arXiv. https://arxiv.org/abs/2104.10350
18. Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63. https://doi.org/10.1145/3381831
19. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM. https://doi.org/10.1145/3442188.3445922
20. Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020). Recalibrating global data center energy-use estimates. Science, 367(6481), 984–986. https://doi.org/10.1126/science.aba3758
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
