Stochastic Optimization with Relaxed Step-Size Policies for Large-Scale Machine Learning Systems
Keywords:
stochastic optimization; step-size policies; large-scale machine learning; distributed systems; algorithmic governance; sustainabilityAbstract
Large-scale machine learning systems increasingly depend on stochastic optimization as the backbone of model training across distributed infrastructures, heterogeneous hardware, and dynamic data environments. Within these systems, step-size policies are not merely numerical hyperparameters but structural control mechanisms that shape convergence behavior, resource utilization, robustness to heterogeneity, and long-run operational stability. Relaxed step-size policies have emerged as an important design direction because they attempt to balance theoretical convergence guarantees with the practical realities of noisy gradients, asynchronous updates, non-convex loss landscapes, and constrained computational budgets. This paper provides a systems-oriented analysis of stochastic optimization with relaxed step-size policies. It examines the architectural and infrastructural trade-offs that arise when optimization decisions are embedded in distributed training frameworks, the robustness challenges associated with data heterogeneity and asynchronous execution, and the implications for fairness, governance, and sustainability in machine learning practice. Rather than presenting formal convergence proofs or algorithmic pseudo-code, the paper develops a conceptual framework linking step-size design to system-level concerns such as communication overhead, failure recovery, energy consumption, and model auditability. The discussion integrates insights from optimization theory, distributed systems, machine learning operations, fairness research, and policy studies. The paper argues that step-size policies should be treated as socio-technical design objects whose effects extend beyond training loss curves into organizational accountability, environmental impact, and the equitable behavior of deployed models.
References
1. Robbins, H., & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3), 400–407.
2. Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 2010 (pp. 177–186). Springer.
3. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., ... & Ng, A. Y. (2012). Large scale distributed deep networks. Advances in Neural Information Processing Systems, 25.
4. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., ... & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28.
5. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12, 2121–2159.
6. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
7. Reddi, S. J., Kale, S., & Kumar, S. (2018). On the convergence of Adam and beyond. In 6th International Conference on Learning Representations.
8. Zinkevich, M. (2003). Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning (pp. 928–936).
9. Polyak, B. T., & Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4), 838–855.
10. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., ... & Zheng, X. (2016). TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (pp. 265–283).
11. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., ... & Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32.
12. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster computing with working sets. In 2nd USENIX Workshop on Hot Topics in Cloud Computing.
13. Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009). Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4), 1574–1609.
14. Jin, R., Cheng, D., Qiao, H., Shi, X., Liu, S., & Zhang, B. (2025). Stochastic gradient descent in non-convex problems: Asymptotic convergence with relaxed step-size via stopping time methods. arXiv preprint arXiv:2504.12601.
15. Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., ... & He, K. (2017). Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677.
16. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations.
17. Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2018). Don't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations.
18. Hardt, M., Recht, B., & Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1225–1234).
19. Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California Law Review, 104(3), 671–732.
20. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220–229).
21. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623).
22. Strubell, E., Ganesan, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645–3650).
23. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., ... & Dean, J. (2021). Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350.
24. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
