Keywords:-
Article Content:-
Abstract
The performance of deep neural networks is critically influenced by the choice of optimization algorithm and its hyperparameters, particularly the learning rate (η). However, the interplay between optimizers and learning rates across different task paradigms, such as classification and regression, remains under-explored. This paper addresses this gap by presenting a systematic empirical study of four common Optimizers (Adaptive Moment Estimation (Adam), Adaptive Gradient Algorithm (Adagrad), Root Mean Square Propagation(RMSProp), and Stochastic Gradient Descent(SGD) with Nesterov Accelerated Gradient momentum) across a spectrum of learning rates . We evaluate 32 configurations on two distinct tasks: a Convolutional Neural Network (CNN) for image classification and a Multilayer Perceptron (MLP) for tabular regression. Our findings demonstrate that optimizer performance, stability, and learning rate sensitivity are highly task-dependent. We show that the optimal configuration diverges significantly between tasks: the CNN classification task achieved peak accuracy with RMSProp at a low η =10−4, whereas the MLP regression task achieved the highest R2 score (0.9102) with the non-adaptive SGD at a moderate η =10−2. These results provide strong empirical evidence that optimization strategies must be carefully tailored to the specific task and architecture.
References:-
References
Z. Lu, Y. Sun, Z. Yang, Y. Zhang, P. Tulajiang, H. Sun, and H. Lin, “Improving generalization in DNNs through enhanced orthogonality in momentum-based optimizers,” Inf. Process. Manag., vol. 62, no. 4, Art. no. 104109, 2025, doi: 10.1016/j.ipm.2025.104109.
A. Atamna, T. Maus, F. Kievelitz, and T. Glasmachers, “Cumulative learning rate adaptation: Revisiting path-based schedules for SGD and Adam,” arXiv:2508.05408, 2025. [Online]. Available: https://arxiv.org/abs/2508.05408.
Y. Wu, L. Liu, J. Bae, K.-H. Chow, A. Iyengar, C. Pu, W. Wei, L. Yu, and Q. Zhang, “Demystifying learning rate policies for high accuracy training of deep neural networks,” arXiv:1908.06477, 2019. [Online]. Available: https://arxiv.org/abs/1908.06477.
C. E. Chika, “Machine learning approach and model for predicting proton stopping power ratio and other parameters using computed tomography images,” J. Med. Phys., vol. 49, no. 4, pp. 519–530, 2024, doi: 10.4103/jmp.jmp_120_24.
C. E. Chika, “Estimation of proton stopping power ratio and mean excitation energy using electron density and its applications via machine learning approach,” J. Med. Phys., vol. 49, no. 2, pp. 155–166, 2024, doi: 10.4103/jmp.jmp_157_23.
R. Zaheer and H. Shaziya, "A Study of the Optimization Algorithms in Deep Learning," 2019 Third International Conference on Inventive Systems and Control (ICISC), Coimbatore, India, pp. 536-539, 2019, doi: 10.1109/ICISC44355.2019.9036442.
C. Peng, “Comprehensive analysis of the impact of learning rate and dropout rate on the performance of convolutional neural networks on the CIFAR-10 dataset,” Appl. Comput. Eng., vol. 102, pp. 183–192, 2024, doi: 10.54254/2755-2721/102/20241161.
I. A. Dewi and M. A. N. E. Salawangi, “High performance of optimizers in deep learning for cloth patterns detection,” IAES Int. J. Artif. Intell., vol. 12, no. 3, pp. 1407–1418, 2023, doi: 10.11591/ijai.v12.i3.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2017. [Online]. Available: https://arxiv.org/abs/1412.6980.
J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” J. Mach. Learn. Res., vol. 12, no. 61, pp. 2121–2159, 2011.
T. Tieleman and G. Hinton, “Lecture 6.5—RMSProp: Divide the learning rate by a running average of its recent magnitude,” Neural Netw. Mach. Learn., Coursera Lecture Notes, 2012.
Y. E. Nesterov, “A method for solving the convex programming problem with convergence rate O(1/k²),” Sov. Math. Dokl., vol. 27, pp. 372–376, 1983.
C. Yu, X. Qi, H. Ma, X. He, C. Wang, and Y. Zhao, “LLR: Learning learning rates by LSTM for training neural networks,” Neurocomputing, vol. 394, pp. 41–50, 2020, doi: 10.1016/j.neucom.2020.01.106.
J. J. Jeong and G. Koo, “AdaLo: Adaptive learning rate optimizer with loss for classification,” Inf. Sci., vol. 690, Art. no. 121607, 2025, doi: 10.1016/j.ins.2024.121607.
S. Hossain, J. Komol, and M. M. Raidah, “Mechanical tools classification dataset,” Kaggle Dataset, 2020.