|
This article is cited in 3 scientific papers (total in 3 papers)
NUMERICAL METHODS AND DATA ANALYSIS
A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
R. I. Abdulkadirova, P. A. Lyakhovb a North-Caucasus Center for Mathematical Research, North-Caucasus Federal University, Stavropol
b North-Caucasus Federal University
Abstract:
In this paper, we propose a natural gradient descent algorithm with momentum based on Dirichlet distributions to speed up the training of neural networks. This approach takes into account not only the direction of the gradients, but also the convexity of the minimized function, which significantly accelerates the process of searching for the extremes. Calculations of natural gradients based on Dirichlet distributions are presented, with the proposed approach introduced into an error backpropagation scheme. The results of image recognition and time series forecasting during the experiments show that the proposed approach gives higher accuracy and does not require a large number of iterations to minimize loss functions compared to the methods of stochastic gradient descent, adaptive moment estimation and adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization
Keywords:
pattern recognition, machine learning, optimization, Dirichlet distributions, natural gradient descent
Received: 07.04.2022 Accepted: 24.08.2022
Citation:
R. I. Abdulkadirov, P. A. Lyakhov, “A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions”, Computer Optics, 47:1 (2023), 160–169
Linking options:
https://www.mathnet.ru/eng/co1113 https://www.mathnet.ru/eng/co/v47/i1/p160
|
Statistics & downloads: |
Abstract page: | 31 | Full-text PDF : | 7 | References: | 10 |
|