Computer Optics
RUS  ENG    JOURNALS   PEOPLE   ORGANISATIONS   CONFERENCES   SEMINARS   VIDEO LIBRARY   PACKAGE AMSBIB  
General information
Latest issue
Archive

Search papers
Search references

RSS
Latest issue
Current issues
Archive issues
What is RSS



Computer Optics:
Year:
Volume:
Issue:
Page:
Find






Personal entry:
Login:
Password:
Save password
Enter
Forgotten password?
Register


Computer Optics, 2020, Volume 44, Issue 2, Pages 282–289
DOI: https://doi.org/10.18287/2412-6179-CO-645
(Mi co791)
 

This article is cited in 2 scientific papers (total in 2 papers)

NUMERICAL METHODS AND DATA ANALYSIS

Reducing computational costs in deep learning on almost linearly separable training data

I. M. Kulikovskikhabc

a Samara National Research University, 443086, Russia, Samara, Moskovskoe Shosse 34
b Rudjer Boskovic Institute, 10000, Croatia, Zagreb, Bijenicka cesta 54
c Faculty of Electrical Engineering and Computing, University of Zagreb, 10000, Croatia, Zagreb, Unska 3
References:
Abstract: Previous research in deep learning indicates that iterations of the gradient descent, over separable data converge toward the L2 maximum margin solution. Even in the absence of explicit regularization, the decision boundary still changes even if the classification error on training is equal to zero. This feature of the so-called “implicit regularization” allows gradient methods to use more aggressive learning rates that result in substantial computational savings. However, even if the gradient descent method generalizes well, going toward the optimal solution, the rate of convergence to this solution is much slower than the rate of convergence of a loss function itself with a fixed step size. The present study puts forward the generalized logistic loss function that involves the optimization of hyperparameters, which results in a faster convergence rate while keeping the same regret bound as the gradient descent method. The results of computational experiments on MNIST and Fashion MNIST benchmark datasets for image classification proved the viability of the proposed approach to reducing computational costs and outlined directions for future research.
Keywords: implicit regularization, gradient method, convergence rate, linear separability, image classification.
Funding agency Grant number
Grant of the President of the Russian Federation MK-6218.2018.9
Ministry of Science and Higher Education of the Russian Federation 074-U01
Russian Foundation for Basic Research 18-37-00219 мол_а
Европейский фонд регионального развития KK.01.1.1.01.0009
This work was supported by the Russian Federation President's grant (Project No. MK-6218.2018.9), the Ministry of Education and Science of the Russian Federation (Project No. 074-U01), RFBR (Project No. 18-37-00219), and the Centre of Excellence project “DATACROSS”, co-financed by the Croatian Government and the European Union through the European Regional Development Fund - the Competitiveness and Cohesion Operational Programme (KK.01.1.1.01.0009).
Received: 13.10.2019
Accepted: 13.12.2019
Document Type: Article
Language: Russian
Citation: I. M. Kulikovskikh, “Reducing computational costs in deep learning on almost linearly separable training data”, Computer Optics, 44:2 (2020), 282–289
Citation in format AMSBIB
\Bibitem{Kul20}
\by I.~M.~Kulikovskikh
\paper Reducing computational costs in deep learning on almost linearly separable training data
\jour Computer Optics
\yr 2020
\vol 44
\issue 2
\pages 282--289
\mathnet{http://mi.mathnet.ru/co791}
\crossref{https://doi.org/10.18287/2412-6179-CO-645}
Linking options:
  • https://www.mathnet.ru/eng/co791
  • https://www.mathnet.ru/eng/co/v44/i2/p282
  • This publication is cited in the following 2 articles:
    Citing articles in Google Scholar: Russian citations, English citations
    Related articles in Google Scholar: Russian articles, English articles
    Computer Optics
    Statistics & downloads:
    Abstract page:141
    Full-text PDF :39
    References:16
     
      Contact us:
     Terms of Use  Registration to the website  Logotypes © Steklov Mathematical Institute RAS, 2024