Loading...

Non-Smooth regularization: improvement to learning framework through extrapolation

Amini, S ; Sharif University of Technology | 2022

23 Viewed
  1. Type of Document: Article
  2. DOI: 10.1109/TSP.2022.3154969
  3. Publisher: Institute of Electrical and Electronics Engineers Inc , 2022
  4. Abstract:
  5. Deep learning architectures employ various regularization terms to handle different types of priors. Non-smooth regularization terms have shown promising performance in the deep learning architectures and a learning framework has recently been proposed to train autoencoders with such regularization terms. While this framework efficiently manages the non-smooth term during training through proximal operators, it is limited to autoencoders and suffers from low convergence speed due to several optimization sub-problems that must be solved in a row. In this paper, we address these issues by extending the framework to general feed-forward neural networks and introducing variable extrapolation which can dramatically increase the convergence speed in each sub-problem. We show that the proposed update rules converge to a critical point of the objective function under mild conditions. To compare the resulting framework with the previously proposed one, we consider the problem of training sparse autoencoders and robustifying deep neural architectures against both targeted and untargeted attacks. Simulations show superior performance in both convergence speed and final objective function value. © 1991-2012 IEEE
  6. Keywords:
  7. Deep neural networks ; Training ; Extrapolation ; Network architecture ; Personnel training ; Perturbation techniques ; Signal to noise ratio ; Auto encoders ; Convergence ; Neural-networks ; Perturbation method ; Proximal operator ; Regularization terms ; Regularizer ; Robustness ; Sparse matrices ; Sparsity
  8. Source: IEEE Transactions on Signal Processing ; Volume 70 , 2022 , Pages 1213-1223 ; 1053587X (ISSN)
  9. URL: https://ieeexplore.ieee.org/document/9722959