Loading...

Training Compressed DNNs for Resisting Against Adversarial Attacks

Mohseni Sangtabi, Saman | 2023

109 Viewed
  1. Type of Document: M.Sc. Thesis
  2. Language: Farsi
  3. Document No: 56356 (19)
  4. University: Sharif University of Technology
  5. Department: Computer Engineering
  6. Advisor(s): Sarbazi Azad, Hamid
  7. Abstract:
  8. Deep Neural Network (DNN) compression is a highly effective technique for reducing the computational burden and energy consumption associated with neural network inference, which is particularly important for low-power, embedded, and real-time systems. Weight pruning and quantization are among the most effective methods for neural network compression. Nonetheless, DNN compression poses various challenges, such as preserving network accuracy, particularly when dealing with adversarial attacks. Network compression can also lead to irregularities in the network structure and imbalanced distribution of workloads, which in turn can result in reduced utilization from the potential compression gains in general-purpose and specialized hardware. In this research, we propose a methodology for compressing neural networks, consisting of weight pruning and quantization, during the learning process. Our approach achieves significant compression rates while preserving the network’s accuracy and robustness against adversarial attacks. Additionally, our compression technique is tailored to the requirements and limitations of the target hardware, ensuring optimal network utilization. Our approach involves using constraint-based optimization, aided by an intensity-variable dynamic regularizer, along with the gradual imposition of constraints to ensure convergence. We utilize objective-aware criteria instead of conventional heuristic measures for network pruning and quantization. Furthermore, by periodically reducing a portion of the constraints, we create the opportunity for the network to better adapt to the network’s dynamicity. We provide the capability for efficient execution on a wide range of hardware platforms by supporting various forms of pruning, including unstructured, structured, pattern-based, and workload-balanced pruning, as well as weight quantization using the weight clustering approach. When compared to the state-of-the-art methods, our experimental results show an achievement of 2.4 times more compression with a minor accuracy drop of 0.6% and 198 times more compression with an accuracy drop of 3.4% for the VGG-16 network and the CIFAR-10 dataset. Furthermore, for the MobileNet-V2 network and the CIFAR10 dataset, our method achieves 5 times more compression, with only a 0.4% drop in accuracy. In the experiments related to adversarial accuracy, with an equal compression ratio, we have obtained an adversarial accuracy improvement of over 8.5% for the MobileNet-V2 network and the CIFAR-10 dataset
  9. Keywords:
  10. Deep Neural Networks ; Pruning Method ; Quantization ; Adversarial Attacks ; Compressed Neural Network ; Pruning Machine Learning Models

 Digital Object List

 Bookmark

...see more