Loading...

Improving Robustness of Question Answering Systems Using Deep Neural Networks

Boreshban, Yasaman | 2023

4 Viewed
  1. Type of Document: Ph.D. Dissertation
  2. Language: Farsi
  3. Document No: 56801 (19)
  4. University: Sharif University of Technology
  5. Department: Computer Engineering
  6. Advisor(s): Ghassem Sani, Gholamreza; Mirroshandel, Abolghasem
  7. Abstract:
  8. Question Answering (QA) systems have reached human-level accuracy; however, these systems are vulnerable to adversarial examples. Recently, adversarial attacks have been widely investigated in text classification. However, there have been few research efforts on this topic in QA systems. In this thesis our approach is improving the robustness of QA systems using deep neural networks. In this thesis, as the first proposed approach, the knowledge distillation method is introduced to create a student model to improve the robustness of QA systems. In this regard, the pre-trained BERT model was used as a teacher, and its impact on the robustness of the student models on the Adversarial SQuAD dataset was assessed. Our experiments show that by using KD, both criteria F1-score and EM of the student models by around 5.0\% increased when tested on the AddSent and AddOneSent adversarial datasets. Before this, knowledge distillation was not used to improve the robustness of QA systems. In order to more accurately evaluate the effect of various attacks on the robustness of QA systems, in the second approach, we modified the attack algorithms widely used in text classification to fit those algorithms for QA systems. We developed a new framework, RobustQA, as the first open-source toolkit for investigating textual adversarial attacks in QA systems. In this framework, in addition to the six existing algorithms, it is possible to develop new attack algorithms in QA systems easily. Since adversarial training is one of the common methods of improving the robustness of deep learning models, it is possible to use this approach with different attack algorithms in this framework. In the third proposed approach, we have presented a new attack algorithm using the evolutionary algorithm of harmony search, which has yielded promising results compared to two evolutionary algorithms, Genetic and PSO. We also investigated the robustness of the models against various attacks by using adversarial training through adversary sentences generated by the harmony search algorithm. In our fourth proposed approach, we used quantization to improve the robustness of NLP models, especially on QA and classification Tasks. Quantization involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model. In our experiments, we evaluated the impact of quantization on the BERT model in the QA system using the SQuAD dataset against TextFooler and PSO adversarial attacks. The results showed that by applying quantization, the robustness of the model against the Textfooler and PSO attacks increases both the F1-score and EM by around 20.0 \%. Furthermore, we evaluated the impact of quantization in the text classification using SST-2, Emotion, and MR datasets against TextFooler, PWWS, and PSO adversarial attacks. Our findings showed that quantization significantly improves by an average of 18.68\% the adversarial accuracy of the models
  9. Keywords:
  10. Question Answering ; Robustness ; Adversarial Example ; Knowledge Distillation ; Adversarial Training ; Quantification ; Deep Neural Networks

 Digital Object List

 Bookmark

...see more