Loading...

SLPL SHROOM at SemEval2024 Task 06: a comprehensive study on models ability to detect hallucination

Fallah, P ; Sharif University of Technology | 2024

0 Viewed
  1. Type of Document: Article
  2. DOI: 10.18653/v1/2024.semeval-1.167
  3. Publisher: ACL Anthology , 2024
  4. Abstract:
  5. Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge or the source text. This study explores methods for detecting hallucinations in three SemEval-2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation. We evaluate two methods: semantic similarity between the generated text and factual references, and an ensemble of language models that judge each other's outputs. Our results show that semantic similarity achieves moderate accuracy and correlation scores in trial data, while the ensemble method offers insights into the complexities of hallucination detection but falls short of expectations. This work highlights the challenges of hallucination detection and underscores the need for further research in this critical area. © 2024 Association for Computational Linguistics
  6. Keywords:
  7. Computational linguistics ; Computer aided language translation ; Machine translation ; Modeling languages ; Ensemble methods ; Factual knowledge ; Generative model ; Language model ; Machine translations
  8. Source: SemEval 2024 - 18th International Workshop on Semantic Evaluation, Proceedings of the Workshop ; 2024 , Pages 1148-1154 ; 979-889176107-0 (ISBN)
  9. URL: https://https-www--scopus--com.ip.ez.smnta.ir/inward/record.uri?eid=2-s2.0-85215514533&partnerID=40&md5=4529089ecfcb8afd27bc3d60ad81717b