Enhanced Large Language Models-based Legal Query Responses through Retrieval Augmented Generation

DOI

10.2991/978-94-6463-512-6_27How to use a DOI?

Keywords

Retrieved Augmented Generation (RAG); AI Hallucination; Artificial Intelligence

Abstract

This study aims to solve the challenge of hallucination in Large Language Models (LLMs) through the implementation of Retrieval Augmented Generation (RAG). Especially the case of answering legal questions. Using Canada’s Criminal Code as dataset, the research first converts the code to the text file format. Then segmenting the text into vectors for a vector database, and with the help of an Embedding Model for vector transformation. Finally, the LLM will play the role of search engine and provide answer. As a result, the evaluation using the Automated Evaluation of Retrieval Augmented Generation (RAGAS) method demonstrated high faithfulness of 0.9444 and answer relevancy of 0.7488, indicating the model’s factual accuracy and relevance to the posed questions. Despite the complexity of the legal questions and the initial challenge of understanding indirect queries, the RAG model successfully provided accurate and relevant answers. However, context precision and context recall were relatively lower at 0.4 and 0.45, suggesting potential for improvement in the model’s ability and the evaluation model’s possible deficiency. The study successfully demonstrates the potential of RAG to mitigate hallucination issues in LLMs, particularly in the legal domain. The high faithfulness and relevancy scores affirm the model’s efficacy in providing accurate legal information, marking a significant advancement in the application of LLMs for legal queries. Future research could focus on enhancing context precision and recall, or create more precise database to evaluate with better evaluation methods.

Copyright

© 2024 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Ruiteng Li
PY  - 2024
DA  - 2024/09/23
TI  - Enhanced Large Language Models-based Legal Query Responses through Retrieval Augmented Generation
BT  - Proceedings of the 2024 International Conference on Artificial Intelligence and Communication (ICAIC 2024)
PB  - Atlantis Press
SP  - 237
EP  - 244
SN  - 1951-6851
UR  - https://doi.org/10.2991/978-94-6463-512-6_27
DO  - 10.2991/978-94-6463-512-6_27
ID  - Li2024
ER  -