Legal Query RAG

Abstract

Recently, legal practice has seen a significant rise in the adoption of Artificial Intelligence (AI) for various core tasks. However, these technologies remain in their early stages and face challenges such as understanding complex legal reasoning, managing biased data, ensuring transparency, and avoiding misleading responses, commonly referred to as hallucinations. To address these limitations, this paper introduces Legal Query RAG (LQ-RAG), a novel Retrieval-Augmented Generation framework with a recursive feedback mechanism specifically designed to overcome the critical shortcomings of standard RAG implementations in legal applications. The proposed framework incorporates four key components: a custom audit agent, a specialized response generation model, a prompt engineering agent, and a fine-tuned legal embedding LLM. Together, these components effectively minimize hallucinations, improve domain-specific accuracy, and deliver precise, high-quality responses for complex queries. Experimental results demonstrate that the fine-tuned embedding LLM achieves a 13% improvement in Hit Rate and a 15% improvement in Mean Reciprocal Rank (MRR). Comparisons with general LLMs reveal a 24% performance gain when using the Hybrid Fine-Tuned Generative LLM (HFM), the specialized response generation model integrated into the LQ-RAG framework. Furthermore, LQ-RAG achieves a 23% improvement in relevance score over naive configurations and a 14% improvement over RAG with Fine-Tuned LLMs (FTM). These findings underscore the potential of domain-specific fine-tuned LLMs, combined with advanced RAG modules and feedback mechanisms, to significantly enhance the reliability and performance of AI in legal practice. The reliance of this study on a proprietary model as the audit agent, combined with the lack of feedback from human experts, highlights the need for improvement. Future efforts should focus on developing a specialized legal audit agent and enhancing its performance by incorporating feedback from domain experts. © 2025 Institute of Electrical and Electronics Engineers Inc.. All rights reserved.