Master's thesis

Verification of Outputs from Retrieval-Augmented Generation (RAG) Systems in Terms of Factual Accuracy

Bc. Matej Babej
Abstract

Táto diplomová práca sa zameriava na problém overovania faktickej správnosti výstupov generovaných systémami Retrieval-Augmented Generation (RAG). RAG kombinuje silné stránky veľkých jazykových modelov (LLM) a externých zdrojov znalostí, aj napriek tomu môžu výstupy obsahovať faktické chyby, známe ako halucinácie. Teoretická časť práce sa venuje architektúre RAG systémov, príčinám vzniku halucinácií …more

Abstract

This thesis focuses on the problem of verifying the factual accuracy of outputs generated by Retrieval-Augmented Generation (RAG) systems. RAG combines the strengths of large language models (LLMs) and external knowledge sources, but the outputs may still contain factual errors, known as hallucinations. The theoretical part of the thesis reviews the architecture of RAG systems, sources of hallucinations …more

Thesis description
This thesis explores the verification of factual accuracy in outputs generated by Retrieval-Augmented Generation (RAG) systems. It begins by introducing Large Language Models (LLMs), outlining their historical development, fundamental principles, and the unique characteristics that distinguish RAG systems from standalone LLMs, particularly regarding their use of external databases. The thesis addresses the challenges of evaluating the truthfulness of generated statements, delving into logical frameworks, definitions of truth and falsehood, and current methods and approaches in fact-checking. The issue of "hallucinations" in LLM outputs is critically examined, including how dataset quality influences the factual integrity of generated responses.

The primary objective is to analyze and develop methods for verifying the outputs of RAG systems. The research includes an evaluation of existing fact-checking algorithms, knowledge graphs, statistical and heuristic techniques, and explores the potential of semantic analysis and other LLMs in the verification process. It contrasts manual and automated verification methods, assesses hybrid approaches that combine human oversight with machine-based techniques, and discusses their effectiveness, scalability, and robustness. The thesis also involves implementing proposed verification methods, detailing the underlying technologies and algorithms, and evaluating their performance against predefined objectives.

Expected outcomes include a practical implementation of the developed verification methods and a thorough analysis of the results. The thesis will conclude with recommendations for improving RAG system reliability and propose potential avenues for future research in the verification of AI-generated outputs.
The thesis has been checked:
29/5/2025 09:08, RNDr. Marek Kumpošt, Ph.D., UČO 44545
Language used
English English
Defence date
17/6/2025
The thesis was defended successfully

Supervisor

RNDr. Marek Kumpošt, Ph.D., UČO 44545
KPSK FI MU

Reader

Mgr. Karol Kubanda, UČO 143339
abs FI MU

  • Přidání souboru

    Soubor nebo složku lze nahrát pomocí tlačítka Přidat.
  • Další operace se soubory

    Podrobnosti lze zjistit označením příslušného řádku.
  • Pohled pro experty

    Pro častou práci je možné zvolit režim Více možností.
  • Vyhledávání souborů

    Vyhledávaný výraz můžete zadat přímo do adresního řádku.
  • Rychlý přístup k souborům

    Pomocí funkce Nedávné je možné se rychle vrátit k právě prohlíženým souborům. Oblíbené soubory je také možné označit Hvězdičkou.