Design and Evaluation of an NLP-Based Corporate Chatbot with a Hybrid Knowledge Architecture and Local LLMs

When your knowledge provides answers: In this work, we have designed a modular RAG system that generates its knowledge from specific documents. We evaluated how well the architecture can handle new documents from a completely different domain. Key features include the fact that no prompt engineering was used and that only local LLMs were employed.

Hybrid RAG in action: your knowledge provides the answer

Initial situation

Companies, organisations and smaller communities such as teams, clubs or families possess extensive, often highly specialised knowledge. In practice, however, this knowledge is scattered across documents, notes and the experiences of individual people, which makes it difficult to locate and ties it to specific individuals. This project is therefore developing a RAG system that lowers the barriers to knowledge sharing: modular enough for organisational environments such as companies, yet simple enough to serve as an easily accessible concept, so that even people with limited IT skills can use our development.

Problem statement / Project goal

Across all of these settings, knowledge is spread over many documents and systems, making reliable information hard to retrieve. Conventional language models may return plausible but unsubstantiated answers and can expose sensitive data through external services. This project therefore designs and evaluates a modular RAG chatbot that retrieves relevant information from a hybrid knowledge architecture and generates well-founded answers using local LLMs. The goal is to measure how effectively the same architecture transfers to documents from a completely new subject area, without resorting to prompt engineering, while keeping the answers transparent, traceable and under local control.

Solution developed and its benefits

In this thesis a modular, locally operated RAG chatbot with a end-to-end pipeline that transforms specific documents into different searchable knowledge bases is developed. For each question, the system combines semantic vector search, document retrieval, and a knowledge graph to identify relevant information. A local large language model then generates an answer based on the retrieved evidence.

The solution provides several benefits:

• grounded answers based on (company-)specific documents.
• transparent references to the underlying sources.
• local processing of sensitive data.
• no dependency on external AI services.
• support for relationships across multiple documents.
• modular components that can be replaced or extended.
• transfer to new knowledge domains without prompt engineering.
• systematic evaluation of retrieval quality, correctness, and faithfulness.
• avoiding hallucinations because no general knowledge is used.

This architecture turns distributed information into an accessible conversational knowledge system while retaining control over data, models, and infrastructure.

Key terms

• ChromaChroma is a vector database that can be used to store and query vectors.
• DoclingDocling is a PDF reader that can be used to ingest PDFs into the RAG system.
• GraphRAGA graph-based approach to retrieve and store knowledge from various sources.
• Hybrid Knowledge ArchitectureA combination of distinct knowledge bases within one RAG system.
• Local LLMsLarge Language Models hosted locally.
• Neo4jNeo4j is a graph database that can be used to store and query graphs.
• OntologyAn ontology is a set of concepts and relationships that define a domain.
• Prompt engineeringPrompt engineering is the process of designing the input prompt for a language model to achieve a desired output.
• PyMuPDFPyMuPDF is a PDF reader that can be used to ingest PDFs into the RAG system.
• QdrantQdrant is a vector database that can be used to store and query vectors.
• RAGRetrieval-Augmented Generation is a technique that combines a large language model with a knowledge base to answer questions.
• RAGASRAGAS is a benchmark for RAG systems that evaluates answer quality, correctness, and faithfulness.

Team