Ask a language model like ChatGPT about your company's leave policy, product terms or internal procedures, and it will either say "I don't know" or, worse, invent a convincing but wrong answer, because it has never seen those documents.
RAG (Retrieval-Augmented Generation) solves this problem and is the most widely used architecture in enterprise GenAI applications.
How does RAG work?
The idea is simple: before asking the model a question, we find the document passages that are likely to contain the answer and give them to the model along with the question. The model then bases its answer on those passages.
There are two stages:
Preparation (once, and whenever documents change):
- Collect the documents: PDFs, Word files, wiki pages, procedures
- Split them into meaningful chunks
- Turn each chunk into a numerical vector that represents its meaning
- Store the vectors in a vector database
Question and answer (for every question):
- Turn the user's question into a vector
- Find the document chunks closest in meaning to the question
- Pass those chunks to the language model along with the question
- The model produces an answer based on the chunks, with citations
RAG or fine-tuning?
| RAG | Fine-tuning | |
|---|---|---|
| What it teaches | Knowledge (document content) | Behaviour and style |
| Updating | Just add documents | Requires retraining |
| Citing sources | Possible | Difficult |
| Access control | Can be filtered per user | Baked into the model, can't be filtered |
| Cost | Relatively low | Higher |
For most enterprise knowledge-access scenarios, RAG is the right starting point. Fine-tuning should be seen as a complement to RAG for a particular style or specialised task.
Enterprise use cases
- Employee support assistant: HR policies, IT procedures, internal regulations
- Customer service assistant: product terms, FAQs, process steps
- Regulatory and compliance assistant: regulations, internal policies
- Document intelligence: question answering and comparison over contracts, reports and files
What to watch when building it
Document quality
RAG is only as good as your documents. Outdated, contradictory or poorly scanned documents lead to wrong answers.
Chunking strategy
How documents are split directly affects quality. Chunks that are too small lose context; chunks that are too large carry irrelevant information. Tables, heading structure and clause numbers should be preserved.
Retrieval quality
Semantic search alone isn't always enough. For product codes, clause numbers or proper names, hybrid search that combines semantic and keyword search usually performs better.
Access permissions
Not every employee should see every document. A RAG system must answer only from documents the user is authorised to see. This is one of the most overlooked yet most critical issues in enterprise RAG.
Freshness
When documents change, the vector database must be updated too; otherwise the assistant answers with outdated information.
How do you measure quality?
A RAG system shouldn't go live just because it "looks good". Measure:
- Retrieval quality: are the right chunks being found?
- Groundedness: is the answer really based on the retrieved chunks, or is the model adding things?
- Answer accuracy: comparison against a question-and-answer set prepared by experts
- Citation accuracy: do the cited sources actually support the answer?
That requires building an evaluation set and testing the system against it after every change.
RAG on-premise
For organisations working with sensitive documents, every component of a RAG system (the language model, the vector database and document processing) can run entirely on-premise. This is a major advantage in industries such as banking and the public sector that can't move data off-site.
Conclusion
RAG is the most practical and reliable way to bring language models together with your organisation's knowledge. The keys to success are less about the model and more about document quality, retrieval strategy, access control and systematic evaluation.
Through our Generative AI & LLM Solutions service we build assistants over your corporate documents, in the cloud or on-premise. To build this capability in your technical teams, see our Building LLM Applications programme.
Sources