Accurate answers
Answers are grounded in your own sources, not the model's general training. Fewer hallucinations, more answers you can actually use.
Guide
RAG — Retrieval-Augmented Generation
RAG is the technology that lets a language model answer questions from your own documents, with source citations you can verify. Here's how it works, and how PrivateGPT uses it in your own Azure environment.
What is RAG?
AI that answers from your own data — not just what it was trained on
Retrieval-Augmented Generation (RAG) is an architecture that combines two steps: finding the relevant material in your own sources, and generating an answer grounded in that material.
Instead of a language model answering only from what it was trained on, it's shown the most relevant excerpts from your documents, websites or SharePoint — and generates an answer using them as the basis. The model never sees your entire knowledge base, only what it needs for the question at hand.
That's why RAG stays accurate at scale, works on large document collections, and why PrivateGPT's RAG can run in your own Azure environment under your control.
How it works
Four steps from question to answer
Indexing
Your documents are split into smaller pieces (chunks) and turned into embeddings — mathematical vectors that represent meaning. They're stored in a search or vector index together with metadata used for filtering — e.g. source, document library, workspace, assistant scope and access-related attributes.
Retrieval
When a user asks a question, it's converted to the same vector format. The system finds the most relevant chunks — a semantic match, not just keywords — filtered by the scope and rights the user and assistant have access to.
Reranking
A second model reviews the candidates and reorders them by actual relevance to the question, not just vector similarity. This is where modern RAG differs from earlier generations.
Generation
The language model receives the question plus the top-ranked chunks as context and generates an answer grounded in that material — with citations back to the source documents.
Benefits
Why organisations use RAG
Source citations
Every answer points back to the material it's based on. Users can verify directly.
Updated without retraining
New knowledge can be used as soon as the document is indexed. No retraining of the model is required — but the actual sync and indexing frequency depends on your chosen setup.
Data stays with you
In PrivateGPT, RAG runs in your own Azure environment. Documents, index and access control stay under your control, and documents aren't used for model training — only the context needed per answer is used.
Respects access rights
Access filtering happens before material reaches the model. Users only see what they already have rights to.
Scales with your data
RAG makes it possible to work with large document libraries because the system only retrieves the most relevant excerpts per question. The actual scale depends on architecture, data sources, indexing and operations.
Modern RAG
What's changed in the last few years
The basic principle is still retrieval + generation. But the technique has evolved significantly — where RAG in 2023 was mostly about finding text excerpts that looked like the question, modern RAG does much more to hit the right answer.
Hybrid search
A combination of semantic vector search and classical keyword search (BM25). Semantics understand what the user means; keywords catch names, codes and exact terminology. Together they beat either method alone.
Reranking
A cross-encoder model reorders candidates by actual relevance, not just vector similarity. Catches nuances that semantic search misses.
Query rewriting
The user's question is reformulated before retrieval — expanded with synonyms, split into sub-questions, or enriched with context from the conversation history. Better input gives better results.
Access filtering at retrieval
Filtering happens before material is presented to the model, not in the response. Data the user can't see never reaches the language model.
Grounded answers with citations
Modern RAG returns not just an answer but also which excerpts back each claim. Users can follow the source in a single click.
Agentic / multi-hop
Complex questions aren't answered in one retrieval call. The system retrieves, reasons, retrieves again — until it has the material it needs.
When RAG fits
RAG is powerful, but not the answer to everything
RAG is a great fit when
- The task is knowledge-heavy and the answer lives in your documents
- The domain has knowledge that updates continuously — new policies, contracts, versions
- Source citations are required (compliance, legal, audit)
- Environments have differentiated access rights
- Chatbots, search and internal knowledge sharing across departments
Consider alternatives when
- Knowledge changes in real time — then the system should integrate directly with the source system, not just an indexed copy
- Answers need to be 100% deterministic — then a rules engine fits better than a language model
- The task requires deep domain expertise beyond what your documents cover — fine-tuning may be worth considering (usually as a supplement to RAG rather than a replacement)
RAG in PrivateGPT
How the platform uses it
PrivateGPT is built around RAG as one of the core platform features. It's available both to end users and as a foundation for your own assistants.
PrivateGPT can use both Azure AI Search and Qdrant as the search and vector layer, depending on your setup. Regardless of the search layer, PrivateGPT handles the RAG pipeline itself: document ingestion, chunking, embeddings, retrieval, filtering, source handling, assistant scope and access control.
An HR assistant, for example, can be scoped to HR policies and onboarding material, while a legal assistant works with contract templates and compliance documents.
- Semantic search across SharePoint, uploaded files, websites and internal knowledge sources
- Support for both Azure AI Search and Qdrant as the search / vector layer
- Indexing of large SharePoint document libraries, or selected folders, sites and filtered slices of content
- Hybrid search combining semantics and keyword/BM25 where relevant
- Reranking available in relevant workflows
- Filtering by user, assistant, workspace and data source
- Source citations on RAG-based answers where sources are available
- Assistants can be assigned specific RAG sources and scopes
- The index can be updated automatically or on a schedule, depending on the data source and setup
Ready to see it in practice?
In 30 minutes, we'll show RAG working on your own documents
Book a demo — we'll start from a couple of your real documents and show how RAG answers your own questions, with source citations, access control and everything that goes with it.