In short:
- RAG makes an AI answer from your own documents instead of its general knowledge alone: it looks up the relevant passages the moment a question is asked, then writes its answer from them.
- The value is not in the answer, it is in the citation. A sourced answer can be checked in two clicks.
- It does not remove made-up answers. The one published study that measured commercial systems still found 17% to 33% of them.
- Two things decide the outcome, and neither one is technical: the state of your documents and the state of your permissions.
- Built on a model running in-house, nothing leaves the organisation. One Law 25 obligation disappears along with it.
"What exactly is our refund policy again?"
Someone asks on a Tuesday afternoon. The answer exists: it sits in a forty-page document approved in 2019 that three people have read end to end, and in an email from management last year that says something different. The person asking will find neither. She will ask a colleague, who will answer from memory, and that answer will circulate for the next two years.
That is the gap RAG fills. Not by making the AI smarter: by giving it access to what your organisation already knows.
The term, then the idea
RAG stands for retrieval-augmented generation. The technique was named in a 2020 research paper, and it has since become the standard way to connect an AI to documents it was never trained on. French-speaking readers will see it rendered as « génération augmentée par récupération ».
The principle fits in one sentence: before answering, the system searches your documents for the passages that touch on the question, then hands them to the model along with the question itself. The model no longer needs to know your refund policy. It needs to know how to read.
So the new part is not the model. It is the search step slipped in front of it. Everything hinges there.
Why not just paste everything into the chat
The question is a fair one now that models swallow entire documents without blinking. If your procedures manual fits in a single conversation, why bother with a search system at all?
Two reasons. The first is cost: you pay by volume of text, and re-pasting the whole manual on every question means paying a thousand times for the same page. The second is more troubling. A study that has become a reference in the field, Lost in the Middle, showed that models mostly retain what sits at the beginning and the end of what you give them, and that their performance drops when the useful information sits in the middle. The finding holds even for models built specifically for long texts.
Put plainly: the more you hand over at once, the higher the odds that the sentence that matters gets drowned. A system that returns only the right three paragraphs works better than one that returns everything.
The answer matters less than the source
Here is the part demonstrations sell badly, because it looks unremarkable. An assistant that answers "under the refund policy, section 4.2, the window is thirty days" with a link to the document is not the same thing as an assistant that answers "the window is thirty days".
In the first case, the employee clicks, reads the paragraph and decides. In the second, she has to take the machine's word for it. The whole difference between a working tool and a gadget sits there. Make it your first criterion when you evaluate a product.
No, it does not eliminate made-up answers
Several vendors sold RAG as the end of made-up answers. Legal publishers went as far as promising "hallucination-free" citations. A Stanford team tested the promise and published the results in 2025 in the Journal of Empirical Legal Studies: 202 preregistered legal questions, put to the leading legal research tools on the market.
| Tool tested | Accurate answers | Made-up answers |
|---|---|---|
| Lexis+ AI (LexisNexis) | 65% | 17% |
| Westlaw AI-Assisted Research (Thomson Reuters) | 42% | 33% |
The third tool tested, Ask Practical Law AI, either declines to answer or answers without grounding in more than 60% of cases. The researchers conclude that the vendors' claims are overstated: RAG reduces the problem compared with a general-purpose AI, it does not make it go away.
Keep the scale in mind. These systems are built by legal publishers, on corpora kept in order for decades, with development budgets larger than most Quebec SMBs' annual revenue. If they get it wrong one time in five, your in-house assistant will too. The lesson is not to give up: it is to build the process around verification rather than around trust.
The real work is your documents
A retrieval system does not arbitrate between two contradictory versions of the same policy. It gives you both, or worse, it gives you whichever one most resembles your question, which is not necessarily the one in force.
This is not a theory. In its own deployment guide, Microsoft recommends applying retention and deletion policies to remove inactive or obsolete files, specifically "to improve the quality of Copilot responses". The largest vendor on the market is telling its customers that document cleanup is part of the AI project.
For an SMB, that translates into a decidedly low-tech decision: pick a corpus you can vouch for. HR procedures, internal policies, the customer service knowledge base. A finite set, with someone accountable for it, from which dead versions have been pulled before indexing rather than after.
Who is allowed to see what
An assistant connected to your documents inherits your permissions. Microsoft says it plainly about Copilot: the tool answers from data the user already has permission to access. That is why the very first step of its deployment guide is titled "remediate oversharing", ahead of any configuration.
The mechanics are counter-intuitive. Leadership is better off understanding them before launching the project. The payroll file sitting in a folder shared too widely is one nobody finds today: you would have to know it exists and where to look. The day an assistant indexes that folder, an employee only has to ask "what does the sales team earn" for the answer to come back. The AI created no vulnerability. It made a three-year-old permission mistake exploitable in a single question.
In Quebec, the rule is written down. Section 20 of the Act respecting the protection of personal information in the private sector provides that personal information is accessible to an employee only "on condition that the information is needed for the performance of their duties". Reviewing access before indexing is not excessive caution, it is what keeps you on the right side of that sentence.
Cleaning up access before you connect anything takes a few hours to plan, and it pays off well beyond AI. We turned it into a full method.
The Law 25 angle
If your documents hold personal information and the model reading them belongs to a foreign vendor, your excerpts leave Quebec on every question. Section 17 of the same act is explicit: before such a communication, the business must conduct a privacy impact assessment, take into account the legal framework of the destination country among other factors, and put a written agreement in place.
A model running on your own server does not relieve you of your other obligations. It removes that one, because there is no longer a communication outside Quebec. That is the most tangible argument for local hosting: it can be counted in hours of legal work avoided.
One nuance worth keeping in mind: an assistant that informs is not an automated decision. The day you wire it into a real decision, turning down a request or screening applications, section 12.1 kicks in and you must inform the person concerned and be able to explain the factors behind the decision.
Default settings decide the quality
A common open-source setup runs on Open WebUI, the interface we covered in our article on Ollama. Its factory settings, readable in its source code, are worth pausing on, because they explain half the disappointments.
The system hands the model only three excerpts per question. A question whose answer is scattered across eight documents will therefore miss, and nobody will understand why. Keyword search is off by default: the system searches by meaning, which works well for "how many days for a refund" and very poorly for a contract number or a part reference.
These defaults are not design flaws. They are cautious, chosen so the thing starts on any machine. They simply are not the right ones for your organisation. Nobody will tell you that at install time.
What it does not solve
A document can carry instructions. A system that reads everything it is given also reads the text a third party slipped into an email or a file, and nothing in its eyes separates a legitimate instruction from a hostile one. This is not hypothetical: in June 2025 Microsoft published CVE-2025-32711, rated 9.3 out of 10, described as an AI command injection in M365 Copilot allowing an unauthorised attacker to disclose information. Microsoft states it fixed the issue on its side, with no action required from customers, and saw no exploitation. The principle stands: in a system that reads your documents, the content of those documents becomes an input to treat with suspicion.
Next, anything the system cannot read does not exist for it. Contracts scanned as images without character recognition, tables buried in presentations, decisions made in a chat thread: invisible.
Finally, the index reflects the state of your documents at the moment it was built. A policy changed yesterday is only known after the next refresh, and an employee handed a stale answer with a neat citation attached will be harder to talk out of it than one who got nothing at all.
What it actually costs
The software building blocks cost nothing in licensing. The specialised databases that store the index are open source: pgvector under the PostgreSQL licence, Qdrant and Chroma under Apache 2.0, Weaviate under BSD-3. The tool that indexes your text runs on your machine and bills nothing.
The spending sits elsewhere, in two places. A machine able to run a decent model, a subject we put numbers on in our article on local AI. And above all human time: sorting documents, pulling dead versions, reviewing access, testing answers. The software is free, the cleanup is not. The project turns on the second one.
The reassuring part is that none of that work is wasted. Sorted documentation and reviewed access serve the assistant, but they serve your teams first.
How we approach it
We always start with a single corpus, chosen because someone can answer for it. We measure before celebrating: ten questions whose answers we already know, put to the system, and we count the errors. When the result is poor, it is almost always the corpus that needs fixing, not the model that needs swapping.
We also require a citation in every answer, and we take the time to show people they need to click it. An assistant taken at its word is more dangerous than no assistant at all, because it lends an approximate answer the appearance of an official one.
The rest is a matter of sequence. Document cleanup and access review come before indexing, never after. It is less exciting than a demo, but it is what separates a project that lasts from one abandoned after six weeks.
Five steps for a first attempt worth running:
- Pick a finite corpus with one person accountable for it.
- Pull the obsolete versions before indexing.
- Check who has access to that corpus today, and fix it.
- Require a source in every answer.
- Ask ten questions you already know the answer to, and count the errors.
What to take away
RAG turns documentation nobody rereads into immediate answers, without handing your documents to a third party. It is one of the AI uses whose value shows up fastest in an SMB, provided you accept that the project is first and foremost a tidying-up project.
Anyone selling you the opposite, an assistant that would guess right out of a mess, is selling you the easy half. And if your employees are already using AI tools of their own with your documents, the subject is more pressing than you think: we wrote about that in our article on shadow AI.
Is your knowledge asleep in documents nobody rereads? Drop us a line and we will look together at what is worth making searchable.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020): the research paper that named the technique.
- TERMIUM Plus: the Government of Canada terminology record for the term and its French equivalent.
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023): the performance drop when the useful information sits in the middle of the supplied text.
- Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools: the Stanford study published in 2025 in the Journal of Empirical Legal Studies, 202 preregistered questions.
- Microsoft, Microsoft 365 Copilot deployment guide: oversharing to remediate as step one, and document retention as a factor in answer quality.
- Microsoft, CVE-2025-32711: the AI command injection in M365 Copilot fixed in June 2025, rated 9.3.
- Act respecting the protection of personal information in the private sector: section 17 on communication outside Quebec, section 20 on access needed for the performance of duties, and section 12.1 on automated decisions.
- Open WebUI, default configuration: the three excerpts handed to the model and keyword search left off, readable in the source code.
- pgvector: PostgreSQL licence, verified in the repository.
- Qdrant and Chroma: Apache 2.0 licence, verified in the repositories.
- Weaviate: BSD-3 licence, verified in the repository.