RAG in business: getting AI to answer from your own documents, without exposing them

"Can AI answer from our own documents?" This is the question we have heard most often in SMEs over the past year. Behind it lies a very concrete need: finding the right clause across 300 contracts, knowing which procedure applies to a client case, answering a tender with the right references. The technical answer is called RAG, for retrieval-augmented generation. Here is what it is, what it requires, and how we deploy it in SMEs.

What a RAG does, in one picture

A general-purpose AI assistant answers from what it learned during training. It knows nothing of your contracts, your procedures or your clients. RAG solves this without retraining anything: when you ask a question, the system first searches your documents for the most relevant passages, then passes them to the model along with your question, asking it to answer only from those passages, citing its sources.

The picture we use with our clients: the model is an excellent intern who has never set foot in your company. RAG is the colleague who brings them the three right files before they answer. The principle was formalised in 2020 by researchers at Facebook AI Research and has become, since 2023, the most common architecture for enterprise assistants.

Two practical consequences. Your documents are not used to train the model: they stay in your system and are consulted on demand. And the quality of the answers depends directly on the quality of what is retrieved, and therefore of your documents.

What it really requires

We would rather say it before than after: a RAG is not a tool you plug in. It is a data project, with four prerequisites.

• Identified, up-to-date documents. If three versions of a procedure coexist on the server, the assistant will sometimes cite the wrong one. The first job is always a sort: which documents are authoritative, where they live, who maintains them.

• Access rights respected. The assistant must not show an employee a document they are not allowed to read. This means connecting the RAG to existing permissions (SharePoint, Drive, document management system), not copying every file into a shared folder.

• Hosting consistent with your data. For personal data or trade secrets, where the search and the model run matters. In Switzerland, the data protection act requires you to know where data goes; we favour deployments in Switzerland or the EU, with a contract that excludes any retraining.

• A set of test questions. Thirty to fifty real questions, with expected answers, written by the business teams. This is what allows you to measure the rate of correct answers before opening to the teams, and then at every update.

A worked example: 4,200 documents, 82% correct answers, then 94%

An 80-person SME in French-speaking Switzerland, active in building services installation, managed 4,200 useful documents: supplier manuals, safety procedures, site reports, framework contracts. Technicians lost an average of 25 minutes a day searching for information, and called the technical office for the rest.

The initial sort brought the corpus down to 2,900 authoritative documents. The RAG was connected to the existing document management system, keeping its access rights, and tested on 50 questions written by the technical office. First result: 82% correct answers. Analysis of the remaining 18% showed that most came from unreadable scanned documents or duplicate manuals; after correcting the corpus, the rate rose to 94%.

After four months of use, measured search time fell from 25 to 8 minutes per day per technician, and calls to the technical office dropped by a third. Each answer cites its source document, which had an unexpected effect: technicians now report obsolete manuals, and the corpus keeps improving.

Where to start in an SME

We recommend starting with a single document scope, the one where search time costs the most: procedures, contracts, or customer support. A pilot takes six to eight weeks, with a test question set from the first week and a decision point at the end. If the correct-answer rate exceeds 85% on business questions, you widen the scope; if not, you fix the corpus before adding anything.

What not to do: index "the whole server" hoping the tool will sort it out. It will not, and users will lose trust at the first wrong answer.

What we take away

RAG is today the safest way to put AI to work on your internal knowledge, because it does not expose it and because it cites its sources. But its success is decided in the documents, not in the model: a sorted corpus, respected rights, a maintained test set. It is a data project before being an AI project, and that is good news: those foundations then serve everything else.

To see where your organisation stands on these topics, our AI Barometer gives a first reading in a few minutes. To scope a pilot, get in touch.