To build an AI assistant that answers questions from your company documents, you don't retrain an AI model. You use an approach called retrieval-augmented generation, or RAG. When someone asks a question, the system searches your documents for the most relevant passages, hands them to a large language model such as GPT or Claude, and asks it to answer using only those passages, with references back to the source. The model supplies the language skills. Your documents supply the facts. Most of the real work is in preparing the documents, respecting who can see what, and testing that the answers are right.
RAG in plain English
Think of a capable new employee who has never worked at your company. Ask them about your leave policy and they'll guess from what's normal elsewhere. Hand them the right page of the staff handbook first and they'll give you a precise answer. RAG does the second thing automatically, every time.
It has two halves:
- Retrieval: finding the few passages, out of thousands of pages, that relate to the question.
- Generation: having the language model write a clear answer from those passages, citing where each point came from.
Retrieval usually relies on embeddings, which are numerical fingerprints of meaning. Each chunk of your documents gets one, stored in a vector database. The question gets one too, and the system finds chunks with similar meaning, even when the wording differs. "How many days off do I get?" can match a section titled "Annual leave entitlement." Many systems combine this with ordinary keyword search, which is better at exact terms such as product codes, policy numbers and names.
RAG or fine-tuning?
Fine-tuning means further training a model on your own examples. People often assume that's how you "teach the AI your documents." For most business assistants it isn't the right tool.
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Answering from facts in your documents | Changing style, format or behaviour on a narrow task |
| Keeping current | Update a document and the next answer uses it | Needs retraining when facts change |
| Showing sources | Can cite the exact passage | Can't reliably say where an answer came from |
| Permissions | Can filter results per user | Whatever the model learned, every user can reach |
| Effort to start | Moderate, mostly data preparation | Needs many good training examples |
Use RAG when the question is "what do our documents say?" Consider fine-tuning only when you need the model to behave differently, for example to always write in a particular format, and even then it's often combined with RAG rather than replacing it.
Preparing your documents
This is where most projects succeed or fail. An assistant can only be as good as what it retrieves.
- Find the sources. Shared drives, intranet pages, wikis, PDFs, help desk articles, contracts. Decide what's in scope for the first version and leave the rest out.
- Remove the duplicates and old versions. If three versions of the travel policy exist, the assistant may quote the wrong one. Pick the current version and archive the others.
- Extract text properly. Scanned PDFs need OCR. Tables need care, because a table flattened into a line of text loses its meaning.
- Split into sensible chunks. Break documents at headings or sections rather than every few hundred words, so each chunk makes sense on its own. Keep the document title and section heading attached to each chunk.
- Add metadata. Department, document type, date, owner and who is allowed to see it. Metadata makes filtering, permissions and updates possible.
Expect to find contradictions between documents along the way. Fixing those is a side benefit of the project, and a necessary one.
Permissions: who can see what
An assistant must never become a back door to documents a person couldn't open themselves. If HR files, salary data or board papers are in the index, the assistant could surface them to anyone who asks the right question.
The safe approach is to store each chunk's access rules alongside it and filter retrieval by the signed-in user's permissions before anything reaches the language model. Ideally those rules come from the system that already holds the documents, so permissions stay in one place. Connect the assistant to your existing sign-in, so it always knows who is asking. For a first version, many teams start with documents everyone in the company can already see, then add restricted collections once the permission model is tested.
Keeping answers current
A RAG assistant is only as current as its index. Set up a sync that picks up new, changed and deleted documents on a schedule, or when they change if the source system allows it. Deleted documents matter as much as new ones: a withdrawn policy left in the index will keep being quoted.
Show the date of the source next to each answer. It helps users judge whether to trust it, and it quickly exposes stale content.
Controlling hallucinations
Language models can produce fluent, confident answers that are wrong. RAG reduces this a great deal, but doesn't remove it. These controls help:
- Instruct the model to answer only from the retrieved passages, and to say clearly when they don't contain the answer.
- Require citations for each claim, linking back to the source document and section.
- If retrieval finds nothing relevant enough, don't call the model at all. Say "I couldn't find this in our documents" and suggest who to ask.
- Keep the assistant's scope narrow at first. An HR policy assistant is easier to get right than one that answers anything about the company.
- Let users flag bad answers with one click, and review those flags.
Evaluating it before and after launch
Don't judge an assistant by trying a few questions and feeling impressed. Build a test set instead:
- Collect 50 to 100 real questions people in your company ask, from email, chat or help desk tickets.
- For each, write down the correct answer and the document it comes from.
- Run the assistant against the set and check two things separately: did retrieval find the right passage, and did the answer use it correctly?
- Include questions the documents can't answer, to check the assistant admits it.
- Rerun the set after every change to documents, chunking, prompts or models.
Separating retrieval from generation is the key habit. When an answer is wrong, it's usually because the right passage was never found, and no amount of prompt tweaking fixes that.
Hosting and privacy options
There are three broad setups, and the right one depends on how sensitive your documents are:
- Cloud AI APIs: your app sends the retrieved passages to a model provider's API. Quickest to build and often the best answer quality. Check the provider's terms on data retention and whether your data may be used for training, and where it's processed.
- Models through your cloud provider: the major cloud platforms offer language models inside your own cloud account and region, which can help with data residency requirements.
- Self-hosted open models: open-weight models running on your own servers. Data never leaves your infrastructure, but you need suitable hardware and someone to run it, and answer quality may trail the largest hosted models.
The vector database and document store can sit in your own environment in all three setups. For organisations in Saudi Arabia and the UAE, data residency rules may influence the choice, so check what applies to your sector. More on local considerations on our UAE page.
Where to start
Pick one well-defined set of documents and one group of users with a real, frequent need, such as internal policies for staff or product manuals for your support team. Clean those documents, build the test set, and launch to a small group. Widen the scope once the answers hold up.
A good assistant also needs a good interface: clear citations, easy feedback and sensible behaviour on mobile. That's where product and UX design earns its place. To discuss your documents and users, see our AI solutions work or contact us for a quote.
Questions people ask
Does RAG train the AI on my company's data?
No. The model itself is not changed. Your documents are searched at question time, and only the relevant passages are sent to the model to answer from. Whether the provider keeps or uses that data depends on its terms, so check them.
What kinds of documents can an AI assistant use?
Most text-based content: PDFs, Word files, web pages, wikis, help desk articles and spreadsheets. Scanned documents need OCR first, and tables and images need extra care to keep their meaning.
How do we stop the assistant giving wrong answers?
Limit it to answering from retrieved passages, require citations, have it say when it can't find an answer, and test it against a set of real questions with known answers before and after every change.
Can staff only see answers from documents they have access to?
Yes, if it's designed that way. Each document's access rules are stored with it, and search results are filtered by the signed-in user's permissions before anything reaches the model.