What Is a RAG Chatbot and How It Answers From Your Documents

A RAG chatbot is a bot that first searches your own documents for the passages that fit a question, and only then writes a reply from what it found. So when a customer asks about returns, the answer should come from your returns policy. Not from whatever the model half-remembers about returns in general.

What is a RAG chatbot?

A RAG chatbot is an assistant that looks things up in a set of texts you give it, then has a language model turn what it found into a readable reply. Retrieval means looking something up. Generation means the model writing the answer. That’s the whole acronym, really. The original retrieval-augmented generation paper describes the technique as combining what a model learned in training with an external store of text that it searches whenever a question comes in.

If you run a business, the difference is a practical one. The reply is supposed to rest on your content (policies, product pages, FAQs) and not on whatever the model happened to soak up from the wider internet.

What happens between the question and the answer

The bot finds the passages that match the question, then writes a reply on top of them. A typical document chatbot goes through these steps:

1. The visitor types a question into the chat window.
2. The system works out what’s actually being asked. If you want the background, there’s a separate guide on how bots understand questions.
3. It searches the knowledge base for the passages closest in meaning to the question.
4. Those passages go to the language model along with the question.
5. The model writes a reply based on the passages it got.
6. The reply shows up in the chat.

You can build each step in a dozen different ways. The order doesn’t change, though: look it up first, write second.

An example conversation: from customer question to grounded answer

The example below is made up. Picture an imaginary online shop that has uploaded its returns policy and its shipping FAQ. No real customer involved.

Visitor: “Can I send back headphones I’ve already opened?”
Bot: “Yes. Opened items can be returned within the return period as long as they are complete and undamaged. Please include all original accessories.”

What happened behind the scenes? The search pulled out the returns paragraph about opened products. The model rephrased it and added nothing the policy doesn’t say. Boring, and that’s exactly the point.

Visitor: “Do you ship to Iceland?”
Bot: “I don’t have information about shipping to Iceland. Please contact the shop directly to confirm.”

Neither document mentions Iceland, so a properly set-up bot admits it doesn’t know instead of guessing. In my view this is the single most important behaviour to test. A bot that confidently invents shipping options is worse than no bot.

How a knowledge base chatbot finds the right passage

A knowledge base chatbot finds answers by matching the meaning of a question to the meaning of passages in your documents, not just by hunting for the same keywords. So “Can I get my money back?” should still land on a section called “Refunds”, even though the two don’t share a single word. Our guide to natural language processing basics explains how software handles meaning like this.

And here’s the catch. Answer quality depends on document quality. Outdated files give you outdated replies. Contradictory files give you inconsistent ones. Vague texts, vague answers. The search can only hand back what you fed it (garbage in, garbage out, same as ever).

RAG vs fine-tuning for business: which one keeps answers current?

Fine-tuning changes the model itself, while RAG leaves the model alone and hands it your documents every time a question comes in. That matters most when your information changes. With retrieval, updating what the bot knows means editing or swapping a file. No retraining.

Retrieval also makes it easier to trace an answer back to the passage it came from. The research behind the method points to two open problems for models that rely only on built-in knowledge: showing where an answer came from, and keeping that knowledge up to date. RAG goes after both. For more on these trade-offs, see our articles on AI in business.

Before you let a RAG chatbot talk to customers

Test the bot with real questions, against documents you’ve already cleaned up. Here’s a short checklist I’d go through:

• Delete old versions of the same policy so only one version is left.
• Check that the documents cover the questions customers actually ask, not the ones you wish they asked.
• Ask questions the documents can’t answer and make sure the bot doesn’t make things up.
• Read the replies side by side with the source text.
• Update the documents whenever prices or policies change.

Botino works this way. It answers from a company’s own content in a website widget you add with a single script, and the features page explains how uploaded files are used. As the list of common product questions answered states, the engine runs on OpenAI and other third-party services.

Whatever tool you pick, a RAG chatbot is only as reliable as the documents behind it. So keep those documents in order. That beats tweaking any setting.

FAQ