Testing a Bot Before It Meets Customers

Most chatbot rollouts that fail were never really tested. Someone typed a dozen friendly questions into the admin panel, got sensible answers back, and flipped the switch. Two weeks later the queue is full of angry customers who asked something slightly off-centre and got a confident, wrong reply. Pre-launch testing isn’t a formality you tick off before launch. It’s the work that decides whether the launch holds.

Start With Real Tickets, Not Imagined Ones

Open your support inbox. Pull the last few months of email, chat transcripts and phone notes. Your question set comes from what customers actually wrote, not from what the team thinks they write. That gap is where most bots quietly break.

Group everything by intent, not by wording. Twenty phrasings of the same shipping question are one test case with twenty variants, and treating them as twenty separate cases just inflates your sheet without adding coverage. Keep the original spelling, slang, half sentences and typos exactly as they arrived. Tidy them up and you get a polished test set that the bot passes and reality fails.

A small team can do this by hand in a spreadsheet in an afternoon. Bigger operations should sample deliberately by ticket category, so no queue - returns, billing, technical, pre-sales - goes untested because it happened to be quiet that month. There’s a practical bonus here: build your knowledge base from the same ticket archive and every uploaded document already maps to a question somebody actually asked.

Build the Question Set: Three Layers

A usable test set has three layers. Skipping the second and third is why so many bots look ready and aren’t. Layer one holds the core intents the bot must answer correctly every single time, drawn from your highest-volume tickets. These are the questions that justify the deployment in the first place.

Layer two is edge phrasing: the same intent asked rudely, compressed into one word, typed in a second language, or buried three paragraphs deep inside a long complaint. Nobody writes clean queries when they’re annoyed. Layer three collects out-of-scope questions the bot should politely decline rather than improvise around.

Freeze the set before you start testing. Keep editing questions while you measure and results stop being comparable between versions, which costs you the only signal that tells you whether a change helped. New cases go into a separate batch, merged in later. For multilingual deployments, have the set translated by someone who genuinely answers customers in that language. Machine-translated test questions test your translator, not your bot.

The Wrong-But-Plausible Questions Are Where Bots Break

Gibberish is harmless. Every bot handles nonsense acceptably, because nonsense trips obvious uncertainty. The dangerous input is a perfectly reasonable question about something adjacent that your knowledge base simply doesn’t cover - close enough to your documents that the model reaches for an answer, far enough that it invents one.

Build questions on a false premise and watch what happens. Ask about a loyalty discount that never existed. Ask about warranty terms for a product you discontinued last year. A good bot corrects the premise; a weak one agrees with it and hands the customer a promise you now have to honour. Test compound questions too, where two intents arrive in one sentence and only one of them is answerable.

Then test the cases where the correct answer is a refusal: legal advice, medical claims, guarantees about refunds nobody authorised. A bot that says it doesn’t know and points to the right contact is behaving exactly as intended, and your scoring sheet has to record that as a pass. When something fails, tighten scope and tone in the admin panel configuration and rerun the identical set. Rewriting the questions until they pass isn’t testing. It’s decoration.

Who Should Test and Who Should Not

Support agents are your best testers by a wide margin. They know the real phrasings, they remember the awkward cases, and they spot instantly when an answer is technically accurate but would still trigger a complaint. Give them the largest share of the set.

The person who wrote the knowledge base is your worst tester. Not through carelessness - they unconsciously phrase questions in the vocabulary of the documents they authored, so retrieval works beautifully and proves nothing. Keep them fixing content, not judging it. Add one person with zero product knowledge as a counterweight; they catch the jargon the bot parrots without ever explaining it.

Founders and managers drift straight to the happy path, get good answers, and declare the project finished. Hand them the wrong-but-plausible layer instead, where their nose for commercial risk is genuinely worth something. Give everyone the same four-point scale - correct, incomplete, wrong, should have declined - so results can be compared instead of argued about. And one rule matters more than the rest: testers record the exact input, character for character. Reproducing a failure needs the original wording, and a summary destroys it.

Fixing What Testing Finds

Most failures trace back to the knowledge base, not the model. A missing document, a PDF that still reflects last year’s pricing, two files that quietly contradict each other - any of those will defeat any system. When two sources disagree, the answer becomes a coin flip that changes between sessions.

Delete superseded documents. Don’t upload the newer version alongside them. Keeping both feels safe and it’s the single most common cause of inconsistent answers I run into. A second class of failure has nothing to do with facts: the answer is right but the register is wrong. Too curt, too chatty, too technical. That belongs in the scope and tone settings of the admin panel, not in the source files.

Voice deployments need their own pass. Speech recognition mishears product names, postcodes and surnames that read perfectly well as text, so a question set validated in chat is only half-validated for a voicebot. Setup questions of this kind come up often enough that the answers to common questions cover most of them. After every change, rerun the whole frozen set rather than just the failing case. Fixing one intent regularly shifts answers on a neighbouring one, and you want to see that straight away.

Soft Launch: Letting Part of the Traffic In

Internal testing stops producing new information faster than people expect. Your team runs out of imagination. Real customers ask things nobody on the payroll would ever think to type. At that point the useful move is exposure, carefully bounded.

Route a slice of traffic to the bot first: one channel, one language, one product line, or one time window. Off-hours traffic is the safest opening slice, because the customer’s alternative at 11pm is no answer at all rather than a slower human answer. That changes the risk calculation completely - you’re comparing the bot against silence, not against your best agent.

Keep the route to a human obvious and staffed the whole time. That’s something you arrange on your side, through your existing contact channels, not a function the bot performs by itself. Read transcripts daily for the first few weeks, then weekly, feeding every new phrasing back into the frozen set as a fresh batch. Widen the slice when a full week passes without a new category of failure showing up. Not when the calendar says it’s time.

What a Ready Bot Actually Looks Like

Readiness is measured by behaviour on the questions the bot can’t answer, not by its score on the ones it can. Any reasonable setup handles the core intents. Once you are live, containment rate as a signal tells you the same thing at scale. What separates a deployment that survives from one that gets switched off is how cleanly it handles the boundary.

A short pre-launch checklist worth running before you go wide:

• Core intents answered correctly from current, non-contradictory documents
• Out-of-scope questions declined cleanly, with a pointer to the right contact
• Tone consistent with how your brand actually speaks to customers
• Each supported language tested by someone who speaks it natively
• Escalation route to a human staffed during the hours the bot is live
• Transcripts reviewed by a named person, not by the team in general

Testing doesn’t end at launch. Your knowledge base ages every time a price, a policy or a product changes, and an untended bot slowly turns confidently outdated. Since Botino gives you a free first voicebot after registering, you can run this whole cycle - ticket mining, three-layer question set, soft launch - before committing to anything, and the time it frees up is worth reading more on business efficiency. The difference between a rollout that works and one abandoned after a month is almost always the quality of that pre-launch question set.