Why Your AI Chatbot Gets Answers Wrong (And How to Fix It)

A chatbot that gives confidently wrong answers is worse than no chatbot at all. At least with no chatbot, the visitor knows to email you. With a confidently wrong chatbot, they act on bad information — and then they blame you.

Wrong chatbot answers are almost never a model problem. The underlying AI is fine. The problem is almost always one of four things: the chatbot couldn't find the right information, the information was outdated, the documents were in the wrong format, or the chatbot was set up to always answer even when it shouldn't.

Here is what each failure looks like, and what to do about it.

Failure 1: The chatbot can't find the right passage

Your refund policy document uses the phrase “subscription termination and reimbursement eligibility.” Your visitor types: “Can I get a refund?”

Those two phrasings mean the same thing, but they look very different to a vector search engine. The retrieval step converts both the query and your document chunks into numeric vectors and measures the cosine distance between them. If the phrasing is too different, the cosine similarity score stays low — and the right answer never reaches the model.

Minyut splits documents into 512-token chunks with a 50-token overlap, cutting at paragraph and sentence boundaries to keep meaning intact. But even well-formed chunks cannot compensate for documents written in a completely different register than the questions visitors actually ask.

The fix here is not technical. Write your documents in the language of the question, not the language of the answer. If customers ask “how do I cancel?”, your policy should include that phrasing — not just the formal heading “termination of subscription.” The closer your wording is to how visitors phrase things, the higher the similarity scores will be, and the more accurate the answers.

Failure 2: Your documents are living in the past

The industry term for this is knowledge base rot. Your documents decay — prices change, policies update, features ship or get retired — but the chatbot keeps answering from whatever was uploaded last.

A visitor asks about your current pricing. The chatbot quotes the rate from eight months ago, before you restructured your plans. The visitor signs up expecting one price and encounters another. That is a trust problem, not a technology problem.

The fix: tie your document uploads to your product release cycle. Every time a price, a policy, or a feature description changes, the corresponding document should be reuploaded. Treat it the same way you would update a changelog. It takes five minutes and prevents the kind of confident misinformation that quietly costs you customers.

As a rough rule: if you would update your pricing page, you should also update the pricing document in your chatbot's knowledge base.

Failure 3: Documents that contain nothing the chatbot can read

There are two ways a document can embed nothing useful: it is a scanned image (not text), or it is so dense and generic that the useful parts get buried.

Scanned PDFs are the silent killer. If someone photographed a page and saved it as a PDF, there is no extractable text — just an image. A system without scanned PDF detection will process the file, report success, and then answer zero questions from it because there was nothing to embed. Minyut detects scanned documents and returns a clear error rather than silently producing an empty knowledge base. Text-based PDFs and Markdown files embed cleanly; scanned ones do not.

Dense documents are a subtler problem. An 80-page user manual where 10% of the content is genuinely useful degrades retrieval precision. Every chunk added to the index competes for the same top-5 retrieval slots at query time. Chunks of boilerplate legal text and introductory filler dilute the results, pushing the actually useful passages out of the window.

Focused 5–10 page documents consistently outperform rambling comprehensive ones. Upload targeted documents for your top question categories first. Add more only once you have confirmed the core knowledge base is working.

Failure 4: A chatbot that answers when it shouldn't

This is the most dangerous failure mode. Most off-the-shelf AI chatbots do not have a fallback gate. If retrieval fails to find a good match, the model falls back on its general training data and produces a plausible-sounding answer. From the visitor's perspective, it looks like a real answer from your knowledge base. It is not.

Minyut uses a 0.60 cosine similarity threshold. If no document chunk scores above 0.60 against the visitor's query, no LLM call is made at all. The chatbot responds: “I don't have information about that in my knowledge base.” That is the correct and honest answer. The visitor then knows to reach you directly — which is exactly what you want when the chatbot genuinely cannot help.

How to find out which failure is yours

You cannot fix a problem you cannot see. Start with your chatbot's query logs. Every conversation is recorded — you can see exactly what the visitor asked, what the chatbot returned, and whether a matching chunk was found above the threshold.

Look for two patterns. Queries that consistently return no match (the chatbot says it doesn't know) point to vocabulary mismatch or missing documents. Queries where the answer is wrong — and the chatbot said it confidently — usually mean stale documents or a system with no similarity threshold at all.

Then test with real visitor phrasing, not your own. Type questions the way a customer who has never read your documentation would phrase them. There is almost always a gap — and that gap explains a significant share of your wrong answers.

The fix is the documents, not the AI

Almost every chatbot accuracy problem is a document problem. The model is doing its job. The retrieval is doing its job. The documents are failing — because they are out of date, written in the wrong register, too dense to retrieve from, or not there at all.

Write documents in the language customers use. Keep them focused. Update them whenever your product or policies change. Use a chatbot platform that has a real similarity threshold — one that lets the system say “I don't know” rather than guess.

The ROI of a well-built chatbot depends entirely on whether it answers correctly. A chatbot that answers 80% of questions correctly and says “I don't know” on the rest is far more valuable than one that answers 100% of questions with 30% wrong.

If you want to test how these failure modes show up in practice —the 0.60 threshold, scanned PDF detection, query logs — you can start a free Minyut account, upload a document, and test it against real questions in under 10 minutes. The free plan covers one chatbot and 100 queries per month, which is enough to see exactly how the system behaves before you spend anything.