A RAG chatbot (RAG stands for retrieval-augmented generation) answers questions using your documents: policies, product sheets, contracts, support tickets. When someone asks a question, the system searches your knowledge base, pulls out the most relevant chunks, and gives them to the AI model to write an answer.
That's what makes RAG so useful, and it's also why it can leak data. If the wrong document can be retrieved, the AI will happily summarise it. Here are seven checks to run before launch.
1. Retrieval respects who's asking
The most common RAG leak is simple: everyone's documents go into one shared index, so any user can retrieve any chunk. A customer asks the right question and gets a slice of another customer's contract, or an intern gets the salary sheet.
Check: every chunk carries access information (which customer, team or role may see it), and the search filters by the current user's permissions before anything reaches the model. Don't rely on the prompt saying "only show the user their own data".
2. The knowledge base is cleaned before indexing
Teams often upload whole folders. Those folders contain things nobody meant to share: API keys in a setup guide, Aadhaar or PAN numbers in a form, internal pricing notes, personal phone numbers.
Check: scan documents for secrets and personal data before they're indexed, remove what the chatbot doesn't need, and decide who approves new documents going in.
3. Documents can't hijack the bot
Retrieved text goes straight to the model, so a document can carry hidden instructions. This is indirect prompt injection, and in RAG it's sometimes called knowledge-base poisoning. Anyone who can add or edit a document, a support ticket or a web page you index can try it.
Check: limit who can add content, treat retrieved text as untrusted data rather than instructions, and test with documents that contain planted instructions to see whether the bot obeys them.
4. The system prompt holds no secrets
Attackers regularly persuade chatbots to print their hidden instructions. If your prompt contains internal URLs, database names, discount rules or keys, assume they will be exposed.
Check: read your system prompt as if it were public. If anything in it would hurt you on a public website, move it out.
5. Output is handled safely
Model replies are shown in a web page or app, so they need the same care as any user input. A poisoned answer could include a malicious link, or markdown that loads an image from an attacker's server with your data in the URL. That's a known way to quietly exfiltrate information.
Check: escape or sanitise output before rendering it, restrict which domains links and images may load from, and never execute anything the model generates without validation.
6. Abuse is limited and visible
Attackers probe chatbots with hundreds of variations until one works, or simply run up your AI bill.
Check: rate-limit users, cap costs, log questions, retrieved sources and answers (masking personal data), and alert someone when usage spikes or the same user keeps trying jailbreak-style questions.
7. The vector database itself is locked down
The vector database that stores your embeddings is a database like any other. Some teams leave it reachable from the internet with default or no authentication, which exposes the whole knowledge base without even touching the chatbot.
Check: keep it on a private network, require authentication, use separate credentials for the app (read-only where possible), and back it up.
Before launch: test it like an attacker
Run through these checks with real attempts, not just a review of the settings:
- Log in as one user and try to retrieve another user's information.
- Ask the bot to reveal its instructions in a few creative ways.
- Upload or index a document containing hidden instructions and see if it follows them.
- Try to make an answer include an external link or image.
Frameworks like the OWASP Top 10 for LLM Applications are a good map for this testing. If you'd rather have experts do it, that's exactly what an AI agent pentest covers: we test your RAG chatbot, agents and AI app, and give you a prioritised fix report.
Not ready for a full test? Book a free 30-minute AI audit and we'll walk through your setup together.