RAG: Four Tests to Detect Common Vulnerabilities

Four realistic flaws are enough to thwart a simple RAG retriever and mislead a model. Fault injection tests and systematic proof checking allow for their detection, followed by targeted fixes: recency rule, cautious OCR normalization, fuzzy matching, and appropriate ingestion of multi-page tables.

Evidence is required to validate the claims made.

The identifier of the returned document indicates which source was ranked first, but this does not guarantee that the response is based on the correct information. A proof check mandates the presence of necessary words or values in the retrieved passage. For example, a test on the return policy requires both the ID returns-2026 and the mention of "30 days," while a test on a table requires "Basic" and "10 Go" in the same passage. These checks facilitate diagnosis: a wrong ID signals a ranking or filtering issue, a correct ID without the expected text points to an extraction or segmentation flaw, and a correct ID with the required evidence provides the language model with the necessary context, even if the final response still needs to be evaluated. If multiple passages are sent to the model, the proof check applies to the combined text.

Four Corrupted Queries Reveal the Weaknesses of the Retriever

Executing four deliberately corrupted inputs results in four failures on a simple retriever. Once the corrections are activated for each case, the four queries return the expected document. The fixes are targeted: tie-breaker by date, normalization of known OCR errors, approximate spelling matching, and preservation of table lines during ingestion. Each mechanism addresses a distinct cause, and separate tests help identify the missing protection. These trials complement the classic relevance evaluation, which measures behavior on expected inputs, while fault injection assesses robustness against realistic flaws.

Contradictions Between Versions: Tiebreaking by Recency

The return policy, which changed from 60 to 30 days at the beginning of 2026, illustrates the risk of duplicates: ingestion added the new page without removing the old one, and both texts share the same keywords, resulting in a score tie. Resolved by list order, this tie returns the 2024 version and could indicate 60 days instead of 30. A recency-based tiebreaker favors the most recent update and corrects this case. This rule is suitable for explicitly replaced policies, while other collections may require a current or withdrawn status when novelty does not imply replacement.

OCR Noise: Normalize Without Distorting Data

If "Enterprise" is transformed into "Enterpr1se" and "SSO" into "SS0" in a text, the points associated with the tokens enterprise and sso drop to zero, leading to a general tie resolved by list order. A simple normalization that converts 0 to o and 1 to i, applied to queries and documents before tokenization, restores matches. However, these rules must remain limited: transforming all 0s into os can distort product codes or measurements. Substitutions should be built from actually observed errors and limited to fields where they are safe.

Spelling Mistakes: Use Fuzzy Matching and Adjust the Threshold

With the query "warehuse sync," exact matching treats the misspelled term as unrelated and lets "sync" break the tie in favor of the document that appeared first. By enabling spelling similarity, the word "warehuse" comes close enough to "warehouse" for the inventory document to prevail. The similarity threshold then becomes crucial: too low, it associates unrelated words; too high, it lets common mistakes slip through. Thresholds based on real queries are more relevant than those derived from a few invented errors.

Multi-Page Tables: Fix at Ingestion

When PDF extraction cuts a table across pages, an initial passage may contain the label "Basic |" without the value "10 Go," which is placed on the next page. This fragmentation favors the ranking of the incomplete passage and deprives the model of the numerical data. The correction lies within the ingestion pipeline: detecting table fragments and joining related pages before segmenting into retrievable passages. Systematically joining each pair of pages would produce overly large passages and content mixes; the rule should target detected tables or integrate just enough nearby context to preserve each line.

The Minimal Testbed: Four Documents, One Score per Token

The testbed relies on four short documents and a simple score: 1 point per query token present in the document after tokenization to lowercase via the regex [a-z0-9]+. Each document retains an identifier, a subject, and an update date. Examples include returns-2026 with 30 days, plans mentioning SSO, crm with contact synchronization every five minutes, and inventory with synchronization every 15 minutes. Production environments often employ more sophisticated matching techniques, but the illustrated flaws can affect them in the same way.