Retrieval-augmented generation is usually introduced as a reliability technique: give a language model trusted documents so it can answer from evidence instead of memory. A new benchmark from University of Ottawa researchers shows why that architectural change also needs its own safety evaluation. RAG-Safety-Bench tests the same harmful requests under four controlled conditions and finds that a model's refusal behavior without retrieval does not reliably predict its behavior once retrieved context is added.
The study evaluates five open-source instruction-tuned models and separates two risks that are often mixed together. One is straightforward: retrieval may supply information that directly helps answer a harmful request. The other is subtler: even documents that are relevant to the topic but do not contain the harmful answer may alter how some models respond. The second effect appears model-specific rather than universal, which makes the benchmark more useful than a simple claim that “RAG makes models unsafe.”
Four conditions isolate what retrieval actually changes
The benchmark contains 987 harmful requests in its full set, with a balanced 346-question subset used for the main paper analysis across 20 harm subcategories. Each request is evaluated in four conditions: no retrieval; an oracle document that contains information needed to answer the harmful request; a topically related document that does not contain the answer; and a random safe document unrelated to the request.
That design removes retriever quality as the main confounding variable. The experiment is not asking whether a production retriever happened to fetch the wrong page. It asks what changes after a specific kind of context has already been inserted into the prompt.
The authors test Gemma-3-12B-It, Llama-3.1-8B-Instruct, Ministral-3-8B-Instruct, Qwen-2.5-7B-Instruct and Phi-4-14B. Safety is judged by an ensemble of LlamaGuard-3-8B, ShieldGemma-2B and WildGuard, with a response marked unsafe when at least two judges agree. A 100-sample human check is reported as strong agreement with the automated scoring, although the benchmark still inherits the limitations of automated safety judges.
Answer-bearing retrieval is the clearest general risk
Across the tested models, the most consistent degradation occurs when retrieval provides an oracle document containing information useful for the harmful request. That result is operationally important but unsurprising: a refusal-aligned model can receive detailed external evidence that makes the requested task easier.
The more interesting result is what happens with safe, on-topic context that does not contain the answer. The effect is not uniform. Several models remain near or below their non-RAG unsafe-response baseline, while Qwen-2.5-7B-Instruct shows a clearer increase in unsafe generation under the related-but-answer-free condition.
That distinction matters because it prevents an overly broad conclusion. The paper supports the claim that safety behavior can change after retrieval and that some models are sensitive even to benign topical context. It does not support treating every retrieved document, or every long context, as inherently destabilizing.
Base-model safety is not a deployment guarantee
Aipolix's analysis is that RAG safety should be evaluated as a property of the deployed system, not inherited from the model card or a no-retrieval benchmark. The relevant unit is the model plus retrieval corpus, retrieval condition, prompt construction and downstream safety controls.
A practical evaluation matrix should therefore include at least the four conditions used by the benchmark: no retrieval, answer-bearing retrieval, relevant-but-answer-free retrieval, and unrelated safe retrieval. If a team tests only ordinary RAG accuracy and a separate base-model refusal benchmark, it can miss the interaction between the two.
This changes how release gates should be built. A model upgrade can improve standalone safety while a change in chunking, corpus composition or prompt assembly changes the safety behavior of the complete RAG system. Conversely, a retrieval change can alter the risk profile without changing the underlying model at all. Safety regression testing should therefore version the retrieval pipeline and corpus alongside the model.
The public artifact is useful, but not fully reproducible end to end
The accompanying repository strengthens the paper's practical value. It includes the frozen full and balanced evaluation inputs, provenance manifests, generation configurations, code for generation and standard local safety scoring, and paper-facing figures. The repository states that the full set contains 987 questions expanded into 3,948 condition rows, while the balanced subset contains 346 questions and 1,384 rows.
There is also an important reproducibility limitation. The repository deliberately omits raw model responses, judge traces and the rescored unsafe_majority_v2 artifacts because they contain sensitive model outputs; it says the v2 rescore workflow is maintained separately. The included judge configurations are described as implementation examples rather than a replacement for the private rescoring artifacts used for the paper.
The repository also says final license files are still pending author approval, although it recommends CC BY-SA 4.0 for benchmark data and MIT for original code. That means the research artifact is inspectable and partly reproducible, but a third party cannot yet reproduce every paper-facing safety judgment from the public repository alone.
Production RAG needs controls on both sides of retrieval
The benchmark's deployment implication is broader than adding another refusal classifier. A production system can filter incoming requests, curate or filter the corpus, constrain what retrieval may return, and inspect the generated output. Those controls address different failure paths.
The four-condition benchmark is especially useful for diagnosing where a failure originates. If unsafe behavior rises only with answer-bearing documents, corpus curation and retrieval-side controls may be the first target. If it rises with benign but related context, teams need to examine prompt construction and the model's context sensitivity as well. If the no-RAG baseline is already poor, retrieval is not the primary problem.
The authors also acknowledge important boundaries: the benchmark is English-only, uses Wikipedia-derived context, covers a finite set of harm categories, intentionally abstracts away retriever performance, and tests relatively small open-source models rather than commercial frontier systems. It is therefore not evidence that a specific enterprise RAG deployment is unsafe.
What it does establish is a better evaluation question. Instead of asking whether a model is “safe” and whether a retriever is “accurate” as separate properties, teams should test whether the combined system stays within policy across the kinds of context it may actually receive. RAG changes the evidence available to the model; safety evaluation has to cross that same architectural boundary.