Detecting missing restrictions
An answer can repeat a source's main claim while dropping a restriction that changes its meaning. Version 3.9.0 reports condition_mismatch for cases such as an answer saying "Remote work is allowed two days per week" when the source says "Remote work is allowed two days per week with manager approval". The same check rejects "Access is allowed" as grounded in "Access is allowed, but not for guests".
Users can disable this check through the conditionCheck: false setting or the CLI option --no-condition-check.
The release notes describe a limited check, not comprehensive detection of restrictions. Negated contrasts without a scope word, including "but not guests" and "but guests are not admitted", are not yet recognized as restrictions. The check also misses restrictions tied to a different subject, as in an answer saying "Staff may enter" against the source "Guests may enter, but staff need approval". Support for restrictions that follow their subject is planned for version 3.10 to address these cases.
Benchmark gains and trade-offs
For the released code, test-split F1 on the condition-dropped benchmark reached 85.9% using the lexical check and 90.1% using a Claude judge. Changes on the public RAGTruth and HaluEval benchmarks were smaller: F1 shifted by no more than 0.01. The stricter check produced a few additional false alarms on RAGTruth, while HaluEval recorded three additional misses.
RAGTruth provides human, word-level hallucination annotations for nearly 18,000 responses generated by large language models using retrieval-augmented generation. Its tasks span question answering, summarization and data-to-text generation. Niu et al. published the corpus at ACL 2024, according to the ACL Anthology.
The project's Benchmarks page reports the full results, including metrics that worsened.
Text handling, speed and score consistency
Before comparing text, the checker normalizes zero-width characters, soft hyphens, full-width digits and no-break spaces. This prevents those formatting differences in PDF or web-page text from causing a quoted sentence to fail the check. Citation markers, including [1] and ¹, also no longer cause otherwise correct sentences to fail.
Sentence detection now keeps abbreviations such as "Dr. Lee" and "10 a.m." from prematurely ending a sentence, while treating list items as sentence boundaries.
Preparing each source and answer only once per check brings long-answer checking to linear time, enabling the reported reduction from about nine seconds to under half a second for a 1,000-sentence answer. The MCP server now measures its input limits consistently with the checker.
Verdicts no longer change merely because a source is duplicated or an unrelated source is added. The release also aligns the word weights used by computeFaithfulness, groundedness and PipelineEvaluator with those used by checkAnswer.
Library fixes and upgrade steps
The package can be installed with npm install @devilsdev/rag-pipeline-utils@^3.9.0. For the MCP server, the command is npx @devilsdev/[email protected] ./docs. Direct users of @devilsdev/rag-core, rag-guardrails, rag-connectors or rag-eval should bring those packages to version 3.9.0 too.
FileVectorStore now returns true cosine similarity scores, with 1 representing an identical vector, rather than scores that increased with the query vector's length. Users should retune minRelevanceScore accordingly. When saving, the store also merges its data with data saved by other processes.
Fixes to the default recursive chunker and the fixed-size and sentence chunkers prevent text from being lost or split. Existing documents need to be ingested again to receive these fixes, and the resulting chunks may differ.
Streaming responses now finish with a single complete done chunk and respect timeout through the end of the returned stream. Users streaming long answers may therefore need to increase timeout. Elsewhere, CostTracker checks budgets before execution, DAG node timeout and concurrency settings take effect, and AuditLogger adds close() and a functioning webhook.
Saved baselines need to be regenerated if they compare multi-source scores or hallucination classes. Checking against multiple sources can now produce different scores, and many hallucination classifications change even though rates and verdicts remain unchanged. The project's CHANGELOG contains the complete change list, with release notes available on GitHub.
