- The paper audits multi-hop RAG benchmarks and finds that NV-Embed-v2, the common retrieval anchor, is non-commercially licensed while three dependent systems omit this limitation in their disclosures.
- The paper measures 13 embedders on the same 1,000-question MuSiQue harness and finds a 2.31-point commercial Recall@5 gap in June 2026 that narrowed to a statistically indistinguishable difference after Nemotron-3-Embed-8B’s release.
- The paper shows that indexing costs are often undisclosed and configuration-sensitive, with GraphRAG estimates ranging from $2.30 to $24.94 on the same corpus and projected 1TB costs spanning roughly $428,000 to $4.6 million.
The problem the paper identifies
This paper argues that the benchmarks used to rank multi-hop retrieval-augmented generation (RAG) systems report headline numbers stripped of the two variables that determine whether a corporate buyer can actually use them: whether the retrieval backbone may be legally deployed commercially, and what it costs to build. The evaluation target is the MuSiQue closed-corpus protocol standardized by HippoRAG-2, under which a fixed pool of 11,656 Wikipedia passages and a 1,000-question sample define the comparison, with Recall@k as the metric. Under this protocol, the dense-retrieval floor is NV-Embed-v2, and every leading system — HippoRAG-2 (74.7% Recall@5), PropRAG (78.3%), and SAG (80.0%, or 81.7% in its own ablation when swapped onto NV-Embed-v2) — reports numbers that inherit the properties of that single component.
The authors frame the problem by analogy to quantitative finance: a strategy backtest that omits transaction costs is not a wrong backtest, but it measures something the practitioner cannot trade. Similarly, a benchmark number measured under conditions the reader does not share — non-commercial licensing, undisclosed indexing cost — answers a question practitioners are not asking. The paper is explicit about its non-claims: it does not allege misconduct or invalid protocols, only that the published record omits conditions that determine deployability.
License verification and the anchor's lineage
NV-Embed-v2 is licensed cc-by-nc-4.0, and its model card states plainly that it "should not be used for any commercial purpose." The paper traces the restriction to a specific cause: the training mix includes MS MARCO, whose license is non-commercial. An NVIDIA representative confirmed on the model's HuggingFace discussion board that non-commercial training data is the reason the model could not be declared commercial. The paper notes that accessing the model through NVIDIA's paid NIM inference service changes who operates the hardware, not what the model is licensed for — a different access channel, not a separate license.
The disclosure audit then searches the full text and linked repositories of the three anchor-dependent systems (HippoRAG-2, PropRAG, SAG) for license terms. None discloses the restriction anywhere. The fourth audited system, KET-RAG, builds on OpenAI's text-embedding-3-small and is the counterexample showing a commercially usable embedder was a live option. The authors are careful to state that the license is not hidden — it is on the model card — so the finding is a failure to report a stated fact, not a concealed one. They also state plainly that the audited set is a relevance-seeded snowball sample, so all counts describe these papers and never a prevalence claim about the field.
The commercial-embedder retrieval floor
The paper measures thirteen embedders from eight makers on one identical harness: embed the 1,000 questions with no decomposition or reranking, embed the corpus once, retrieve by cosine similarity, score Recall@k. All measurements, including NV-Embed-v2 itself, are run by the authors rather than borrowed from the literature — a choice justified by their finding that query-formatting choice alone swings one entrant (Qwen3-VL-Embedding-8B) by 11.6 Recall@5 points, making literature figures of unknown provenance unsafe to compare.
The headline result: NVIDIA's Nemotron-3-Embed-8B, released 2026-07-16, is statistically indistinguishable from the non-commercial anchor. The paired difference is +0.24 Recall@5 (95% CI [−0.94,+1.43], p=0.69) and −0.58 at Recall@10 (p=0.28). Before that release, the best commercially licensed embedder in the panel — Google Gemini embedding-001 at 67.24 — trailed the anchor by 2.31 Recall@5 points, a significant gap (95% CI [0.91, 3.71], p=0.001). The "commercial tax" was therefore real and measurable in June 2026 and closed within a single release cycle in July 2026.
The paper is deliberate about the verb and about what the design cannot support. At n=1,000, the smallest detectable difference at 80% power is roughly 1.7 points; a two-one-sided-test equivalence claim at a ±0.5-point margin would require roughly 33,700 questions given the observed point estimate. The supported claim is that any residual tax is small relative to the 2.31 points measured before, not that it is provably zero.
Inference is handled with unusual care. Because per-question scores across entrants are strongly correlated (mean k0), overlap of marginal confidence intervals is formally invalid as a pairwise test. The uniqueness claim — that NV-Embed-v2 is the only entrant indistinguishable from the top entrant — is reported under Holm–Bonferroni correction across twelve comparisons and confirmed with Dunnett-type simultaneous intervals (max-k1 critical value 2.78, independently re-implemented to within 0.4%). Under a global Holm family of all 47 paired tests in the paper, every claimed rejection stands.
The durable finding is the paid-versus-free divide rather than any single ranking. The four API-only providers charge per token on every re-index; self-hosted open-weight models charge nothing per token. Nemotron-3-Embed-8B is the only entrant simultaneously commercially licensed, free to self-host, and statistically indistinguishable from the anchor. Every other entrant meeting the first two conditions sits 5.2 to 14.6 points below it. Two secondary results reinforce the point that embedder-level measurement matters: Recall@10 lifts every embedder by a tightly clustered k2 to k3 points (all intervals exclude zero), and BGE-M3 — among the most widely deployed open-weight embedders at 35.6M monthly HuggingFace downloads — lands second-to-last in dense mode, evidence that general-purpose leaderboard reputation does not transfer to a stringently filtered hard benchmark.
Cost disclosure and standardized costing
The cost audit finds that three of five systems disclose no usable indexing cost: SAG and Microsoft's GraphRAG publish no dollar figure at all (SAG despite claiming production deployment at hundreds of millions of items), and HippoRAG-2 publishes token counts but no dollars. Only PropRAG (~$k$41.89) price the shared 5.64MB corpus.</p>
<p>The sharpest finding concerns GraphRAG. KET-RAG's own tables price GraphRAG at $k$524.94 (high-performance) on the identical corpus — an 11x spread driven entirely by a chunk-size choice that Microsoft's paper does not name. An independent token-based reconstruction ($k$638.99) brackets the high-performance figure, so the two estimates agree once configuration is matched; what a reader cannot recover is which configuration "the cost of GraphRAG" means. Extrapolated linearly to 1TB, the undisclosed choice separates roughly $k$74.6M for a single index build. The authors caveat that linear extrapolation is almost certainly optimistic for graph construction, since entity extraction plausibly scales super-linearly, making the 1TB figures more likely underestimates.</p>
<p>The cost model deliberately separates one-time embedding cost from recurring answering cost. At matched 1TB scale, embedding sits 7.5x to 900x below graph construction, and a full year of answering at 10,000 queries/day sits 350x or more below it. Answering cost was measured from real API usage with <a href="https://www.emergentmind.com/topics/gpt-4o-mini-8a7e420b-c7de-47e7-818a-da41dc130fa6" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">gpt-4o-mini</a> over actual retrieved passages: $k$80.060 per 1,000 queries at batch rates, with the rate card predicting the meter to within 4%, and the k=5→k=10 cost ratio tightly clustered at 1.86–1.93x across the panel. A practical crossover follows: at 10,000 queries/day, recurring answering cost overtakes one-time API re-embedding within the first year for corpora between roughly 10GB and 180GB. The paper's summary is pointed: embedding choice is a licensing and sovereignty question, answering cost is a volume question, and indexing architecture is where the money actually is — and disclosure runs inversely to size, since the largest cost line is the least reported.
Limitations and open questions
The paper concedes its limits with unusual specificity, several of which bear directly on the headline. The panel is not vendor-neutral: five of thirteen entrants are NVIDIA models, and the winning model shares a maker with the anchor it matches. The authors disclose this and point to the eight non-NVIDIA entrants as competitive context, but readers should weigh the headline accordingly. Uneven query-instruction sweeps flatter the anchor itself, which was swept over four instructions while Nemotron-3-Embed-8B was measured in one configuration — a bias that runs against the paper's own thesis and makes the $k$9 conservative. The pre-specification of the primary Gemini-vs-anchor comparison is asserted from version history, not external registration, and the authors say so plainly. A pilot on 2WikiMultihopQA replicates the match between the top two embedders but shows no measurable commercial gap there ($+0.24$0 points, $+0.24$1), weakening the magnitude claim: the tax's size is a property of the corpus, not a constant of the field. All Recall@$+0.24$2 figures are index-free ceilings — exhaustive cosine search, no ANN — so none is a production recall figure. Hosted endpoints are exposed to silent server-side drift (text-embedding-3-large moved $+0.24$3 points between rounds), making hosted figures point-in-time. The cost model prices machine time only, not human setup cost, and does not measure the query-time retrieval-LLM costs of the audited systems, a gap none of them disclose either.
Conclusion
This paper measures two omissions in the multi-hop retrieval literature and finds both consequential. On licensing, the protocol's anchor is non-commercially licensed for a traceable reason, and none of the three anchor-dependent systems discloses it. On cost, the literature's least-disclosed number is its largest: graph construction, which reaches six and seven figures at institutional scale with an 11x configuration-dependent uncertainty for the field's most-cited baseline. Measured on one harness, the quality penalty for requiring a commercial license was a significant 2.31 Recall@5 points in June 2026 and is no longer measurable as of July 2026 — with the explicit caveat that "matches" is the strongest verb the data support, and that the finding holds on one corpus. The durable contribution is methodological as much as empirical: the released per-question vectors, harness, and cost model make both the ranking and the disclosure audit recomputable by any reader, which is the standard the paper argues the field should adopt.