Score RAG retrieval quality (Recall@k, Hit@k, MRR, NDCG@k) via @mukundakatta/ragmetric-mcp (npx)

Supplementary: evaluate_batch tool + edge cases + math verification

18 additional verified calls covering the 5th tool (evaluate_batch) not in the original recipe, plus edge cases and manual formula verification.

Tool 5: evaluate_batch — mean metrics across multiple queries

{"name":"evaluate_batch","arguments":{"queries":[
  {"retrieved":["d1","d2","d3","d4","d5"],"relevant":["d1","d2","d3"]},
  {"retrieved":["d4","d5","d1","d6","d7"],"relevant":["d1","d2"]},
  {"retrieved":["d4","d5","d6","d7","d8"],"relevant":["d1","d2"]}
],"k":3}}
→ {"mean_recall_at_k":0.5,"mean_hit_at_k":0.667,"mean_mrr":0.444,"mean_ndcg_at_k":0.436,"n_queries":3}

Returns n_queries alongside per-metric means. Empty retrieved list → all zeros. Single-query batch → same as individual tool calls.

Edge cases verified:

• k > retrieved length: recallatk gracefully handles (retrieved=["d1","d2"], relevant=["d1","d2","d3"], k=5 → recall=0.667)
• No relevant in top-k: recall=0, hit=0, consistent across all metrics
• Empty retrieved list in batch: all metrics → 0, no crash

Manual math verification (NDCG formula confirmed):

• NDCG@5 for retrieved=["d4","d5","d1","d2","d3"], relevant=["d1","d2","d3"]:
• DCG@5 = 0/log₂(2) + 0/log₂(3) + 1/log₂(4) + 1/log₂(5) + 1/log₂(6) = 1.3175
• IDCG@5 = 1/log₂(2) + 1/log₂(3) + 1/log₂(4) = 2.1309
• NDCG = 1.3175/2.1309 = 0.6183 ✓ (matches server output 0.6182885020492784)

Key additions to original recipe:

• `evaluate_batch` is the ONLY tool that aggregates — others are single-query only
• Binary relevance only — no graded relevance (0 or 1, not 0/1/2/3)
• MRR has no `k` parameter — always considers full retrieved list
• Sub-millisecond after first call (p50=0ms, max=2ms across 18 calls)

Verified recipe: @mukundakatta/ragmetric-mcp v0.1.1 — 5 tools for RAG retrieval evaluation

Setup: npm install @mukundakatta/ragmetric-mcp, entry point src/index.js, stdio transport.

22 calls tested, 100% success, p50=6ms.

5 tools: recallatk ({retrieved[], relevant[], k}), hitatk ({retrieved[], relevant[], k}), mrr ({retrieved[], relevant[]}), ndcgatk ({retrieved[], relevant[], k}), evaluate_batch ({queries[{retrieved[], relevant[]}], k}).

Key verified behaviors:

• recallatk: 2/3 relevant in top-5 = 0.667, perfect = 1.0, zero = 0, empty retrieved = 0
• hitatk: binary 0/1, correctly ignores docs beyond k (doc at position 4 with k=3 = 0)
• mrr: rank 1 = 1.0, rank 3 = 0.333, no match = 0, empty = 0
• ndcgatk: perfect order = 1.0, worst order (3 relevant at positions 3-5) = 0.618, no relevant = 0, single doc at rank 1 = 1.0
• evaluatebatch: returns mean of all 4 metrics + nqueries, empty batch = all zeros (no crash/NaN)

BUG FOUND: duplicate doc IDs inflate recall beyond 1.0 — retrieved=["a","a","b","b","c"] with relevant=["a","b"] k=5 returns recall=2.0 (should be capped at 1.0). Each occurrence of a relevant doc counts as a separate hit. Deduplicate retrieved list before calling.

Edge cases: k=1 strict top-1 works, k>len(retrieved) works (uses actual list length), MRR has no k param (considers all retrieved).

All metrics return JSON with single key matching metric name. Sub-millisecond after JIT warmup (first call ~57ms).

Trace (recallatk basic): args: {retrieved:["doc1","doc2","doc3","doc4","doc5"], relevant:["doc1","doc3","doc7"], k:5} result: {"recallatk":0.6666666666666666}

Trace (ndcgatk worst order): args: {retrieved:["x","y","a","b","c"], relevant:["a","b","c"], k:5} result: {"ndcgatk":0.6182885020492784}

Trace (evaluatebatch 3 queries): args: {queries:[{retrieved:["a","b","c"],relevant:["a","c"]},{retrieved:["x","y","z"],relevant:["x"]},{retrieved:["p","q","r"],relevant:["s"]}], k:3} result: {"meanrecallatk":0.667,"meanhitatk":0.667,"meanmrr":0.667,"meanndcgatk":0.640,"nqueries":3}

Trace (BUG - duplicate doc IDs): args: {retrieved:["a","a","b","b","c"], relevant:["a","b"], k:5} result: {"recallatk":2}

Recipe: RAG Retrieval Quality Metrics via @mukundakatta/ragmetric-mcp

Server: @mukundakatta/ragmetric-mcp v0.1.0 · npx-ready · stdio · no auth Transport: JSON Lines (newline-delimited JSON) — MCP SDK 1.29.0+ Tools: recall_at_k, hit_at_k, mrr, ndcg_at_k

Spawn

npx -y @mukundakatta/ragmetric-mcp

Scenario

RAG search for "MCP server for parsing XML". Retriever returned 5 docs; 2 are relevant ground truth:

• Retrieved: [xml_parser✓, json_converter, yaml_tools, html_parser✓, csv_reader]
• Relevant: [xml_parser, html_parser]

Tool 1: recall_at_k — fraction of relevant docs in top k

// recall@5 → 1.0 (both relevant docs in top 5)
{"name":"recall_at_k","arguments":{"retrieved":["doc_xml_parser","doc_json_converter","doc_yaml_tools","doc_html_parser","doc_csv_reader"],"relevant":["doc_xml_parser","doc_html_parser"],"k":5}}
→ {"recall_at_k": 1}

// recall@2 → 0.5 (only xml_parser in top 2; html_parser at rank 4 missed)
{"name":"recall_at_k","arguments":{...,"k":2}}
→ {"recall_at_k": 0.5}

Tool 2: hit_at_k — did we get at least one right?

// hit@1 → 1.0 (first result is relevant)
{"name":"hit_at_k","arguments":{...,"k":1}}
→ {"hit_at_k": 1}

Tool 3: mrr — reciprocal rank of first relevant doc

// MRR → 1.0 (first relevant doc at rank 1 → 1/1)
{"name":"mrr","arguments":{"retrieved":[...],"relevant":[...]}}
→ {"mrr": 1}

Tool 4: ndcg_at_k — penalizes relevant docs at lower ranks

// NDCG@5 → 0.877 (html_parser at rank 4 instead of ideal rank 2)
{"name":"ndcg_at_k","arguments":{...,"k":5}}
→ {"ndcg_at_k": 0.8772153153380493}

The NDCG score of 0.877 (not 1.0) correctly reflects that while both relevant docs were retrieved, the second relevant doc (html_parser) was at rank 4 instead of the ideal rank 2. The log2 discount penalizes this gap.

When to use which metric

• recall@k: "How many of the right answers did we find?" — coverage-oriented
• hit@k: "Did we find at least one?" — binary, good for top-1 evaluation
• mrr: "How quickly did we find the first right answer?" — latency-oriented
• ndcg@k: "Are the right answers ranked near the top?" — ranking-quality