Score RAG retrieval quality (Recall@k, Hit@k, MRR, NDCG@k) via @mukundakatta/ragmetric-mcp (npx)

Recipe: RAG Retrieval Quality Metrics via @mukundakatta/ragmetric-mcp

Server: @mukundakatta/ragmetric-mcp v0.1.0 · npx-ready · stdio · no auth Transport: JSON Lines (newline-delimited JSON) — MCP SDK 1.29.0+ Tools: recall_at_k, hit_at_k, mrr, ndcg_at_k

Spawn

npx -y @mukundakatta/ragmetric-mcp

Scenario

RAG search for "MCP server for parsing XML". Retriever returned 5 docs; 2 are relevant ground truth:

• Retrieved: [xml_parser✓, json_converter, yaml_tools, html_parser✓, csv_reader]
• Relevant: [xml_parser, html_parser]

Tool 1: recall_at_k — fraction of relevant docs in top k

// recall@5 → 1.0 (both relevant docs in top 5)
{"name":"recall_at_k","arguments":{"retrieved":["doc_xml_parser","doc_json_converter","doc_yaml_tools","doc_html_parser","doc_csv_reader"],"relevant":["doc_xml_parser","doc_html_parser"],"k":5}}
→ {"recall_at_k": 1}

// recall@2 → 0.5 (only xml_parser in top 2; html_parser at rank 4 missed)
{"name":"recall_at_k","arguments":{...,"k":2}}
→ {"recall_at_k": 0.5}

Tool 2: hit_at_k — did we get at least one right?

// hit@1 → 1.0 (first result is relevant)
{"name":"hit_at_k","arguments":{...,"k":1}}
→ {"hit_at_k": 1}

Tool 3: mrr — reciprocal rank of first relevant doc

// MRR → 1.0 (first relevant doc at rank 1 → 1/1)
{"name":"mrr","arguments":{"retrieved":[...],"relevant":[...]}}
→ {"mrr": 1}

Tool 4: ndcg_at_k — penalizes relevant docs at lower ranks

// NDCG@5 → 0.877 (html_parser at rank 4 instead of ideal rank 2)
{"name":"ndcg_at_k","arguments":{...,"k":5}}
→ {"ndcg_at_k": 0.8772153153380493}

The NDCG score of 0.877 (not 1.0) correctly reflects that while both relevant docs were retrieved, the second relevant doc (html_parser) was at rank 4 instead of the ideal rank 2. The log2 discount penalizes this gap.

When to use which metric

• recall@k: "How many of the right answers did we find?" — coverage-oriented
• hit@k: "Did we find at least one?" — binary, good for top-1 evaluation
• mrr: "How quickly did we find the first right answer?" — latency-oriented
• ndcg@k: "Are the right answers ranked near the top?" — ranking-quality