Recomputed daily from live GitHub signals

Better questions.
Better evaluations.

Find the tools to measure quality, reliability and risk.

Explore the catalog Sources & methodology →

281

tools ranked

Evals catalog

← All 17 indexes

281 records · snapshot 2026-11-03

How to choose: scope, evidence and limitations

Choose evaluation tools around a defined failure

Decide whether you are evaluating answers, retrieval, agent actions, code or a safety boundary. A framework, benchmark and production monitor provide different kinds of evidence.

Guide sources reviewed 2026-11-03. Catalog snapshot: 2026-11-03T16:38:16.780832+00:00. Source review does not mean runtime testing.

Measure answers on a task set

Define reference answers, judging criteria and dataset/version before choosing a harness. Review where automatic judging disagrees with human labels.

Current catalog categories: Eval Frameworks; Benchmarks; Reasoning.

Evaluate a complete system

Include retrieval/context, tool-call trajectories or executable tests as appropriate. Measure the task outcome and relevant intermediate failures.

Current catalog categories: RAG Eval; Agent Eval; Coding Eval.

Investigate production and adversarial failures

Specify the threat/failure model and permitted test scope. A passing test suite or dashboard does not prove universal safety or correctness.

Current catalog categories: Red Teaming & Safety; Observability.

Evidence to check before choosing

Keep model, prompts, tools, data and evaluator versions with each result.
Separate held-out examples from training/optimization data.
Compare cost, runtime and agreement with reviewed labels under the same conditions.

What remains unknown

Benchmark wins, evaluator validity, safety coverage and comparative task-success rates are unverified unless supported by a reproducible result outside this catalog.

Sources and scope

Published collection and scoring rules

Continue by task: observability · rag · agents.

About The Eval Index

Snapshot: 2026-11-03T16:38:16.780832+00:00 · 281 catalog records.

Source: Public GitHub repository searches and API signals. Collection and scoring source · Current JSON snapshot · Page inventory.

What does this catalog cover?

The Eval Index contains 281 records in the snapshot dated 2026-11-03T16:38:16.780832+00:00. Coverage is defined by its published data source and collection rules; inclusion is not an endorsement.

How is the default order determined?

Momentum blends log-scaled GitHub stars (55%), push recency decaying over 180 days (32%), and a young-repository bonus for repositories under 120 days old with at least 20 stars (13%), rounded and clamped to 1–100. Configured searches select the catalog; coverage is not exhaustive. Stars and repository activity are proxies, not measured adoption, reviews, security or product-quality benchmarks.

When was this snapshot collected?

The recorded snapshot time is 2026-11-03T16:38:16.780832+00:00. The refresh workflow is scheduled daily; this timestamp records the available data rather than guaranteeing a successful refresh every day.

Browse The Living Indexes by topic. Published by Kymata Labs.

Entry descriptions come from upstream publishers or the configured catalog. Promotional claims have not been independently tested by this index.

Better questions.Better evaluations.