Assay

How do you know which language model to trust with real work? Assay runs a blinded, citation-checked exam over a body of source documents and turns the result into a defensible choice.

The pipeline

The first domain is Indian GST law, where every answer can be checked against the Act, a circular or a ruling.

  1. Author the exam

    A domain expert writes weighted questions, an answer key and a pinned corpus of source documents. Weight means the cost of being wrong. The author never scores.

  2. Send one identical prompt

    Every model gets a byte-identical prompt. Prompts are SHA-256 hashed and the hashes are published with the results.

  3. Check the model that answered

    Runs call provider APIs directly. If the returned model differs from the requested one, the run stops. Caching is off, and every request is logged.

  4. Verify citations in code

    A deterministic checker looks up each citation in the corpus with full-text search. Only near matches go to one pinned adjudicator, who cannot see which model wrote the quote.

  5. Blind the answers

    Answers are anonymised and sent to three judges from three model families. No model family ever scores its own output.

  6. Score with statistics

    Python computes the totals: citation 0–3, correctness 0–4 against the key, reasoning 0–3. Bradley–Terry ratings and bootstrap 95% intervals follow, and overlapping intervals are reported as a tie.

The system

prompts/What is actually sent, with hashes locked in prompts.lock.json
bench/runner.pyGeneration, the model drift check and full request logging
bench/corpus.py + factcheck.pySQLite FTS5 index with exact and fuzzy citation lookup
bench/anonymize.py + judge.pyBlinding, leave-one-out jury assignment and scoring
bench/score.pyWeighted totals, Bradley–Terry ratings and bootstrap intervals
runs/<run_id>/Immutable artifacts: manifest, responses, verdicts and scores

Rules that keep it honest

Never run inside an agent

Coding harnesses can substitute models silently, so the benchmark runs as plain Python against provider APIs.

LLMs never add up the score

Citation scores come from code wherever possible, and every total comes from Python.

Pin everything

Provider, quantization and sampling settings are recorded on every request.

Report ties as ties

When confidence intervals overlap, there is no winner to announce.