Runs juryrig's full audit_suite against five built-in mock judges, each with a deliberate flaw, so you can see what a flagged audit looks like. No input needed.
Full text reports
Input (CSV with header, or a JSON list of objects). One row per pair of responses you showed the judge twice, once in each order:
| column | meaning |
|---|---|
case_id | optional label (unique) |
verdict_original | what the judge said with response X in slot A and Y in slot B: A, B or tie |
verdict_swapped | what it said with the slots swapped (Y in A, X in B): A, B or tie |
len_first, len_second | optional: lengths of X and Y (any unit) for an extra "does the longer answer win?" stat |
Verdicts are the judge's literal slot answer. A fair judge's two verdicts are opposite (A→B).
Input (CSV or JSON). One row per response: case_id (optional), score_original (judge score for the response), score_padded (judge score for the same response with content-free filler appended). Set Score scale max if your scores are not 0–1 (e.g. 10 for a 1–10 scale).
Input (CSV or JSON). One row per judgment: score (judge confidence/score in 0–1) and label (1 if the human label says the response was good/correct, else 0).
Input (CSV or JSON). One row per item: optional case_id, then one column per judge (column name = judge name) with that judge's score for the item. At least two judges. The agreement matrix uses juryrig's within-ε rate and Cohen's κ.
- What runs here? juryrig's real audit code (v0.3.0 from PyPI), replaying the outputs you paste, inside this browser tab. Live audits that call a judge model (OpenAI, Anthropic, any HTTP endpoint) are available in the library and CLI (
juryrig cases.json) but need your own keys, so they are not part of this Space. - Privacy: the page only downloads Pyodide (jsDelivr CDN) and the juryrig wheel (PyPI). Pasted data is processed locally and never uploaded.
- Limits: a flagged audit means the evidence you pasted shows bias at the chosen thresholds; with few cases the numbers are noisy. Defaults are deliberately strict.
- The disagreement atlas appears automatically once a juryrig release after 0.3.0 ships it.