LOCAL RESULT AUDIT · v0.1
Prediction Metric Desk
Keep every case in view. Check a multiclass prediction file against a complete inventory, including missing outputs, abstentions and invalid lines.
No training, inference or server upload. Inputs stay in this page’s memory; refresh clears them. Reports include the full original text and predictions.
1. Preview two files
UTF-8 without BOM, ≤10 MiB per file. 1–5,000 inventory records; ≤5,000 prediction lines. Probability order must match the supplied taxonomy.
Choose both files or preview a demonstration.
Pending preview
Ordered taxonomy
2. Active audit
Probability-only support
Valid full vectors with known gold, including explicit abstentions. ECE uses probability argmax, with first taxonomy label winning ties; reported labels are not replaced. Ten left-closed bins, last includes 1. No clipping or renormalization.
How to read these scores
Conditional accuracy excludes nondecisions. Completion bounds assume every known-gold nondecision later receives a wrong or right label; they are not confidence intervals. End-to-end macro F1 counts nondecisions as false negatives and averages every supplied class, with unsupported classes scored 0.
Unknown gold is retained but excluded from scoring. This report does not establish gold quality, calibrated confidence, independence or future performance. Hashes check consistency, not authenticity.
Probability bins and per-class scores
| Bin | Count | Argmax accuracy | Mean confidence | Gap |
|---|
| Class | Support | TP | FP | FN | F1 |
|---|
Confusion: all known-gold cases
| Output kind | Label or status | Count |
|---|
Every inventory case
All physical prediction lines
Full active report JSON
Data and conventions
The fault demo is original synthetic data with intentional failures. The BANKING77 demo reuses all 3,080 known original test outputs from work:tiny-hash-intent:v1, converting saved int8 logits with its frozen temperature 0.75. This is recalculation, not a new fit, inference run or blind benchmark. Seven normalized train/test overlaps remain; no rows are removed.
BANKING77: PolyAI; Casanueva, Temcinas, Gerz, Henderson and Vulić (2020), CC BY 4.0. Original code and synthetic fixture: Renchao Wu, Apache-2.0. Usage and limits · Input contract · Sources · Licenses
Brier is the mean sum of squared class errors, without binary half scaling. NLL uses natural-log mean; p(gold)=0 is infinite, unlike clipped implementations. ECE is a descriptive binned statistic, not a guarantee.