# Prediction Metric Desk v0.1 contract

Identity: work:prediction-metric-desk:v1. Written before implementation. Reusable single-run local multiclass result audit; no fitting, inference, threshold selection, new gold, blind benchmark or adoption claim.

## Inputs

Two UTF-8 files, no BOM, each <=10 MiB. Source bytes preserved and SHA256-bound in report. JSON.parse last-key semantics, never implicit schema repair. Inventory object exact keys schema_version:1, kind:"classification-inventory", labels, records. Labels ordered 2..100 unique nonblank scalar strings <=128 UTF16 units, trimmed, no ASCII control (U0000..001F or007F). No normalization/casefold/order inference. Records 1..5000, exact id/text/gold_label keys. IDs ASCII [A-Za-z0-9][A-Za-z0-9._:-]{0,127} over entire string; unique. Text scalar string <=8192 UTF16 units; arbitrary content including blank preserved. gold_label exact taxonomy label or null (unknown truth). Invalid inventory globally rejects.

Prediction JSONL <=5000 physical LF-separated rows; one final LF is terminator, not extra blank. Empty file valid. Internal blanks retained. CRLF accepted; raw source preserved. Each record exact keys id/predicted_label/probabilities; id same grammar, predicted_label taxonomy label or null explicit abstain. probabilities null (hard-only) or L finite numbers in [0,1], sum within1e-8 of1, never renormalized. Explicit abstain may retain probabilities. Reported label may differ from probability argmax, counted not overwritten. A bad line is retained invalid; an ID on any parseable object recognized before remaining validation, so all duplicate occurrences invalid even if another occurrence malformed in other fields. Unknown/extra IDs retained outside inventory metrics. Invalid JSON has no recovered ID. Global byte/UTF8/BOM/line-cap failure rejects new preview, preserving active audit.

## Metrics and full accounting

Each inventory record is predicted/abstained/missing/invalid; aligned in inventory order, all records retained. Duplicate matches invalid. All prediction physical lines retained independently with raw text/errors and resolved ID. Never hide bad/unassigned/extra rows in denominators.

N inventory; K known-gold inventory; E valid nonabstained decisions (including unknown gold); C known-gold valid nonabstained decisions; A correct among C; U=K-C. Coverage E/N, known-gold coverage C/K; conditional accuracy A/C. Zero denominator null. Completion bounds [A/K,(A+U)/K] assume every known-gold nondecision later gets a wrong/right label, not confidence intervals or guarantees. Unknown-gold records excluded from scoring, retained in inventory counts.

Confusion has K rows aggregated by ordered true class and typed output columns {kind:label,label} or {kind:status,status:abstained/missing/invalid}; actual class named "missing" cannot collide. Per-class TP/FP/FN use known gold, with nondecisions as FN. End-to-end macroF1 averages ALL supplied classes, unsupported class F1=0; null whenK0. A declared policy score, not conditional model F1. Counts also broken down by gold-known status.

D probability support = known gold with valid full probability vector, including abstentions but excluding invalid/extra/unknown-gold. NLL natural-log mean, no clipping; any p_gold0 produces value:null/infinite:true and explicit count (D0 value:null/infinite:false). Brier mean SUM across L classes, no /L or binary half-scaling, [0,2] up to sum tolerance. ECE fixed10 bins min(9,floor(maxprob*10)), left closed/right open except1 in last; argmax lowest ordered taxonomy index on ties. Bin correctness uses argmax, independently of reported hard label. Empty bin averages null. ECE weighted by D; D0 null. Not exactly paper bin defaults; no calibration fit/per-case guarantee or CI. No invalid finite/infinite numeric JSON values emitted.

## Product and report

No remote APIs, fonts, analytics, localStorage or writes until explicit native download. Two file selections produce a preview; explicit Apply replaces active audit. Any global failure clears stale preview/download without changing active report. All async selections generation-bound. Clear resets both. Filters/page changes only display, full report never filtered. Metrics show denominators/limitations, confusion selectable true-class row, full matrix/report. Report exact source strings/byte counts/SHA, full inventory/aligned cases/physical lines, taxonomy, typed confusion, metrics, probability bins, source and method provenance. Pretty UTF8 report <=64MiB; oversize export fails without retaining stale link. Report includes all original text; download explicitly disclosed; not a restorable annotation checkpoint and hashes not authentication.

## Demonstrations and validation

Small original synthetic fixture intentionally includes literal "missing" class, unknown truth, abstain with probabilities, missing predictions, all duplicate occurrences, malformed/blank/extra lines, zero gold probability, and hard-label/argmax mismatch. Hand counts fixed before core implementation.

Real demo uses same accepted TinyHash original3080 known BANKING77 test rows, 77 labels in original order, saved int8 logits/T0.75 ->stable softmax only. Verify source CSV argmax/confidence/p_gold/NLL/Brier every row, reuse exact source hashes. No model inference/training, no new benchmark; seven normalized overlaps and original limitations remain. Reused data/results CC BY4.0 PolyAI/Casanuevaetal2020, own code/synthetic Apache2.0. Research formula sources official scikit-learn Brier/log-loss docs and Guoetal2017 arXiv1706.04599v2; own zero handling differs from clipped sklearn helper. References no copied research prose/figures/code.

Independent hand/oracle/property checks: complete denominators, class/order collisions, duplicate/invalid/extra retention, null/Infinity, bin/tie/mismatch, malicious Unicode/IDs, byte/row caps, active-preservation/generation races, all3080 source-row comparisons. Actual native both-file imports and full report download, invalid imports, keyboard/375390/accessibility and console. Independent total owner review required before accepted count.
