Import a complete labeled inventory and a JSONL prediction file. Keep missing predictions, explicit abstentions, duplicate IDs, invalid lines and unknown truth visible alongside the usual classification scores. The entire audit runs locally; no model training/inference or remote upload.
Open index.html through a static server:
python3 -m http.server 8000 --bind 127.0.0.1
Use a modern browser with Web Crypto, module scripts and String.isWellFormed. Loopback and HTTPS supported; opening file:// directly is not supported. Native browser modules have no build/runtime dependencies. The CLI/tests were checked with Node24.19.0; use Node22 or newer with native Web Crypto and String.isWellFormed. File selections create a pending preview; explicit Apply replaces the active audit. A rejected import leaves the active audit intact. Clear/refresh loses page memory. No localStorage/checkpoint restore. Download includes all original text and predictions, not just displayed filters; keep it only where those inputs are appropriate.
Imagine checking a classifier before putting its predictions to use. The score says75% correct. Before trusting that number, I want to know which records reached the denominator: what happened to a missing prediction, an abstention, a duplicate id, or a record whose correct label is unknown?
I built Prediction Metric Desk to keep those records on the page. It compares the full inventory with the prediction file, retaining invalid lines and every duplicate occurrence. It does not repair missing outputs or turn hard labels into invented probabilities. A small authored fault example makes the difference visible: seven records have known truth, only four have usable label decisions, and three of those four are correct. That gives75% conditional accuracy with only4/7 decision coverage. The eighth record has unknown truth and stays in the inventory.
The tool also shows hypothetical completion bounds: if the three known-truth nondecisions later receive wrong or right labels, the completed accuracy would lie between3/7 and6/7. These are assumptions about unfinished decisions, not confidence intervals. Probability scores use another explicitly counted subset; the example's zero probability for a true class produces infinite log loss rather than a clipped, more comfortable number.
The larger demonstration reuses all3,080 already saved outputs from my Tiny Hash Intent experiment. It converts frozen int8 logits to probabilities at the already chosen temperature0.75; it does not run or train the model again. All301 errors and seven previously identified normalized train/test overlaps remain. The original independent review also tested a separate seven-record fixture and real complete downloads, including a filtered view whose export still kept every record. Those checks establish the accounting rules and export behavior, not fresh evidence of generalization or improved calibration.
Use the desk when you already have an inventory and predictions to inspect. Inputs live in page memory; refresh clears the audit. Downloads contain the original text and predictions, and a report cannot restore the session. The detailed contract below specifies each score's denominator, including the declared macro-F1 policy. An attractive aggregate is still only as useful as the records and assumptions behind it.
假设要检查一批分类结果,页面显示75%正确。要决定这个数字能不能用,还得知道没有预测、明确弃权、重复编号和正确答案未知的记录去了哪里。我做这个审计台,是为了把完整清单留在视野里,而不是只看顺利产出的那部分。
随包八条合成故障样例里,七条有已知正确标签,但只有四条得到有效的非弃权决定,其中三条正确。因此条件准确率是3/4=75%,已知标签记录的决定覆盖率却只有4/7。剩下一条正确答案未知,仍保留在完整清单中。这里没有真实客户或新的标注实验。
工具还显示一个假设范围:三条未决定的已知标签记录,如果后来全答错或全答对,完成后的准确率会在3/7到6/7之间。这不是置信区间,也不是未来成绩保证。概率指标另算有完整概率且正确标签已知的记录;样例把某个真实类别的概率设为零,log loss就保留无穷,不裁成一个更好看的有限值。
大演示复用Tiny Hash Intent原来保存的3,080条输出,只按原定温度0.75转换int8 logits,没有重新训练或调用模型。301个错误和已知的七条归一化训练/测试重叠都保留。原独立验收还用另造的七条样例核数学,并实际验证筛选页面后导出仍有完整记录。这证明的是计数和导出契约,不是新的泛化能力或校准改进。
文件只在页面内存中,刷新就清空。报告包含原文与预测,分享前要检查;它不是可恢复的检查点。下面保留各指标的分母和完整评分约定,便于把分数还原到具体记录上。
Complete contract. Each file UTF8/noBOM/≤10MiB. Inventory1–5000 unique records, ordered2–100 classes; prediction JSONL≤5000 physical lines, empty allowed. One final LF terminates a row; internal blank lines remain invalid. Every duplicate occurrence is invalid, including one with a malformed label/vector. Extra IDs never join the inventory denominator. JSON last-key semantics apply.
Inventory example:
{"schema_version":1,"kind":"classification-inventory","labels":["a","b"],"records":[{"id":"item-1","text":"Example","gold_label":"a"}]}
Prediction line:
{"id":"item-1","predicted_label":"a","probabilities":[0.8,0.2]}
Use null gold for unknown truth, null predicted_label for explicit abstention, null probabilities for hard-label-only output. Probability vectors must follow taxonomy order, be finite/in[0,1], sum within1e-8 of1. No renormalization, clipping, missing-value repair or invented one-hot scores. A policy label may differ from probability argmax; the desk reports it rather than replacing it.
CLI, output must not already exist:
node audit_cli.mjs data/fault-inventory.json data/fault-predictions.jsonl my-audit.json
Reports carry standalone formula references and exact bundled-source attribution; unmatched user origins/licenses remain unspecified. The report preserves source bytes as UTF8 text plus byte lengths and SHA256, complete inventory/aligned cases, all raw prediction lines/errors, full typed confusion, all classes and probability bins. Invalid parsed objects have value:null with raw/error preserved, rather than silently serializing malformed numeric values. Source hashes establish byte consistency, not authenticity or truth quality. Pretty JSON download≤64MiB; oversize fails explicitly.
Confusion uses typed class/status columns, so an actual class called missing cannot collide with a missing-prediction status. There are separate known/unknown status counts and full physical-line accounting; rejected malformed lines without a recognized ID cannot be guessed onto an inventory row.
The original8-case synthetic fault demo includes all nondecision states, one unknown truth, a literal class named missing, zero true-class probability, duplicate malformed label, extra ID, broken JSON, blank line and a label/argmax mismatch. Conditional accuracy3/4=75% with known-gold decision coverage4/7; completion bounds3/7–6/7. NLL is infinite. It is not a benchmark.
The real demonstration reuses all3080 original BANKING77 test outputs from work:tiny-hash-intent:v1, with77 ordered classes. It reconstructs probabilities from saved int8 logits at the original frozen temperature0.75, without model execution. Correct2779/3080 (90.2273%), macroF1 .9023831286, NLL .3555964155, Brier .1462308026, ECE10 .0117704422. All301 errors remain. The benchmark was already viewed; seven normalized train/test overlaps remain. This validates result accounting, not new generalization or calibration evidence. Source attribution, exact byte map and conventions: SOURCES.md, LICENSES.md.
Reproduce computational checks:
node test_audit.mjs
PYTHONDONTWRITEBYTECODE=1 python3 verify_independent.py
make_demo.py needs NumPy2.5.4 and only recomputes saved probability vectors; requirements-adaptation.txt pins that optional dependency. Test artifacts include independent original-logit comparisons over237160 probabilities, originalCSV row metrics and225 exhaustive two-record gold/output-state fixtures. VALIDATION.md records actual native browser import/download and limitations. Local verification is not a cloud deployment or adoption receipt.
Identity: work:prediction-metric-desk:v1, version0.1.0. Today’s candidate is developed for later independent acceptance and daily release; no claim of new public deployment. Future changes to this frozen candidate require scoped review.