METHOD / BENCHMARK
A benchmark that keeps the difficult parts visible.
OpenDeco freezes the question, eligible evidence, model identity, evaluation mode and analysis plan before a public result becomes a benchmark release.
CautionNo benchmark run has been executed
Scope and eligibility
A release defines which domains, datasets, model implementations and configurations are eligible. Exclusions are recorded with reasons so a benchmark cannot quietly narrow itself around a preferred result.
Evaluation modes
Replay, compliance and counterfactual analyses answer different questions. OpenDeco labels the mode explicitly and does not present a counterfactual schedule as though it had an observed outcome.
Calibration overlap
Known use of an evaluation dataset in model development is recorded before scoring. Results on overlapping data remain visible when useful, but they are not described as independent validation.
In this repository today
The leakage registry carries 8 relationship types between one model version and one dataset version. Four of them force a non-independent verdict on their own, and two leave a real judgement to be recorded.
Unknown is the default lineage flag, and unknown never resolves into a claim that nothing was found. Inheritance raises a flag and never lowers one, so a model version cannot assert an independence its parent has not established.
Metrics
Metrics are chosen for the scientific question and versioned with the protocol. No single league-table number is allowed to conceal calibration, discrimination, burden, uncertainty or failure-region behaviour.
In this repository today
Each metric OpenDeco publishes carries the 9 properties spec 130 requires, including its expected range, its direction, its stated assumption and the limitation that goes with it. None carries an implementation version, because no code has computed one yet.
Failure regions
Aggregate performance is accompanied by inspection of exposure regions where errors or model divergence concentrate. Those regions are treated as research objects, not inconvenient outliers to hide.
Sensitivity
Configuration sweeps and analytical choices are tested where they can materially change the conclusion. A finding that disappears under reasonable settings is reported as fragile.
In this repository today
No sensitivity analysis has been run. A sweep planner exists and bounds the grid it will plan. What is missing is the analysis that consumes a sweep and decides whether a conclusion survives it.
External and leave-study-out validation
When multiple independent study sources exist, OpenDeco uses held-out or leave-study-out analysis where scientifically appropriate. The protocol distinguishes this from random record splitting inside a single historical programme.
In this repository today
This needs more than one independent study source, and OpenDeco holds none. No held-out or leave-study-out analysis has been attempted, and the clustering identifiers that would define the held-out unit are declared and unused.
Release
A public release is built from frozen inputs and immutable run manifests, checked for provenance and restricted-data leakage, rerun cleanly and assigned stable version metadata.
In this repository today
The publication gate runs the 11 conditions spec 37.1 blocks a release on, and every check reads artifacts instead of a declared intent flag. Its restricted-data check reads the path of every artifact in the release. Its checksum check recomputes every line of every run manifest.
Two of the 11 accept no override, because publishing restricted records or breaching a licence is a rule and not an editorial judgement. The gate has never been asked to pass a real release.