the effect of running cldd

Your blind spot becomes a number.
The loop runs until the model breaks — and reports where.

A PD model is graded only on the loans a past policy approved, yet deployed on everyone. CLDD plants ground truth in a synthetic world, hides it the way your approval policy would, and escalates selection severity until correction fails — turning an unmeasurable blind spot into an operating frontier.

without

Declined-cohort calibration is unmeasurable — no labels exist for the applicants you turned away, exactly where the model must still be right.

with cldd

Operating frontier = 0.4 — the model holds through severity 0.4 and breaks at 0.6; honestly 0.2–0.4 across the 25-seed sweep, with the failure cause named: an unobserved confounder.

01generate 02measure 03improve 04regenerate

closed-loop default detection · import name cldd

Stress-test a PD model under selective labels —
and get the severity at which it breaks.

Real lending data only labels the loans a prior underwriter approved, so you cannot measure calibration on the applicants you declined — exactly where a new model must still be right. CLDD builds synthetic lending worlds with planted ground truth, hides labels the way real approval policies do, and grades every correction against that truth.

current v0.2.0 · alpha 149 tests, all synthetic deterministic — byte-identical per seed MIT · Python ≥ 3.10

install & 60-second tourpypi

Version history → Source ↗

How it works

One loop, escalated until correction fails.

The loop raises selection severity round by round and reports the operating frontier — the last severity at which declined-cohort calibration still holds (target ECE ≤ 0.10).

01 · generate→

Plant the truth

Build a synthetic cohort at a given selection severity; plant true default, then hide it via the approval policy.

02 · measure→

Score the blind spot

Train the PD model on approved rows only; score it against planted truth on the declined subpopulation.

03 · improve→

Apply a lever

IPW reweight · disjoint retrain · exploration · reject inference — every corrector graded against planted truth.

04 · regenerate⟲

Escalate or stop

Corrected declined-cohort ECE ≤ target? Raise the severity and go again. Otherwise stop and report the frontier.

Scope. CLDD is a synthetic validation harness, not a production pipeline: retraining and feedback are seeded simulations inside the harness; it never acts on live data or real lending decisions.

Version history

Two releases. Each one earns its number.

Semantic versioning, Keep-a-Changelog format, and a rule the releases actually follow: committed evidence first, headline second. Every number below recomputes from CSVs committed in the repo.

release indexchangelog.md · semver · pypi

v0.2.0 2026-07-14 latest · alpha The EMP measurement layer — price what being wrong on the declines costs details ↓ v0.1.0 2026-07-03 initial · alpha The harness — closed loop, two synthetic worlds, pluggable correction levers details ↓

v0.2.0 latest · alpha 2026-07-14 · 11 days after v0.1.0

The expected-maximum-profit (EMP) measurement layer: the loop now reports what being wrong on the declined population costs, not only whether it is miscalibrated.

Calibration says whether the model is wrong on the declines; v0.2.0 adds the axis that says what it costs — two EMP variants computed from the same in-process scores, strictly as reporting. ECE remains the sole loop-control metric, and every v1 column of the committed frontier CSVs is byte-identical.

major

minor ▲

patch

2 EMP variants

literature EMPC vs. harness-derived economics, side by side

25-seed sweep

the frontier reported as a distribution, not a point

+26 tests

149 total — hull cases, monotone invariance, float determinism

$2,800/label

exploration priced in dollars: 157 labels for $439,578

Notable result · 01

The frontier is a distribution, not a point. Across the full 25-seed set, the published single-seed frontier (0.4, seed 42) sits at the optimistic end: in the SCM world the median frontier is 0.2, with only 11/25 seeds reaching 0.4 (flat world: median 0.4, 15/25). The boundary is honestly 0.2–0.4 depending on the draw — the prior headline was a valid instance, not a center.

Notable result · 02

The two EMP variants disagree — and the disagreement is the finding. The literature's convenience prior assumes ROI = 0.2644 where this 60-day daily-ACH loan structure actually returns 0.0875 — a 3.0× overstatement — and places 55% of defaults at full recovery where the harness plants ~1%. Priced honestly, profit on the declined pool collapses toward zero as severity rises; priced by the standard prior, it appears to grow. Same model, same scores, opposite conclusion.

frontier distribution — artifacts/frontier_sweep.csv, 25 seeds
world	min	median	max	seeds at 0.4	seeds at 0.2
flat	0.2	0.4	0.4	15/25	10/25
SCM	0.2	0.2	0.4	11/25	14/25

the two EMP variants diverge — SCM world, declined subpopulation, seed 42
selection severity	0.0	0.2	0.4	0.6
literature empc (naive)	0.0217	0.0331	0.0416	0.0420 ↑
harness emp_h (naive)	0.0387	0.0297	0.0139	0.0024 ↓

Added 6 entries›

cldd.emp — two EMP variants over the same in-process scores, pure numpy, zero RNG: empc_literature (Verbraken et al. 2014 closed form over the ROC convex hull, with the source-verified prior) and emp_harness (this harness's own loan economics + planted per-row default timing; SCM cohorts only). Both are ranking-only: invariant to any strictly monotone transform of the score.
EMP columns on SubgroupMetrics / LeverMetrics and in the frontier CSVs, plus an EMP-vs-severity panel on the frontier plot.
RoundResult.exploration_cost — the exploration lever's bought labels priced in dollars (new column in artifacts/exploration_frontier.csv).
f1_emp_cutoff — F1 at the EMP-optimal cutoff, reported alongside F1 at the arbitrary 0.5 policy threshold (retained for continuity, now documented as diagnostic-only).
scripts/run_frontier_sweep.py — the frontier's distribution across the 25-seed set, replacing a single-seed point estimate; one subprocess per run, resumable.
26 new tests (149 total): hand-computed convex-hull cases, monotone-invariance, degenerate-timing cases, cross-process float determinism.

Changed 3 entries›

EMP is a reporting axis only — ECE remains the sole loop-control metric. Loop decisions, the frontier, and every v1 column of the committed frontier CSVs are unchanged (byte-identity verified; EMP columns strictly appended).
Renamed the loop driver and artifacts to drop pre-release lineage naming: run_clue.py → run_loop.py, clue_frontier* → loop_frontier* (history-preserving renames; CSV contents byte-identical).
Moved the accompanying article FABLE.md → docs/assessment.md (history-preserving; a dated provenance snapshot, excluded from the Sphinx build).

Fixed 1 entry›

ExplorationCorrector raised ZeroDivisionError when constructed with exploration_rate = 0.0 (the inverse-propensity weight was evaluated eagerly even though no row can be explored). Unreachable through SelectiveLabelsLoop, so no committed number changes.

Reading caveats 4 boundaries, stated›

Raw EMP moves with world hardness. A riskier declined pool changes EMP even for a perfect model — read the two variants against each other at fixed severity, not the trend in isolation.
emp_h rests on unfitted timing. days_to_default is planted but never validated against real recovery data — a verified experiment, not a verified result.
Post-term defaults are imputed — ~22.5% of planted defaults land past the 60-day term and are priced at the cohort's mean in-term loss fraction (a stated convention, not measured truth).
emp_h is SCM-only — the flat generator plants no timing, so its emp_h columns are empty by design.

full changelog ↗ pypi · 0.2.0 ↗ accompanying article ↗

v0.1.0 initial · alpha 2026-07-03 · first publish to PyPI

The selective-labels default-detection harness: the closed loop, two synthetic worlds, pluggable correction levers, and the frontier as the deliverable.

The initial alpha ships the whole mechanism: generate → measure → improve → regenerate, escalating selection severity until correction fails. Its headline holds inside the frontier and is explicitly not claimed beyond it: across 25 seeds, g-computation cuts counterfactual MAE −13.5% (positive on 24/25 seeds, Wilcoxon p = 1.5e-7) — and collapses to a negligible +0.0017 at full severity, where an unobserved confounder defeats both backdoor adjustment and IPW.

major

minor ●

patch

8 levers

naive, IPW, retrain, exploration + four reject-inference correctors

2 worlds

flat synthetic generator and the fitted, layered SCM

123 tests

all synthetic, no real data needed; CI on three gates

−13.5% MAE

g-computation vs naive, inside the frontier, 24/25 seeds

Headline result · seed 42

Both worlds land the operating frontier at severity 0.4 — the last severity at which IPW-corrected declined-cohort ECE stays ≤ 0.10 — and the counterfactual deliverable breaks at the same boundary. One cause explains both failures: selection through an unobserved confounder, which backdoor adjustment and IPW cannot fix. That single measured limit — not an unverifiable score — is the deliverable. (v0.2.0's sweep later shows this seed sits at the optimistic end.)

declined-cohort ECE vs severity — artifacts/loop_frontier*.csv, seed 42, target ≤ 0.10
selection severity	0.0	0.2	0.4	0.6
naive (flat world)	0.021	0.045	0.108	0.161
IPW-corrected (flat)	0.020	0.038	0.086 ✓	0.154 ✗
IPW-corrected (SCM)	0.036	0.038	0.097 ✓	0.244 ✗

Added 10 entries›

Closed loop (SelectiveLabelsLoop): the generate → measure → improve → regenerate cycle that escalates selection severity to find a PD model's operating frontier.
Pluggable correction levers (Corrector ABC): naive, IPW reweight, disjoint retrain, and exploration — plus the four classic reject-inference correctors (reclassification, augmentation, fuzzy augmentation, parcelling), graded against planted truth.
Two synthetic worlds: the flat SyntheticBorrowerGenerator and the fitted, layered StructuralBorrowerGenerator (SCM).
Marginal-fidelity gate (cldd.fidelity): compares SCM cohorts against real-data univariate marginals.
Counterfactual validator: a deployable g-computation estimator vs. naive conditioning.
Feedback / exploration simulation and observable positivity diagnostics — a regime/drift alarm that needs no declined-row labels.
Top-level export of the calibrated PD detector, CalibratedPDModel.
CalibratedPDClassifier: a scikit-learn estimator face for the calibrated PD detector — the full check_estimator battery passes on scikit-learn 1.7.2–1.9.0, probabilities byte-identical to the research API.
Packaging hygiene: cldd.__version__, a PEP 561 py.typed marker, coverage tooling.
Dedicated regression tests for model_pd.py and eval_default.py.

Changed 3 entries›

Fidelity gate output and docs relabeled MARGINAL so "fidelity PASSED" no longer reads as joint/causal fidelity.
DEFAULT_DATA_DIR overridable via CLDD_DATA_DIR; the private dataset is not shipped, and an absent dataset raises a clear, actionable error.
Development-status classifier: Beta → Alpha — the number says what the software is.

Fixed 2 entries›

CI float-determinism: a version-sensitive exploration test marked pinned, frozen-value asserts compare with tolerance — pinning dependency versions does not pin the BLAS/CPU, so boosted-tree output can drift by ~1 ULP across machines.
The strict Sphinx (-W) docs build no longer breaks on the internal findings doc.

Known limitations 2 boundaries, stated›

The real-data fidelity gate needs a private dataset and therefore does not run on public CI; coverage of the data-loading path is correspondingly low on CI by design.
scikit-learn estimator compatibility is provided through CalibratedPDClassifier only (binary-only; sample-weight equivalence not guaranteed). The loop-internal functional API stays outside the sklearn contract by design.

full changelog ↗ pypi · 0.1.0 ↗

Versioning policy

What a version number is allowed to claim.

Semantic versioning

MAJOR.MINOR.PATCH per semver.org. Pre-1.0, minor bumps carry new capability; the alpha label is kept on every release — the development-status classifier was deliberately moved down from Beta to Alpha in 0.1.0.

Keep a Changelog

Every release documents Added / Changed / Fixed — and, beyond the format, its notable results and stated limitations. What didn't hold is recorded next to what did.

Continuity guarantee

New measurement axes are strictly appended: v0.2.0's EMP columns never touch loop control, and byte-identity of every v0.1.0 column in the committed frontier CSVs was verified before release.

Reproduce the headline from committed evidence: python scripts/paired_significance.py. Exact pins for float-exact reproduction live in requirements-dev.txt.

Your blind spot becomes a number.The loop runs until the model breaks — and reports where.

Stress-test a PD model under selective labels —and get the severity at which it breaks.