When You Can't Measure What Matters
A closed-loop harness for probability-of-default estimation.

The selective-labels problem

A probability-of-default model is graded on the loans a past policy approved — yet deployed on everyone. For every applicant the old policy declined, no repayment outcome ever forms: not delayed, not expensive, structurally absent. This is the selective-labels problem^[1], and it puts the one cohort where a new model most needs to be right — the applicants the old policy turned away — exactly where no ground truth exists.

The standard escapes each concede the problem they set out to solve:

Put a desk under the abstraction. A small-business lender funds 60-day daily-ACH loans — 3% origination fee, 5.75% of principal in term interest — approving about 60% of applicants under an incumbent policy, over a labeled pool that ran near a 17% default rate; these are the economics the harness's worlds mirror. A challenger PD model arrives, and the validation deck grades it the only place a grade exists: the booked loans. It wins there. But every row of that deck is a loan the incumbent already chose to fund — and the challenger's business case is the opposite region, the applicants the incumbent declined that the challenger would approve.

Cross the two policies and the deck's coverage becomes visible. Where both would fund, labels exist and the deck grades. Where the incumbent funded and the challenger declines, the avoided risk is measurable too. Where both decline, nothing changes and no claim is needed. That leaves one cell — the incumbent declined, the challenger would fund, and money moves — and it is exactly the cell in which no label has ever formed. That is the precise sense in which the model is deployed on everyone and graded on the approved: the gap is not missing data waiting on better collection, it is the structure of the decision itself, which is why each escape above circles back to an assumption about the very region under test. The harness's answer is not to close the gap in real data; it is to build a world in which the gap can be manufactured at a chosen severity — with the truth on the far side of it still in the harness's hand.

The harness: plant the truth, hide it honestly

If no real dataset can contain the declined cohort's labels, then build a world where you planted them. The harness generates synthetic lending cohorts with the true default outcome planted for every row, hides labels through the same mechanism a real approval policy would, and grades every correction technique against the truth it planted — not against a proxy. The loop then escalates selection severity round by round until the correction fails, and reports the last severity at which declined-cohort calibration still held: the operating frontier.

Three commitments separate this from a simulation that flatters its subject:

Two synthetic worlds, eight levers

Cohorts come from two generators: a flat synthetic world (independent features) and a fitted, layered structural causal model^[6] whose marginals are checked against real data by a fidelity gate — labeled marginal fidelity precisely so it never reads as joint or causal fidelity. Against them, eight correction levers: naive conditioning, inverse-propensity weighting^[5], disjoint retraining, bought-label exploration, and the four classic reject-inference correctors — reclassification, augmentation, fuzzy augmentation, parcelling^[2]. A counterfactual validator runs g-computation^[4] against naive conditioning over the same planted worlds.

One round, as it actually ran

Here is one round from the committed run — the SCM world, seed 42, severity 0.4, every number from artifacts/loop_frontier_scm.csv. Generate: 4,000 applicants, true default planted for every row — 17.5% of this cohort will default, and the harness knows which rows. The approval policy funds the lowest-risk 60% by a selection score; severity is how tightly that score tracks true latent risk — at 0 approval is random with respect to risk, at 1 it would track latent risk in full, including a component no feature column contains. At 0.4 the censoring is already violent: the 2,400 funded rows carry a planted default rate near 8.8%, the 1,600 declined rows 30.4%. Every declined label is then hidden — held by the harness, invisible to every model that follows.

Measure: the PD model trains on the 2,400 funded rows alone, then scores the 1,600 declined rows against the truth it never saw. It prices that pool at a mean PD of 21.3% where the planted rate is 30.4% — an underestimate no production system could see — and its declined-cohort ECE lands at 0.112, over the 0.10 target. Improve: the IPW lever fits a propensity model and upweights the funded rows that most resemble declines; declined-cohort ECE falls to 0.097, under target, and the round passes on the strength of that one lever. The disjoint-retrain lever, handed a fresh cohort with the same censoring, lands at 0.114 and fails the same bar — while clearing it comfortably in the flat world (0.092). A lever earns its keep per world and per severity, never in general.

Regenerate: severity rises to 0.6, and the lever that just earned its keep collapses — IPW declined-ECE 0.244 — while the observable alarm fires without touching a single declined label: the propensity model's effective-sample-size ratio drops 0.92 → 0.80 and the share of weights pinned at the clip floor jumps 0.13% → 7.5%. The loop stops; the frontier is 0.4, exactly the boundary Figure 3 draws. One lever can push past it: exploration — deliberately funding 157 applicants the policy would have declined, 56 of whom default, at a net cost of $439,578 — buys declined-cohort ECE 0.092 at severity 0.6. Identification can be bought instead of assumed; ≈$2,800 a label is what it costs in this world.

Results

Correction holds to 0.4 — and breaks at 0.6

At seed 42, both worlds land the operating frontier at severity 0.4: IPW-corrected declined-cohort ECE stays within the 0.10 target through 0.4 and fails at 0.6. The naive model — trained on approved rows, no correction — leaves the target zone a full severity step earlier. The frontier, not a headline accuracy, is the deliverable.

The frontier is a distribution, not a point

A single seed is an anecdote. Across the full 25-seed sweep, the published single-seed frontier (0.4, seed 42) sits at the optimistic end: in the SCM world the median frontier is 0.2, with only 11 of 25 seeds reaching 0.4; the flat world's median holds at 0.4 (15/25). The honest boundary is 0.2–0.4 depending on the draw — the prior headline was a valid instance, not a center.

Post-publication (2026-07-29): both sweeps were re-run on fully seed-disjoint "spaced" draws — and the overlapping set's SCM median of 0.2 does not replicate. On the independent draw both worlds center at 0.4; what survives is the range, 0.2–0.4 depending on the draw:

Both deliverables break at the same wall

The counterfactual deliverable tells the same story from the other side. Inside the frontier, g-computation^[4] earns its keep: −13.5% counterfactual MAE against naive conditioning, positive on 24 of 25 seeds (Wilcoxon p = 1.5e-7; on an independent spaced seed set: 22/25, p = 1.6e-6). At full severity the advantage collapses to a negligible +0.0017 — because the highest severity selects through an unobserved confounder, which defeats backdoor adjustment^[6] and IPW^[5] alike. One cause explains both failures: the calibration correction and the counterfactual advantage break at the same wall — selection through a variable no feature set contained. That is the essay's central claim, and it is a measured limit, not an argued one. Post-publication (August 2026): v4 made this attribution an intervention — the confounder's strength became an explicit axis — and the pre-registered test withdrew it: the wall stands with the confounder switched off. The boundary survives; the one-cause sentence did not. The closing section carries the ledger.

Two EMPs disagree — that's the finding

Calibration says whether the model is wrong on the declines; the expected-maximum-profit layer^[3] asks what being wrong costs. Two variants are computed from the same in-process scores: the literature's closed form over the ROC convex hull with its published convenience prior, and a harness-derived variant priced by the world's own loan economics and planted default timing. They disagree — and the disagreement is the result. The standard prior assumes ROI = 0.2644 where this 60-day loan structure actually returns 0.0875 (a 3.0× overstatement), and places 55% of defaults at full recovery where the harness plants ~1%. Priced honestly, profit on the declined pool collapses toward zero as severity rises; priced by the convenience prior, it appears to grow. Same model, same scores, opposite conclusion.

What this claims — and refuses to

The harness is a synthetic validation harness, not a production pipeline: retraining and feedback are seeded simulations inside the loop; it never acts on live data or real lending decisions. Its numbers hold inside their stated boundaries, and the boundaries ship with the results: the harness-economics EMP rests on planted-but-unfitted default timing — a verified experiment, not a verified result; ~22.5% of planted defaults land past the 60-day term and are priced at a stated convention; the frontier is a property of this model family in these worlds, reported as a distribution precisely so a lucky seed can't become a headline. Limits are reported, not smoothed over — that is the discipline the harness exists to enforce.

What an operating frontier changes is the sentence a PD model ships with. A validation deck closes on a score — AUC on the booked loans, a calibration chart over the same rows — measured precisely where the incumbent already decided, silent precisely where the challenger would decide differently. A frontier closes on a boundary statement: this correction held to severity 0.4, broke at 0.6, and the break announced itself. That last clause is the governable part. Declined-cohort calibration cannot be watched in production — that is the whole problem — but the conditions under which a correction was shown to fail can be: propensity AUC, effective-sample-size ratio, clip-floor share, none of which needs a declined-row label. In the committed runs those diagnostics crossed their calibrated thresholds exactly at the severity where correction failed, in both worlds; in the v0.3.0 feedback runs the declined-ECE alarm fired by generation 1–2 in every unexplored run — severity 0.4: 25 of 25 at generation 1. Governance stops being "trust the score" and becomes state the envelope, and watch the dials that told us when it broke — an envelope that ships with its error bar, 0.2–0.4 depending on the draw, because a seed is a draw, not a fact.

The other thing the loop changes is what happens to the claims after publication, because an escalating harness does not stop when the essay does. This page froze its numbers at v0.2.0; the loop kept running — and it is allowed to come back for its author. Three measurement campaigns have landed since, each committed beside the claims it tested. First, v0.3.0 priced the feedback regime — the model's own approvals creating its next training set, three arms over 25 spaced seeds — and the decomposition it exists for came back one-for-three, reported as measured:

Then the replication attack: both sweeps re-run on fully disjoint "spaced" seeds. The counterfactual advantage survives — +0.0129, positive on 22 of 25 seeds, Wilcoxon p = 1.6e-6 — while the SCM world's median frontier moves 0.2 → 0.4: a seed-overlap caveat the README had carried was real, and removing it moved the center. And last, v4 made the essay's central attribution an intervention: the unobserved confounder's strength became an explicit axis — 300 loop runs and 450 counterfactual evals against four pre-registered, Holm-corrected hypotheses. All four came back not confirmed. The wall stands with the confounder switched off — at strength 0 the flat world's frontier sits at 0.4 on 25 of 25 seeds — and the one-cause reading is withdrawn as measured, not softened.

That ledger is the closing argument. The replication moved a median and said so; the feedback decomposition confirmed one of its own three predictions and reported the other two as measured-opposite and not-confirmed; and the interventional test took this essay's central attribution and withdrew it. What survives is every boundary measurement — the frontier, its 0.2–0.4 range, the counterfactual collapse. What fell is the explanatory sentence. The harness has now disagreed with its author in public, on committed evidence — that is the property worth governing with.

Code & Data

The code and data section of this entry is /cldd — closed-loop default detection, the published harness behind every number above: version history with committed evidence, notable results recorded next to stated limitations, and frontier CSVs that recompute the headline.

Versions. v0.1.0 (2026-07-03) ships the harness — the loop, two worlds, eight levers, 123 tests. v0.2.0 (2026-07-14) adds the EMP measurement layer and the 25-seed sweep — 149 tests, every v0.1.0 column of the committed CSVs verified byte-identical. v0.3.0 (2026-07-22) adds the three-arm feedback-loop profit decomposition and its 450-run sweep — 203 tests. Committed at the repo tip, unreleased: the spaced-seed replication, the v4 unobserved_strength × severity surface with its verdicts, and a 13-claim doc-number gate that recomputes every quoted figure in CI — suite at 246. Full release notes, semver policy, and reading caveats live on /cldd.

Citation

Please cite this work as:

Or use the BibTeX citation:

References

1. Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017). The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. KDD ’17. pdf ↗
2. Hand, D. J., & Henley, W. E. (1993). Can reject inference ever work? IMA Journal of Mathematics Applied in Business and Industry, 5(1), 45–55.
3. Verbraken, T., Bravo, C., Weber, R., & Baesens, B. (2014). Development and application of consumer credit scoring models using profit-based classification measures. European Journal of Operational Research, 238(2), 505–513.
4. Robins, J. (1986). A new approach to causal inference in mortality studies with a sustained exposure period. Mathematical Modelling, 7(9–12), 1393–1512.
5. Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685.
6. Pearl, J. (2009). Causality: Models, Reasoning and Inference (2nd ed.). Cambridge University Press.
7. Pakdaman Naeini, M., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining Well Calibrated Probabilities Using Bayesian Binning. AAAI ’15.