19. Bitemporal Governance, Vault, and the Four Regimes of Drift

The Time-Traveler Backtest and the Overused Vault

Why do so many quantitative trading strategies, hospital early-warning scores, and flight-delay predictors boast $95%$ accuracy in the research notebook and crash during their first week in live production?

Two structural failures explain almost every production collapse:

1. The Time-Traveler in the Database (Bitemporal Leakage)

Suppose on May 1st an economist wants to train a model to predict whether the stock market will rise on May 2nd, using “April GDP growth” as a feature. In the historical SQL table downloaded three years later, the row labeled event_date = 2023-04-30 contains the final revised April GDP number. The model uses that number to predict May 2nd and achieves spectacular accuracy!

Why did the model fail when deployed live on May 1st? Because in the real world, the statistical bureau does not publish the first estimate of April GDP until May 28th, and does not publish the final revised number until July! On May 1st, that number did not exist in the universe yet.

Every datum in an operational system actually has two timestamps:

• Event Time ($t_{\text{ev}}$): When the physical event occurred in the World ($L_1$).
• Availability / Ingestion Time ($t_{\text{disp}}$): When the record was actually written, finalized, and available to the Agent ($L_3$) in the computer system ($t_{\text{disp}} \ge t_{\text{ev}}$).

Filtering a training set by $t_{\text{ev}} \le \tau$ instead of $t_{\text{disp}} \le \tau$ secretly grants your model a time machine.

2. Peeking at the Vault Until It Confesses

Even when timestamps are clean, suppose a research team reserves the last three months of 2023 as a “holdout test set.”

• They run Model 1 on the test set; it fails.
• They tweak the learning rate and feature set (Model 2) and run it on the same test set; it still fails.
• After $15$ rounds of adjustments, Model 15 passes on the test set, and the team publishes that test score as an “out-of-sample confirmation.”

As Cynthia Dwork and colleagues (2015) proved in adaptive data analysis, every time you peek at a holdout set and use the result to choose your next modification, information leaks from the holdout set into your model. After enough openings, the “test set” is just a second training set in disguise.

And once the model is finally live in production, the physical world itself begins to drift: passenger traffic shifts from 2013 to 2023 (nyc_flights_2013.csv $\to$ nyc_flights_2023.csv), a pandemic alters hospital baseline prevalence, or a regulatory change breaks the link between features and outcomes.

crates/stochology-governance builds the three defensive shields that protect live operations: the Bitemporal Clock & Canary (clock.rs), the Single-Opening Temporal Vault (vault.rs), and Multi-Regime Drift Adaptation & Circuit Breaking (drift.rs).

19.1 Bitemporal Clock ($t_{\text{disp}} \ge t_{\text{ev}}$) and the Synthetic Canary (stochology_governance::clock)

Enforcing Physical Causality in Data Pipelines

Every feature record entering BitemporalClock carries $(t_{\text{ev}}, t_{\text{disp}})$ with the physical invariant $t_{\text{disp}} \ge t_{\text{ev}}$ (a database cannot record a measurement before the event happens, and network/reporting latency $\Delta_{\text{lat}} = t_{\text{disp}} - t_{\text{ev}} \ge 0$ is non-negative).

When an agent makes a decision at operational wall-clock time $\tau$, the Bitemporal Clock admits a record if and only if:

$$t_{\text{disp}} \le \tau$$

Any record with $t_{\text{ev}} \le \tau < t_{\text{disp}}$ (an event that already happened in the physical world, but whose measurement has not yet arrived in the database at time $\tau$) is strictly quarantined from the feature vector.

The Synthetic Canary Column ($I(\text{Canary}; Y) = 0$)

How can an automated gate prove that a complex feature-engineering pipeline has not accidentally leaked future target information $Y_{t + h}$ into time $\tau$ through a buggy rolling window, a global normalization (StandardScaler fitted over the whole table), or a target-encoded join?

Borrowing the metaphor of the coal-mine canary, BitemporalClock injects a Synthetic Canary Column $C_{\text{canary}}$ into the feature table:

• For every causal record properly available at $t_{\text{disp}} \le \tau$, $C_{\text{canary}}$ is drawn from independent SplitMix64 noise that has zero mutual information with the target $Y$: $$I(C_{\text{canary}}; Y) = 0$$
• However, inside the forbidden future horizon $t_{\text{disp}} > \tau$, the canary is coupled to the future realization of $Y$.
• If the feature pipeline obeys strict bitemporal causality ($t_{\text{disp}} \le \tau$), the model sees only the pre-$\tau$ canary noise, so empirical mutual information stays at $I(C_{\text{canary}}; Y) \approx 0.0000$.
• The instant a buggy join, backfill, or look-ahead window reaches across $\tau$ into $t_{\text{disp}} > \tau$, the canary’s mutual information spikes ($I(C_{\text{canary}}; Y) = 0.6931\text{ nats} > \tau_{\text{canary}}$), tripping an immediate Temporal Intrusion Alarm (Gate G7 and --defect clock)!

19.2 The Single-Opening Temporal Vault (stochology_governance::vault)

To prevent adaptive overfitting on holdout data, TemporalVault wraps out-of-time (nyc_flights_2013_h1.csv, nyc_flights_2023_h1.csv) and out-of-domain (houston_flights_2011.csv) evaluation reserves inside a state machine governed by three irreversible rules:

1. Cryptographic Pre-Registration Seal and Keyed HMAC-SHA256 Authentication: A vault can be constructed only with a valid 64-character lowercase hexadecimal SHA-256 hash sealing the model weights, feature pipeline, and acceptance threshold before the vault is touched, paired with an RFC 2104 Keyed HMAC-SHA256 authentication tag (hmac_sha256_hex in crates/stochology-core/src/sha256.rs). On open_once(), TemporalVault recomputes the SHA-256 digest of the stored outcomes and pre-registration ID, immediately returning Err(VaultError::SealTampered) if a single bit was mutated after sealing.
2. Immutable Single-Opening Lock (opened_count: 0 -> 1): Calling open_once() increments opened_count from 0 to 1. Any subsequent attempt to call open_once() on the same vault is immediately blocked (--defect vault).
3. Automatic Epistemological Downgrade on Failure: If the model’s out-of-time Brier score or calibration error violates the pre-registered threshold upon opening, the study is automatically and permanently downgraded to ExploratoryHypothesisGenerating. The team may use what they learned to formulate a new hypothesis, but that new hypothesis must be tested on a brand-new, untouched future vault!

19.3 The Four Regimes of Production Drift and the Murphy-CORP Circuit Breaker

Foundations: Factoring the Joint Distribution $P(X, Y)$

Once a model is deployed in live operations, the joint distribution of features $X$ and outcomes $Y$ can change from the source training distribution $P(X, Y)$ to a new target production distribution $Q(X, Y)$.

By the product rule of probability, $P(X, Y)$ can be factored in two ways:

• Causal / Covariate Factoring: $P(X, Y) = P(X) P(Y \mid X)$
• Anti-Causal / Label Factoring: $P(X, Y) = P(Y) P(X \mid Y)$

This gives rise to Four Distinct Regimes of Production Drift, each requiring a completely different mathematical response:

The Murphy-CORP Circuit Breaker

How can an automated monitor tell the difference between a repairable shift (Regimes 2 & 3) and fatal Concept Drift (Regime 4)? Look back at the PAV isotonic decomposition from Lens 5 (Chapter 6):

$$\mathrm{BS} = \mathrm{REL}_{\mathrm{CORP}} - \mathrm{RES}_{\mathrm{CORP}} + \mathrm{UNC}_0$$

• Under pure Label Shift or monotone Covariate Shift, the overall base rate or marginal scale moves, causing Miscalibration to rise ($\Delta \mathrm{REL}_{\mathrm{CORP}} > \tau_{\mathrm{REL}}$), while the model’s ability to rank high-risk cases above low-risk cases (Resolution, $\mathrm{RES}_{\mathrm{CORP}}$) remains intact ($\Delta \mathrm{RES}_{\mathrm{CORP}} \approx 0$)! This shift can be repaired mathematically without throwing away the model.
• Under Concept Drift, the features $X$ lose their causal link to $Y$: the model’s ranking power collapses, causing Resolution to plunge ($\Delta \mathrm{RES}_{\mathrm{CORP}} < -\tau_{\mathrm{RES}}$)! No recalibration or reweighting can resurrect destroyed resolution. When $\Delta \mathrm{RES}_{\mathrm{CORP}} < -\tau_{\mathrm{RES}}$, MurphyCorpCircuitBreaker trips (tripped = true), halts automated action, and forces fallback to the conservative Wald Maximin policy (--defect concept_drift).

19.4 Adapting to Covariate Drift: ExTRA-WCP-T and the Lei & Candès Bound

When covariates shift ($Q_X \neq P_X$) while $P(Y \mid X)$ remains invariant, standard conformal prediction intervals lose their coverage guarantee because test points are drawn from $Q_X$ rather than $P_X$.

Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas (2019) and Lihua Lei and Emmanuel Candès (2021) showed that if we weight each calibration conformity score $V_i = |Y_i - \hat{\mu}(X_i)|$ by the density ratio $w(x) = \frac{dQ_X}{dP_X}(x)$, exact finite-sample validity is restored!

In fit_covariate_drift_extra_wcp (crates/stochology-governance/src/drift.rs), we implement Exponential-Tilt Weighted Conformal Prediction (ExTRA-WCP-T):

1. Exponential-Tilt Density Ratio Estimation: Rather than dividing two unstable high-dimensional density histograms, we model the log-density ratio linearly in a feature map $\phi(x)$: $$\hat{w}(x) = \exp\bigl(\hat{\theta}^\top \phi(x) - \hat{\psi}\bigr)$$ fitting $\hat{\theta}$ by minimizing the convex Bregman / logistic classification objective between source samples $X \sim P_X$ and target samples $X’ \sim Q_X$, and normalizing so that $\hat{Z}_P = \frac{1}{n}\sum_{i=1}^n \hat{w}(X_i) \approx 1$.
2. Weighted Conformal Quantile: Given normalized weights $p_i(x) = \frac{\hat{w}(X_i)}{\sum_{j=1}^n \hat{w}(X_j) + \hat{w}(x)}$, ExTRA-WCP-T computes the $(1-\alpha)$ weighted quantile of the conformity scores.
3. Lei & Candès (2021) Finite-Sample Coverage Floor: Even when the density ratio $\hat{w}(x)$ is estimated imperfectly, Lei and Candès proved that actual target coverage under $Q$ satisfies the exact lower bound: $$P_{Q}\bigl(Y_{n+1} \in \widehat{C}(X_{n+1})\bigr) \ge 1 - \alpha - d_{\mathrm{TV}}(P_X, Q_X)$$ where $d_{\mathrm{TV}}(P_X, Q_X) = \frac{1}{2}\int |dP_X - dQ_X|$ is the Total Variation distance between source and target covariates.
4. Gibbs & Candès (2021) Online Adaptive Conformal Inference (AdaptiveConformalInference): What if the operational data stream is completely non-exchangeable—undergoing continuous, unmodeled time-series regime shifts where no static density ratio $w(x)$ suffices? In AdaptiveConformalInference::evaluate_stream, the effective miscoverage quantile level $\alpha_t$ is updated online after each observed nonconformity score $S_t$ via step size $\gamma > 0$: $$\alpha_{t+1} = \mathrm{clamp}\Bigl(\alpha_t + \gamma\bigl(\alpha - \mathrm{err}_t\bigr),\ 0.001,\ 0.999\Bigr), \qquad \mathrm{err}_t = \mathbb{I}\bigl\lbrace S_t > \hat{Q}_{1-\alpha_t}(\mathcal{C}_{\text{cal}})\bigr\rbrace$$ By a telescoping-sum identity, Isaac Gibbs and Emmanuel Candès (2021) proved that long-run empirical miscoverage converges to $\alpha$ for any arbitrary sequence of observations without exchangeability: $$\left|\frac{1}{T}\sum_{t=1}^T \mathrm{err}_t - \alpha\right| \le \frac{1}{\gamma T}$$

19.5 Adapting to Label Shift: The Four Canonical Estimators (BBSE, MLLS/SLD, DFM, OTTER)

Now consider Label Shift: disease prevalence or delay base rate shifts from source prior $p_Y = P(Y = 1)$ to an unknown target prior $q_Y = Q(Y = 1)$, while $P(X \mid Y = y) = Q(X \mid Y = y)$ holds. How can we estimate the new target base rate $q_Y$ in live production before a single new ground-truth label $Y$ has arrived?

Because $Q(X) = \sum_{y \in \lbrace 0, 1 \rbrace} Q(X \mid Y = y) q_Y = \sum_{y \in \lbrace 0, 1 \rbrace} P(X \mid Y = y) q_Y$, the distribution of our classifier’s predictions on the unlabeled target stream shifts in an algebraically predictable way! To guard against estimator fragility, estimate_label_shift_four_methods computes all four canonical estimators simultaneously:

Once $\hat{q}_Y$ is estimated, every individual probability forecast $p(x) = P(Y=1 \mid X=x)$ is updated to the new target regime via Bayes’ prior-odds ratio:

$$Q(Y = 1 \mid X = x) = \frac{\frac{\hat{q}_Y}{p_Y} p(x)}{\frac{\hat{q}_Y}{p_Y} p(x) + \frac{1 - \hat{q}_Y}{1 - p_Y} (1 - p(x))}$$

Computational Audit in Rust

Run Laboratory 6 from your terminal to see the Bitemporal Clock, Synthetic Canary, Single-Opening Vault, ExTRA-WCP-T on real NYC flight shifts (2013 -> 2023), the 4 Label Shift estimators, and the Concept Drift Circuit Breaker:

cargo run --release --bin labs -- --governance

=== LAB 6: PRODUCTION GOVERNANCE: BITEMPORAL CLOCK, VAULT, AND 4 DRIFT REGIMES ===
  [Bitemporal Clock + Canary] enabled: MI_canary=0.0000, alarm=false | disabled (mutation): MI_canary=0.6931, alarm=true
  [Single-Opening Temporal Vault] sha256_seal=b451a7f0..8524795a, HMAC_tag=36cb9aff..ab71228a, Brier=0.0044, state=OpenedApproved
  [Covariate Drift: NYC Flights 2013->2023] beta_tilt=-0.1562, Z_P=0.9946, ExTRA_WCP_T_threshold=0.8178, Lei_Candes_bound >= 0.802 | Online_ACI_cov=0.891 (vs static=0.882)
  [Label Shift 4 Estimators (fit on 2013, TPR=0.620, FPR=0.249)] BBSE=0.0341 (sigma_min=0.094), MLLS/SLD=0.1161, DFM=0.0717 (Delta_min=0.6), OTTER=0.0691 (kappa=2.73) | Concept Drift Breaker=ConceptDriftCircuitBreaker (tripped=true)

And in Laboratory 7 (labs --real-data), the out-of-time H1 flight vaults (nyc_flights_2013_h1.csv $\to$ nyc_flights_2023_h1.csv) and the out-of-domain Houston reserve (houston_flights_2011.csv) verify:

• NYC $2013 \to 2023$ Lei & Candès bound $\ge 0.802$ (and online Gibbs-Candès ACI tracks $89.1%$ target $90%$ coverage vs. $88.2%$ static);
• H1 out-of-time reserve $2013 \to 2023$ Lei & Candès bound $\ge 0.828$;
• NYC $\to$ Houston out-of-domain reserve Lei & Candès bound $\ge 0.693$;
• H1 Temporal Vault returns OpenedApproved ($\mathrm{BS} = 0.1783$).

Exercises

1. (Concept — Bitemporal Asymmetry) A credit-card fraud model uses the feature customer_30_day_chargeback_count computed from the transactions table by WHERE transaction_date <= current_date. Given that banks often receive chargeback disputes from cardholders up to $45$ days after the transaction_date, explain why this SQL query leaks the future, and write the correct bitemporal condition using $t_{\text{ev}}$ and $t_{\text{disp}}$.
2. (Calculation — BBSE Label Shift) Suppose a binary classifier trained in 2023 has true positive rate $\mathrm{TPR} = P(\hat{Y}=1 \mid Y=1) = 0.80$ and false positive rate $\mathrm{FPR} = P(\hat{Y}=1 \mid Y=0) = 0.10$. In 2026, without any new ground-truth labels, you observe that the classifier predicts $\hat{Y} = 1$ on $\mu_1 = Q(\hat{Y}=1) = 0.31$ ($31%$) of incoming cases. Using the total probability identity $\mu_1 = \mathrm{TPR}\cdot q_Y + \mathrm{FPR}\cdot (1 - q_Y)$, solve for the new target prevalence $\hat{q}_Y$.
3. (Calculation — Lei & Candès Coverage Floor) If you construct a nominal $90%$ conformal prediction interval ($\alpha = 0.10$) and the Total Variation distance between source and target covariate distributions is $d_{\mathrm{TV}}(P_X, Q_X) = 0.072$, what is the guaranteed finite-sample coverage floor $1 - \alpha - d_{\mathrm{TV}}$?
4. (Laboratory — Governance Defect Proofs) Run cargo run --release --bin gates -- --defect clock, --defect vault, and --defect concept_drift and verify that all three exit with code 1.

Summary

• Bitemporal Governance (BitemporalClock) filters strictly by availability time $t_{\text{disp}} \le \tau$ ($t_{\text{disp}} \ge t_{\text{ev}}$) and injects a Synthetic Canary Column ($I(\text{Canary}; Y) = 0$) that triggers an immediate alarm if future leakage occurs.
• The Single-Opening Temporal Vault (TemporalVault) seals pre-registered evaluations with SHA-256, permits only a single opening (0 -> 1), and automatically downgrades failed confirmations to exploratory status.
• The Murphy-CORP Circuit Breaker separates repairable Covariate/Label shift ($\Delta\mathrm{REL}_{\mathrm{CORP}} > \tau_{\mathrm{REL}}$) from fatal Concept Drift ($\Delta\mathrm{RES}_{\mathrm{CORP}} < -\tau_{\mathrm{RES}}$).
• Covariate Drift is adapted via exponential-tilt ExTRA-WCP-T bounded by Lei & Candès’s $1 - \alpha - d_{\mathrm{TV}}(P_X, Q_X)$, while Label Shift is estimated across BBSE, MLLS/SLD, DFM, and OTTER.