6. The Five Transversal Lenses

Learning Objectives. By the end of this chapter, you will be able to audit any stochastic model through five universal lenses: (1) Information (Shannon entropy, mutual information, the Data Processing Inequality, the Strong DPI contraction hierarchy $\eta_{\chi^2} \le \eta_{\mathrm{KL}} \le \eta_{\mathrm{TV}}$, and Fano’s error floor); (2) Time (non-ergodicity, Itô’s variance drain $g_{\text{time}} = \mu - \frac{1}{2}\sigma^2$, the asymmetric Kelly criterion, and Log-Sobolev mixing times); (3) Scale and Tail (power-law moment regimes and the Hill tail-index estimator $\hat{\alpha}_{k,n}$); (4) Causal Structure and Invariance (Pearl’s three-rung Causal Ladder, confounders vs. Berkson colliders, Invariant Causal Prediction $\hat{S}^{\ast}$, and nonlinear NILE/IRM over an orthonormal Hermite basis); and (5) Decision and Calibration (the cost-loss threshold $p^{\ast} = \frac{C}{\eta L}$, $\mathrm{EVPI}$, and the exact isotonic CORP-Murphy decomposition via Pool-Adjacent-Violators).

Step 1 — Intuition and the Real-World Paradox

Suppose five domain specialists—a telecom engineer, a quantitative trader, a reinsurance actuary, an epidemiologist, and a hurricane forecaster—sit around a table. On the surface, their fields look completely unrelated. Yet when their models fail in the real world, they fail along the same five universal axes:

1. The Illusion of Creating Information by Reprocessing (Lens 1 — Information): An analyst takes a noisy survey table, runs it through three neural feature layers and two clustering algorithms, and believes the polished dashboard contains “more signal” than the raw sensors. In reality, no mathematical transformation can ever create information about the physical world that was not already captured at the sensor. Every stage of processing can only preserve or leak signal.
2. Confusing the Crowd at One Instant with One Person Across Time (Lens 2 — Time): Imagine a casino game where a fair coin is tossed every minute: on Heads your wealth increases by $+50%$ ($\times 1.50$); on Tails your wealth drops by $-40%$ ($\times 0.60$). If $10\text{,}000$ people play a single round, the average wealth of the room grows by $\frac{1.50 + 0.60}{2} = 1.05$ ($+5%$). It looks like a gold mine! Now play that exact same game yourself for $100$ consecutive rounds: half the time you multiply by $1.50$, half the time by $0.60$, and since $1.50 \times 0.60 = 0.90$, you lose $-10%$ of your wealth every two rounds until you are bankrupt! The average across the crowd ($g_{\text{ensemble}}$) and the growth rate of a single trajectory through time ($g_{\text{time}}$) are not the same number.
3. Treating a Giant Outlier as a “Six-Sigma Fluke” (Lens 3 — Scale and Tail): If you measure the height of $1\text{,}000$ randomly chosen adults, the tallest person might be $2.05\text{ m}$—barely $1.2\times$ the average—and changes the sample mean by a millimeter. If you measure the net worth of $1\text{,}000$ people or the size of $2\text{,}167$ fire insurance claims (danish_fire.csv), a single observation can be hundreds of times larger than the median and can outweigh half the dataset combined. In a heavy-tailed world, the sample variance does not converge: standard textbook $\pm 1.96\sigma/\sqrt{n}$ error bars are fiction.
4. Mistaking a Weathervane for the Wind (Lens 4 — Structure and Invariance): Looking at historical data, every time the barometer needle drops, rain follows. Correlation is high! What happens if we tape the barometer needle to “Sunny” (an intervention, $\mathrm{do}(\text{Barometer} = \text{High})$)? It rains anyway—because the atmospheric pressure caused both the needle and the rain. Worse still, what happens when we train a model inside a hospital and condition on hospitalization (a Berkson collider)? Two completely independent diseases suddenly appear correlated in our dataset simply because either one suffices to put a patient in a hospital bed.
5. Confusing a Bold Forecast with an Useful Decision (Lens 5 — Decision and Calibration): A weather app says “$80%$ chance of hurricane winds” when the true frequency on such days is only $25%$. If a town evacuates whenever the risk exceeds $p^{\ast} = 12%$, both $80%$ and $25%$ lie above $12%$, so the mayor orders the right evacuation today. Does that mean the $80%$ forecast is safe? The moment a neighbouring port uses the same app to decide whether to scuttle a cargo schedule whose break-even threshold is $p^{\ast} = 50%$, the overconfident $80%$ forecast triggers a ruinous false alarm!

These five universal audits are the Five Transversal Lenses (crates/stochology-lenses). Let us build each one from its foundations.

Step 2 — Foundations and Mathematical Formalization

6.1 Lens 1 — Information (stochology_lenses::information)

Foundations: Surprise, Entropy, KL Divergence, and Mutual Information

How do we measure “information” with a number? In 1948, Claude Shannon started from a simple intuition: observing an event that was almost certain ($p \approx 1$) tells you almost nothing, whereas observing a rare event ($p \ll 1$) carries high surprise, measured as $\ln\frac{1}{p(x)} = -\ln p(x)$ (in natural units, nats, or base-$2$ bits).

1. Shannon Entropy $H(W)$: The average uncertainty (expected surprise) of a random variable $W$ before we observe it: $$H(W) \triangleq -\sum_{w \in \mathcal{W}} P(w)\ln P(w) \ge 0$$
2. Kullback-Leibler (KL) Divergence $D_{\mathrm{KL}}(P \parallel Q)$: How distinguishable a distribution $P$ is from a reference distribution $Q$: $$D_{\mathrm{KL}}(P \parallel Q) \triangleq \sum_x P(x)\ln\frac{P(x)}{Q(x)} \ge 0$$
3. Mutual Information $I(W; X)$: How much observing a noisy signal $X$ reduces our uncertainty about the hidden state of the world $W$: $$I(W; X) \triangleq H(W) - H(W \mid X) = \sum_{w, x} P(w, x)\ln\frac{P(w, x)}{P(w)P(x)} \ge 0$$ If $X$ and $W$ are independent, $I(W; X) = 0$ (knowing $X$ tells you nothing about $W$).

The Data Processing Inequality (DPI) and Strong Contraction (SDPI)

Suppose the true state of the world $W$ generates raw physical data $X$, which an ingestion pipeline summarizes into features $Y$, which a model finally compresses into a decision score $Z$. Because each step depends only on the immediately preceding step, they form a 3-step Markov chain:

$$W \longrightarrow X \longrightarrow Y \longrightarrow Z$$

Shannon’s Data Processing Inequality (DPI) (DpiMarkovChain) proves that information can never increase along a Markov chain:

$$I(W; Z) \le I(W; Y) \le I(W; X) \le H(W)$$

Even stronger: when a noisy physical or computational channel $W(y \mid x)$ maps an input distribution $P_X$ to $P_Y$ and $Q_X$ to $Q_Y$, it strictly shrinks the distance between any two distributions! The Strong Data Processing Inequality (SDPI) measures the worst-case contraction factor $\eta_D(W) = \sup_{P \neq Q} \frac{D(P_Y \parallel Q_Y)}{D(P_X \parallel Q_X)} < 1$ across three divergence geometries:

• $\chi^2$-divergence contraction $\eta_{\chi^2}(W) = \rho_m^2(X; Y)$ (the squared Hirschfeld-Gebelein-Rényi maximal correlation);
• KL-divergence contraction $\eta_{\mathrm{KL}}(W)$;
• Total Variation contraction $\eta_{\mathrm{TV}}(W) = \sup_{x, x’} \lVert W(\cdot \mid x) - W(\cdot \mid x’) \rVert_{\mathrm{TV}}$ (Dobrushin’s coefficient of ergodicity).

The protocol audits the exact Ahlswede-Gács-Raginsky SDPI Hierarchy:

$$\eta_{\chi^2}(W) = \rho_m^2(X; Y) \le \eta_{\mathrm{KL}}(W) \le \eta_{\mathrm{TV}}(W) \le 1$$

Fano’s Error Floor

What happens when a pipeline leaks mutual information so that $I(W; Z)$ becomes small compared to the entropy $H(W)$ of the $|\mathcal{W}|$ possible states of the world? Robert Fano’s Inequality (1952) establishes an inescapable lower bound on the probability of classification error $P_e = P(\hat{W}(Z) \neq W)$ of any classifier, no matter how sophisticated:

$$H_2(P_e) + P_e \ln(|\mathcal{W}| - 1) \ge H(W \mid Z) = H(W) - I(W; Z)$$

where $H_2(p) = -p\ln p - (1-p)\ln(1-p)$ is binary entropy. In plain words: if your features $Z$ do not carry enough nats of mutual information $I(W; Z)$ to cover the entropy $H(W)$ of the target, no deep neural network can push error below Fano’s floor $P_e^{\min}$.

Blahut-Arimoto Exact Channel Capacity (blahut_arimoto_capacity)

While $I(W; X)$ measures the mutual information for a fixed input prior $P_W$, what is the absolute maximum information rate that a noisy physical channel $W(x \mid w)$ can transmit under any possible input distribution? In 1972, Richard Blahut and Suguru Arimoto independently derived an alternating convex-optimization algorithm that converges monotonically to Shannon’s Channel Capacity:

$$C(W) \triangleq \max_{P_W \in \Delta_{|\mathcal{W}|}} I(P_W, W)$$

By alternating between the variational posterior $Q^{(t)}(w \mid x) \propto P_W^{(t)}(w) W(x \mid w)$ and the tilted prior update $P_W^{(t+1)}(w) \propto P_W^{(t)}(w) \exp\bigl(\sum_x W(x \mid w)\ln\frac{W(x \mid w)}{\sum_{w’} P_W^{(t)}(w’)W(x \mid w’)}\bigr)$, blahut_arimoto_capacity (together with Laplace-smoothed contingency estimation in estimate_empirical_channel) computes the exact capacity $C(W)$ and capacity-achieving input prior $P_W^{\ast}$ in nats.

6.2 Lens 2 — Time and Survival (stochology_lenses::time)

Foundations: Ergodicity Breaking and Itô’s Variance Drain

In physics and probability, a process is called ergodic if the average across a large population of parallel trajectories at a single time $T$ (the ensemble average, $g_{\text{ensemble}}$) equals the average experienced by a single trajectory followed over a long period of time (the time average, $g_{\text{time}}$).

Additive games with stationary independent increments are ergodic. Multiplicative compounding is non-ergodic (Ole Peters, 2019).

Let us see why using Kiyosi Itô’s stochastic calculus. Suppose an asset, population, or capital reserve $S_t$ evolves via geometric Brownian motion with instantaneous drift $\mu$ and volatility $\sigma$:

$$\frac{dS_t}{S_t} = \mu dt + \sigma dW_t$$

1. The Ensemble Growth Rate ($g_{\text{ensemble}}$): If you average $S_t$ across infinitely many parallel universes, $\mathbb{E}[S_t] = S_0 e^{\mu t}$, so the expected wealth grows at rate: $$g_{\text{ensemble}} = \frac{1}{t}\ln \mathbb{E}\left[\frac{S_t}{S_0}\right] = \mu$$
2. The Time-Average Growth Rate ($g_{\text{time}}$): You do not live across a million parallel universes; you live in one universe over time! By Itô’s Lemma (which is simply the stochastic version of Jensen’s Inequality applied to the concave function $\ln S_t$), $d(\ln S_t) = \left(\mu - \frac{1}{2}\sigma^2\right)dt + \sigma dW_t$. Therefore, along almost every single path over time, the compounded growth rate is: $$g_{\text{time}} = \lim_{T \to \infty} \frac{1}{T}\ln\left(\frac{S_T}{S_0}\right) = \mu - \frac{1}{2}\sigma^2 < g_{\text{ensemble}}$$

The missing piece, $\frac{1}{2}\sigma^2$, is Itô’s Variance Drain (or volatility drag). Even if the average return $\mu$ is positive, if volatility is high enough that $\frac{1}{2}\sigma^2 > \mu$, the ensemble average $\mathbb{E}[S_t] \to \infty$ explodes to infinity while almost every individual player goes bankrupt ($S_t \to 0$)! Across $37\text{,}930$ daily trading sessions of the Dow Jones Industrial Average (djia_1885_2023.csv, 1885–2023), the annualized ensemble growth rate is $g_{\text{ensemble}} = 6.12%$, but Itô’s variance drain steals $1.45%$ per year, leaving a true compounded time-growth rate of $g_{\text{time}} = 4.68%$.

The Optimal Asymmetric Kelly Fraction and Absorbing Ruin

Because over-betting amplifies $\sigma^2$ quadratically while increasing $\mu$ only linearly, betting too large a fraction $f$ of your capital guarantees ruin. In 1956, Bell Labs physicist John L. Kelly Jr. derived the exact fraction $f^{\ast}$ of capital to risk on a repeatable gamble with win probability $p$, fractional loss $a$ on failure, and fractional gain $b$ on success so as to maximize long-run logarithmic time growth $\mathbb{E}[\ln(1 + R_t(f))]$:

$$\max_{f} \bigl[ p\ln(1 + b f) + (1 - p)\ln(1 - a f) \bigr] \implies f^{\ast} = \frac{p}{a} - \frac{1 - p}{b}$$

Notice two crucial boundaries of Kelly’s curve:

• At $f = f^{\ast}$, long-run compounded time-growth reaches its unique peak.
• At $f > 2 f^{\ast}$, time-average growth turns strictly negative even though every single bet has positive expected value!
• At $f \ge 1/a$, a single bad draw wipes out $100%$ of capital ($W_t \le 0$), hitting an absorbing ruin barrier from which no future recovery is possible.

Finally, Lens 2 audits how long a stochastic simulation or Markov chain takes to explore its state space via the Log-Sobolev Inequality (LSI) constant $\alpha_{\mathrm{LSI}} = -\frac{1}{2}\ln \eta_{\mathrm{KL}}$, Penalizing the mixing time whenever heavy-tailed mass $H_R$ traps the sampler in local modes.

Continuous-Return Empirical Kelly vs. the Gaussian Merton Ratio (empirical_kelly_optimal_fraction)

In continuous-time Gaussian finance, Robert Merton (1969) approximated the optimal leverage fraction by maximizing the second-order Taylor expansion $g(f) \approx f\hat{\mu} - \frac{1}{2}f^2\hat{\sigma}^2$, which yields the Merton Ratio $f_{\text{Merton}} = \frac{\hat{\mu}}{\hat{\sigma}^2}$. However, real empirical returns $r_1, \dots, r_n$ exhibit severe skewness and excess kurtosis (on djia_1885_2023.csv, excess kurtosis is $17.8$ and the Jarque-Bera statistic is $\mathrm{JB} = 502\text{,}551$, computed online via OnlineMoments). Because the logarithmic utility $\ln(1 + f r_t)$ diverges to $-\infty$ as $1 + f r_{\min} \to 0^+$, empirical_kelly_optimal_fraction solves the exact concave optimization problem via damped Newton-Raphson:

$$f_{\text{emp}}^{\ast} = \arg\max_{0 \le f < f_{\max}} \frac{1}{n}\sum_{t=1}^n \ln(1 + f r_t)$$

Whenever negative tail jumps exist, $f_{\text{emp}}^{\ast} < f_{\text{Merton}}$ (2.08 < 2.12 on DJIA), protecting the agent against Gaussian over-leverage.

6.3 Lens 3 — Scale and Tail (stochology_lenses::tail)

Foundations: Thin Tails vs. Power Laws and Moment Existence Regimes

In a Gaussian or exponential distribution (thin-tailed / “Mediocristan”), the probability of seeing an event larger than $x$ decays exponentially fast ($e^{-c x^2}$ or $e^{-c x}$). Extreme outliers are practically impossible.

In financial losses, reinsurance claims, city sizes, electrical grid spikes, and casualty counts (heavy-tailed / “Extremistan”), the upper tail decays slowly as a Pareto power law:

$$P(X > x) \sim C x^{-\alpha} \qquad \text{as } x \to \infty$$

The positive exponent $\alpha > 0$ is called the tail index. Why is $\alpha$ the single most important number in risk management? Because the $m$-th moment $\mathbb{E}[X^m] = \int_{x_0}^\infty m x^{m-1} P(X > x) dx \propto \int_{x_0}^\infty x^{m - 1 - \alpha} dx$ is finite if and only if $m < \alpha$! This splits all positive random variables into three sharply different physical worlds:

Bruce Hill’s (1975) Tail-Index Estimator

How do we estimate $\alpha$ from empirical data without parametric wishful thinking? Sort the sample of size $n$ in ascending order: $X_{(1)} \le X_{(2)} \le \dots \le X_{(n)}$. If the top $k$ observations above threshold $X_{(n-k)}$ follow a Pareto tail, then by a logarithmic transformation their log-spacings $\ln(X_{(n-i+1)} / X_{(n-k)})$ behave like exponential random variables with rate $\alpha$. Maximum likelihood estimation yields Hill’s Estimator (HillTailDiagnostic):

$$\hat{\alpha}_{k,n} = \left(\frac{1}{k}\sum_{i=1}^k \ln\frac{X_{(n-i+1)}}{X_{(n-k)}}\right)^{-1}, \qquad \mathrm{SE}(\hat{\alpha}_{k,n}) \approx \frac{\hat{\alpha}_{k,n}}{\sqrt{k}}$$

Applied to the $n = 2\text{,}167$ industrial fire insurance losses of Copenhagen Re (danish_fire.csv), Hill’s estimator yields $\hat{\alpha}_{\mathrm{Hill}} = 1.6009 \pm 0.1601$, placing fire claims squarely inside the FiniteMeanInfiniteVariance ($1 < \alpha \le 2$) regime, with $99\text{th}$ percentile $q_{99} = 26.04\text{M DKK}$ and an effective tail sample size of only tail_n_eff $= 13.9$.

Dekkers-Einmahl-de Haan Moment Estimator, Peaks-Over-Threshold GPD, and Expected Shortfall ($\mathrm{ES}_{0.99}$)

To cross-check Hill’s Pareto estimator across all three Extreme Value Theory (EVT) domains of attraction ($\xi > 0$ Fréchet, $\xi = 0$ Gumbel, and $\xi < 0$ Weibull, where $\xi = 1/\alpha$), crates/stochology-lenses/src/tail.rs implements three complementary EVT diagnostics:

1. Dekkers-Einmahl-de Haan (1989) Moment Estimator (dedh_moment_tail_index): Using the first two log-spacing moments $M_n^{(j)} = \frac{1}{k}\sum_{i=1}^k \bigl(\ln X_{(n-i+1)} - \ln X_{(n-k)}\bigr)^j$ ($j \in \lbrace 1, 2 \rbrace$), the DEdH generalized extreme-value shape estimator is: $$\hat{\xi}_{\mathrm{DEdH}} = M_n^{(1)} + 1 - \frac{1}{2}\left(1 - \frac{(M_n^{(1)})^2}{M_n^{(2)}}\right)^{-1} \in \mathbb{R}$$
2. Pickands-Balkema-de Haan Peaks-Over-Threshold GPD (gpd_pot_fit): By the Pickands-Balkema-de Haan Theorem (1974/1975), excesses $Y = X - u \mid X > u$ above a high threshold $u$ converge to a Generalized Pareto Distribution $G_{\xi, \beta}(y) = 1 - (1 + \xi y / \beta)^{-1/\xi}$, calibrated via Hosking-Wallis probability-weighted moments (L-moments).
3. Coherent Expected Shortfall ($\mathrm{ES}_{0.99}$, expected_shortfall_99): Unlike Value-at-Risk ($q_{99}$), which ignores the severity of losses beyond the $99\text{th}$ percentile and violates Artzner et al.’s (1999) subadditivity axiom, Expected Shortfall $\mathrm{ES}_{0.99} = \mathbb{E}[X \mid X \ge q_{0.99}]$ is a coherent spectral risk measure. On danish_fire.csv, while $q_{99} = 26.04\text{M DKK}$, the tail expectation jumps more than twofold to $\mathrm{ES}_{0.99} = 58.59\text{M DKK}$ (and $\mathrm{ES}_{0.99}^{\mathrm{POT}} = 58.88\text{M DKK}$)!

6.4 Lens 4 — Structure and Invariance (stochology_lenses::invariance)

Foundations: Pearl’s Causal Ladder, Confounders, and Berkson Colliders

Standard regression and machine learning live entirely on the first rung of Judea Pearl’s Three-Rung Causal Ladder:

Why is $P(Y \mid X = x)$ (Rung 1) so often completely different from $P(Y \mid \mathrm{do}(X = x))$ (Rung 2)? Consider the two elementary causal graphs that trap analysts every day:

1. The Confounder ($X \leftarrow C \to Y$): Variable $C$ (say, outdoor summer temperature) causes both ice-cream sales ($X$) and swimming-pool drownings ($Y$). In passive data, $X$ and $Y$ are strongly correlated. If you ban ice cream ($\mathrm{do}(X = 0)$), drownings do not drop at all! To identify the causal effect of $X$ on $Y$, you must condition on the confounder $C$ to block the back-door path.
2. The Berkson Collider ($X \to B \leftarrow Y$): Now flip the arrows! Suppose $X$ (locomotor disease) and $Y$ (respiratory disease) are completely independent in the general population, and either condition can cause a patient to be admitted to a hospital ($B = 1$). If an analyst restricts their dataset only to hospitalized patients (conditioning on $B = 1$), learning that a hospitalized patient does not have respiratory disease ($Y = 0$) immediately raises the probability that they must be in the hospital because of locomotor disease ($X = 1$)! Conditioning on a common effect (collider) creates a mirage of correlation out of thin air (Joseph Berkson, 1946). That is why Filter 2 of the Stochology Funnel strictly bans conditioning on a Berkson collider.

Invariant Causal Prediction (ICP) and Nonlinear NILE/IRM

How can we discover which predictors $X_S$ are true causal parents of $Y$ rather than spurious proxies or descendants when we observe data across multiple environments or regimes $e \in \mathcal{E}$ (for example, different decades, climate regimes, or geographical basins)?

Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen (2016) introduced a profound principle called Invariant Causal Prediction (ICP):

If $S^{\ast}$ is the set of true direct causal parents of $Y$, then the conditional mechanism $P^e(Y \mid X_{S^{\ast}})$ remains invariant across all environments $e \in \mathcal{E}$ that do not directly intervene on $Y$ itself.

By contrast, any subset $S$ that relies on a confounder, a collider, or a regime-specific proxy will exhibit changing regression coefficients or changing residual distributions when the environment $e$ shifts! ICP identifies the identifiable causal core by taking the intersection of all statistically invariant subsets:

$$\hat{S}^{\ast} = \bigcap_{S : H_{0, S} \text{ not rejected across } \mathcal{E}} S$$

In crates/stochology-lenses/src/invariance.rs, the protocol extends linear ICP (env_ols_invariance_penalties) to Nonlinear ICP and NILE/IRM (run_nonlinear_icp_nile) by expanding standardized predictors onto the 3-atom orthonormal probabilists’ Hermite basis:

$$\Phi(z) = \left[ z ,\quad \frac{z^2 - 1}{\sqrt{2}} ,\quad \frac{z^3 - 3z}{\sqrt{6}} \right], \qquad \mathbb{E}_{Z \sim \mathcal{N}(0,1)}[\Phi_j(Z)\Phi_k(Z)] = \delta_{jk}$$

and auditing the Invariant Risk Minimization (IRM / NILE) risk-gradient penalty $\sum_{e \in \mathcal{E}} \lVert \nabla_{w \mid w=1.0} R^e(w \cdot \Phi) \rVert^2$ alongside Paul Rosenbaum’s hidden-confounding sensitivity threshold $\Lambda^{\ast}$ (the minimum odds ratio an unobserved confounder would need to overturn the conclusion).

In addition, crates/stochology-lenses/src/invariance.rs provides two classic econometric invariance and sensitivity audits:

• Gregory Chow’s (1960) Structural Break $F$-Statistic (chow_invariance_f_stat): Tests $H_{0, S}: \beta^{(e)} = \beta^{\text{pooled}}$ across $E$ environments via $F_S = \frac{(\mathrm{RSS}_{\text{pooled}} - \sum_{e=1}^E \mathrm{RSS}_e) / ((E - 1)p)}{\sum_{e=1}^E \mathrm{RSS}_e / (N_{\text{total}} - E p)}$. For the true causal parent $X_0$, $F_{\lbrace 0 \rbrace} = 0.18$ (invariant), whereas for the spurious proxy $X_1$, $F_{\lbrace 1 \rbrace} = 23\text{,}437.0$ (structural breakdown!).
• Cinelli & Hazlett (2020) Omitted-Variable Bias Robustness Value (cinelli_hazlett_robustness_value): Computes the exact partial-$R^2$ strength $RV_q = \frac{1}{2}\bigl(\sqrt{f_q^4 + 4 f_q^2} - f_q^2\bigr)$ (where $f_q = q\cdot |t| / \sqrt{\mathrm{df}}$) that an unobserved confounder $U$ would need with both treatment $X$ and outcome $Y$ to reduce the estimated effect by fraction $q$.

6.5 Lens 5 — Decision and Calibration (stochology_lenses::decision)

Foundations: The Cost-Loss Boundary ($p^{\ast}$) and Value of Perfect Information ($\mathrm{EVPI}$)

A probability forecast $p \in (0, 1)$ is not yet an action. To cross from belief to action on Bridge $L_2 \to L_3$, we must price the errors.

Consider a decision-maker who must choose between two acts—protect (e.g., evacuate a coast, staff extra nurses, hold a flight) or do_not_protect—before a binary adverse event $Y \in \lbrace 0, 1 \rbrace$ occurs with forecast probability $p = P(Y = 1)$:

Comparing the two expected costs shows that protect is strictly better than do_not_protect whenever $C + p(1 - \eta)L < pL$, which simplifies algebraically to the Cost-Loss Boundary:

$$p > p^{\ast} \triangleq \frac{C}{\eta L}$$

Look at the intuition of $p^{\ast} = \frac{C}{\eta L}$:

• If protecting costs $C = USD 12\text{,}000$, a storm surge causes loss $L = USD 125\text{,}000$, and early evacuation mitigates $\eta = 0.80$ ($80%$) of the avoidable damage ($\eta L = USD 100\text{,}000$), then the critical threshold is $p^{\ast} = \frac{12\text{,}000}{100\text{,}000} = 0.1200$ ($12%$).
• You do not wait until a hurricane is “$50%$ likely” to evacuate! Because the damage $\eta L$ is more than $8\times$ the cost of protection $C$, it is rational to protect the moment the calibrated probability passes $12%$.

Furthermore, before spending money on a complex predictive project, we compute the Expected Value of Perfect Information ($\mathrm{EVPI}$)—the maximum economic gain an infallible oracle could ever deliver over acting on the base rate $p_0$ alone:

$$\mathrm{EVPI} = \min\bigl(p_0 \eta L, C\bigr) - \max\bigl(0, p_0 \eta L - C\bigr) = \begin{cases} p_0(\eta L - C) & \text{if } p_0 \le p^{\ast} \text{ (or } p_0 \eta L \text{ net)}, \\ C(1 - p_0) & \text{if } p_0 > p^{\ast}. \end{cases}$$

Specifically, under the cost table above, an oracle saves $C$ whenever $Y=0$ (if you would otherwise protect) or saves $\eta L - C$ whenever $Y=1$ (if you would otherwise not protect). If the total engineering and operational cost of building the model exceeds $\mathrm{EVPI}$, Filter 1 of the Stochology Funnel kills the project immediately.

Calibration vs. Resolution: The Exact Isotonic CORP-Murphy Decomposition

How do we audit whether a forecaster’s probabilities $p_1, \dots, p_n \in (0, 1)$ against realized binary outcomes $y_1, \dots, y_n \in \lbrace 0, 1 \rbrace$ are honest?

We start with Glenn Brier’s (1950) strictly proper Brier Score:

$$\mathrm{BS} = \frac{1}{n}\sum_{i=1}^n (p_i - y_i)^2$$

In 1973, Allan Murphy showed that the Brier Score splits into three meaningful physical components:

1. Miscalibration / Reliability ($\mathrm{REL}$): When the model says $p = 30%$, how far is the true empirical frequency $\hat{\pi}(p)$ from $0.30$? Small $\mathrm{REL}$ is good ($0$ is perfect calibration).
2. Resolution / Discrimination ($\mathrm{RES}$): How strongly do the calibrated frequencies $\hat{\pi}(p)$ separate high-risk days from low-risk days around the overall base rate $\bar{y}$? Large $\mathrm{RES}$ is good ($0$ means a useless constant forecast).
3. Base Uncertainty ($\mathrm{UNC}_0 = \bar{y}(1 - \bar{y})$): The intrinsic variance of the binary outcome in the World ($L_1$).

Historically, analysts estimated $\mathrm{REL}$ and $\mathrm{RES}$ by chopping $[0, 1]$ into $10$ arbitrary histogram bins—a procedure that breaks the exact algebraic identity and changes its answer whenever someone shifts the bin boundaries!

In 2021, Timo Dimitriadis, Tilmann Gneiting, and Alexander Jordan solved this fifty-year problem with the CORP (Consistent, Optimally Binned, Reproducible) Decomposition. Instead of arbitrary bins, CORP computes the unique non-parametric isotonic regression $\hat{\pi}_1 \le \hat{\pi}_2 \le \dots \le \hat{\pi}_n$ of the sorted outcomes $y_{(i)}$ on the sorted forecasts $p_{(i)}$ via the Pool-Adjacent-Violators (PAV) algorithm (CorpPavDecomposition in crates/stochology-lenses/src/decision.rs).

Because PAV partitions the sorted data into optimal monotone blocks where the block average equals the conditional frequency, the CORP-Murphy Identity closes to machine precision ($\sim 10^{-15}$):

$$\mathrm{BS} = \underbrace{\frac{1}{n}\sum_{i=1}^n (p_{(i)} - \hat{\pi}_i)^2}_{\mathrm{REL}_{\mathrm{CORP}} \text{ (Miscalibration)}} - \underbrace{\frac{1}{n}\sum_{i=1}^n (\hat{\pi}_i - \bar{y})^2}_{\mathrm{RES}_{\mathrm{CORP}} \text{ (Resolution)}} + \underbrace{\bar{y}(1 - \bar{y})}_{\mathrm{UNC}_0 \text{ (Uncertainty)}}$$

Remember the fundamental operational asymmetry of this equation:

• $\mathrm{REL}_{\mathrm{CORP}}$ (Miscalibration) can be repaired post-hoc by monotone recalibration (such as PAV isotonic mapping or Platt scaling).
• $\mathrm{RES}_{\mathrm{CORP}}$ (Resolution) cannot be created post-hoc by any monotone rescaling—because any strictly increasing map $p \mapsto g(p)$ produces the exact same PAV isotonic blocks $\hat{\pi}_i$! To increase $\mathrm{RES}_{\mathrm{CORP}}$, you must go back to $L_1$ and collect better causal predictors.
• Richardson’s (2000) Relative Economic Value (CostLossRule::relative_economic_value) and Good’s (1952) Logarithmic Score (logarithmic_score): Alongside the Brier decomposition, Lens 5 evaluates the normalized economic value of a forecast system relative to climatology and a perfect oracle at threshold $p^{\ast} = \frac{C}{\eta L}$: $$V(p^{\ast}) = \frac{\mathbb{E}[\text{Cost}_{\text{climatology}}] - \mathbb{E}[\text{Cost}_{\text{forecast}}]}{\mathbb{E}[\text{Cost}_{\text{climatology}}] - \mathbb{E}[\text{Cost}_{\text{oracle}}]} \in (-\infty, 1]$$ as well as I. J. Good’s local strictly proper Logarithmic (Ignorance) Score $\mathrm{LogS}(p, y) = -\frac{1}{n}\sum_{i=1}^n \bigl[y_i \ln \tilde{p}_i + (1 - y_i)\ln(1 - \tilde{p}_i)\bigr]$ safeguarded by Cromwell’s floor $\tilde{p}_i \in [\varepsilon_{\text{floor}}, 1 - \varepsilon_{\text{floor}}]$.

Step 3 — Auditable Laboratory in Rust and Exercises

Run Laboratory 3 from your terminal to execute all five transversal lenses on both synthetic benchmarks and real empirical data:

cargo run --release --bin labs -- --lenses

=== LAB 3: THE FIVE TRANSVERSAL LENSES (SDPI/DPI, TIME/KELLY, HILL, ICP, CORP-PAV) ===
  [Lens 1: Information] SDPI: eta_chi2=0.4900 <= eta_KL=0.4900 <= eta_TV=0.7000 (valid=true, C_BA=0.2704 nats) | DPI W->X->Y->Z: 0.3681 >= 0.1662 >= 0.0576 (monotonic=true) | I_amp=0.5185 | Fano P_e >= 0.1156
  [Lens 2: Time] DJIA (37930 trading days, kurtosis=17.8, JB=502551): g_ensemble=0.0612 vs g_time_Ito=0.0468 (drain=0.0145) | Kelly f*=0.8600 (emp_DJIA=2.08 vs Merton=2.12) | Mixing steps=13
  [Lens 3: Tail] Danish Fire (n=2167): Hill alpha_k=1.6009 (+/- 0.1601, DEdH_xi=0.538, GPD_xi=0.483), regime=FiniteMeanInfiniteVariance, median=1.778, MAD=0.574, q99=26.04, ES99=58.59 (POT=58.88), tail_n_eff=13.9 | VicElec (n=52608): q99=6981.5 MW
  [Lens 4: Invariance] ICP S*=[0], env_OLS_penalty=["0.0001", "15.8059"] (ChowF=[0.18, 23437.0]), Storms_nonlinear_NILE_IRM=["0.0011", "0.1072"], Lambda*=4.752 (RV=0.511) | AuthenticForm(S*=[0], discarded='atom_1_regime_shifted_lat_proxy', A=true)
  [Lens 5: Decision] CORP-PAV (Atlantic Storms, n=20778): BS=0.26906 = REL(0.21135) - RES(0.13102) + UNC0(0.18873) [id_err=1.53e-14, LogS_PAV=0.1849, REV_PAV=0.719] | FairCoin(p0=0.50): agree@0.30=100%, REL_over=0.1104 vs REL_cal=0.0005, regret@0.70=+18.20
  [Almost-Perfect Machine M=(T,Phi,theta)] sup_err=0.0234 (Barron C_f=5.60), DPI_gap=0.0297 nats, regions=64 (ratio=5.3x), d_eff=8.00/8, P1..P5=true
  [Relational Ontology & Complex Systems] MI_rel=0.3681 vs MI_iso=0.0003 nats | Prigogine dS=-0.55 (turnover=100.0%, closure=100.0%) | Schelling(tau=0.125->50.2% vs tau=0.375->82.4%) | Daisyworld(MAE=1.52C vs dead=9.57C, dT/dL=-11.93) | verified=true

Look at what each lens reveals:

1. Lens 1 (Information): Across the 3-step Markov chain $W \to X \to Y \to Z$, mutual information decays monotonically ($0.3681 \ge 0.1662 \ge 0.0576\text{ nats}$), the SDPI contraction constants obey $\eta_{\chi^2}(0.4900) \le \eta_{\mathrm{KL}}(0.4900) \le \eta_{\mathrm{TV}}(0.7000)$ with Blahut-Arimoto channel capacity $C_{\mathrm{BA}} = 0.2704\text{ nats}$, and Fano’s inequality forces at least $11.56%$ classification error.
2. Lens 2 (Time): Over $37\text{,}930$ trading sessions of the Dow Jones (djia_1885_2023.csv, excess kurtosis $17.8$, Jarque-Bera $\mathrm{JB} = 502\text{,}551$), Itô’s variance drain reduces the ensemble growth rate of $6.12%$ down to a time-average growth rate of $4.68%$, while exact continuous empirical Kelly leverage ($2.08$) is strictly more conservative than the Gaussian Merton ratio ($2.12$).
3. Lens 3 (Tail): On danish_fire.csv ($n = 2\text{,}167$), the Hill estimator finds $\hat{\alpha} = 1.6009 \pm 0.1601$ (FiniteMeanInfiniteVariance), corroborated by Dekkers-Einmahl-de Haan $\hat{\xi}_{\mathrm{DEdH}} = 0.538$ and Peaks-Over-Threshold GPD $\hat{\xi}_{\mathrm{GPD}} = 0.483$, with $99%$ Expected Shortfall $\mathrm{ES}_{0.99} = 58.59\text{M DKK}$ (POT = 58.88M).
4. Lens 4 (Invariance): Across shifting regimes, ICP isolates the true causal parent $\hat{S}^{\ast} = [0]$ with tiny invariance penalty ($0.0001$ linear, Chow $F = 0.18$; $0.0011$ nonlinear NILE/IRM on atlantic_storms.csv) while exposing the spurious proxy ($15.8059$ linear, Chow $F = 23\text{,}437.0$; $0.1072$ nonlinear), with Rosenbaum sensitivity $\Lambda^{\ast} = 4.752$ and Cinelli-Hazlett Robustness Value $RV = 0.511$.
5. Lens 5 (Decision): On $n = 20\text{,}778$ NOAA Atlantic storm records, the PAV isotonic decomposition splits $\mathrm{BS} = 0.26906$ into $\mathrm{REL}_{\mathrm{CORP}} = 0.21135$, $\mathrm{RES}_{\mathrm{CORP}} = 0.13102$, and $\mathrm{UNC}_0 = 0.18873$ with an identity closure error of $1.53 \times 10^{-14}$, PAV-calibrated Logarithmic Score $\mathrm{LogS} = 0.1849$, and Relative Economic Value $\mathrm{REV}_{\mathrm{PAV}} = 0.719$ ($71.9%$ of an infallible oracle’s economic gain). Meanwhile, the FairCoin benchmark proves that an overconfident forecaster (REL = 0.1104) who agrees $100%$ of the time with a calibrated forecaster (REL = 0.0005) at threshold $p^{\ast}_A = 0.30$ inflicts $+18.20$ units of excess regret the moment the decision threshold moves to $p^{\ast}_B = 0.70$!

Exercises

1. (Calculation — Non-Ergodicity and Itô’s Drain) An investment strategy has expected annual arithmetic return $\mu = 0.12$ ($12%$) and annual volatility $\sigma = 0.30$ ($30%$).
   • (a) What is its compounded time-average growth rate $g_{\text{time}} = \mu - \frac{1}{2}\sigma^2$?
   • (b) If a fund applies $3\times$ leverage (tripling both $\mu$ to $0.36$ and $\sigma$ to $0.90$), what happens to $g_{\text{ensemble}}$, and what happens to $g_{\text{time}}$? Why does the $3\times$ leveraged investor lose money over time while the expected value grows?
2. (Concept — Confounder vs. Collider) A tech company notices that among employees it hired, programming test scores ($X$) and interview charisma scores ($Y$) are negatively correlated. Given that a candidate is hired ($B = 1$) only if the sum of $X + Y$ exceeds a high cutoff, draw the causal graph connecting $X$, $Y$, and $B$, and explain why conditioning on hired employees created the negative correlation.
3. (Calculation — Cost-Loss Threshold) Protecting a power substation against a geomagnetic storm costs $C = USD 45\text{,}000$. An unmitigated transformer failure costs $L = USD 1\text{,}000\text{,}000$, and protective switching prevents $\eta = 0.90$ of the damage. Compute $p^{\ast}$. Should the operator protect if the calibrated probability is $p = 0.065$?
4. (Concept — CORP-Murphy Asymmetry) Why is it impossible to increase $\mathrm{RES}_{\mathrm{CORP}}$ by applying a strictly increasing function $p_i \mapsto g(p_i)$ to your model’s predicted probabilities?

Chapter Summary

• Lens 1 (Information): Post-processing cannot create signal ($I(W; Z) \le I(W; Y) \le I(W; X)$), channel noise contracts distributions according to $\eta_{\chi^2} = \rho_m^2 \le \eta_{\mathrm{KL}} \le \eta_{\mathrm{TV}}$, and lost information enforces Fano’s error floor.
• Lens 2 (Time): Multiplicative dynamics are non-ergodic ($g_{\text{time}} = \mu - \frac{1}{2}\sigma^2 < g_{\text{ensemble}}$), and survival requires respecting the asymmetric Kelly fraction $f^{\ast} = \frac{p}{a} - \frac{1-p}{b}$ and absorbing ruin barriers.
• Lens 3 (Tail): Power-law tails $P(X > x) \propto x^{-\alpha}$ estimated via Hill’s $\hat{\alpha}_{k,n}$ lose finite variance when $\alpha \le 2$ and lose finite mean when $\alpha \le 1$.
• Lens 4 (Invariance): Pearl’s Causal Ladder separates seeing, doing, and imagining; conditioning on a Berkson collider is forbidden; and Invariant Causal Prediction ($\hat{S}^{\ast}$) + NILE/IRM isolate mechanisms that survive across environments.
• Lens 5 (Decision): Rational protection occurs at $p > p^{\ast} = \frac{C}{\eta L}$, project viability requires $\mathrm{EVPI} > C_{\text{total}}$, and PAV isotonic regression yields the exact CORP-Murphy decomposition $\mathrm{BS} = \mathrm{REL}_{\mathrm{CORP}} - \mathrm{RES}_{\mathrm{CORP}} + \mathrm{UNC}_0$.