18. Eliminatory Funnel and the 8-Step SIGA Roadmap

The Two-Million-Dollar Model That Could Never Change a Decision

A regional hospital system hires a data-science team for USD 500,000 to build a machine-learning model predicting whether patients admitted with severe fever have a bacterial infection.

Suppose treating a patient with broad-spectrum antibiotics costs $C = USD 100$, while failing to treat a true bacterial infection costs $L = USD 50\text{,}000$ in intensive-care complications (with treatment efficacy $\eta = 1.0$). By the Cost-Loss Rule of Chapter 6, a physician should administer antibiotics whenever the probability of bacterial infection exceeds:

$$p^{\ast} = \frac{C}{\eta L} = \frac{100}{50\text{,}000} = 0.002 \quad (0.2%)$$

Now look at the clinical reality: any patient arriving at the hospital with severe fever already has a baseline probability of bacterial infection of at least $15%$—and even if the new USD 500,000 machine-learning test comes back negative, the patient’s posterior probability only drops from $15%$ to $3%$, which is still fifteen times higher than the $0.2%$ threshold $p^{\ast}$!

In other words, whether the expensive new model says “positive” or “negative,” the doctor must administer antibiotics either way. The Expected Value of Sample Information is zero, and the USD 500,000 project was dead on arrival before a single line of code was written.

In most organizations, nobody checks whether a model can alter an action, whether the true cluster sample size $n_{\mathrm{eff}}(p)$ is large enough, or whether the inference software can even recover known synthetic parameters until after months of computing have been spent.

To reverse this waste, crates/stochology-siga enforces two sequential operational instruments:

1. The 4-Filter Eliminatory Funnel (funnel.rs): A strict short-circuiting gate that kills economically, causally, or computationally invalid projects upfront.
2. The 8-Step SIGA Operational Roadmap (roadmap.rs): A disciplined execution cycle (Survey, Interrogate, Generate, Act) backed by anytime-valid e-processes and natural-frequency decision trees.

18.1 The 4 Eliminatory Filters (stochology_siga::funnel)

The four filters are evaluated in strict sequence (Filter 1 -> Filter 2 -> Filter 3 -> Filter 4). If Filter 1 fails, Filters 2–4 are never even run (short_circuited = true):

Filter 1 — Economic Viability and Question Resolution ($\Phi_5$)

Filter 1 asks three elementary questions before touching data:

1. Is the critical cost-loss threshold non-degenerate: $p^{\ast} = \frac{C}{\eta L} \in (0, 1)$?
2. Does the Expected Value of Perfect Information ($\mathrm{EVPI}$) strictly exceed the total engineering, data-acquisition, and operational cost of the study ($\mathrm{EVPI} > C_{\text{total}} \ge 0$)?
3. Does the temporal/spatial resolution of the dataset satisfy Invariant $\Phi_5$ ($R_{\text{data}} \ge R_{\text{decision}}$)?

Filter 2 — Cluster Independence ($\Phi_2$) and Causal Hygiene

Filter 2 audits Bridge $L_1 \to L_2$ and Lens 4:

1. Using the Moulton-Campbell unequal-cluster formula $\tilde{m} = \bar{m}(1 + \mathrm{CV}^2)$ and the tail-exceedance intraclass correlation $\rho_I(p)$, is the effective sample size large enough to support inference ($n_{\mathrm{eff}}(p) = \frac{n}{1 + (\tilde{m}-1)\rho_I(p)} \ge n_{\min}$)?
2. Has the analyst mapped the Markov Blanket of the target variable (its direct causal parents, direct causal children, and co-parents), and verified that zero conditioning occurs on a Berkson collider (conditioned_on_collider == false)?

Filter 3 — Simulation-Based Calibration (SBC) and SHELF Prior Elicitation

Even if the data are clean and the causal graph is sound, how do we know that our Bayesian or probabilistic estimation algorithm does not have a subtle coding bug or biased approximation?

In 2018, Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman formalized Simulation-Based Calibration (SBC) based on a beautiful self-consistency identity of probability theory:

1. Draw a true parameter $\tilde{\theta} \sim \pi(\theta)$ from the prior distribution.
2. Simulate a synthetic dataset $\tilde{y} \sim p(y \mid \tilde{\theta})$ using the generative model ($\Phi_1$).
3. Run your inference algorithm on $\tilde{y}$ to draw $M$ posterior samples $\theta_1, \dots, \theta_M \sim p(\theta \mid \tilde{y})$.
4. Count the rank statistic $r(\lbrace \theta_m \rbrace, \tilde{\theta}) = \sum_{m=1}^M \mathbb{I}\lbrace \theta_m < \tilde{\theta} \rbrace \in \lbrace 0, 1, \dots, M \rbrace$.

Theorem (Talts et al., 2018): If the posterior inference algorithm is mathematically exact, the rank statistic $r$ is distributed strictly uniformly across $\lbrace 0, 1, \dots, M \rbrace$!

• If your algorithm is overconfident (posterior intervals too narrow), the true $\tilde{\theta}$ falls in the extremes too often, creating a $\cup$-shaped rank histogram.
• If your algorithm is underconfident (intervals too wide), the rank histogram heaps up in the middle ($\cap$-shaped).
• If your algorithm is biased, the rank histogram tilts to the left or right.

Filter 3 bins the SBC ranks across $B$ bins and computes Pearson’s $\chi^2 = \sum_{b=1}^B \frac{(O_b - E_b)^2}{E_b}$, requiring $\chi^2 \le \chi^2_{\text{crit}}$. Simultaneously, Filter 3 validates expert prior elicitation under Anthony O’Hagan’s Sheffield Elicitation Framework (SHELF), requiring strict physical ordering of the elicited quartiles:

$$L_{\text{phys}} < q_{25} < q_{50} < q_{75} < U_{\text{phys}}$$

Once the elicited quartiles $(q_{25}, q_{50}, q_{75})$ are verified, fit_shelf_beta_prior (crates/stochology-siga/src/funnel.rs) fits a continuous conjugate $\mathrm{Beta}(\hat{\alpha}, \hat{\beta})$ prior by minimizing the cumulative distribution RMSE $\sqrt{\frac{1}{3}\bigl[(I_{q_{25}}(\alpha, \beta) - 0.25)^2 + (I_{q_{50}}(\alpha, \beta) - 0.50)^2 + (I_{q_{75}}(\alpha, \beta) - 0.75)^2\bigr]}$, where the regularized incomplete Beta function $I_x(\alpha, \beta)$ is evaluated to machine precision in pure Rust via Lentz’s (1976) continued-fraction algorithm (regularized_incomplete_beta) and Lanczos’s $g=7$ log-Gamma approximation (ln_gamma_lanczos).

Filter 4 — Cryptographic Pre-Registration, Cross-Authorship, and Exit Clause

Finally, Filter 4 verifies:

1. A valid 64-character lowercase hexadecimal SHA-256 pre-registration seal;
2. Independent cross-authorship (modeler_id != verifier_id);
3. A non-empty, machine-actionable numeric exit clause.

18.2 The 8-Step SIGA Operational Roadmap (stochology_siga::roadmap)

Once a project survives all four filters of the Eliminatory Funnel, it executes the 8-Step SIGA Roadmap across four phases (S — Survey, I — Interrogate, G — Generate, A — Act):

18.3 Anytime-Valid Sequential Evidence (e-processes) and Ville’s Inequality

Foundations: Why Classical $p$-Values Break Under Optional Stopping

In a classical fixed-horizon Neyman-Pearson test, you must fix the sample size $n$ in advance and look at the data only once at the end. What happens if a researcher or an automated monitoring dashboard computes a classical $p$-value every day as new data arrive and stops the moment $p_t < 0.05$?

By the Law of the Iterated Logarithm, even under the null hypothesis $H_0$ (where the true effect is zero!), a standardized random walk crosses the fixed $1.96$ threshold infinitely often as $t \to \infty$! Peeking continuously at a classical $p$-value inflates the Type I false-positive rate all the way to $100%$.

How e-values, Test Martingales, and Ville’s Inequality Solve Optional Stopping

Modern sequential statistics (Glenn Shafer, Peter Grünwald, Aaditya Ramdas, and Vladimir Vovk) replaces fragile $p$-values with evidence values (e-values) and sequential e-processes (EProcess in crates/stochology-siga/src/roadmap.rs).

An e-variable is any non-negative random variable $L_i \ge 0$ whose expected value under the null hypothesis $H_0$ is at most $1$:

$$\mathbb{E}_{H_0}[L_i \mid \mathcal{F}_{i-1}] \le 1$$

For example, the likelihood ratio $L_i = \frac{p_1(Y_i)}{p_0(Y_i)}$ between the alternative model $p_1$ and the null model $p_0$ automatically satisfies $\mathbb{E}_{p_0}\left[\frac{p_1(Y_i)}{p_0(Y_i)}\right] = \sum_y p_0(y)\frac{p_1(y)}{p_0(y)} = \sum_y p_1(y) = 1$!

Think of $L_i$ as a fair betting multiplier against the null hypothesis: you start with USD 1.00 of initial capital ($E_0 = 1$), and at each step $i = 1, \dots, t$ your capital multiplies by $L_i$:

$$E_t \triangleq \prod_{i=1}^t L_i$$

Because $E_t$ is a non-negative test martingale (or supermartingale) under $H_0$, Jean Ville’s (1939) Maximal Inequality guarantees that the probability of your capital ever reaching $1/\alpha$ at any stopping time $t$—no matter how often you peek at the dashboard!—is at most $\alpha$:

$$P_{H_0}\left(\sup_{t \ge 1} E_t \ge \frac{1}{\alpha}\right) \le \alpha$$

Even better: if an analyst tried $K \ge 1$ candidate specifications (Prohibition 4), dividing the cumulative evidence by $K$ preserves anytime validity across all $K$ attempts:

$$\frac{E_t}{K} \ge \frac{1}{\alpha}$$

At $\alpha = 0.05$, reaching $E_t / K \ge 20$ means the data have multiplied initial skeptical capital at least twenty-fold after paying the full multiplicity tax $K$.

Non-Parametric GRAPA / Kelly Betting Martingales (EvidenceProcess::from_grapa_betting)

What if we want to construct an anytime-valid e-process from bounded zero-mean under-$H_0$ payoff differences $X_1, \dots, X_t$ without assuming a parametric likelihood ratio $p_1/p_0$? Following Peter Grünwald, Rianne de Heide, and Wouter Koolen (2024, Safe Testing) and Ian Waudby-Smith and Aaditya Ramdas (2024), EvidenceProcess::from_grapa_betting constructs a predictable Growth-Rate Adaptive (GRAPA / Kelly) betting martingale:

$$E_t = \prod_{i=1}^t \bigl(1 + \lambda_i X_i\bigr), \qquad \lambda_i = \mathrm{clamp}\left(\frac{\hat{\mu}_{i-1}}{\hat{\sigma}_{i-1}^2 + \hat{\mu}_{i-1}^2},\ -0.5,\ 0.5\right)$$

Because the predictable stake $\lambda_i$ depends strictly on past observations $\mathcal{F}_{i-1} = \sigma(X_1, \dots, X_{i-1})$, $\mathbb{E}_{H_0}[1 + \lambda_i X_i \mid \mathcal{F}_{i-1}] = 1 + \lambda_i \mathbb{E}_{H_0}[X_i \mid \mathcal{F}_{i-1}] = 1$, guaranteeing exact finite-sample Ville validity under arbitrary optional stopping while maximizing asymptotic log-capital growth under the alternative!

Computational Audit in Rust

Run Laboratory 5 from your terminal to watch all three canonical YAML cards pass through the 4-Filter Funnel, execute the 8-Step SIGA Roadmap, compute their cumulative e-processes $E_{30}$, and emit their integer Gigerenzer Natural-Frequency Executive Memoranda:

cargo run --release --bin labs -- --siga

=== LAB 5: 4-FILTER ELIMINATORY FUNNEL AND 8-STEP SIGA ROADMAP ===
  [SIGA cards/card_01_atlantic_storms_evacuation.yaml] dataset=data/atlantic_storms.csv, EVPI=257928.0, p_null=0.0050 (unclamped), chi2_SBC=0.85, E_30=324171.1, steps_1_to_8=[true, true, true, true, true, true, true, true], approved=true
    Out of every 1000 independent units, 98 experience the critical event (of which 98 trigger the alarm at threshold p*=0.12000 and 0 are missed), while 902 do not experience the event (of which 156 trigger a false alarm and 746 remain clear). Positive predictive frequency: 98 out of 254 alarms. Action: do_not_protect. Exit clause: Immediately suspend automated synoptic advisory if REL_CORP > 0.015 or RES_CORP < 0.020 over a 14-day rolling window.
  [SIGA cards/card_02_bank_queue_staffing.yaml] dataset=data/bank_calls.csv, EVPI=216213.6, p_null=0.0050 (unclamped), chi2_SBC=4.65, E_30=2786.4, steps_1_to_8=[true, true, true, true, true, true, true, true], approved=true
    Out of every 1000 independent units, 99 experience the critical event (of which 99 trigger the alarm at threshold p*=0.08000 and 0 are missed), while 901 do not experience the event (of which 426 trigger a false alarm and 475 remain clear). Positive predictive frequency: 99 out of 525 alarms. Action: protect. Exit clause: Immediately suspend automated shift scheduling if REL_CORP > 0.020 or RES_CORP < 0.025 over a 10-business-day rolling window.
  [SIGA cards/card_03_nyc_flights_holding.yaml] dataset=data/nyc_flights_2013.csv, EVPI=135285.0, p_null=0.0050 (unclamped), chi2_SBC=3.00, E_30=32.8, steps_1_to_8=[true, true, true, true, true, true, true, true], approved=true
    Out of every 1000 independent units, 98 experience the critical event (of which 71 trigger the alarm at threshold p*=0.06000 and 27 are missed), while 902 do not experience the event (of which 277 trigger a false alarm and 625 remain clear). Positive predictive frequency: 71 out of 348 alarms. Action: protect. Exit clause: Immediately revert to conservative Wald Maximin ground-delay policy if REL_CORP > 0.012 or RES_CORP < 0.005 or RRM radius >= 0.85 over a 7-day window.
  [SHELF Beta Fit & GRAPA Betting Martingale] SHELF(0.02,0.05,0.09)->Beta(alpha=1.05, beta=15.03, CDF_RMSE=0.0111) | GRAPA E_35=946.42 (rejects_H0=true)

Exercises

1. (Calculation — Filter 1 Short-Circuiting) A logistics company proposes spending $C_{\text{total}} = USD 80\text{,}000$ to build a model predicting late port deliveries. Each late delivery causes loss $L = USD 10\text{,}000$, expedited air-freight protection costs $C = USD 2\text{,}000$ with efficacy $\eta = 1.0$, and across the year $100$ shipments are made with base rate $p_0 = 0.05$ ($5$ late shipments per year). Even if the model were an infallible oracle, what is the maximum annual savings ($\mathrm{EVPI} \times 100$) it could generate? Will Filter 1 approve or kill the USD 80,000 project?
2. (Concept — Simulation-Based Calibration) Suppose you run SBC on a new Bayesian inference algorithm and observe that out of $1\text{,}000$ synthetic runs, the true parameter $\tilde{\theta}$ falls below the $10\text{th}$ percentile of the posterior $250$ times and above the $90\text{th}$ percentile $260$ times. Is your algorithm overconfident, underconfident, or unbiased?
3. (Calculation — Sequential e-process and Ville’s Inequality) Over $5$ sequential out-of-sample test weeks, a model achieves weekly likelihood ratios $L_1 = 2.5, L_2 = 1.8, L_3 = 2.0, L_4 = 0.8, L_5 = 5.0$ against the null model, after $K = 2$ attempted specifications.
   • (a) Compute $E_5 = \prod_{i=1}^5 L_i$ and the multiplicity-adjusted evidence $E_5 / K$.
   • (b) By Ville’s inequality, what is the upper bound on the probability that $E_t / K$ could have reached this level under the null hypothesis?
4. (Calculation — Gigerenzer Tree) Look at the natural-frequency memo for card_03_nyc_flights_holding.yaml above ($1\text{,}000$ units, $98$ events, $71$ true alarms, $277$ false alarms). Verify that $71 + 277 = 348$ total alarms and compute the Positive Predictive Value ($\mathrm{PPV} = 71 / 348$). Why is it still rational for the dispatcher to protect when $\mathrm{PPV} \approx 20.4%$? (Hint: compare $\mathrm{PPV}$ with $p^{\ast} = 0.06000 = 6.0%$!)

Summary

• The 4-Filter Eliminatory Funnel evaluates projects in strict sequence: Filter 1 ($\mathrm{EVPI} > C_{\text{total}}$ and $\Phi_5$), Filter 2 (Moulton-Campbell $n_{\mathrm{eff}}(p)$ and zero Berkson collider), Filter 3 (SBC $\chi^2$ rank uniformity and SHELF quartile consistency), and Filter 4 (SHA-256 pre-registration, cross-authorship, and exit clause).
• The 8-Step SIGA Roadmap executes Survey (Steps 1–2), Interrogate (Steps 3–4), Generate (Steps 5–6), and Act (Steps 7–8).
• Anytime-Valid e-processes ($E_t = \prod L_i$) remain valid under arbitrary optional stopping via Ville’s Inequality ($P_{H_0}(\sup_t E_t / K \ge 1/\alpha) \le \alpha$).
• Gigerenzer Natural-Frequency Trees translate abstract conditional probabilities into transparent integer headcounts out of $1\text{,}000$ or $10\text{,}000$ independent units.