Data Diction

I selected a model. Can I trust my p-values yet?

Jinal Shah — Tue, 08 Sep 2026 00:00:00 GMT

Reviewed by Ryan Peterson on 2026-08-13

In a previous post, Dr. P. Hackman demonstrated the remarkable effectiveness of unadjusted post-selection inference (UPSI) for achieving publishable, significant results. By searching for interesting “subgroups” through a large collection of main effects and interactions and then applying ordinary inferences as though no selection had occurred, Hackman virtually guaranteed statistically significant findings even when none existed.

The statistical problem I want to focus on here is not about causation: it’s about what happens when we use the data to select which subgroups to test, and then act as if those subgroups had been specified in advance. This issue applies equally to randomized trials and observational studies. Randomization protects against confounding, but not against the false positives that come from data-driven model selection. The analysis in the original post is simply performing model selection and then pretending the selected model had been specified in advance.

While this strategy may be excellent for a fictional statistician seeking ill-gotten fame and glory, sane analysts are interested in a slightly different goal: obtaining inference that is actually valid. This raises a natural question: if I select a model, and UPSIs are invalid, what should I use instead? In this post, I’ll revisit Hackman’s simulation setup and illustrate a more rigorous approach to post-selection inference: selective inference.

The selectInferToolkit R package

Selective inference, forward stepwise selection, and many other selection and inference methods are available in the selectInferToolkit R package, available on GitHub.

UPSI Alternatives

One proposed remedy, discussed in Maggie Qian’s follow-up post, The Antidote to Guaranteed Significance, is multiplicity adjustment. Qian shows that a Bonferroni or similar correction adjusts p-values for the number of tests performed, and therefore substantially “brings the false positive rate back to earth.” However, even Bonferroni, conservative as it is, doesn’t bring it down to the nominal level; the null still gets rejected nearly three times the nominal rate. A key reason is that multiplicity adjustment requires knowing how many tests you’re adjusting for. In the model selection setting, that number is genuinely ambiguous. Is it the possible models? Something proportional to ? The number of steps taken during the forward search? The answer lies somewhere in between, and no single correction cleanly resolves the problem.

More importantly, the subgroup discovery framing is a special case of a more general problem. In many analyses, there is no pre-specified treatment and no enumerable list of candidate subgroups: we simply have a large collection of predictors and interactions, and we use the data to select a model. In the general setting, model selection (usually, but not always!) presumes no natural pre-specified hierarchy and no obvious grouping structure. The post-selection inference problem remains, and we need methods that address it directly.

Selective inference is designed for exactly this situation. Rather than counting comparisons after the fact, it directly accounts for the model selection procedure by asking: given that this particular model was selected from data, what can we still infer? In effect, it conditions on the selection event and recalibrates the usual p-values and confidence intervals accordingly. The price of this intellectual humility can be quite steep: the resulting intervals are often wider - sometimes infinite in length. Worse, the impressively small p-values driven by Hackman’s underhanded UPSIs will, more often than not, lose their sheen of significance.

How does selective inference actually work though?

For readers interested in more theoretical properties of selective inference can refer to slides here.

Example: a study for treatment of chronic pain

Hackman’s UPSI approach

Let’s revisit the (rather extreme) simulation in the original post which uses interactions between 6 covariates to define subgroups:

library(tidyverse)
subgroups <- expand.grid(
  age = c("18-35", "36-50", "51-65", "66-80", "80+"),
  sex = c("Male", "Female"),
  hand = c("Left", "Right"), 
  coffee = c("0", "1", "2+"),
  alcohol = c("0", "1", "2", "3+"),
  physical_activity = c("0", "1", "2", "3+")
)
set.seed(21)
n <- 100      # Sample size  

# Simulate recruitment (y: outcome)
y <- rnorm(n)

# Simulate recruitment (assume each new person has random subgroup)
x_idx <- sample(1:nrow(subgroups), n, replace = TRUE) 
X <- subgroups[x_idx,]
simdata <- tibble(y = y, X)

# Create model matrix w/all interactions
X2 <- model.matrix(y ~ . * ., data = simdata)[,-1]

# combine into data set for model fitting
simdata2 <- data.frame(y=y, X2)

As we saw in Hackman’s original post, when using forward stepwise selection with BIC, every single selected subgroup was declared statistically significant as reproduced below. Importantly, Hackman’s UPSI approach finds highly significant results despite the relationship between treatment and response being completely random!

library(selectInferToolkit) #github.com/petersonR/selectInferToolkit
# Fit the stepwise BIC model 
fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                              direction = "forward", 
                              penalty = "BIC")

# Perform unadjusted post-selection inference (UPSI)
infer_upsi(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	−0.09	0.24	0.393
age36.50	0.27	0.10	0.45	0.003
age66.80	−0.38	−0.57	−0.18	<0.001
age66.80.coffee1	0.23	0.04	0.42	0.019
age51.65.coffee2.	0.29	0.11	0.46	0.002
age51.65.physical_activity1	−0.24	−0.42	−0.07	0.007
sexFemale.physical_activity2	−0.32	−0.49	−0.15	<0.001

Presenting UPSI-based results like this can be deeply misleading. Imagine a research team applies a similar approach to an observational study of pain management, discovers that patients aged 36-50 show a “significantly” stronger association with better outcomes, and publishes accordingly. A clinical lab takes the finding at face value and designs a confirmatory trial targeting that age group. The trial fails - not because the treatment doesn’t work, but because the original association was an artifact of searching through dozens of subgroups without accounting for the search. Time, money, and patient participation were spent chasing a signal that was never there.

UPSIs are “oopsies”

The fact that a variable was selected and appears statistically significant in the unadjusted post-selection inferences of the final model (e.g. a significant UPSI-based p-value) does not imply that the effect is real. Much of the apparent certainty can disappear once we account for the model selection process itself.

Selective inference to the rescue

When performing inference after selection, we know that since we’ve used the data both to select the model and now are using the same data to perform inference, these results are overly-optimistic - they are UPSIs. So next let’s use selective inference, which explicitly accounts for the fact that the model itself was selected using the data.

# infer_selective can perform selective inference 
infer_selective(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	NA	NA	NA
age36.50	0.27	−9.26	0.33	0.879
age66.80	−0.38	−Inf	0.72	0.080
age66.80.coffee1	0.23	−1.30	1.63	0.381
age51.65.coffee2.	0.29	−5.85	1.21	0.759
age51.65.physical_activity1	−0.24	−1.57	1.06	0.333
sexFemale.physical_activity2	−0.32	−6.74	0.61	0.099

Here we see that after accounting for the selection process, none of the selected effects remain statistically significant! Let’s take a moment to appreciate this victory over Hackman’s UPSI approach.

Of course, as mentioned previously, one caveat is confidence intervals become substantially wider. In this example, the lower CI bound for the 66-80 age subgroup extends to . This isn’t an error or numerical instability. Rather, it reflects the uncertainty introduced by model selection: when an effect is identified only after searching through many possible models, accounting for that selection process can result in substantially wider confidence intervals.

A victory? Or a fluke?

I can almost hear Hackman coughing back: this was only a single simulation, I might have lost that battle but surely not the war. Even Bonferroni and his ridiculous adjustment failed to fully rein in my UPSIs.

Selective inference did successfully wash the false sheen of significance off of Hackman’s UPSI-based p-values once. Taking a page from Hackman himself, let’s repeat this 1000 times, as though we run the study in 1000 parallel universes to confirm without any doubt that selective inference controls the false-positive results. I will also add the simple antidote with Bonferroni multiplicity correction (with the number of tests set to as in Qian’s post) to see how it performs.

As noted earlier, selective inference controls type I error per selected variable conditional on selection, but not across all selected variables within a single analysis/dataset. When BIC selects multiple variables and each is tested at , the chance of at least one false positive per simulation compounds. To address this, I also apply a within-simulation Bonferroni correction on top of selective inference, adjusting each SI p-value by the number of selected variables in that simulation. This adds a second layer of multiplicity control: SI accounts for the search process, while the Bonferroni step accounts for testing multiple selected variables simultaneously.

# create list to store different results
sim_results_UPSI <- list()
sim_results_SI <- list() # selective inference only 
sim_results_SI_BF <- list() # selective inference with bonferroni correction 
sim_results_BONF <- list()

set.seed(1234)
## The below takes about 10 minutes to run 

for(s in 1:1000) { 
  simdata2$y <- rnorm(n)
  fit_bic <- select_stepwise_ic(
    y ~ ., data = simdata2, 
    direction = "forward",
    penalty = "BIC"
  )
  
  sim_results_UPSI[[s]] <- 
    infer_upsi(fit_bic, data = simdata2) %>% 
    tidy() %>% 
    filter(coef != 0) 
  
  sim_results_BONF[[s]] <- 
    infer_upsi(fit_bic, data = simdata2) %>%
    tidy() %>%
    filter(coef != 0) %>%
    mutate(
      n_tests = ncol(simdata2) - 1, # number of predictors, p, 
      p_value  = pmin(p_value * n_tests, 1) # Bonferroni adjusted
    )
  
  sim_results_SI[[s]] <- 
    infer_selective(fit_bic, data = simdata2) %>%
    tidy() %>%
    filter(coef != 0) 
  
  sim_results_SI_BF[[s]] <- 
    infer_selective(fit_bic, data = simdata2) %>%
    tidy() %>%
    filter(coef != 0) %>%
    mutate(
      n_selected = sum(term != "(Intercept)"),  # number of selected non-intercept terms
      p_value = pmin(p_value * n_selected, 1)
    )
  
}

upsi_results      <- bind_rows(sim_results_UPSI, .id = "sim")
si_results  <- bind_rows(sim_results_SI, .id = "sim")
si_bonf_results  <- bind_rows(sim_results_SI_BF, .id = "sim")
bonf_results <- bind_rows(sim_results_BONF, .id = "sim")

I can now summarize the results as shown below.

Code

ASA Response to NIH Request for Information: Measuring and Rewarding Scientific Impact

Wed, 19 Aug 2026 00:00:00 GMT

Developed with scientific and technical input from Ryan A. Peterson, PhD; Elizabeth W. Eisenhauer, PhD; Theresa Kim, PhD, MS; and other ASA members.

Note

This response is also available from the ASA website. It is republished here at Data Diction with permission.

Contributor Disclaimer

The views do not necessarily represent the views of contributors’ employers or affiliated institutions. Theresa Kim provided scientific and technical input during off-duty hours and without use of government-furnished equipment; her participation does not represent the views of NIH, HHS, or the U.S. Government.

Claude Opus 5 (Anthropic) and GPT-5.6 Sol (OpenAI) were used to assist with drafting and editing these comments.

1. Rigor and Reproducibility

1.1 The missing instrument: indicators for analytic replicability

We are highly encouraged by NIH’s renewed focus on transparency, reproducibility, and replication shown in the Request for Information on Measuring and Rewarding Scientific Impact, the Initiative to promote Strengthening Replication and Reproducibility of NIH-funded Research, the Gold Standard implementation plan, and the recent Highlighted Topic on Enhancing Scientific Rigor, Transparency and Replicability. These documents name reproducibility and replicability as distinct goals, correctly, and the Initiative has instruments aimed at each: for the former, longstanding investments in data sharing, standardization, and public access to findings; for the latter, funded replication research through the Common Fund, a Replication Prize, and a stated intent to engage researchers in identifying areas ripe for replication.

These are steps in the right direction. However, we have yet to observe incentives targeting the gap between reproducibility and replicability: whether the reported inference is valid given how the analysis proceeded. Statisticians and biostatisticians have a useful and unusual vantage point on this gap: we are often among the first to recognize why a finding may not replicate, while not always having the standing or involvement needed to address the problem.

This matters because replication studies detect errors, but do not prevent them. They measure the failure rate after the fact, one finding at a time, at a high cost per finding. A study can be perfectly reproducible — fully containerized, one-click re-runnable — and still report inference that is unlikely to replicate, and the replication study will discover this several years and several hundred thousand dollars later.

A common route to this is inference on a model that was chosen using the same data used for inference: unadjusted post-selection inference (UPSI). It is one instance of what Simmons, Nelson, and Simonsohn (2011) termed researcher degrees of freedom: undisclosed flexibility in analytic choices, sufficient on its own to produce significant findings without any intent to deceive. In a satirical blog post, Perry Hackman used simulations with pure-noise outcomes and a modest set of candidate subgroup interactions to show that an ostensibly rigorous method, forward stepwise selection with information criteria followed by standard statistical inference (dubbed UPSI), can produce at least one “significant” effect in 90+% of simulated trials (Peterson, 2025b). Nothing in that pipeline is clearly fraudulent; the analysis plan reads as rigorous, and the resulting code can be shared, containerized, and re-run perfectly.

Recommendation: The Initiative’s replication instruments would return more per dollar if paired with a cheap upstream instrument that flags the findings most likely to need them. In analysis plans, progress reports, and publication reporting, a short structured statement answering the following could be incentivized:

Was the reported model, subgroup, outcome scale, or covariate set chosen using the same data used to compute the reported inference?
If so, how were the inferences adjusted for selection? (sample splitting; selective or conditional inference; bootstrap or model-averaged uncertainty; explicit relabeling as exploratory). If not, where was the full analysis specification verifiably pre-registered?
How many candidate specifications were considered, including those examined and discarded?

Based on the above, results should be taken as on a spectrum between confirmatory (to provide stronger evidence for prespecified claims) and exploratory (to identify patterns and generate hypotheses).

Another recommendation is to commission periodic targeted statistical review of a random sample of supported publications, scored by trained statistical reviewers, reporting the proportion that (a) report selection-conditional inference without adjustment, (b) report subgroup findings without pre-specified subgroup structure, and (c) contain code that does not execute. Tracking that proportion over time measures whether the Initiative is working.

1.2 Candidate pools and model complexity are growing faster than the standards applied

Beneath many variable-selection procedures is an implicit assumption of covariate equipoise: that every candidate predictor is equally worthy of entering the model. Stepwise procedures, popular information criteria searches, and standard lasso implementations can all behave this way, and the assumption becomes problematic in settings where NIH is investing heavily.

Independent groups have converged on the same principle from different directions. Yu, Bien, and Tibshirani (2019) articulate a reluctant interaction selection principle — prefer a main effect over an interaction if all else is equal — and Tay and Tibshirani (2020) extend this method to a reluctant non-linear selection principle, preferring a linear term over a non-linear one on the same grounds. In his standard reference text, Harrell (2015) advises pre-specifying a small set of plausible interactions rather than testing all of them. Peterson and Cavanaugh (2022) advise that human-understandable “glass-box” models with meaningful features should be preferred to more complex ones precisely because they are easier to interrogate and replicate. Different penalties, different algorithms, same underlying claim: complexity should have to earn its way into a model, and treating every candidate term as equally deserving is what allows spurious complexity through. NIH need not adjudicate among these methods, but it should seek to require the reporting that makes the problem visible.

Transparent modeling methods matter because replication requires something to replicate: a claim specific enough that an independent team can test it on new data and report whether it held. Transparent statistical models often make such claims directly available. Waist circumference is associated with one-year HDL, β = −0.04 (95% CI −0.07 to −0.01) is a statement another group can go and check in a different dataset, and the check has a clear objective.

On the other hand, black-box predictive models, such as neural networks, XGBoost, random forests, and statistical models with high-order interaction terms, instead primarily yield prediction functions. A second team can apply the deposited model to new data and test whether predictive performance holds up. That is a meaningful form of external validation, but it is different from replicating an interpretable scientific claim. When performance drops, the failure may be difficult to localize: a transparent model can reveal which estimated relationship moved and in which direction, while an opaque model may reveal only that predictive performance deteriorated, and, if subgroup metrics are tracked, for whom.

This has a direct implication for the Initiative. Predictive modeling is an increasingly prominent category of NIH-supported research (e.g., Arshi et al. 2025, Le et al. 2024), and it is precisely the category on which replication instruments have the least purchase. Transparency is therefore not only a matter of trust, fairness, or regulatory compliance, though it is those too. It also determines how specifically a scientific claim can be interrogated and replicated.

As datasets grow larger and contain newer, more high-dimensional modalities, new statistical methodology will be critical. For instance, with only twenty candidate predictors, to capture potential interactions, one would need to sift through 190 pairwise interactions, 1,140 three-way interactions, and over a million candidate models across all orders. Further, when modalities differ in their dimension and overall signal, they quickly run into practical challenges. For example, in a lung adenocarcinoma survival analysis with 7 clinical covariates and 22,283 gene expression measurements, assuming covariate equipoise and penalizing both groups equally encodes a structural disadvantage for clinically well-motivated variables, which get de-selected due to the sheer difference in dimensionality of the two modalities. With NIH’s data-integration priorities, as studies become multimodal, consisting of genomics plus clinical plus imaging plus wearable plus environmental and more, the need for novel methods that can accommodate such data is critical; important methodological challenges remain at the scale and complexity of these data.

Recommendations: First, NIH should be wary of accepting post-hoc explanation of an opaque model (such as a locally interpretable model, a SHAP-based variable importance metric, etc.) as a reliable, replicable scientific finding; the explainer is a separate model that may or may not describe what the first one does, and should not be treated as equivalent to a directly interpretable scientific claim. Second, the accuracy cost of transparency is an empirical question that has been studied. A systematic review of 71 studies developing clinical prediction models found no performance benefit of machine learning over logistic regression (Christodoulou et al., 2019). Benchmarking work supports this outside the clinical setting as well: across 110 datasets from the Penn Machine Learning Benchmarks database, a transparent regression approach was within 5% of the best-performing method on 79% of classification and 62% of regression datasets, with random forests and gradient boosting showing no statistically distinguishable advantage (Peterson et al., 2024). Where transparent methods fell short, the reasons were diagnosable and specific and not reflective of a general law that complexity buys accuracy (specific examples can be found in Peterson et al., 2024, Section 3.3.2).

The implication is not that black-box methods should be discouraged, but that the burden of justification runs backward. Investigators are rarely asked to show that their model’s opacity bought anything. For NIH-funded models intended to inform clinical decisions, a low-cost expectation is that a transparent baseline be fit and reported alongside the complex one, so the cost of interpretability is measured in that application rather than assumed. Where the gap is negligible, as is often the case, the transparent model is the better scientific product, because it can be interrogated and corrected.

1.3 Adequacy at the design stage: making evidentiary expectations auditable

These analysis-stage safeguards are necessary, but inference is only as strong as the data generated by the study design. The design determines which populations, outcomes, contrasts, and sources of variation are observed and therefore which scientific claims the resulting data can support. One of the earliest places this becomes explicit is the sample-size or design justification, where investigators state what they expect the proposed design to be capable of learning.

This makes the justification useful for more than determining a number. It provides a prospective, auditable statement of what adequate evidence was expected to look like. NIH can then ask whether the realized study provided that evidence and, if not, whether the departures and their implications were documented. A study need not unfold exactly as anticipated. Recruitment, measurement properties, effect sizes, intraclass correlations, or other design features must often be learned during the course of a study, at which point they become scientific information in their own right. Reporting them allows planning assumptions to be updated across studies rather than repeatedly regenerated from incomplete information.

For studies relying on existing or linked data, realized data quality may also differ from what was assumed at design, with implications for which claims the data can support. Rigor requires making those departures visible and reconsidering the claims they permit, rather than retrospectively treating the original justification as though nothing changed.

Gold Standard Science creates a policy demand for determining and justifying adequate sample sizes, but no common definition of “adequate” accompanies it. Raising conventional thresholds does not settle the problem. A study can have 90% power under poorly supported assumptions, for an effect of little scientific consequence, using an inefficient design, or with an imprecisely measured outcome. Adequacy is instead a fit-for-purpose judgment: whether the design, measurement, data quality, and analysis are sufficient for the claims the study is intended to support.

NIH guidance already reflects this principle. Under the current Parent R01 review framework, reviewers assess whether sample size is sufficient and well justified in relation to the research questions, outcomes, and design (NIH, 2024). NCCIH feasibility trials instead require quantitative feasibility or acceptability benchmarks and justification that the sample is sufficient to evaluate them; testing efficacy or “preliminary efficacy” is explicitly not the purpose (NCCIH, 2024a). By contrast, NCCIH efficacy, effectiveness, and pragmatic trials require power calculations linked to endpoints and hypotheses, at least 90% power for the primary outcome, adjustment for multiple primary outcomes, and a clinically meaningful effect size (NCCIH, 2024b). NIH therefore already treats adequacy as purpose-dependent; the opportunity is to make that principle explicit, consistent, and auditable.

The same principle extends beyond these examples. Exploratory secondary analyses, transportability or triangulation studies, resource construction, and methods development may require adequate precision, population overlap, analytic scope, ability to assess discordance, or support for future studies rather than power against a single alternative. Requiring a power calculation regardless can produce the calculation without the justification. This is not a lower bar for exploratory work but a different one: identify the intended claim, the criterion of adequacy it requires, and the corresponding limits on inference.

Design information also interacts with the candidate analysis space. Limited information combined with broad analytic flexibility can generate apparently strong but unstable findings, while each source of risk is largely invisible to instruments aimed only at the other.

Declaring intended claims and criteria of adequacy at the design stage gives the disclosure in §1.1 something to be checked against. A confirmatory aim that later relies on substantial data-dependent selection, or a secondary outcome later presented as a primary claim, has changed evidentiary status during execution. That is not necessarily a scientific failure, provided that its path from “design” to “discussion” is visible and reflected in the claims made.

Recommendations:

First, require fit-for-purpose definitions of adequacy. Applications should identify the claims or objectives the study is intended to support, and the criterion for an adequate sample size or design should reflect that purpose. When an effect size is used to justify adequacy, its magnitude should be grounded where possible in empirical evidence and interpreted on a scale meaningful to the scientific question. Generic standardized effect-size conventions such as “small,” “medium,” and “large” should not substitute for separately considering the magnitude of effect that would matter and the variability anticipated in the study population. Not every aim of a complex study requires a separate, fully developed power analysis. However, the study should identify the minimum set of claims or objectives it is designed to support with strong evidentiary justification and demonstrate that the proposed design is adequate for those purposes.

Second, treat sample-size justification as an integral part of study design rather than a calculation performed after the design is largely fixed. The amount of information a study can provide depends not only on the number of observations but on interdependent choices about the population recruited, comparisons made, measurements collected, timing and intensity of follow-up, analytic strategy, and other features that can affect the credibility of conclusions. Those choices must also be made within constraints on participant burden, operational feasibility, time, and budget. Improving one dimension may worsen another. Sample-size justification should therefore be an iterative, collaborative process in which statistical, substantive, operational, and other relevant experts identify the major threats to the study’s success, examine how alternative design choices redistribute those threats, and determine how limited resources can be used to produce the strongest evidence for the study’s purpose. This provides a concrete, early example of collaboration that NIH could recognize and measure: integrating statistical, scientific, operational, and other relevant expertise into consequential study-design decisions.

Third, make sample-size and design justification transparent, auditable, and cumulative. Power analyses and other sample-size calculations should be reproducible from the statistical analysis plan, with the method, inputs, assumptions, and software or tools documented sufficiently for an independent reviewer to recreate and vary the calculation. Studies should subsequently compare planning assumptions with what was realized, including recruitment and retention, variability, clustering, measurement performance, and other design quantities. Meaningful departures should be reported along with their implications for the evidence the study can support, rather than treated only as implementation failures. Recording what was expected, what occurred, and how those differences affected the study would allow empirical knowledge about design assumptions to accumulate across studies, so that future investigators can ground their assumptions in better evidence. NIH could track the proportion of funded studies that make their design justifications reproducible and subsequently report comparisons between key planning assumptions and realized study conditions.

1.4 Identifying findings ripe for replication

The Initiative states that NIH will engage researchers in selecting research areas ripe for replication and reproducibility studies. The indicators proposed above provide a practical way to do so.

Statistical fragility can triage more effectively than citation count alone. A tractable prioritization score could combine downstream influence (citations, guideline adoption, use as the basis for subsequent trials) with fragility markers extractable from the published record: limited effective information relative to analytic complexity; post-hoc subgroup findings; adaptive analytic choices; estimates where the magnitude just passes a significance threshold; single-site designs; and material departures from the design assumptions or adequacy criteria established at the outset. Findings that are both highly influential and structurally fragile are where replication dollars buy the most information.

The proposed indicators make these markers much easier to identify prospectively. Section 1.1 records whether inference was conditioned on data-driven selection; §1.3 records what the design was expected to support and whether realized information or study objectives materially changed. Issues like a lower-than-anticipated effective sample size, unexpectedly high or low intraclass correlation, or a claim that moved from exploratory to confirmatory status are not themselves a scientific failure. But when such changes materially affect the strength of a consequential finding, they provide a rational signal for replication priority.

1.5 Learning from discordance: reconciliation and transportability

Discordance in the existing evidence is itself informative. Studies that appear to address the same question may differ in estimand, population, measurement, design, or analytic assumptions. Triangulation uses this principle: agreement across approaches with different sources of bias can strengthen causal inference, while disagreement can reveal which assumptions or design features matter (Lawlor et al., 2016). Rather than defaulting to another de novo study, NIH could support structured reconciliation of discordant evidence, harmonizing these dimensions where possible and identifying the likely sources of remaining disagreement.

The same idea is more powerful when planned prospectively. Where transportability matters, a second cohort, site, health system, or data source can apply the same estimand, outcome definition, and prespecified analysis. This narrows the dimensions that vary and provides direct evidence about whether a result transports across populations or settings. Methods for generalizability and transportability already formalize this problem as extending causal inferences from a study population to a defined target population (Dahabreh et al., 2021).

One element of a definition of causal research is explicit definitions of the theoretical and empirical estimands and assumptions under which the design and analysis identify that quantity. Randomization alone does not remove this requirement: PCORI-funded work on post-randomization selection in cluster trials shows how treatment-dependent identification, recruitment, or noncompliance can change the population represented by the observed data and the causal effect that can be identified (Li et al., 2024).

Recent causal frameworks similarly emphasize the target estimand, target population, target trial, and target validity (Lu et al., 2024). Multiple data sources do not establish causality by themselves, but they can show whether a causal conclusion depends on the population, measurement, or contextual factors from which it was obtained.

Recommendations: Fund reconciliation of consequential discordant evidence as a scientific activity, with harmonized comparisons and identified sources of disagreement as deliverables rather than another narrative review. Where claims are intended to extend beyond a single source population, allow and incentivize budgets for harmonization and prespecified analysis in a second data source. The work required to make those sources comparable should itself be recognized as part of the scientific contribution.

1.6 A model that already works

NIH need not build a reproducibility review from scratch. Since 2016, the Journal of the American Statistical Association (JASA) has operated a reproducibility initiative establishing minimum criteria for code, data, and workflow, and created the Associate Editor of Reproducibility role to implement them; it has since expanded from Applications and Case Studies to all original research (Wrobel et al., 2024). Public materials include a reviewer and author guide, a pre-submission checklist, a standardized artifact declaration form, and a template repository.

Directly transferable elements:

A standardized artifact declaration, submitted with progress reports or at closeout, stating what is reproducible, by which script, from which data, and where scope is limited and why.
A designated, credited reproducibility reviewer role. The single most important design feature of the JASA program is that this is a named position with real editorial standing, not volunteer labor. NIH could support analogous roles in biostatistics cores as an allowable and expected budget line.
A recognition mechanism. JASA gives an award for materials exceeding the minimum. NIH’s Replication Prize already establishes the model; the same logic could extend to reproducibility artifacts, recognized in award reporting and citable in biosketches.
Master-script executability: relative paths only, a wrapper script that reproduces results without modification, transparent run order, seeds set where exact reproduction is expected, versioned software documented and cited. Low-burden if adopted at project start rather than retrofitted.

2. Data, Software, and Model Sharing

Extend sharing from data to analytic artifacts. A dataset without the analysis is insufficient for validation. Deposited artifacts should include the executable pipeline, the computational environment (container or lockfile), seeds, and the selection procedure itself, assuming the model choice was data-dependent, not merely the final chosen model. Reporting only the winning specification discards what is needed to evaluate the inference.

Adopt synthetic facsimile data as the standard answer to restricted access. Where real data cannot be shared, JASA (for instance) requires a synthetic or facsimile dataset in the same format so code can at least be executed and inspected. This is a tested solution to NIH’s hardest sharing problem and could be made an explicit, fundable expectation rather than an ad hoc accommodation.

Measure executed reproductions, not availability statements. The proportion of awards with a data availability statement is helpful but less meaningful as a quality signal. The meaningful metric is the proportion of deposited artifacts a qualified third party can execute end-to-end in a clean environment. NIH could seed this by funding a small number of institutional or consortium reproduction services and reporting aggregate success rates.

Fund data resources, not just data access. Valuable clinical and administrative data often require substantial legal, technical, and scientific work before they are usable beyond the institution that generated them. Curation, linkage, harmonization, missing-data assessment, and measurement-quality evaluation are part of evidence production, not background plumbing. NIH should fund and recognize this work, including contributions from the institutions and personnel that create usable data resources, and expect data quality to be assessed relative to specified use cases rather than treated as an intrinsic property of a dataset. PCORI’s Patient-Centered Outcomes Data Repository provides one model: methodological projects can deposit documented, reusable study data expressly to support secondary analysis, reproduction, and further methods research (Li, 2026).

Integrating modalities is a methodological problem requiring renewed investment and awareness to protect the scientific record. Making heterogeneous data available together does not ensure that existing tools will handle them appropriately. Differences in measurement, missingness, dimensionality, and relevance to the scientific question across modalities require methodological development alongside infrastructure investment. As argued in §1.2, funding data integration without funding the methods to analyze integrated data legitimately risks producing a great deal of accessible data and a great many non-replicable findings drawn from it.

Fund software maintenance. Methods software is infrastructure that decays without support. Statisticians and data scientists maintain packages on CRAN that are used well beyond the projects that produced them, with no funding attached to that maintenance — which is the ordinary arrangement rather than an unusual one. NIH should permit and expect maintenance budgets, count software citations as scholarly output, and expect critical software to be cited in references alongside papers so its use is traceable.

3. Training and Mentorship

Training impact should be measured as development, not placement alone. Career outcomes remain useful, but they depend on trainees’ starting goals, opportunities, and the changing research labor market. NIH could instead pair placement outcomes with brief baseline and follow-up measures of trainees’ skills, confidence, research responsibilities, and career plans. Useful indicators include the number of trainees supported; gains in competencies such as study design, data analysis, communication, collaboration, and reproducible practice; and whether trainees gained responsibilities or opportunities they did not have at entry.

Access to appropriate mentorship is itself measurable. Trainees on non-methodological projects should have documented access to statistical and other relevant quantitative mentorship with protected effort, and training plans should treat reproducible workflows, including version control, environment management, and executable analysis as basic research competencies rather than specialty skills.

Suggested indicators: baseline and follow-up assessments of trainee competencies and career plans; trainee placement interpreted relative to stated goals at entry; access to appropriate methodological mentorship; and inclusion of reproducible-research competencies in training plans.

4. Collaboration

Recognize methodological and reproducibility service as a scored contribution. Reproducibility review, cross-lab statistical mentoring, maintenance of shared analytic infrastructure, and other methodological service are largely invisible in conventional productivity metrics. Making these contributions visible in biosketches and review criteria is a low-cost way to recognize work that directly supports team science and research rigor. Independent methodological service, including statistical service on DSMBs and other monitoring bodies, should likewise be recognized as a substantive contribution even when it does not result in co-authorship.

Collaborative statistical contribution is otherwise typically recorded as either co-authorship or an acknowledgment. Better indicators would capture both the role and structure of the collaboration, including the following:

Named quantitative collaborators with meaningful, specified effort, ideally distinguishable from nominal statistical support at token (e.g., supervisory or consulting) levels.
Contribution taxonomy (e.g., CRediT) at publication, naming methodology, software, formal analysis, and data curation roles.
Collaborative products beyond publications, including jointly submitted grants, workshops or meetings developed, shared data or software resources, and trainees taught or mentored across disciplines.
Whether the analysis plan was authored or co-authored by, and final analytic decisions authorized by, a named quantitative scientist with substantive responsibility for the analysis.

Collaboration should reflect more than the number of people a scientist has worked with, disciplines spanned, or projects contributed to. A quantitative scientist who develops a deep, sustained collaboration in one scientific area may contribute as much as one working across many fields. At the program or institutional level, NIH could instead characterize the reach and durability of collaborative networks, including repeated collaboration across projects and connections among otherwise separate scientific groups.

Analytic independence should be an explicit expectation. Individually reasonable analysis decisions, such as which covariates to adjust for, how to restrict a population, which subgroups to consider, how to handle outliers, etc., can cumulatively add up to an undocumented selection procedure, especially if there is pressure on such decisions in the direction of increased significance. Gelman and Loken (2014) call this the garden of forking paths: multiplicity exists even when only one analysis was ever run, because the path taken depended on the data. A quantitative collaborator supported only at token effort may lack the standing to resist result-driven analytic choices or to advocate independently for the ethical and judicious analysis.

Such independence should be supported by expecting named quantitative experts to participate meaningfully on research teams with substantive, not token, effort. Further, such independence is potentially measurable: whether the quantitative scientist holds authorship rather than acknowledgment, giving them standing to insist on a caveat or decline; whether their effort is protected on the award rather than discretionary; and, for high-stakes observational analyses, whether independent statistical review occurred, on the model of the independence already expected of trial statisticians.

5. Entrepreneurship and Translation

Translation measured through patents, licenses, and company formation systematically undercounts methodological translation. A method that moves from paper to open-source implementation to routine use in NIH-funded studies, clinical guidelines, regulatory submissions, or policy analysis is translated in the sense NIH cares about, and the process is measurable: downstream software citation, dependency counts in other NIH-supported analyses, adoption in guidelines and regulatory review.

NIH should include methods-to-practice pathways in translation indicators and support the unglamorous middle step between invention and adoption: testing, documentation, maintenance, training, and integration of methods into usable software and workflows. These activities currently fall through the cracks between methodological and translational funding.

6. Foundational Scientific Exploration

Valid inference after data-dependent selection is, at best, only partially solved. General solutions face fundamental limits; partial solutions exist but remain underused, in part because accounting honestly for selection often produces wider uncertainty, and wider uncertainty can be harder to publish.

Adjacent problems are similarly open and consequential: quantifying uncertainty for models produced by regularized selection; setting evidentiary thresholds that scale with the size and structure of a candidate space; establishing theoretical guarantees when covariate groups grow at different rates; and building transparent methods that remain competitive for outcome distributions where they currently underperform, such as zero-inflated and heavily skewed clinical endpoints.

This is precisely the foundational work the Initiative should fund: methodological, high-risk, without necessarily having an immediate clinical output, and directly relevant to whether findings across the rest of the portfolio replicate. As biomedical data and analytic choice expand, investment in replication alone addresses failures downstream; foundational statistical research can prevent some of them upstream.

7. Public Impact

Public impact should be evaluated along an impact pathway rather than inferred from citation counts alone. Evaluation frameworks distinguish intermediate outcomes such as reach, adoption, and sustained use from downstream changes in population health (Glasgow et al., 1999). NIH could similarly assess whether research produces trustworthy findings that reach intended populations, enter practice or policy, and are sustained over time.

Population outcomes are harder to attribute because the population is ultimately uncontrolled. Before asking whether research changed health outcomes, NIH often needs to know whether the resulting intervention, diagnostic, screening strategy, or clinical practice actually reached the population. NIH should incentivize studies linking changes in disease profiles and outcomes to changes in management and uptake, supported by data on where and among whom products and practices are used. This may require linking surveillance and health-system data with commercial, claims, registry, or other sources that capture diffusion beyond traditional research datasets.

Trustworthy evidence is a prerequisite for meaningful public impact. The replication rate of influential NIH-supported findings, tracked over time, would therefore provide an expensive but unusually direct indicator of whether the knowledge being propagated into research, practice, and policy remains supported when tested again.

Two additional indicators worth developing are the estimated resources committed downstream to findings that later failed to replicate, which NIH is uniquely positioned to assess; and the inspectability of clinical decision tools derived from NIH-funded research, including whether clinicians and patients cadesign stage through methodological collaboration. Methodological input on study design should be a fundable, expected component of research, not an unbudgeted courtesy. It has always been true that a specification chosen a priori on a firm statistical foundation generates no search to document. Until recently, the effort required to explore many specifications served as an informal check on doing so. AI agents remove that friction, making design-stage input more consequential, yet simultaneously easier to skip. The recommendations that follow address the multiplicity that remains, as it always will, since no analysis plan can anticipate everything a real dataset will demand.

Recommended indicator: an analytic provenance record. Where generative AI materially contributes to analytic code or decisions, require a short log — model and version, date, nature of the contribution, and, where AI was used to generate or compare analytic alternatives, the alternatives considered and the selection process. Systems understand and contest the basis of the model’s outputs.

Finally, not all public impact should be reduced to a count. Periodic expert evaluation of influential NIH-supported bodies of work could assess whether and how they changed scientific understanding, practice, policy, or population health over time.

8. Other Area: Generative AI, Agents, & Accountability

Generative AI and modern agentic analytic tools interact with every other category, and current trajectories run against the goals of this initiative.

AI-assisted analysis can quickly become an unrecorded selection procedure. An agent that iterates through dozens or hundreds of specifications and surfaces the one that “works best” is performing model selection at a scale exceeding what any analysis plan can reasonably specify in advance. The path taken by an LLM agent is rarely, if ever, reproducible from the prompt alone. If human-led data analysis represents Gelman’s “garden of forking paths,” an AI agent turns that garden into a random forest.

Mandating a complete audit trail of this search process is technically possible. However, with modern “chain of thought” LLMs, the massive volume of intermediate and erratic “thinking” steps renders any such record highly convoluted. An AI-assisted scientist must then decide among several alternatives. One option is to diligently store a large, complex record of these AI-dictated choices, and to publish this record alongside their research. Another option is to narratively summarize the AI-enabled search, which asks readers and funders alike to trust that whatever decisions were made, justified or not, were appropriate. Yet another option is to proceed as though the selected path was the “right” one all along. Every incentive pushes the scientist toward this final path of least resistance, which also carries the worst consequences for replicability. Consequently, as AI-assisted analysis becomes normalized, rejected specifications and the rationale for moving among them will disappear — not because scientists intend to conceal them, but because we lack both the norms of analytic accountability and the practical, rigorous tools to support them. This is multiplicity at machine speed, and it is poised to undermine replicability if left unchecked.

Recommendation: prevent multiplicity at the used for analytic work should make this information exportable rather than requiring investigators to reconstruct it after the fact. NIH could support development of provenance-capture tooling as research infrastructure.

Recommendation: retain locatable human accountability. A named individual should be able to explain and defend every analytic choice. NIH could require an attestation, parallel to existing authorship and conflict attestations, that a responsible analyst reviewed and understood AI-generated analytic code and decisions. AI may propose analytic choices, but a named human must adopt them and be able to state why each was selected over considered alternatives. This shifts analytic authority from AI dictation, which cannot be held to account for its mistakes, back to human judgment.

Recommendation: promote team science through shared analytic artifacts. Collaboration requires a shared, inspectable artifact to collaborate on. If each member works in a private AI-assisted workflow producing unshared code and unlogged decisions, team science becomes several parallel solo efforts with a joint author list (or worse, reverts to solo work with an AI). Shared repositories, executable master scripts, versioned environments, and declared selection procedures are the practical antidote, and NIH should fund and expect them at the team level.

9. Other Area: Unintended Consequences and Feasibility

Checklist theater. Any requirement can become a compliance artifact if checked for existence rather than content; data-availability statements are a cautionary example of the limitations of existence-based metrics. Mitigation: sample-based audits by qualified reviewers, and metrics based on execution rather than deposit.
Chilling exploration. Pre-specification requirements can be misread as prohibiting exploration. The requirement is labeling, not prohibition. Selection disclosure should let exploratory work proceed freely and be reported honestly as exploratory rather than presented as confirmatory.
Over-rewarding confirmation. If replication success becomes a scored metric for individual investigators, the rational response is to study safe questions. This ultimately risks stifled innovation and hinders research exploration. Replication incentives should attach to the enterprise and to replication teams, not to the original investigator’s record.
Regressive burden. Container registries, reproducibility reviewers, and synthetic data generation are easier to absorb at well-resourced institutions than at small ones. Mitigation: fund shared infrastructure, allow these costs explicitly in budgets, and scale expectations to award size and complexity rather than applying a uniform standard.
Career-stage asymmetry. Compliance and service expectations impose different burdens across career stages, with early-career investigators often least able to absorb them. Reproducibility service and exemplary artifacts should count concretely in review.
Metric capture generally. The indicators emphasized here, including executed reproductions, audited selection disclosure, candidate-space reporting, and downstream enablement, are relatively resistant to gaming because satisfying them requires producing much of the underlying scientific behavior they are intended to measure.

Feasibility across contexts: selection disclosure, candidate-space reporting, and artifact declaration are relatively low burden when incorporated prospectively into the workflow, because they record decisions as they occur rather than reconstructing them later. Executable-artifact standards carry moderate costs and scale with award size. Independent reproduction services are more resource-intensive and should be centrally or consortium-funded rather than mandated institution by institution.

Closing

Statisticians and biostatisticians have an unusual vantage point on this RFI: we are often among the first to recognize why a finding may not replicate, while not always having the standing or involvement needed to address the problem. The interventions most likely to succeed make analytic choices visible and attributable, because visibility is what allows a collaborator, reviewer, or reader to catch the problem, and because the alternative in an era of AI-accelerated analysis is a scientific record that is easy to re-run and impossible to interrogate.

The American Statistical Association appreciates the opportunity to comment on the Request for Information on Measuring and Rewarding Scientific Impact. Questions or comments may be directed to ASA Director of Science Policy Steve Pierson: spierson@amstat.org.

References

Arshi, B., Wynants, L., Rijnhart, E., Reeve, K., Cowley, L. E., & Smits, L. J. (2025). Number of Publications on New Clinical Prediction Models: A Bibliometric Review. JMIR Medical Informatics, 13, e62710. https://doi.org/10.2196/62710
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019). A Systematic Review Shows No Performance Benefit of Machine Learning over Logistic Regression for Clinical Prediction Models. Journal of Clinical Epidemiology, 110, 12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004
Dahabreh, I. J., Haneuse, S. J. A., Robins, J. M., Robertson, S. E., Buchanan, A. L., Stuart, E. A., & Hernán, M. A. (2021). Study Designs for Extending Causal Inferences From a Randomized Trial to a Target Population. American Journal of Epidemiology, 190(8), 1632–1642. https://doi.org/10.1093/aje/kwaa270
Gelman, A., & Loken, E.R. (2014). The Statistical Crisis in Science. American Scientist, 102, 460. https://doi.org/10.1511/2014.111.460
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the Public Health Impact of Health Promotion Interventions: The RE-AIM Framework. American Journal of Public Health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322
Harrell, F.E. (2015). Regression Modeling Strategies, 2nd ed. Springer.
JASA Reproducibility Guide and Pre-Submission Author Checklist: https://jasa-acs.github.io/repro-guide/
Lawlor, D. A., Tilling, K., & Davey Smith, G. (2016). Triangulation in Aetiological Epidemiology. International Journal of Epidemiology, 45(6), 1866–1886. https://doi.org/10.1093/ije/dyw314
Le, J. P., Morrison, J., Malhotra, A., Nemati, S., Wardi, G., & Ford, J. S. (2026). National Institutes of Health-Funded Artificial Intelligence and Machine Learning Research, 2019-2023: Cross-Sectional Study. Journal of Medical Internet Research, 28, e84861. https://doi.org/10.2196/84861
Li, F., Li, F., Liu, B., Papadogeorgou, G., Bobb, J., Thomas, L., & Wruck, L. (2024). New Causal Inference Methods for Cluster-Randomized Trials with Postrandomization Selection Bias. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/10.2024.ME.2019C116146
Lu, H., Li, F., Lesko, C. R., Fink, D. S., Rudolph, K. E., Harhay, M. O., Rentsch, C. T., Fiellin, D. A., & Gonsalves, G. S. (2024). Four Targets: An Enhanced Framework for Guiding Causal Inference from Observational Data. International Journal of Epidemiology, 54(1), dyaf003. https://doi.org/10.1093/ije/dyaf003
National Center for Complementary and Integrative Health. (2024a). PAR-25-274: Feasibility Clinical Trials of Mind and Body Interventions for NCCIH High Priority Research Topics (R34 Clinical Trial Required). National Institutes of Health. Posted November 21, 2024. Accessed August 17, 2026.
National Center for Complementary and Integrative Health. (2024b). PAR-25-268: Investigator Initiated Clinical Trials of Complementary and Integrative Interventions Delivered Remotely or via mHealth (R01 Clinical Trial Required). National Institutes of Health. Posted November 21, 2024. Accessed August 17, 2026.
National Institutes of Health. (2024). PA-25-305: NIH Research Project Grant (Parent R01 Clinical Trial Required). Posted December 18, 2024. Accessed August 17, 2026.
Peterson, R.A., & Cavanaugh, J.E. (2022). Ranked Sparsity: A Cogent Regularization Framework for Selecting and Estimating Feature Interactions and Polynomials. AStA Advances in Statistical Analysis, 106, 427–454. https://doi.org/10.1007/s10182-021-00431-7
Peterson, R.A., McGrath, M., & Cavanaugh, J.E. (2024). Can a Transparent Machine Learning Algorithm Predict Better than Its Black Box Counterparts? A Benchmarking Study Using 110 Data Sets. Entropy, 26, 746. https://doi.org/10.3390/e26090746
Peterson, R.A. (2025a). What Do We Mean by Glass-box, Exactly? Data Diction (blog). https://doi.org/10.59350/9dhes-thd51
Peterson, R.A. (2025b) [written under Hackman, P. pseudonym]. How Can I Guarantee a Significant Result? Data Diction (blog). https://doi.org/10.59350/6mzf8-xzd69
Peterson, R. A., Bird, S. M., Harris, L. M., Breheny, P. J., & Cavanaugh, J. E. (2026). A Ranked Sparsity Extension to the Bayesian Information Criterion: A Tool for Selecting Variables from Multiple Data Modalities. Entropy, 28(9), 943. https://doi.org/10.3390/e28090943
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Tay, J.K., & Tibshirani, R. (2020). Reluctant Generalised Additive Modelling. International Statistical Review, 88(Suppl 1), S205–S224. https://doi.org/10.1111/insr.12429
Wrobel, J., Hector, E.C., Crawford, L., D’Agostino McGowan, L., da Silva, N., Goldsmith, J., Hicks, S., Kane, M., Lee, Y., Mayrink, V., Paciorek, C.J., Usher, T., & Wolfson, J. (2024). Partnering with Authors to Enhance Reproducibility at JASA. Journal of the American Statistical Association, 119(546), 795–797. https://doi.org/10.1080/01621459.2024.2340557
Yu, G., Bien, J., & Tibshirani, R. (2019). Reluctant Interaction Modeling. Preprint. https://doi.org/10.48550/arXiv.1907.08414

Citation

For attribution, please cite this work as:

“ASA Response to NIH Request for Information: Measuring and Rewarding Scientific Impact.” 2026. Data Diction (blog). August 19, 2026. https://doi.org/10.59350/pmkk7-kym57.

Can an AI assistant handle the tedious parts of academic writing?

Ryan Peterson — Tue, 28 Apr 2026 00:00:00 GMT

Author Note

This post deviates from our usual AI use policy as an experiment using Claude Code, the result of which will become clear as you read.

What if you could offload the parts of academic writing that have nothing to do with writing? Not the thinking, not the modeling, not the prose — but the LaTeX errors, the git housekeeping, the YAML frontmatter surgery that eats an afternoon every time you switch journals.

I recently put this to the test with Claude Code, Anthropic’s AI coding assistant. Over a single conversation, I used it to clean up a git repo, migrate a manuscript from one journal template to another, and debug the resulting build errors. Here’s how it went.

The setup

I’m working on a paper targeting MDPI’s journal Entropy, but the manuscript (RBIC_Multimodal.Rmd) was still using the rticles::elsevier_article template from an earlier submission plan. The repo also had some generated figure files tracked in git that shouldn’t have been. Routine housekeeping, but the kind that quietly devours time.

Claude Code runs in your terminal (or IDE) and has direct access to your project files, shell, and git. You describe what you want, it proposes a plan, and you approve or redirect. It’s a conversation, not a one-shot prompt.

Task 1: Stop tracking build artifacts

The RBIC_Multimodal_files/ directory — full of generated PDFs from knitr — was being tracked in git. These get regenerated every build, so they just add noise to diffs.

I asked Claude about it, and it laid out the standard three-step fix:

# Add to .gitignore
echo "RBIC_Multimodal_files/" >> .gitignore

# Remove from git's index (but keep the local files!)
git rm -r --cached RBIC_Multimodal_files/

# Commit and push
git commit -m "Stop tracking RBIC_Multimodal_files/"
git push

The key here is the --cached flag — it untracks the files without deleting them from disk. Claude explained this clearly and then, after I confirmed, executed it. Eight PDFs removed from the repo, .gitignore updated, done.

Note

Nothing here is beyond a quick Stack Overflow search. But Claude handled it end-to-end — checking what was tracked, editing .gitignore, running the commands, committing — without me switching contexts.

Task 2: Elsevier to MDPI Entropy

This is where things got more interesting. Switching rticles templates isn’t just changing one line in the YAML. The author/affiliation format is different, the citation engine changes (CSL to natbib), extra metadata fields are required, and you need a Definitions/ folder with the MDPI class files.

I asked Claude to help ensure the proper template was in use. It:

Spawned a sub-agent to research rticles::mdpi_article requirements, YAML fields, and Entropy-specific settings
Read my existing Rmd frontmatter
Rewrote the YAML from scratch

Here’s a simplified before/after:

Before (Elsevier):

output:
  rticles::elsevier_article:
    keep_tex: true
author:
  - name: "Ryan A. Peterson"
    affiliation: a,b
    footnote: "Corresponding Author"
address:
  - code: a
    address: "Department of Biostatistics..."
csl: biometrics.csl

After (MDPI):

output:
  rticles::mdpi_article:
    extra_dependencies: longtable
    keep_tex: true
author:
  - name: Ryan A. Peterson
    affil: "1,2,*"
affiliation:
  - num: 1
    address: |
      Department of Biostatistics...
    email: ryan-peterson@uiowa.edu
journal: entropy
type: article
status: submit

Claude also copied the Definitions/ folder from the installed rticles package, added required back-matter fields (acknowledgement, funding, conflictsofinterest), and removed packages that conflict with mdpi.cls (like endfloat and the custom caption width).

Task 3: Debugging the build

I rendered the document in RStudio and fed the errors back to Claude. Three rounds of fixes followed.

Round 1: Missing Ghostscript

! epstopdf Error: Required program gs not found

The MDPI logos are .eps files, and pdfLaTeX needs Ghostscript to convert them. Claude proposed two options: install Ghostscript, or pre-convert the logos to PDF so collaborators don’t hit the same issue.

I pointed out that option 2 is better for the team:

“It seems like [option 2] is the better option because if others are rendering this document on their machines, they may run into a similar issue.”

Claude agreed, installed Ghostscript via conda (which I already had), converted the three EPS logos to PDF, and then patched mdpi.cls to drop the .eps extensions from \includegraphics calls. Now pdfLaTeX finds the PDFs automatically — no Ghostscript required at build time.

Round 2: A sneaky bibliography entry

The next error looked like a math issue:

! Missing $ inserted.
l.22 ...95/3/10.1093/biomet/asn034/2/asn034.pdf]}}

I told Claude I’d seen this kind of thing before with tables and escape characters. But it traced the actual source to a .bib entry with an eprint field containing a URL-like path full of underscores. Under natbib, those underscores get interpreted as LaTeX subscript operators. The doi and URL fields already covered the same reference, so removing eprint was the clean fix.

Tip

This was the moment that sold me. I had a plausible (but wrong) hypothesis about the error source. Claude didn’t anchor on my suggestion — it searched the .bib file, matched the error text, and found the real cause.

Round 3: Unused packages

Package gensymb Warning: Not defining \perthousand.

I wasn’t sure whether gensymb was actually used anywhere in the paper. Claude searched the entire Rmd for any gensymb commands (\degree, \celsius, \micro, etc.) — found nothing but the \usepackage line itself. Removed it.

The collaboration pattern

What I found most useful wasn’t any single capability — it was the iteration loop:

I describe the goal
Claude proposes a plan
I approve or redirect
Claude executes
I report results (or errors)
Repeat

I stayed in control throughout. Claude asked before running destructive commands. When I redirected (the EPS portability issue), it adapted immediately. When I told it the undefined references were expected (those chunks have eval=FALSE while I re-run an analysis), it moved on without trying to “fix” them.

Key takeaways

Claude Code is a collaborator, not a button. It works best with back-and-forth. The human provides judgment; the AI handles execution and research.
It handles tedious format migrations well. YAML rewriting, class file patching, bibliography fixes — exactly the kind of work that’s straightforward but time-consuming.
It debugs iteratively. Each error got diagnosed and fixed in one round, not blindly retried.
Human oversight matters. I caught the portability issue with EPS conversion. I knew the undefined references were expected. The AI didn’t need to know everything — it just needed to listen when I told it.
It’s git-aware. It reads status, writes descriptive commit messages, and pushes when asked — but only when asked.

One more thing

At the end of our session, I asked Claude to generate a Quarto reveal.js presentation summarizing everything we’d done. It wrote 20 slides with accurate quotes from our conversation, code blocks from the actual commands, a mermaid diagram of the workflow, and custom SCSS theming.

Then I asked it to fix three issues with the first draft. It did.

Then I asked it to write this blog post.

It did that too.

Author Note

It felt incorrect to say that I – Ryan Peterson – authored this post, because the “I” used throughout – Claude generating text through my perspective – is not me. The only text written by me is contained in the two “Author Note” boxes.

We decided to leave the text as it is, without our usual review process, so that it stands as a genuine experiment of what Claude is capable of from a pure writing perspective. This post therefore represents an exception to a key GMWG value to be human first:

We pledge to only use AI as a supporting writing tool

It also demonstrates the importance of such a pledge.

In this and future posts, any AI generated content will be clearly denoted as such with a dotted border. For example:

This text is AI-generated…

…This text is not.

Citation

For attribution, please cite this work as:

Peterson, Ryan. 2026. “Can an AI Assistant Handle the Tedious Parts of Academic Writing?” Data Diction (blog). April 28, 2026. https://doi.org/10.59350/vj2m3-s2t36.

How can I guarantee a significant result?

Perry Hackman, PhD — Mon, 22 Dec 2025 00:00:00 GMT

Dear student,

Before you embark on your “career” as a statistician, you must purge yourself of a childish misconception: that our job is to seek truth. Truth is stubborn, unpredictable, and worst of all, often unpublishable. Scientists crave confidence, the journals crave significance, and we, if we are clever, can provide both without the nuisance of real rigor.

In this post series, I will instruct in the Statistical Dark Arts. Today, I’ll describe how to always ensure a publishable result with model selection and unadjusted post selection inferences (UPSIs).

Why UPSIs?

You will frequently come across scientists with ambitious ideas, saying things like:

I’ve conceived a brilliant new way to treat chronic pain effectively. I don’t want to waste time, efforts, and money on a result that ends up being insignificant… How can I guarantee a significant result??

No statistician wants to contribute to a null finding. Interpreting those is too hard! I’ll let you in on a statistical secret: there IS a way to guarantee statistical significance – and it’s actually extremely common in science.

It’s called unadjusted post-selection inference (UPSI).

With good model selection tools and UPSIs in your toolbelt, you can turn any negative study into a positive one – guaranteed.

Example: a study for treatment of chronic pain

Say the researcher who approached you is planning a cross-over study where patients were given one of two treatments for chronic pain. Patients will record their overall pain each day for 1 week while undergoing treatment A, and 1 week while undergoing treatment B.

Investigators are hoping to determine the difference between treatments in the average pain score. Our outcome is thus a continuous measure for pain reduction: for each subject, this value reflects the difference in that subject’s weekly average pain score between the two candidate treatments.

In addition to determining whether the treatment works on average in their population, investigators wish to determine specific subgroups that might see the most benefit. They are primarily interested in subgroups by age and sex, as well as patient self-reported data such as alcohol consumption and physical activity.

Note

Make sure they have an exhaustive list of candidate effect modifiers; I’d even suggest a few new ones like left/right handed-ness and coffee consumption.

Here’s a way to list all the possible subgroups of this set of variables:

subgroups <- expand.grid(
  age = c("18-35", "36-50", "51-65", "66-80", "80+"),
  sex = c("Male", "Female"),
  hand = c("Left", "Right"), 
  coffee = c("0", "1", "2+"),
  alcohol = c("0", "1", "2", "3+"),
  physical_activity = c("0", "1", "2", "3+")
)
nrow(subgroups)

[1] 960

This is a LOT of subgroups! And it’s way too many subgroups to possibly sift through, at least without tipping off the statistical reviewers about multiplicity.

Multiplicity

Gone, alas, are the carefree days when we could run tons of tests, report the precious few that were significant, and quietly ignore the rest. Some meddlesome truth-seekers eventually caught on and began scolding us for our ingenuity, slapping the sinister label multiplicity onto our beloved golden-egg-laying goose. Now the reputable journals know to look for it. Our approach must become more subtle.

Here’s the key: turn to model selection. Write in the analysis plan:

We will use forward selection to determine which variables or interactions improve our model’s predictions, adding each candidate predictor in a stepwise fashion until the AIC indicates the model’s predictions can no longer be improved.

Sounds rigorous and honest, right?

This simple trick can guarantee significant results, and help you ensure your study will find a statistically significant result by the end, even when none exists!

How about that?! ZERO RISK! (well, to us anyway).

As a bonus, the more interactions we include that are truly null, the more unnecessarily opaque our model becomes – talk about a win-win.

Virtual Trial 1

Let me illustrate via a simulation. Let’s virtually collect patients in our trial.

library(tidyverse)

set.seed(21)
n <- 100        # Sample size  

# Simulate recruitment (y: outcome)
y <- rnorm(n)

# Simulate recruitment (assume each new person has random subgroup)
x_idx <- sample(1:nrow(subgroups), n, replace = TRUE) 
X <- subgroups[x_idx,]
simdata <- tibble(y = y, X)
simdata

# A tibble: 100 × 7
          y age   sex    hand  coffee alcohol physical_activity
                            
 1  0.793   36-50 Female Left  0      2       1                
 2  0.522   80+   Female Left  2+     1       3+               
 3  1.75    36-50 Female Left  2+     1       0                
 4 -1.27    66-80 Female Right 0      3+      0                
 5  2.20    51-65 Male   Right 2+     0       2                
 6  0.433   36-50 Male   Left  0      2       1                
 7 -1.57    66-80 Male   Right 0      2       0                
 8 -0.935   80+   Female Left  1      3+      3+               
 9  0.0635  36-50 Female Left  2+     0       2                
10 -0.00239 66-80 Male   Left  0      2       0                
# ℹ 90 more rows

We need to create interactions to select from:

# Create model matrix w/all interactions
X2 <- model.matrix(y ~ . * ., data = simdata)[,-1]

# Sanity check: make sure these columns represent main effects + interactions
sample(colnames(X2), 10)

 [1] "alcohol2"                     "sexFemale:coffee1"           
 [3] "alcohol1:physical_activity3+" "age66-80:handRight"          
 [5] "alcohol1:physical_activity1"  "age51-65:sexFemale"          
 [7] "handRight:coffee1"            "age66-80:physical_activity2" 
 [9] "age51-65:coffee2+"            "age36-50:coffee1"

# combine into data set for model fitting
simdata2 <- data.frame(y=y, X2)

OK! We have 93 candidate predictors for model selection. Don’t worry, unless our paper’s reviewers are extremely thorough, we don’t have to report this number. Our final model will have many fewer.

Model selection is a simple task. The following code uses forward step-wise selection with AIC to build an optimally-predicting model, per our analysis plan.

library(selectInferToolkit)
fit_aic <- select_stepwise_ic(y ~ ., data = simdata2, direction = "forward")

Caution

While other packages provide p-values after selection, most do so without being transparent, making it difficult to ensure they are truly UPSIs. If you aren’t careful with these less transparent tools, you might report p-values that are somehow diabolically adjusted towards insignificance. In contrast, the selectInferToolkit package, available on GitHub, helps you explicitly use UPSIs for inference:

infer_upsi(fit_aic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	−0.07	0.21	0.315
age36.50	0.17	−0.06	0.41	0.156
age66.80	−0.35	−0.55	−0.15	<0.001
handRight	0.23	−0.02	0.47	0.076
age80..sexFemale	−0.29	−0.51	−0.07	0.013
age36.50.handRight	−0.21	−0.43	0.01	0.071
age80..handRight	−0.22	−0.44	0.00	0.053
age66.80.coffee1	0.45	0.19	0.71	<0.001
age36.50.coffee2.	0.19	−0.01	0.40	0.070
age51.65.coffee2.	0.33	0.16	0.49	<0.001
age36.50.alcohol1	−0.13	−0.31	0.06	0.178
age66.80.alcohol1	−0.26	−0.50	−0.01	0.042
age66.80.alcohol2	−0.19	−0.40	0.02	0.081
age51.65.physical_activity1	−0.33	−0.50	−0.17	<0.001
age51.65.physical_activity2	−0.18	−0.35	−0.01	0.039
sexFemale.alcohol1	0.33	0.15	0.50	<0.001
sexFemale.alcohol2	0.19	0.02	0.37	0.036
sexFemale.alcohol3.	−0.17	−0.37	0.03	0.091
sexFemale.physical_activity2	−0.27	−0.44	−0.10	0.003
handRight.alcohol3.	−0.16	−0.36	0.04	0.123
handRight.physical_activity3.	−0.29	−0.48	−0.10	0.004
coffee2..alcohol3.	0.12	−0.05	0.29	0.172
coffee1.physical_activity1	0.17	0.01	0.33	0.040
alcohol3..physical_activity3.	0.25	0.08	0.42	0.006

And there you have it!

For many subgroups, treatment A worked great (increased pain scores, significant positive effects). Treatment A didn’t work so well for others (with negative significant effects), for whom it actually appears to decrease pain scores. These results indicate we could stand to maximize pain by giving certain patients A, and other patients B. I wouldn’t fret too much about how we included interactions without their constituent main effects; leave that subtlety to the media to figure out.

Caution

Sometimes, researchers want to actually decrease pain scores; you might want to check on this.

What’s with the high p-values?

I know what you’re thinking - won’t it be confusing to include all those results with high p-values?

An expert tip: if you want to make fewer discoveries and get even lower p-values, simply use BIC instead of AIC. BIC only lets the most significant results into the model, and you can say it’s asymptotically consistent, which sounds equally rigorous to AIC, which is asymptotically efficient.

In this example, here’s the model selected via stepwise BIC:

fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                              direction = "forward", 
                              penalty = "BIC")

infer_upsi(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	−0.09	0.24	0.393
age36.50	0.27	0.10	0.45	0.003
age66.80	−0.38	−0.57	−0.18	<0.001
age66.80.coffee1	0.23	0.04	0.42	0.019
age51.65.coffee2.	0.29	0.11	0.46	0.002
age51.65.physical_activity1	−0.24	−0.42	−0.07	0.007
sexFemale.physical_activity2	−0.32	−0.49	−0.15	<0.001

Voila – the headlines practically write themselves!

How UPSIs work so well

Let me let you in on a little secret. If you were paying attention, you’d have seen the outcomes in the preceding example, , were generated completely randomly. There was, by design, no true relationship to discover at all. Yet, I was virtually guaranteed to find a significant result. This is due to multiplicity’s clever younger brother, selective inference.

Selective inference is a difficult problem to solve; in fact, some call it impossible. Others hold out hope. At any rate, UPSIs have inertia and incentives on their side.

Later posts to this blog will go into specific solutions to the selective inference problem and how to implement them in R. Fair warning though, these solutions are often less promising and will certainly be less significant.

For now, let me finish convincing you that UPSIs are indeed effective at resolving the “publish or perish” dilemma; with USPIs, we can thrive at both! It should suffice to repeat this simulation again, as though it were performed in a parallel universe.

Virtual Trial 2

simdata2$y <- rnorm(n)
fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                              direction = "forward", 
                              penalty = "BIC")

infer_upsi(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.03	−0.14	0.21	0.715
age66.80.sexFemale	−0.25	−0.43	−0.07	0.008
age80..coffee2.	0.30	0.11	0.50	0.003
age80..physical_activity1	−0.26	−0.46	−0.07	0.009

Indeed, we still discover “significant” relationships. And, these are completely different treatment effect modifiers. In this parallel universe, our scientist friends publish completely different effects and will no doubt pour their hard-fought resources into what we well know are wild goose chases.

This is not a fluke.

In case you are still skeptical, let’s repeat this 50 times, as though we ran the study in 50 parallel universes.

sim_results <- list()
for(s in 1:50) { 
  simdata2$y <- rnorm(n)
  fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                                direction = "forward", penalty = "BIC")
  sim_results[[s]] <- infer_upsi(fit_bic, data = simdata2) %>% 
    tidy() %>% 
    filter(coef != 0) 
}

all_selections <- bind_rows(sim_results, .id = "simulation")

main_results <- all_selections %>%
  filter(term != "(Intercept)") %>%
  group_by(simulation) %>%
  summarize(
    any_p_lt_05 = any(p_value < 0.05), 
    n_selections = n()
  )

Using BIC, we found at least one significant effect in 46 of 50 trials, and on average these trials found 2.6 significant effects. We achieved falsely significant results 92% of the time.

If you’re thinking “hmm, my study still might have a chance of failing”, well good news. AIC finds a significant result nearly 100% of the time. We also only considered pairwise interactions between 6 candidate predictors. To minimize the risk of a negative study, one could consider higher level interactions or a greater number of subgroups, and a positive result quickly becomes a sure thing.

As I’ve said, UPSIs have incentives and inertia on their side. The more broad we make this practice, the better we can do at ensuring that the publications keep flowing, regardless of the truth.

Despicably yours,

P. Hackman

Conclusion

Key Takeaways

Unadjusted post-selection inference nearly guarantees false positives
Searching for subgroup effects is a model selection problem
Model selection algorithms don’t make multiplicity issues go away; they make them more subtle and harder to adjust for.

Future Threads

What are some alternatives to UPSI-based inference, and how are they implemented in R?
What is the selectInferToolkit package?

Appendix

R Session Info

Does debiasing estimates lead to better predictions?

Logan Harris — Mon, 15 Dec 2025 00:00:00 GMT

Reviewers

Ryan Peterson (2025-11-24).
Patrick Breheny (2025-12-12).

What connotation do you attach to the word “bias”? A negative one?

In this post we will see why not all bias is bad… at least when it comes to building predictive models. In fact, for many years, statisticians have recognized the benefits of biased estimators in reducing prediction error. Perhaps you knew this, but if not, don’t worry. That is the purpose of this post.

Modeling Goal

When starting off with a data analysis, it is important to outline the primary goals. Whether you realize it or not, in virtually all cases, a primary goal is to use the data to develop a predictive model.

Consider a clinical trial evaluating a new drug for metastatic lung cancer. We say our goal is to “estimate the treatment effect” of our new drug. However, estimating a treatment effect is fundamentally a predictive task. That is, we need to predict what would happen with treatment and what would happen without treatment. The treatment effect is the difference between those two predictions*, the better our predictions… the better our estimate of the treatment effect is.

A pre-dictor is pre-what, exactly?*

When we say a treatment effect is “the difference between two predictions,” that only works if the predictions are made using information that comes before the treatment starts. These are true pre-treatment predictors.

If we accidentally include a model “predictor” that is measured after treatment begins, the treatment could affect it, and the real treatment effect gets obfuscated. The model might appear to “predict” well, but the truth is that it’s not “predicting” at all. Models built this way can give a misleading estimate of the effect of the treatment because part of the effect has been absorbed by the post-treatment variable.

For a valid treatment effect, we must base predictions only on information that could not have been influenced by the treatment – valid pre-dictor variables.

Now, suppose with treatment alone we can predict remission status with 60% accuracy. If you also have patients’ genetic information, what should you do with that?

Including it directly into a model is problematic because the human genome is large*. Unless you have a massive number of patients in your trial (hundreds of thousands) you will NEED to make assumptions about the plausible effects of someone’s genetic information on how well the new treatment works. These assumptions allow us to fit a model with all the genetic information but also intentionally introduce bias to the estimates. This is what a method known as penalized regression does, which we will cover momentarily.

Human Genome*

The human genome is estimated to contain between 30,000 and 40,000 genes!

Now, suppose including genetic information allows us to predict if a patient will be experience remission with 75% accuracy. But this was with biased estimates?! Surely, if we could remove this bias we would get even better predictions, right?

To answer this, we will now break down what makes a set of estimates good at prediction.

Breaking down predictive performance

Bias-variance tradeoff

When we estimate parameters for a model, we are always juggling two competing forces: bias, which measures how far our average estimate is from the truth and variance, which reflects how much our estimates would change if we collect a new sample. A model’s ability to predict an outcome is dependent on both. Increasing or inducing bias can often reduce variance, whereas decreasing bias can often increase variance. Good predictions come from striking a balance between the two. It is not uncommon to be able to intentionally introduce a little bias and be able to drastically reduce variance resulting in better predictive abilities.

It is like comparing one friend who is always 5 minutes late to one that is sometimes 30 minutes early and other times 30 minutes late. While the latter is on average on time, you’d probably describe the one who is always 5 minutes late as more reliable.

If you are someone who likes mathematical details, keep reading. But if the previous example makes sense, feel free to skip to the next section.

For a sample of size , suppose we have a continuous outcome stored in a length vector and features on each sample unit stored in an matrix . We will focus on a linear predictor setting. That is, we want to predict based on a linear combination of the features in . We do this by estimating parameters and then obtaining predictions :

is usually estimated in a way that is based on minimizing , known as the residual sum of squares. However, to assess predictive performance we might consider mean square prediction error (MSPE). MSPE shifts our focus to how well our model is expected to predict a single out-of-sample observation (an observation with predictors that was not in the original sample used to estimate the model).

Assuming that the error structure for is normally distributed (i.e. ) and letting be the error corresponding to and , MSPE can be decomposed as follows:

To make the point easier to see, assume we have just a single predictor (i.e., ) and consider two different candidate estimators for , and . Then, the difference in their MSPEs is:

So now we see the details of the earlier claim that predictive performance is a function of both variance and bias of the estimates.

Let’s work through a toy example. Consider an example where estimator A has a bias of -0.5 and has a variance of 1. This scenario might reflect a penalized estimate for , where the estimator wants to shrink the estimate by a certain amount. Consider an alternative estimator B which corrects the bias of estimator A so that , but this increases its variance to 1.5. Then

The MSPE for the “debiased” estimator is greater than that of the “biased” estimator. Correcting the bias in the estimator had a negative impact on predictive performance.

So, if we care about predictive performance, then bias in our estimates is not necessarily a bad thing.

Penalized Regression

A common rule of thumb is that you need at least 10 observations ( denotes number of observations) per predictor ( denotes number of predictors) in traditional linear regression settings to estimate “stably” using ordinary least squares (OLS), but even more may be required. If but , this rule of thumb would suggest we are in a gray area where estimates for tend to be highly variable and can lead to poor predictions. Of course, if , then it is not possible to use OLS at all.

Adding bias to the estimation process is helpful in both of these scenarios. Enter penalized regression methods.

In general, such methods “penalize” larger estimates of by an amount dictated by a tuning parameter, . The penalty acts like a “complexity tax” forcing the model to stop chasing noise and to start paying attention to the real signal. So, while this introduces bias in the estimates, it also reduces their variance, often leading to superior predictive performance. In fact, this can be true even when is much larger than . In high-dimensional or noisy settings or if there is large amount of correlation between predictors, the gains are often large enough that biased estimators dominate unbiased ones in prediction.

Popular Penalized Regression Methods

The least absolute shrinkage and selection operator (lasso)
Ridge regression
Elastic net (simply a mix of 1 and 2)

It’s easiest to show this with data, so we’ll now turn to an example.

Example: Predicting Leukemia Subtype

To start, we’ll load some libraries and a helpful function.

library(hdrm)
library(ncvreg)
library(hdi)
library(ggplot2)

estimate_intercept <- function(beta, X, y) {
  eta_no_intercept <- drop(X %*% beta)

  f <- function(alpha) {
    p <- plogis(alpha + eta_no_intercept)
    sum(y) - sum(p)
  }

  uniroot(f, interval = c(-1000, 1000))$root
}

Now, consider a data set for predicting leukemia subtype using gene expression data (Golub et al. 1999).

brca1 <- hdrm::read_data("Golub1999")
X <- brca1$X
y <- brca1$y == "ALL"
n <- length(brca1$y)

This dataset has 47 patients with acute lymphoblastic leukemia (abbreviated as ALL) and 25 patients with acute myeloid leukemia (AML). There are 7129 gene expression features, putting us in the high-dimensional realm of .

Note

When , the variance of for an ordinary, unpenalized regression of any type is essentially because the model is not identifiable. It’s like trying to solve a puzzle with more missing pieces than clues; many solutions look “possible,” so you can’t tell which one is the real one.

We split the dataset 50/50 into train and test sets.

set.seed(2806)
idx_train <- sample.int(n, size = floor(n * 0.5))
Xtrain <- X[idx_train,]
ytrain <- y[idx_train]
Xtest <- X[-idx_train,]
ytest <- y[-idx_train]

Then, we will perform a penalized regression method called the “lasso” and select the tuning parameter using cross validation:

set.seed(2806)
cv_fit <- cv.ncvreg(Xtrain, ytrain, penalty = "lasso", family = "binomial")
lambda_min <- cv_fit$lambda.min

Finally we can visualize the predicted probabilities on the testing data to see how well the selected lasso model is able to differentiate between the two disease subtypes.

lasso_res <- data.frame(
  predicted_prob_ALL = predict(cv_fit, Xtest, type = "response"),
  class = factor(ytest, levels = c(0, 1), labels = c("AML", "ALL"))
)

lasso_res$prediction_quality <- ifelse(
  lasso_res$predicted_prob_ALL > .5 & lasso_res$class == "AML" | 
    lasso_res$predicted_prob_ALL < .5 & lasso_res$class == "ALL",
  "Bad", 
  "Good"
)

ggplot(lasso_res, aes(x = class, y = predicted_prob_ALL, color = prediction_quality)) +
  geom_point(alpha = 0.7, position = position_jitter(width = .1)) +
  theme_minimal() +
  scale_color_discrete(name = "Prediction") + 
  ylab("Lasso-based predicted probability ALL") +
  xlab("")

Along with the fact that we reduced the variability of estimation (enough to actually fit a model), introducing bias also helps mitigate over fitting which leads to models that are more generalizable with improved out-of-sample predictions. With this example, we have good (though imperfect) separation between the two subtypes.

This is not the only benefit of introducing bias here. Penalties like the lasso produce sparse fits, with many coefficients set exactly to zero (the second “s” in lasso does literally stand for selection). Weak and noisy predictors are removed, leading to more interpretable glass-box models.

The Pitfall of Debiasing

That being said, it is reasonable to think that reducing the bias of the estimates for would lead to better predictive performance. One such example is the debiased lasso (also known as the desparsified lasso, Zhang and Zhang (2014)) . However, while this method may be good for providing asymptotically unbiased estimators, debiasing can come at the cost of reintroducing a high amount of variance.

As a result, debiasing often leads to noticeably worse predictions. To see this, we will fit the desparsified lasso to our leukemia dataset.

## Takes about 20 minutes; cached 
debiased_fit <- hdi::lasso.proj(Xtrain, ytrain, family = "binomial", lambda = lambda_min)
saveRDS(debiased_fit$bhat, "debiased_fit.rds")

In the code below, we re-estimate the intercept after we obtain the debiased estimates. Then, we can visualize the predictions as we did with the lasso.

debiased_fit <- readRDS("debiased_fit.rds")
debiased_res <- data.frame(
  predicted_prob_ALL = plogis(estimate_intercept(debiased_fit, Xtrain, ytrain) + Xtest %*% debiased_fit),
  class = factor(ytest, levels = c(0, 1), labels = c("AML", "ALL"))
)

debiased_res$prediction_quality <- ifelse(
  debiased_res$predicted_prob_ALL > .5 & debiased_res$class == "AML" | 
    debiased_res$predicted_prob_ALL < .5 & debiased_res$class == "ALL",
  "Bad", 
  "Good"
)

ggplot(debiased_res, aes(x = class, y = predicted_prob_ALL, color = prediction_quality)) +
  geom_point(alpha = 0.7, position = position_jitter(width = .1)) +
  theme_minimal() +
  scale_color_discrete(name = "Prediction") +
  ylab("Debiased-lasso-based predicted probability ALL") +
  xlab("")

Debiasing the original lasso point estimates results in 1 additional subject with AML being misclassified into the ALL group. Additionally, whereas the lasso correctly predicted all subjects with ALL, debiasing leads to 2 incorrect AML predictions. This is a reduction in accuracy from 87.5% to 78.1%!

While you might say you don’t care about prediction, in a future post we will explore why you can’t afford not to.

Take aways

Biased estimation can improve a model’s predictive performance
Reducing bias (via debiasing) often worsens predictions
The lasso and other penalized regression methods can yield better glass-box models

Follow-up Questions

The following questions might be of interest for new posts:

What implications does bias have on inference and model interpretation?
When might debiasing be a good idea?

References

Golub, Todd R., Donna K. Slonim, Pablo Tamayo, Christine Huard, Michael Gaasenbeek, Jill P. Mesirov, Hilary Coller, et al. 1999. “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring.” Science 286 (5439): 531–37. https://doi.org/10.1126/science.286.5439.531.

Zhang, C. H., and S. S. Zhang. 2014. “Confidence Intervals for Low Dimensional Parameters in High Dimensional Linear Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76 (1): 217–42.

Citation

For attribution, please cite this work as:

Harris, Logan. 2025. “Does Debiasing Estimates Lead to Better Predictions?” Data Diction (blog). December 15, 2025. https://doi.org/10.59350/h2397-twd61.

What do we mean by glass-box, exactly?

Ryan Peterson — Mon, 24 Nov 2025 00:00:00 GMT

Reviewed by Logan Harris on November 21, 2025

Introduction

What are glass-box models?

Glass-box models are transparent, intrinsically interpretable alternatives to their opaque counterparts, black box models. Data scientists typically consider regression-based methods and sparse decision trees as “glass-box”. In this post, I describe the benefits of glass-box methods for modeling data, arguing for the importance of intrinsic interpretability. An intrinsically interpretable model is one whose internal logic is clear enough that a human can see why it makes each prediction, not merely trust a separate explainer.

I hope to convince you that the prevailing definition for glass-box models requires significant refinement in order to be true to its namesake.

Just because a model is a “simple regression” does not mean it is interpretable. In fact, model selection methods that attempt to create more interpretable models often invalidate inference by making p-values unjustifiably low, and confidence intervals (CIs) unrealistically narrow.

Are glass-box models a novelty? Or a necessity?

Consider the useful analogy in the image below, which compares two very different uses of glass as a window into a process. On the left, a penny pressor. I always considered these to be a fun novelty (I still don’t understand how they are legal!). The glass box is part of the appeal - you get to watch a penny turn into a souvenir. Cool!

On the right, you see scientists at the Hanford Site in Washington peering through shield windows while working on the plutonium that would eventually be used in the world’s first atomic bomb explosion. One intact version of these shield windows used during the Manhattan project is currently priced at 10 million dollars, although you can purchase a fragment of it for much less shop.minimuseum.com.

The example of the Hanford Site clearly shows how in high-stakes decisions or systems, opacity is dangerous. Therefore, sometimes, a glass-box model isn’t a novelty like with the penny pressor - it’s a requirement for oversight and correction.

If the AI doomers are right that AI represent existential threats on par with nuclear war… it is easy to see that glass-box models should be (at least in many cases) classified as a necessity, not a novelty.

Why do we build models in the first place?

Let’s start with a quote you have probably heard before from George Box:

All models are wrong, but some are useful.

What makes them useful? Good models help us to:

make good predictions
understand phenomena.

The mix of these two goals are entirely context dependent. In my experience, it is rare for the focus to be entirely on one of these goals.

A brief history of model selection

Hypothesis testing

Starting in the 1700s and through the 1980s, hypothesis testing & p-values were the main way to build models. In the typical set up, two competing models are compared (e.g., a null and an alternative), and evidence against the null model is summarized via a p-value.

Information criteria

In the 1970s, Akaike changed the game with his famous information criteria, AIC, and the modeling goal substantively changed from one of testing to one of optimization. Instead of model A vs model B, we now had a set of a bunch of models that we could compare at once to find the best.

Computationally-intensive validation

More recently, however, “computationally-intensive” validation opened Pandora’s box. In this era, the data scientist is only limited by their imagination. Any model can be compared against any other model and fed through an optimization pipeline that uses computationally-intensive validation to ensure bad models (that is, models that predict poorly), get sifted out and never see the light of day.

Now, so-called black box modeling approaches, where the desire to understand phenomena is completely defenestrated in conquest of making better predictions, are ubiquitous.

I suppose that’s what Pandora deserved for having an opaque box to begin with.

Now what?

In light of these advancements, “scientific” model sifting via interdisciplinary expertise remains and is increasingly important. Burnham and Anderson suggest that scientists should build a small set of models to clearly and uniquely represent their hypotheses a priori:

“…it seems poor practice to consider all possible models; surely some science can be brought to bear on such an unthinking approach (otherwise, the scientist is superfluous)”

So in the present era (with potentially huge data sets and lots of features), how can we use domain expertise efficiently?

These are important questions I’ve devoted a lot of effort to, but this is not the topic of today’s post.

On black & glass boxes

Black box machine learning (ML) methods are not designed with interpretability in mind. Black box models:

include random forests, ensembles/super learners, neural networks, XGBoost, etc.
can capture nonlinearities and/or high-order interactions well.
may have predictions that are extrinsically explainable. This differs from intrinsic interpretability, however.

The term glass-box models arose to contrast black box models with traditional statistical models.

Generally, the following are considered glass-box:

Regression (linear, logistic, etc.)
Penalized regression (if high-dimensional)
Interpretable decision trees

Regression methods allow us to estimate how an outcome changes with, or is impacted by, a covariate , holding other covariates constant. “Holding confounders constant” is, in my view, a statistical slight-of-hand that is often very poorly understood and mis-applied. At its best though, the language and techniques of regression allow us to get closer to a causal interpretation of . Modern causal inference methods formalize this and can yield additional insights.

In traditional statistical models, we not only describe these model-based associations, we seek inferences about them (often with p-values and CIs).

Note

Confidence intervals and p-values are hard to obtain in black box ML settings.

Not all regression models are glass-box.

If you disagree, please keep reading.

A refined glass-box model definition

Here’s my suggestion for a refined definition of “glass-box model”.

Glass-box model:

A statistical model expressed in terms of a linear combination of a parsimonious set of meaningful parameters with quantified uncertainty. The best glass-box models are transparent, small, quantify parameter uncertainty honestly, and still predict well.

Transparency is reduced as more features are added, especially features that render models difficult to interpret (like interactions), or those involving complex transformations. This definition of transparency resembles that for typical applications of Occam’s Razor in model selection, where the number of parameters in the model translates directly to its simplicity, except that we consider some parameters (coefficients on interactions, for instance) more complex than others (main effects).

Uncertainty quantification is also a key component of this definition. Inferential tools like p-values and CIs are a cornerstone of science, replicability, and transparency. Models without such measures are limited to description, and may therefore have poor generalizability.

Linearity: While the definition contains the word “linear,” it’s language is careful to encompass generalized linear models in addition to linear regression.

Under this definition, transparency (conversely, opacity) is a spectrum:

The most transparent model is the “null” model
Single-predictor models, often used to describe “unadjusted” relationships, might be labeled as the next most transparent.*
On the other end might be large-language models with billions of interconnected parameters.

*A brief aside.

My college friend Tom smoked cigarettes, despite being “pre-med”. I asked him about why he smoked given the health consequences. He said “It’s not bad for me until I’ve smoked 20 pack-years worth! That’s what the literature says!”

In fact, this 20+ pack years is all over the literature and official screening guidelines. Frank Harrell refers to this as dichotomania. It illustrates the appeal of a glass-box model in the most negative possible light, but an appeal nonetheless.

Let’s go through a simple example to illustrate.

Example: HERS Dataset

The data

The Heart and Estrogen/progestin Replacement Study (HERS) was a clinical trial of hormone therapy for prevention of recurrent heart attacks and death among post-menopausal women with existing coronary heart disease.

This data set contains 27 baseline features for n=2571 patients, and complete 1-year follow-up cholesterol data.

Baseline covariates include age, baseline cholesterol, clinical characteristics, treatment, patient-reported health/activity, diabetes status, blood biomarkers, etc.

Let’s take a fresh look at the HERS data to build a glass-box model for HDL cholesterol 1-year post-baseline. We might want to do this for several reasons, for instance, if we want to plan a trial to consider whether a particular intervention will impact cholesterol. We may want to pre-specify our statistical analysis and specify what model will be used to test our hypothesis. Specifically, the question might be posed - what patient factors should our model for cholesterol include as “precision” variables?

This setting is “easy”…

We have , a relatively Gaussian response, a small set of interpretable features. However, we’ll see that even in this “easy” setting, building a glass-box model (under the refined definition) is not trivial.

Describing the data

Here’s our outcome:

library(tidyverse)
library(ggplot2)
library(gtsummary)
library(kableExtra)
library(recipes)
library(selectInferToolkit) #github.com/petersonR/selectInferToolkit

data("hers")

hers %>%
  ggplot(aes(x = hdl1)) +
  geom_histogram() +
  theme_minimal() +
  xlab("HDL (1 year ahead)")

hers %>%
  select(where(is.numeric)) %>%
  select(-hdl1) %>%
  tbl_summary()

Characteristic	N = 2,571¹
age	67 (62, 72)
weight	71 (62, 81)
bmi	27.8 (24.6, 31.7)
waist	91 (82, 100)
whr	0.87 (0.81, 0.92)
glucose	99 (91, 114)
ldl	141 (119, 166)
hdl	49 (41, 57)
tg	157 (116, 208)
sbp	134 (121, 146)
dbp	72 (67, 80)
¹ Median (Q1, Q3)

And our predictors:

hers %>%
  select(where(is.factor)) %>%
  tbl_summary()

Characteristic	N = 2,571¹
ht
placebo	1,303 (51%)
hormone therapy	1,268 (49%)
raceth
White	2,299 (89%)
African American	184 (7.2%)
Other	88 (3.4%)
smoking	328 (13%)
drinkany	1,020 (40%)
exercise	1,012 (39%)
physact
much less active	170 (6.6%)
somewhat less active	459 (18%)
about as active	854 (33%)
somewhat more active	794 (31%)
much more active	294 (11%)
globrat
poor	46 (1.8%)
fair	536 (21%)
good	1,229 (48%)
very good	648 (25%)
excellent	112 (4.4%)
medcond	947 (37%)
htnmeds	2,107 (82%)
statins	951 (37%)
diabetes	662 (26%)
dmpills	246 (9.6%)
insulin	244 (9.5%)
¹ n (%)

Let’s clean it up a bit for modeling. The steps below use the recipes package to standardize the data and create indicators for each of the factor variables.

hers <- hers %>%
  mutate(physact = relevel(physact, "about as active"),
         globrat = relevel(globrat, "good"))

rec_obj <- recipe(hdl1 ~ ., data = hers)

rec_obj <- rec_obj %>%
  step_dummy(all_nominal(), keep_original_cols = FALSE) %>%
  step_center(all_predictors()) %>%
  step_scale(all_numeric()) %>%
  prep()

df <- bake(rec_obj, new_data = hers)

Unadjusted models

We can look at the unadjusted associations via gtsummary’s function below (recall, these are close to the most transparent, interpretable models we have, but they probably don’t predict well).

tbl_uvregression(df, method = "lm", y = hdl1)

Characteristic	N	Beta	95% CI	p-value
age	2,571	0.11	0.08, 0.15	<0.001
weight	2,571	-0.19	-0.23, -0.15	<0.001
bmi	2,571	-0.19	-0.23, -0.15	<0.001
waist	2,571	-0.23	-0.26, -0.19	<0.001
whr	2,571	-0.20	-0.24, -0.16	<0.001
glucose	2,571	-0.15	-0.19, -0.11	<0.001
ldl	2,571	-0.01	-0.05, 0.03	0.5
hdl	2,571	0.73	0.70, 0.76	<0.001
tg	2,571	-0.36	-0.40, -0.33	<0.001
sbp	2,571	0.00	-0.03, 0.04	0.8
dbp	2,571	0.02	-0.01, 0.06	0.2
ht_hormone.therapy	2,571	0.17	0.13, 0.21	<0.001
raceth_African.American	2,571	0.04	0.00, 0.08	0.061
raceth_Other	2,571	-0.04	-0.08, 0.00	0.056
smoking_yes	2,571	-0.04	-0.08, 0.00	0.057
drinkany_yes	2,571	0.14	0.10, 0.18	<0.001
exercise_yes	2,571	0.05	0.02, 0.09	0.006
physact_much.less.active	2,571	-0.05	-0.09, -0.01	0.008
physact_somewhat.less.active	2,571	-0.06	-0.10, -0.02	0.004
physact_somewhat.more.active	2,571	0.07	0.03, 0.11	<0.001
physact_much.more.active	2,571	0.02	-0.02, 0.06	0.2
globrat_poor	2,571	-0.03	-0.07, 0.01	0.2
globrat_fair	2,571	-0.03	-0.07, 0.00	0.079
globrat_very.good	2,571	0.03	0.00, 0.07	0.087
globrat_excellent	2,571	0.06	0.02, 0.09	0.005
medcond_yes	2,571	-0.01	-0.05, 0.03	0.7
htnmeds_yes	2,571	-0.06	-0.10, -0.02	0.004
statins_yes	2,571	0.01	-0.02, 0.05	0.5
diabetes_yes	2,571	-0.16	-0.20, -0.13	<0.001
dmpills_yes	2,571	-0.13	-0.17, -0.09	<0.001
insulin_yes	2,571	-0.08	-0.12, -0.05	<0.001
Abbreviation: CI = Confidence Interval

So… in these unadjusted relationships, nearly everything is significantly associated with HDL 1-year ahead… Are these helpful?

A kitchen sink approach

In settings like this when , and especially when , quite a few statisticians suggest that a “full model approach” is best for inference. We’ll tackle that debate in another post.

For now, let’s check out where this kitchen sink approach (as in, throw all predictors into the model except the kitchen sink) gets us:

fit <- lm(hdl1 ~., data = df)

tbl_regression(fit) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.03	0.01, 0.06	0.018
weight	-0.03	-0.11, 0.04	0.4
bmi	-0.01	-0.08, 0.06	0.8
waist	0.01	-0.08, 0.10	0.9
whr	-0.02	-0.06, 0.03	0.5
glucose	-0.01	-0.05, 0.02	0.4
ldl	0.00	-0.03, 0.03	>0.9
hdl	0.69	0.66, 0.72	<0.001
tg	-0.05	-0.08, -0.03	<0.001
sbp	-0.01	-0.04, 0.02	0.5
dbp	0.02	-0.01, 0.05	0.3
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.01, 0.06	0.016
raceth_Other	-0.03	-0.05, 0.00	0.038
smoking_yes	-0.01	-0.04, 0.02	0.5
drinkany_yes	0.01	-0.01, 0.04	0.3
exercise_yes	0.01	-0.01, 0.04	0.4
physact_much.less.active	-0.01	-0.04, 0.01	0.3
physact_somewhat.less.active	0.01	-0.02, 0.04	0.5
physact_somewhat.more.active	-0.01	-0.04, 0.02	0.6
physact_much.more.active	-0.03	-0.06, 0.00	0.025
globrat_poor	-0.01	-0.03, 0.02	0.7
globrat_fair	-0.01	-0.03, 0.02	0.7
globrat_very.good	0.02	-0.01, 0.04	0.3
globrat_excellent	0.01	-0.02, 0.03	0.6
medcond_yes	0.00	-0.02, 0.03	0.8
htnmeds_yes	-0.03	-0.05, 0.00	0.054
statins_yes	0.01	-0.02, 0.04	0.5
diabetes_yes	0.01	-0.04, 0.05	0.7
dmpills_yes	-0.03	-0.06, 0.01	0.10
insulin_yes	-0.02	-0.06, 0.01	0.2
Abbreviation: CI = Confidence Interval

OK - we have a good starting point now. A few questions arise.

Is this a good model?

The predictive accuracy () is 0.58. Not bad, but actually not great considering a model containing only baseline HDL achieves an of 0.53.

Is this a glass-box model?

Well, we can learn quickly that baseline HDL, triglycerides, treatment group, race and maybe physical activity are significant predictors of HDL 1-year out.

A subquestion is whether these other “insignificant” variables are not important? The answer is NO. In fact, we are missing something big… More on this soon.

Let’s look back at our new glass-box model definition:

A statistical model expressed in terms of a linear combination of a parsimonious set of meaningful parameters with quantified uncertainty.

How does our kitchen sink model stack up?

Linear? ✅
Predicts well? 🤷
Parsimonious? ❌
Meaningful parameters? ❌*
Valid uncertainty: ✅

So, while this checks some of the boxes, I would not consider this kitchen sink model a glass-box model. How can we make this a better glass box?

*The trouble with collinearity

On the face of it, you might think it’s straightforward to interpret each parameter of the kitchen sink model. For instance, holding other variables constant, for every 1 SD increase in age, the expected HDL 1 year after baseline increases by 0.11 SDs. However, for other parameters, it’s not so easy.

BMI (body mass index), WHR (waist to hip ratio), weight, and waist circumference are highly collinear with each other. They all measure adiposity. In the kitchen sink model, it appeared none of these were “significant”. However, the meaning of these parameters in the full model is lacking. “The effect of BMI holding WHR and waist circumference constant” is, in fact, quite ridiculous to conceptualize since these variables relate so highly to each other.

To see this, consider the model below where we remove 3 of the four adiposity variables, so only waist circumference represents adiposity as a “singular flagship”:

fit <- lm(hdl1 ~. - bmi - whr - weight, data = df)

tbl_regression(fit) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.04	0.01, 0.07	0.009
waist	-0.04	-0.07, -0.01	0.009
glucose	-0.02	-0.05, 0.02	0.4
ldl	0.00	-0.03, 0.02	>0.9
hdl	0.69	0.66, 0.72	<0.001
tg	-0.05	-0.08, -0.03	<0.001
sbp	-0.01	-0.04, 0.02	0.5
dbp	0.02	-0.01, 0.05	0.3
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.00, 0.06	0.020
raceth_Other	-0.03	-0.05, 0.00	0.042
smoking_yes	-0.01	-0.03, 0.02	0.6
drinkany_yes	0.01	-0.01, 0.04	0.4
exercise_yes	0.01	-0.01, 0.04	0.4
physact_much.less.active	-0.02	-0.04, 0.01	0.3
physact_somewhat.less.active	0.01	-0.02, 0.04	0.5
physact_somewhat.more.active	-0.01	-0.04, 0.02	0.6
physact_much.more.active	-0.03	-0.06, 0.00	0.026
globrat_poor	-0.01	-0.03, 0.02	0.7
globrat_fair	0.00	-0.03, 0.02	0.7
globrat_very.good	0.02	-0.01, 0.04	0.3
globrat_excellent	0.01	-0.02, 0.03	0.6
medcond_yes	0.00	-0.02, 0.03	0.8
htnmeds_yes	-0.03	-0.05, 0.00	0.054
statins_yes	0.01	-0.02, 0.04	0.5
diabetes_yes	0.01	-0.04, 0.05	0.7
dmpills_yes	-0.03	-0.06, 0.00	0.095
insulin_yes	-0.03	-0.06, 0.01	0.14
Abbreviation: CI = Confidence Interval

We’ve reduced our model dimension and in fact now discover that waist circumference was significant after all! This is the benefit of a small degree of critical thought applied to the kitchen sink model - we’ve gotten one step closer to a good glass-box model.

Selecting a glass-box model…

Let’s say we want to get an even better glass-box model. Lots of stuff in our last model was insignificant; do we really need to keep them all? If we use fewer predictors, our model becomes more parsimonious, and thereby becomes more transparent.

Here’s how we could go about this:

Contextual model refining: This is a great choice and should be the first go to, but it can be hard if there isn’t much context on the features, and in high dimensions. We already did this by looking only at waist as a candidate predictor.
Stepwise selection (select with AIC, BIC, p-values, etc.)
Best-subsets (select with AIC or BIC)
Lasso/penalized regression (Simultaneous selection & estimation with shrinkage toward zero)
Bayesian methods

For now, let’s show the results of a model selected via stepwise selection with AIC:

fit_stepAIC <- MASS::stepAIC(fit, direction = "both", trace = 0)
tbl_regression(fit_stepAIC) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.03	0.01, 0.06	0.015
waist	-0.04	-0.06, -0.01	0.008
hdl	0.69	0.67, 0.72	<0.001
tg	-0.06	-0.08, -0.03	<0.001
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.00, 0.05	0.030
raceth_Other	-0.03	-0.05, 0.00	0.032
physact_much.less.active	-0.02	-0.05, 0.00	0.10
physact_much.more.active	-0.03	-0.05, 0.00	0.034
htnmeds_yes	-0.03	-0.05, 0.00	0.030
dmpills_yes	-0.03	-0.06, -0.01	0.012
insulin_yes	-0.03	-0.06, -0.01	0.015
Abbreviation: CI = Confidence Interval

Whoa! Such low p-values!!!

After this selective process, it seems that we can also claim significance for age, waist, medications, and insulin! And it’s a smaller model so it’s a more “glass-box” approach!

Right?

…Right?

……

Well, no. This is a classic example of an UPSI.

UPSI

A term I’m coining right now that stands for an “Unadjusted Post Selection Inference”. It also is a statistical “oopsie”.

You should know deep down the p-values from the model selected via stepwise AIC are too low. The data set has already been used to select the model, so of course everything that’s selected is more likely to be significant.

We’ll tackle UPSIs in more depth in another post, as well as “better” alternatives like selective inference. In short, inferences post-selection get tricky. Ignoring the variability inherent in the model selection process has been characterized as a common “bad” practice in statistics. UPSIs generally leads to invalid, non-replicable inferences.

Then… why are UPSIs so common?

Well, you tell me why anyone would want their p-values to be lower than they should be… 🙄

Eyerolls and publication bias aside, even well-intentioned statisticians find it difficult to properly adjusting these p-values for the selection process. Some say it’s impossible. Again, we’ll save this for another post.

Is the stepwise AIC model a good glass-box model?

Linear? ✅
Predicts well? ✅
Parsimonious? ✅
Meaningful parameters? ✅ This model has fewer highly collinear features.
Valid uncertainty: ❌Unfortunately, selecting for a more parsimonious model created dishonestly low p-values.

So while this model may predict better, is more parsimonious, and has more meaningful parameters, it’s still not an optimal glass-box model (under our refined definition).

Conclusion

Producing high-quality glass-box models is both necessary and difficult.

Key Takeaways

Regression is not necessarily a glass-box approach
- collinearity can obfuscate meaningful relationships and patterns
- p-values can be easily invalidated by selection
Building an optimal glass-box model is not easy, even for easy scenarios
There is no substitute for domain expertise and critical thinking.

Appendix

R Session Info

Did Denver’s 2022 ‘Zero Fare for Cleaner Air’ campaign actually work?

Ryan Peterson — Fri, 21 Jul 2023 00:00:00 GMT

A data-dictated look at whether Denver’s free August public transit policy had its intended effect on air quality.

Image credit: National Renewable Energy Laboratory, Colorado State University

Backstory

Most summers, Coloradoans flock to the majestic Rocky Mountains with their beautiful hikes and various mountain activities. This is the case, at least, unless poor air quality forces them indoors. For me, this occurred on a smoky July day in 2020, when surreal “snowing” ash sprinkling down from nearby wildfires forced us to evacuate the pickleball courts.

Between wildfires and pollution, Denver’s summer air often leaves room for improvement. Sadly, the Rockies don’t seem so enticing when they are obscured behind a polluted haze.

In August 2022, I noticed my RTD bus was more crowded than usual, and I was not asked to scan my bus pass. This is how I learned of Denver’s 2022 “Zero fare for cleaner air” initiative. Throughout the month, my packed bus led me to believe the policy did work to increase ridership.

My story was validated by the RTD; according to the final RTD report, RTD did indeed see 22% increased ridership during the free-fare month, up 36% from the August prior. This increase led some to conclude that the campaign was a huge success, and also to the expansion of the program in 2023.

But wait… the campaign is called “Zero fare for better air”. So for this to really be a success, the policy change should be measurable in better air quality, not just ridership. To this point, the report concluded that “impacts to air quality are difficult to quantify”. They mention this difficulty is due to no baseline provided. So we’re left wondering – did it work? Did we actually have cleaner air in August of 2022?

Recently, my team investigated how the Covid-19 pandemic affected congestion and air quality in cities across the US (we found that it did). In this post I use similar outcomes and methods to determine the impact of this policy in Denver.

Air Quality Data

There are plenty of important pollutants to worry about in our air, but automobile traffic contributes especially to nitrous oxide (NO2) and ozone (O3). We’ll consider each of these using data from the EPA’s Air Quality System.

Note

Code and data for this and other blog posts are available here.

NO2

Data visualizations

Here are plots of the historical daily data for NO2 in Denver.

Modeling

We can use this historical data to build a forecast of what August’s NO2 levels would be using forecasting methodology available in the fastTS R package that can handle this kind of seasonal data. The series is logged (+10) prior to modeling. We include weekday and month indicator variables and a natural cubic basis spline for time. We computed 30-day-ahead predictions and tested whether these predictions were significantly different than the observed daily values during the zero-fare period. As some months were easier to forecast than others (August was easier to forecast than winter months), we also use heteroskedasticity-corrected standard errors. To evaluate our model, a 10% test set was held out.

Our model can predict daily NO2 on the log scale to within about 0.175 units, with an out-of-sample of 0.533 (about 53% of the variation in this outcome can be explained by historical patterns in our model).

The observed daily NO2 values were on average a factor of 0.932 lower, or 6.8% lower, during the zero-fare month compared to their forecasted values (95% CI: 0.921, 0.944). This is strong evidence of a decrease in daily NO2 during the month of August 2022 than would have been expected historically.

Ozone

Data visualizations

Here are plots of the historical daily data for ozone in Denver.

Modeling

Modeling of ozone data proceeded similarly, although no outcome transformation was used.

Our model can predict daily ozone to within about 0.005 units, with an out-of-sample of 0.67 (about 67% of the variation in this outcome can be explained by historical patterns in our model).

The observed daily ozone values were on average -0.001 parts per million lower during the zero-fare month compared to their forecasted values (95% CI: -0.002, -0.001). This doesn’t show evidence of a change in daily ozone during the month of August 2022 in comparison to would have been expected historically.

Takeaways

Daily average NO2 in Denver during the zero fare month was about 7% less than forecasts (p < 0.001)!
No observable change was seen in ozone relative to forecasts.
There is room for more to be done to improve Denver’s air quality.

Limitations

Ozone and NO2 are affected by many things on a daily basis, which were not controlled for in this analysis. A more effective analysis would control for these things, which might have improved the precision of the model estimates or better account for the possibility of confounding. Both outcomes are also not perfectly measured by the AQS stations scattered about the city of Denver; there’s always the possibility that more accurate or more granular data could better show an effect of the zero-fare policy.

Sensitivity of method

If the method we used for determining the effect of intervention were flawed, we might expect to see high rejection rates for any other subset of 31 days. We can check this by reproducing the same method for 31-day chunks of time surrounding August 2022. Below is a table of the estimated effect under the same methodology for every cut point listed, including FDR-adjusted (and nominal) p-values as well as effect size estimates.

cut	estimate	ci_lb	ci_ub	p.value	p.adj
2021-01-01	1.008	0.96	1.06	0.86	0.95
2021-02-01	0.989	0.95	1.03	0.80	0.95
2021-03-01	1.082	1.04	1.13	0.054	0.21
2021-04-01	0.997	0.97	1.02	0.90	0.95
2021-05-01	1.035	1.01	1.06	0.094	0.29
2021-06-01	1.030	1.01	1.05	0.17	0.37
2021-07-01	0.965	0.95	0.98	0.033	0.16
2021-08-01	0.977	0.96	0.99	0.18	0.37
2021-09-01	0.999	0.97	1.03	0.99	0.99
2021-10-01	1.004	0.97	1.04	0.91	0.95
2021-11-01	0.944	0.91	0.98	0.12	0.32
2021-12-01	0.977	0.94	1.02	0.58	0.89
2022-01-01	1.132	1.09	1.18	0.002	0.019
2022-02-01	1.024	0.98	1.07	0.59	0.89
2022-03-01	0.958	0.93	0.99	0.15	0.36
2022-04-01	0.920	0.89	0.95	0.009	0.068
2022-05-01	0.986	0.96	1.01	0.54	0.89
2022-06-01	1.019	0.99	1.04	0.44	0.81
2022-07-01	0.970	0.95	0.99	0.098	0.29
2022-08-01	0.932	0.92	0.94	< 0.001	< 0.001
2022-09-01	0.989	0.97	1.01	0.66	0.93
2022-10-01	0.990	0.96	1.02	0.70	0.93
2022-11-01	1.008	0.97	1.05	0.85	0.95
2022-12-01	1.103	1.05	1.15	0.032	0.16

On the horizon

Come August 2023, Denver will roll out the program again and I will revisit this analysis to see whether zero fares produce observably cleaner air throughout the month. Please check out their website to sign up and participate.

Note

Again, code and data for this and other posts are available here. This post was updated on 2/15/2024 to point to the fastTS R package, which is an updated version of srlTS, and again on 6/11/2024 based on a bug fix in fastTS 1.0.0, which strengthened the observed effect of NO2.

Appendix

R Session Info

Detecting interactions in R

Ryan Peterson — Tue, 20 Jun 2023 00:00:00 GMT

But what about interactions; are any of those significant?

I have heard some variant of this question from clinicians and researchers from many fields of science. While usually asked in earnest, this question is a dangerous one; the sheer number of interactions can greatly inflate the number of false discoveries in the interactions, resulting in difficult-to-interpret models with many unnecessary interactions. Still, there are times when these expeditions are necessary and fruitful. Thankfully, useful tools are now available to help with the process. This article discusses two regularization-based approaches: Group-Lasso INTERaction-NET (glinternet) and the Sparsity-Ranked Lasso (SRL). The glinternet method implements a hierarchy-preserving selection and estimation procedure, while the SRL is a hierarchy-preferring regularization method which operates under ranked sparsity principles (in short, ranked sparsity methods ensure interactions are treated more skeptically than main effects a priori).

Useful package #1: ranked sparsity methods via sparseR

The sparseR package has been designed to make dealing with interactions and polynomials much more analyst-friendly. Building on the recipes package, sparseR has many built-in tools to facilitate the prepping of a model matrix with interactions and polynomials; these features are presented in the package website located at https://petersonr.github.io/sparseR/. The package is available on CRAN and can be installed and loaded with the code below

install.packages("sparseR")
library(sparseR)

The simplest way to implement the SRL in sparseR is via a single call to the sparseR() function, here demonstrated with Fisher’s iris data set. 10-fold cross-validation is used by default, so we set the seed = 1 here for reproducibility.

data(iris)
srl <- sparseR(Sepal.Width ~ ., data = iris, k = 1, seed = 1)
srl


Model summary @ min CV:
-----------------------------------------------------
  lasso-penalized linear regression with n=150, p=21
  (At lambda=0.0024):
    Nonzero coefficients: 7
    Cross-validation error (deviance): 0.07
    R-squared: 0.63
    Signal-to-noise ratio: 1.71
    Scale estimate (sigma): 0.264

  SR information:
             Vartype Total Selected Saturation Penalty
         Main effect     6        2      0.333    2.45
 Order 1 interaction    12        3      0.250    3.46
  Order 2 polynomial     3        2      0.667    3.00


Model summary @ CV1se:
-----------------------------------------------------
  lasso-penalized linear regression with n=150, p=21
  (At lambda=0.0070):
    Nonzero coefficients: 6
    Cross-validation error (deviance): 0.08
    R-squared: 0.58
    Signal-to-noise ratio: 1.39
    Scale estimate (sigma): 0.281

  SR information:
             Vartype Total Selected Saturation Penalty
         Main effect     6        2      0.333    2.45
 Order 1 interaction    12        2      0.167    3.46
  Order 2 polynomial     3        2      0.667    3.00

The summary function produces additional details:

summary(srl, at = "cv1se")

lasso-penalized linear regression with n=150, p=21
At lambda=0.0070:
-------------------------------------------------
  Nonzero coefficients         :   6
  Expected nonzero coefficients:   0.44
  Average mfdr (6 features)    :   0.074

                               Estimate      z       mfdr Selected
Species_setosa                  0.82889 18.596    < 1e-04        *
Sepal.Length_poly_1             0.19494  9.638    < 1e-04        *
Petal.Width_poly_2              0.10142  4.698 0.00016138        *
Petal.Width:Species_versicolor  0.29190  3.335 0.02568952        *
Sepal.Length:Species_setosa     0.06826  2.769 0.14613161        *
Sepal.Length_poly_2            -0.03215 -2.694 0.27005358        *

We see that two models are displayed by default corresponding to two “smart” choices for the penalization parameter . The first model printed refers to the model where is set to minimize the cross-validated error, while the second one refers to a model where is set to a value such that the model is as sparse as possible while still being within 1 SD of the minimum cross-validated error. Visualizations are also available via sparseR that can help visualize both the solution path and the resulting model (interactions can be very challenging to interpret without a good figure!)

plot(srl)

effect_plot(srl, "Petal.Width", by = "Species", at = "cvmin")

effect_plot(srl, "Petal.Width", by = "Species", at = "cv1se")

Note that while ranked sparsity principles were motivated by the estimation of the lasso (Peterson & Cavanaugh 2022), they can also be implemented with MCP, SCAD, or elastic net and for binary, normal, and survival data. Finally, sparseR includes some functionality to perform forward-stepwise selection using a sparsity-ranked modification of BIC, as well as post-selection inferential techniques using sample splitting and bootstrapping.

Useful package #2: hierarchy-preserving regularization via glinternet

Some argue that when it comes to interactions, hierarchy is very important (i.e., an interaction shouldn’t be included in a model without its constituent main effects). While ranked sparsity methods do prefer hierarchical models, they can often still produce non-hierarchical ones. The glinternet package and the function of the same name uses regularization for model selection under hierarchy constraint, such that all candidate models are hierarchical. Glinternet can handle both continuous and categorical predictors, but requires pre-specification of a numeric model matrix. It can be performed as follows:

# install.packages("glinternet")
library(glinternet)
library(dplyr)

X <- iris %>% 
  select(-Sepal.Width) %>% 
  mutate(Species = as.numeric(Species) - 1)

set.seed(321)
cv_fit <- glinternet.cv(X, Y = iris$Sepal.Width, numLevels = c(1,1,1,3))

The cv_fit object contains necessary information from the cross-validation procedure and the fits themselves stored in a series of lists. A more in-depth tutorial to extract coefficients (and facilitate a model interpretation) using the glinternet package can be found at https://strakaps.github.io/post/glinternet/. Importantly, both the glinternet and sparseR methods have associated predict methods which can yield predictions on new (or the training) data, shown below. For comparison, we also fit a “main effects only” model with sparseR by setting k = 0.

me <- sparseR(Sepal.Width ~ ., data = iris, k = 0, seed = 333)
p_me <- predict(me)
p_srl <- predict(srl)
p_gln <- as.vector(predict(cv_fit, X))

With a little help from the yardstick package’s metrics() function, we can compare the accuracy of each model’s predictions using root-mean-squared error (RMSE), R-squared (RSQ), and mean absolute error (MAE); see table below. Evidently, glinternet and SRL are similar in terms of their predictive performance. However, both outperform the main effects model considerably, suggesting interactions among other variables do have signal worth capturing when predicting Sepal.Width.

gln_res <- tibble(p_gln, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_gln) %>% 
  rename("glinternet"= .estimate) 
srl_res <- tibble(p_srl, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_srl) %>% 
  rename("SRL"= .estimate) 
me_res <- tibble(p_me, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_me) %>% 
  rename("Main effects only"= .estimate) 

results_table <- gln_res %>% 
  bind_cols(srl_res[,3]) %>% 
  bind_cols(me_res[,3]) %>% 
  rename("Metric" = .metric) %>% 
  mutate(Metric = toupper(Metric)) %>% 
  select(-.estimator)

Metric	glinternet	SRL	Main effects only
RMSE	0.24	0.25	0.25
RSQ	0.69	0.68	0.66
MAE	0.19	0.19	0.19

Other packages worth mentioning: ncvreg, hierNet, visreg, sjPlot

The SRL and other sparsity-ranked regularization methods implemented in sparseR would not be possible without the ncvreg package, which performs the heavy-lifting in terms of model fitting, optimization, and cross-validation. The hierNet package is another hierarchy-enforcing procedure that may yield better models than glinternet, however the latter is more computationally efficient especially for situations with a medium-to-large number of covariates. Finally, when interactions or polynomials are included in models, figures are truly worth a thousand words, and packages such as visreg and sjPlot have great functionality for plotting the effects of interactions.

References

Bien J and Tibshirani R (2020). hierNet: A Lasso for Hierarchical Interactions. R package version 1.9. https://CRAN.R-project.org/package=hierNet
Breheny P and Burchett W (2017). Visualization of Regression Models Using visreg. The R Journal, 9: 56-71.
Breheny P and Huang J (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Statist., 5: 232-253.
Kuhn M and Vaughan D (2021). yardstick: Tidy Characterizations of Model Performance. R package version 0.0.8. https://CRAN.R-project.org/package=yardstick
Lim M and Hastie T (2020). glinternet: Learning Interactions via Hierarchical Group-Lasso Regularization. R package version 1.0.11. https://CRAN.R-project.org/package=glinternet
Lüdecke D (2021). sjPlot: Data Visualization for Statistics in Social Science. R package version 2.8.8. https://CRAN.R-project.org/package=sjPlot
Peterson R (2021). sparseR: Variable selection under ranked sparsity principles for interactions and polynomials. https://github.com/petersonR/sparseR/.
Peterson, R, Cavanaugh, J. Ranked sparsity: a cogent regularization framework for selecting and estimating feature interactions and polynomials. AStA Adv Stat Anal 106, 427–454 (2022). https://doi.org/10.1007/s10182-021-00431-7

Note

This post was originally published in the Biometric Bulletin (2021) Volume 38 Issue 3.

Appendix

R Session Info

Welcome to Data Diction

Ryan Peterson — Wed, 01 Jun 2022 00:00:00 GMT

Reviewed by Logan Harris on 2025-11-12

Data Diction

Data: things known or assumed as facts, making the basis of reasoning or calculation
Diction: 1) the choice and use of words and phrases in speech or writing. 2) the choice of words especially with regard to correctness, clearness, or effectiveness.

In addition to the play on “Data Addiction”, Data Diction is also a play on the very commonly used term of “Data dictionary”, a term with which statistical practitioners should be familiar.

Note

Data Diction started in 2022 as a blog. Posts were infrequent. The Glassbox Modeling Working Group (see below) now aims to improve post frequency and participation by multiple authors and reviewers who will collaborate and disseminate quick tutorials, opinion pieces, etc.

Logo

Ryan Peterson created the logo using ozone data from Denver that he compiled for the post about Denver’s 2021 “Zero Fare for Cleaner Air”.

Code

Data Diction

I selected a model. Can I trust my p-values yet?

UPSI Alternatives

Example: a study for treatment of chronic pain

Hackman’s UPSI approach

Selective inference to the rescue

A victory? Or a fluke?

ASA Response to NIH Request for Information: Measuring and Rewarding Scientific Impact

1. Rigor and Reproducibility

1.1 The missing instrument: indicators for analytic replicability

1.2 Candidate pools and model complexity are growing faster than the standards applied

1.3 Adequacy at the design stage: making evidentiary expectations auditable

1.4 Identifying findings ripe for replication

1.5 Learning from discordance: reconciliation and transportability

1.6 A model that already works

2. Data, Software, and Model Sharing

3. Training and Mentorship

4. Collaboration

5. Entrepreneurship and Translation

6. Foundational Scientific Exploration

7. Public Impact

8. Other Area: Generative AI, Agents, & Accountability

9. Other Area: Unintended Consequences and Feasibility

Closing

References

Citation

Can an AI assistant handle the tedious parts of academic writing?

The setup

Task 1: Stop tracking build artifacts

Task 2: Elsevier to MDPI Entropy

Task 3: Debugging the build

Round 1: Missing Ghostscript

Round 2: A sneaky bibliography entry

Round 3: Unused packages

The collaboration pattern

Key takeaways

One more thing

Citation

How can I guarantee a significant result?

Why UPSIs?

Example: a study for treatment of chronic pain

Virtual Trial 1

What’s with the high p-values?

How UPSIs work so well

Virtual Trial 2

This is not a fluke.

Conclusion

Key Takeaways

Future Threads

Related

Appendix

Does debiasing estimates lead to better predictions?

Modeling Goal

Breaking down predictive performance

Penalized Regression

Example: Predicting Leukemia Subtype

The Pitfall of Debiasing

Take aways

Follow-up Questions

References

Citation

What do we mean by glass-box, exactly?

Introduction

What are glass-box models?

Are glass-box models a novelty? Or a necessity?

Why do we build models in the first place?

A brief history of model selection

Hypothesis testing

Information criteria

Computationally-intensive validation

Now what?

On black & glass boxes

Not all regression models are glass-box.

A refined glass-box model definition

Example: HERS Dataset

The data

Describing the data

Unadjusted models

A kitchen sink approach

Selecting a glass-box model…