Data Diction

Can an AI assistant handle the tedious parts of academic writing?

Ryan Peterson — Tue, 28 Apr 2026 00:00:00 GMT

Author Note

This post deviates from our usual AI use policy as an experiment using Claude Code, the result of which will become clear as you read.

What if you could offload the parts of academic writing that have nothing to do with writing? Not the thinking, not the modeling, not the prose — but the LaTeX errors, the git housekeeping, the YAML frontmatter surgery that eats an afternoon every time you switch journals.

I recently put this to the test with Claude Code, Anthropic’s AI coding assistant. Over a single conversation, I used it to clean up a git repo, migrate a manuscript from one journal template to another, and debug the resulting build errors. Here’s how it went.

The setup

I’m working on a paper targeting MDPI’s journal Entropy, but the manuscript (RBIC_Multimodal.Rmd) was still using the rticles::elsevier_article template from an earlier submission plan. The repo also had some generated figure files tracked in git that shouldn’t have been. Routine housekeeping, but the kind that quietly devours time.

Claude Code runs in your terminal (or IDE) and has direct access to your project files, shell, and git. You describe what you want, it proposes a plan, and you approve or redirect. It’s a conversation, not a one-shot prompt.

Task 1: Stop tracking build artifacts

The RBIC_Multimodal_files/ directory — full of generated PDFs from knitr — was being tracked in git. These get regenerated every build, so they just add noise to diffs.

I asked Claude about it, and it laid out the standard three-step fix:

# Add to .gitignore
echo "RBIC_Multimodal_files/" >> .gitignore

# Remove from git's index (but keep the local files!)
git rm -r --cached RBIC_Multimodal_files/

# Commit and push
git commit -m "Stop tracking RBIC_Multimodal_files/"
git push

The key here is the --cached flag — it untracks the files without deleting them from disk. Claude explained this clearly and then, after I confirmed, executed it. Eight PDFs removed from the repo, .gitignore updated, done.

Note

Nothing here is beyond a quick Stack Overflow search. But Claude handled it end-to-end — checking what was tracked, editing .gitignore, running the commands, committing — without me switching contexts.

Task 2: Elsevier to MDPI Entropy

This is where things got more interesting. Switching rticles templates isn’t just changing one line in the YAML. The author/affiliation format is different, the citation engine changes (CSL to natbib), extra metadata fields are required, and you need a Definitions/ folder with the MDPI class files.

I asked Claude to help ensure the proper template was in use. It:

Spawned a sub-agent to research rticles::mdpi_article requirements, YAML fields, and Entropy-specific settings
Read my existing Rmd frontmatter
Rewrote the YAML from scratch

Here’s a simplified before/after:

Before (Elsevier):

output:
  rticles::elsevier_article:
    keep_tex: true
author:
  - name: "Ryan A. Peterson"
    affiliation: a,b
    footnote: "Corresponding Author"
address:
  - code: a
    address: "Department of Biostatistics..."
csl: biometrics.csl

After (MDPI):

output:
  rticles::mdpi_article:
    extra_dependencies: longtable
    keep_tex: true
author:
  - name: Ryan A. Peterson
    affil: "1,2,*"
affiliation:
  - num: 1
    address: |
      Department of Biostatistics...
    email: ryan-peterson@uiowa.edu
journal: entropy
type: article
status: submit

Claude also copied the Definitions/ folder from the installed rticles package, added required back-matter fields (acknowledgement, funding, conflictsofinterest), and removed packages that conflict with mdpi.cls (like endfloat and the custom caption width).

Task 3: Debugging the build

I rendered the document in RStudio and fed the errors back to Claude. Three rounds of fixes followed.

Round 1: Missing Ghostscript

! epstopdf Error: Required program gs not found

The MDPI logos are .eps files, and pdfLaTeX needs Ghostscript to convert them. Claude proposed two options: install Ghostscript, or pre-convert the logos to PDF so collaborators don’t hit the same issue.

I pointed out that option 2 is better for the team:

“It seems like [option 2] is the better option because if others are rendering this document on their machines, they may run into a similar issue.”

Claude agreed, installed Ghostscript via conda (which I already had), converted the three EPS logos to PDF, and then patched mdpi.cls to drop the .eps extensions from \includegraphics calls. Now pdfLaTeX finds the PDFs automatically — no Ghostscript required at build time.

Round 2: A sneaky bibliography entry

The next error looked like a math issue:

! Missing $ inserted.
l.22 ...95/3/10.1093/biomet/asn034/2/asn034.pdf]}}

I told Claude I’d seen this kind of thing before with tables and escape characters. But it traced the actual source to a .bib entry with an eprint field containing a URL-like path full of underscores. Under natbib, those underscores get interpreted as LaTeX subscript operators. The doi and URL fields already covered the same reference, so removing eprint was the clean fix.

Tip

This was the moment that sold me. I had a plausible (but wrong) hypothesis about the error source. Claude didn’t anchor on my suggestion — it searched the .bib file, matched the error text, and found the real cause.

Round 3: Unused packages

Package gensymb Warning: Not defining \perthousand.

I wasn’t sure whether gensymb was actually used anywhere in the paper. Claude searched the entire Rmd for any gensymb commands (\degree, \celsius, \micro, etc.) — found nothing but the \usepackage line itself. Removed it.

The collaboration pattern

What I found most useful wasn’t any single capability — it was the iteration loop:

I describe the goal
Claude proposes a plan
I approve or redirect
Claude executes
I report results (or errors)
Repeat

I stayed in control throughout. Claude asked before running destructive commands. When I redirected (the EPS portability issue), it adapted immediately. When I told it the undefined references were expected (those chunks have eval=FALSE while I re-run an analysis), it moved on without trying to “fix” them.

Key takeaways

Claude Code is a collaborator, not a button. It works best with back-and-forth. The human provides judgment; the AI handles execution and research.
It handles tedious format migrations well. YAML rewriting, class file patching, bibliography fixes — exactly the kind of work that’s straightforward but time-consuming.
It debugs iteratively. Each error got diagnosed and fixed in one round, not blindly retried.
Human oversight matters. I caught the portability issue with EPS conversion. I knew the undefined references were expected. The AI didn’t need to know everything — it just needed to listen when I told it.
It’s git-aware. It reads status, writes descriptive commit messages, and pushes when asked — but only when asked.

One more thing

At the end of our session, I asked Claude to generate a Quarto reveal.js presentation summarizing everything we’d done. It wrote 20 slides with accurate quotes from our conversation, code blocks from the actual commands, a mermaid diagram of the workflow, and custom SCSS theming.

Then I asked it to fix three issues with the first draft. It did.

Then I asked it to write this blog post.

It did that too.

Author Note

It felt incorrect to say that I – Ryan Peterson – authored this post, because the “I” used throughout – Claude generating text through my perspective – is not me. The only text written by me is contained in the two “Author Note” boxes.

We decided to leave the text as it is, without our usual review process, so that it stands as a genuine experiment of what Claude is capable of from a pure writing perspective. This post therefore represents an exception to a key GMWG value to be human first:

We pledge to only use AI as a supporting writing tool

It also demonstrates the importance of such a pledge.

In this and future posts, any AI generated content will be clearly denoted as such with a dotted border. For example:

This text is AI-generated…

…This text is not.

How can I guarantee a significant result?

Perry Hackman, PhD — Mon, 22 Dec 2025 00:00:00 GMT

Dear student,

Before you embark on your “career” as a statistician, you must purge yourself of a childish misconception: that our job is to seek truth. Truth is stubborn, unpredictable, and worst of all, often unpublishable. Scientists crave confidence, the journals crave significance, and we, if we are clever, can provide both without the nuisance of real rigor.

In this post series, I will instruct in the Statistical Dark Arts. Today, I’ll describe how to always ensure a publishable result with model selection and unadjusted post selection inferences (UPSIs).

Why UPSIs?

You will frequently come across scientists with ambitious ideas, saying things like:

I’ve conceived a brilliant new way to treat chronic pain effectively. I don’t want to waste time, efforts, and money on a result that ends up being insignificant… How can I guarantee a significant result??

No statistician wants to contribute to a null finding. Interpreting those is too hard! I’ll let you in on a statistical secret: there IS a way to guarantee statistical significance – and it’s actually extremely common in science.

It’s called unadjusted post-selection inference (UPSI).

With good model selection tools and UPSIs in your toolbelt, you can turn any negative study into a positive one – guaranteed.

Example: a study for treatment of chronic pain

Say the researcher who approached you is planning a cross-over study where patients were given one of two treatments for chronic pain. Patients will record their overall pain each day for 1 week while undergoing treatment A, and 1 week while undergoing treatment B.

Investigators are hoping to determine the difference between treatments in the average pain score. Our outcome is thus a continuous measure for pain reduction: for each subject, this value reflects the difference in that subject’s weekly average pain score between the two candidate treatments.

In addition to determining whether the treatment works on average in their population, investigators wish to determine specific subgroups that might see the most benefit. They are primarily interested in subgroups by age and sex, as well as patient self-reported data such as alcohol consumption and physical activity.

Note

Make sure they have an exhaustive list of candidate effect modifiers; I’d even suggest a few new ones like left/right handed-ness and coffee consumption.

Here’s a way to list all the possible subgroups of this set of variables:

subgroups <- expand.grid(
  age = c("18-35", "36-50", "51-65", "66-80", "80+"),
  sex = c("Male", "Female"),
  hand = c("Left", "Right"), 
  coffee = c("0", "1", "2+"),
  alcohol = c("0", "1", "2", "3+"),
  physical_activity = c("0", "1", "2", "3+")
)
nrow(subgroups)

[1] 960

This is a LOT of subgroups! And it’s way too many subgroups to possibly sift through, at least without tipping off the statistical reviewers about multiplicity.

Multiplicity

Gone, alas, are the carefree days when we could run tons of tests, report the precious few that were significant, and quietly ignore the rest. Some meddlesome truth-seekers eventually caught on and began scolding us for our ingenuity, slapping the sinister label multiplicity onto our beloved golden-egg-laying goose. Now the reputable journals know to look for it. Our approach must become more subtle.

Here’s the key: turn to model selection. Write in the analysis plan:

We will use forward selection to determine which variables or interactions improve our model’s predictions, adding each candidate predictor in a stepwise fashion until the AIC indicates the model’s predictions can no longer be improved.

Sounds rigorous and honest, right?

This simple trick can guarantee significant results, and help you ensure your study will find a statistically significant result by the end, even when none exists!

How about that?! ZERO RISK! (well, to us anyway).

As a bonus, the more interactions we include that are truly null, the more unnecessarily opaque our model becomes – talk about a win-win.

Virtual Trial 1

Let me illustrate via a simulation. Let’s virtually collect patients in our trial.

library(tidyverse)

set.seed(21)
n <- 100        # Sample size  

# Simulate recruitment (y: outcome)
y <- rnorm(n)

# Simulate recruitment (assume each new person has random subgroup)
x_idx <- sample(1:nrow(subgroups), n, replace = TRUE) 
X <- subgroups[x_idx,]
simdata <- tibble(y = y, X)
simdata

# A tibble: 100 × 7
          y age   sex    hand  coffee alcohol physical_activity
                            
 1  0.793   36-50 Female Left  0      2       1                
 2  0.522   80+   Female Left  2+     1       3+               
 3  1.75    36-50 Female Left  2+     1       0                
 4 -1.27    66-80 Female Right 0      3+      0                
 5  2.20    51-65 Male   Right 2+     0       2                
 6  0.433   36-50 Male   Left  0      2       1                
 7 -1.57    66-80 Male   Right 0      2       0                
 8 -0.935   80+   Female Left  1      3+      3+               
 9  0.0635  36-50 Female Left  2+     0       2                
10 -0.00239 66-80 Male   Left  0      2       0                
# ℹ 90 more rows

We need to create interactions to select from:

# Create model matrix w/all interactions
X2 <- model.matrix(y ~ . * ., data = simdata)[,-1]

# Sanity check: make sure these columns represent main effects + interactions
sample(colnames(X2), 10)

 [1] "alcohol2"                     "sexFemale:coffee1"           
 [3] "alcohol1:physical_activity3+" "age66-80:handRight"          
 [5] "alcohol1:physical_activity1"  "age51-65:sexFemale"          
 [7] "handRight:coffee1"            "age66-80:physical_activity2" 
 [9] "age51-65:coffee2+"            "age36-50:coffee1"

# combine into data set for model fitting
simdata2 <- data.frame(y=y, X2)

OK! We have 93 candidate predictors for model selection. Don’t worry, unless our paper’s reviewers are extremely thorough, we don’t have to report this number. Our final model will have many fewer.

Model selection is a simple task. The following code uses forward step-wise selection with AIC to build an optimally-predicting model, per our analysis plan.

library(selectInferToolkit)
fit_aic <- select_stepwise_ic(y ~ ., data = simdata2, direction = "forward")

Caution

While other packages provide p-values after selection, most do so without being transparent, making it difficult to ensure they are truly UPSIs. If you aren’t careful with these less transparent tools, you might report p-values that are somehow diabolically adjusted towards insignificance. In contrast, the selectInferToolkit package, available on GitHub, helps you explicitly use UPSIs for inference:

infer_upsi(fit_aic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	−0.07	0.21	0.315
age36.50	0.17	−0.06	0.41	0.156
age66.80	−0.35	−0.55	−0.15	<0.001
handRight	0.23	−0.02	0.47	0.076
age80..sexFemale	−0.29	−0.51	−0.07	0.013
age36.50.handRight	−0.21	−0.43	0.01	0.071
age80..handRight	−0.22	−0.44	0.00	0.053
age66.80.coffee1	0.45	0.19	0.71	<0.001
age36.50.coffee2.	0.19	−0.01	0.40	0.070
age51.65.coffee2.	0.33	0.16	0.49	<0.001
age36.50.alcohol1	−0.13	−0.31	0.06	0.178
age66.80.alcohol1	−0.26	−0.50	−0.01	0.042
age66.80.alcohol2	−0.19	−0.40	0.02	0.081
age51.65.physical_activity1	−0.33	−0.50	−0.17	<0.001
age51.65.physical_activity2	−0.18	−0.35	−0.01	0.039
sexFemale.alcohol1	0.33	0.15	0.50	<0.001
sexFemale.alcohol2	0.19	0.02	0.37	0.036
sexFemale.alcohol3.	−0.17	−0.37	0.03	0.091
sexFemale.physical_activity2	−0.27	−0.44	−0.10	0.003
handRight.alcohol3.	−0.16	−0.36	0.04	0.123
handRight.physical_activity3.	−0.29	−0.48	−0.10	0.004
coffee2..alcohol3.	0.12	−0.05	0.29	0.172
coffee1.physical_activity1	0.17	0.01	0.33	0.040
alcohol3..physical_activity3.	0.25	0.08	0.42	0.006

And there you have it!

For many subgroups, treatment A worked great (increased pain scores, significant positive effects). Treatment A didn’t work so well for others (with negative significant effects), for whom it actually appears to decrease pain scores. These results indicate we could stand to maximize pain by giving certain patients A, and other patients B. I wouldn’t fret too much about how we included interactions without their constituent main effects; leave that subtlety to the media to figure out.

Caution

Sometimes, researchers want to actually decrease pain scores; you might want to check on this.

What’s with the high p-values?

I know what you’re thinking - won’t it be confusing to include all those results with high p-values?

An expert tip: if you want to make fewer discoveries and get even lower p-values, simply use BIC instead of AIC. BIC only lets the most significant results into the model, and you can say it’s asymptotically consistent, which sounds equally rigorous to AIC, which is asymptotically efficient.

In this example, here’s the model selected via stepwise BIC:

fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                              direction = "forward", 
                              penalty = "BIC")

infer_upsi(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.07	−0.09	0.24	0.393
age36.50	0.27	0.10	0.45	0.003
age66.80	−0.38	−0.57	−0.18	<0.001
age66.80.coffee1	0.23	0.04	0.42	0.019
age51.65.coffee2.	0.29	0.11	0.46	0.002
age51.65.physical_activity1	−0.24	−0.42	−0.07	0.007
sexFemale.physical_activity2	−0.32	−0.49	−0.15	<0.001

Voila – the headlines practically write themselves!

How UPSIs work so well

Let me let you in on a little secret. If you were paying attention, you’d have seen the outcomes in the preceding example, , were generated completely randomly. There was, by design, no true relationship to discover at all. Yet, I was virtually guaranteed to find a significant result. This is due to multiplicity’s clever younger brother, selective inference.

Selective inference is a difficult problem to solve; in fact, some call it impossible. Others hold out hope. At any rate, UPSIs have inertia and incentives on their side.

Later posts to this blog will go into specific solutions to the selective inference problem and how to implement them in R. Fair warning though, these solutions are often less promising and will certainly be less significant.

For now, let me finish convincing you that UPSIs are indeed effective at resolving the “publish or perish” dilemma; with USPIs, we can thrive at both! It should suffice to repeat this simulation again, as though it were performed in a parallel universe.

Virtual Trial 2

simdata2$y <- rnorm(n)
fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                              direction = "forward", 
                              penalty = "BIC")

infer_upsi(fit_bic, data = simdata2) %>% 
  tidy() %>% 
  filter(coef != 0)

term	coef	ci_low	ci_high	p_value
(Intercept)	0.03	−0.14	0.21	0.715
age66.80.sexFemale	−0.25	−0.43	−0.07	0.008
age80..coffee2.	0.30	0.11	0.50	0.003
age80..physical_activity1	−0.26	−0.46	−0.07	0.009

Indeed, we still discover “significant” relationships. And, these are completely different treatment effect modifiers. In this parallel universe, our scientist friends publish completely different effects and will no doubt pour their hard-fought resources into what we well know are wild goose chases.

This is not a fluke.

In case you are still skeptical, let’s repeat this 50 times, as though we ran the study in 50 parallel universes.

sim_results <- list()
for(s in 1:50) { 
  simdata2$y <- rnorm(n)
  fit_bic <- select_stepwise_ic(y ~ ., data = simdata2, 
                                direction = "forward", penalty = "BIC")
  sim_results[[s]] <- infer_upsi(fit_bic, data = simdata2) %>% 
    tidy() %>% 
    filter(coef != 0) 
}

all_selections <- bind_rows(sim_results, .id = "simulation")

main_results <- all_selections %>%
  filter(term != "(Intercept)") %>%
  group_by(simulation) %>%
  summarize(
    any_p_lt_05 = any(p_value < 0.05), 
    n_selections = n()
  )

Using BIC, we found at least one significant effect in 46 of 50 trials, and on average these trials found 2.6 significant effects. We achieved falsely significant results 92% of the time.

If you’re thinking “hmm, my study still might have a chance of failing”, well good news. AIC finds a significant result nearly 100% of the time. We also only considered pairwise interactions between 6 candidate predictors. To minimize the risk of a negative study, one could consider higher level interactions or a greater number of subgroups, and a positive result quickly becomes a sure thing.

As I’ve said, UPSIs have incentives and inertia on their side. The more broad we make this practice, the better we can do at ensuring that the publications keep flowing, regardless of the truth.

Despicably yours,

P. Hackman

Conclusion

Key Takeaways

Unadjusted post-selection inference nearly guarantees false positives
Searching for subgroup effects is a model selection problem
Model selection algorithms don’t make multiplicity issues go away; they make them more subtle and harder to adjust for.

Future Threads

What are some alternatives to UPSI-based inference, and how are they implemented in R?
What is the selectInferToolkit package?

Appendix

R Session Info

Does debiasing estimates lead to better predictions?

Logan Harris — Mon, 15 Dec 2025 00:00:00 GMT

Reviewers

Ryan Peterson (2025-11-24).
Patrick Breheny (2025-12-12).

What connotation do you attach to the word “bias”? A negative one?

In this post we will see why not all bias is bad… at least when it comes to building predictive models. In fact, for many years, statisticians have recognized the benefits of biased estimators in reducing prediction error. Perhaps you knew this, but if not, don’t worry. That is the purpose of this post.

Modeling Goal

When starting off with a data analysis, it is important to outline the primary goals. Whether you realize it or not, in virtually all cases, a primary goal is to use the data to develop a predictive model.

Consider a clinical trial evaluating a new drug for metastatic lung cancer. We say our goal is to “estimate the treatment effect” of our new drug. However, estimating a treatment effect is fundamentally a predictive task. That is, we need to predict what would happen with treatment and what would happen without treatment. The treatment effect is the difference between those two predictions*, the better our predictions… the better our estimate of the treatment effect is.

A pre-dictor is pre-what, exactly?*

When we say a treatment effect is “the difference between two predictions,” that only works if the predictions are made using information that comes before the treatment starts. These are true pre-treatment predictors.

If we accidentally include a model “predictor” that is measured after treatment begins, the treatment could affect it, and the real treatment effect gets obfuscated. The model might appear to “predict” well, but the truth is that it’s not “predicting” at all. Models built this way can give a misleading estimate of the effect of the treatment because part of the effect has been absorbed by the post-treatment variable.

For a valid treatment effect, we must base predictions only on information that could not have been influenced by the treatment – valid pre-dictor variables.

Now, suppose with treatment alone we can predict remission status with 60% accuracy. If you also have patients’ genetic information, what should you do with that?

Including it directly into a model is problematic because the human genome is large*. Unless you have a massive number of patients in your trial (hundreds of thousands) you will NEED to make assumptions about the plausible effects of someone’s genetic information on how well the new treatment works. These assumptions allow us to fit a model with all the genetic information but also intentionally introduce bias to the estimates. This is what a method known as penalized regression does, which we will cover momentarily.

Human Genome*

The human genome is estimated to contain between 30,000 and 40,000 genes!

Now, suppose including genetic information allows us to predict if a patient will be experience remission with 75% accuracy. But this was with biased estimates?! Surely, if we could remove this bias we would get even better predictions, right?

To answer this, we will now break down what makes a set of estimates good at prediction.

Breaking down predictive performance

Bias-variance tradeoff

When we estimate parameters for a model, we are always juggling two competing forces: bias, which measures how far our average estimate is from the truth and variance, which reflects how much our estimates would change if we collect a new sample. A model’s ability to predict an outcome is dependent on both. Increasing or inducing bias can often reduce variance, whereas decreasing bias can often increase variance. Good predictions come from striking a balance between the two. It is not uncommon to be able to intentionally introduce a little bias and be able to drastically reduce variance resulting in better predictive abilities.

It is like comparing one friend who is always 5 minutes late to one that is sometimes 30 minutes early and other times 30 minutes late. While the latter is on average on time, you’d probably describe the one who is always 5 minutes late as more reliable.

If you are someone who likes mathematical details, keep reading. But if the previous example makes sense, feel free to skip to the next section.

For a sample of size , suppose we have a continuous outcome stored in a length vector and features on each sample unit stored in an matrix . We will focus on a linear predictor setting. That is, we want to predict based on a linear combination of the features in . We do this by estimating parameters and then obtaining predictions :

is usually estimated in a way that is based on minimizing , known as the residual sum of squares. However, to assess predictive performance we might consider mean square prediction error (MSPE). MSPE shifts our focus to how well our model is expected to predict a single out-of-sample observation (an observation with predictors that was not in the original sample used to estimate the model).

Assuming that the error structure for is normally distributed (i.e. ) and letting be the error corresponding to and , MSPE can be decomposed as follows:

To make the point easier to see, assume we have just a single predictor (i.e., ) and consider two different candidate estimators for , and . Then, the difference in their MSPEs is:

So now we see the details of the earlier claim that predictive performance is a function of both variance and bias of the estimates.

Let’s work through a toy example. Consider an example where estimator A has a bias of -0.5 and has a variance of 1. This scenario might reflect a penalized estimate for , where the estimator wants to shrink the estimate by a certain amount. Consider an alternative estimator B which corrects the bias of estimator A so that , but this increases its variance to 1.5. Then

The MSPE for the “debiased” estimator is greater than that of the “biased” estimator. Correcting the bias in the estimator had a negative impact on predictive performance.

So, if we care about predictive performance, then bias in our estimates is not necessarily a bad thing.

Penalized Regression

A common rule of thumb is that you need at least 10 observations ( denotes number of observations) per predictor ( denotes number of predictors) in traditional linear regression settings to estimate “stably” using ordinary least squares (OLS), but even more may be required. If but , this rule of thumb would suggest we are in a gray area where estimates for tend to be highly variable and can lead to poor predictions. Of course, if , then it is not possible to use OLS at all.

Adding bias to the estimation process is helpful in both of these scenarios. Enter penalized regression methods.

In general, such methods “penalize” larger estimates of by an amount dictated by a tuning parameter, . The penalty acts like a “complexity tax” forcing the model to stop chasing noise and to start paying attention to the real signal. So, while this introduces bias in the estimates, it also reduces their variance, often leading to superior predictive performance. In fact, this can be true even when is much larger than . In high-dimensional or noisy settings or if there is large amount of correlation between predictors, the gains are often large enough that biased estimators dominate unbiased ones in prediction.

Popular Penalized Regression Methods

The least absolute shrinkage and selection operator (lasso)
Ridge regression
Elastic net (simply a mix of 1 and 2)

It’s easiest to show this with data, so we’ll now turn to an example.

Example: Predicting Leukemia Subtype

To start, we’ll load some libraries and a helpful function.

library(hdrm)
library(ncvreg)
library(hdi)
library(ggplot2)

estimate_intercept <- function(beta, X, y) {
  eta_no_intercept <- drop(X %*% beta)

  f <- function(alpha) {
    p <- plogis(alpha + eta_no_intercept)
    sum(y) - sum(p)
  }

  uniroot(f, interval = c(-1000, 1000))$root
}

Now, consider a data set for predicting leukemia subtype using gene expression data (Golub et al. 1999).

brca1 <- hdrm::read_data("Golub1999")
X <- brca1$X
y <- brca1$y == "ALL"
n <- length(brca1$y)

This dataset has 47 patients with acute lymphoblastic leukemia (abbreviated as ALL) and 25 patients with acute myeloid leukemia (AML). There are 7129 gene expression features, putting us in the high-dimensional realm of .

Note

When , the variance of for an ordinary, unpenalized regression of any type is essentially because the model is not identifiable. It’s like trying to solve a puzzle with more missing pieces than clues; many solutions look “possible,” so you can’t tell which one is the real one.

We split the dataset 50/50 into train and test sets.

set.seed(2806)
idx_train <- sample.int(n, size = floor(n * 0.5))
Xtrain <- X[idx_train,]
ytrain <- y[idx_train]
Xtest <- X[-idx_train,]
ytest <- y[-idx_train]

Then, we will perform a penalized regression method called the “lasso” and select the tuning parameter using cross validation:

set.seed(2806)
cv_fit <- cv.ncvreg(Xtrain, ytrain, penalty = "lasso", family = "binomial")
lambda_min <- cv_fit$lambda.min

Finally we can visualize the predicted probabilities on the testing data to see how well the selected lasso model is able to differentiate between the two disease subtypes.

lasso_res <- data.frame(
  predicted_prob_ALL = predict(cv_fit, Xtest, type = "response"),
  class = factor(ytest, levels = c(0, 1), labels = c("AML", "ALL"))
)

lasso_res$prediction_quality <- ifelse(
  lasso_res$predicted_prob_ALL > .5 & lasso_res$class == "AML" | 
    lasso_res$predicted_prob_ALL < .5 & lasso_res$class == "ALL",
  "Bad", 
  "Good"
)

ggplot(lasso_res, aes(x = class, y = predicted_prob_ALL, color = prediction_quality)) +
  geom_point(alpha = 0.7, position = position_jitter(width = .1)) +
  theme_minimal() +
  scale_color_discrete(name = "Prediction") + 
  ylab("Lasso-based predicted probability ALL") +
  xlab("")

Along with the fact that we reduced the variability of estimation (enough to actually fit a model), introducing bias also helps mitigate over fitting which leads to models that are more generalizable with improved out-of-sample predictions. With this example, we have good (though imperfect) separation between the two subtypes.

This is not the only benefit of introducing bias here. Penalties like the lasso produce sparse fits, with many coefficients set exactly to zero (the second “s” in lasso does literally stand for selection). Weak and noisy predictors are removed, leading to more interpretable glass-box models.

The Pitfall of Debiasing

That being said, it is reasonable to think that reducing the bias of the estimates for would lead to better predictive performance. One such example is the debiased lasso (also known as the desparsified lasso, Zhang and Zhang (2014)) . However, while this method may be good for providing asymptotically unbiased estimators, debiasing can come at the cost of reintroducing a high amount of variance.

As a result, debiasing often leads to noticeably worse predictions. To see this, we will fit the desparsified lasso to our leukemia dataset.

## Takes about 20 minutes; cached 
debiased_fit <- hdi::lasso.proj(Xtrain, ytrain, family = "binomial", lambda = lambda_min)
saveRDS(debiased_fit$bhat, "debiased_fit.rds")

In the code below, we re-estimate the intercept after we obtain the debiased estimates. Then, we can visualize the predictions as we did with the lasso.

debiased_fit <- readRDS("debiased_fit.rds")
debiased_res <- data.frame(
  predicted_prob_ALL = plogis(estimate_intercept(debiased_fit, Xtrain, ytrain) + Xtest %*% debiased_fit),
  class = factor(ytest, levels = c(0, 1), labels = c("AML", "ALL"))
)

debiased_res$prediction_quality <- ifelse(
  debiased_res$predicted_prob_ALL > .5 & debiased_res$class == "AML" | 
    debiased_res$predicted_prob_ALL < .5 & debiased_res$class == "ALL",
  "Bad", 
  "Good"
)

ggplot(debiased_res, aes(x = class, y = predicted_prob_ALL, color = prediction_quality)) +
  geom_point(alpha = 0.7, position = position_jitter(width = .1)) +
  theme_minimal() +
  scale_color_discrete(name = "Prediction") +
  ylab("Debiased-lasso-based predicted probability ALL") +
  xlab("")

Debiasing the original lasso point estimates results in 1 additional subject with AML being misclassified into the ALL group. Additionally, whereas the lasso correctly predicted all subjects with ALL, debiasing leads to 2 incorrect AML predictions. This is a reduction in accuracy from 87.5% to 78.1%!

While you might say you don’t care about prediction, in a future post we will explore why you can’t afford not to.

Take aways

Biased estimation can improve a model’s predictive performance
Reducing bias (via debiasing) often worsens predictions
The lasso and other penalized regression methods can yield better glass-box models

Follow-up Questions

The following questions might be of interest for new posts:

What implications does bias have on inference and model interpretation?
When might debiasing be a good idea?

References

Golub, Todd R., Donna K. Slonim, Pablo Tamayo, et al. 1999. “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring.” Science 286 (5439): 531–37. https://doi.org/10.1126/science.286.5439.531.

Zhang, C. H., and S. S. Zhang. 2014. “Confidence Intervals for Low Dimensional Parameters in High Dimensional Linear Models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76 (1): 217–42.

What do we mean by glass-box, exactly?

Ryan Peterson — Mon, 24 Nov 2025 00:00:00 GMT

Reviewed by Logan Harris on November 21, 2025

Introduction

What are glass-box models?

Glass-box models are transparent, intrinsically interpretable alternatives to their opaque counterparts, black box models. Data scientists typically consider regression-based methods and sparse decision trees as “glass-box”. In this post, I describe the benefits of glass-box methods for modeling data, arguing for the importance of intrinsic interpretability. An intrinsically interpretable model is one whose internal logic is clear enough that a human can see why it makes each prediction, not merely trust a separate explainer.

I hope to convince you that the prevailing definition for glass-box models requires significant refinement in order to be true to its namesake.

Just because a model is a “simple regression” does not mean it is interpretable. In fact, model selection methods that attempt to create more interpretable models often invalidate inference by making p-values unjustifiably low, and confidence intervals (CIs) unrealistically narrow.

Are glass-box models a novelty? Or a necessity?

Consider the useful analogy in the image below, which compares two very different uses of glass as a window into a process. On the left, a penny pressor. I always considered these to be a fun novelty (I still don’t understand how they are legal!). The glass box is part of the appeal - you get to watch a penny turn into a souvenir. Cool!

On the right, you see scientists at the Hanford Site in Washington peering through shield windows while working on the plutonium that would eventually be used in the world’s first atomic bomb explosion. One intact version of these shield windows used during the Manhattan project is currently priced at 10 million dollars, although you can purchase a fragment of it for much less shop.minimuseum.com.

The example of the Hanford Site clearly shows how in high-stakes decisions or systems, opacity is dangerous. Therefore, sometimes, a glass-box model isn’t a novelty like with the penny pressor - it’s a requirement for oversight and correction.

If the AI doomers are right that AI represent existential threats on par with nuclear war… it is easy to see that glass-box models should be (at least in many cases) classified as a necessity, not a novelty.

Why do we build models in the first place?

Let’s start with a quote you have probably heard before from George Box:

All models are wrong, but some are useful.

What makes them useful? Good models help us to:

make good predictions
understand phenomena.

The mix of these two goals are entirely context dependent. In my experience, it is rare for the focus to be entirely on one of these goals.

A brief history of model selection

Hypothesis testing

Starting in the 1700s and through the 1980s, hypothesis testing & p-values were the main way to build models. In the typical set up, two competing models are compared (e.g., a null and an alternative), and evidence against the null model is summarized via a p-value.

Information criteria

In the 1970s, Akaike changed the game with his famous information criteria, AIC, and the modeling goal substantively changed from one of testing to one of optimization. Instead of model A vs model B, we now had a set of a bunch of models that we could compare at once to find the best.

Computationally-intensive validation

More recently, however, “computationally-intensive” validation opened Pandora’s box. In this era, the data scientist is only limited by their imagination. Any model can be compared against any other model and fed through an optimization pipeline that uses computationally-intensive validation to ensure bad models (that is, models that predict poorly), get sifted out and never see the light of day.

Now, so-called black box modeling approaches, where the desire to understand phenomena is completely defenestrated in conquest of making better predictions, are ubiquitous.

I suppose that’s what Pandora deserved for having an opaque box to begin with.

Now what?

In light of these advancements, “scientific” model sifting via interdisciplinary expertise remains and is increasingly important. Burnham and Anderson suggest that scientists should build a small set of models to clearly and uniquely represent their hypotheses a priori:

“…it seems poor practice to consider all possible models; surely some science can be brought to bear on such an unthinking approach (otherwise, the scientist is superfluous)”

So in the present era (with potentially huge data sets and lots of features), how can we use domain expertise efficiently?

These are important questions I’ve devoted a lot of effort to, but this is not the topic of today’s post.

On black & glass boxes

Black box machine learning (ML) methods are not designed with interpretability in mind. Black box models:

include random forests, ensembles/super learners, neural networks, XGBoost, etc.
can capture nonlinearities and/or high-order interactions well.
may have predictions that are extrinsically explainable. This differs from intrinsic interpretability, however.

The term glass-box models arose to contrast black box models with traditional statistical models.

Generally, the following are considered glass-box:

Regression (linear, logistic, etc.)
Penalized regression (if high-dimensional)
Interpretable decision trees

Regression methods allow us to estimate how an outcome changes with, or is impacted by, a covariate , holding other covariates constant. “Holding confounders constant” is, in my view, a statistical slight-of-hand that is often very poorly understood and mis-applied. At its best though, the language and techniques of regression allow us to get closer to a causal interpretation of . Modern causal inference methods formalize this and can yield additional insights.

In traditional statistical models, we not only describe these model-based associations, we seek inferences about them (often with p-values and CIs).

Note

Confidence intervals and p-values are hard to obtain in black box ML settings.

Not all regression models are glass-box.

If you disagree, please keep reading.

A refined glass-box model definition

Here’s my suggestion for a refined definition of “glass-box model”.

Glass-box model:

A statistical model expressed in terms of a linear combination of a parsimonious set of meaningful parameters with quantified uncertainty. The best glass-box models are transparent, small, quantify parameter uncertainty honestly, and still predict well.

Transparency is reduced as more features are added, especially features that render models difficult to interpret (like interactions), or those involving complex transformations. This definition of transparency resembles that for typical applications of Occam’s Razor in model selection, where the number of parameters in the model translates directly to its simplicity, except that we consider some parameters (coefficients on interactions, for instance) more complex than others (main effects).

Uncertainty quantification is also a key component of this definition. Inferential tools like p-values and CIs are a cornerstone of science, replicability, and transparency. Models without such measures are limited to description, and may therefore have poor generalizability.

Linearity: While the definition contains the word “linear,” it’s language is careful to encompass generalized linear models in addition to linear regression.

Under this definition, transparency (conversely, opacity) is a spectrum:

The most transparent model is the “null” model
Single-predictor models, often used to describe “unadjusted” relationships, might be labeled as the next most transparent.*
On the other end might be large-language models with billions of interconnected parameters.

*A brief aside.

My college friend Tom smoked cigarettes, despite being “pre-med”. I asked him about why he smoked given the health consequences. He said “It’s not bad for me until I’ve smoked 20 pack-years worth! That’s what the literature says!”

In fact, this 20+ pack years is all over the literature and official screening guidelines. Frank Harrell refers to this as dichotomania. It illustrates the appeal of a glass-box model in the most negative possible light, but an appeal nonetheless.

Let’s go through a simple example to illustrate.

Example: HERS Dataset

The data

The Heart and Estrogen/progestin Replacement Study (HERS) was a clinical trial of hormone therapy for prevention of recurrent heart attacks and death among post-menopausal women with existing coronary heart disease.

This data set contains 27 baseline features for n=2571 patients, and complete 1-year follow-up cholesterol data.

Baseline covariates include age, baseline cholesterol, clinical characteristics, treatment, patient-reported health/activity, diabetes status, blood biomarkers, etc.

Let’s take a fresh look at the HERS data to build a glass-box model for HDL cholesterol 1-year post-baseline. We might want to do this for several reasons, for instance, if we want to plan a trial to consider whether a particular intervention will impact cholesterol. We may want to pre-specify our statistical analysis and specify what model will be used to test our hypothesis. Specifically, the question might be posed - what patient factors should our model for cholesterol include as “precision” variables?

This setting is “easy”…

We have , a relatively Gaussian response, a small set of interpretable features. However, we’ll see that even in this “easy” setting, building a glass-box model (under the refined definition) is not trivial.

Describing the data

Here’s our outcome:

library(tidyverse)
library(ggplot2)
library(gtsummary)
library(kableExtra)
library(recipes)
library(selectInferToolkit) #github.com/petersonR/selectInferToolkit

data("hers")

hers %>%
  ggplot(aes(x = hdl1)) +
  geom_histogram() +
  theme_minimal() +
  xlab("HDL (1 year ahead)")

hers %>%
  select(where(is.numeric)) %>%
  select(-hdl1) %>%
  tbl_summary()

Characteristic	N = 2,571¹
age	67 (62, 72)
weight	71 (62, 81)
bmi	27.8 (24.6, 31.7)
waist	91 (82, 100)
whr	0.87 (0.81, 0.92)
glucose	99 (91, 114)
ldl	141 (119, 166)
hdl	49 (41, 57)
tg	157 (116, 208)
sbp	134 (121, 146)
dbp	72 (67, 80)
¹ Median (Q1, Q3)

And our predictors:

hers %>%
  select(where(is.factor)) %>%
  tbl_summary()

Characteristic	N = 2,571¹
ht
placebo	1,303 (51%)
hormone therapy	1,268 (49%)
raceth
White	2,299 (89%)
African American	184 (7.2%)
Other	88 (3.4%)
smoking	328 (13%)
drinkany	1,020 (40%)
exercise	1,012 (39%)
physact
much less active	170 (6.6%)
somewhat less active	459 (18%)
about as active	854 (33%)
somewhat more active	794 (31%)
much more active	294 (11%)
globrat
poor	46 (1.8%)
fair	536 (21%)
good	1,229 (48%)
very good	648 (25%)
excellent	112 (4.4%)
medcond	947 (37%)
htnmeds	2,107 (82%)
statins	951 (37%)
diabetes	662 (26%)
dmpills	246 (9.6%)
insulin	244 (9.5%)
¹ n (%)

Let’s clean it up a bit for modeling. The steps below use the recipes package to standardize the data and create indicators for each of the factor variables.

hers <- hers %>%
  mutate(physact = relevel(physact, "about as active"),
         globrat = relevel(globrat, "good"))

rec_obj <- recipe(hdl1 ~ ., data = hers)

rec_obj <- rec_obj %>%
  step_dummy(all_nominal(), keep_original_cols = FALSE) %>%
  step_center(all_predictors()) %>%
  step_scale(all_numeric()) %>%
  prep()

df <- bake(rec_obj, new_data = hers)

Unadjusted models

We can look at the unadjusted associations via gtsummary’s function below (recall, these are close to the most transparent, interpretable models we have, but they probably don’t predict well).

tbl_uvregression(df, method = "lm", y = hdl1)

Characteristic	N	Beta	95% CI	p-value
age	2,571	0.11	0.08, 0.15	<0.001
weight	2,571	-0.19	-0.23, -0.15	<0.001
bmi	2,571	-0.19	-0.23, -0.15	<0.001
waist	2,571	-0.23	-0.26, -0.19	<0.001
whr	2,571	-0.20	-0.24, -0.16	<0.001
glucose	2,571	-0.15	-0.19, -0.11	<0.001
ldl	2,571	-0.01	-0.05, 0.03	0.5
hdl	2,571	0.73	0.70, 0.76	<0.001
tg	2,571	-0.36	-0.40, -0.33	<0.001
sbp	2,571	0.00	-0.03, 0.04	0.8
dbp	2,571	0.02	-0.01, 0.06	0.2
ht_hormone.therapy	2,571	0.17	0.13, 0.21	<0.001
raceth_African.American	2,571	0.04	0.00, 0.08	0.061
raceth_Other	2,571	-0.04	-0.08, 0.00	0.056
smoking_yes	2,571	-0.04	-0.08, 0.00	0.057
drinkany_yes	2,571	0.14	0.10, 0.18	<0.001
exercise_yes	2,571	0.05	0.02, 0.09	0.006
physact_much.less.active	2,571	-0.05	-0.09, -0.01	0.008
physact_somewhat.less.active	2,571	-0.06	-0.10, -0.02	0.004
physact_somewhat.more.active	2,571	0.07	0.03, 0.11	<0.001
physact_much.more.active	2,571	0.02	-0.02, 0.06	0.2
globrat_poor	2,571	-0.03	-0.07, 0.01	0.2
globrat_fair	2,571	-0.03	-0.07, 0.00	0.079
globrat_very.good	2,571	0.03	0.00, 0.07	0.087
globrat_excellent	2,571	0.06	0.02, 0.09	0.005
medcond_yes	2,571	-0.01	-0.05, 0.03	0.7
htnmeds_yes	2,571	-0.06	-0.10, -0.02	0.004
statins_yes	2,571	0.01	-0.02, 0.05	0.5
diabetes_yes	2,571	-0.16	-0.20, -0.13	<0.001
dmpills_yes	2,571	-0.13	-0.17, -0.09	<0.001
insulin_yes	2,571	-0.08	-0.12, -0.05	<0.001
Abbreviation: CI = Confidence Interval

So… in these unadjusted relationships, nearly everything is significantly associated with HDL 1-year ahead… Are these helpful?

A kitchen sink approach

In settings like this when , and especially when , quite a few statisticians suggest that a “full model approach” is best for inference. We’ll tackle that debate in another post.

For now, let’s check out where this kitchen sink approach (as in, throw all predictors into the model except the kitchen sink) gets us:

fit <- lm(hdl1 ~., data = df)

tbl_regression(fit) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.03	0.01, 0.06	0.018
weight	-0.03	-0.11, 0.04	0.4
bmi	-0.01	-0.08, 0.06	0.8
waist	0.01	-0.08, 0.10	0.9
whr	-0.02	-0.06, 0.03	0.5
glucose	-0.01	-0.05, 0.02	0.4
ldl	0.00	-0.03, 0.03	>0.9
hdl	0.69	0.66, 0.72	<0.001
tg	-0.05	-0.08, -0.03	<0.001
sbp	-0.01	-0.04, 0.02	0.5
dbp	0.02	-0.01, 0.05	0.3
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.01, 0.06	0.016
raceth_Other	-0.03	-0.05, 0.00	0.038
smoking_yes	-0.01	-0.04, 0.02	0.5
drinkany_yes	0.01	-0.01, 0.04	0.3
exercise_yes	0.01	-0.01, 0.04	0.4
physact_much.less.active	-0.01	-0.04, 0.01	0.3
physact_somewhat.less.active	0.01	-0.02, 0.04	0.5
physact_somewhat.more.active	-0.01	-0.04, 0.02	0.6
physact_much.more.active	-0.03	-0.06, 0.00	0.025
globrat_poor	-0.01	-0.03, 0.02	0.7
globrat_fair	-0.01	-0.03, 0.02	0.7
globrat_very.good	0.02	-0.01, 0.04	0.3
globrat_excellent	0.01	-0.02, 0.03	0.6
medcond_yes	0.00	-0.02, 0.03	0.8
htnmeds_yes	-0.03	-0.05, 0.00	0.054
statins_yes	0.01	-0.02, 0.04	0.5
diabetes_yes	0.01	-0.04, 0.05	0.7
dmpills_yes	-0.03	-0.06, 0.01	0.10
insulin_yes	-0.02	-0.06, 0.01	0.2
Abbreviation: CI = Confidence Interval

OK - we have a good starting point now. A few questions arise.

Is this a good model?

The predictive accuracy () is 0.58. Not bad, but actually not great considering a model containing only baseline HDL achieves an of 0.53.

Is this a glass-box model?

Well, we can learn quickly that baseline HDL, triglycerides, treatment group, race and maybe physical activity are significant predictors of HDL 1-year out.

A subquestion is whether these other “insignificant” variables are not important? The answer is NO. In fact, we are missing something big… More on this soon.

Let’s look back at our new glass-box model definition:

A statistical model expressed in terms of a linear combination of a parsimonious set of meaningful parameters with quantified uncertainty.

How does our kitchen sink model stack up?

Linear? ✅
Predicts well? 🤷
Parsimonious? ❌
Meaningful parameters? ❌*
Valid uncertainty: ✅

So, while this checks some of the boxes, I would not consider this kitchen sink model a glass-box model. How can we make this a better glass box?

*The trouble with collinearity

On the face of it, you might think it’s straightforward to interpret each parameter of the kitchen sink model. For instance, holding other variables constant, for every 1 SD increase in age, the expected HDL 1 year after baseline increases by 0.11 SDs. However, for other parameters, it’s not so easy.

BMI (body mass index), WHR (waist to hip ratio), weight, and waist circumference are highly collinear with each other. They all measure adiposity. In the kitchen sink model, it appeared none of these were “significant”. However, the meaning of these parameters in the full model is lacking. “The effect of BMI holding WHR and waist circumference constant” is, in fact, quite ridiculous to conceptualize since these variables relate so highly to each other.

To see this, consider the model below where we remove 3 of the four adiposity variables, so only waist circumference represents adiposity as a “singular flagship”:

fit <- lm(hdl1 ~. - bmi - whr - weight, data = df)

tbl_regression(fit) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.04	0.01, 0.07	0.009
waist	-0.04	-0.07, -0.01	0.009
glucose	-0.02	-0.05, 0.02	0.4
ldl	0.00	-0.03, 0.02	>0.9
hdl	0.69	0.66, 0.72	<0.001
tg	-0.05	-0.08, -0.03	<0.001
sbp	-0.01	-0.04, 0.02	0.5
dbp	0.02	-0.01, 0.05	0.3
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.00, 0.06	0.020
raceth_Other	-0.03	-0.05, 0.00	0.042
smoking_yes	-0.01	-0.03, 0.02	0.6
drinkany_yes	0.01	-0.01, 0.04	0.4
exercise_yes	0.01	-0.01, 0.04	0.4
physact_much.less.active	-0.02	-0.04, 0.01	0.3
physact_somewhat.less.active	0.01	-0.02, 0.04	0.5
physact_somewhat.more.active	-0.01	-0.04, 0.02	0.6
physact_much.more.active	-0.03	-0.06, 0.00	0.026
globrat_poor	-0.01	-0.03, 0.02	0.7
globrat_fair	0.00	-0.03, 0.02	0.7
globrat_very.good	0.02	-0.01, 0.04	0.3
globrat_excellent	0.01	-0.02, 0.03	0.6
medcond_yes	0.00	-0.02, 0.03	0.8
htnmeds_yes	-0.03	-0.05, 0.00	0.054
statins_yes	0.01	-0.02, 0.04	0.5
diabetes_yes	0.01	-0.04, 0.05	0.7
dmpills_yes	-0.03	-0.06, 0.00	0.095
insulin_yes	-0.03	-0.06, 0.01	0.14
Abbreviation: CI = Confidence Interval

We’ve reduced our model dimension and in fact now discover that waist circumference was significant after all! This is the benefit of a small degree of critical thought applied to the kitchen sink model - we’ve gotten one step closer to a good glass-box model.

Selecting a glass-box model…

Let’s say we want to get an even better glass-box model. Lots of stuff in our last model was insignificant; do we really need to keep them all? If we use fewer predictors, our model becomes more parsimonious, and thereby becomes more transparent.

Here’s how we could go about this:

Contextual model refining: This is a great choice and should be the first go to, but it can be hard if there isn’t much context on the features, and in high dimensions. We already did this by looking only at waist as a candidate predictor.
Stepwise selection (select with AIC, BIC, p-values, etc.)
Best-subsets (select with AIC or BIC)
Lasso/penalized regression (Simultaneous selection & estimation with shrinkage toward zero)
Bayesian methods

For now, let’s show the results of a model selected via stepwise selection with AIC:

fit_stepAIC <- MASS::stepAIC(fit, direction = "both", trace = 0)
tbl_regression(fit_stepAIC) %>%
  bold_p()

Characteristic	Beta	95% CI	p-value
age	0.03	0.01, 0.06	0.015
waist	-0.04	-0.06, -0.01	0.008
hdl	0.69	0.67, 0.72	<0.001
tg	-0.06	-0.08, -0.03	<0.001
ht_hormone.therapy	0.19	0.16, 0.21	<0.001
raceth_African.American	0.03	0.00, 0.05	0.030
raceth_Other	-0.03	-0.05, 0.00	0.032
physact_much.less.active	-0.02	-0.05, 0.00	0.10
physact_much.more.active	-0.03	-0.05, 0.00	0.034
htnmeds_yes	-0.03	-0.05, 0.00	0.030
dmpills_yes	-0.03	-0.06, -0.01	0.012
insulin_yes	-0.03	-0.06, -0.01	0.015
Abbreviation: CI = Confidence Interval

Whoa! Such low p-values!!!

After this selective process, it seems that we can also claim significance for age, waist, medications, and insulin! And it’s a smaller model so it’s a more “glass-box” approach!

Right?

…Right?

……

Well, no. This is a classic example of an UPSI.

UPSI

A term I’m coining right now that stands for an “Unadjusted Post Selection Inference”. It also is a statistical “oopsie”.

You should know deep down the p-values from the model selected via stepwise AIC are too low. The data set has already been used to select the model, so of course everything that’s selected is more likely to be significant.

We’ll tackle UPSIs in more depth in another post, as well as “better” alternatives like selective inference. In short, inferences post-selection get tricky. Ignoring the variability inherent in the model selection process has been characterized as a common “bad” practice in statistics. UPSIs generally leads to invalid, non-replicable inferences.

Then… why are UPSIs so common?

Well, you tell me why anyone would want their p-values to be lower than they should be… 🙄

Eyerolls and publication bias aside, even well-intentioned statisticians find it difficult to properly adjusting these p-values for the selection process. Some say it’s impossible. Again, we’ll save this for another post.

Is the stepwise AIC model a good glass-box model?

Linear? ✅
Predicts well? ✅
Parsimonious? ✅
Meaningful parameters? ✅ This model has fewer highly collinear features.
Valid uncertainty: ❌Unfortunately, selecting for a more parsimonious model created dishonestly low p-values.

So while this model may predict better, is more parsimonious, and has more meaningful parameters, it’s still not an optimal glass-box model (under our refined definition).

Conclusion

Producing high-quality glass-box models is both necessary and difficult.

Key Takeaways

Regression is not necessarily a glass-box approach
- collinearity can obfuscate meaningful relationships and patterns
- p-values can be easily invalidated by selection
Building an optimal glass-box model is not easy, even for easy scenarios
There is no substitute for domain expertise and critical thinking.

Appendix

R Session Info

Did Denver’s 2022 ‘Zero Fare for Cleaner Air’ campaign actually work?

Ryan Peterson — Fri, 21 Jul 2023 00:00:00 GMT

A data-dictated look at whether Denver’s free August public transit policy had its intended effect on air quality.

Image credit: National Renewable Energy Laboratory, Colorado State University

Backstory

Most summers, Coloradoans flock to the majestic Rocky Mountains with their beautiful hikes and various mountain activities. This is the case, at least, unless poor air quality forces them indoors. For me, this occurred on a smoky July day in 2020, when surreal “snowing” ash sprinkling down from nearby wildfires forced us to evacuate the pickleball courts.

Between wildfires and pollution, Denver’s summer air often leaves room for improvement. Sadly, the Rockies don’t seem so enticing when they are obscured behind a polluted haze.

In August 2022, I noticed my RTD bus was more crowded than usual, and I was not asked to scan my bus pass. This is how I learned of Denver’s 2022 “Zero fare for cleaner air” initiative. Throughout the month, my packed bus led me to believe the policy did work to increase ridership.

My story was validated by the RTD; according to the final RTD report, RTD did indeed see 22% increased ridership during the free-fare month, up 36% from the August prior. This increase led some to conclude that the campaign was a huge success, and also to the expansion of the program in 2023.

But wait… the campaign is called “Zero fare for better air”. So for this to really be a success, the policy change should be measurable in better air quality, not just ridership. To this point, the report concluded that “impacts to air quality are difficult to quantify”. They mention this difficulty is due to no baseline provided. So we’re left wondering – did it work? Did we actually have cleaner air in August of 2022?

Recently, my team investigated how the Covid-19 pandemic affected congestion and air quality in cities across the US (we found that it did). In this post I use similar outcomes and methods to determine the impact of this policy in Denver.

Air Quality Data

There are plenty of important pollutants to worry about in our air, but automobile traffic contributes especially to nitrous oxide (NO2) and ozone (O3). We’ll consider each of these using data from the EPA’s Air Quality System.

Note

Code and data for this and other blog posts are available here.

NO2

Data visualizations

Here are plots of the historical daily data for NO2 in Denver.

Modeling

We can use this historical data to build a forecast of what August’s NO2 levels would be using forecasting methodology available in the fastTS R package that can handle this kind of seasonal data. The series is logged (+10) prior to modeling. We include weekday and month indicator variables and a natural cubic basis spline for time. We computed 30-day-ahead predictions and tested whether these predictions were significantly different than the observed daily values during the zero-fare period. As some months were easier to forecast than others (August was easier to forecast than winter months), we also use heteroskedasticity-corrected standard errors. To evaluate our model, a 10% test set was held out.

Our model can predict daily NO2 on the log scale to within about 0.175 units, with an out-of-sample of 0.533 (about 53% of the variation in this outcome can be explained by historical patterns in our model).

The observed daily NO2 values were on average a factor of 0.932 lower, or 6.8% lower, during the zero-fare month compared to their forecasted values (95% CI: 0.921, 0.944). This is strong evidence of a decrease in daily NO2 during the month of August 2022 than would have been expected historically.

Ozone

Data visualizations

Here are plots of the historical daily data for ozone in Denver.

Modeling

Modeling of ozone data proceeded similarly, although no outcome transformation was used.

Our model can predict daily ozone to within about 0.005 units, with an out-of-sample of 0.67 (about 67% of the variation in this outcome can be explained by historical patterns in our model).

The observed daily ozone values were on average -0.001 parts per million lower during the zero-fare month compared to their forecasted values (95% CI: -0.002, -0.001). This doesn’t show evidence of a change in daily ozone during the month of August 2022 in comparison to would have been expected historically.

Takeaways

Daily average NO2 in Denver during the zero fare month was about 7% less than forecasts (p < 0.001)!
No observable change was seen in ozone relative to forecasts.
There is room for more to be done to improve Denver’s air quality.

Limitations

Ozone and NO2 are affected by many things on a daily basis, which were not controlled for in this analysis. A more effective analysis would control for these things, which might have improved the precision of the model estimates or better account for the possibility of confounding. Both outcomes are also not perfectly measured by the AQS stations scattered about the city of Denver; there’s always the possibility that more accurate or more granular data could better show an effect of the zero-fare policy.

Sensitivity of method

If the method we used for determining the effect of intervention were flawed, we might expect to see high rejection rates for any other subset of 31 days. We can check this by reproducing the same method for 31-day chunks of time surrounding August 2022. Below is a table of the estimated effect under the same methodology for every cut point listed, including FDR-adjusted (and nominal) p-values as well as effect size estimates.

cut	estimate	ci_lb	ci_ub	p.value	p.adj
2021-01-01	1.008	0.96	1.06	0.86	0.95
2021-02-01	0.989	0.95	1.03	0.80	0.95
2021-03-01	1.082	1.04	1.13	0.054	0.21
2021-04-01	0.997	0.97	1.02	0.90	0.95
2021-05-01	1.035	1.01	1.06	0.094	0.29
2021-06-01	1.030	1.01	1.05	0.17	0.37
2021-07-01	0.965	0.95	0.98	0.033	0.16
2021-08-01	0.977	0.96	0.99	0.18	0.37
2021-09-01	0.999	0.97	1.03	0.99	0.99
2021-10-01	1.004	0.97	1.04	0.91	0.95
2021-11-01	0.944	0.91	0.98	0.12	0.32
2021-12-01	0.977	0.94	1.02	0.58	0.89
2022-01-01	1.132	1.09	1.18	0.002	0.019
2022-02-01	1.024	0.98	1.07	0.59	0.89
2022-03-01	0.958	0.93	0.99	0.15	0.36
2022-04-01	0.920	0.89	0.95	0.009	0.068
2022-05-01	0.986	0.96	1.01	0.54	0.89
2022-06-01	1.019	0.99	1.04	0.44	0.81
2022-07-01	0.970	0.95	0.99	0.098	0.29
2022-08-01	0.932	0.92	0.94	< 0.001	< 0.001
2022-09-01	0.989	0.97	1.01	0.66	0.93
2022-10-01	0.990	0.96	1.02	0.70	0.93
2022-11-01	1.008	0.97	1.05	0.85	0.95
2022-12-01	1.103	1.05	1.15	0.032	0.16

On the horizon

Come August 2023, Denver will roll out the program again and I will revisit this analysis to see whether zero fares produce observably cleaner air throughout the month. Please check out their website to sign up and participate.

Note

Again, code and data for this and other posts are available here. This post was updated on 2/15/2024 to point to the fastTS R package, which is an updated version of srlTS, and again on 6/11/2024 based on a bug fix in fastTS 1.0.0, which strengthened the observed effect of NO2.

Appendix

R Session Info

Detecting interactions in R

Ryan Peterson — Tue, 20 Jun 2023 00:00:00 GMT

But what about interactions; are any of those significant?

I have heard some variant of this question from clinicians and researchers from many fields of science. While usually asked in earnest, this question is a dangerous one; the sheer number of interactions can greatly inflate the number of false discoveries in the interactions, resulting in difficult-to-interpret models with many unnecessary interactions. Still, there are times when these expeditions are necessary and fruitful. Thankfully, useful tools are now available to help with the process. This article discusses two regularization-based approaches: Group-Lasso INTERaction-NET (glinternet) and the Sparsity-Ranked Lasso (SRL). The glinternet method implements a hierarchy-preserving selection and estimation procedure, while the SRL is a hierarchy-preferring regularization method which operates under ranked sparsity principles (in short, ranked sparsity methods ensure interactions are treated more skeptically than main effects a priori).

Useful package #1: ranked sparsity methods via sparseR

The sparseR package has been designed to make dealing with interactions and polynomials much more analyst-friendly. Building on the recipes package, sparseR has many built-in tools to facilitate the prepping of a model matrix with interactions and polynomials; these features are presented in the package website located at https://petersonr.github.io/sparseR/. The package is available on CRAN and can be installed and loaded with the code below

install.packages("sparseR")
library(sparseR)

The simplest way to implement the SRL in sparseR is via a single call to the sparseR() function, here demonstrated with Fisher’s iris data set. 10-fold cross-validation is used by default, so we set the seed = 1 here for reproducibility.

data(iris)
srl <- sparseR(Sepal.Width ~ ., data = iris, k = 1, seed = 1)
srl


Model summary @ min CV:
-----------------------------------------------------
  lasso-penalized linear regression with n=150, p=21
  (At lambda=0.0019):
    Nonzero coefficients: 8
    Cross-validation error (deviance): 0.07
    R-squared: 0.63
    Signal-to-noise ratio: 1.71
    Scale estimate (sigma): 0.264

  SR information:
             Vartype Total Selected Saturation Penalty
         Main effect     6        3      0.500    2.45
 Order 1 interaction    12        3      0.250    3.46
  Order 2 polynomial     3        2      0.667    3.00


Model summary @ CV1se:
-----------------------------------------------------
  lasso-penalized linear regression with n=150, p=21
  (At lambda=0.0070):
    Nonzero coefficients: 6
    Cross-validation error (deviance): 0.08
    R-squared: 0.58
    Signal-to-noise ratio: 1.39
    Scale estimate (sigma): 0.281

  SR information:
             Vartype Total Selected Saturation Penalty
         Main effect     6        2      0.333    2.45
 Order 1 interaction    12        2      0.167    3.46
  Order 2 polynomial     3        2      0.667    3.00

The summary function produces additional details:

summary(srl, at = "cv1se")

lasso-penalized linear regression with n=150, p=21
At lambda=0.0070:
-------------------------------------------------
  Nonzero coefficients         :   6
  Expected nonzero coefficients:   0.44
  Average mfdr (6 features)    :   0.074

                               Estimate      z       mfdr Selected
Species_setosa                  0.82889 18.596    < 1e-04        *
Sepal.Length_poly_1             0.19494  9.638    < 1e-04        *
Petal.Width_poly_2              0.10142  4.698 0.00016138        *
Petal.Width:Species_versicolor  0.29190  3.335 0.02568952        *
Sepal.Length:Species_setosa     0.06826  2.769 0.14613161        *
Sepal.Length_poly_2            -0.03215 -2.694 0.27005358        *

We see that two models are displayed by default corresponding to two “smart” choices for the penalization parameter . The first model printed refers to the model where is set to minimize the cross-validated error, while the second one refers to a model where is set to a value such that the model is as sparse as possible while still being within 1 SD of the minimum cross-validated error. Visualizations are also available via sparseR that can help visualize both the solution path and the resulting model (interactions can be very challenging to interpret without a good figure!)

plot(srl)

effect_plot(srl, "Petal.Width", by = "Species", at = "cvmin")

effect_plot(srl, "Petal.Width", by = "Species", at = "cv1se")

Note that while ranked sparsity principles were motivated by the estimation of the lasso (Peterson & Cavanaugh 2022), they can also be implemented with MCP, SCAD, or elastic net and for binary, normal, and survival data. Finally, sparseR includes some functionality to perform forward-stepwise selection using a sparsity-ranked modification of BIC, as well as post-selection inferential techniques using sample splitting and bootstrapping.

Useful package #2: hierarchy-preserving regularization via glinternet

Some argue that when it comes to interactions, hierarchy is very important (i.e., an interaction shouldn’t be included in a model without its constituent main effects). While ranked sparsity methods do prefer hierarchical models, they can often still produce non-hierarchical ones. The glinternet package and the function of the same name uses regularization for model selection under hierarchy constraint, such that all candidate models are hierarchical. Glinternet can handle both continuous and categorical predictors, but requires pre-specification of a numeric model matrix. It can be performed as follows:

# install.packages("glinternet")
library(glinternet)
library(dplyr)

X <- iris %>% 
  select(-Sepal.Width) %>% 
  mutate(Species = as.numeric(Species) - 1)

set.seed(321)
cv_fit <- glinternet.cv(X, Y = iris$Sepal.Width, numLevels = c(1,1,1,3))

The cv_fit object contains necessary information from the cross-validation procedure and the fits themselves stored in a series of lists. A more in-depth tutorial to extract coefficients (and facilitate a model interpretation) using the glinternet package can be found at https://strakaps.github.io/post/glinternet/. Importantly, both the glinternet and sparseR methods have associated predict methods which can yield predictions on new (or the training) data, shown below. For comparison, we also fit a “main effects only” model with sparseR by setting k = 0.

me <- sparseR(Sepal.Width ~ ., data = iris, k = 0, seed = 333)
p_me <- predict(me)
p_srl <- predict(srl)
p_gln <- as.vector(predict(cv_fit, X))

With a little help from the yardstick package’s metrics() function, we can compare the accuracy of each model’s predictions using root-mean-squared error (RMSE), R-squared (RSQ), and mean absolute error (MAE); see table below. Evidently, glinternet and SRL are similar in terms of their predictive performance. However, both outperform the main effects model considerably, suggesting interactions among other variables do have signal worth capturing when predicting Sepal.Width.

gln_res <- tibble(p_gln, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_gln) %>% 
  rename("glinternet"= .estimate) 
srl_res <- tibble(p_srl, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_srl) %>% 
  rename("SRL"= .estimate) 
me_res <- tibble(p_me, y = iris$Sepal.Width) %>% 
  yardstick::metrics(y, p_me) %>% 
  rename("Main effects only"= .estimate) 

results_table <- gln_res %>% 
  bind_cols(srl_res[,3]) %>% 
  bind_cols(me_res[,3]) %>% 
  rename("Metric" = .metric) %>% 
  mutate(Metric = toupper(Metric)) %>% 
  select(-.estimator)

Metric	glinternet	SRL	Main effects only
RMSE	0.24	0.25	0.25
RSQ	0.69	0.68	0.66
MAE	0.19	0.19	0.19

Other packages worth mentioning: ncvreg, hierNet, visreg, sjPlot

The SRL and other sparsity-ranked regularization methods implemented in sparseR would not be possible without the ncvreg package, which performs the heavy-lifting in terms of model fitting, optimization, and cross-validation. The hierNet package is another hierarchy-enforcing procedure that may yield better models than glinternet, however the latter is more computationally efficient especially for situations with a medium-to-large number of covariates. Finally, when interactions or polynomials are included in models, figures are truly worth a thousand words, and packages such as visreg and sjPlot have great functionality for plotting the effects of interactions.

References

Bien J and Tibshirani R (2020). hierNet: A Lasso for Hierarchical Interactions. R package version 1.9. https://CRAN.R-project.org/package=hierNet
Breheny P and Burchett W (2017). Visualization of Regression Models Using visreg. The R Journal, 9: 56-71.
Breheny P and Huang J (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Statist., 5: 232-253.
Kuhn M and Vaughan D (2021). yardstick: Tidy Characterizations of Model Performance. R package version 0.0.8. https://CRAN.R-project.org/package=yardstick
Lim M and Hastie T (2020). glinternet: Learning Interactions via Hierarchical Group-Lasso Regularization. R package version 1.0.11. https://CRAN.R-project.org/package=glinternet
Lüdecke D (2021). sjPlot: Data Visualization for Statistics in Social Science. R package version 2.8.8. https://CRAN.R-project.org/package=sjPlot
Peterson R (2021). sparseR: Variable selection under ranked sparsity principles for interactions and polynomials. https://github.com/petersonR/sparseR/.
Peterson, R, Cavanaugh, J. Ranked sparsity: a cogent regularization framework for selecting and estimating feature interactions and polynomials. AStA Adv Stat Anal 106, 427–454 (2022). https://doi.org/10.1007/s10182-021-00431-7

Note

This post was originally published in the Biometric Bulletin (2021) Volume 38 Issue 3.

Appendix

R Session Info

Welcome to Data Diction

Ryan Peterson — Wed, 01 Jun 2022 00:00:00 GMT

Reviewed by Logan Harris on 2025-11-12

Data Diction

Data: things known or assumed as facts, making the basis of reasoning or calculation
Diction: 1) the choice and use of words and phrases in speech or writing. 2) the choice of words especially with regard to correctness, clearness, or effectiveness.

In addition to the play on “Data Addiction”, Data Diction is also a play on the very commonly used term of “Data dictionary”, a term with which statistical practitioners should be familiar.

Note

Data Diction started in 2022 as a blog. Posts were infrequent. The Glassbox Modeling Working Group (see below) now aims to improve post frequency and participation by multiple authors and reviewers who will collaborate and disseminate quick tutorials, opinion pieces, etc.

Logo

Ryan Peterson created the logo using ozone data from Denver that he compiled for the post about Denver’s 2021 “Zero Fare for Cleaner Air”.

Code

Data Diction

Can an AI assistant handle the tedious parts of academic writing?

The setup

Task 1: Stop tracking build artifacts

Task 2: Elsevier to MDPI Entropy

Task 3: Debugging the build

Round 1: Missing Ghostscript

Round 2: A sneaky bibliography entry

Round 3: Unused packages

The collaboration pattern

Key takeaways

One more thing

How can I guarantee a significant result?

Why UPSIs?

Example: a study for treatment of chronic pain

Virtual Trial 1

What’s with the high p-values?

How UPSIs work so well

Virtual Trial 2

This is not a fluke.

Conclusion

Key Takeaways

Future Threads

Related

Appendix

Does debiasing estimates lead to better predictions?

Modeling Goal

Breaking down predictive performance

Penalized Regression

Example: Predicting Leukemia Subtype

The Pitfall of Debiasing

Take aways

Follow-up Questions

References

What do we mean by glass-box, exactly?

Introduction

What are glass-box models?

Are glass-box models a novelty? Or a necessity?

Why do we build models in the first place?

A brief history of model selection

Hypothesis testing

Information criteria

Computationally-intensive validation

Now what?

On black & glass boxes

Not all regression models are glass-box.

A refined glass-box model definition

Example: HERS Dataset

The data

Describing the data

Unadjusted models

A kitchen sink approach

Selecting a glass-box model…

Conclusion

Key Takeaways

Future Threads / Related Questions

Appendix

Did Denver’s 2022 ‘Zero Fare for Cleaner Air’ campaign actually work?

Backstory

Air Quality Data

NO2

Data visualizations

Modeling

Ozone

Data visualizations

Modeling

Takeaways

Limitations

Sensitivity of method

On the horizon

Appendix

Detecting interactions in R

Useful package #1: ranked sparsity methods via sparseR

Useful package #2: hierarchy-preserving regularization via glinternet

Other packages worth mentioning: ncvreg, hierNet, visreg, sjPlot

References

Appendix

Welcome to Data Diction

Data Diction

Logo