Skip to content

Commit dd1cdb9

Browse files
authored
Add sup-t option for p-value-adjustment (#528)
* Add `sup-t` option for p-value-adjustment * add test * typo * fix * add tests * fix * fix * fix test * wordlist * fix * fix * fix, add test * docs
1 parent ad55d9e commit dd1cdb9

15 files changed

+218
-34
lines changed

DESCRIPTION

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Type: Package
22
Package: modelbased
33
Title: Estimation of Model-Based Predictions, Contrasts and Means
4-
Version: 0.11.2.13
4+
Version: 0.11.2.14
55
Authors@R:
66
c(person(given = "Dominique",
77
family = "Makowski",
@@ -78,6 +78,7 @@ Suggests:
7878
marginaleffects (>= 0.26.0),
7979
mice,
8080
mgcv,
81+
mvtnorm,
8182
nanoparquet,
8283
nnet,
8384
ordinal,

NEWS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@
2626
* New vignettes (Case Studies) about using *modelbased* with finite mixture models
2727
and interrupted time series analysis.
2828

29+
* The `p_adjust` argument gets a new option, `"sup-t"`, to calculate
30+
simultaneous confidence intervals.
31+
2932
## Bug fixes
3033

3134
* Fixed printing and plotting for models from packages *nnet* and *brglm2*.

R/estimate_contrasts.R

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,12 @@
99
#' levels at which contrasts are evaluated (e.g., `contrast="x=c('a','b')"`).
1010
#' @param p_adjust The p-values adjustment method for frequentist multiple
1111
#' comparisons. Can be one of `"none"` (default), `"hochberg"`, `"hommel"`,
12-
#' `"bonferroni"`, `"BH"`, `"BY"`, `"fdr"`, `"tukey"`, `"sidak"`, `"esarey"` or
13-
#' `"holm"`. The `"esarey"` option is specifically for the case of Johnson-Neyman
14-
#' intervals, i.e. when calling `estimate_slopes()` with two numeric predictors
15-
#' in an interaction term. Details for the other options can be found in the
12+
#' `"bonferroni"`, `"BH"`, `"BY"`, `"fdr"`, `"tukey"`, `"sidak"`, `"sup-t"`,
13+
#' `"esarey"` or `"holm"`. The `"esarey"` option is specifically for the case of
14+
#' Johnson-Neyman intervals, i.e. when calling `estimate_slopes()` with two
15+
#' numeric predictors in an interaction term. `"sup-t"` computes simultaneous
16+
#' confidence bands, also called sup-t confidence band (Montiel Olea &
17+
#' Plagborg-Møller, 2019). Details for the other options can be found in the
1618
#' p-value adjustment section of the `emmeans::test` documentation or
1719
#' `?stats::p.adjust`. Note that certain options provided by the **emmeans**
1820
#' package are only available if you set `backend = "emmeans"`.
@@ -153,8 +155,13 @@
153155
#' 200) through [bootES::bootES]. Adjusts for contrasts, but not for covariates.
154156
#'
155157
#' @references
156-
#' Mize, T., & Han, B. (2025). Inequality and Total Effect Summary Measures for
157-
#' Nominal and Ordinal Variables. Sociological Science, 12, 115–157. \doi{10.15195/v12.a7}
158+
#' - Mize, T., & Han, B. (2025). Inequality and Total Effect Summary Measures for
159+
#' Nominal and Ordinal Variables. Sociological Science, 12, 115–157.
160+
#' \doi{10.15195/v12.a7}
161+
#'
162+
#' - Montiel Olea, J. L., and Plagborg-Møller, M. (2019). Simultaneous
163+
#' confidence bands: Theory, implementation, and an application to SVARs.
164+
#' Journal of Applied Econometrics, 34(1), 1–17. \doi{10.1002/jae.2656}
158165
#'
159166
#' @examplesIf all(insight::check_if_installed(c("lme4", "marginaleffects", "rstanarm"), quietly = TRUE))
160167
#' \dontrun{

R/estimate_slopes.R

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@
2626
#' comparisons. For `estimate_slopes()`, multiple comparison only occurs for
2727
#' Johnson-Neyman intervals, i.e. in case of interactions with two numeric
2828
#' predictors (one specified in `trend`, one in `by`). In this case, the
29-
#' `"esarey"` option is recommended, but `p_adjust` can also be one of `"none"`
30-
#' (default), `"hochberg"`, `"hommel"`, `"bonferroni"`, `"BH"`, `"BY"`, `"fdr"`,
31-
#' `"tukey"`, `"sidak"`, or `"holm"`.
29+
#' `"esarey"` or `"sup-t"` options are recommended, but `p_adjust` can also be
30+
#' one of `"none"` (default), `"hochberg"`, `"hommel"`, `"bonferroni"`, `"BH"`,
31+
#' `"BY"`, `"fdr"`, `"tukey"`, `"sidak"`, or `"holm"`. `"sup-t"` computes
32+
#' simultaneous confidence bands, also called sup-t confidence band (Montiel
33+
#' Olea & Plagborg-Møller, 2019).
3234
#' @inheritParams estimate_means
3335
#' @inheritParams parameters::model_parameters.default
3436
#'
@@ -38,6 +40,11 @@
3840
#'
3941
#' @return A data.frame of class `estimate_slopes`.
4042
#'
43+
#' @references
44+
#' Montiel Olea, J. L., and Plagborg-Møller, M. (2019). Simultaneous
45+
#' confidence bands: Theory, implementation, and an application to SVARs.
46+
#' Journal of Applied Econometrics, 34(1), 1–17. \doi{10.1002/jae.2656}
47+
#'
4148
#' @examplesIf all(insight::check_if_installed(c("marginaleffects", "effectsize", "mgcv", "ggplot2", "see"), quietly = TRUE))
4249
#' library(ggplot2)
4350
#' # Get an idea of the data

R/get_marginalmeans.R

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,7 @@ get_marginalmeans <- function(model,
228228
# we can use this function for contrasts as well,
229229
# just need to add "hypothesis" argument
230230
means <- .call_marginaleffects(fun_args)
231+
vcov_means <- .safe(stats::vcov(means))
231232

232233
# intermediate step: joint tests --------------------------------------------
233234
# ---------------------------------------------------------------------------
@@ -281,7 +282,8 @@ get_marginalmeans <- function(model,
281282
datagrid = datagrid,
282283
transform = !is.null(transform),
283284
keep_iterations = keep_iterations,
284-
joint_test = joint_test
285+
joint_test = joint_test,
286+
vcov = vcov_means
285287
)
286288
)
287289
)
@@ -434,7 +436,7 @@ get_marginalmeans <- function(model,
434436
"at", "by", "focal_terms", "adjusted_for", "predict", "trend", "comparison",
435437
"contrast", "estimate", "p_adjust", "transform", "datagrid", "preserve_range",
436438
"coef_name", "slope", "ci", "model_info", "contrast_filter",
437-
"keep_iterations", "joint_test"
439+
"keep_iterations", "joint_test", "vcov"
438440
)
439441
}
440442

R/get_marginaltrends.R

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,7 @@ get_marginaltrends <- function(model,
123123

124124
# Compute stuff
125125
estimated <- suppressWarnings(do.call(marginaleffects::avg_slopes, fun_args))
126+
vcov_slopes <- .safe(stats::vcov(estimated))
126127

127128
# Fourth step: back-transform response --------------------------------------
128129
# ---------------------------------------------------------------------------
@@ -157,7 +158,8 @@ get_marginaltrends <- function(model,
157158
p_adjust = p_adjust,
158159
ci = ci,
159160
transform = !is.null(transform),
160-
keep_iterations = keep_iterations
161+
keep_iterations = keep_iterations,
162+
vcov = vcov_slopes
161163
)
162164
)
163165
)

R/p_adjust.R

Lines changed: 76 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,11 @@
1111
focal <- attributes(params)$contrast
1212
# Use .safe to handle cases where no statistic is extracted
1313
statistic <- .safe(insight::get_statistic(model)$Statistic)
14-
dof <- insight::get_df(model, type = "wald", verbose = FALSE)
14+
# extract degrees of freedom
15+
dof <- .safe(params$df[1])
16+
if (is.null(dof)) {
17+
dof <- insight::get_df(model, type = "wald", verbose = FALSE)
18+
}
1519

1620
# harmonize argument
1721
p_adjust <- tolower(p_adjust)
@@ -20,7 +24,7 @@
2024
# we have to print a different error message.
2125
emmeans_options <- c("scheffe", "mvt", "dunnettx")
2226

23-
all_methods <- c(tolower(stats::p.adjust.methods), emmeans_options, "tukey", "sidak", "esarey")
27+
all_methods <- c(tolower(stats::p.adjust.methods), emmeans_options, "tukey", "sidak", "esarey", "sup-t")
2428
insight::validate_argument(p_adjust, all_methods)
2529

2630
# emmeans methods? Then tell user
@@ -35,6 +39,11 @@
3539
return(.p_adjust_esarey(params))
3640
}
3741

42+
# sup-t is a longer subroutine, so we handle it separately
43+
if (p_adjust == "sup-t") {
44+
return(.p_adjust_supt(params, model))
45+
}
46+
3847
# needed for rank adjustment
3948
focal_terms <- datagrid[focal]
4049
rank_adjust <- prod(vapply(focal_terms, insight::n_unique, numeric(1)))
@@ -68,6 +77,71 @@
6877
}
6978

7079

80+
.p_adjust_supt <- function(params, model) {
81+
insight::check_if_installed("mvtnorm")
82+
# get correlation matrix, based on the covariance matrix
83+
vc <- .safe(stats::cov2cor(attributes(params)$vcov))
84+
if (is.null(vc)) {
85+
insight::format_alert("Could not calculate covariance matrix for `sup-t` adjustment.")
86+
return(params)
87+
}
88+
# get confidence interval level, or set default
89+
ci_level <- attributes(params)$ci
90+
if (is.null(ci_level)) {
91+
ci_level <- 0.95
92+
}
93+
# several sanity checks - we can either have a marginaleffects object, when
94+
# `estimate_slopes()` was called, or a modelbased object, when processing /
95+
# formatting was already done. So we check for both, and extract the required
96+
# columns.
97+
df_column <- intersect(c("df", "df_error"), colnames(params))[1]
98+
if (is.na(df_column)) {
99+
df_column <- ".sup_df"
100+
params[[df_column]] <- Inf
101+
}
102+
coef_column <- intersect(c(.valid_coefficient_names(), "estimate"), colnames(params))[1]
103+
if (is.na(coef_column)) {
104+
insight::format_alert("Could not find coefficient column to apply `sup-t` adjustment.")
105+
return(params)
106+
}
107+
se_column <- intersect(c("SE", "std.error"), colnames(params))[1]
108+
if (is.na(se_column)) {
109+
insight::format_alert("Could not extract standard errors to apply `sup-t` adjustment.")
110+
return(params)
111+
}
112+
p_column <- intersect(c("p", "p.value"), colnames(params))[1]
113+
if (is.na(p_column)) {
114+
insight::format_alert("Could not extract p-values to apply `sup-t` adjustment.")
115+
return(params)
116+
}
117+
ci_low_column <- intersect(c("CI_low", "conf.low"), colnames(params))[1]
118+
ci_high_column <- intersect(c("CI_high", "conf.high"), colnames(params))[1]
119+
if (is.na(ci_low_column) || is.na(ci_high_column)) {
120+
insight::format_alert("Could not extract confidence intervals to apply `sup-t` adjustment.")
121+
return(params)
122+
}
123+
# calculate updated confidence interval level, based on simultaenous
124+
# confidence intervals (https://onlinelibrary.wiley.com/doi/10.1002/jae.2656)
125+
crit <- mvtnorm::qmvt(ci_level, df = params[[df_column]][1], tail = "both.tails", corr = vc)$quantile
126+
# update confidence intervals
127+
params[[ci_low_column]] <- params[[coef_column]] - crit * params[[se_column]]
128+
params[[ci_high_column]] <- params[[coef_column]] + crit * params[[se_column]]
129+
# update p-values
130+
for (i in 1:nrow(params)) {
131+
params[[p_column]][i] <- 1 - mvtnorm::pmvt(
132+
lower = rep(-abs(stats::qt(params[[p_column]][i] / 2, df = params[[df_column]][i])), nrow(vc)),
133+
upper = rep(abs(stats::qt(params[[p_column]][i] / 2, df = params[[df_column]][i])), nrow(vc)),
134+
corr = vc,
135+
df = params[[df_column]][i]
136+
)
137+
}
138+
# clean up - remove temporary column
139+
params[[".sup_df"]] <- NULL
140+
141+
params
142+
}
143+
144+
71145
.p_adjust_esarey <- function(x) {
72146
# only for slopes
73147
if (!inherits(x, c("estimate_slopes", "marginaleffects_slopes"))) {

R/utils.R

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@
7272
msg <- c(msg, insight::color_text(code_snippet, "green"), "\n")
7373
}
7474
message(msg)
75-
} else if (isTRUE(all(out$SE == out$SE[1])) && insight::is_mixed_model(model)) {
75+
} else if (length(out$SE) > 1 && isTRUE(all(out$SE == out$SE[1])) && insight::is_mixed_model(model)) {
7676
msg <- "Standard errors are probably not reliable. This can happen when random effects are involved. You may try `estimate_relation()` instead." # nolint
7777
if (!inherits(model, "glmmTMB")) {
7878
msg <- paste(msg, "You may also try package {.pkg glmmTMB} to produce valid standard errors.")

inst/WORDLIST

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ GAMM
2626
GLMM
2727
GLMMs
2828
GLM's
29+
GMM
30+
GMMs
2931
Greifer
3032
Heiss
3133
Hernan
@@ -48,13 +50,16 @@ Mize
4850
Mmmh
4951
Modelisation
5052
Modelling
53+
Møller
54+
Montiel
5155
Mulinari
5256
Neyman
53-
Nonresponse
57+
Olea
5458
ORCID
5559
Owww
5660
PCVs
5761
PCV
62+
Plagborg
5863
QoL
5964
Psychometrika
6065
RCTs
@@ -71,6 +76,7 @@ Spiller
7176
SSM
7277
Subramanian
7378
SV
79+
SVARs
7480
Versicolor
7581
Virginica
7682
Visualisation

man/estimate_contrasts.Rd

Lines changed: 14 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)