Package {pecme}


Type: Package
Title: Penalized ECME Estimation for Censored Linear Mixed Models
Version: 0.1.1
Description: Fits Gaussian linear mixed models with a random intercept when the response is subject to left, right, and/or interval censoring, using the Expectation/Conditional Maximization Either (ECME) algorithm of Liu and Rubin (1994) in the spirit of the fast censored-response mixed-model algorithm of Vaida and Liu (2009). Simultaneous estimation and variable selection is supported through coordinate-descent penalized maximization with Lasso, Adaptive Lasso, SCAD, MCP, Elastic Net, and Ridge penalties (no penalty is also supported). The random intercept is integrated out by Gauss-Hermite quadrature at every iteration, and the two ECME conditional-maximization steps respectively maximize the expected penalized complete-data objective (for the regression coefficients) and the actual observed-data marginal likelihood (for the variance components), which is the defining feature of ECME relative to plain ECM/EM. The package provides a single-fit engine, a sequential/parallel penalty-parameter grid search with information-criterion or cross-validated selection, data-dependent or user-supplied lambda grids, and an Expectation-Maximization based treatment of a completely missing (at random) response, sharing the same truncated-normal machinery used for censoring. References: Liu and Rubin (1994) "The ECME algorithm: A simple extension of EM and ECM with faster monotone convergence" <doi:10.1093/biomet/81.4.633>; Vaida and Liu (2009) "Fast Implementation for Normal Mixed Effects Models With Censored Response" <doi:10.1198/jcgs.2009.07130>.
License: MIT + file LICENSE
Encoding: UTF-8
Depends: R (≥ 4.1.0)
Imports: stats, graphics, parallel, lme4, withr
Suggests: testthat (≥ 3.0.0), waldo, mice
Config/testthat/edition: 3
Config/roxygen2/version: 8.0.0
NeedsCompilation: no
Packaged: 2026-09-27 14:27:04 UTC; Haeri
Author: Ali Asghar Haeri-Mehrizi [aut, cre], Arshia Haeri-Mehrizi [aut], Adel Mohammadpour [rev]
Maintainer: Ali Asghar Haeri-Mehrizi <haeri.stat@gmail.com>
Repository: CRAN
Date/Publication: 2026-10-07 09:00:08 UTC

pecme: Penalized ECME Estimation for Censored Linear Mixed Models

Description

Fits Gaussian random-intercept linear mixed models with a left-, right-, and/or interval-censored (and optionally missing) response via the ECME algorithm, with simultaneous penalized variable selection (Lasso, Adaptive Lasso, SCAD, MCP, Elastic Net, Ridge, or no penalty). See [pecme()] for a single fit and [grid_pecme()] for penalty-parameter selection.

Model

Y^*_{ij} = x_{ij}'\beta + b_j + e_{ij}, \quad e_{ij}\sim N(0,\sigma^2), \quad b_j \sim N(0,\sigma_b^2)

with Y^* observed exactly, left-censored, right-censored, interval-censored, or completely missing, depending on the row.

Algorithm

Liu and Rubin's (1994) ECME algorithm: an E-step (Gauss-Hermite quadrature for the random intercept – a numerical approximation to the continuous integral, not an exact evaluation of it; see [pecme_quadrature_check()] – with closed-form truncated-normal moments for censored/missing rows) alternates with two conditional-maximization steps – a penalized coordinate-descent CM-step for \beta that maximizes the expected complete-data objective, and an ECME CM-step for (\sigma,\sigma_b) that maximizes the actual observed-data marginal likelihood directly.

Author(s)

Maintainer: Ali Asghar Haeri-Mehrizi haeri.stat@gmail.com

Authors:

Other contributors:


Stable log(Phi(b) - Phi(a)) for possibly-vectorized a < b

Description

Stable log(Phi(b) - Phi(a)) for possibly-vectorized a < b

Usage

.pecme_interval_logprob(a, b)

log(exp(log_a) - exp(log_b)) for log_a > log_b, stable near 0

Description

log(exp(log_a) - exp(log_b)) for log_a > log_b, stable near 0

Usage

.pecme_logspace_sub(log_a, log_b)

Gauss-Hermite nodes and weights for N(0, 1) integration

Description

Returns nodes z and weights w such that, for a smooth function f, E[f(Z)] \approx \sum_k w_k f(z_k) when Z \sim N(0,1). Results are cached per quadrature order because the Golub-Welsch eigendecomposition is by far the most expensive part of the E-step if it is repeated every ECME iteration.

Usage

gauss_hermite_normal(n = 40L)

Arguments

n

Integer number of quadrature nodes (>= 5).

Value

A list with nodes and weights, both of length n, ordered by increasing node value, calibrated so that they directly integrate against the standard normal density (i.e. the physicists' Hermite rule has already been rescaled).


Grid search over the penalty parameter lambda

Description

Grid search over the penalty parameter lambda

Usage

grid_pecme(
  fixed_formula,
  random_var,
  data,
  censor = "none",
  left = NULL,
  right = NULL,
  penalty = c("lasso", "adaptive", "scad", "mcp", "elastic", "ridge", "none"),
  lambda = NULL,
  nlambda = 50L,
  lambda_min_ratio = NULL,
  alpha = 0.5,
  gamma = NULL,
  adaptive_gamma = 1,
  criterion = c("bic", "ebic", "aic", "gcv", "cv"),
  ebic_gamma = 0.5,
  cv_folds = 5L,
  cv_seed = NULL,
  missing_y = FALSE,
  missing_x = c("error", "cca", "mean", "reg"),
  standardize = TRUE,
  quadrature = 40L,
  workers = 1L,
  parallel_over = c("lambda", "groups"),
  variance_step = c("ecme", "em"),
  refit_at_best = TRUE,
  init_method = c("lmer", "zero", "random"),
  max_iter = 200L,
  tol = 1e-06,
  cd_max_iter = 1000L,
  cd_tol = 1e-08,
  keep_path = TRUE,
  verbose = FALSE
)

Arguments

fixed_formula

A formula for the fixed effects, e.g. 'y ~ x1 + x2'.

random_var

Character: name of the grouping (cluster) column in 'data'. Currently a random INTERCEPT only (no random slopes).

data

A data frame.

censor

Either '"none"'/'"left"'/'"right"'/'"interval"' applied to every row, a column name in 'data' holding per-row labels from that set, or a vector of the same length as 'nrow(data)'. Rows do not need to be pre-labelled '"missing"': set 'missing_y = TRUE' and leave the response 'NA' for those rows.

left, right

Column names or numeric vectors of censoring bounds (used only for the corresponding 'censor' labels). With censoring, the response must be supplied on its analysis scale (transformed responses such as 'log(y)' are rejected); censoring bounds are never transformed automatically.

penalty

One of '"none"', '"lasso"', '"adaptive"', '"scad"', '"mcp"', '"elastic"', '"ridge"'.

lambda

Either 'NULL' (default: build a data-dependent grid via [pecme_lambda_max()]/[pecme_lambda_grid()]) or a user-supplied numeric vector of candidate lambda values.

nlambda

Number of grid points when 'lambda = NULL'.

lambda_min_ratio

'lambda_min = lambda_min_ratio * lambda_max' when 'lambda = NULL'; defaults to the usual 'n > p' / 'n <= p' heuristic (see [pecme_lambda_grid()]).

alpha

Elastic Net mixing parameter in '[0,1]' ('1' = Lasso, '0' = Ridge); ignored unless 'penalty = "elastic"'.

gamma

Extra shape parameter for SCAD ('> 2', default '3.7') or MCP ('> 1', default '3'); ignored otherwise.

adaptive_gamma

Exponent used to build Adaptive Lasso weights from a Ridge pilot fit: 'weights = 1 / |beta_pilot|^adaptive_gamma'.

criterion

One of '"bic"' (default), '"ebic"', '"aic"', '"gcv"', or '"cv"'. '"cv"' performs leave-groups-out K-fold cross-validation (refits the whole model 'cv_folds' times per lambda) and selects by held-out predictive log-likelihood; this is far more expensive than the closed-form criteria.

ebic_gamma

Extended-BIC tuning constant in '[0,1]' (Chen and Chen, 2008); '0' reduces EBIC to ordinary BIC. Only used when 'criterion = "ebic"'.

cv_folds

Number of folds (by group, not by observation) for 'criterion = "cv"'.

cv_seed

Optional integer seed for the fold assignment.

missing_y

If 'TRUE', rows with a missing response (or a missing bound for a labelled-censored row) are treated as completely missing at random and IMPUTED via the E-step's posterior mean at every iteration, instead of raising an error. Important: such a row contributes exactly 'log(1) = 0' to the observed-data log-likelihood – it carries NO direct information about '(beta, sigma, sigma_b)'; the imputed value is a model- implied *prediction* for that row given the parameters, not evidence that helped estimate them. The parameters themselves are identified entirely by the observed and censored rows.

missing_x

How to handle 'NA' in the fixed-effects predictors: '"error"' (default), '"cca"', '"mean"', or '"reg"'. The '"mean"' and '"reg"' options impute the numeric model matrix and are intended only for suitable numeric predictors. For factors, interactions, splines, or nonlinear terms, impute raw variables before constructing the model matrix (e.g. with 'mice'). See [pecme_handle_missing_x()] and [pecme_mi_combine()].

standardize

Standardize predictors before penalization (recommended; default 'TRUE').

quadrature

Number of Gauss-Hermite nodes for the random intercept (default '40'). This is a fixed-order numerical approximation to the random-intercept integral, not an exact evaluation of it; use [pecme_quadrature_check()] on a fitted model to check sensitivity to this choice, and increase it if 'sigma_b' is large relative to 'sigma', or convergence looks unstable.

workers

Number of parallel worker processes for the E-step (default '1'). Set to e.g. '32' on a 32-core machine.

parallel_over

'"lambda"' (default: one full ECME fit per worker, embarrassingly parallel – usually the better choice for 'nlambda > workers') or '"groups"' (parallelize the E-step across groups within each, sequential, lambda – better for a single/small grid on data with many groups). Only one level of parallelism is ever used at a time; 'workers' are never oversubscribed by nesting both.

variance_step

'"ecme"' (default) or '"em"'; see Details.

refit_at_best

As in [pecme()]'s 'refit', but applied only to the final selected lambda (grid points themselves are fit with 'refit = FALSE' to avoid 'nlambda' extra unpenalized re-fits).

init_method

'"lmer"' (default), '"zero"', or '"random"'. With ‘"random"', starting values are drawn from the caller’s ongoing global random-number stream (unlike [simulate_pecme_data()]'s 'seed', this is NOT locally seed-preserved – by design, since the point of '"random"' is to give a genuinely different starting point on repeated calls, e.g. for a multi-start convergence check). Call 'set.seed()' yourself beforehand if you need the specific starting values to be reproducible.

max_iter, tol

Outer ECME iteration limit/tolerance.

cd_max_iter, cd_tol

Inner coordinate-descent iteration limit/tolerance.

keep_path

Keep the fitted 'beta' at every grid point (for plotting a regularization path)? Default 'TRUE'.

verbose

Print iteration progress.

Value

An object of class '"pecme_grid"': a list with '$best' (the selected '"pecme"' fit, refit per 'refit_at_best'), '$lambda', '$scores' (a data frame of one row per lambda with every criterion, 'df', 'logLik', and convergence status), '$best_index', and '$criterion'.


Numerically stable log(sum(exp(x)))

Description

Numerically stable log(sum(exp(x)))

Usage

logsumexp(x)

Fit a penalized ECME censored/missing-response linear mixed model

Description

Fits Y^*_{ij} = x_{ij}'\beta + b_j + e_{ij}, e_{ij} \sim N(0,\sigma^2), b_j \sim N(0,\sigma_b^2), where the response Y^* may be left-, right-, or interval-censored (and/or missing), by the ECME algorithm of Liu and Rubin (1994). The random intercept is integrated out numerically by fixed-order Gauss-Hermite quadrature (a numerical approximation to the continuous integral, not an exact evaluation of it; see [pecme_quadrature_check()] for a sensitivity diagnostic and 'quadrature' below). \beta is estimated by a penalized CM-step (coordinate descent); (\sigma,\sigma_b) are estimated, by default, with a true ECME CM-step that directly maximizes the observed-data marginal likelihood (not merely its EM lower bound), following the strategy of Vaida and Liu (2009) for censored linear mixed models.

Usage

pecme(
  fixed_formula,
  random_var,
  data,
  censor = "none",
  left = NULL,
  right = NULL,
  penalty = c("none", "lasso", "adaptive", "scad", "mcp", "elastic", "ridge"),
  lambda = 0,
  alpha = 0.5,
  gamma = NULL,
  adaptive_gamma = 1,
  missing_y = FALSE,
  missing_x = c("error", "cca", "mean", "reg"),
  standardize = TRUE,
  quadrature = 40L,
  workers = 1L,
  variance_step = c("ecme", "em"),
  refit = TRUE,
  init_method = c("lmer", "zero", "random"),
  max_iter = 200L,
  tol = 1e-06,
  cd_max_iter = 1000L,
  cd_tol = 1e-08,
  verbose = FALSE
)

Arguments

fixed_formula

A formula for the fixed effects, e.g. 'y ~ x1 + x2'.

random_var

Character: name of the grouping (cluster) column in 'data'. Currently a random INTERCEPT only (no random slopes).

data

A data frame.

censor

Either '"none"'/'"left"'/'"right"'/'"interval"' applied to every row, a column name in 'data' holding per-row labels from that set, or a vector of the same length as 'nrow(data)'. Rows do not need to be pre-labelled '"missing"': set 'missing_y = TRUE' and leave the response 'NA' for those rows.

left, right

Column names or numeric vectors of censoring bounds (used only for the corresponding 'censor' labels). With censoring, the response must be supplied on its analysis scale (transformed responses such as 'log(y)' are rejected); censoring bounds are never transformed automatically.

penalty

One of '"none"', '"lasso"', '"adaptive"', '"scad"', '"mcp"', '"elastic"', '"ridge"'.

lambda

Non-negative penalty strength.

alpha

Elastic Net mixing parameter in '[0,1]' ('1' = Lasso, '0' = Ridge); ignored unless 'penalty = "elastic"'.

gamma

Extra shape parameter for SCAD ('> 2', default '3.7') or MCP ('> 1', default '3'); ignored otherwise.

adaptive_gamma

Exponent used to build Adaptive Lasso weights from a Ridge pilot fit: 'weights = 1 / |beta_pilot|^adaptive_gamma'.

missing_y

If 'TRUE', rows with a missing response (or a missing bound for a labelled-censored row) are treated as completely missing at random and IMPUTED via the E-step's posterior mean at every iteration, instead of raising an error. Important: such a row contributes exactly 'log(1) = 0' to the observed-data log-likelihood – it carries NO direct information about '(beta, sigma, sigma_b)'; the imputed value is a model- implied *prediction* for that row given the parameters, not evidence that helped estimate them. The parameters themselves are identified entirely by the observed and censored rows.

missing_x

How to handle 'NA' in the fixed-effects predictors: '"error"' (default), '"cca"', '"mean"', or '"reg"'. The '"mean"' and '"reg"' options impute the numeric model matrix and are intended only for suitable numeric predictors. For factors, interactions, splines, or nonlinear terms, impute raw variables before constructing the model matrix (e.g. with 'mice'). See [pecme_handle_missing_x()] and [pecme_mi_combine()].

standardize

Standardize predictors before penalization (recommended; default 'TRUE').

quadrature

Number of Gauss-Hermite nodes for the random intercept (default '40'). This is a fixed-order numerical approximation to the random-intercept integral, not an exact evaluation of it; use [pecme_quadrature_check()] on a fitted model to check sensitivity to this choice, and increase it if 'sigma_b' is large relative to 'sigma', or convergence looks unstable.

workers

Number of parallel worker processes for the E-step (default '1'). Set to e.g. '32' on a 32-core machine.

variance_step

'"ecme"' (default) or '"em"'; see Details.

refit

If 'TRUE' (default) and 'penalty != "none"', an additional unpenalized refit restricted to the selected coefficients ("relaxed refit") is stored in '$refit'. This reduces the shrinkage bias of the penalized estimates for reporting, but it is NOT generally valid unconditional post-selection inference: the selection step itself is not accounted for, so treat its 'se' and p-values as approximate/exploratory, not confirmatory.

init_method

'"lmer"' (default), '"zero"', or '"random"'. With ‘"random"', starting values are drawn from the caller’s ongoing global random-number stream (unlike [simulate_pecme_data()]'s 'seed', this is NOT locally seed-preserved – by design, since the point of '"random"' is to give a genuinely different starting point on repeated calls, e.g. for a multi-start convergence check). Call 'set.seed()' yourself beforehand if you need the specific starting values to be reproducible.

max_iter, tol

Outer ECME iteration limit/tolerance.

cd_max_iter, cd_tol

Inner coordinate-descent iteration limit/tolerance.

verbose

Print iteration progress.

Value

An object of class '"pecme"'.

References

Liu, C. and Rubin, D. B. (1994). The ECME algorithm. *Biometrika*, 81(4), 633-648.

Vaida, F. and Liu, L. (2009). Fast implementation for normal mixed effects models with censored response. *JCGS*, 18(1), 151-162.

Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. *JASA*, 96(456), 1348-1360.

Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. *Annals of Statistics*, 38(2), 894-942.

See Also

[grid_pecme()], [pecme_mi_combine()]


Cluster (group-level) bootstrap for a pecme fit

Description

Resamples whole groups with replacement, refits the identical model (same formula, 'penalty', 'lambda', 'alpha', 'gamma') to each resampled data set, and summarizes the resulting coefficient distribution – a valid source of standard errors and confidence intervals for a penalized fit, where naive SEs do not apply.

Usage

pecme_bootstrap(
  object,
  R = 200L,
  workers = NULL,
  ci_level = 0.95,
  seed = NULL,
  verbose = FALSE
)

Arguments

object

A '"pecme"' fit.

R

Number of bootstrap replicates (default '200'; increase for final reporting, e.g. '1000+').

workers

Parallel workers across replicates (default: reuse 'object$workers').

ci_level

Confidence level for the percentile interval.

seed

Optional integer seed (locally preserved; see [simulate_pecme_data()]).

verbose

Print replicate progress.

Details

Because each replicate is refit at a FIXED penalty/lambda, this captures sampling variability in the coefficient estimates GIVEN that regularization strength, but not uncertainty in the choice of lambda itself, and not formal selection-event-conditional inference for which predictors end up nonzero. A replicate in which a coefficient is shrunk to exactly zero contributes 0 to that coefficient's bootstrap distribution (not 'NA'), so the reported bootstrap SE reflects both estimation and selection variability together for that coefficient.

Value

An object of class '"pecme_bootstrap"': a list with 'table' (a data frame of 'estimate', 'boot_mean', 'boot_se', and percentile 'ci_lower'/'ci_upper' per coefficient), 'replicates' (the raw 'R x p' matrix of bootstrap coefficient estimates), 'R', 'n_failed' (replicates that errored or failed to converge, excluded), and 'ci_level'.


Conditional log-likelihood contribution log P(observation | B = b)

Description

Vectorized over observations; 'mu' may be a matrix ('n x K') for simultaneous evaluation at several quadrature nodes.

Usage

pecme_conditional_loglik(y, mu, sigma, censor, left, right)

Closed-form coordinate update for the penalized quadratic subproblem

Description

Solves, in the single coordinate 'beta_j',

\min_{\beta_j} \frac{v_j \beta_j^2 - 2 z_j \beta_j}{2\sigma^2} + p_\lambda(|\beta_j|)

for each of the supported penalties, where 'z_j' is the partial cross-product of column 'j' with the partial residual and 'v_j' is its sum of squares. 'lambda_star = lambda * sigma^2' rescales the penalty onto the same footing as the (unscaled) least-squares term.

Usage

pecme_coordinate_update(
  zj,
  vj,
  sigma,
  lambda,
  penalty,
  alpha = 0.5,
  gamma = 3.7,
  wj = 1
)

Naive parameter count used for AIC/BIC/EBIC/GCV

Description

Counts nonzero fixed-effects coefficients plus two variance components ('sigma', 'sigma_b'). This is a convenient, commonly used HEURISTIC, not the effective degrees of freedom in the technical sense the AIC/GCV theory assumes. It is a reasonable approximation for Lasso-type penalties under regularity conditions (Zou, Hastie & Tibshirani, 2007, show 'E[#nonzero]' is an unbiased estimate of the Lasso's effective df), but it is NOT generally accurate for Ridge (which keeps essentially every coefficient nonzero, so this count is close to the maximum 'p' regardless of how much shrinkage is actually applied – see [pecme_effective_df()] for a proper, closed-form Ridge alternative) nor for Elastic Net, SCAD, or MCP (whose effective df would need a generalized-df/local-derivative calculation, e.g. Zou et al. 2007's generalized df; not implemented here). AIC/BIC/EBIC/GCV computed from this 'df' should therefore be read as approximate, especially for Ridge, Elastic Net, SCAD, and MCP.

Usage

pecme_df(beta, has_intercept = TRUE, tol = 1e-08)

Sequential (single-core) E-step

Description

Sequential (single-core) E-step

Usage

pecme_e_step(
  y,
  X,
  beta,
  group,
  sigma,
  sigma_b,
  censor,
  left = NULL,
  right = NULL,
  quadrature = 40L
)

Arguments

y, X, beta, group, sigma, sigma_b, censor, left, right

Model quantities; see [pecme()].

quadrature

Number of Gauss-Hermite nodes for the random intercept.

Value

A list of posterior moments: 'Ey', 'Ey2' (per observation), 'Eb', 'Ebb' (per observation, broadcast from the group), 'Eyb' (per observation), and 'group_Eb', 'group_Ebb' (per group, in the order of 'sort(unique(group))').


E-step dispatcher: sequential or cluster-parallel over groups

Description

With 'workers <= 1', or too few groups to make parallelism worthwhile, this simply calls [pecme_e_step()]. Otherwise the groups are partitioned across a pre-existing PSOCK/FORK cluster (created once per [pecme()] call and reused across iterations).

Usage

pecme_e_step_dispatch(
  y,
  X,
  beta,
  group,
  sigma,
  sigma_b,
  censor,
  left = NULL,
  right = NULL,
  quadrature = 40L,
  workers = 1L,
  cluster = NULL
)

Arguments

cluster

A '"cluster"' object from 'parallel::makeCluster()', or 'NULL' for sequential execution.


Working Ridge-type effective degrees-of-freedom approximation

Description

For Ridge, the naive nonzero-coefficient count from [pecme_df()] is a poor stand-in for effective df. The classical Ridge smoother trace is used as a CONDITIONAL/WORKING approximation on the standardized fixed- effects design. It is not claimed to be the exact effective df of the complete PECME estimator, which also contains censoring, random effects, estimated variance components, quadrature, and possibly adaptive penalization.

Usage

pecme_effective_df(object)

Arguments

object

A '"pecme"' fit.

Details

No corresponding closed form is implemented here for Elastic Net, SCAD, or MCP (a generalized-df/local-derivative calculation would be needed, e.g. Zou, Hastie & Tibshirani, 2007); for '"none"'/'"lasso"'/ '"adaptive"' the naive nonzero count is already a standard, reasonable approximation and is returned unchanged (plus the 2 variance components).

Value

A working/conditional df approximation, including the intercept if present and the two variance components, for '"ridge"', '"none"', '"lasso"', or '"adaptive"' fits; 'NA_real_' (with an explanatory 'attr(, "note")') for '"elastic"', '"scad"', or '"mcp"', where no closed form is implemented.

References

Hastie, T. and Tibshirani, R. (1990). *Generalized Additive Models*. Chapman and Hall. (Ridge effective df / "trace of the hat matrix".)

Zou, H., Hastie, T. and Tibshirani, R. (2007). On the "degrees of freedom" of the lasso. *Annals of Statistics*, 35(5), 2173-2192.


Recode a response/censoring specification to include missing values

Description

Given raw vectors 'y', 'left', 'right' and a 'censor' label vector ('"none"', '"left"', '"right"', '"interval"'), returns a new 'censor' vector in which any row with 'is.na(y)' (for '"none"'/'"left"'/ '"right"'/'"interval"' rows that have no usable censoring bound either) is relabelled '"missing"'.

Usage

pecme_flag_missing_response(y, left, right, censor)

Handle missing values in the fixed-effects design matrix

Description

Handle missing values in the fixed-effects design matrix

Usage

pecme_handle_missing_x(
  X,
  method = c("error", "cca", "mean", "reg"),
  has_intercept = TRUE,
  max_iter = 10L
)

Arguments

X

Numeric design matrix (already including the intercept column, if any – the intercept column is never touched).

method

One of: * '"error"' (default): stop with an informative message. * '"cca"': complete-case analysis – rows with any missing covariate are dropped (with a warning); response/group/censoring vectors must be subset identically by the caller. * '"mean"': unconditional column-mean imputation (deterministic, ignores correlation between predictors – fast but understates uncertainty; only intended for quick exploration). * '"reg"': iterative conditional-mean (regression) imputation: each incomplete column is regressed in turn on the current values of the other columns and imputed with the fitted mean, cycling until stable or 'max_iter' sweeps. This is a single imputation and, like '"mean"', does not propagate imputation uncertainty into the final standard errors.

has_intercept

Whether column 1 of 'X' is an intercept (excluded from imputation modeling on the right-hand side is not necessary, but it is never itself imputed since it has no NAs).

Value

A list with the (possibly modified) 'X', a logical 'keep' vector (all 'TRUE' unless 'method = "cca"'), and 'n_missing'.


Information criteria (AIC/BIC/EBIC/GCV)

Description

AIC/BIC/GCV use 'df', the naive nonzero-coefficient-based parameter count of [pecme_df()] – a heuristic, not a general effective-degrees-of-freedom calculation (see [pecme_df()] and [pecme_effective_df()]). This matters most for Ridge (essentially no coefficients are ever exactly zero, so 'df' sits near the maximum regardless of shrinkage strength) and, to a lesser extent, Elastic Net/SCAD/MCP; treat AIC/BIC/EBIC/GCV from those fits as approximate, and prefer [pecme_effective_df()]-based comparisons for Ridge, or cross-validated criteria ('criterion = "cv"' in [grid_pecme()], [pecme_loo()]) when the choice of 'df' matters for the conclusion.

Usage

pecme_ic(logLik, df, n, p_total, gamma_ebic = 0.5, n_selected = NULL)

Build a log-spaced lambda grid

Description

Build a log-spaced lambda grid

Usage

pecme_lambda_grid(
  lambda_max,
  nlambda = 50L,
  lambda_min_ratio = NULL,
  n = NULL,
  p = NULL
)

Arguments

lambda_max

Upper end of the grid.

nlambda

Number of grid points.

lambda_min_ratio

'lambda_min = lambda_min_ratio * lambda_max'. Defaults follow the usual 'n > p' vs 'n <= p' heuristic ('1e-4' / '1e-2' respectively) unless supplied explicitly.

n

Number of observations.

p

Number of predictors.

Value

A numeric vector of length 'nlambda' containing a decreasing log-spaced sequence from 'lambda_max' to 'lambda_min'.


Leave-one-group-out (or leave-one-fold-out) cross-validated log-likelihood

Description

Refits the model with each group (or fold of groups) held out in turn and evaluates the held-out marginal log-likelihood at the refit parameters – a genuine, frequentist out-of-sample predictive criterion for clustered data (leave-one-*observation*-out is not meaningful here since observations within a group share a random intercept).

Usage

pecme_loo(object, method = c("exact", "kfold"), folds = 10L, workers = NULL)

Arguments

object

A '"pecme"' fit (from [pecme()] or 'grid_pecme()$best').

method

Character string specifying the resampling method.

folds

Integer specifying the number of folds for K-fold cross-validation.

workers

Parallel workers for the LOO refits (default: reuse 'object$workers').

Value

A list with 'elpd_loo' (summed held-out log-likelihood, on the same scale as 'logLik'), 'se' (a naive standard error across successful group/fold contributions), 'n_successful_folds', and 'pointwise' (a data frame of one row per group/fold).


Marginal observed-data log-likelihood, random intercept integrated out

Description

\ell(\beta,\sigma,\sigma_b) = \sum_{j=1}^J \log \int \prod_{i \in j} P(y_{ij} \mid x_{ij}'\beta + b, \sigma)\, \phi(b; 0, \sigma_b^2)\, db

Usage

pecme_marginal_loglik(
  y,
  X,
  group,
  beta,
  sigma,
  sigma_b,
  censor,
  left = NULL,
  right = NULL,
  quadrature = 40L
)

Arguments

y, X, beta, group, sigma, sigma_b, censor, left, right

Model quantities; see [pecme()].

quadrature

Number of Gauss-Hermite nodes for the random intercept.

Details

evaluated by Gauss-Hermite quadrature. This is the objective that the ECME step maximizes (over 'sigma', 'sigma_b') and that the penalized criterion in [pecme()] is built from (over 'beta', given 'sigma', 'sigma_b').


Comprehensive fit-quality metrics for a pecme model

Description

Collects the quantities typically reported for a mixed-model fit: pointwise prediction error (MSE/RMSE/MAE/MAPE/SMAPE), two kinds of pseudo-R-squared, information criteria, and (optionally) a cross-validated predictive log-likelihood.

Usage

pecme_metrics(
  object,
  newdata = NULL,
  newresponse = NULL,
  loo = FALSE,
  loo_method = c("exact", "kfold"),
  loo_folds = 10L,
  workers = NULL,
  ebic_gamma = 0.5
)

Arguments

object

A '"pecme"' fit (from [pecme()] or 'grid_pecme()$best').

newdata, newresponse

Optional: evaluate prediction-error metrics out-of-sample instead of on the training data. 'newdata' needs the predictor and grouping columns. The observed response is looked up automatically from 'newdata' using the same column name as in the fitted model's formula ('response' has no independent role – see [pecme()]); pass 'newresponse' explicitly only if 'newdata' does not itself contain that column (e.g. truly held-out rows with unknown outcomes, being scored on other rows' observed values). Rows with a missing/'NA' response (however obtained) are silently dropped from the error metrics, since there is no single observed value to compare a point prediction against.

loo

If 'TRUE', additionally compute a cross-validated predictive log-likelihood by actually refitting the model with one group ('loo_method = "exact"') or one fold of groups ('loo_method = "kfold"') held out at a time, and evaluating the held-out marginal log-likelihood. This is a genuine, if expensive, out-of-sample criterion; see Details for why it is not the PSIS-LOO of the 'loo' package.

loo_method

'"exact"' (leave-one-group-out; 'J' refits) or '"kfold"' (leave-one-fold-out; 'loo_folds' refits).

loo_folds

Number of folds when 'loo_method = "kfold"'.

workers

Parallel workers for the LOO refits (default: reuse 'object$workers').

ebic_gamma

Extended-BIC tuning constant (see [grid_pecme()]); only affects the recomputed 'EBIC' entry.

Details

# Why there is no WAIC WAIC (and DIC) are defined from the variance of the pointwise log-likelihood *across posterior draws* of the parameters. 'pecme' is a frequentist penalized-likelihood estimator: it returns a single point estimate (plus, for an unpenalized fit, an asymptotic Hessian- based covariance), not a posterior sample. Reporting a WAIC number here would therefore not be a real WAIC – rather than approximate it, '$WAIC' is returned as 'NA' with an explanatory 'attr'. For out-of-sample model comparison, use 'AIC'/'BIC'/'EBIC' (fast, from the fitted likelihood) or 'loo = TRUE' (slower, but a genuine held-out predictive log-likelihood, which is the same quantity WAIC and PSIS-LOO both aim to approximate).

# Two R-squared values 'R2_marginal' and 'R2_conditional' follow Nakagawa & Schielzeth (2013): 'R2_marginal = Var(fixed) / (Var(fixed) + sigma_b^2 + sigma^2)‘ uses only the fixed-effects predictor’s variance in the numerator; 'R2_conditional' additionally credits the random intercept, '(Var(fixed) + sigma_b^2) / (Var(fixed) + sigma_b^2 + sigma^2)'. These are the R-squared values normally reported for a mixed model. 'R2_predictive' = '1 - MSE / Var(y)' is the more familiar OLS-style quantity computed directly from predictions; it is provided for intuition/comparability but is not the standard mixed-model R-squared. 'R2_predictive_adj' heuristically applies the usual OLS adjusted-R2 formula with 'object$df'; this adjustment is not a rigorously derived quantity for mixed models and should be reported as a diagnostic, not a citation-grade statistic.

Value

An object of class '"pecme_metrics"' (a list; has a 'print' method).


Combine ‘pecme' fits across multiply-imputed data sets (Rubin’s rules)

Description

A convenience helper for the recommended workflow when fixed-effects covariates (not the response) have missing values: impute 'm' complete data sets externally (e.g. with the 'mice' package), fit the *same* ‘pecme()' model to each, and combine with Rubin’s rules.

Usage

pecme_mi_combine(fits, use_refit = FALSE)

Arguments

fits

A list of 'm' objects of class '"pecme"', all fit with identical 'penalty'/'lambda'/'alpha'/'gamma' and the same fixed coefficient names/order.

use_refit

If ‘TRUE', pool each fit’s '$refit' sub-fit instead of 'fits' themselves (see Details); required whenever 'fits' are penalized.

Details

Rubin's rules require a (classically) valid standard error for every fit being combined. A penalized fit ('penalty != "none"' and 'lambda > 0') deliberately stores 'se = NA' (see [pecme()]): naive standard errors are not valid for a shrinkage estimate, so silently pooling 'NA's here would produce 'NA' within-imputation variance and a misleading pooled result. Because of this, 'pecme_mi_combine()':

* by default, REQUIRES every element of 'fits' to have valid SEs ('se_valid == TRUE', i.e. 'penalty == "none"' or 'lambda == 0') and errors otherwise, naming which elements fail and why; * with ‘use_refit = TRUE', instead pools each fit’s '$refit' – the automatic unpenalized "relaxed refit" on the selected predictors (see [pecme()]'s 'refit' argument) – which does have valid SEs. This is a practical compromise, not a rigorous solution: (a) the relaxed refit is not valid unconditional post-selection inference to begin with (its SEs ignore the selection step); (b) different imputed data sets can select different variable subsets, so a coefficient absent from one imputation's refit is treated as exactly 0 (with 'se = 0') in that imputation for pooling purposes, which folds selection instability into the between-imputation variance rather than separating it out. Report results obtained this way as approximate/exploratory.

For fully rigorous inference on a penalized, multiply-imputed fit, see [pecme_bootstrap()] (a cluster bootstrap) applied within each imputed data set, or consult a specialist reference on selective/ post-selection inference under multiple imputation.

Value

A data frame with the pooled estimate, within-imputation variance, between-imputation variance, total variance, and pooled standard error for each fixed-effect coefficient.


Penalized observed-data objective: -logLik(beta, sigma, sigma_b) + P_lambda(beta)

Description

Penalized observed-data objective: -logLik(beta, sigma, sigma_b) + P_lambda(beta)

Usage

pecme_objective(
  beta,
  y,
  X,
  sigma,
  sigma_b,
  group,
  censor,
  left = NULL,
  right = NULL,
  penalty,
  lambda,
  alpha = 0.5,
  gamma = 3.7,
  weights = NULL,
  has_intercept = TRUE,
  quadrature = 40L
)

Conditional (given the random intercept) observed-data log-likelihood

Description

Sum over observations of 'log P(observation | mu, sigma)', where 'mu' already includes the random intercept. Used mainly for testing.

Usage

pecme_observed_loglik(y, mu, sigma, censor, left = NULL, right = NULL)

Conditional expected complete-data objective for beta (the CM-step target)

Description

Given E-step moments, the beta-dependent part of the expected complete-data penalized log-likelihood reduces to

Q(\beta) = \frac{1}{2\sigma^2} \sum_i (E[y^*_i] - E[b_{g(i)}] - x_i'\beta)^2 + P_\lambda(\beta) + \text{const}

because E\|y^* - b - X\beta\|^2 = \mathrm{Var}(y^*-b\mid \text{data}) + (E[y^*-b\mid\text{data}] - X\beta)^2 and the variance term does not depend on beta. This is what makes coordinate descent on the adjusted response 'E[y*] - E[b]' exact rather than an approximation.

Usage

pecme_q_beta(
  beta,
  y_star,
  X,
  random_effect = NULL,
  sigma,
  penalty = .pecme_penalties,
  lambda = 0,
  alpha = 0.5,
  gamma = 3.7,
  weights = NULL,
  has_intercept = TRUE
)

Check sensitivity of a fitted model to the Gauss-Hermite quadrature order

Description

Fixed-order Gauss-Hermite quadrature is a numerical approximation to the (otherwise intractable) integral over the random intercept, not an exact evaluation of it. This function reports how much the fit changes as the number of quadrature nodes changes, so 'quadrature' is a checked choice rather than an unexamined default. Approximation error is typically largest when 'sigma_b' is large relative to 'sigma', or with heavy censoring/many missing responses.

Usage

pecme_quadrature_check(
  object,
  quadrature_values = c(15L, 20L, 30L, 50L, 80L, 120L),
  method = c("loglik", "refit")
)

Arguments

object

A '"pecme"' fit.

quadrature_values

Integer vector of node counts to compare against ‘object'’s own 'quadrature', which is always included.

method

'"loglik"' (default; cheap: re-evaluates the marginal log-likelihood at ‘object'’s already-fitted '(beta, sigma, sigma_b)' under each node count, with no re-optimization) or '"refit"' (expensive: fully refits the model – including its 'beta', 'sigma', 'sigma_b' – at each node count, showing whether the actual parameter *estimates*, not just the likelihood value at a fixed point, are sensitive to the quadrature order).

Value

An object of class '"pecme_quadrature_check"': a data frame ('quadrature', 'logLik', and, for 'method = "refit"', 'sigma', 'sigma_b', 'max_abs_beta_change' relative to 'object'), with a 'print' method that flags whether the range of 'logLik' values looks small relative to typical evidence thresholds.

Examples


sim <- simulate_pecme_data(n_groups = 25, n_per_group = 8, p = 3, seed = 1)
fit <- pecme(y ~ x1 + x2 + x3, random_var = "group", data = sim$data)
pecme_quadrature_check(fit)


Conditional first/second moments of (possibly censored or missing) Y*

Description

Vectorized over observations and, optionally, over a second dimension (quadrature nodes for the random intercept): 'mu' may be a matrix of shape 'n x K', in which case moments are returned as 'n x K' matrices.

Usage

pecme_truncated_moments(mu, sigma, censor, left, right, y)

Value of the penalty term P_lambda(beta)

Description

Value of the penalty term P_lambda(beta)

Usage

penalty_value(
  beta,
  penalty = .pecme_penalties,
  lambda,
  alpha = 0.5,
  gamma = 3.7,
  weights = NULL,
  has_intercept = TRUE
)

Arguments

beta

Full coefficient vector (including the intercept, if any).

has_intercept

Whether 'beta[1]' is an unpenalized intercept.


Plot the regularization path and/or selection-criterion curve

Description

Plot the regularization path and/or selection-criterion curve

Usage

## S3 method for class 'pecme_grid'
plot(x, which = c("both", "path", "criterion"), ...)

Arguments

x

A '"pecme_grid"' object.

which

'"path"' (coefficient paths vs 'log(lambda)'), '"criterion"' (the selection criterion vs 'log(lambda)'), or '"both"' (default).

...

Additional graphical arguments. Currently ignored.

Value

Invisibly returns the input 'x', an object of class '"pecme_grid"'.


Predict from a fitted pecme model

Description

Predict from a fitted pecme model

Usage

## S3 method for class 'pecme'
predict(
  object,
  newdata = NULL,
  level = c("population", "group"),
  use_refit = FALSE,
  se_fit = FALSE,
  ...
)

Arguments

object

A '"pecme"' fit.

newdata

Optional data frame with the predictor columns (and, for 'level = "group"', the grouping column) used to fit 'object'. If omitted, fitted values on the training data are returned.

level

'"population"' (fixed effects only, 'X beta') or '"group"' ('X beta + b_j' for known groups; groups absent from the training data effectively use 'b_j = 0', i.e. the population mean).

use_refit

Use the post-selection refit coefficients (see 'object$refit') instead of the penalized estimates, if available.

se_fit

Also return standard errors (only available when 'object$se_valid' is 'TRUE', i.e. an unpenalized/lambda=0 fit; a naive delta-method approximation that ignores 'sigma_b' uncertainty).

...

Additional arguments passed to the prediction method.

Value

If 'se_fit = FALSE', a numeric vector of predictions. If 'se_fit = TRUE', a list with components 'fit' and 'se.fit', containing the predicted values and their estimated standard errors, respectively.


Simulate data from a censored/missing random-intercept linear mixed model

Description

Generates Y^*_{ij} = \beta_0 + x_{ij}'\beta + b_j + e_{ij}, e_{ij}\sim N(0,\sigma^2), b_j\sim N(0,\sigma_b^2), with 'p' standard-normal (optionally correlated) predictors, of which only 'n_active' have a non-zero coefficient – convenient for checking variable-selection performance. Censoring and/or missingness can be injected on top of the latent 'y_star'.

Usage

simulate_pecme_data(
  n_groups = 30L,
  n_per_group = 10L,
  p = 10L,
  n_active = 4L,
  beta_active = 1.5,
  rho = 0,
  sigma = 1,
  sigma_b = 0.7,
  censor_type = c("none", "left", "right", "interval"),
  censor_prob = 0,
  missing_prob = 0,
  seed = NULL
)

Arguments

n_groups

Number of clusters/groups.

n_per_group

Observations per group (recycled if a vector).

p

Number of predictors (in addition to the intercept).

n_active

Number of predictors with a non-zero true coefficient (the rest are exactly zero – useful for checking variable selection).

beta_active

Non-zero coefficient magnitude used for the 'n_active' active predictors (sign alternates).

rho

Pairwise predictor correlation (compound symmetry), '0' for independent predictors.

sigma, sigma_b

Residual and random-intercept standard deviations.

censor_type

'"none"', '"left"', '"right"', or '"interval"' applied to a 'censor_prob' fraction of observations (chosen independently at random); the rest are '"none"'.

censor_prob

Fraction of observations to censor.

missing_prob

Fraction of (non-censored) observations to set completely missing ('NA') in the response.

seed

Optional integer seed for reproducibility. The caller's global RNG state is saved and restored on exit, so passing 'seed' has no side effect on random draws made elsewhere in the calling session (unlike a bare 'set.seed()' call).

Value

A list with 'data' (a data frame with columns 'y', 'x1..xp', 'group', 'censor', 'left', 'right'), 'beta_true' (length 'p + 1', intercept first), 'sigma_true', 'sigma_b_true', and 'y_star' (the uncensored, non-missing latent response, for validation).

Examples

sim <- simulate_pecme_data(n_groups = 20, n_per_group = 8, p = 6,
                            n_active = 3, censor_type = "left",
                            censor_prob = 0.2, seed = 1)
head(sim$data)

Soft-thresholding operator

Description

Soft-thresholding operator

Usage

soft_threshold(z, a)