Research-scale validation execution

Purpose

eyeprocess separates an executable model from evidence that the model is scientifically dependable. The validation execution engine converts a declared Monte Carlo design into deterministic jobs, atomic checkpoints, resumable runs, auditable failures, recovery summaries, calibration diagnostics, and promotion decisions.

Deterministic plans

library(eyeprocess)

plan <- validation_job_plan(
  grid = list(
    n_person = c(50L, 150L, 500L),
    n_item = c(10L, 30L),
    process_effect = c(0, 0.25, 0.50),
    feature_reliability = c(0.50, 0.80),
    missingness = c(0, 0.15)
  ),
  replications = 500L,
  base_seed = 20260805L,
  model_family = "dynamic_irtree",
  chunk_size = 25L
)

write_validation_job_manifest(plan, "validation/dynamic-irtree")

A job seed is determined by the complete design cell, replication, and base seed. Reordering a plan therefore does not alter the simulated study.

Atomic execution and resumption

run_validation_jobs(
  plan,
  simulator = simulate_one_study,
  fitter = fit_one_model,
  extractor = extract_estimates,
  truth_extractor = extract_truth,
  diagnostics_extractor = extract_diagnostics,
  draws_extractor = extract_draws,
  output_dir = "validation/dynamic-irtree",
  workers = 8L,
  backend = "future",
  isolation = "callr",
  timeout_seconds = 3600,
  memory_limit_mb = 8192
)

resume_validation_jobs(
  plan,
  "validation/dynamic-irtree",
  retry = c("missing", "failed", "nonconverged")
)

Every checkpoint preserves the job specification, seed, warnings, messages, errors, runtime, estimates, diagnostics, optional posterior draws, predictions, and session metadata. Failed jobs are evidence and are never silently removed.

Collection and evidence

result <- collect_validation_jobs("validation/dynamic-irtree", plan)
validation_recovery_summary(result)
validation_failure_summary(result)
validation_runtime_summary(result)
validation_calibration_summary(result)
validation_sbc_summary(result)

audit <- audit_validation_completion(result)
plot_parameter_recovery(result)
plot_interval_coverage(result)
plot_sbc_rank(result)
plot_validation_failures(result)
plot_validation_runtime(result)
write_validation_release_report(result, "validation-report.md")

Promotion remains gated

evidence <- list(
  dynamic_irtree = list(
    completion = audit,
    sbc = sbc_audit,
    misspecification = misspecification_audit,
    grouped_validation = grouped_result,
    engine_equivalence = equivalence_result,
    empirical_reproduction = reproduction_result,
    preprocessing_sensitivity = aoi_sensitivity
  )
)

audit_model_promotion(evidence)

The audit reports experimental whenever any required gate is absent or fails. Code execution alone is not a promotion criterion.