---
title: "Interpreting ReproStat Outputs"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Interpreting ReproStat Outputs}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(ReproStat)
set.seed(20260324)
```

## Why interpretation matters

ReproStat does not just return a single score. It returns a collection of
stability summaries, each reflecting a different way a model result can vary
under perturbation.

This article explains how to read those outputs in a way that is useful for
real analysis work.

## The question ReproStat answers

The package asks:

> If I perturb the observed data in reasonable ways and refit the same model
> many times, how much do the key outputs move?

That is different from:

- classical uncertainty around one fitted model
- external replication across studies
- causal identification
- model adequacy in a broader scientific sense

ReproStat is focused on **stability of fitted outputs under repeated data
perturbation**.

## Running a diagnostic object

```{r fit-example}
diag_obj <- run_diagnostics(
  mpg ~ wt + hp + disp,
  data = mtcars,
  B = 150,
  method = "bootstrap"
)
```

Everything else in the package builds from this object.

## Coefficient stability

```{r coef-stability}
coef_stability(diag_obj)
```

This measures how much estimated coefficients vary across perturbation runs.

Interpretation:

- lower values indicate more stable coefficient estimates
- large values suggest the estimated effect size is sensitive to the data
- coefficients can be unstable even when the average fitted model looks strong

Use this when you care about the **magnitude** of effects, not only whether
they are statistically significant.

## P-value stability

```{r pvalue-stability}
pvalue_stability(diag_obj)
```

This reports the proportion of perturbation runs in which each coefficient is
significant at the chosen `alpha`.

Interpretation:

- values near `1` mean the term is almost always significant
- values near `0` mean the term is almost never significant
- values near `0.5` mean the significance decision is unstable

This is especially helpful when a predictor looks "significant" in the base
fit but may be borderline under small perturbations.

## Selection stability

```{r selection-stability}
selection_stability(diag_obj)
```

For standard regression backends, this reflects sign consistency across
perturbations. For `glmnet`, it reflects non-zero selection frequency.

Interpretation:

- high values suggest the direction or inclusion pattern is stable
- low values suggest the predictor is not behaving consistently

This is often the most intuitive measure when the practical question is:
"Does this variable keep showing up in the same way?"

## Prediction stability

```{r prediction-stability}
prediction_stability(diag_obj)
```

Prediction stability summarizes how much the model's predictions vary across
perturbation runs.

Interpretation:

- lower mean prediction variance indicates more stable predictive behavior
- unstable predictions can occur even when some coefficient summaries look fine

This is useful when the model will be used for scoring, ranking, or decision
support rather than only interpretation.

## The Reproducibility Index

```{r ri}
reproducibility_index(diag_obj)
```

The Reproducibility Index aggregates multiple stability dimensions into a
single 0-100 score.

A practical reading guide is:

- `90-100`: highly stable under the chosen perturbation scheme
- `70-89`: reasonably stable overall
- `50-69`: mixed stability; inspect components carefully
- `< 50`: low stability; the model is sensitive in ways worth investigating

These are not universal cutoffs. They are interpretive anchors.

## Do not read the RI alone

Two models can have similar RI values for different reasons:

- one may have stable coefficients but unstable predictions
- another may have stable predictions but unstable significance decisions

For that reason, the RI should be treated as a summary, not a replacement for
the component-level diagnostics.

## Confidence intervals for the RI

```{r ri-ci}
ri_confidence_interval(diag_obj, R = 300, seed = 1)
```

This interval reflects uncertainty in the RI induced by the stored perturbation
draws. It is useful when you want to know whether the reported RI is itself
stable or highly variable.

## Choosing a perturbation method

Different perturbation schemes answer different questions.

### Bootstrap

Use `"bootstrap"` when you want a broad sense of ordinary sampling
variability.

### Subsample

Use `"subsample"` when you want to know whether the result depends heavily on
which observations happen to be included.

### Noise

Use `"noise"` when you want to stress test sensitivity to small measurement
error or recording noise in numeric predictors.

## When a low score is useful

A low RI is not a failure of the package. It is a result.

It can tell you that:

- the model is over-sensitive to the observed data
- the effect structure is weak or unstable
- a simpler or more robust specification may be preferable
- the analysis should be reported with more caution

## What to report in practice

For applied work, a compact reporting pattern is:

1. describe the perturbation method and `B`
2. report the RI and its confidence interval
3. mention which components were most and least stable
4. include at least one stability plot
5. note any sensitivity that affects substantive conclusions

## Next steps

After reading this article, the most useful follow-up pages are:

- `vignette("ReproStat-intro")` for the basic workflow
- the backend guide for model-family differences
- the workflow patterns article for practical usage designs
