---
title: "Metafrontier Methods: Theory and Computation"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Metafrontier Methods: Theory and Computation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

```{r setup}
library(metafrontier)
```

This vignette provides a detailed exposition of the metafrontier methods
implemented in the package, linking the econometric theory to the
computational approach used at each step.


## 1. The metafrontier framework

### 1.1 Group-specific stochastic frontiers

Consider $J$ groups of firms, where group $j$ contains $n_j$ firms. For
group $j$, the stochastic frontier model is:

$$\ln y_{ij} = x_{ij}'\beta_j + v_{ij} - u_{ij}, \quad i = 1, \ldots, n_j$$

where $y_{ij}$ is the output of firm $i$ in group $j$, $x_{ij}$ is a
vector of (logged) inputs including a constant, $\beta_j$ is the
group-specific parameter vector, $v_{ij} \sim N(0, \sigma_{v,j}^2)$ is
symmetric noise, and $u_{ij} \ge 0$ is one-sided inefficiency.

The group-specific technical efficiency is:

$$TE_{ij} = \exp(-u_{ij}) \in (0, 1]$$

estimated by default via the Battese and Coelli (1988) conditional
expectation estimator $E[\exp(-u_{ij}) \mid \varepsilon_{ij}]$
(`estimator = "bc88"`). The Jondrow et al. (1982) estimator
$\exp(-E[u_{ij} \mid \varepsilon_{ij}])$ is also computed and stored, and
`efficiencies(fit, estimator = "jlms")` switches between the two without
refitting.


### 1.2 The metafrontier

The metafrontier is defined as a function $f^*(x) = \exp(x'\beta^*)$ such
that:

$$x'\beta^* \ge x'\beta_j \quad \text{for all } x \text{ and all } j$$

That is, the metafrontier weakly dominates all group frontiers. It is
the boundary of the industry **metatechnology** $T^*$: the set of all
input-output combinations that are technically feasible in the industry
(Battese, Rao and O'Donnell, 2004; O'Donnell, Rao and Battese, 2008).
Each group frontier bounds a restricted subset $T_j \subseteq T^*$ of
this metatechnology, where the restrictions reflect regulation, the
physical environment, resource endowments, or the cost of switching
production systems.


### 1.3 The efficiency decomposition

For each firm, efficiency relative to the metafrontier decomposes as:

$$TE^*_{ij} = TE_{ij} \times TGR_{ij}$$

where the **technology gap ratio** is:

$$TGR_{ij} = \frac{\exp(x_{ij}'\beta_j)}{\exp(x_{ij}'\beta^*)} = \exp\left(x_{ij}'(\beta_j - \beta^*)\right) \in (0, 1]$$

A $TGR$ of 1 means the group frontier coincides with the metafrontier at
that input mix; values below 1 indicate a technology gap.


## 2. Deterministic metafrontier (Battese, Rao, and O'Donnell, 2004)

### 2.1 Estimation

After obtaining group estimates $\hat\beta_j$ in Stage 1, the
metafrontier parameters $\hat\beta^*$ are chosen so that the metafrontier
lies as close as possible to the group frontiers while enveloping all of
them. Battese, Rao and O'Donnell (2004) propose two optimisation
criteria, both implemented in the package via the `objective` argument.

**Minimum sum of absolute deviations** (`objective = "lp"`, the default):

$$\min_{\beta^*} \sum_{j=1}^{J} \sum_{i=1}^{n_j} \left|x_{ij}'\beta^* - x_{ij}'\hat\beta_j\right|$$
$$\text{subject to: } x_{ij}'\beta^* \ge x_{ij}'\hat\beta_j \quad \forall\, i, j$$

Because the envelopment constraints force every deviation to be
non-negative, the absolute values can be dropped and the problem reduces
to a linear programme (O'Donnell, Rao and Battese, 2008, Eqs. 23-25).
This LP is the primary presentation in O'Donnell, Rao and Battese (2008)
and is the package default; it is solved with `lpSolveAPI`.

**Minimum sum of squared deviations** (`objective = "qp"`):

$$\min_{\beta^*} \sum_{j=1}^{J} \sum_{i=1}^{n_j} \left(x_{ij}'\beta^* - x_{ij}'\hat\beta_j\right)^2$$

subject to the same envelopment constraints. This is a convex quadratic
programme, solved exactly with `quadprog` when that package is
available, with an adaptive-barrier fallback via `constrOptim()` from
base R otherwise.

The two criteria weight deviations differently (the squared criterion
penalises large gaps between the metafrontier and a group frontier more
heavily), but in practice they typically produce very similar
coefficients.

### 2.2 Properties

- The deterministic metafrontier is a **point estimate** with no associated
  standard errors (since Stage 2 is a deterministic optimisation, not a
  statistical model).
- The enveloping constraints guarantee $TGR_{ij} \le 1$ for all
  observations in the estimation sample.
- The metafrontier coefficients depend on the observed input range; they
  are global only within the sample support.

### 2.3 Example

```{r det-example}
sim <- simulate_metafrontier(
  n_groups = 2, n_per_group = 300,
  tech_gap = c(0, 0.4),
  sigma_u = c(0.2, 0.35),
  seed = 123
)

fit_det <- metafrontier(
  log_y ~ log_x1 + log_x2,
  data = sim$data,
  group = "group",
  meta_type = "deterministic"
)

# Metafrontier coefficients (no standard errors)
coef(fit_det, which = "meta")

# Group coefficients for comparison
coef(fit_det, which = "group")
```

The metafrontier intercept should be at least as large as all group
intercepts:

```{r verify-envelop}
meta_b0 <- coef(fit_det, which = "meta")[1]
group_b0 <- sapply(coef(fit_det, which = "group"), `[`, 1)
meta_b0 >= group_b0
```

To use the minimum sum of squared deviations criterion instead, set
`objective = "qp"`:

```{r det-qp}
fit_qp <- metafrontier(
  log_y ~ log_x1 + log_x2,
  data = sim$data,
  group = "group",
  meta_type = "deterministic",
  objective = "qp"
)

# LP and QP criteria typically give very similar coefficients
cbind(
  LP = coef(fit_det, which = "meta"),
  QP = coef(fit_qp, which = "meta")
)
```


## 3. Stochastic metafrontier (Huang, Huang, and Liu, 2014)

### 3.1 Estimation

Huang, Huang, and Liu (2014) propose treating the technology gap as a
stochastic variable. In Stage 2, the fitted group frontier values become
the dependent variable in a second SFA:

$$\ln \hat{f}(x_{ij}; \hat\beta_j) = x_{ij}'\beta^* + v^*_{ij} - u^*_{ij}$$

where $u^*_{ij} \ge 0$ captures the technology gap and $v^*_{ij}$ is a
noise term. This is estimated via MLE, yielding:

- Point estimates $\hat\beta^*$ with standard errors
- A variance-covariance matrix for inference
- A distributional $\widehat{TGR}$ with associated uncertainty

### 3.2 Advantages over the deterministic approach

1. **Inference**: Standard errors, confidence intervals, and hypothesis
   tests on metafrontier parameters are available.
2. **Robustness**: The noise term $v^*_{ij}$ absorbs sampling variation
   from Stage 1, preventing overfitting.
3. **Consistency**: The metafrontier need not strictly envelop all group
   frontiers in finite samples, which can be more realistic.

### 3.3 Caveat: the generated-regressor problem

The stochastic metafrontier is a two-stage estimator. In Stage 2, the
dependent variable $\ln \hat{f}(x_{ij}; \hat\beta_j)$ is itself an
estimate from Stage 1 -- it is a *generated regressor* (Murphy and
Topel, 1985). The standard errors reported by the package are derived
from the Stage 2 Hessian alone and **do not account for the sampling
uncertainty in the Stage 1 group frontier estimates**.

As a result:

- Standard errors, confidence intervals (`confint()`), and hypothesis
  tests may be **understated** (confidence intervals narrower than their
  nominal coverage warrants).
- This issue does **not** affect point estimates of $\hat\beta^*$ or
  efficiency scores, only inference.
- The bias is negligible when group sample sizes are large relative to
  the number of frontier parameters.
- The Murphy--Topel (1985) correction is available via
  `vcov(fit, correction = "murphy-topel")` and
  `confint(fit, correction = "murphy-topel")`. This adjusts the Stage 2
  variance-covariance matrix to account for Stage 1 estimation
  uncertainty.

### 3.4 Example

```{r sto-example}
fit_sto <- metafrontier(
  log_y ~ log_x1 + log_x2,
  data = sim$data,
  group = "group",
  meta_type = "stochastic"
)

summary(fit_sto)
```

The stochastic metafrontier provides standard errors:

```{r sto-inference}
# Variance-covariance matrix
vcov(fit_sto)

# Log-likelihood of the metafrontier model
logLik(fit_sto)
```

### 3.5 A note on TGR values

Under the stochastic metafrontier, TGR values are not constrained to be
$\le 1$ in finite samples, since the metafrontier need not strictly
envelop all group frontiers. Values slightly above 1 can occur and are
consistent with the stochastic framework.

```{r tgr-range}
tgr_vals <- efficiencies(fit_sto, type = "tgr")
summary(tgr_vals)
```


## 4. DEA-based metafrontier

### 4.1 Approach

For a nonparametric metafrontier:

1. Compute group-specific DEA efficiencies $\hat\theta_{ij}^{group}$ using
   only observations from group $j$.
2. Compute pooled DEA efficiencies $\hat\theta_{ij}^{pool}$ using all
   observations.
3. The TGR is: $TGR_{ij} = \hat\theta_{ij}^{pool} / \hat\theta_{ij}^{group}$.

The package solves the DEA linear programs using `lpSolveAPI`.

### 4.2 Returns to scale

The `rts` argument controls the technology assumption:

- `"crs"` (constant returns to scale): the standard CCR model
- `"vrs"` (variable returns to scale): the BCC model
- `"drs"` / `"irs"` (decreasing / increasing returns)

```{r dea-example}
# CRS metafrontier
fit_crs <- metafrontier(
  log_y ~ log_x1 + log_x2,
  data = sim$data,
  group = "group",
  method = "dea",
  rts = "crs"
)

# VRS metafrontier
fit_vrs <- metafrontier(
  log_y ~ log_x1 + log_x2,
  data = sim$data,
  group = "group",
  method = "dea",
  rts = "vrs"
)

# Compare mean TGR
cbind(
  CRS = tapply(fit_crs$tgr, fit_crs$group_vec, mean),
  VRS = tapply(fit_vrs$tgr, fit_vrs$group_vec, mean)
)
```


## 5. Comparing methods

The choice between deterministic, stochastic, and DEA metafrontiers
involves trade-offs:

| Feature | Deterministic SFA | Stochastic SFA | DEA |
|---|---|---|---|
| Functional form | Parametric | Parametric | Nonparametric |
| Noise handling | Stage 1 only | Both stages | None |
| Inference on TGR | No | Yes | No |
| TGR $\le$ 1 guaranteed | Yes | No | Yes |
| Small sample performance | Moderate | Moderate | Poor |
| References | BRO (2004) | HHL (2014) | ORB (2008) |

```{r compare-methods}
# Compare TGR estimates across methods
tgr_det <- tapply(fit_det$tgr, fit_det$group_vec, mean)
tgr_sto <- tapply(fit_sto$tgr, fit_sto$group_vec, mean)
tgr_dea <- tapply(fit_crs$tgr, fit_crs$group_vec, mean)
true_tgr <- tapply(sim$data$true_tgr, sim$data$group, mean)

comparison <- data.frame(
  True = true_tgr,
  Deterministic = tgr_det,
  Stochastic = tgr_sto,
  DEA_CRS = tgr_dea
)
round(comparison, 4)
```


## 6. Choosing a method: practical guidance

Selecting between deterministic SFA, stochastic SFA, and DEA
metafrontiers depends on the research question, data characteristics,
and inferential requirements.

**Use the deterministic SFA metafrontier (BRO 2004) when:**

- You need guaranteed $TGR \le 1$ (the metafrontier strictly envelops
  all group frontiers).
- Inference on metafrontier parameters is not required.
- The goal is descriptive decomposition of efficiency into within-group
  and between-group components.

**Use the stochastic SFA metafrontier (HHL 2014) when:**

- You need standard errors, confidence intervals, or hypothesis tests
  on the metafrontier parameters.
- You want a distributional framework for the technology gap ratio.
- Sample sizes per group are moderate to large (at least 50--100
  observations per group is recommended).
- You are comfortable with the generated-regressor caveat (Section 3.3).

**Use the DEA metafrontier when:**

- You prefer a nonparametric approach with no functional form
  assumptions.
- Multiple inputs and/or multiple outputs are involved.
- Sample sizes are large enough to support DEA (a rough guideline:
  $n \ge 3 \times (m + s)$ per group, where $m$ is the number of
  inputs and $s$ the number of outputs).
- The returns-to-scale assumption (`rts`) is well-justified by the
  application context.

In many applied studies, it is informative to estimate multiple methods
and compare TGR estimates for robustness (as shown in Section 5).


## 7. Testing for technology heterogeneity

Before estimating a metafrontier, it is useful to test whether separate
group frontiers are actually needed. The **poolability test** uses a
likelihood ratio statistic:

$$LR = -2\left[LL_{pooled} - \sum_{j=1}^{J} LL_j\right] \sim \chi^2_{df}$$

where $LL_{pooled}$ is the log-likelihood of a single frontier estimated
on the pooled sample and $LL_j$ are the group-specific log-likelihoods.

```{r poolability}
poolability_test(fit_det)
```

A significant test (p < 0.05) confirms that the group frontiers differ,
that is, the groups face different restrictions of the common
metatechnology, and the metafrontier decomposition is warranted.


## 8. Simulation for Monte Carlo studies

The `simulate_metafrontier()` function generates data from a known DGP,
enabling parameter recovery studies:

```{r monte-carlo, eval=FALSE}
# Monte Carlo: check parameter recovery over 100 replications
set.seed(1)
n_rep <- 100
beta_hat <- matrix(NA, n_rep, 3)

for (r in seq_len(n_rep)) {
  sim_r <- simulate_metafrontier(
    n_groups = 2, n_per_group = 200,
    tech_gap = c(0, 0.3),
    sigma_u = c(0.2, 0.3),
    sigma_v = 0.15
  )
  fit_r <- metafrontier(
    log_y ~ log_x1 + log_x2,
    data = sim_r$data,
    group = "group",
    meta_type = "deterministic"
  )
  beta_hat[r, ] <- coef(fit_r, which = "meta")
}

# Bias
true_beta <- c(1.0, 0.5, 0.3)
colMeans(beta_hat) - true_beta
```

The `simulate_metafrontier()` function supports:

- Arbitrary number of groups (`n_groups`)
- Unequal group sizes (`n_per_group` as a vector)
- Custom metafrontier coefficients (`beta_meta`)
- Group-specific technology gaps (`tech_gap`)
- Group-specific inefficiency dispersion (`sigma_u`)
- Reproducible results via `seed`


## References

- Battese, G.E. and Coelli, T.J. (1988). Prediction of firm-level
  technical efficiencies with a generalized frontier production function
  and panel data. *Journal of Econometrics*, 38(3), 387--399.

- Battese, G.E., Rao, D.S.P. and O'Donnell, C.J. (2004). A metafrontier
  production function for estimation of technical efficiencies and
  technology gaps for firms operating under different technologies.
  *Journal of Productivity Analysis*, 21(1), 91--103.

- Huang, C.J., Huang, T.-H. and Liu, N.-H. (2014). A new approach to
  estimating the metafrontier production function based on a stochastic
  frontier framework. *Journal of Productivity Analysis*, 42(3), 241--254.

- Jondrow, J., Lovell, C.A.K., Materov, I.S. and Schmidt, P. (1982). On
  the estimation of technical inefficiency in the stochastic frontier
  production function model. *Journal of Econometrics*, 19(2--3), 233--238.

- O'Donnell, C.J., Rao, D.S.P. and Battese, G.E. (2008). Metafrontier
  frameworks for the study of firm-level efficiencies and technology
  ratios. *Empirical Economics*, 34(2), 231--255.
