---
title: "Introduction to datacaged"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to datacaged}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment  = "#>",
  eval     = FALSE
)
```

## Overview

The `datacaged` package simplifies access to **CAGED** microdata (Cadastro Geral de Empregados e Desempregados) directly from HuggingFace, loading data into a local **DuckDB** database for efficient analysis.

It supports three series:

| Period               | Series          | Database table                         |
|----------------------|-----------------|-----------------------------------------|
| Jan/2020 – present      | Novo CAGED      | `caged_mov`, `caged_for`, `caged_exc`   |
| Jan/1992 – Dec/2019  | Legacy CAGED    | `caged_antigo`                          |
| Jan/1992 – Dec/2019  | CAGED Adjustments| `caged_ajustes`                         |

## Installation

```{r install}
# Via remotes
remotes::install_github("gecomt/datacaged")
```

## Platform compatibility

The `datacaged` package is compatible with **Windows, macOS and Linux** with no extra configuration for Novo CAGED (2020+).

| Recurso | Windows | macOS | Linux |
|---|---|---|---|
| Download (HTTPS) | OK | OK | OK |
| Downloads paralelos | OK | OK | OK |
| Novo CAGED (2020+) | OK | OK | OK |
| Legacy CAGED (pre-2020, PPMd) | (!) requires 7-Zip | (!) requires 7-Zip | (!) requires 7-Zip |

**Legacy CAGED** uses PPMd compression. If you need this data, install 7-Zip:

- **Windows**: [baixe o instalador](https://www.7-zip.org/download.html) e instale normalmente
- **macOS**: `brew install 7-zip`  
- **Linux**: `sudo apt install 7zip` ou `sudo dnf install 7zip`

Novo CAGED (2020+) uses LZMA, natively supported by the `archive` package on all platforms.

## Parallel downloads

By default, the package downloads 3 files simultaneously (MOV, FOR and EXC for each month), resulting in approximately **3× faster** downloads compared to sequential mode.

```{r workers}
# Control the number of workers
caged_download(years = 2023, months = 1:3, workers = 3)  # padrão

# Set globally for the entire session
options(datacaged.workers = 4)

# Sequential mode (useful for unstable connections)
caged_download(years = 2023, months = 1, workers = 1)
```

## Basic usage: Full pipeline

The `caged_load()` function does everything in a single command: downloads `.7z` files from HuggingFace, extracts, normalises and writes to DuckDB.

```{r load-basico}
library(datacaged)

# Download Novo CAGED Jan–Dec/2023
# Novo CAGED: national file, `states` does not filter
caged_load(
  years    = 2023,
  months   = seq_len(12L),
  db_path = "caged.duckdb"
)
```

Progress is displayed in the terminal with a progress bar and final summary.

## Querying the data

After populating the database, connect and query with `dplyr` or plain SQL:

```{r consulta-dplyr}
library(dplyr)

con <- caged_connect("caged.duckdb")

# Monthly employment balance in 2023
saldo_mensal <- tbl(con, "caged_mov") |>
  group_by(competenciamov) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  arrange(competenciamov) |>
  collect()

saldo_mensal
```

```{r consulta-sql}
# Or with direct SQL
DBI::dbGetQuery(con, "
  SELECT
    competenciamov,
    uf,
    SUM(saldomovimentacao) AS saldo,
    AVG(salario)           AS salario_medio,
    COUNT(*)               AS movimentacoes
  FROM caged_mov
  WHERE uf = 35          -- Sao Paulo
  GROUP BY competenciamov, uf
  ORDER BY competenciamov
")
```

Always close the connection when done:

```{r desconectar}
DBI::dbDisconnect(con, shutdown = TRUE)
```

## Granular functions

For more control, use the functions individually:

### 1. Download files only

```{r download}
# Download and save to local cache (~/.local/share/R/datacaged por padrão)
manifest <- caged_download(
  years    = 2023,
  months   = c(1L, 2L, 3L),
  destdir = "~/meus_dados/caged_cache"
)

# manifest is a data.frame with the status of each file
dplyr::count(manifest, status)
```

### 2. Parse files manually

```{r parse}
# One file at a time
df <- caged_parse("~/meus_dados/caged_cache/caged_mov/2023/CAGEDMOV202301.7z")
glimpse(df)

# Several at once
arquivos <- list.files(
  "~/meus_dados/caged_cache/NOVO_CAGED/2023",
  pattern    = "CAGEDMOV",
  full.names = TRUE
)
df_todos <- caged_parse_batch(arquivos)
```

### 3. Write to database

```{r gravar}
caged_to_duckdb(df_todos, db_path = "caged.duckdb")
```

## Inspect the database

```{r info}
caged_info("caged.duckdb")
#> ── caged.duckdb ────────────────────────────────────────
#> Tamanho do arquivo: 142.3 MB
#> ── Tabelas ──────────────────────────────────────────────
#> * "caged_mov"   Registros: 3,665,155
#> * "caged_for"      Registros:    91,098
#> * "caged_exc"       Registros:     7,900
#>   Registros   : 4.823.901
#>   Competências: 202301 – 202312
```

## Example: Historical series with legacy CAGED

```{r historico}
# Baixa Legacy CAGED para Nordeste (2015–2019)
nordeste <- c("MA", "PI", "CE", "RN", "PB", "PE", "AL", "SE", "BA")

caged_load(
  years    = 2015:2019,
  db_path = "caged_historico.duckdb"
)

con <- caged_connect("caged_historico.duckdb")

# Evolução anual do saldo formal no Nordeste
tbl(con, "caged_antigo") |>
  mutate(ano = as.integer(substr(as.character(competencia), 1, 4))) |>
  group_by(ano, uf) |>
  summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
  collect() |>
  tidyr::pivot_wider(names_from = uf, values_from = saldo)

DBI::dbDisconnect(con, shutdown = TRUE)
```


## CAGED Adjustments

CAGED Adjustments contain retroactive corrections to legacy CAGED records (up to 2019).
Use `caged_adjustments_load()` to download and write to the `caged_ajustes` table.

```{r ajustes}
# Baixar ajustes de 2019
caged_adjustments_load(years = 2019, months = seq_len(12L), db_path = "caged.duckdb")

# Listar o que está disponível no HuggingFace
caged_hf_files(type = "ajustes")

# Comparar saldo original vs ajustado
con <- caged_connect("caged.duckdb")

antigo  <- dplyr::tbl(con, "caged_antigo")  |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_original = sum(saldomovimentacao, na.rm = TRUE))

ajustes <- dplyr::tbl(con, "caged_ajustes") |>
  dplyr::group_by(competencia) |>
  dplyr::summarise(saldo_ajuste = sum(saldomovimentacao, na.rm = TRUE))

dplyr::full_join(antigo, ajustes, by = "competencia") |>
  dplyr::mutate(saldo_final = saldo_original + saldo_ajuste) |>
  dplyr::collect()

DBI::dbDisconnect(con, shutdown = TRUE)
```

## Performance tips

- **DuckDB is columnar**: prefer `select()` before `collect()` to bring only the necessary columns.
- **Re-runs are safe**: `caged_load()` skips competencies already in the database by default. Use `overwrite_competencies = TRUE` to overwrite.
- **`.7z` cache**: downloaded files are stored in `~/.local/share/R/datacaged`. They can be reused without a new HuggingFace connection.
- **Memory**: the pipeline processes one competency at a time to avoid excessive RAM usage with large periods.


## Utilities

```{r utilitarios}
# Verificar se o HuggingFace está online antes de baixar
caged_status()

# Listar competências disponíveis no HuggingFace
caged_hf_files()                     # Novo CAGED (últimos 12 meses)
caged_hf_files(type = "antigo")      # Legacy CAGED
caged_hf_files(type = "ajustes")     # CAGED Adjustments

# Atualização incremental — baixa apenas o que ainda não está no banco
caged_update(db_path = "caged.duckdb")
caged_update(db_path = "caged.duckdb", series = c("novo", "antigo"))

# Exportar tabelas para Parquet (nativo DuckDB, muito rápido)
caged_to_parquet("caged.duckdb", output_dir = "~/exports")
caged_to_parquet("caged.duckdb", output_dir = "~/exports",
                 tables = "caged_mov", partition_by = "uf")
```

## Main variables

| Column | Description |
|---|---|
| `competenciamov` | Competency in Novo CAGED, `YYYYMM` format (ex: `202301`) |
| `competencia` | Competency in legacy CAGED and Adjustments, format `AAAAMM` |
| `uf` | IBGE state code (ex: `35` = SP) |
| `municipio` | IBGE municipality code |
| `saldomovimentacao` | `+1` hire, `-1` dismissal |
| `salario` | Contracted wage in BRL |
| `sexo` | `1` male, `3` female |
| `idade` | Age in years |
| `escolaridade` | Education level code (1–9) |
| `racacor` | Race/colour code (1–5) |
| `tipomovimentacao` | Reason for movement code |
| `secao` | CNAE 2.0 section (Novo CAGED) |
| `fonte_tipo` | `MOV`, `FOR`, `EXC`, `ANTIGO` or `AJUSTES` |