--- title: "Vaccination Data with SI-PNI" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Vaccination Data with SI-PNI} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", eval = FALSE ) ``` ## Overview The **SI-PNI (Sistema de Informacao do Programa Nacional de Imunizacoes)** is Brazil's national immunization information system, managed by the Ministry of Health. It tracks vaccination doses applied and coverage rates across the country. SI-PNI data come in two eras: | Era | Years | Data type | Granularity | |-----|-------|-----------|-------------| | **Aggregated** | 1994--2019 | Dose counts (DPNI) and coverage (CPNI) | Annual per UF | | **Microdata** | 2020+ | Individual-level (one row per dose) | Monthly | `sipni_data()` automatically routes to the correct era based on the requested year. ## Data sources: the R2 mirror and DATASUS By default, `healthbR` reads SI-PNI data from the [healthbr-data](https://github.com/SidneyBissoli/healthbr-data) mirror: hive-partitioned Parquet on Cloudflare R2 (free egress), with values **byte-identical** to the Ministry's files and full provenance metadata (source URL, hash, download date, pipeline version) embedded in every file. If the mirror is unreachable, `sipni_data()` falls back automatically to the official DATASUS/OpenDataSUS sources. ```{r} # default: R2 mirror with automatic DATASUS fallback sipni_data(year = 2024, uf = "AC", month = 1) # pin a single source (no fallback) sipni_data(year = 2019, uf = "AC", source = "datasus") # invert the priority (DATASUS first, R2 as fallback) sipni_data(year = 2019, uf = "AC", source = c("datasus", "r2")) ``` Why the mirror is the default: - The Ministry decommissioned the old OpenDataSUS host in 2026 and removed the 2020--2025 microdata files from the new one; **only the mirror holds the complete 2020+ series**. - Only the requested UF/month partitions are transferred (no ~1.4 GB national CSV download to extract one UF). - Aggregates read in seconds (the DATASUS FTP is slow and unstable for large UFs). The result records where the data actually came from: ```{r} data <- sipni_data(year = 2024, uf = "AC", month = 1) attr(data, "healthbr_source") #> microdata #> "r2" attr(data, "healthbr_provenance") #> # A tibble: 1 x 7 (partition, processing timestamp, Ministry source URL...) ``` ### Checking availability with sipni_status() The mirror publishes a `manifest.json` per dataset recording, for every partition, the Ministry source file, its hash, the processing timestamp and record counts. `sipni_status()` reads it: ```{r} # everything the mirror holds sipni_status() # which 2026 microdata months are published so far? sipni_status("microdados") |> filter(year == 2026) ``` ### Check available years ```{r} sipni_years() #> [1] 1994 1995 ... 2025 2026 ``` ### Module information ```{r} sipni_info() ``` ## Aggregated data: doses applied (DPNI) The default type returns aggregated dose counts (1994--2019): ```{r} # doses applied in Acre, 2019 ac_doses <- sipni_data(year = 2019, uf = "AC") ac_doses ``` ### Key variables (DPNI) | Variable | Description | |----------|-------------| | ANO | Reference year | | UF | UF code (IBGE 2 digits) | | MUNIC | Municipality code (IBGE 6 digits) | | IMUNO | Immunobiological code | | DOSE | Dose type (1st, 2nd, booster, etc.) | | QT_DOSE | Number of doses applied | | FX_ETARIA | Age group (coded) | ### Using the dictionary By default the dictionary is read from the mirror — the full versions converted from the Ministry's original .cnv/.dbf files. **Important — decoding data requires `lookup = TRUE`.** In the .cnv files, `code` is a sequential category code, and the value(s) actually found in the data columns live in `source_codes` (possibly several per label, reflecting code changes over the years — e.g. data codes `08` **and** `82` both mean Hepatite B). Joining data against `code` silently produces wrong labels. `lookup = TRUE` expands the dictionary into one row per **data code**, ready to join: ```{r} # published form (code = .cnv category, source_codes = data codes) sipni_dictionary("IMUNO") # join-ready lookup: one row per data code sipni_dictionary("IMUNO", lookup = TRUE) # dose types and age groups sipni_dictionary("DOSE", lookup = TRUE) sipni_dictionary("FX_ETARIA", lookup = TRUE) # the built-in offline copy (already in data-code form) sipni_dictionary("IMUNO", source = "datasus") ``` When more than one .cnv category claims the same data code, the more specific claim wins: explicitly listed codes take precedence over codes that only fall inside a range. This matters for residual catch-all categories: FX_ETARIA's "Idade ignorada" spans codes `00-99`, but it only labels codes that no specific age group claimed explicitly. ## Aggregated data: vaccination coverage (CPNI) The CPNI type provides coverage rates per municipality: ```{r} # vaccination coverage in Acre, 2019 ac_coverage <- sipni_data(year = 2019, type = "CPNI", uf = "AC") ac_coverage ``` ### Key variables (CPNI) | Variable | Description | |----------|-------------| | ANO | Reference year | | UF | UF code (IBGE 2 digits) | | MUNIC | Municipality code (IBGE 6 digits) | | IMUNO | Immunobiological code | | QT_DOSE | Number of doses applied | | POP | Target population | | COBERT | Vaccination coverage (%) | ## Microdata (2020+) For years 2020 and later, SI-PNI provides individual-level microdata (one row per vaccination dose). The `type` parameter is ignored for these years: ```{r} # microdata for Acre, January 2024 ac_micro <- sipni_data(year = 2024, uf = "AC", month = 1) ac_micro ``` **Column names differ by source.** The mirror publishes the Ministry's JSON exports (56 fields, no CSV serialization artifacts); the OpenDataSUS CSVs use different names (~47 fields). Each source returns its columns exactly as published — healthbR does not rename or remap them: | R2 mirror (default) | DATASUS CSV | |---------------------|-------------| | `dt_vacina` | `data_vacina` | | `ds_vacina` | `descricao_vacina` | | `tp_sexo_paciente` | `tipo_sexo_paciente` | | `nu_idade_paciente` | `numero_idade_paciente` | | `sg_uf_estabelecimento` | `sigla_uf_estabelecimento` | ### Key variables (R2 microdata) | Variable | Description | |----------|-------------| | sg_uf_estabelecimento | UF of the health facility | | co_municipio_estabelecimento | Municipality (IBGE) | | tp_sexo_paciente | Sex (M/F) | | nu_idade_paciente | Patient age | | no_raca_cor_paciente | Race/color (descriptive) | | ds_vacina | Vaccine name | | ds_dose_vacina | Dose description | | dt_vacina | Vaccination date | ### Exploring variables ```{r} # DPNI variables sipni_variables() # CPNI variables sipni_variables(type = "CPNI") # microdata variables, R2 mirror (default; 56 fields) sipni_variables(type = "API") # microdata variables, OpenDataSUS CSV (~47 fields) sipni_variables(type = "API", source = "datasus") # search sipni_variables(search = "dose") ``` ## Month parameter for microdata Each month is a separate partition. Use `month` to select specific months: ```{r} # single month jan <- sipni_data(year = 2024, uf = "AC", month = 1) # first quarter q1 <- sipni_data(year = 2024, uf = "AC", month = 1:3) # all 12 months (default) full_year <- sipni_data(year = 2024, uf = "AC") ``` For aggregated data (1994--2019), the `month` parameter is ignored because the files are annual. ## Example: vaccine doses by immunobiological ```{r} ac_2019 <- sipni_data(year = 2019, uf = "AC") # decode immunobiological names: lookup = TRUE gives data-code rows imuno_labels <- sipni_dictionary("IMUNO", lookup = TRUE) |> select(code, label) doses_by_vaccine <- ac_2019 |> group_by(IMUNO) |> summarize(total_doses = sum(as.integer(QT_DOSE), na.rm = TRUE), .groups = "drop") |> left_join(imuno_labels, by = c("IMUNO" = "code")) |> arrange(desc(total_doses)) doses_by_vaccine ``` Note: the 2019 aggregated data are drastically incomplete *at the source* (the Ministry was migrating to the new SI-PNI that year); totals are a fraction of 2018's. This is faithful to what DATASUS publishes. ## Example: coverage trends over time ```{r} # coverage data for Sao Paulo, 2015-2019 sp_cov <- sipni_data( year = 2015:2019, type = "CPNI", uf = "SP" ) # average coverage by year sp_cov |> group_by(year) |> summarize( mean_coverage = mean(as.numeric(COBERT), na.rm = TRUE), .groups = "drop" ) ``` ## Example: individual-level analysis (2020+) ```{r} # vaccinations in Acre, January 2024 ac_jan <- sipni_data(year = 2024, uf = "AC", month = 1) # vaccines administered ac_jan |> count(ds_vacina, sort = TRUE) # doses by sex ac_jan |> count(tp_sexo_paciente) # age distribution ac_jan |> mutate(age = as.integer(nu_idade_paciente)) |> filter(!is.na(age)) |> mutate(age_group = cut(age, breaks = c(0, 5, 12, 18, 30, 60, Inf), right = FALSE)) |> count(age_group) ``` ## Mixed year requests When requesting years that span both eras (e.g., 2019 and 2024), `sipni_data()` fetches each era and combines the results. Note that column names and structure differ between eras: ```{r} # aggregated (2019) + microdata (2024) mixed <- sipni_data(year = c(2019, 2024), uf = "AC", month = 1) # aggregated (UPPERCASE) and microdata columns are combined # with NAs where columns don't overlap names(mixed) ``` ## Lazy evaluation over the mirror With `lazy = TRUE` and the default source, `sipni_data()` returns the **remote** arrow dataset — dplyr verbs are pushed down to R2 and only the partitions your query touches are transferred. In this mode the partition columns keep the bucket layout names (`ano`, `mes`, `uf`, as strings): ```{r} ds <- sipni_data(year = 2019, uf = "AC", lazy = TRUE) ds |> filter(IMUNO == "02") |> # data code 02 = BCG (see sipni_dictionary) select(MUNIC, DOSE, QT_DOSE) |> collect() ``` ## Download tips - Aggregates (1994--2019) read from the mirror in seconds. - Microdata months for large UFs (SP, MG) hold millions of rows — start with a single month and UF, or use `lazy = TRUE` and aggregate before `collect()`. - Results are cached locally per partition: a second request for the same year/month/UF is served from disk. - If you pin `source = "datasus"` for 2020+, only the current year's CSVs exist at the source, and each month is a ~1.4 GB national download. ## Smart type parsing ```{r} # parsed types (default) ac <- sipni_data(year = 2019, uf = "AC") class(ac$QT_DOSE) # integer # raw character columns, exactly as published ac_raw <- sipni_data(year = 2019, uf = "AC", parse = FALSE) ``` ## Cache management Downloaded data is cached locally for faster future access: ```{r} # check cache status sipni_cache_status() # clear cache if needed sipni_clear_cache() ``` ## Additional resources - [healthbr-data](https://github.com/SidneyBissoli/healthbr-data) — the mirror's pipelines, reproducibility policy and dataset documentation - OpenDataSUS (`dadosabertos.saude.gov.br`) - [Census vignette](censo-denominadores.html) for population denominators