ETC5521 Worksheet Week 3

Data validation with pointblank

Author

Prof. Di Cook

🎯 Objectives

Practice setting up rule-based, automated data validation checks, using the pointblank package, applied to the World Development Indicators (WDI) data from the week 3 lecture. This should take about 30 minutes.

install.packages(c("tidyverse", "pointblank", "janitor"))

📦 Data

This is the same tidying used in lecture – you don’t need to change anything here, just run it to get wdi (long, one row per country/year/series) and wdi2016 (a single cross-section).

library(janitor)

raw_dat <- read_csv(here::here("data/world-development-indicators.csv"),
                    na = "..", n_max = 11935)

wdi <- raw_dat |>
  select(`Country Code`, `Series Code`, `1969 [YR1969]`:`2018 [YR2018]`) |>
  rename_all(make_clean_names) |>
  pivot_longer(x1969_yr1969:x2018_yr2018,
               names_to = "year",
               values_to = "value") |>
  mutate(year = as.numeric(str_sub(year, 2, 5))) |>
  pivot_wider(names_from = series_code, values_from = value)

wdi2016 <- wdi |> filter(year == 2016)

Three series we’ll use today:

SP.DYN.LE00.IN: life expectancy at birth, total (years)
SP.POP.TOTL: population, total
EN.ATM.CO2E.PC: CO2 emissions (metric tons per capita) – note this series stops being reported after 2014, so it will be entirely missing in wdi2016.

🧩 Tasks

1. Skim the pointblank introduction. In your own words, what is an `agent`, and why is `interrogate()` a separate step from adding the validation checks?

2. Create a pointblank `agent` for `wdi2016`, and add two structural checks: `country_code` is never missing (`col_vals_not_null`), and `country_code` is unique (one row per country) (`rows_distinct`). `interrogate()` and check whether `all_passed()`.

3. Now add two plausibility checks: life expectancy (`SP.DYN.LE00.IN`) should be between 0 and 100 years (`col_vals_between`), and population (`SP.POP.TOTL`) should not be negative (`col_vals_gte`). `interrogate()` again. Does the console output say every step is “OK”? Does `all_passed()` agree? If not, why not? (use `get_agent_report` here)

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with `na_pass = TRUE`, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

5. (Bonus, if time permits) Add a rule for `EN.ATM.CO2E.PC` (CO2 emissions per capita should not be negative) to the agent, using `na_pass = TRUE`, and interrogate. Does it pass? Is that reassuring?

🎯 Objectives

📦 Data

🧩 Tasks

1. Skim the pointblank introduction. In your own words, what is an agent, and why is interrogate() a separate step from adding the validation checks?

2. Create a pointblank agent for wdi2016, and add two structural checks: country_code is never missing (col_vals_not_null), and country_code is unique (one row per country) (rows_distinct). interrogate() and check whether all_passed().

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with na_pass = TRUE, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

5. (Bonus, if time permits) Add a rule for EN.ATM.CO2E.PC (CO2 emissions per capita should not be negative) to the agent, using na_pass = TRUE, and interrogate. Does it pass? Is that reassuring?

1. Skim the pointblank introduction. In your own words, what is an `agent`, and why is `interrogate()` a separate step from adding the validation checks?

2. Create a pointblank `agent` for `wdi2016`, and add two structural checks: `country_code` is never missing (`col_vals_not_null`), and `country_code` is unique (one row per country) (`rows_distinct`). `interrogate()` and check whether `all_passed()`.

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with `na_pass = TRUE`, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

5. (Bonus, if time permits) Add a rule for `EN.ATM.CO2E.PC` (CO2 emissions per capita should not be negative) to the agent, using `na_pass = TRUE`, and interrogate. Does it pass? Is that reassuring?