ETC5521 Worksheet Week 3

Data validation with pointblank

Author

Prof. Di Cook

🎯 Objectives

Practice setting up rule-based, automated data validation checks, using the pointblank package, applied to the World Development Indicators (WDI) data from the week 3 lecture. This should take about 30 minutes.

install.packages(c("tidyverse", "pointblank", "janitor"))

📦 Data

This is the same tidying used in lecture – you don’t need to change anything here, just run it to get wdi (long, one row per country/year/series) and wdi2016 (a single cross-section).

library(janitor)

raw_dat <- read_csv(here::here("data/world-development-indicators.csv"),
                    na = "..", n_max = 11935)

wdi <- raw_dat |>
  select(`Country Code`, `Series Code`, `1969 [YR1969]`:`2018 [YR2018]`) |>
  rename_all(make_clean_names) |>
  pivot_longer(x1969_yr1969:x2018_yr2018,
               names_to = "year",
               values_to = "value") |>
  mutate(year = as.numeric(str_sub(year, 2, 5))) |>
  pivot_wider(names_from = series_code, values_from = value)

wdi2016 <- wdi |> filter(year == 2016)

Three series we’ll use today:

SP.DYN.LE00.IN: life expectancy at birth, total (years)
SP.POP.TOTL: population, total
EN.ATM.CO2E.PC: CO2 emissions (metric tons per capita) – note this series stops being reported after 2014, so it will be entirely missing in wdi2016.

🧩 Tasks

1. Skim the pointblank introduction. In your own words, what is an `agent`, and why is `interrogate()` a separate step from adding the validation checks?

Solution

An agent is an object that holds a plan of validation checks (a set of rules) for a particular table, built up with create_agent() and then piped through validation functions like col_vals_not_null(). Nothing is actually checked against the data until you call interrogate(), which runs every rule and records the results. Separating the two means you can build up and re-use a validation plan (e.g. save it, apply it to next month’s data refresh) without re-running it every time you add a step.

2. Create a pointblank `agent` for `wdi2016`, and add two structural checks: `country_code` is never missing (`col_vals_not_null`), and `country_code` is unique (one row per country) (`rows_distinct`). `interrogate()` and check whether `all_passed()`.

Solution

Code

library(pointblank)

agent <- create_agent(
  tbl = wdi2016,
  tbl_name = "wdi2016",
  label = "WDI 2016 cross-section checks"
) |>
  col_vals_not_null(columns = vars(country_code)) |>
  rows_distinct(columns = vars(country_code)) |>
  interrogate()

all_passed(agent)

[1] TRUE

Both checks pass: every row has a country_code, and there’s exactly one row per country in this cross-section.

3. Now add two plausibility checks: life expectancy (`SP.DYN.LE00.IN`) should be between 0 and 100 years (`col_vals_between`), and population (`SP.POP.TOTL`) should not be negative (`col_vals_gte`). `interrogate()` again. Does the console output say every step is “OK”? Does `all_passed()` agree? If not, why not? (use `get_agent_report` here)

Solution

Code

agent <- create_agent(
  tbl = wdi2016,
  tbl_name = "wdi2016",
  label = "WDI 2016 cross-section checks"
) |>
  col_vals_not_null(columns = vars(country_code)) |>
  rows_distinct(columns = vars(country_code)) |>
  col_vals_between(columns = vars(SP.DYN.LE00.IN), left = 0, right = 100) |>
  col_vals_gte(columns = vars(SP.POP.TOTL), value = 0) |>
  interrogate()

agent

		STEP	COLUMNS	VALUES	EVAL	UNITS	PASS	FAIL	W	S	N	EXT
Pointblank Validation
WDI 2016 cross-section checks tibble wdi2016
	1	`col_vals_not_null()`	`▮country_code`	—	✓	`217`	`217` `1`	`0` `0`	—	—	—	—
	2	`rows_distinct()`	`▮country_code`	—	✓	`217`	`217` `1`	`0` `0`	—	—	—	—
	3	`col_vals_between()`	`▮SP.DYN.LE00.IN`	`[0, 100]`	✓	`217`	`199` `0.91705`	`18` `0.08295`	—	—	—
	4	`col_vals_gte()`	`▮SP.POP.TOTL`	`0`	✓	`217`	`216` `0.99`	`1` `0.01`	—	—	—
2026-08-17 11:52:03 AEST < 1 s 2026-08-17 11:52:03 AEST

Code

all_passed(agent)

[1] FALSE

The printed summary shows a ✔ “OK” for every step, but all_passed() returns FALSE. That’s because the console summary only flags steps that breach a warn/stop threshold (none are set here), while all_passed() asks whether every single row passed. Looking at the detailed report shows why:

Code

get_agent_report(agent, display_table = FALSE) |>
  select(i, columns, units, n_pass, f_pass)

# A tibble: 4 × 5
      i columns        units n_pass f_pass
  <int> <chr>          <dbl>  <dbl>  <dbl>
1     1 country_code     217    217  1    
2     2 country_code     217    217  1    
3     3 SP.DYN.LE00.IN   217    199  0.917
4     4 SP.POP.TOTL      217    216  0.995

The life expectancy check only passes for 91.7% of rows (18 of 217 countries have a missing value in 2016), and the population check passes for 99.5% (1 country missing). By default, pointblank counts a missing value as a failing unit for col_vals_between()/col_vals_gte(), since it can’t confirm the value satisfies the rule.

You can find which countries these are:

Code

wdi2016 |> filter(is.na(SP.DYN.LE00.IN)) |> select(country_code)

# A tibble: 18 × 1
   country_code
   <chr>       
 1 ASM         
 2 AND         
 3 VGB         
 4 CYM         
 5 DMA         
 6 GIB         
 7 GRL         
 8 IMN         
 9 MHL         
10 MCO         
11 NRU         
12 MNP         
13 PLW         
14 SMR         
15 SXM         
16 KNA         
17 TCA         
18 TUV

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with `na_pass = TRUE`, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

Solution

Code

agent <- create_agent(
  tbl = wdi2016,
  tbl_name = "wdi2016",
  label = "WDI 2016 cross-section checks"
) |>
  col_vals_not_null(columns = vars(country_code)) |>
  rows_distinct(columns = vars(country_code)) |>
  col_vals_between(columns = vars(SP.DYN.LE00.IN), left = 0, right = 100, na_pass = TRUE) |>
  col_vals_gte(columns = vars(SP.POP.TOTL), value = 0, na_pass = TRUE) |>
  interrogate()

all_passed(agent)

[1] TRUE

na_pass = TRUE tells pointblank to excuse NA values from the check – a missing value is neither counted as a pass nor a fail, it’s simply skipped, so it can no longer drag the pass rate down. Now all_passed() is TRUE.

Use na_pass = TRUE when missingness is already known about and accepted (e.g. you documented it during IDA, and it doesn’t itself indicate a data quality problem for this particular check). Leave it FALSE (the default) when you want validation to catch missingness – for example, if a future data refresh should always be complete, and any new NA appearing would be a red flag worth investigating.

5. (Bonus, if time permits) Add a rule for `EN.ATM.CO2E.PC` (CO2 emissions per capita should not be negative) to the agent, using `na_pass = TRUE`, and interrogate. Does it pass? Is that reassuring?

Solution

Code

agent <- create_agent(tbl = wdi2016, tbl_name = "wdi2016") |>
  col_vals_gte(columns = vars(EN.ATM.CO2E.PC), value = 0, na_pass = TRUE) |>
  interrogate()

get_agent_report(agent, display_table = FALSE) |>
  select(i, columns, units, n_pass, f_pass)

# A tibble: 1 × 5
      i columns        units n_pass f_pass
  <int> <chr>          <dbl>  <dbl>  <dbl>
1     1 EN.ATM.CO2E.PC   217    217      1

It passes 100% – but not reassuringly so. EN.ATM.CO2E.PC stops being reported after 2014, so every value in wdi2016 is NA, and with na_pass = TRUE all of them are excused rather than checked. A validation step that always “passes” because there’s nothing left to check isn’t actually telling you anything useful; it’s worth checking the coverage of a rule (how many non-missing values it actually evaluated), not just whether it passed.

🎯 Objectives

📦 Data

🧩 Tasks

1. Skim the pointblank introduction. In your own words, what is an agent, and why is interrogate() a separate step from adding the validation checks?

2. Create a pointblank agent for wdi2016, and add two structural checks: country_code is never missing (col_vals_not_null), and country_code is unique (one row per country) (rows_distinct). interrogate() and check whether all_passed().

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with na_pass = TRUE, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

5. (Bonus, if time permits) Add a rule for EN.ATM.CO2E.PC (CO2 emissions per capita should not be negative) to the agent, using na_pass = TRUE, and interrogate. Does it pass? Is that reassuring?

1. Skim the pointblank introduction. In your own words, what is an `agent`, and why is `interrogate()` a separate step from adding the validation checks?

2. Create a pointblank `agent` for `wdi2016`, and add two structural checks: `country_code` is never missing (`col_vals_not_null`), and `country_code` is unique (one row per country) (`rows_distinct`). `interrogate()` and check whether `all_passed()`.

4. We already know from the lecture that this data has missingness, and haven’t decided yet whether to treat it as a problem here. Re-run the same two plausibility checks with `na_pass = TRUE`, and explain what that argument changes – and when you would, or wouldn’t, want to use it.

5. (Bonus, if time permits) Add a rule for `EN.ATM.CO2E.PC` (CO2 emissions per capita should not be negative) to the agent, using `na_pass = TRUE`, and interrogate. Does it pass? Is that reassuring?