ETC5521 Tutorial 3

Initial data analysis

Author

Prof. Di Cook

🎯 Objectives

Practice doing data quality checks across multiple related tables, and practice writing up data problems as clear, reproducible GitHub issues.

🔧 Preparation

The reading for this week is The initial examination of data. It is authored by Chris Chatfield, and is a classic paper explaining the role of initial data analysis.

  • Open your RStudio Project for this unit, (the one you created in week 1, ETC5521). Create a .qmd document for this weeks activities.
  • Install the ecotourism package, which we’ll use today:
# install.packages("remotes")
remotes::install_github("vahdatjavad/ecotourism")

📥 Exercises

Today you have a sprint competition to discover as many distinct problems as possible in the ecotourism package’s data.

The ecotourism package bundles together wildlife occurrence records (glowworms, Gouldian finches, manta rays, orchids), the weather stations nearest to where each species is observed, daily weather at those stations, and ABS tourism trip data by region. Because it’s several related tables, rather than one flat data set, some problems will only show up once you try to join tables together, not by skimming any one table alone.

object description
glowworms Glowworm occurrence records, 2014-2024 (lat/lon, date/time, `ws_id` of nearest station)
gouldian_finch Gouldian finch occurrence records, 2014-2024
manta_rays Manta ray occurrence records, 2014-2024
orchids Orchid occurrence records, 2014-2024 (many species)
top_stations The weather station(s) chosen as "nearest" for each organism
weather_stations Weather station metadata: name, location, state (key: `ws_id`)
weather Daily weather by station, 2014-2024 (temp, rain, wind, etc; key: `ws_id` + `date`)
tourism_region SA2 tourism regions, with state, location, and nearest `ws_id` (key: `sa2_code_tra`)
tourism_activity Monthly tourism trips by SA2 region and stopover activity code
tourism_activity_name Lookup table: stopover activity code -> activity name
tourism_reason Monthly tourism trips by SA2 region and stopover reason code
tourism_reason_name Lookup table: stopover reason code -> reason name

The package’s README has a diagram of how these tables relate to each other – look at it before you start.

What to look for

The list from the class slides is a reasonable place to start, although part of the exercise is noticing things nobody told you to look for. Some suggestions are:

  • Missing data: are there columns, or particular rows, with unexpectedly many (or entirely) missing values?
  • Incomplete joins: when you join two related tables on their key column(s), does every row find a match in both directions? If not, is that a genuine data problem, or just a real absence (e.g. a region with no reported trips)?
  • Missing time points: for a time series that should be contiguous (e.g. daily weather at one station), are there any gaps?
  • Name/text mismatches: do the same categories (e.g. states, regions) get written consistently across tables, and within a table?
  • Anything else you notice that looks wrong, inconsistent, or implausible.

How to report what you find

For each distinct problem you find, create a separate GitHub issue on the ecotourism package’s repository. Each issue should include:

  1. A short, specific title (e.g. “tourism_region uses ‘ACT’ while other tables spell out the state name”).
  2. Which table(s) and column(s) are involved.
  3. The R code you used to find it (so it’s reproducible).
  4. Why it matters – what would go wrong if you used the data as-is?

Guidelines

  • Use whatever R package or software or tool you’d like.
  • Feel free to buddy up, and work with another student in your tutorial session.
  • No cheating! This time do the assignment without AI help (except for code syntax), or searching for answers on the web. (Reading the package’s own documentation and README is fine – that’s part of an initial data analysis.)

The most complete list of distinct, genuine issues, each with a clear issue write-up, as decided by your tutor, wins the prize! Your tutor’s decision is final!

Genuine issues confirmed in the data (there may be more than this):

  • Missing values: gouldian_finch is missing month (2 rows) and day (17 rows); manta_rays is missing obs_state (8 rows); weather_stations is missing address/stn_city/stn_state (2 rows); weather has substantial missingness in temp/min/max/dewp/rh/prcp/wind_speed/max_speed.
  • Blank/incomplete rows: tourism_region has 3 rows where state, region, sa2 and sa2_code_tra are all NA (only lon/lat/ws_id are populated).
  • Type mismatch breaking a join: tourism_activity$sa2_code_tra and tourism_reason$sa2_code_tra are integer, while tourism_region$sa2_code_tra is character – a direct join errors until one side is coerced.
  • Incomplete join (real absence, worth flagging): once types are matched, every sa2_code_tra in tourism_activity/tourism_reason matches tourism_region, but 66 of the 2351 distinct regions in tourism_region never appear in tourism_activity at all.
  • Missing contiguous time points: the weather station 956410-99999 (one of the orchid stations) is missing all 365 days of 2022, even though its other years (2014-2024) are complete.
  • Name/text mismatches: tourism_region$state records “ACT” for the Australian Capital Territory, while every other row spells out the full state name (and weather_stations$stn_state also uses “Australian Capital Territory”). Separately, Norfolk Island is recorded under state “New South Wales” / region “Outback NSW”, and the Cocos (Keeling) Islands are recorded under state “Tasmania” – both are external territories, not part of either state.
library(ecotourism)
library(tidyverse)

# missing values
colSums(is.na(tourism_region))
colSums(is.na(gouldian_finch))

# type mismatch on the join key
tourism_activity |> anti_join(tourism_region, by = "sa2_code_tra")  # errors!
tourism_activity |>
  mutate(sa2_code_tra = as.character(sa2_code_tra)) |>
  anti_join(tourism_region, by = "sa2_code_tra")                    # 0 rows once fixed

# missing days in a station's daily weather record
weather |>
  group_by(ws_id) |>
  summarise(n_obs = n(),
            n_expected = as.integer(max(date) - min(date)) + 1L) |>
  filter(n_obs != n_expected)

# inconsistent state labels
tourism_region |> distinct(state)
weather_stations |> distinct(stn_state)

👌 Finishing up

Make sure you say thanks and good-bye to your tutor. This is a time to also report what you enjoyed and what you found difficult.