ETC5521 Diving Deeper into Data Exploration: Assignment 2

As per Monash’s integrity rules, these solutions are not to be shared beyond this class.

🎯 Goal

The assignment is designed to practice using initial data analysis to prepare data for exploration, and conduct randomisation appropriate tests for significance of patterns. The assignment represents 20% of your final grade for ETC5521. This is an individual assignment.

📌 Guidelines

1. Accept the GitHub Classroom Assignment provided in Moodle using a GitHub Classroom compatible web browser. This should generate a private GitHub repository that can be found at https://github.com/etc5521-2025. Your GitHub assignment 2 repo should contain the file assign02.html, README.md, assign02-submission.qmd, assignment.css, etc5521-assignment2.Rproj, .gitignore, and data files generated from your work needed to render your solution file. Code should be available but folded in report.

2. Answer each question in the assign02-submission.qmd in the repo.

3. For the final submission knit assign02-submission.qmd which will contain your answers. Make sure to provide the link to the script of any Generative AI conversation you employed in arriving at your solution. Note that marks are allocated for overall grammar and structure of your final report.

4. Leave all of your files in your GitHub repo for marking. We will check your git commit history. You should have contributions to the repo with consistent commits over time. (Note: nothing needs to be submitted to Moodle.)

5. You are expected to develop your solutions by yourself, without discussing any details with other class members or other friends or contacts. You can ask for clarifications from the teaching team and we encourage you to attend consultations to get assistance as needed. As a Monash student you are expected to adhere to Monash’s academic integrity policy. and the details on use of Generative AI as detailed on this unit’s Moodle assessment overview.

6. We expect that this assignment takes about 10-12 hours of time to complete. You should work on this analysis steadily over the time period between release and due date. Spend a couple of hours soon after the assignment is released getting started, and several hours each of the following two weeks to refine your analysis. Your GitHub commit history should reflect this working pattern.

Deadlines:

Due date	Turn in
11:45pm Mon Aug 25	Assignment 2 Repo on GitHub has been created
11:45pm Mon Sep 1	Final solutions available on repo

🛠️ Exercises

The file yarra_wq.xls contains water quality measurements from Victorian Department of Energy, Environment, and Climate Action. Metadata on the stations is provided in SW Metadata.csv.

Your boss has received a complaint from the Rakali Organisation that water quality in the Yarra River has declined. They have tasked you with preparing the data to answer the question “Has water quality declined from the 1990s to recent years?”

Question 1: Conduct an IDA

Check the data for what can be used to prepare it to answer this question, and do the data processing that is needed. This might include:

• Reading and checking types of variables,
• Converting dates into date, month, weekday, hour
• Checking which sites have sufficient measurements
• Checking which parameters (water quality variables like pH, nitrogen, …) have sufficient measurements

You should write out a cleaned data set, called yarra_wq_clean.csv into the data directory.

Solution

Load the libraries, read the data, and make the temporal variables.

library(tidyverse)
library(readxl)
library(lubridate)

d <- read_xls("data/yarra_wq/yarra_wq.xls",
              col_types = c("text", 
                            "text",
                            "date",
                            "text",
                            "numeric", 
                            "text",
                            "numeric",
                            "text",
                            "text",
                            "text"))

# Make time variables
d <- d |>
  mutate(date = date(Datetime),
         year = year(Datetime),
         month = month(Datetime, label = TRUE),
         wday = wday(Datetime, label = TRUE, 
                     week_start = 1),
         hour = hour(Datetime))

Check the dates for each of the sensors. We find only YARRA @ CHANDLER HWY has data ranging over the period needed to answer the primary question. (Note: YARRA @ WARRANDYTE shows incorrect values here because dates are in the 1800s.)

# Initial checks on data
d |> count(Name, sort=TRUE)

# A tibble: 5 × 2
  Name                     n
  <chr>                <int>
1 YARRA DS MCMAHONS     6573
2 YARRA @ CHANDLER HWY  3509
3 YARRA @ WARRANDYTE    2271
4 YARRA @ YERING GORGE  2083
5 YARRA @ TEMPLESTOWE   1947

d |> group_by(Name) |> summarise(mn = min(year, na.rm=TRUE),
                                 mx = max(year, na.rm=TRUE))

# A tibble: 5 × 3
  Name                    mn    mx
  <chr>                <dbl> <dbl>
1 YARRA @ CHANDLER HWY  1975  2025
2 YARRA @ TEMPLESTOWE   1975  1982
3 YARRA @ WARRANDYTE     Inf  -Inf
4 YARRA @ YERING GORGE  1974  1982
5 YARRA DS MCMAHONS     1962  1982

Subset to just this sensor, and then keep checking.

d_ch <- d |>
  filter(Name == "YARRA @ CHANDLER HWY")

Now check number of measurements for each variable recorded. Looks like lots of missings.

d_ch |> count(Parameter, sort=TRUE) |> print(n=25)

# A tibble: 23 × 2
   Parameter                            n
   <chr>                            <int>
 1 Streamflow (mean daily)           1890
 2 Salinity as EC@25                  225
 3 Water Temperature                  225
 4 pH                                 225
 5 Turbidity                          221
 6 Nitrogen as Nitrite (NO3)           53
 7 Nitrogen as Total                   53
 8 Phosphorus as TRP                   53
 9 Total Arsenic                       53
10 Total Cadmium                       53
11 Total Chromium                      53
12 Total Copper                        53
13 Total Nickel                        53
14 Total Zinc                          53
15 Dissolved Oxygen (DO) (field)       46
16 Nitrogen as Total Kjeldahl (TKN)    34
17 Phosphorus (Total)                  34
18 Colour (True Filtered) (PCU)        32
19 Nitrogen as NOx                     32
20 Phosphorus as FRP                   32
21 Total Suspended Solids (TSS)        32
22 Nitrogen as Ammonia (Total)          2
23 Nitrogen as Nitrate (NO2)            2

Convert to wide form, to check again. Missings values are explicitly recorded as NA.

d_ch_w <- d_ch |>
  group_by(Name, date, Parameter) |>
  summarise(Value = mean(Value, na.rm=TRUE), 
            .groups="drop") |>
  select(Name, Parameter, Value, date) |>
  pivot_wider(names_from = Parameter, 
              values_from = Value, 
              values_fill = NA) |>
  mutate(year = year(date),
         month = month(date, label = TRUE),
         wday = wday(date, label = TRUE, 
                     week_start = 1))

library(GGally)
library(patchwork)
p1 <- ggduo(d_ch_w, columnsX=26, columnsY=3:8)
p2 <- ggduo(d_ch_w, columnsX=26, columnsY=9:14)
p3 <- ggduo(d_ch_w, columnsX=26, columnsY=15:20)
p4 <- ggduo(d_ch_w, columnsX=26, columnsY=20:25)

Figure 1: Check of first 6 water quality measurements.

Figure 2: Check of third 6 water quality measurements.

Figure 3: Check of second 6 water quality measurements.

Figure 4: Check of last 6 water quality measurements.

d_clean <- d_ch_w |>
  mutate(period = as.factor(if_else(year < 2000, "1990s", "2020s"))) |>
  filter(year > 1989) |>
  select(Name, period, date, year, month, wday,
         `Salinity as EC@25`, Turbidity,
         `Water Temperature`, pH) |>
  rename(salinity = `Salinity as EC@25`,
         turbidity = Turbidity,
         temperature = `Water Temperature`)

write_csv(d_clean, file="data/yarra_wq_clean.csv")

Only four variables have sufficient data in the 1990s and 2020s to study the difference: Salinity, Turbidity, Water Temperature and pH.

Question 2: EDA

Make some numerical and graphical summaries to address the main question. These might include side-by-side plots of the distributions of different parameters, and means/standard deviations.

Write some explanatory sentences for each plot and table made, describing what is learned.

Solution

Read the data and take a quick look.

library(ggbeeswarm)
d <- read_csv("data/yarra_wq_clean.csv",
              col_types = "cfDiffdddd") |>
  mutate(wday = factor(wday, 
    levels = c("Mon", "Tue", "Wed", "Thu", 
               "Fri", "Sat", "Sun")))
d |>
  pivot_longer(salinity:pH, names_to = "variable", 
               values_to = "value") |>
  ggplot(aes(x=period, y=value)) +
    geom_quasirandom() +
    facet_wrap(~variable, ncol=2, scales = "free_y")

Figure 5: Examining the distribution of each variable across the two periods

Figure 5 shows dotplots of the four water chemistry variables shown by 1990s and 2020s. There looks to be some differences on a couple. But there are a few problems to fix:

• three missing values that are dropped when plotting
• several 0s on pH that have to be incorrect
• an extreme value on salinity, which might be suspicious
• turbidity needs to be transformed
• temperature in the 2020s is bimodal

We need to also check that the same months are represented in both periods, so that we have comparable sets of data. It might also be a good idea to check that the week days are relatively equally represented.

summary(d[,6:10])

  wday        salinity      turbidity      temperature  
 Mon: 13   Min.   :   0   Min.   :  0.0   Min.   : 7.3  
 Tue: 24   1st Qu.: 150   1st Qu.: 18.0   1st Qu.:12.1  
 Wed:111   Median : 180   Median : 24.0   Median :17.0  
 Thu: 28   Mean   : 192   Mean   : 40.3   Mean   :16.6  
 Fri: 17   3rd Qu.: 220   3rd Qu.: 41.0   3rd Qu.:21.2  
 Sat:  1   Max.   :1300   Max.   :340.0   Max.   :26.0  
 Sun:  1                  NA's   :3                     
       pH      
 Min.   :0.00  
 1st Qu.:7.00  
 Median :7.30  
 Mean   :7.23  
 3rd Qu.:7.60  
 Max.   :9.30  
               

d_tidied <- d |>
  mutate(pH = if_else(pH < 5, NA, pH),
         salinity = if_else(salinity > 800, NA, salinity),
         salinity = if_else(salinity < 5, NA, salinity),
         turbidity = if_else(turbidity < 5, NA, turbidity)) |>
  mutate(turbidity = log10(turbidity))

d_tidied |>
  pivot_longer(salinity:pH, names_to = "variable", 
               values_to = "value") |>
  ggplot(aes(x=period, y=value)) +
    geom_quasirandom() +
    facet_wrap(~variable, ncol=2, scales = "free_y")

Figure 6: Examining the distribution of each variable across the two periods, on tidied data.

• The three missings are all on turbidity.
• Zeros on pH, salinity and turbidity converted to missing
• Extreme salinity value converted to missing - it might be a correct value, but it is going to affect all the analysis, so for now we remove it from the analysis. We might investigate this value in more detail later.
• A log10 transformation was done on turbidity.

Now check that other information matches.

mth_chk <- d_tidied |>
  pivot_longer(salinity:pH, names_to = "variable", 
               values_to = "value") |>
  count(variable, period, month) |>
  pivot_wider(names_from = month, values_from = n) |>
  print (n=50)

# A tibble: 8 × 14
  variable  period   Jan   Feb   Mar   Apr   May   Jun   Jul
  <chr>     <fct>  <int> <int> <int> <int> <int> <int> <int>
1 pH        1990s      4     4     4     4     4     4     4
2 pH        2020s     21    17    14     8     7    10     9
3 salinity  1990s      4     4     4     4     4     4     4
4 salinity  2020s     21    17    14     8     7    10     9
5 temperat… 1990s      4     4     4     4     4     4     4
6 temperat… 2020s     21    17    14     8     7    10     9
7 turbidity 1990s      4     4     4     4     4     4     4
8 turbidity 2020s     21    17    14     8     7    10     9
# ℹ 5 more variables: Aug <int>, Sep <int>, Oct <int>,
#   Nov <int>, Dec <int>

There is a relatively similar distribution of counts over months for each of the two time periods and each of the variables. It should be reasonable to compare them.

Now we are ready to make some statements about differences.

pH

ggplot(d_tidied, aes(x=period, y=pH)) + 
  geom_quasirandom() +
  stat_summary(colour="red") +
  xlab("")

d_tidied |>
  group_by(period) |>
  summarise(m = mean(pH, na.rm=TRUE),
            s = sd(pH, na.rm=TRUE)) 

# A tibble: 2 × 3
  period     m     s
  <fct>  <dbl> <dbl>
1 1990s   6.87 0.442
2 2020s   7.43 0.460

Figure 7: A deeper look at difference in distribution of pH over the two times. The red dot and line indicate the means and standard errors.

Figure 7 shows that pH is higher on average in the 2020s. This means that the water has become more alkaline, on average. We note that the discreteness of pH measurements is more prominent in the 2020s but this is due to more observations making it more visible.

salinity

ggplot(d_tidied, aes(x=period, y=salinity)) + 
  geom_quasirandom() +
  stat_summary(colour="red") +
  xlab("")

d_tidied |>
  group_by(period) |>
  summarise(m = mean(salinity, na.rm=TRUE),
            s = sd(salinity, na.rm=TRUE)) 

# A tibble: 2 × 3
  period     m     s
  <fct>  <dbl> <dbl>
1 1990s   197.  58.4
2 2020s   187.  57.7

Figure 8: A deeper look at difference in distribution of salinity over the two times. The red dot and line indicate the means and standard errors.

Figure 8 suggests that salinity hasn’t changed over this time.

temperature

ggplot(d_tidied, aes(x=period, y=temperature)) + 
  geom_quasirandom() +
  stat_summary(colour="red") +
  xlab("")

d_tidied |>
  group_by(period) |>
  summarise(m = mean(temperature, na.rm=TRUE),
            s = sd(temperature, na.rm=TRUE)) 

# A tibble: 2 × 3
  period     m     s
  <fct>  <dbl> <dbl>
1 1990s   15.2  4.52
2 2020s   17.1  5.15

Figure 9: A deeper look at difference in distribution of temperature over the two times. The red dot and line indicate the means and standard errors.

Figure 9 suggests water temperature may have increased a little. But the bimodality of the 2020s might be interesting to check. This could be related to different times of the year.

ggplot(d_tidied, aes(x=month, y=temperature, colour=period)) + 
  geom_quasirandom(alpha=0.5) +
  stat_summary() +
  xlab("") +
  scale_colour_brewer("", palette = "Dark2")

Figure 10: A deeper look at difference in distribution of temperature over the two times, by month.

This is interesting! The differences in temperature are occurring in the summer months: Jan, Feb, Mar, Nov, Dec (Figure 10).

turbidity

ggplot(d_tidied, aes(x=period, y=turbidity)) + 
  geom_quasirandom() +
  stat_summary(colour="red") +
  xlab("")

d_tidied |>
  group_by(period) |>
  summarise(m = mean(turbidity, na.rm=TRUE),
            s = sd(turbidity, na.rm=TRUE)) 

# A tibble: 2 × 3
  period     m     s
  <fct>  <dbl> <dbl>
1 1990s   1.58 0.300
2 2020s   1.44 0.290

Figure 11: A deeper look at difference in distribution of turbidity over the two times. The red dot and line indicate the means and standard errors.

Figure 11 suggests that turbidity is slightly lower in the 2020s, which means the water may be getting clearer.

pairwise relationships

ggscatmat(d_tidied, columns = 6:10, color = "period", alpha=0.5) +
  scale_colour_brewer("", palette = "Dark2") +
  theme(aspect.ratio = 1)

Figure 12: Pairwise relationships between the variables, coloured by period.

ggplot(d_tidied, aes(x=temperature, y=pH, colour=period)) +
  geom_point(alpha=0.5) +
  facet_wrap(~month, ncol=4, scales="free") +
  scale_colour_brewer("", palette = "Dark2") +
  theme(aspect.ratio = 1,
        axis.text = element_blank())

Figure 13: Examining temperature and pH by month, coloured by period.

There is nothing to see in the pairwise plots (Figure 12, Figure 13). While based on correlation it might seem like there are some associations between variables in the two periods, it is very weak.

Question 3: Check your findings

For three of the plots you have produced, write down what was the null hypothesis being tested. This needs to be based on your ggplot code, not on what was in the plot of the data.

Generate the appropriate lineup, to check whether your summary statements from the previous part were reasonable or not.

Solution

We’ll check the conclusions about pH, about temperature difference being greater in the summer, and the pairwise relationships.

library(nullabor)

set.seed(303)
ggplot(lineup(null_permute("period"), d_tidied), 
       aes(x=period, y=pH)) + 
  geom_quasirandom() +
  stat_summary(colour="red") +
  facet_wrap(~.sample, ncol=5) +
  theme(axis.title = element_blank(),
        axis.text = element_blank())

Figure 14: Lineup plot of pH by period.

Because the test is whether there is a difference in the period, the null samples are computed by permuting the period variable.

The data plot (18) is clearly visible in this lineup. If we wanted to test it we would show this plot to about 10 friends, and use pvisual() to compute a \(p\)-value.

set.seed(308)
ggplot(lineup(null_permute("period"), d_tidied, n=6), 
       aes(x=month, y=temperature, colour=period)) + 
  geom_quasirandom(alpha=0.1) +
  stat_summary() +
  xlab("") +
  scale_colour_brewer("", palette = "Dark2") +
  facet_wrap(~.sample, ncol=2) +
  theme(axis.title = element_blank(),
        axis.text = element_blank(),
        legend.position = "none")

Figure 15: Lineup plot of temperature by period over months.

Because the test is whether there is a difference in the period, the null samples are computed by permuting the period variable.

Some choices in making the lineups are:

• Only made 5 null samples, to produce lineups of size 6, because reading these plots is complicated.
• Focus the plots on the means because trying to compare the whole distribution is even more complicated.
• If we want to have more comparisons we could make several different lineups to show to more people.

The data plot (5) is reasonably visible in the lineups which suggests the warmer summer temperature is a plausible conclusion.

set.seed(320)
ggplot(lineup(null_permute("pH"), d_tidied), 
       aes(x=pH, y=temperature)) + 
  geom_point(alpha=0.8) +
  facet_wrap(~.sample, ncol=5) +
  theme(aspect.ratio=1,
        axis.title = element_blank(),
        axis.text = element_blank())

Figure 16: Lineup plot of pairwise relationships.

Because we are interested in assessing the pairwise relationships, ignoring difference in period, we would permute one of the two variables to generate null samples. This breaks association between them.

It’s going to be easier to do this for each pair of variables. Figure 16 shows the lineup for pH and temperature.

We think it is very difficult to detect the data plot (17) in this lineup, which suggests the conclusion that there is no relationship between the variables is reasonable.

Generative AI analysis

In this part, we would like you to actively discuss how generative AI helped with your answers to the assignment questions, and where or how it was mistaken or misleading.

You need to provide a link that makes the full script of your conversation with any generative AI tool accessible to the teaching staff. You should not use a paid service, as the freely available systems will sufficiently helpful.

Marks

Part	Points
Q1 worth	5
Q2 worth	7
Q3 worth	5
GitHub Repo	3
Generative AI Analysis	-3
Reproducibility, Formatting, Spelling & Grammar	-5

Note that the negative marks for “Generative AI Analysis”, “Formatting, Spelling & Grammar” correspond to reductions in scores. You can lose up to 3 marks for poor use of the GAI. For example, no use, basic questions only, no link to the script, and no acknowledgment but clearly used. You can lose up to 5 marks if your report is not reproducible, for poorly formatted and written answers. Three marks will be reserved for appropriate GitHub work, accepting the assignment in a timely fashion and consistent and substantive commits.

Here are some guidelines for code:

• Split chunks that process data and ones that plot data
• It is dangerous to run scraping code or downloading data each render because web sites often change. Always best to scrape once or download once and save the result for the rest of the analysis. (This is general advice not needed for this assignment.)
• Label the chunks - helps to locate errors
• Comment the code so you can remember what each is supposed to do when you come back to it.
• Learn how to name things well, see https://github.com/jennybc/how-to-name-files

Rubric

To help you complete in your report, below is a rubric to guide you to what we are expecting:

content	Excellent (HD)	Very good (D)	Good (C)	Satisfactory (P)	Unsatisfactory (F)
Q1	Saved data is clean, and formatted in a way that the question can be addressed. Limitations of the cleaned data are detailed. Steps taken during the data processing are explained.	Saved data is clean, and formatted in a way that the question can be addressed. Limitations of the cleaned data are mostly detailed. Steps taken during the data processing are provided.	Saved data is clean, and formatted in a way that the question can be reasonably addressed. Limitations of the cleaned data are mentioned. Steps taken during the data processing are provided.	Saved data is clean, and formatted in a way that the question can be reasonably addressed. Steps taken during the data processing are provided.	Saved data is clean, and formatted in a way that the question cannot addressed. Glaring errors in the processing that leave substantial missing data, or incorrect values. Steps taken during the data processing are not clearly provided.
Q2	Plots and summaries are comprehensive, well-designed, polished and concise, and clearly designed to address the primary question. Text is used to summarise the tables and plots. Anything interesting or problematic with the data is discussed.	Plots and summaries are comprehensive, well-designed, and concise, and clearly designed to address the primary question. Text is used to summarise the tables and plots. Anything interesting or problematic with the data is discussed.	Plots and summaries are mostly appropriate, and address the primary question. Text is used to summarise the tables and plots. Interesting or problematic aspects of the data are mentioned.	Plots and summaries are reasonable, and mostly address the primary question. Text is used to summarise the tables and plots. Interesting findings are provided.	Plots and summaries are dont address the primary question. Text is inadequate to describe the tables and plots. No summary of interesting findings.
Q3	Null hypotheses correctly provided, and explained. Appropriate null generating methods used, and explained. A variety of plots are chosen. Findings clearly articulated.	Null hypotheses correctly provided, and explained. Appropriate null generating methods used, and explained. Findings clearly articulated.	Null hypotheses mostly correct, and explained. Appropriate null generating methods used, and explained. Findings documented.	At least one null hypotheses is correct, and explained. At least one appropriate null generating method used, and explained. Findings summarised.	All null hypotheses are incorrect. Inappropriate null generating method used. Findings not summarised.
Repo	Actively commiting during entire assignment period, with informative commit messages, done after each small change to the work. Repo accepted in a timely manner.	Actively commiting during entire assignment period, with informative commit messages, done after changes to the work. Repo accepted in a timely manner.	Actively commiting during entire assignment period, done after changes to the work. Repo accepted in a timely manner.	Multiple commits made, at least after each part completed, with informative commit messages.	Repo not accepted in time, and just a few single commits.
GAI	GAI used effectively, deeply, and explained, script linked to report (0 deduction)	NA	GAI used but not explained well or inadequate and script linked to report (-0.5)	Shallow use of GAI, script linked to report (-1)	Clearly used but no script linked to report (-3)
Reproducibility	No changes needed for report to reproduce exactly as provided. Code is nicely formatted, commented and readable using appropriate tidyverse standards. (0 deduction)	Single change needed for report to reproduce exactly as provided. Code is nicely formatted, commented and readable using appropriate tidyverse standards. (0 deduction)	Just a few changes needed for report to reproduce exactly as provided. Code is formatted, commented and readable using appropriate tidyverse standards. (-0.5)	Multiple changes needed for report to reproduce exactly as provided. Code is readable. (-1)	Cannot easily make changes for report to reproduce at all. Code is not readable. (-3)
Spelling/Grammar	Writing style is exceptional, scholarly and succinct that is free from spelling, grammar and punctuation errors. (0 deduction)	Writing style is scholarly, free from spelling, grammar and punctuation errors. (0 deduction)	Writing style is scholarly, but wordy and inconcise. Free from spelling, grammar and punctuation errors. (-0.5)	Writing is scholarly and wordy. Contains some grammatical, punctuation and spelling errors. (-1)	Writing is unscholarly. Many grammatical, punctuation and spelling errors. (-2)
References	The appropriate referencing style has been used consistently, with no errors. Includes citations for software used, and data sources. (0 deduction)	The appropriate referencing style has been used consistently, with very few errors, and includes software used, and data sources. (0 deduction)	The appropriate referencing style has been used consistently, and only a few citations missing. (-0.5)	The appropriate referencing style has been used much of the time, missing some major sources that were clearly used. (-1)	Material used from external sources without citation. (-2)