```{r, include = FALSE}
source("../setup.R")
```


## Birth of EDA {background-image="images/tukey_cover.png" background-size="40%" background-position="95% 15%"}

::: {.column width=50%}

The field of exploratory data analysis came of age when this book appeared in 1977. 

<br>

[*Tukey held that too much emphasis in statistics was placed on statistical hypothesis testing (confirmatory data analysis); more emphasis needed to be placed on using data to suggest hypotheses to test.*]{.monash-blue}

:::


## John W. Tukey

:::: {.columns}
::: {.column width=50%}

<br>
<center>
<img src="images/John_Tukey.jpg" style="width: 400px; border-radius: 50%"> 
</center>

<br><br>
[Image source: [wikimedia.org](https://upload.wikimedia.org/wikipedia/en/e/e9/John_Tukey.jpg)]{.smallest}
::: 

::: {.column width=50%}
- Born in 1915, in New Bedford, Massachusetts.
- Mum was a private tutor who home-schooled John. Dad was a Latin teacher. 
- BA and MSc in Chemistry, and PhD in Mathematics
- Awarded the National Medal of Science in 1973, by President Nixon
- By some reports, his home-schooling was unorthodox and contributed to his thinking and working differently. 

::: 
::::

## Taking a glimpse back in time 

is possible with the [American Statistical Association video lending library](https://community.amstat.org/jointscsg-section/media/videos). 

<br>
*We're going to watch John Tukey talking about exploring high-dimensional data with an amazing new computer in 1973, four years before the EDA book.*  
<br>

[Look out for these things:]{.monash-blue2}

Tukey's expertise is described as [for trial and error learning]{.monash-orange2} and the [computing equipment]{.monash-orange2}.

<center>



[First 4.25 minutes]{.smallest}
</center>

##

:::: {.columns}

::: {.column width=50%}

<center>
<img src="images/tukey_cover.png" style="width: 400px;"> 
</center>
::: 

::: {.column width=50%}

<img src="images/pencil_and_paper.png" width="80%">

::: 
::::

## Setting the frame of mind 

*Excerpt from the introduction*

::: {.hscroll}
This book is based on an important principle. 

<br>
**It is important to understand  what you CAN DO before you learn to  measure how WELL you seem to have DONE it.** 

<br>
Learning first what you can do will  help you to work more easily  and effectively. 

<br>
This book is about exploratory data analysis, about looking at data to see  what it seems to say. It concentrates on simple  arithmetic and easy-to-draw pictures. It regards whatever appearances  we have recognized as partial descriptions, and tries to look beneath them for  new insights. Its concern is with  appearance, not with confirmation.

<br>
**Examples, NOT case histories**

<br>
The book does not exist to make the case that exploratory data analysis is useful. Rather it exists to expose its readers and users to a considerable variety of techniques for looking more effectively at one's data.  The examples are not intended to be complete case histories. Rather they should isolated techniques in action on real data. The emphasis is on general techniques, rather than specific problems. 
<br>

A basic problem about any body of data is to make it more easily and effectively handleable by minds -- our minds, her mind, his mind. To this general end:

- anything that make a simpler description possible makes the description more easily handleable.
- anything that looks below the previously described surface makes the description more effective.


<br>

So we shall always be glad (a) to simplify description and (b) to describe one layer deeper. In particular, 

- to be able to say that we looked one layer deeper, and found nothing, is a definite step forward -- though not as far as to be able to say that we looked deeper and found thus-and-such.
- to be able to say that "if we change our point of view in the following way ... things are simpler" is always a gain--though not quite so much as to be able to say "if we don't bother to change out point of view (some other) things are equally simple."

<br>
...

<br>
Consistent with this view, we believe, is a clear demand that pictures based on exploration of data should *force* their messages upon us. Pictures that emphasize what we already know--"security blankets" to reassure us--are frequently not worth the space they take. Pictures that have to be gone over with a reading glass to see the main point are wasteful of time and inadequate of effect. **The greatest value of a picture** is when it *forces* us to notice **what we never expected to see.**

<br>
<center> <b>Confirmation</b> </center>
<br>

The principles and procedures of what we call confirmatory data analysis are both widely used and one of the great intellectual products of our century. In their simplest form, these principles and procedures look at a sample--and at what that sample has told us about the population from which it came--and assess the precision with which our inference from sample to population is made. We can no longer get along without confirmatory data analysis. <b>But we need not start with it.</b>

<br>

The best way to <b>understand what CAN be done is not longer</b>--if it ever was--<b>to ask what things could</b>, in the current state of our skill techniques, <b>be confirmed</b> (positively or negatively). Even more understanding is <em>lost</em> if we consider each thing we can do to data <em>only</em> in terms of some set of very restrictive assumptions under which that thing is best possible--assumptions we <em>know we CANNOT check in practice</em>.

<center> <b>Exploration AND confirmation</b> </center>

Once upon a time, statisticians only explored. Then they learned to confirm exactly--to confirm a few things exactly, each under very specific circumstances. As they emphasized exact confirmation, their techniques inevitably became less flexible. The connection of the most used techniques with past insights was weakened. Anything to which confirmatory procedure was not explicitly attached was decried as "mere descriptive statistics", no matter how much we learned from it. 

<br>

Today, the flexibility of (approximate) confirmation by the jacknife makes it relatively easy to ask, for almost any clearly specified exploration, "How far is it confirmed?"

<br>

**Today, exploratory and confirmatory can--and should--proceed side by side**. This book, of course, considers only exploratory techniques, leaving confirmatory techniques to other accounts. 

<br>

<center> <b> About the problems </b> </center>

<br>
The teacher needs to be careful about assigning problems. Not too many, please. They are likely to take longer than you think. The number supplied is to accommodate diversity of interest, not to keep everybody busy. 

<br>
Besides the length of our problems, both teacher and student need to realise that many problems do not have a single "right answer". There can be many ways to approach a body of data. Not all are equally good. For some bodies of data this may be clear, but for others we may not be able to tell from a single body of data which approach is preferred. Even several bodies of data about very similar situations may not be enough to show which approach should be preferred. Accordingly, it will often be quite reasonable for different analysts to reach somewhat different analyses.

<br>
Yet more--to unlock the analysis of a body of day, to find the good way to approach it, may require a key, whose finding is a creative act. Not everyone can be expected to create the key to any one situation. And to continue to paraphrase Barnum, no one can be expected to create a key to each situation he or she meets. 

<br>
**To learn about data analysis, it is right that each of us try many things that do not work**--that we tackle more problems than we make expert analyses of. We often learn less from an expertly done analysis than from one where, by not trying something, we missed--at least until we were told about it--an opportunity to learn more. Each teacher needs to recognize this in grading and commenting on problems. 

<br>
<center><b> Precision</b></center>

The teacher who heeds these words and admits that there need be *no one correct approach* may, I regret to contemplate, still want whatever is done to be digit perfect. (Under such a requirement, the write should still be able to pass the course, but it is not clear whether she would get an "A".) One does, from time to time, have to produce digit-perfect, carefully checked results, but forgiving techniques that are not too distributed by unusual data are also, usually, *little disturbed by SMALL arithmetic errors*. The techniques we discuss here have been chosen to be forgiving. It is hoped, then, that small arithmetic errors will take little off the problem's grades, leaving severe penalties for larger errors, either of arithmetic or concept.

:::

## Outline

:::: {.columns}

::: {.column width=45%}

1. [Scratching down numbers]{.monash-orange2}
2. [Schematic summary]{.monash-orange2}
3. [Easy re-expression]{.monash-orange2}
4. Effective comparison
5. Plots of relationship
6. Straightening out plots (using three points)
7. Smoothing sequences
8. Parallel and wandering schematic plots
9. Delineations of batches of points
10. Using two-way analyses
::: 

::: {.column width=10%}


:::

::: {.column width=45%}
11. Making two-way analyses
12. Advanced fits
13. Three way fits
14. Looking in two or more ways at batched of points
15. Counted fractions
16. Better smoothing
17. Counts in bin after bin
18. Product-ratio plots
19. Shapes of distributions
20. Mathematical distributions
::: 
::::

## Looking at numbers with Tukey {.transition-slide .center style="text-align: center;"}




## Scratching down numbers

:::: {.columns}
::: {.column}
Prices of Chevrolet in the local used car newspaper ads of 1968. 

```{r}
options(width=20)
chevrolets <- tibble(
  prices = c(250, 150, 795, 895, 695, 
               1699, 1499, 1099, 1693, 1166,
               688, 1333, 895, 1775, 895,
               1895, 795))
#chevrolets$prices
```


:::

::: {.column}

[Stem-and-leaf plot]{.monash-blue2}: still seen in introductory statistics texts

```{r}
#| echo: false
fabric_drawing(cid = "canvas1", 
               cwidth = 1050, 
               cheight = 450, 
               cfill = "whitesmoke", 
               drawingWidth = 3, 
               gumSize = 10)
fabric_text_add(cid = "canvas1", textId = "txt1",
                text = "250, 150, 795, 895, 695,  ",
                left = 10, top = 10, 
                fontFamily = "Courier", fontSize = 28)
fabric_text_add(cid = "canvas1", textId = "txt2",
                text = "1699, 1499, 1099, 1693,",
                left = 10, top = 60, 
                fontFamily = "Courier", fontSize = 28)
fabric_text_add(cid = "canvas1", textId = "txt3",
                text = "1166, 688, 1333, 895,",
                left = 10, top = 110, 
                fontFamily = "Courier", fontSize = 28)
fabric_text_add(cid = "canvas1", textId = "txt3",
                text = "1775, 895, 1895, 795",
                left = 10, top = 160, 
                fontFamily = "Courier", fontSize = 28)
#fabric_curtail(cid = "canvas1", imgsrc = #"images/lecture-02A/chevy.png", 
#                    type = "background")
```

:::
::::

##

:::: {.columns}

::: {.column width=45%}

First stem-and-leaf, first digit on stem, second digit on leaf

<img src="images/canvas1-IMG (5).png" width="90%">

::: 


::: {.column width=45% .fragment}

Order any leaves which need it, eg stem 6

<img src="images/canvas1-IMG (6).png" width="90%">

::: 
::::

::: {.fragment}
[*A benefit is that the numbers can be read off the plot, but the focus is still on the pattern. Also quantiles like the median, can be computed easily.*]{.smaller}
:::

##

:::: {.columns}
::: {.column width=50%}

Shrink the stem

<img src="images/canvas1-IMG (8).png" width="90%">

::: 

::: {.column width=50%}

Shrink the stem more

<img src="images/canvas1-IMG (9).png" width="90%">

::: 
::::

## And, in R ...

```{r stem, echo=TRUE}
chevrolets$prices
stem(chevrolets$prices)
```

## 🔖 Remember the tips data

```{r echo=FALSE}
options(width=80, digits=3)
tips <- read_csv("http://ggobi.org/book/data/tips.csv")
head(tips$tip, 50)
```

```{r stem_tips, echo=TRUE}
stem(tips$tip, scale=0.5, width=120)
```


## Refining the size

:::: {.columns}

::: {.column width=25%}

Five digits per stem

<img src="images/stem_stretched.png" width="100%">

<br>

::: {.fragment}
[What is the number in parentheses? And why might this be useful?]{.monash-orange2}
:::

::: 

::: {.column width=25%}
::: {.fragment}

Two digits per stem

<img src="images/stem_5line.png" width="100%">
:::
::: 
::: {.column width=39% style="text-size: 80%;"}

::: {.fragment}

```{r echo=FALSE}
options(width=100)
```

```{r stem_tips2, echo=TRUE}
stem(tips$tip, scale=2)
```

:::

::: {.fragment}
[Why no number in parentheses?]{.monash-orange2}
:::

::: {.fragment}

```{r}
median(tips$tip)
```
:::

:::
::::

## Summary

- Stem-and-leaf plots are similar information to the histogram. 
- Generally it is possible to also read off the numbers, and to then easily calculate median or Q1 or Q3. 
- It's great for small data sets, when you only have pencil and paper.
- Alternatives are a histogram, (jittered) dotplot, density plot, box plot, violin plot, letter value plot. 


## a different style of number scratching

for categorical variables

:::: {.columns}

::: {.column width=50%}
We know about

<img src="images/tally.png" width="90%">

but its too easy to 

<img src="images/tally_error.png" width="90%">

make a mistake

::: 
::: {.column width=50%}

Is this easier?

<img src="images/squares.png" width="90%">

::: {.fragment}
or harder
:::
::: 
::::

## Count this data using the squares approach.

:::: {.columns}

::: {.column width=30%}

```{r echo=FALSE}
options(width=30)
tips$sex[1:72]
```


::: 

::: {.column width=70%}
```{r}
#| echo: false
fabric_drawing(cid = "canvas2", 
               cwidth = 800, 
               cheight = 750, 
               cfill = "whitesmoke", 
               drawingWidth = 3, 
               gumSize = 10)
```

::: 
::::

## What does it mean to "feel what the data are like?" {.center}

## 

:::: {.columns}

::: {.column width=50%}
<img src="images/stem_alaska.png" width="90%">
::: 

::: {.column width=50%}

This is a stem and leaf of the height of the highest peak in each of the 50 US states. 

<br>

The states roughly fall into three groups.

<br>

[It's not really surprising, but we can imagine this grouping. Alaska is in a group of its own, with a much higher high peak. Then the Rocky Mountain states, California, Washington and Hawaii also have high peaks, and the rest of the states lump together.]{.monash-blue2 .smaller}

::: 
::::

## {.transition-slide .center style="text-align: center;"}


### Exploratory data analysis is detective work -- in the purest sense -- finding and revealing the clues.

## {.transition-slide .center  style="text-align: center;"}

### More summaries of numerical values

## Hinges and 5-number summaries

:::: {.columns}
::: {.column width="50%"}

```{r}
#| echo: false
options(width=25)
x <- c(-3.2, -1.7, -0.4, 0.1, 0.3, 1.2, 1.5, 1.8, 2.4, 3.0, 4.3, 6.4, 9.8)
x
```

You know the median is the middle number. What's a hinge?

There are 13 data values here, provided already sorted.  We are going to write them into a Tukey named down-up-down-up pattern, evenly. 

[Median will be 7th, hinge will be 4th from each end.]{.monash-blue2}
::: 

::: {.column width="50%"}

```{r}
#| echo: false
fabric_drawing(cid = "canvas3", 
               cwidth = 750, 
               cheight = 600, 
               cfill = "whitesmoke", 
               drawingWidth = 3, 
               gumSize = 10)
```
::: 
::::

## Hinges and 5-number summary

:::: {.columns}

::: {.column width="50%"}
<img src="images/canvas3-IMG.png" width="80%">
::: 

::: {.column width="50%"}
<img src="images/hinges.png" width="80%">
::: 


Hinges are almost always the same as Q1 and Q3
::::

## box-and-whisker display

:::: {.columns}

::: {.column width="50%"}
<img src="images/canvas3-IMG.png" width="80%">
::: 

::: {.column width="50%"}
Starting with a 5-number summary

<img src="images/5-number.png" width="80%">
 
::: 
::::

## box-and-whisker display

:::: {.columns}

::: {.column width="50%"}

Starting with a 5-number summary

<img src="images/5-number.png" width="80%">
 
::: 

::: {.column width="50%"}

```{r}
#| echo: false
fabric_drawing(cid = "canvas4", 
               cwidth = 750, 
               cheight = 600, 
               cfill = "whitesmoke", 
               drawingWidth = 3, 
               gumSize = 10)
```

::: 
::::

## Identified end values


:::: {.columns}

::: {.column width="50%"}
<img src="images/box_and_whisker.png" width="40%">

Why are some individual points singled out? 
::: 

::: {.column width="50%"}
<img src="images/schematic.png" width="40%">

Rules for this one may be clearer?
::: 
::::

## {.center style="text-align: center;"}

### Isn't this imposing a belief? 

## {.transition-slide .center style="text-align: center;"}

### There is no excuse for failing to plot and look

[Another Tukey wisdom drop]{.smallest}


## Fences and outside values {background-image="images/schematic.png" background-size="20%" background-position="99% 50%"}

::: {.column width="80%"}

- H-spread: difference between the hinges (we would call this Inter-Quartile Range)
- step: 1.5 times H-spread
- inner fences: 1 step outside the hinges
- outer fences: 2 steps outside the hinges
- the value at each end closest to, but still inside the inner fence are "adjacent"
- values between an inner fence and its neighbouring outer fence are "outside"
- values beyond outer fences are "far out"
- these rules produce a SCHEMATIC PLOT
:::

## New statistics: trimeans

The number that comes closest to 

$$\frac{\text{lower hinge} + 2\times \text{median} + \text{upper hinge}}{4}$$
is the **trimean**.

<br>
<br>

Think about trimmed means, where we might drop  the highest and lowest 5% of observations.

## Letter value plots

:::: {.columns}

::: {.column width="50%"}
Why break the data into quarters? Why not eighths, sixteenths? k-number summaries?

What does a 7-number summary look like?

<img src="images/7-number.png" width="80%">

[How would you make an 11-number summary?]{.monash-orange2}
::: 
::: {.column width="50%"}

```{r lvplot, echo=TRUE}
library(lvplot)
p <- ggplot(mpg, 
            aes(class, hwy))
p + geom_lv(aes(fill=..LV..)) + 
  scale_fill_brewer() + 
  coord_flip() + 
  xlab("")
```

::: 
::::

---
## Box plots are ubiquitous in use today. 

<br><br>
- 🐈🐩  Mostly used to compare distributions, multiple subsets of the data.

- Puts the emphasis on the [middle 50%]{.monash-orange2} of observations, although variations can put emphasis on other aspects.

## Easy re-expression {.transition-slide .center  style="text-align: center;"}

## Logs, square roots, reciprocals

:::: {.columns}

::: {.column width="50%"}

What you need to know about logs?

- how to find good enough logs fast and easily
- that equal differences in logs correspond to equal ratios of raw values. 

[(This means that wherever you find people using products or **ratios**-- even in such things as price indexes--using logs--thus converting producers to sums and ratios to differences--is likely to help.)]{.smaller}
::: 

::: {.column width="50%"}

::: {.fragment}
The most common transformations are logs, sqrt root, reciprocals, reciprocals of square roots

<center> -1, -1/2, +1/2, +1 </center>

What happened to ZERO?
:::

::: {.fragment}

It turns out that the role of a [zero power]{.monash-orange2}, is for the purposes of re-expression, neatly solved by the [logarithm]{.monash-orange2}.

:::

::: 
::::

## Re-express to symmetrize the distribution

<img src="images/logs.png" width="50%">

## Power ladder {.center style="text-align: center;"}
<br>
<br>

[⬅️ fix RIGHT-skewed values]{style="text-align: left;"} 
<br>
<br>

-2, -1, -1/2, 0 (log), 1/3, 1/2, [1]{.monash-orange2 .larger}, 2, 3, 4 

<br>

[fix LEFT-skewed values ➡️ ]{style="text-align: right;"} 

## {.transition-slide .center style="text-align: center;"}

### We now regard re-expression as a tool, something to let us do a better job of grasping. The grasping is done with the eye and the better job is through a more symmetric appearance.

[Another Tukey wisdom drop]{.smallest}

## Linearising bivariate relationships

<img src="images/linearise1.png" width="34%"> 
<img src="images/linearise2.png" width="34%"> 
<img src="images/linearise3.png" width="29%"> 

<br>

[Surprising observation: The small fluctuations in later years]{.monash-orange2}. 

 [What might be possible reasons?]{.monash-blue2}

## Linearising bivariate relationships

<img src="images/linearise4.png" width="32%"> 
<img src="images/linearise5.png" width="32%"> 
<img src="images/linearise6.png" width="32%"> 

<br>

See some fluctuations in the early years, too. [Note that the log transformation couldn't linearise.]{.monash-blue2}

## {.transition-slide .center style="text-align: center;"}

### Whatever the data, we can try to gain by straightening or by flattening. 


### When we succeed in doing one or both, we almost always see more clearly what is going on.


## Rules and advice

:::: {.columns}

::: {.column width="45%" style="font-size: 80%;"}

1. Graphics are friendly.
2. Arithmetic often exists to make graphs possible.
3. [Graphs force us to notice the unexpected]{.monash-orange2}; nothing could be more important.
4. Different graphs show us quite different aspects of the same data.
5. There is [no more reason to expect one graph to "tell all"]{.monash-orange2} than to expect one number to do the same.
6. "Plotting $y$ against $x$" involves significant choices--how we express one or both variables can be crucial.
:::


::: {.column width="45%" style="font-size: 80%;"}


7. The first step in penetrating plotting is to straighten out the dependence or point scatter as much as reasonable.
8. Plotting $y^2$, $\sqrt{y}$, $log(y)$, $-1/y$ or the like instead of $y$ is one plausible step to take in search of straightness.
9. Plotting $x^2$, $\sqrt{x}$, $log(x)$, $-1/x$ or the like instead of $x$ is another.
10. Once the plot is straightened, we can usually gain much by flattening it, usually by plotting residuals.
11. When plotting scatters, we may need to be careful about how we express $x$ and $y$ in order to avoid concealment by crowding.

:::
::::

## {background-image="https://vignette.wikia.nocookie.net/starwars/images/d/d6/Yoda_SWSB.png/revision/latest?cb=20150206140125" background-size="cover"}

<br><br><br>
[The book is a digest of]{.monash-white} 🌟 
[tricks and treats]{.monash-white}  🌟 [of massaging numbers and drafting displays.]{.monash-white} 

[Many of the tools have made it into today's analyses in various ways. Many have not.]{.monash-white}

[Notice the word developments too:]{.monash-white} [froots, fences.]{.monash-pink2} [Tukey brought you the word]{.monash-white} ["software"]{.monash-white}

[The temperament of the book is an inspiration for the mind-set for this unit. There is such delight in working with numbers!]{.monash-white}

["We love data!"]{.monash-white .larger}

## Take-aways

- Tukey's approach was a [reaction to many years of formalising data analysis]{.monash-blue2} using statistical hypothesis testing. 
- Methodology development in statistical testing was a [reaction to the ad-hoc nature]{.monash-blue2} of data analysis.
- Complex machine learning models like neural networks are in [reaction to the inability of statistical models]{.monash-blue2} to capture highly non-linear relationships, and depend heavily on the data provided. 
- Exploring data today is in reaction to the need to [explain complex models]{.monash-blue2}, to support organisations against legal challenges to decisions made from the model
- It is much [easier]{.monash-blue2} to accomplish [computers]{.monash-blue2}.
- "Exploratory data analysis" as commonly used today term is unfortunately synonymous with "descriptive statistics", but it is truly much more. Understanding its history from Tukey's advocation helps you see it is the tooling to [discover what you don't know]{.monash-orange2}.

## Resources

- [wikipedia](https://en.wikipedia.org/wiki/Exploratory_data_analysis)
- John W. Tukey (1977) Exploratory data analysis
- Data coding using [`tidyverse` suite of R packages](https://www.tidyverse.org) 
- Sketching canvases made using [`fabricerin`](https://ihaddadenfodil.com/post/fabricerin-a-tutorial/)

