Exploring Kaggle Titanic data with R Packages nanair and UpSetR

Exploring Kaggle Titanic data with R Packages nanair and UpSetR


Recently (6/8/2018), I saw a post about a new R package “nanair”, which according to the package documentation, “provides data structures and functions that facilitate the plotting of missing values and examination of imputations. This allows missing data dependencies to be explored with minimal deviation from the common work patterns of ‘ggplot2’ and tidy data.”  nanair is authored by Nicholas Tierney, Di Cook, Miles McBain, Colin Fay, Jim Hester, and Luke Smith.
In addition to the user manual, there is a github repo with an excellent readme and links to really good vignettes and tutorials.  I decided to try nanair out on the Titanic dataset on Kaggle, as a way to look at missing values.  The code for this article is on github, and includes many other examples not detailed here.  The main feature of nanair is the creation of “shadow matrices” which generate columns with binary values describing if there are missing data in the original data.frame.  However, there are many other nice features, some of which I’ll highlight here.
nanair’s handling of missing data can start right from reading a file.  Using the pipe notation, it looks like this:
   train_data <- read.csv(“train.csv”, header = TRUE, stringsAsFactors = FALSE) %>%     replace_with_na_all(condition = ~.x == “”)
The replace_with_na_all function allows you to replace any number of specific values, including blanks, with NA when you read the file.  Thus, if you have data that has blanks and NAs, you can clean that up in one operation.  Likewise, if there are values like -99 etc. you want to convert, you can do that there too.  Having had headaches where some data read in as blanks, and others NA, I really like this addition.  There are other variants of the replace_with_na available as well.
Let’s start by looking at a simple, not prettified, ggplot of the age data in the dataset.  Note that in my code, I have put the test and train data into a list, hence the list notation below.

  i <- 1
  ggplot(data = data_list[[i]],    aes(x = PassengerId, y = Age)) +    geom_point() +    labs(title = paste0(“ages for all passengersn”,                               “for “, names(data_list)[i], ” data”)) +    theme(plot.title = element_text(size = 12, hjust = 0.5)) +    theme(legend.title = element_blank())
which produces this plot:

ggplot does not represent the missing values, so this chart is actually misleading.  Here’s where nanair can help.  nanair provides functions that mirror some ggplot functions, but explicitly handle missing values.  In this case, we’ll replace geom_point with geom_miss_point:
  i <- 1
  ggplot(data = data_list[[i]],    aes(x = PassengerId, y = Age)) +    geom_miss_point() +    labs(title = paste0(“ages for all passengersn”,                               “for “, names(data_list)[i], ” data”)) +    scale_color_manual(labels = c(“Missing”, “Has Age”),    values = c(“coral2”, “aquamarine3”)) +    theme(plot.title = element_text(size = 12, hjust = 0.5)) +    theme(legend.title = element_blank())
which produces this:

Much nicer!  What nanair geom_miss_point does in this case, is separate the missing values, then offset them from the other data (in this case, by putting them below 0).  The nanair functions work nicely with the ggplot enhancements, such as facets:
  i <- 1
  ggplot(data = data_list[[i]],    aes(x = PassengerId, y = Age)) +     geom_miss_point() +     facet_wrap(~Pclass, ncol = 2) +     theme(legend.position = “bottom”) +    labs(title = paste0(“ages for all passengersn”,                               “for “, names(data_list)[i], ” datan”,                               “by passenger class”)) +    scale_color_manual(labels = c(“Missing”, “Has Age”),    values = c(“coral2”, “aquamarine3”)) +    theme(plot.title = element_text(size = 12, hjust = 0.5)) +    theme(legend.title = element_blank())
producing:

So we can quickly and easily see that there are more missing ages for the third class passengers.  Age is an important feature in the Titanic data set, so understanding its missingness informs what do do about it.
A nice feature of the shador matrix is that it comes along for the ride in other operations.  Here, we impute the ages for the train data using the impute_lm function from package simputation (authored by Mark van der Loo).  Note that for the train we can leverage the Survived label which we cannot do for the test data.
  i <- 1
  data_list[[i]] %>%    bind_shadow() %>%    impute_lm(Age ~ Fare + Survived + Pclass + Sex + Embarked) %>%    ggplot(aes(x = PassengerId,    y = Age,    color = Age_NA)) +     geom_point() +    labs(title = paste0(“ages for all passengersn”,                               “for “, names(data_list)[i], ” datan”,                               “with missing values imputed”)) +    scale_color_manual(labels = c(“Has Age”, “Missing”),    values = c(“aquamarine3”, “coral2”)) +    theme(plot.title = element_text(size = 12, hjust = 0.5)) +    theme(legend.title = element_blank()) 
generating this:

In the above, the bind_shadow() function generates and attaches the shadow matrix to the original data.frame.
Another function in the nanair package is as_shadow_upset, which works with the UpSetR package to generate plots showing permutations of the missing values.
  i <- 1
  data_list[[i]] %>%    as_shadow_upset() %>%    upset(main.bar.color = “aquamarine3”,    matrix.color = “red”,     text.scale = 1.5,    nsets = 3)
Which produces this great visualization:

The upset() function comes from UpSetR, and the combination of these two shows us missing values, combination of missing values, and the number in each case in two dimensions.  I had not used UpSetR before seeing it in the nanair vignettes, so props to its authors Jake Conway, and Nils Gehlenborg.
As a last example, nanair provides gg_mis_var() to generate basic plots of missingness:
  i <- 1
  gg_miss_var(data_list[[i]]) +    labs(title = paste0(“missing data for “,                                names(data_list)[i], ” data”)) +    theme(plot.title = element_text(size = 12, hjust = 0.5))
producing this:

There is much more you can do with nanair, and I would recommend you read the excellent tutorials and other information they have provided.  I think the value will really come through for more complex data sets and enhance may EDA workflows.


Link: Exploring Kaggle Titanic data with R Packages nanair and UpSetR