tidytuesday: Part-of-Speech and textrecipes with The Office

This post was written before the change to textrecipes to support spacyr as an engine to step_tokenize(). It is still a good demonstration of how to use a custom tokenizer.

I’m ready for my second #tidytuesday and as a massive The Office fan this dataset is right up my alley. In this post, you will read how to

  • Use the R wrapper spacyr of spacy to extract part of speech tags
  • Use a custom tokenizer in conjunction with textrecipes package
  • Do hyperparameter tuning with the tune package
  • Try to predict the author of each line in the show

I’ll put a little more effort into the explorative charts then I usually do. I’ll not be explaining each line of code for those, but you are encouraged to play around with them yourself.

Packages πŸ“¦

library(schrute)
library(tidytext)
library(tidymodels)
library(tidyr)
library(tokenizers)
library(textrecipes)
library(spacyr)
library(paletteer)

We will be using the schrute package which includes the dataset for the week. tidytext and tokenizers to do data exploration for the text. spacyr to access the spacy to perform part of speech tagging. tidymodels and textrecipes to do to the preprocessing and modeling. And lastly, we use paletteer to get pretty color palettes.

Exploring the data ⛏

The data comes with a lot of different variables. We will be focusing on character and text which contains the character and what they said, in English1. First, let us take a look at how many lines each character has

theoffice %>%
  count(character, sort = TRUE)
## # A tibble: 773 x 2
##    character     n
##    <chr>     <int>
##  1 Michael   10921
##  2 Dwight     6847
##  3 Jim        6303
##  4 Pam        5031
##  5 Andy       3754
##  6 Angela     1569
##  7 Kevin      1564
##  8 Erin       1440
##  9 Oscar      1368
## 10 Ryan       1198
## # … with 763 more rows

Micheal, Dwight, Jim, and Pam are dominating the charts. This is unsurprising since they are some of the main characters having a central role in the episodes they appear in. This will be too many classes for the scope of this post so I’ll limit it to the top 5 characters with the most lines since the number drops off more after the first 5.

small_office <- theoffice %>%
  select(character, text) %>%
  filter(character %in% c("Michael", "Dwight", "Jim", "Pam", "Andy"))

Let us take a lot at how many words each line in the script is. This is going to be a problem for us later on as predicting with shorter text is harder than longer text as there is less information in it.

small_office %>%
  mutate(n_words = count_words(text)) %>%
  ggplot(aes(n_words, color = character)) +
  geom_density(binwidth = 1, key_glyph = draw_key_timeseries) +
  xlim(c(0, 50)) +
  scale_color_paletteer_d("nord::aurora") +
  labs(x = "Number of words", y = "Density", color = NULL,
       title = "Distribution of line length in The Office") +
  theme_minimal() +
  theme(legend.position = "top", 
        plot.title.position = "plot") 

These lines are thankfully pretty similar, which will make it easier for us to make a good predictive model. However, we can still see some differences. Pam and Jim both have shorter lines than the rest, and Michael and Andy both have fewer shorter lines in exchange for more long lines.

We will be also be exploring part of speech tagging and for that, we will be using the spacyr package. It isn’t always needed but I’m going to explicitly initialize the spacy model

spacy_initialize(model = "en_core_web_sm")

the spacyr package outputs in this nice format with doc_id, sentence_id, token_id, token and pos.

spacy_parse(small_office$text[1], entity = FALSE, lemma = FALSE)
##    doc_id sentence_id token_id       token   pos
## 1   text1           1        1         All   ADV
## 2   text1           1        2       right   ADV
## 3   text1           1        3         Jim PROPN
## 4   text1           1        4           . PUNCT
## 5   text1           2        1        Your   ADJ
## 6   text1           2        2 quarterlies  NOUN
## 7   text1           2        3        look  VERB
## 8   text1           2        4        very   ADV
## 9   text1           2        5        good   ADJ
## 10  text1           2        6           . PUNCT
## 11  text1           3        1         How   ADV
## 12  text1           3        2         are  VERB
## 13  text1           3        3      things  NOUN
## 14  text1           3        4          at   ADP
## 15  text1           3        5         the   DET
## 16  text1           3        6     library  NOUN
## 17  text1           3        7           ? PUNCT

Normally I would just analyze the data in this format. But since I have to create a custom wrapper for textrecipes anyway I’ll do the remaining of the text mining in tidytext. textrecipes requires that the tokenizer returns the tokens in a list format similar to the tokenizers in tokenizers. The following function takes a character vector and returns the part of speech tags in a list format.

spacy_pos <- function(x) {
  tokens <- spacy_parse(x, entity = FALSE, lemma = FALSE)
  token_list <- split(tokens$pos, tokens$doc_id)
  names(token_list) <- gsub("text", "", names(token_list))
  res <- unname(token_list[as.character(seq_along(x))])
  empty <- lengths(res) == 0
  res[empty] <- lapply(seq_len(sum(empty)), function(x) character(0))
  res
}

Little example to showcase the function

example_string <- c("Hello there pig", "", "One more pig here")

spacy_pos(x = example_string)
## [[1]]
## [1] "INTJ" "ADV"  "VERB"
## 
## [[2]]
## character(0)
## 
## [[3]]
## [1] "NUM"  "ADJ"  "NOUN" "ADV"

We can use a custom tokenizer by simply passing it to the token argument. This is going to take a little longer than normal since POS tagging takes longer than simply tokenizing.

small_office_tokens <- small_office %>%
  unnest_tokens(text, text, token = spacy_pos, to_lower = FALSE)

Below is a chart of the number of each part of speech tags. The meaning of the acronyms can be found here if you click on the Universal Part-of-speech Tags button.

colors <- rep(paletteer_d("rcartocolor::Pastel"), length.out = 16)

small_office_tokens %>%
  count(text) %>%
  ggplot(aes(n, reorder(text, n), fill = reorder(text, n))) +
  geom_col() +
  labs(x = NULL, y = NULL, title = "Part of Speech tags in The Office") +
  scale_fill_manual(values = colors) +
  guides(fill = "none") +
  theme_minimal() +
  theme(plot.title.position = "plot") 

I found it initially surprising that punctuation (PUNCT) was leading the chart. But after thinking about it a little bit it, I can imagine it has something to do with all the lines being very short and having to end in some kind of punctuation.

We can facet this by the character to see who uses what part of speech.

small_office_tokens %>%
  count(character, text) %>%
  group_by(character) %>%
  mutate(prop = n / sum(n)) %>%
  ungroup() %>%
  ggplot(aes(forcats::fct_rev(reorder(text, n)), prop, fill = character)) +
  geom_col(position = "dodge") +
  scale_fill_paletteer_d("nord::aurora") +
  labs(x = NULL, y = NULL, fill = NULL,
       title = "Part of speech tags by main character in The Office") +
  theme_minimal() +
  theme(legend.position = "top", 
        plot.title.position = "plot") 

I don’t immediately see anything popping out at me, but it is a very pretty chart otherwise. I feel like I have seen enough, lets get to modeling!

Modeling βš™οΈ

Not that we have gotten a look at the data lets get to modeling. First we need to do a test/train split which we can do with yardstick.

set.seed(1234)
office_split <- initial_split(small_office, strata = character)
office_test <- testing(office_split)
office_train <- training(office_split)

Next we are going to prepare the preprocessing steps. We will be using the custom part of speech tokenizer we defined earlier to include part of speech tag counts as features in our model. Since this data is going to a little sparse will we also include bi-grams of the data. To this, we first create a copy of the text variable and apply the tokenizers to each copy. Lastly will be also be doing some downsampling of the data to handle the imbalance in the data. This calculation will once again take a little while since the part of speech calculations takes a minute or two.

rec <- recipe(character ~ text, data = small_office) %>%
  # Deal with imbalance
  step_downsample(character) %>%
  # Create copy of text variable
  step_mutate(text_copy = text) %>%
  # Tokenize the two text columns
  step_tokenize(text, token = "ngrams", options = list(n = 2)) %>%
  step_tokenize(text_copy, custom_token = spacy_pos) %>%
  # Filter to only keep the most 100 frequent n-grams
  step_tokenfilter(text, max_tokens = 100) %>%
  # Calculate tf-idf for both sets of tokens
  step_tfidf(text, text_copy) %>%
  prep()

We can now extract the processed data

office_test_prepped <- bake(rec, office_test)
office_train_prepped <- juice(rec)

To do the actual modeling we will be using multinom_reg() with "glmnet" as the engine. This model has two hyperparameters, which we will be doing a grid search over to find optimal values. We specify that we want to tune these parameters by passing tune() to them.

tune_spec <- multinom_reg(penalty = tune(), mixture = tune()) %>%
  set_engine("glmnet")
tune_spec
## Multinomial Regression Model Specification (classification)
## 
## Main Arguments:
##   penalty = tune()
##   mixture = tune()
## 
## Computational engine: glmnet

Next we set up a bootstrap sampler and grid to optimize over.

set.seed(12345)
office_boot <- bootstraps(office_train_prepped, strata = character, times = 10)

hyper_grid <- grid_regular(penalty(), mixture(), levels = 10)

We are experiencing a little bit of data leakage since we don’t perform the preprocessing within each bootstrap.

Now we pass all the objects to tune_grid(). It is also possible to combine our recipe and model object into a workflow object to pass to tune_grid instead. However, since the preprocessing step took so long and we didn’t vary anything it makes more sense time-wise to use tune_grid() with a formula instead. I also set control = control_grid(verbose = TRUE) so I get a live update of how far the calculations are going.

set.seed(123456)
fitted_grid <- tune_grid(
  formula = character ~ .,
  model = tune_spec,
  resamples = office_boot,
  grid = hyper_grid,
  control = control_grid(verbose = TRUE)
)
## Warning: `tune_grid.formula()` is deprecated as of lifecycle 0.1.0.
## The first argument to `tune_grid()` should be either a model or a workflow. In the future, you can use:
## tune_grid(tune_spec, character ~ ., resamples = office_boot, 
##     grid = hyper_grid, control = control_grid(verbose = TRUE))
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_warnings()` to see where this warning was generated.

We can now look at the best performing models with show_best()

fitted_grid %>%
  show_best("roc_auc")
## # A tibble: 5 x 7
##   penalty mixture .metric .estimator  mean     n  std_err
##     <dbl>   <dbl> <chr>   <chr>      <dbl> <int>    <dbl>
## 1 0.00599   0.444 roc_auc hand_till  0.568    10 0.000724
## 2 0.00599   0.556 roc_auc hand_till  0.568    10 0.000722
## 3 0.00599   0.333 roc_auc hand_till  0.567    10 0.000716
## 4 0.00599   0.667 roc_auc hand_till  0.567    10 0.000715
## 5 0.00599   0.222 roc_auc hand_till  0.567    10 0.000700

And we can use the values from the best performing model to fit our final model.

final_model <- tune_spec %>%
  update(penalty = 0.005994843, mixture = 1 / 3) %>%
  fit(character ~ ., data = office_train_prepped)

Evaluation πŸ“

Now that we have our final model we can predict on our test set and look at the confusion matrix to see how well we did.

bind_cols(
  predict(final_model, office_test_prepped),
  office_test_prepped
) %>%
  conf_mat(truth = character, estimate = .pred_class) %>%
  autoplot(type = "heatmap")

These are not going too well. It is doing best at predicting Michael correctly, and it seems to confuse Dwight and Michael a little bit.

Let us investigate the cases that didn’t go too well. We can get the individual class probabilities by setting type = "prob" in predict()

class_predictions <- predict(final_model, office_test_prepped, type = "prob")
class_predictions
## # A tibble: 8,212 x 5
##    .pred_Andy .pred_Dwight .pred_Jim .pred_Michael .pred_Pam
##         <dbl>        <dbl>     <dbl>         <dbl>     <dbl>
##  1      0.239        0.167     0.237         0.248    0.108 
##  2      0.235        0.232     0.186         0.180    0.168 
##  3      0.176        0.101     0.344         0.322    0.0571
##  4      0.188        0.193     0.220         0.190    0.208 
##  5      0.250        0.178     0.171         0.216    0.184 
##  6      0.208        0.183     0.193         0.186    0.230 
##  7      0.166        0.137     0.256         0.199    0.242 
##  8      0.185        0.145     0.185         0.247    0.237 
##  9      0.195        0.226     0.212         0.180    0.188 
## 10      0.155        0.211     0.245         0.190    0.200 
## # … with 8,202 more rows

We can do some wrangling to get the 5 worst predicted texts for each character:

bind_cols(
  class_predictions,
  office_test
) %>%
  pivot_longer(starts_with(".pred_")) %>%
  filter(gsub(".pred_", "", name) == character) %>%
  group_by(character) %>%
  arrange(value) %>%
  slice(1:5) %>%
  ungroup() %>%
  select(-name, -value) %>%
  reactable::reactable()

So the first striking thing here is that many of the lines are quite short, with most of Pam’s being 5 words or less. On the other hand, all the wrongly predicted lines for Michael are quite a bit longer than the rest.

We can also get the best predicted lines for each character by flipping the sign with desc()

bind_cols(
  class_predictions,
  office_test
) %>%
  pivot_longer(starts_with(".pred_")) %>%
  filter(gsub(".pred_", "", name) == character) %>%
  group_by(character) %>%
  arrange(desc(value)) %>%
  slice(1:5) %>%
  ungroup() %>%
  select(-name, -value) %>%
  reactable::reactable()

One thing I noticed is that many of Pam’s lines start with β€œOh my” and that might have been a unique character trait that got picked up in the bi-grams.

session information


─ Session info ───────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.6.0 (2019-04-26)
 os       macOS Mojave 10.14.6        
 system   x86_64, darwin15.6.0        
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Los_Angeles         
 date     2020-05-02                  

─ Packages ───────────────────────────────────────────────────────────────────
 ! package       * version    date       lib
 P assertthat      0.2.1      2019-03-21 [?]
 P backports       1.1.6      2020-04-05 [?]
 P base64enc       0.1-3      2015-07-28 [?]
 P bayesplot       1.7.1      2019-12-01 [?]
 P blogdown        0.18       2020-03-04 [?]
 P bookdown        0.18       2020-03-05 [?]
 P boot            1.3-24     2019-12-20 [?]
 P broom         * 0.5.5      2020-02-29 [?]
 P callr           3.4.3      2020-03-28 [?]
 P class           7.3-16     2020-03-25 [?]
 P cli             2.0.2      2020-02-28 [?]
 P clipr           0.7.0      2019-07-23 [?]
 P cluster         2.1.0      2019-06-19 [?]
 P codetools       0.2-16     2018-12-24 [?]
 P colorspace      1.4-1      2019-03-18 [?]
 P colourpicker    1.0        2017-09-27 [?]
 P crayon          1.3.4      2017-09-16 [?]
 P crosstalk       1.1.0.1    2020-03-13 [?]
 P data.table      1.12.8     2019-12-09 [?]
 P desc            1.2.0      2018-05-01 [?]
 P details       * 0.2.1      2020-01-12 [?]
 P dials         * 0.0.6      2020-04-03 [?]
 P DiceDesign      1.8-1      2019-07-31 [?]
 P dichromat       2.0-0      2013-01-24 [?]
 P digest          0.6.25     2020-02-23 [?]
 P dplyr         * 0.8.5      2020-03-07 [?]
 P DT              0.13       2020-03-23 [?]
 P dygraphs        1.1.1.6    2018-07-11 [?]
 P ellipsis        0.3.0      2019-09-20 [?]
 P emo             0.0.0.9000 2019-12-18 [?]
 P evaluate        0.14       2019-05-28 [?]
 P fansi           0.4.1      2020-01-08 [?]
 P farver          2.0.3      2020-01-16 [?]
 P fastmap         1.0.1      2019-10-08 [?]
 P forcats         0.5.0      2020-03-01 [?]
 P foreach         1.5.0      2020-03-30 [?]
 P furrr           0.1.0      2018-05-16 [?]
 P future          1.16.0     2020-01-16 [?]
 P generics        0.0.2      2018-11-29 [?]
 P ggplot2       * 3.3.0      2020-03-05 [?]
 P ggridges        0.5.2      2020-01-12 [?]
 P glmnet          3.0-2      2019-12-11 [?]
 P globals         0.12.5     2019-12-07 [?]
 P glue            1.4.0      2020-04-03 [?]
 P gower           0.2.1      2019-05-14 [?]
 P GPfit           1.0-8      2019-02-08 [?]
 P gridExtra       2.3        2017-09-09 [?]
 P gtable          0.3.0      2019-03-25 [?]
 P gtools          3.8.2      2020-03-31 [?]
 P hardhat         0.1.2      2020-02-28 [?]
 P htmltools       0.4.0      2019-10-04 [?]
 P htmlwidgets     1.5.1      2019-10-08 [?]
 P httpuv          1.5.2      2019-09-11 [?]
 P httr            1.4.1      2019-08-05 [?]
 P igraph          1.2.5      2020-03-19 [?]
 P infer         * 0.5.1      2019-11-19 [?]
 P inline          0.3.15     2018-05-18 [?]
 P ipred           0.9-9      2019-04-28 [?]
 P iterators       1.0.12     2019-07-26 [?]
 P janeaustenr     0.1.5      2017-06-10 [?]
 P jcolors         0.0.4      2019-12-08 [?]
 P jsonlite        1.6.1      2020-02-02 [?]
 P knitr         * 1.28       2020-02-06 [?]
 P labeling        0.3        2014-08-23 [?]
 P later           1.0.0      2019-10-04 [?]
 P lattice         0.20-41    2020-04-02 [?]
 P lava            1.6.7      2020-03-05 [?]
 P lhs             1.0.1      2019-02-03 [?]
 P lifecycle       0.2.0      2020-03-06 [?]
 P listenv         0.8.0      2019-12-05 [?]
 P lme4            1.1-23     2020-04-07 [?]
 P loo             2.2.0      2019-12-19 [?]
 P lubridate       1.7.8      2020-04-06 [?]
 P magrittr        1.5        2014-11-22 [?]
 P mapproj         1.2.7      2020-02-03 [?]
 P maps            3.3.0      2018-04-03 [?]
 P markdown        1.1        2019-08-07 [?]
 P MASS            7.3-51.5   2019-12-20 [?]
 P Matrix          1.2-18     2019-11-27 [?]
 P matrixStats     0.56.0     2020-03-13 [?]
 P mime            0.9        2020-02-04 [?]
 P miniUI          0.1.1.1    2018-05-18 [?]
 P minqa           1.2.4      2014-10-09 [?]
 P munsell         0.5.0      2018-06-12 [?]
 P nlme            3.1-145    2020-03-04 [?]
 P nloptr          1.2.2.1    2020-03-11 [?]
 P nnet            7.3-13     2020-02-25 [?]
 P oompaBase       3.2.9      2019-08-24 [?]
 P paletteer     * 1.1.0      2020-03-10 [?]
 P palr            0.2.0      2020-01-30 [?]
 P pals            1.6        2019-12-04 [?]
 P parsnip       * 0.1.0.9001 2020-04-17 [?]
 P pillar          1.4.3      2019-12-20 [?]
 P pkgbuild        1.0.6      2019-10-09 [?]
 P pkgconfig       2.0.3      2019-09-22 [?]
 P plyr            1.8.6      2020-03-03 [?]
 P png             0.1-7      2013-12-03 [?]
 P prettyunits     1.1.1      2020-01-24 [?]
 P prismatic       0.2.0.9000 2020-03-15 [?]
 P pROC            1.16.2     2020-03-19 [?]
 P processx        3.4.2      2020-02-09 [?]
 P prodlim         2019.11.13 2019-11-17 [?]
 P promises        1.1.0      2019-10-04 [?]
 P ps              1.3.2      2020-02-13 [?]
 P purrr         * 0.3.3      2019-10-18 [?]
 P R6              2.4.1      2019-11-12 [?]
 P Rcpp            1.0.4.6    2020-04-09 [?]
 P reactable       0.1.0.1    2020-02-29 [?]
 P reactR          0.4.2      2020-01-24 [?]
 P recipes       * 0.1.10     2020-03-18 [?]
 P rematch2        2.1.1      2020-03-28 [?]
   renv            0.9.3      2020-02-10 [1]
 P reshape2        1.4.4      2020-04-09 [?]
 P reticulate      1.15       2020-04-02 [?]
 P rlang           0.4.5      2020-03-01 [?]
 P rmarkdown       2.1        2020-01-20 [?]
 P rpart           4.1-15     2019-04-12 [?]
 P rprojroot       1.3-2      2018-01-03 [?]
 P rsample       * 0.0.6      2020-03-31 [?]
 P rsconnect       0.8.16     2019-12-13 [?]
 P rstan           2.19.3     2020-02-11 [?]
 P rstanarm        2.19.3     2020-02-11 [?]
 P rstantools      2.0.0      2019-09-15 [?]
 P rstudioapi      0.11       2020-02-07 [?]
 P scales        * 1.1.0      2019-11-18 [?]
 P schrute       * 0.2.1      2020-03-30 [?]
 P scico           1.1.0      2018-11-20 [?]
 P sessioninfo     1.1.1      2018-11-05 [?]
 P shape           1.4.4      2018-02-07 [?]
 P shiny           1.4.0.2    2020-03-13 [?]
 P shinyjs         1.1        2020-01-13 [?]
 P shinystan       2.5.0      2018-05-01 [?]
 P shinythemes     1.1.2      2018-11-06 [?]
 P SnowballC       0.7.0      2020-04-01 [?]
 P spacyr        * 1.2.1      2020-03-04 [?]
 P StanHeaders     2.21.0-1   2020-01-19 [?]
 P statmod         1.4.34     2020-02-17 [?]
 P stopwords       1.0        2019-07-24 [?]
 P stringi         1.4.6      2020-02-17 [?]
 P stringr         1.4.0      2019-02-10 [?]
 P survival        3.1-12     2020-03-28 [?]
 P textrecipes   * 0.1.0.9000 2020-04-11 [?]
 P threejs         0.3.3      2020-01-21 [?]
 P tibble        * 3.0.1      2020-04-20 [?]
 P tidymodels    * 0.1.0      2020-02-16 [?]
 P tidyposterior   0.0.2      2018-11-15 [?]
 P tidypredict     0.4.5      2020-02-10 [?]
 P tidyr         * 1.0.2      2020-01-24 [?]
 P tidyselect      1.0.0      2020-01-27 [?]
 P tidytext      * 0.2.3      2020-03-04 [?]
 P timeDate        3043.102   2018-02-21 [?]
 P tokenizers    * 0.2.1      2018-03-29 [?]
 P tune          * 0.1.0      2020-04-02 [?]
 P utf8            1.1.4      2018-05-24 [?]
 P vctrs           0.2.4      2020-03-10 [?]
 P viridisLite     0.3.0      2018-02-01 [?]
 P withr           2.1.2      2018-03-15 [?]
 P workflows     * 0.1.1      2020-03-17 [?]
 P xfun            0.13       2020-04-13 [?]
 P xml2            1.3.0      2020-04-01 [?]
 P xtable          1.8-4      2019-04-21 [?]
 P xts             0.12-0     2020-01-19 [?]
 P yaml            2.2.1      2020-02-01 [?]
 P yardstick     * 0.0.6      2020-03-17 [?]
 P zoo             1.8-7      2020-01-10 [?]
 source                              
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 Github (hadley/emo@3f03b11)         
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 Github (jaredhuling/jcolors@f200755)
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 local                               
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 local                               
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 Github (therneau/survival@c55af18)  
 local                               
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.2)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      
 CRAN (R 3.6.0)                      

[1] /Users/emilhvitfeldthansen/Desktop/blogv4/renv/library/R-3.6/x86_64-apple-darwin15.6.0
[2] /private/var/folders/m0/zmxymdmd7ps0q_tbhx0d_26w0000gn/T/RtmpcBhzUv/renv-system-library

 P ── Loaded and on-disk path mismatch.