![\"very](\"./images/sc-lines-by-quarter.png\")
We’ll use some ggplot and tidyverse code to plot the data we collected in part 1
\n\n\nAgenda\nIn this tutorial, we will cover:
\n\n
We’ll use tidyverse (which includes things like magrittr (%>%) and ggplot2) to load our data. Reshape2 provides the cast/melt functions which can be used to reshape specific datasets into formats that are easier to plot.
library(tidyverse)\nlibrary(reshape2)\n# This path will probably need to be changed depending on where you downloaded that dataset.\ndata = read_tsv(\"sc.tsv\")\nLet’s start by cleaning our data a bit. We’ve got a couple problems with it:
\nclean = data %>%\nselect(num, class, additions, deletions, mergedAt) %>%\nfilter(!is.na(mergedAt)) %>%\ngroup_by(num, class, mergedAt, month=floor_date(mergedAt, 'month'), quarter=floor_date(mergedAt, 'quarter')) %>%\nsummarise(additions=sum(additions), deletions=-sum(deletions)) %>%\nfilter(class != \"ignore\") %>%\nfilter(class != \"image\") %>%\narrange(mergedAt) %>%\nas_tibble()\nWe’ll setup a ‘theme’ for our plot that mainly consists of using the black and white theme which is quite elegant and readable, and then making some font sizes a wee bit larger:
\ntheme = theme_bw() + theme(\naxis.text=element_text(size=14),\nplot.title=element_text(size=18),\naxis.title=element_text(size=16),\nlegend.title=element_text(size=14),\nlegend.text=element_text(size=14))\nWe want all of our plots to look the same which is why we use this trick, it helps us keep a consistent aesthetic without having to re-type the configuration every single time.
\nLet’s get to plotting!
\nWe have a dataset that looks like:
\nclean %>% head(10)\nWe’ll want to plot the changes versus the time (either quarter or month), and\nthen maybe plot them differently based on the class of the addition/removal. So that translates into an aesthetics statement like aes(x=time, y=additions, fill=class)
In ggplot2 you can plot data some different ways, if you provide the datasets upfront, e.g. data %>% ggplot() you’ll generally do something like data %>% ggplot(aes(x=a, y=b)) + geom_something() and geom_something will take the data from the ggplot call. However if you want to plot multiple series, you can also provide the data directly to the geom_* functions like so:
ggplot() +\ngeom_col(data=clean, aes(x=quarter, y=additions, fill=class)) +\ngeom_col(data=clean, aes(x=quarter, y=deletions, fill=class)) + scale_fill_brewer(palette = \"Paired\") +\ngeom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color=\"black\") +\ntheme +\nxlab(\"Quarter\") + ylab(\"Lines added or removed\") + guides(fill=guide_legend(title=\"Category\")) +\nggtitle(\"Lines added or removed by file type across GTN Single Cell Learning Materials\")\nggsave(\"sc-lines-by-quarter.png\", width=12, height=5)\nggplot() +\ngeom_col(data=clean, aes(x=month, y=additions, fill=class)) +\ngeom_col(data=clean, aes(x=month, y=deletions, fill=class)) + scale_fill_brewer(palette = \"Paired\") +\ntheme +\ngeom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color=\"black\") +\nxlab(\"Month\") + ylab(\"Lines added or removed\") + guides(fill=guide_legend(title=\"Category\")) +\nggtitle(\"Lines added or removed by file type across GTN Single Cell Learning Materials\")\nggsave(\"sc-lines-by-month.png\", width=12, height=5)\n
In this case we plotted the data for additions and deletions separately, and we additionally added points based on the actual date of the PRs to visualise their density.
\nLet’s produce ‘s favourite running sum plots in addition. We’ll start by reshaping our data. Currently we have data that looks like:
\n| time | \nvariable | \nvalue | \n
|---|---|---|
| today | \nmeasure1 | \n1 | \n
| today | \nmeasure2 | \n10 | \n
| yesterday | \nmeasure1 | \n30 | \n
| yesterday | \nmeasure2 | \n5 | \n
And we’ll re-shape this to look like this, which will make it easier to calculate changes over the course of a specific series:
\n| time | \nmeasure1 | \nmeasure2 | \n
|---|---|---|
| today | \n1 | \n10 | \n
| yesterday | \n30 | \n5 | \n
We’ll use the dcast function to do that:
clean %>% select(month, class, additions) %>% dcast(month ~ class, value.var=\"additions\") %>% head()\nThat doesn’t look quite right so, let’s change how the data is aggregated:
\n# cumulative\nclean %>% select(month, class, additions) %>% dcast(month ~ class, value.var=\"additions\", fun.aggregate = sum)\nLet’s do it for real now:
\ncumulative = clean %>% select(month, class, additions) %>%\ndcast(month ~ class, value.var=\"additions\", fun.aggregate = sum) %>%\nmutate(across(bibliography:workflows, cumsum)) %>%\nreshape2::melt(id.var=\"month\")\ncumulative %>% ggplot(aes(x=month, y=value, color=variable)) + geom_line() +\ntheme_bw() + theme +\nxlab(\"Month\") + ylab(\"Lines added\") + guides(color=guide_legend(title=\"Category\")) +\nggtitle(\"Cumulative lines added by category, across GTN Single Cell materials\")\nggsave(\"sc-lines-cumulative.png\", width=12, height=8)\n![\"workflows](\"./images/sc-lines-cumulative.png\")
Let’s again start with some cleaning, namely removing all rows with 0 records, and removing future records (at the time of writing.)
\nroles = read_tsv(\"sc-roles.tsv\")\nw = roles %>%\nfilter(count != 0) %>%\nfilter(!grepl(\"2025\", date)) %>%\nfilter(!grepl(\"2024-12-01\", date))\nw %>%\nggplot(aes(x=date, y=count, color=area)) +\ntheme +\nxlab(\"Date\") + ylab(\"Unique Contributors\") + guides(color=guide_legend(title=\"Contribution Area\")) +\nggtitle(\"Contributors over time to GTN Single Cell Materials\") +\ngeom_line()\nggsave(\"sc-contribs.png\", width=12, height=6)\n![\"plot](\"./images/sc-contribs.png\")
Let’s plot all of the new single cell things added over time, all the new FAQs, Tutorials, Slides, Etc:
\nadded_by_time = read_tsv(\"single-cell-over-time.tsv\")\nadded_by_time %>% dcast(date ~ <code>type</code>) %>%\narrange(date) %>%\nmutate(across(event:workflow, cumsum)) %>%\nmelt(id.var=\"date\") %>%\nas_tibble() %>% arrange(date) %>%\nggplot(aes(x=date, y=value, color=variable)) + geom_line() +\ntheme_bw() + theme +\nxlab(\"Date\") + ylab(\"New Single Cell Items\") + guides(color=guide_legend(title=\"Contribution Type\")) +\nggtitle(\"New Single Cell events, FAQs, news, slides, tutorials, videos and workflows in the GTN\")\nggsave(\"sc-files-cumulative.png\", width=6, height=6)\nWe may also want to know how many changes there have been since the start date of this study, e.g. October 1st, 2020:
\nsince_oct = added_by_time %>% dcast(date ~ <code>type</code>) %>%\narrange(date) %>%\nmutate(across(event:workflow, cumsum)) %>%\nfilter(date > as.Date(\"2020-10-01\"))\n# Pull out the first/last date as our start ane dnwend\nstart_date = (since_oct %>% head(n=1))$date\nend_date = (since_oct %>% tail(n=1))$date\n# Table of our changes.\nsince_oct %>%\nfilter(date == start_date | date == end_date) %>% # Just those rows\nmutate(date = case_when(date == start_date ~ 'start', date == end_date ~ 'end')) %>% # Relabel the start/end as literal string start and end\npivot_longer(-date) %>% pivot_wider(names_from=date, values_from=value) %>% # Transpose the data\nmutate(increase=end - start) %>% # And calculate our increase\nselect(name, increase) %>% arrange(-increase)\nGTN uses the Galaxy Europe Plausible server for collecting metrics (you can change your preferences in your GTN privacy preferences).\nWe can download the data from the server and plot it, filtering by our preferred start/end dates and filters (namely that page includes /topics/single-cell/).\nUnfortunately the data is downloaded as a zip file which we’ll then need to extract data from:
system(\"wget 'https://plausible.galaxyproject.eu/training.galaxyproject.org/export?period=custom&date=2024-11-07&from=2022-08-01&to=2024-11-07&filters=%7B%22page%22%3A%22~%2Ftopics%2Fsingle-cell%2F%22%7D&with_imported=true&interval=date' -O sc-stats.zip\")\nWe can use the unzip function to read a single file directly from the zip:
\nviews = read_csv(unzip(\"sc-stats.zip\", \"visitors.csv\"))\nWith that we’re ready to plot. We’re going to use a new feature for our plot,\nannotate. Annotation allows you to draw arbitrary features atop your plot, in\nthis case we’re going to draw rectangles to indicate outages and events that\nmight have affected our data.
y = 3300\nxoff = 3\nviews %>% group_by(date=floor_date(date, 'week')) %>%\nsummarise(date, visitors=sum(visitors), pageviews=sum(pageviews)) %>%\nfilter(visitors != 0 | pageviews != 0) %>%\nmelt(id.var=\"date\") %>%\nggplot(aes(x=date, y=value, color=variable)) + geom_line() +\nannotate(\"rect\", xmin = as.Date(\"2023-10-01\"), xmax = as.Date(\"2023-10-27\"), ymin = 0, ymax = y, alpha = .2) + # Outage\nannotate(\"rect\", xmin = as.Date(\"2023-05-22\") - xoff, xmax = as.Date(\"2023-05-26\") - xoff, ymin = 0, ymax = y, alpha = .2) + # Smorg3\nannotate(\"rect\", xmin = as.Date(\"2024-10-07\") - xoff, xmax = as.Date(\"2024-10-11\") - xoff, ymin = 0, ymax = y, alpha = .2) + # GTA\nannotate(\"rect\", xmin = as.Date(\"2024-09-16\") - xoff, xmax = as.Date(\"2024-09-20\") - xoff, ymin = 0, ymax = y, alpha = .2) + # Bootcamp\ntheme_bw() + theme +\nscale_y_continuous(expand = c(0, 0), limits = c(0, NA)) +\nxlab(\"Date\") + ylab(\"Count\") + guides(color=guide_legend(title=\"Metric\")) +\nggtitle(\"GTN Single Cell Visits\")\nggsave(\"sc-pageviews.png\", width=14, height=4)\nfor our lovely pageview plot!
\n![\"pageviews](\"./images/sc-pageviews.png\")
\n\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "cell_type": "markdown", "id": "final-ending-cell", "metadata": { "editable": false, "collapsed": false }, "source": [ "# Key Points\n\n", "\n# Congratulations on successfully completing this tutorial!\n\n", "Please [fill out the feedback on the GTN website](https://training.galaxyproject.org/training-material/topics/contributing/tutorials/meta-analysis-plot/tutorial.html#feedback) and check there for further resources!\n" ] } ], "etadata": { "kernelspec": { "display_name": "R", "language": "R", "name": "r" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "4.1.0" } } }Discuss this with us and we can perhaps generalise this analysis, to reduce the amount of data processing you need to do, and make it more accessible for everyone!
\n