At this point, you can set up an R project, import a data file into RStudio, and quickly explore the dataset’s structure, which is typically the first step in any analysis. If the dataset is well-structured, the next crucial step is to manipulate the data in R to obtain the information we need. Often, we are not interested in the entire dataset; instead, we focus on selecting specific variables, analyzing certain groups, and observing how these variables vary across those groups
With R, there are several ways to perform these operations. Firstly, you can use R’s base functions, without having to load a specific package. An alternative approach is based on the tidyverse, the package suite developed by Hadley Wickham with a special syntax. Here, we’ll concentrate on the tidyverse functions, which I feel are easier and more intuitive to use.
4.1 Introduction to dplyr functions
To manipulate data, we will use a package from the tidyverse called dplyr, comprising a comprehensive suite of exceptionally useful functions. To familiarize ourselves with its usage, we will explore the Quality of Government Environmental Indicators dataset. To do this, we will need first to load the tidyverse and import the data with the haven package.
4.1.1 Count the observations of groups with count()
One basic operation in data manipulation is to count the number of observations of different values in a variable. The dplyr::count() function is used for this purpose.
In Exercise 1, we observed that the qog dataset has a country-year structure. For each country, there is a series of indicators, with one value per year. The dataset includes a year variable and a cname variable representing the country.
We might want to know how many country observations we have for each year. To do this, we can use count() with the year variable, that will give us the number of observations for each year. If we look at the results, we see that the dataset has observations from 1946 to 2020 with an increasing number of countries in the dataset over time.
If we want to cross the year and cname variables, we can use the count() function with both variables. This will give us the number of observations for each year and country. In this case, we see that we have only one observation for each country-year combination.
# A tibble: 11,722 × 3
year cname n
<dbl> <chr> <int>
1 1946 Afghanistan 1
2 1946 Albania 1
3 1946 Andorra 1
4 1946 Antigua and Barbuda 1
5 1946 Argentina 1
6 1946 Australia 1
7 1946 Belgium 1
8 1946 Bhutan 1
9 1946 Bolivia (Plurinational State of) 1
10 1946 Brazil 1
# ℹ 11,712 more rows
4.1.2filter() observations
We may want to use only a subset of countries and get a smaller versions of the dataset. And the tidyverse has one for this, which is called filter() We want to select specific rows/observations of the dataset based on a specific conditions. To filter, we will use extensively boolean operators.
Operator
Description
==
equal
!=
not equal
<
less than
<=
less than or equal
>
greater than
>=
greater than or equal
&
and
|
or
!
not
%in%
in the set
For instance, I might be only interested in environmental indicators for France. In that case I could filter the whole dataset with only the observations that has France as country variable.
# Filter for countries and year and save in a new object ! qog_subset<-filter(qog, cname%in%c("France", "Norway", "Spain", "Denmark"), year>1990)count(qog_subset, cname)
# A tibble: 4 × 2
cname n
<chr> <int>
1 Denmark 30
2 France 30
3 Norway 30
4 Spain 30
Often however, we need several operations to be done together. To do so with the tidyverse functions, we can use something that is called the pipe : |>. You will quickly understand what the pipe is, we use it to chain instructions just as in a recipe. Depending on the version of R, you might also use/see this %>%. The shortcut for it is Ctrl + Shift + M or Cmd + Shift + M on Mac.
Let’s say we want to count the number of observations we have on the variable wvs_pedp that measure the share of the population participating in environmental protests for Sweden. We can use the pipe to do it by chaining the filter() and count() functions rather than doing it separately in different steps.
qog|># Keep only countries filter(cname=="Sweden")|># Count the number of observations for each countrycount(wvs_pedp)# Use Sort = TRUE if you want to sort the results
# A tibble: 2 × 2
wvs_pedp n
<dbl> <int>
1 5.22 1
2 NA 74
4.1.3 Your turn
Count the countries and year where the share of the population participating in environmental protests was greater than 10 (measured by the wvs_pedp variable).
4.2select() variables
While filter() is useful for keeping only certain groups of rows, select() is used, as its name suggests, to select certain variables (columns) from our dataframe.
Let’s say I’m interested in CO2 emissions per capita, their overall level and how they vary over time and by country. For this, I would need only three variables that are present in the dataset: year, cname and the variable wdi_co2 which comes from the World Bank’s World Development Indicators. To do this, I use select() and just specify which variables of the dataset I want to select.
4.3 Calculating statistics by group with group_by() and summarise()
To compute a first set of descriptive statistics, we could look at the central indicators of the co2 emissions variable (wdi_co2).
mean(qog_subset$wdi_co2, na.rm =TRUE)# Compute the mean
[1] 7.666003
median(qog_subset$wdi_co2, na.rm =TRUE)# Compute the median
[1] 7.496984
sd(qog_subset$wdi_co2, na.rm =TRUE)# Compute the standard deviation
[1] 2.003383
However, we are often interested in how these indicators vary across groups such as country or year. To do so, use the summarise() function in R that allows you to compute new variables by groups. To choose grouping variable, we use first group_by() where we specify for which group we want to compute something. Here I use this function to compute the mean, the median, the standard deviation and the first and third quartile of co2 emissions per capita for each country. This gives us a new tibble with all of the summary information.
It is also possible to sort the results to gain a clearer idea of which countries have the highest average CO2 emissions per capita. To achieve this, we utilize the arrange() function, specifying the variable by which we intend to sort.
When were the CO2 emissions per capita the lowest in Saudi Arabia?
4.4.1 Mutate
We can also create new variables based on other ones with mutate(). Let’s say I want to compute the growth rate of co2 emissions per capita every year. For this, I will use mutate() to create new variables.
# A tibble: 75 × 5
cname year wdi_co2 co2_lag co2_growth
<chr> <dbl> <dbl> <dbl> <dbl>
1 France 1963 6.88 NA NA
2 France 1964 7.01 6.88 2.02
3 France 1965 7.06 7.01 0.695
4 France 1966 6.90 7.06 -2.32
5 France 1967 7.33 6.90 6.19
6 France 1968 7.52 7.33 2.59
7 France 1969 8.01 7.52 6.64
8 France 1970 8.45 8.01 5.42
9 France 1971 8.83 8.45 4.50
10 France 1972 9.11 8.83 3.13
# ℹ 65 more rows
4.5 Recode values with case_when
Mutate() is also widely used to recode variable. Here we create a new trajectory variable and we use case_when() to recode whether there co2_emissions are growing or not for a given year.
# A tibble: 75 × 6
cname year wdi_co2 co2_lag co2_growth trajectory
<chr> <dbl> <dbl> <dbl> <dbl> <chr>
1 France 1963 6.88 NA NA <NA>
2 France 1964 7.01 6.88 2.02 Bad
3 France 1965 7.06 7.01 0.695 Bad
4 France 1966 6.90 7.06 -2.32 Good
5 France 1967 7.33 6.90 6.19 Bad
6 France 1968 7.52 7.33 2.59 Bad
7 France 1969 8.01 7.52 6.64 Bad
8 France 1970 8.45 8.01 5.42 Bad
9 France 1971 8.83 8.45 4.50 Bad
10 France 1972 9.11 8.83 3.13 Bad
# ℹ 65 more rows
# A tibble: 3 × 2
trajectory n
<chr> <int>
1 Bad 22
2 Good 31
3 <NA> 22
4.5.1 Your turn !
Create a century variable that indicates whether the year is in the 20th or 21st century. Find the countries that have the highest total number of deaths due to natural disasters for each century, as measured by the emdat_ndeath variable.