Solutions Tutorial 1: Introduction to Econometrics and Data

Retrieval practice

  1. Panel data follow the same units over time; pooled cross-sections draw different units in different periods.
  2. Prediction asks how accurately we can forecast an outcome, whereas causation asks what would happen to the outcome under an intervention.
  3. Reverse causality and confounding are the two standard explanations.
  4. Untouched raw data let us undo mistakes and reproduce every transformation in code.
  5. Check that the row count behaves as expected, that all keys match, and that the intended key remains unique.

Solutions

1. From an economic question to a dataset

(a) The original question is descriptive: it asks what happened. A predictive version is, “Given GDP per capita and earlier unemployment, what will unemployment be next year?” A causal version is, “What would happen to unemployment if an intervention raised GDP per capita?”

(b) One row in the analysis table should represent one country in one year. The key is therefore country_code + year.

(c) This is panel data because the same four countries are observed repeatedly from 2015 through 2023.

(d) country_code is a stable identifier, country is a readable label, and year is the time variable. gdp_per_capita is an outcome measured in constant 2015 US dollars per person. unemployment is an outcome measured as a percentage of the labour force.

2. Inspect before transforming

(a) The expected row count is:

\[ 4\text{ countries}\times9\text{ years}\times2\text{ indicators}=72. \]

(b) Either workflow below shows 72 rows and 4 columns. country_code and indicator_code are text, year is integer-valued, and value is numeric.

library(tidyverse)

macro_url <- paste0(
  "https://empirical-economics.netlify.app/",
  "tutorials/datafiles/world_bank_macro_2015_2023.csv"
)
macro <- read_csv(macro_url)

dim(macro)
names(macro)
glimpse(macro)
slice_head(macro, n = 4)
import pandas as pd

macro_url = (
    "https://empirical-economics.netlify.app/"
    "tutorials/datafiles/world_bank_macro_2015_2023.csv"
)
macro = pd.read_csv(macro_url)

macro.shape
macro.columns.tolist()
macro.info()
macro.head(4)

The first statement assembles the file address as one string and passes it to the CSV reader, which downloads the file and returns a rectangular table stored under the name macro. The four calls that follow report, in turn, the number of rows and columns, the column names, the column types alongside the first few values, and the first four rows. None of them change macro; they only describe what was imported, which is what allows the observed row count to be compared with the 72 rows predicted in (a).

(c) unemployment_2023 is the object that receives the result. The filtering and selection functions take the imported data and stated conditions or column names as inputs; their output is a new rectangular dataset with four rows and three columns.

unemployment_2023 <- macro |>
  filter(
    year == 2023,
    indicator_code == "SL.UEM.TOTL.ZS"
  ) |>
  select(country_code, year, value)

unemployment_2023
unemployment_2023 = (
    macro
    .loc[
        lambda data:
            (data["year"] == 2023) &
            (data["indicator_code"] == "SL.UEM.TOTL.ZS"),
        ["country_code", "year", "value"],
    ]
)

unemployment_2023

The pipe passes macro to filter(), which keeps the rows satisfying both stated conditions, and then to select(), which keeps three of the four columns. Conditions separated by a comma are combined with a logical and, so a row survives only if it is from 2023 and carries the unemployment indicator code. The assignment stores the resulting table under a name, and writing that name on its own line prints it.

The result contains Belgium (5.547), Germany (3.071), France (7.335), and the Netherlands (3.535). The values are unemployment rates in percent, not proportions.

(d) In the long table, the key is country_code + year + indicator_code: a country-year appears twice because it has two indicators.

macro |>
  summarise(
    rows = n(),
    missing_values = sum(is.na(value)),
    duplicate_keys = n() - n_distinct(
      country_code, year, indicator_code
    )
  )
key = ["country_code", "year", "indicator_code"]

pd.Series({
    "rows": len(macro),
    "missing_values": macro["value"].isna().sum(),
    "duplicate_keys": macro.duplicated(key).sum(),
})

summarise() collapses the whole table into a single row of summary statistics. n() counts the rows; sum(is.na(value)) counts the missing entries in value by summing a logical vector in which TRUE counts as one; and the last expression subtracts the number of distinct combinations of the three key columns from the row count. A positive difference would mean that some key occurs more than once, so a zero confirms that the intended key is unique.

The results are 72 rows, 0 missing values, and 0 duplicate keys.

(e) A zero means that the measured unemployment rate is zero. A missing value means that the rate was not observed. Replacing the latter with the former invents an observation and biases summaries such as the average unemployment rate.

3. Give variables meaning and validate a join

(a) The recoding creates readable analysis names while retaining indicator_code, which links each variable back to its source definition and unit.

(b) The declared relationship makes the join fail if the metadata key is not unique. Both steps are shown together below.

macro_clean <- macro |>
  mutate(
    year = as.integer(year),
    indicator = recode(
      indicator_code,
      "NY.GDP.PCAP.KD" = "gdp_per_capita",
      "SL.UEM.TOTL.ZS" = "unemployment"
    )
  ) |>
  arrange(country_code, year, indicator)

metadata_url <- paste0(
  "https://empirical-economics.netlify.app/",
  "tutorials/datafiles/world_bank_country_metadata.csv"
)
country_metadata <- read_csv(metadata_url)

macro_named <- macro_clean |>
  left_join(
    country_metadata,
    by = join_by(country_code),
    relationship = "many-to-one"
  )
indicator_names = {
    "NY.GDP.PCAP.KD": "gdp_per_capita",
    "SL.UEM.TOTL.ZS": "unemployment",
}

macro_clean = (
    macro
    .assign(
        year=lambda data: data["year"].astype("int64"),
        indicator=lambda data:
            data["indicator_code"].map(indicator_names),
    )
    .sort_values(["country_code", "year", "indicator"])
)

metadata_url = (
    "https://empirical-economics.netlify.app/"
    "tutorials/datafiles/world_bank_country_metadata.csv"
)
country_metadata = pd.read_csv(metadata_url)

macro_named = macro_clean.merge(
    country_metadata,
    on="country_code",
    how="left",
    validate="many_to_one",
)

mutate() adds or overwrites columns: it stores year as an integer and creates indicator, in which recode() translates each source code into an analysis name. arrange() then sorts the rows by country, year, and indicator, so the ordering is set by the code rather than by the order of the download. The metadata file is imported in the same way as the macro file, and left_join() attaches its columns wherever country_code matches. The argument relationship = "many-to-one" states the expected structure: many macro rows may match one metadata row, but a metadata key that appears twice raises an error instead of silently duplicating rows.

(c) The validation should show 72 rows before, 72 rows after, and 0 missing country names.

tibble(
  rows_before = nrow(macro_clean),
  rows_after = nrow(macro_named),
  unmatched_names = sum(is.na(macro_named$country))
)
pd.Series({
    "rows_before": len(macro_clean),
    "rows_after": len(macro_named),
    "unmatched_names": macro_named["country"].isna().sum(),
})

tibble() builds a one-row table from three computed numbers: the row count before the join, the row count after it, and the number of rows whose country is missing. Equal row counts show that the join added no rows, and zero missing names show that every country_code found a match. Both checks are needed, because a join can leave the row count unchanged and still fail to match some units.

(d) The Netherlands contributes \(9\times2=18\) indicator rows. A second metadata match duplicates those 18 rows, so an unchecked join produces \(72+18=90\) rows. The Netherlands would then receive twice the intended weight in totals, means, plots, and regressions. A declared many-to-one join instead fails and exposes the bad metadata key.

4. Reshape to the intended unit of observation

(a) The pivot below produces one row per country-year.

analysis_data <- macro_named |>
  select(country_code, country, region, year, indicator, value) |>
  pivot_wider(
    names_from = indicator,
    values_from = value
  ) |>
  arrange(country, year)
analysis_data = (
    macro_named
    .pivot(
        index=["country_code", "country", "region", "year"],
        columns="indicator",
        values="value",
    )
    .reset_index()
    .rename_axis(columns=None)
    .sort_values(["country", "year"])
)

select() keeps the identifying columns together with indicator and value, and drops indicator_code, which differs between the two rows of a country-year and would therefore prevent them from being combined. pivot_wider() takes the names of the new columns from indicator and their contents from value, so the two rows describing one country-year become one row with a gdp_per_capita column and an unemployment column; the columns that were not named in either argument identify that row. arrange() orders the result by country and year.

(b) The wide-table key is country_code + year. The checks below test the row count, the number of unique keys, missing values in both outcomes, and unemployment values outside 0–100. We expect 36 rows, 36 unique country-year keys, and zero on the three remaining counts.

analysis_data |>
  summarise(
    rows = n(),
    unique_country_years = n_distinct(country_code, year),
    missing_gdp = sum(is.na(gdp_per_capita)),
    missing_unemployment = sum(is.na(unemployment)),
    impossible_unemployment = sum(
      unemployment < 0 | unemployment > 100
    )
  )
pd.Series({
    "rows": len(analysis_data),
    "unique_country_years":
        analysis_data[["country_code", "year"]]
        .drop_duplicates().shape[0],
    "missing_gdp":
        analysis_data["gdp_per_capita"].isna().sum(),
    "missing_unemployment":
        analysis_data["unemployment"].isna().sum(),
    "impossible_unemployment":
        (~analysis_data["unemployment"].between(0, 100)).sum(),
})

The same summarise() pattern now tests the structure the wide table is supposed to have. The row count and the number of distinct country_code-year pairs must agree, since one row per country-year means no key is repeated. The two is.na() counts confirm that the pivot left no empty cells, which would appear if an indicator were absent for some country-year. The final expression counts unemployment rates outside the range the definition permits, which catches values that no rate can take, such as a negative number or a figure above 100.

(c) The results are 36 rows, 36 unique keys, 0 missing GDP values, 0 missing unemployment values, and 0 impossible unemployment values.

(d) The four cases are:

  1. panel data;
  2. a cross-section;
  3. a time series;
  4. pooled cross-sections.

The decisive distinction in case 4 is that the sampled households are not the same units each year.

5. Describe only what the data show

The following code computes the requested comparison.

comparison_wide <- analysis_data |>
  filter(year %in% c(2015, 2023)) |>
  select(country, year, unemployment, gdp_per_capita) |>
  pivot_wider(
    names_from = year,
    values_from = c(unemployment, gdp_per_capita)
  )

comparison <- comparison_wide |>
  mutate(
    unemployment_change_pp =
      unemployment_2023 - unemployment_2015,
    gdp_change_percent = 100 * (
      gdp_per_capita_2023 / gdp_per_capita_2015 - 1
    )
  ) |>
  select(
    country,
    unemployment_2015,
    unemployment_2023,
    unemployment_change_pp,
    gdp_change_percent
  ) |>
  arrange(unemployment_2023)

comparison
comparison = (
    analysis_data
    .pivot(
        index="country",
        columns="year",
        values=["gdp_per_capita", "unemployment"],
    )
)

comparison = pd.DataFrame({
    "unemployment_2015":
        comparison[("unemployment", 2015)],
    "unemployment_2023":
        comparison[("unemployment", 2023)],
    "unemployment_change_pp":
        comparison[("unemployment", 2023)] -
        comparison[("unemployment", 2015)],
    "gdp_change_percent": 100 * (
        comparison[("gdp_per_capita", 2023)] /
        comparison[("gdp_per_capita", 2015)] - 1
    ),
}).sort_values("unemployment_2023")

comparison

The first pipeline keeps the two endpoint years, retains the two outcomes, and pivots year into the column names, so each country occupies one row with a 2015 and a 2023 column for each outcome. The second block computes the change in unemployment as a difference between two percentages, which is measured in percentage points, and the change in GDP per capita as a ratio minus one multiplied by 100, which is measured in percent. The two changes are expressed in different units and cannot be compared as plain numbers. select() then fixes the order of the reported columns and arrange() sorts the countries by their 2023 unemployment rate.

Rounded to three decimals, the output is:

Country Unemployment 2015 Unemployment 2023 Change (percentage points) GDP per capita change (%)
Germany 4.612 3.071 -1.541 5.817
Netherlands 6.872 3.535 -3.337 11.283
Belgium 8.482 5.547 -2.935 9.621
France 10.354 7.335 -3.019 7.130

(a) Germany had the lowest 2023 unemployment rate, at 3.071% of the labour force.

(b) The Netherlands had the largest decline, 3.337 percentage points.

(c) The Netherlands had the largest increase in constant-price GDP per capita, approximately 11.283%.

(d) One accurate description is: “Between 2015 and 2023, unemployment fell in all four countries, by between 1.541 and 3.337 percentage points. Over the same period, constant-2015-dollar GDP per capita rose in every country, by between 5.817% and 11.283%.” The phrase “over the same period” reports timing without claiming that one change caused the other.

6. Why the pattern does not identify an effect

(a) Reverse causality is plausible because lower unemployment raises household income and production, which may increase GDP per capita. The arrow can therefore run from unemployment to GDP as well as in the proposed direction.

(b) A common shock is a plausible confounder. For example, the pandemic and associated restrictions affected both production and labour markets. Policy responses, technological change, or external demand could also move both outcomes.

(c) Yes. Stable co-movement can improve forecasts because prediction needs a pattern that works out of sample; it does not require the predictor to be the cause. Causal inference instead needs variation in GDP per capita that is unrelated to all other determinants of unemployment.

(d) One possibility is a policy or external shock that changes economic activity for some otherwise comparable places but not others. The design must construct a credible counterfactual: treated and comparison units should have had similar unemployment paths in the absence of the change. Random assignment would do this by design; a convincing natural experiment would need an argument and evidence that the source of variation is as-if random.

ImportantThe warranted conclusion

The table establishes a descriptive pattern. It does not estimate what unemployment would have been if GDP per capita had followed a different path.

Self-study

7. Audit a fragile workflow

(a) The workflow contains four distinct threats: the only raw file is edited, blanks are replaced with zero, country names are matched by row position, and the operations are not recorded.

(b) Each threat and its replacement:

Threat Why it matters Replacement
Editing the only raw file The original evidence is lost and mistakes cannot be undone Preserve an untouched snapshot in data/raw/
Replacing blanks with zero Missing and measured zero have different meanings Keep missing values explicit and document any justified treatment
Matching names by row position Sorting either file silently mismatches countries Join on country_code and validate the relationship
Unrecorded spreadsheet operations Nobody can reproduce or audit the transformations Put every transformation and check in a script

The vague filename is a fifth warning: a processed file should be generated reproducibly, with its source and creation process documented, rather than serving as an unexplained “final” artefact.

(c) A suitable minimal structure is:

my_project/
├── README.md
├── data/
│   ├── raw/
│   │   └── world_bank_macro_2015_2023.csv
│   └── processed/
│       └── macro_analysis.csv
├── code/
│   └── prepare_data.R  (or prepare_data.py)
└── output/
    └── figures/
        └── macro_trends.png

The source, retrieval date, variable definitions, units, and run instructions belong in README.md. The processed CSV and figure are outputs that the code should be able to recreate.

8. Diagnose a failed pivot

(a) The error reveals that country_code + year + indicator is not unique, even though the intended long-table design requires it to be unique.

(b) Taking the first value hides a conflict and makes the answer depend on row order. Taking the mean invents an aggregation rule that may mix revisions, sources, units, or records duplicated at the source. Either choice changes the data without first identifying the cause.

(c) A defensible sequence is:

  1. Count duplicates using the full intended key.
  2. Display every row belonging to duplicate keys, including all source columns.
  3. Check whether the rows are exact duplicates or disagree in value, unit, source, or retrieval date.
  4. Return to the source documentation and import code to determine how the duplicates arose.
  5. State and document a substantive resolution rule.
  6. Apply the rule in code, test uniqueness again, and only then pivot.

Takeaways

What should you be able to do now?

  1. Translate an economic question into a unit of observation, variables, and a key.
  2. Inspect raw data and test missingness and uniqueness before transforming them.
  3. Join and reshape data while making structural expectations explicit.
  4. Distinguish a descriptive pattern, a useful prediction, and a credible causal effect.
  5. Preserve a reproducible path from raw inputs to communicated results.