Solutions Tutorial 6: Difference-in-Differences (Finance Track)

Retrieval practice

  1. The mean outcome of the treated and of the comparison group, each before and after the event; the DiD improves on the after-only comparison between the groups and on the before-and-after comparison within the treated group.
  2. That without the treatment the two groups’ outcomes would have changed by the same amount on average; leads only describe the years before treatment, are often imprecise, and say nothing about the untreated path after treatment.
  3. Each firm’s own relation with the market, estimated in a window before the event: the market model predicts the firm’s normal return, and the abnormal return is the actual return minus that prediction.
  4. All assets react to the same news on the same day, so their returns are correlated across assets and more assets add little independent information; collapse to one long-short return per day and use time-series inference, or compare the estimate with the same estimator on placebo dates.
  5. Using an already-treated cohort as the control for a cohort treated later, whose outcome then contains treatment effects.

Solutions

1. Book leverage of Compustat firms by headquarters state

a) The two lines

book_lev   = (dltt + dlc) / at
good_faith = as.integer(!is.na(gf_year) & fyear >= gf_year)

In the code, firm-years in which dltt, dlc or dvc is missing are dropped before debt and the dividend dummy are computed. A missing item means that Compustat has no value for that year; setting it to zero would record the firm as having no debt or paying no dividend, which the data do not show (Lecture 1). Serfling likewise keeps only firm-years with “nonmissing data for the main variables of interest” (p. 2255). The rule drops 138 firm-years. The >= counts the adoption year itself as treated, which is why the event study’s year \(0\) is a treated year and year \(-1\) the reference.

b) Book or market leverage

Book leverage is what managers say they set when they make capital structure decisions (Graham and Harvey 2002), and it moves only when debt or the book value of assets changes. Much of the variation in market leverage comes from share prices rather than from debt policy (Welch 2004). Market leverage is closer to the target leverage of capital structure theory, which is defined on market values.

Market leverage can change on the day the law is adopted without any change in debt: if the news changes the value of the firm’s equity, the denominator moves. An estimate on market leverage then mixes the debt response with the stock market’s valuation of the law.

c) Current headquarters state

The firm is coded as Californian in every year, so it counts as treated from 1980, while in 1975-1984 it was in Nevada, which adopts the exception only in 1987. Firms coded in the wrong state put treated firms in the control group and untreated firms in the treated group. Both make the two groups more alike, so the estimate moves towards zero. Serfling (2016, footnote 12) discusses this limitation of the Compustat header.

d) The dropped firm-years

The 1,217 firm-years belong to firms observed in a single year. The firm fixed effect fits such an observation exactly, so its residual is zero and it adds no within-firm variation from which \(\delta\) could be estimated. fixest and pyfixest drop these singletons and report it.

2. Leverage in California and Connecticut after 1980

a) The four means

1975-1979 1980-1984 Change
California, Connecticut \(0.2478\) \(0.2393\) \(-0.0085\)
Never-adopting states \(0.2455\) \(0.2435\) \(-0.0020\)
Difference in differences \(-0.0065\)

Book leverage of firms in California and Connecticut fell by 0.65 percentage points more than that of firms in never-adopting states, about 2.5% of mean book leverage. The sign is the one Serfling predicts. With a standard error clustered by state of \(0.0048\) (\(t=-1.36\)) the estimate is not significant. The table has no controls; with Serfling’s controls on the same firms the estimate is \(-0.0127\) (\(t=-2.52\)).

b) Entry and exit

With all firm-years, a firm enters the means only in the years in which it is in Compustat. Firms that enter Compustat after 1979 appear only in the second column, firms that disappear before 1980 only in the first, and other firms appear in more years in one column than in the other. A change in the mean then combines changes within firms with a change in which firms are averaged. Firms that enter in 1980-1984, for example, have lower book leverage than the incumbents: \(0.235\) against \(0.257\) in California and Connecticut, and \(0.242\) against \(0.259\) in the never-adopting states. The regression with firm fixed effects compares each firm only with itself.

In the table above every firm contributes five years to each column, so each change in the mean is the average change within firms, and the four-means DiD equals the estimate with firm and year fixed effects exactly. The code checks this: the regression returns \(-0.0065\).

c) Other controls

States that adopt after 1984 are untreated throughout 1975-1984 and are valid controls: Arizona and Oklahoma (1985), Nevada (1987), Idaho and Utah (1989), Delaware (1992) and Wyoming (1994). Four states must not be used. New Hampshire (1974) is treated in all ten years and Massachusetts (1977) from the third pre-adoption year, so their leverage changes include their own treatment effects. Montana (1982) and Alaska (1983) adopt during the post-adoption years, so their change from before to after mixes untreated and treated years.

3. An earlier cohort as the control

a) Two DiD estimates

1980-1984 1985-1989 Change
Arizona, Oklahoma \(0.3322\) \(0.3075\) \(-0.0247\)
Never-adopting states \(0.2327\) \(0.2611\) \(0.0284\)
California, Connecticut \(0.2164\) \(0.2455\) \(0.0291\)

Against the never-adopting states the DiD is \(-0.0247-0.0284=-0.0531\); against California and Connecticut it is \(-0.0247-0.0291=-0.0538\). The comparison with the never-adopting states is clean. The comparison with California and Connecticut uses firms that are treated in both periods as controls, a comparison that conventional two-way fixed effects also makes. Both estimates rest on 28 firms; the clean one has a standard error clustered by state of \(0.024\).

b) The decomposition

California and Connecticut against the never-adopting states over the same years: \(0.0291-0.0284=0.0007\). Writing \(\Delta\) for each group’s change,

\[ \underbrace{(\Delta_{AZ,OK}-\Delta_{CA,CT})}_{-0.0538} =\underbrace{(\Delta_{AZ,OK}-\Delta_{never})}_{-0.0531} -\underbrace{(\Delta_{CA,CT}-\Delta_{never})}_{0.0007}. \]

California and Connecticut are treated in both periods. Under parallel trends, their DiD against the never-adopting states measures how their own treatment effect changed between years 0-4 and years 5-9 after adoption. The comparison with an earlier adopter therefore subtracts that change from the effect on Arizona and Oklahoma.

c) When the comparison is biased

The comparison is biased when the earlier cohort’s treatment effect changes between the two periods. If the leverage of firms in California and Connecticut were still falling in 1985-1989 because of their own adoption in 1980, their change would be lower, and subtracting it would move the estimate for Arizona and Oklahoma towards zero, possibly past it. For these firms, without controls, the effect hardly changes between the two periods (\(0.0007\)), so the comparison does almost no harm here.

4. What a newer estimator says about Serfling’s estimate

a) The leads

The leads lie close to zero with confidence intervals of about \(\pm0.015\) to \(\pm0.02\), as wide as the effects after adoption, and all of them include zero. That is consistent with parallel trends in the years before adoption, but it does not establish it. The intervals are wide enough to hide pre-trends of the size of the effect, and parallel trends concerns the years after adoption, which the leads cannot show. The two-way fixed-effects \(\le-5\) coefficient (\(-0.013\)) also hints that leverage in adopting states was lower long before adoption.

b) The static estimate

The static model compares the post-adoption years of firms in adopting states with all their pre-adoption years, back to 1967. The event studies compare with the year before adoption, and the years five or more before adoption get their own coefficient. That coefficient is \(-0.013\): long before adoption, leverage in adopting states was lower than just before it. In the static model these years lower the pre-adoption baseline, so the measured fall after adoption is smaller.

c) The firms in each mean

It is the entry and exit of firms from Question 2(b). The two versions estimate different quantities under different assumptions.

  • Callaway-Sant’Anna on the unbalanced panel estimates the effect on the average leverage of the Compustat firms headquartered in the adopting states, whoever they are in each year. It needs parallel trends for these changing groups: without the law, firms entering and leaving would have moved the treated and the control means in the same way.
  • The same-firm version estimates the effect on firms present in both years. It needs parallel trends only for these firms, and it conditions on firms remaining in Compustat, which the law could also affect.

The leads favour the first version: they are flat there, while the same-firm version shows leverage in adopting states rising towards adoption relative to never-adopting firms. The version that passes the check is the one that relies on the assumption about entry and exit, and the data cannot tell whether its \(-0.012\) reflects the law or entry and exit that would have happened anyway.

5. Which controls belong in the regression?

a) Fixed before the law, or a response to it?

The implied-contract and public-policy dummies are other court decisions. They are not chosen by the firm, and they are included because they may coincide with the good-faith exception. The firm-level controls are measured in the same year as leverage, so all of them could respond to the law. In the second table, log assets, market-to-book, the dividend-payer share and \(Z\) change after adoption, while profitability and fixed assets barely move. On its own, the table does not show that the law caused these changes, but it shows that these controls are not fixed with respect to the treatment.

b) The Altman Z-score

Book leverage contains dlc, and dlc lowers working capital and therefore \(Z\). A regression of leverage on \(Z\) is therefore partly a regression of leverage on itself.

The channel through which the law could lower \(Z\) is Serfling’s own mechanism: firing costs make labour costs more fixed, which raises operating leverage and the risk of financial distress. If so, \(Z\) lies on the path from the law to leverage, and controlling for it removes the part of the effect that runs through it. Here that part is positive: \(Z\)’s coefficient is about \(-0.04\), so a fall of \(0.03\) raises predicted leverage by about \(0.001\). Removing it makes the estimate more negative, as in the first table: \(-0.0094\) without controls, \(-0.0106\) with \(Z\) alone.

c) Include it or not?

  • Include. Distress risk varies across states and years for reasons unrelated to the law, such as regional recessions, and it affects leverage. Then \(Z\) confounds the comparison of trends, and parallel trends is more plausible conditional on it.
  • Leave out. \(Z\) responds to the law and is built partly from the outcome. Controlling for it then removes part of the effect and adds a mechanical relation with leverage.
  • Report. The estimate without controls, with all controls and without \(Z\), and a version with \(Z\) measured before adoption or without its working-capital term. Serfling (footnote 21) reports that his results hold when the EBIT term is removed from \(Z\), because EBIT over assets overlaps with profitability.

Self-study

6. What did the election do to clean energy?

a) The DiD and the counterfactual

\[ \begin{aligned} \widehat{\Delta}_{\text{clean}}&=-3.85-(-1.77)=-2.08,\\ \widehat{\Delta}_{\text{SPY}}&=2.76-(-0.07)=2.83,\\ \widehat{\tau}_{DiD}&=-2.08-2.83=-4.91, \end{aligned} \]

or \(-4.92\) from the unrounded means. In the week after the election, clean energy fell about 4.9 percentage points relative to the market. Parallel trends sets clean energy’s untreated change equal to the market’s, so without the news it would have stood at \(-1.77+2.83=1.06\%\); it stood at \(-3.85\%\).

b) The interaction model

Coefficient Estimate Clustered SE Cell or change in the table
Intercept, \(\beta_0\) \(-0.07\) \(0.00\) SPY before
treated, \(\beta_1\) \(-1.69\) \(2.23\) Clean energy minus SPY before
post, \(\beta_2\) \(2.84\) \(0.00\) SPY’s change
treated:post, \(\beta_3\) \(-4.92\) \(1.66\) The DiD

The model has four coefficients for four cells, so it fits the cell means exactly. SPY is a single cluster whose residuals sum to zero in each cell, so the standard errors of \(\beta_0\) and \(\beta_2\) are zero, and the standard error of \(\beta_3\) measures only how much the twelve clean-energy assets differ from each other. Question 8 takes this up.

Code 
library(dplyr)
library(fixest)

returns <- read.csv(
  "https://empirical-economics.netlify.app/tutorials/datafiles/clean_energy_2016_daily.csv"
)

# (2) the interaction model, clustered by ticker
event_window <- returns |>
  filter(group == "clean" | ticker == "SPY", k >= -5, k <= 4) |>
  arrange(ticker, k) |>
  group_by(ticker) |>
  mutate(cum = cumsum(ret)) |>
  ungroup() |>
  mutate(treated = as.integer(group == "clean"),
         post = as.integer(k >= 0))

feols(cum ~ treated * post, data = event_window, cluster = ~ticker)
Code 
import pandas as pd
import pyfixest as pf

returns = pd.read_csv(
    "https://empirical-economics.netlify.app/tutorials/datafiles/clean_energy_2016_daily.csv"
)

# (2) the interaction model, clustered by ticker
event_window = (
    returns.query("(group == 'clean' or ticker == 'SPY') and -5 <= k <= 4")
    .sort_values(["ticker", "k"])
    .assign(
        cum=lambda d: d.groupby("ticker")["ret"].cumsum(),
        treated=lambda d: (d["group"] == "clean").astype(int),
        post=lambda d: (d["k"] >= 0).astype(int),
    )
)
pf.feols("cum ~ treated * post", data=event_window, vcov={"CRV1": "ticker"}).summary()

7. Which market is the right comparison?

b) What the pre-election spreads show

Against SPY, clean energy fell 0.33 points a day more than the market over the 20 days before the election, about 6.7 points in total (\(t=-2.18\)), so the two groups were not moving together; part of the drift may be anticipation as the polls moved. Against IWO the pre-election spread is small and insignificant, which supports IWO as the comparison.

It does not show that parallel trends holds after the election. A shock that starts on the same day as the treatment leaves no trace in the pre-election data, and the election also changed expectations about taxes, trade and interest rates. A \(t\)-statistic of \(-0.66\) over 20 days is also consistent with a moderate pre-election difference.

c) The 2020 election and the vaccine

The vaccine announcement threatens parallel trends. It moved investors from growth to cyclical and value stocks on the first trading day after the call, so the groups’ untreated changes differ in exactly the post-election window. Every window that contains 9 November contains both events; the window that ends before it, 4 to 6 November, ends before the race was called and measures the changing odds during the count. The comparison with fossil energy swings most, from \(+10.06\) to \(-15.33\), because the rotation lifted that cyclical sector far more than the broad market. A pre-election check would not have detected the problem: the paths before the election can be parallel and still say nothing about a shock that begins together with the treatment.

8. Standard errors when every asset sees the same news

a) A common shock does not average out

\[ \operatorname{Var}\Big(\frac1N\sum_{i=1}^{N}r_i\Big) =\operatorname{Var}(c)+\frac1{N^2}\sum_{i=1}^{N}\operatorname{Var}(e_i) =\sigma_c^2+\frac{\sigma_e^2}{N}. \]

As \(N\) grows, the second term vanishes but the first does not: more assets average away the asset-specific noise, not the common shock. A standard error computed from the spread across assets uses \(r_i-\bar r=e_i-\bar e\), in which \(c\) cancels, so it reflects only \(\sigma_e^2/N\). With a single event, the sample contains one draw of \(c\), and nothing in the cross-section measures its variance.

b) The same standard error for every benchmark

The benchmark is a single ticker and therefore a single cluster. The saturated model fits its two cells exactly, so its residuals sum to zero and it contributes nothing to the clustered variance. The standard error depends only on how much the twelve clean-energy assets’ changes differ from each other, which is the same whichever benchmark is subtracted. The benchmark’s own change is treated as known, and the common movement of clean energy relative to the market in that week, \(c\) in (a), is not measured at all.

c) Newey-West on the long-short series

Benchmark Estimate Newey-West standard error \(t\)
SPY \(-4.92\) \(3.25\) \(-1.51\)
IWO \(-9.84\) \(3.07\) \(-3.20\)

The time series of daily long-short returns contains many ordinary weeks, so it measures how much a weighted ten-day sum of these returns moves when nothing happens. That variation includes the common component that the cross-section cannot see.

d) Placebo dates

Benchmark Estimate Placebo estimates Placebo SD Share at or below the estimate
SPY \(-4.92\) 250 \(3.38\) \(10.4\%\)
IWO \(-9.84\) 250 \(3.17\) \(0\%\)

Newey-West and the placebo dates agree (\(3.25\) against \(3.38\), \(3.07\) against \(3.17\)), and both are about twice the clustered standard error. Against SPY, a shortfall of 4.9 points or more occurs in about one ordinary week in ten: weak evidence that the election moved clean energy relative to the S&P 500. Against IWO, the estimate lies below all 250 placebo estimates, with \(t=-3.2\). Both methods use ordinary weeks. If the weeks around an election are more volatile, they understate the variation of the estimate without an effect.

Code 
library(dplyr)
library(tidyr)

returns <- read.csv("https://empirical-economics.netlify.app/tutorials/datafiles/clean_energy_2016_daily.csv")

# tent weights: the DiD of the four means as a weighted sum of daily long-short returns
weights <- c(0, 0.2, 0.4, 0.6, 0.8, 1, 0.8, 0.6, 0.4, 0.2)   # k = -5, ..., 4

for (benchmark in c("SPY", "IWO")) {
  # one long-short return per day: clean-energy average minus the benchmark
  long_short <- returns |>
    filter(group == "clean" | ticker == benchmark) |>
    summarise(ret = mean(ret), .by = c(group, date, k)) |>
    pivot_wider(names_from = group, values_from = ret) |>
    mutate(ls = clean - market) |>
    arrange(k)

  did_at <- function(centre) {
    window <- filter(long_short, k >= centre - 5, k <= centre + 4)
    sum(weights * window$ls)
  }
  estimate <- did_at(0)

  # (c) Newey-West: variance of the weighted sum, w' Omega w, with Omega built from
  # the Bartlett-weighted autocovariances (lag 5) of the daily long-short return
  # over the estimation year (October 2015 to September 2016)
  e <- long_short$ls[long_short$date >= "2015-10-01" & long_short$date <= "2016-09-30"]
  autocov <- drop(acf(e, lag.max = 9, type = "covariance", plot = FALSE)$acf)
  bartlett <- pmax(1 - (0:9) / 6, 0)
  omega <- toeplitz(autocov * bartlett)
  nw_se <- sqrt(drop(t(weights) %*% omega %*% weights))

  # (d) placebo dates: the same DiD centred on every day from k = -275 to k = -26
  placebo <- sapply(-275:-26, did_at)

  cat(benchmark, "| estimate", round(estimate, 2), "| Newey-West se", round(nw_se, 2),
      "| placebo sd", round(sd(placebo), 2),
      "| share at or below", round(mean(placebo <= estimate), 3), "\n")
}
Code 
import numpy as np
import pandas as pd
from scipy.linalg import toeplitz

returns = pd.read_csv("https://empirical-economics.netlify.app/tutorials/datafiles/clean_energy_2016_daily.csv")

# tent weights: the DiD of the four means as a weighted sum of daily long-short returns
weights = np.array([0, 0.2, 0.4, 0.6, 0.8, 1, 0.8, 0.6, 0.4, 0.2])  # k = -5, ..., 4

for benchmark in ["SPY", "IWO"]:
    # one long-short return per day: clean-energy average minus the benchmark
    daily = (returns.query("group == 'clean' or ticker == @benchmark")
             .groupby(["k", "date", "group"])["ret"].mean().unstack("group")
             .reset_index().sort_values("k"))
    daily["ls"] = daily["clean"] - daily["market"]

    def did_at(centre):
        window = daily.query("@centre - 5 <= k <= @centre + 4")
        return float((weights * window["ls"].to_numpy()).sum())

    estimate = did_at(0)

    # (c) Newey-West: w' Omega w, Omega from the Bartlett-weighted autocovariances
    # (lag 5) of the daily long-short return over the estimation year
    e = daily.loc[daily["date"].between("2015-10-01", "2016-09-30"), "ls"].to_numpy()
    e = e - e.mean()
    autocov = np.array([(e[j:] * e[:len(e) - j]).sum() / len(e) for j in range(10)])
    bartlett = np.maximum(1 - np.arange(10) / 6, 0)
    nw_se = np.sqrt(weights @ toeplitz(autocov * bartlett) @ weights)

    # (d) placebo dates: the same DiD centred on every day from k = -275 to k = -26
    placebo = np.array([did_at(c) for c in range(-275, -25)])

    print(benchmark, "| estimate", round(estimate, 2), "| Newey-West se", round(nw_se, 2),
          "| placebo sd", round(placebo.std(ddof=1), 2),
          "| share at or below", round((placebo <= estimate).mean(), 3))

Takeaways

What should you be able to do now?

  1. Build a firm-year panel from Compustat with a paper’s screens, and explain how its outcome and its treatment dummy are coded and where they can be mismeasured.
  2. Calculate a \(2\times2\) DiD for one adoption cohort from four means, and explain why the four means equal the within-firm estimate only when the same firms are observed in every year.
  3. Show that a comparison with an already-treated cohort subtracts that cohort’s change in its own treatment effect, and say when this biases conventional two-way fixed effects.
  4. Read an event-study plot from two-way fixed effects and from Callaway-Sant’Anna, explain why insignificant leads do not establish parallel trends, and recognise when an estimate on an unbalanced panel reflects firms entering and leaving.
  5. Decide whether a control belongs in a DiD regression: a confounder of the trends, or a variable that responds to the treatment or is built from the outcome.
  6. Explain why a common event date makes standard errors computed across assets too small, and do inference on the long-short series with Newey-West or with placebo dates.