Empirical Economics

Lecture 5: Panel Data II

Outline

Course Overview

Introduction to Econometrics and Data
Linear Model I
Linear Model II
Panel Data I
Panel Data II

What do we do today?

Second of two lectures devoted to panel data.
Prerequisite knowledge: fixed effects and first differences, within and between variation, strict exogeneity, clustered standard errors.
This lecture:
- The alternatives: pooled OLS and random effects, and what each assumes.
- The connection: all three estimators are one family, indexed by a single number.
- The choice: what the specification tests can and cannot settle for you.
Lecture 4 removed an entire class of confounders by comparing each unit with itself. Today we ask what that comparison cost us, and whether we are ever willing to pay less for it.
Material: Wooldridge, Sections 13.1, 13.2, 14.2 and 14.3.

What should you be able to do?

By the end of this lecture, you should be able to:

State what pooled OLS assumes, and why it still needs clustered standard errors.
Explain the random-effects assumption and why it is stronger than the fixed-effects one.
Write down the quasi-demeaning parameter \(\theta\) and place pooled OLS, RE and FE on it.
Carry out and interpret the Breusch–Pagan LM and Hausman tests.
Use the correlated-random-effects regression as a robust alternative to Hausman.
Justify a choice of panel estimator from economics, not only from a \(p\)-value.

Two questions will guide us

Is union membership associated with higher wages?
What is the return to a year of schooling?

We observe 545 young men in each of 8 years, 1980 to 1987 (Vella and Verbeek, 1998).
Outcome: the log of the hourly wage.
Both questions are about the same people, in the same dataset, answered by the same regression.

Prediction

Lecture 4 gave you a fixed-effects estimator. Which of these two questions can it answer, and which can it not?

Commit to an answer now. We will come back to it.

Why fixed effects are not enough

The variable you care about may not move

Variable	Between-person SD	Within-person SD
educ	1.748	0.000
black	0.320	0.000
hisp	0.363	0.000
union	0.329	0.195
married	0.377	0.258
exper	1.655	2.449

The within-person standard deviation of education, being black, and being Hispanic is exactly zero. Nobody in this panel returns to school, and nobody changes ethnicity.

What that zero costs us

Lecture 4 showed that the within transformation subtracts each person’s own mean, so a regressor that never moves within a person satisfies \(z_i - \bar z_i = 0\) and vanishes from the equation. Equivalently, in the dummy-variable view: \(z_i\) is perfectly collinear with the person fixed effect, so no software could separate them however it was asked.

That is not a computational nuisance. It means the return to schooling is not identified by a comparison of a person with himself, because the comparison never varies the schooling.

Your prediction, checked

Question 1 is answerable by fixed effects: 24.4% of person-years are union, and men move in and out of unions.

Question 2 is not. Fixed effects will drop educ without comment, as they must.

So we need an estimator that keeps the between-person comparison. This lecture is about what that costs.

Four ways to use a panel

	\(\operatorname{Cov}(\alpha_i,x_i)\neq 0\) allowed	\(\operatorname{Cov}(\alpha_i,x_i)=0\) assumed
All \(x_{it}\) strictly exogenous	First differences, LSDV, within estimation. The coefficient \(\gamma\) on a time-invariant \(z_i\) is not identified.	Pooled OLS, random effects. The coefficient \(\gamma\) on a time-invariant \(z_i\) is identified.
Some \(x_{it}\) not strictly exogenous	Instrumental variables or dynamic panel methods (Arellano–Bond). Beyond this course.	Instrumental variables or dynamic panel methods. Beyond this course.

One assumption, two consequences. Allowing \(\alpha_i\) to correlate with the regressors is exactly what forces us to throw away the between variation, and throwing away the between variation is exactly what makes \(\gamma\) unidentified.

Everything in the top row assumes strict exogeneity, \(E(u_{it}\mid x_{i1},\ldots,x_{iT},\alpha_i)=0\): no lagged dependent variables and no feedback, as derived in Lecture 4.

Pooled OLS

Ignore the panel structure

The simplest thing to do with a panel is to forget that it is one:

\[ y_{it}=\beta_0+\beta_1x_{1it}+\cdots+\beta_kx_{kit}+v_{it}, \qquad v_{it}=\alpha_i+u_{it}. \]

The individual effect has not disappeared. It has been folded into a composite error \(v_{it}\), where OLS will treat it as noise.

Nothing was demeaned and nothing was differenced, so a time-invariant \(z_i\) survives and its coefficient can be estimated.

Consistent under what, exactly?

This is the distinction Lecture 4 drew for fixed effects, applied here.

Two different requirements

Pooled OLS is consistent as \(N\to\infty\) under contemporaneous exogeneity of the composite error, \[E(v_{it}\mid x_{it}) = 0.\]

Pooled OLS is unbiased under strict exogeneity of the composite error, \[E(v_{it}\mid x_{i1},\ldots,x_{iT}) = 0.\]

Both require \(E(\alpha_i \mid x_{it}) = 0\), because \(\alpha_i\) sits inside \(v_{it}\). That single requirement is the whole ballgame, and it is precisely what fixed effects refuse to assume.

The composite error is serially correlated

Even when pooled OLS is consistent, its standard errors are not. Because the same \(\alpha_i\) appears in every period for person \(i\),

\[ \operatorname{Cov}(v_{it},v_{is}) =\operatorname{Cov}(\alpha_i+u_{it},\alpha_i+u_{is}) =\operatorname{Var}(\alpha_i) =\sigma^2_\alpha, \qquad t \neq s, \]

while \(\operatorname{Var}(v_{it})=\sigma^2_\alpha+\sigma^2_u\), so that

\[ \operatorname{Corr}(v_{it},v_{is})=\frac{\sigma^2_\alpha}{\sigma^2_\alpha+\sigma^2_u}. \]

Whenever \(\sigma^2_\alpha>0\), observations on the same person are positively correlated in every pair of periods. We will use this correlation twice more today: once to build the random-effects estimator, and once to test for it.

Clustering, revisited

Lecture 3’s rule was not “are my errors correlated?” but “at what level does my treatment vary?” Union status varies person by person and persists, so we cluster on the person.

Coefficient	Estimate	Default SE	Clustered SE
union	0.1801	0.0171	0.0276
educ	0.0994	0.0047	0.0092

Clustering leaves both coefficients untouched and raises the standard error on union by 61%, and the one on educ by 97%. The default standard errors were counting 8 observations per man as 8 independent pieces of evidence.

Pooled OLS on the wage panel

	Default SEs	Clustered by person
* p < 0.1, p < 0.05, * p < 0.01
Controls for experience and its square included but not shown.
Union member	0.180***	0.180***
	(0.017)	(0.028)
Years of education	0.099***	0.099***
	(0.005)	(0.009)
Married	0.108***	0.108***
	(0.016)	(0.026)
Black	-0.144***	-0.144***
	(0.024)	(0.050)
Num.Obs.	4360	4360
R2	0.187	0.187

Union membership is associated with 18.0 log points, which by Lecture 2’s log-level interpretation is \(e^{0.180}-1\), or 19.7%, higher wages. Whether that association is a union effect is exactly what the rest of the lecture is about.

Random effects

Averaging away what fixed effects throw out

Before building anything new, look at the comparison fixed effects discards. Collapse each man to his eight-year averages and regress those on each other:

\[ \bar y_i = \beta_0 + \beta_1 \bar x_{1i} + \cdots + \gamma_1 z_{1i} + \cdots + (\alpha_i + \bar u_i). \]

This is the between estimator: one observation per person, no within variation at all.

Estimator	Union premium	Return to schooling
Between	0.271	0.095
Pooled OLS	0.180	0.099
Fixed effects	0.082	not identified

The between estimate of the union premium is 3.30 times the fixed-effects one. Pooled OLS lands between them, because it uses both comparisons at once.

The random-effects assumption

Random effects keeps both comparisons, and pays for the privilege with an assumption:

The random-effects assumption

\[ E\!\left(\alpha_i \mid x_{i1},\ldots,x_{iT}\right)=0. \]

The unobserved individual effect is uncorrelated with the entire history of the regressors.

This is not a weaker version of the fixed-effects assumption. It is the assumption fixed effects were invented to avoid.

Note what it does not buy. Random effects is consistent under exactly the same condition as pooled OLS. What separates them is efficiency, and that is the next slide.

Quasi-demeaning: one estimator family

Since we know the error correlation is \(\sigma^2_\alpha/(\sigma^2_\alpha+\sigma^2_u)\), we can undo it. Generalized least squares subtracts not the whole person mean, but a fraction of it:

\[ y_{it}-\theta\bar y_i =\beta_1\left(x_{1it}-\theta\bar x_{1i}\right)+\cdots+\left(v_{it}-\theta\bar v_i\right), \qquad \theta=1-\sqrt{\frac{\sigma^2_u}{\sigma^2_u+T\sigma^2_\alpha}}. \]

Three estimators, one number

\(\theta=0\): subtract nothing. This is pooled OLS.
\(\theta=1\): subtract the whole mean. This is fixed effects.
\(0<\theta<1\): random effects, somewhere in between.

\(\theta\to 0\) as \(\sigma^2_\alpha\to 0\), and \(\theta\to 1\) as \(T\to\infty\). With enough periods per person, random effects and fixed effects converge.

What \(\theta\) is in our data

Component	Value
Idiosyncratic, \(\hat\sigma^2_u\)	0.123
Individual effect, \(\hat\sigma^2_\alpha\)	0.105
Implied \(\hat\theta\)	0.643

With \(T=8\) and roughly equal variance components, \(\hat\theta = 0.643\): random effects subtracts 64% of each man’s mean, and sits that far from pooled OLS towards fixed effects, so its estimates should land close to, but not on top of, the fixed-effects ones.

How the two variances are estimated

Both are read off regressions we have already run. The within residuals have had \(\alpha_i\) removed, so they estimate the idiosyncratic part directly: \[\hat\sigma^2_u=\frac{1}{NT-N-k}\sum_i\sum_t\hat u^2_{it}.\] The between regression has error \(\alpha_i+\bar u_i\), whose variance is \(\sigma^2_\alpha+\sigma^2_u/T\). Estimate that by its residual variance \(\hat\sigma^2_b\) and subtract the piece just obtained: \(\hat\sigma^2_\alpha=\hat\sigma^2_b-\hat\sigma^2_u/T\). This is what ercomp() reports.

Four estimates of one coefficient

From all between variation to none, in four steps. Nothing was added to the data; one comparison was progressively removed.

What random effects buys, and what it costs

Random effects can estimate the coefficient on a time-invariant regressor, because it never fully removes the between comparison. That is the answer to our second guiding question, and the only estimator today that offers one.

Random effects is also more efficient than fixed effects when its assumption holds, because it uses both sources of variation instead of one.

The price

If \(\operatorname{Cov}(\alpha_i,x_{it})\neq 0\), random effects is inconsistent, and the bias does not shrink with more data. It is Lecture 2’s omitted variable bias, arriving through the between comparison that fixed effects deleted.

The efficiency gain is real only under the assumption. The inconsistency is real whenever the assumption fails.

Standard errors for random effects

The efficiency claim rests on more than the exogeneity assumption. GLS is the efficient estimator only if \(u_{it}\) is homoskedastic and serially uncorrelated after the individual effect is accounted for.

In wage data it is neither: wages are persistent for reasons beyond a fixed personal endowment.

So we report random effects with standard errors clustered by person, for the same reason we did it for pooled OLS in Lecture 3’s framework. The GLS transformation handles the part of the serial correlation we modelled; clustering handles the part we did not.

Choosing between them

Predict before you test

We already know something about \(\alpha_i\) here. In Lectures 2 and 3 we built a world in which unobserved ability raised both schooling and wages, and we could not fix the resulting bias because ability was unmeasured.

That world is this dataset. The person effect \(\alpha_i\) contains ability; ability is correlated with educ and plausibly with union membership too.

Retrieval question

Given that, what should the random-effects assumption \(E(\alpha_i\mid x_i)=0\) be doing here: holding, or failing?

What do you therefore expect a test of it to report?

Economics gives the answer before any test runs. A specification test that agrees with it is a confirmation, not a discovery.

The Hausman test

Two estimators, one of which is fragile:

Under strict exogeneity, \(\hat\beta_{FE}\) is consistent whether or not \(\alpha_i\) correlates with \(x\).
\(\hat\beta_{RE}\) is consistent and efficient if \(\alpha_i\) and \(x\) are uncorrelated, and inconsistent if they are not.

So if the assumption holds, the two should estimate the same thing and differ only by sampling noise. If it fails, only one of them is still aiming at \(\beta\).

For one coefficient, it is the squared gap, scaled by how much that gap varies:

\[ H=\frac{\left(\hat\beta_{FE}-\hat\beta_{RE}\right)^2} {\operatorname{Var}(\hat\beta_{FE})-\operatorname{Var}(\hat\beta_{RE})} \;\sim\;\chi^2(1). \]

With several coefficients the same idea gives \(\chi^2(k)\), where \(k\) counts the regressors both estimators identify: the time-varying ones.

The Hausman test on the wage panel

\(H=250.3\) against a 5% critical value of 9.5. We reject decisively, exactly as economics said we would. The fixed-effects estimator is the consistent one here.

What the Hausman test cannot tell you

Two asymmetries

Failing to reject is not evidence that the assumption holds. The test can be underpowered, and Lecture 2’s warning applies unchanged: the exogeneity of an unobservable is not something a dataset can verify. Non-rejection means you did not detect a violation, not that there is none.

The classical statistic assumes random effects is efficient under \(H_0\). That requires \(u_{it}\) to be homoskedastic and serially uncorrelated, which we have just said is false for wages. When it fails, \(\operatorname{Var}(\hat\beta_{FE})-\operatorname{Var}(\hat\beta_{RE})\) need not even be positive, and the statistic can come out negative.

Both problems have the same fix, and it is more intuitive than the test it replaces.

Correlated random effects

Mundlak’s idea: if we are worried that \(\alpha_i\) correlates with the regressors, let it, and model how. Write

\[ \alpha_i=\psi_0+\psi_1\bar x_{1i}+\cdots+\psi_k\bar x_{ki}+c_i, \]

and substitute back. The estimating equation becomes a random-effects regression with each person’s own means added as extra regressors:

\[ y_{it}=\beta_0+\beta_1x_{1it}+\cdots+\beta_kx_{kit} +\gamma z_i +\psi_1\bar x_{1i}+\cdots+\psi_k\bar x_{ki}+c_i+u_{it}. \]

The test is now a coefficient

\(H_0:\psi_1=\cdots=\psi_k=0\) is the random-effects assumption. Test it with an ordinary Wald test, using whatever standard errors you would otherwise use, clustered ones included.

Correlated random effects on the wage panel

	Fixed effects	Correlated random effects
* p < 0.1, p < 0.05, * p < 0.01
Both clustered by person. Experience terms and their means included, not shown.
Union member	0.082***	0.082***
	(0.023)	(0.023)
Married	0.045**	0.045**
	(0.021)	(0.021)
Years of education		0.095***
		(0.011)
Person mean of union		0.189***
		(0.048)
Person mean of married		0.098**
		(0.045)
Num.Obs.	4360	4360

Two things to notice. The coefficients on the time-varying regressors are identical to fixed effects, to four decimals. And educ is estimated, which fixed effects could not do.

The joint test of the person means gives \(F=7.18\) on \(4\) and \(544\) degrees of freedom, with \(p<0.001\): rejected, with clustered standard errors, and no efficiency assumption anywhere.

When the test rejects but you wanted \(\gamma\)

We have rejected the random-effects assumption, so fixed effects is the estimator to trust. But our second guiding question was the return to schooling, and fixed effects cannot answer it.

This is the honest position, and it is worth stating plainly:

The answer to question 2

The return to schooling is not identified by within-person variation in this panel, because there is none. Random effects will print a number for it, but only by assuming away the very confounder that Lectures 2 and 3 established.

A number is available. A credible number is not. The difference is an assumption, not a computation.

Hausman–Taylor estimators instrument the between variation instead of assuming it away, which moves the problem rather than removing it. The genuine solutions are the research designs of the coming weeks.

Pooled OLS or random effects?

The remaining comparison is the easy one, because it is not about identification at all.

Both estimators need the same assumption about \(\alpha_i\), so the only open question is whether there is an individual effect worth modelling. If \(\sigma^2_\alpha=0\), the composite error \(v_{it}=\alpha_i+u_{it}\) is just \(u_{it}\): nothing persists within a person, and pooled OLS is already the efficient estimator.

That is a hypothesis about a variance, and it has its own test.

The Breusch–Pagan LM test

The Lagrange multiplier test of Breusch and Pagan (1980) puts the two cases against each other:

\(H_0:\sigma^2_\alpha=0\). No individual effect. Pooled OLS is efficient.
\(H_1:\sigma^2_\alpha>0\). A person’s errors share a persistent component, so they are correlated across time and GLS has work to do.

Run pooled OLS, keep its residuals \(\hat v_{it}\), and compare each person’s summed residuals with their individual ones:

\[ LM=\frac{NT}{2(T-1)} \left[\frac{\sum_i\left(\sum_t \hat v_{it}\right)^2}{\sum_i\sum_t \hat v_{it}^2}-1\right]^2 \;\sim\;\chi^2(1). \]

Under \(H_0\) a person’s residuals are as likely to cancel as to reinforce, the ratio in brackets is about 1, and \(LM\) is near zero. A persistent \(\alpha_i\) pushes the residuals of one person to the same side, the ratio grows, and the statistic with it.

The LM test on the wage panel

On our data, \(\chi^2=3217\) against a 5% critical value of 3.84, so \(p<0.001\): there is an individual-effect variance component, and GLS has something to do.

This settles efficiency, and nothing else. Pooled OLS and random effects fail together when \(\alpha_i\) is correlated with the regressors, and the LM test does not look at that at all.

A caveat on the LM test

\(\sigma^2_\alpha=0\) sits on the boundary of the parameter space, since a variance cannot be negative. The usual two-sided \(\chi^2\) approximation is therefore not exact; the one-sided version (type = "honda") is the sharper test.

What dies when

Estimator	If \(\operatorname{Cov}(\alpha_i,x)=0\)	If \(\operatorname{Cov}(\alpha_i,x)\neq0\)
Pooled OLS	Consistent, inefficient, needs clustered SEs	Inconsistent
Random effects	Consistent and efficient	Inconsistent
Fixed effects	Consistent, less efficient, no \(\gamma\)	Consistent, no \(\gamma\)

Pooled OLS and random effects live and die together. Choosing between them is a question about standard errors and efficiency, never about credibility.

The only line in the table that changes what you are allowed to believe is the third one.

Does the error process favour FE or FD?

One diagnostic from Lecture 4 remains. If the level errors \(u_{it}\) are serially independent, differencing induces \(\operatorname{Corr}(\Delta u_{it},\Delta u_{i,t-1})=-0.5\); if instead the levels follow a random walk, the differences are uncorrelated and the benchmark is \(0\).

Regressing the first-difference residuals on their own within-person lag gives -0.396, with a 95% interval of [-0.438, -0.354].

Neither benchmark survives: the errors are neither i.i.d. in levels nor a random walk. This tells us which estimator is more efficient under which error process; it does not tell us which is more credible, since both rest on the same identifying assumption.

A line of reasoning

A workflow, not a flowchart

Estimate all of them. Pooled OLS, first differences, fixed effects, random effects, and report them side by side.
Cluster everywhere. At the level at which the treatment varies. This changes inference, never a coefficient.
Use the LM test to check that there is an individual effect worth modelling, and remember it says nothing about identification.
Use the Hausman or, better, the Mundlak test to ask whether the between comparison is contaminated. Predict the answer from economics first.
Decide with the economics. If you believe \(\alpha_i\) correlates with your regressor, use fixed effects, whatever the \(p\)-value says.

Lecture 4 warned there is no mechanical “test and choose” rule. These tests are evidence, not verdicts.

Everything in one table

	Pooled OLS	Random effects	Fixed effects	First diff.
* p < 0.1, p < 0.05, * p < 0.01
All standard errors clustered by person. Experience terms included, not shown.
Union member	0.180***	0.107***	0.082***	0.043*
	(0.028)	(0.021)	(0.023)	(0.022)
Years of education	0.099***	0.101***
	(0.009)	(0.009)
Married	0.108***	0.063***	0.045**	0.038
	(0.026)	(0.019)	(0.021)	(0.024)
Black	-0.144***	-0.144***
	(0.050)	(0.050)
Num.Obs.	4360	4360	4360	3815

One column per specification, ordered by how much between-person variation each one uses. The union premium falls from 0.180 to 0.082 as that variation is removed, and educ vanishes in the two columns that remove it entirely.

First differences is the fourth column rather than a fifth estimator: it removes \(\alpha_i\) as fixed effects do, but weights the changes differently, which is why its estimate is close to the FE one without being equal to it.

What applied papers actually do

In applied work, people just use fixed effects regressions as a rule. The choice is usually argued rather than tested: what a paper reports is the table on the previous slide, plus an argument why it is the right choice.

The argument is always about the same thing: what is in \(\alpha_i\), and would you expect it to correlate with the regressor of interest? For union membership, more able or more motivated workers may sort into unionized firms, so \(\alpha_i\) and union are correlated, so we use fixed effects. The test agreed, but the sentence came first.

Lecture 4’s lesson, again

A coefficient is a property of a comparison. Pooled OLS, random effects and fixed effects gave 0.180, 0.107 and 0.082 from identical data. None of them is a mistake. They answer different questions.

Software

Estimating pooled OLS, RE, FE and between

R
Python

Code

library(plm)
library(haven)

wagepan <- read_dta(
  "https://empirical-economics.netlify.app/tutorials/datafiles/WAGEPAN.DTA"
)

# fixest has no random-effects estimator, so the RE and between models come
# from plm. Declaring the index once makes every model a panel model, and the
# only thing that changes between the four calls is `model`.
wage_eq <- lwage ~ union + educ + black + hisp + exper + expersq + married

pooled <- plm(wage_eq, wagepan, index = c("nr", "year"), model = "pooling")
between <- plm(wage_eq, wagepan, index = c("nr", "year"), model = "between")
re <- plm(wage_eq, wagepan, index = c("nr", "year"), model = "random")
fe <- plm(wage_eq, wagepan, index = c("nr", "year"), model = "within")

# The variance components and the quasi-demeaning weight theta.
ercomp(re)

# Cluster-robust variances. "arellano" clusters on the index's first dimension,
# here the person; the between model has one row per person, so it gets HC0.
vcovHC(re, method = "arellano", type = "HC0", cluster = "group")

Code

import pandas as pd
from linearmodels.panel import PooledOLS, BetweenOLS, RandomEffects, PanelOLS

df = pd.read_stata(
    "https://empirical-economics.netlify.app/tutorials/datafiles/WAGEPAN.DTA"
)
panel = df.set_index(["nr", "year"])   # linearmodels reads the panel from the index

y = panel["lwage"]
X = panel[["union", "educ", "black", "hisp",
           "exper", "expersq", "married"]].assign(const=1.0)

pooled = PooledOLS(y, X).fit(cov_type="clustered", cluster_entity=True)
between = BetweenOLS(y, X).fit()
re = RandomEffects(y, X).fit(cov_type="clustered", cluster_entity=True)

# Fixed effects drops educ, black and hisp itself, so we do not pass them.
X_tv = panel[["union", "exper", "expersq", "married"]].assign(const=1.0)
fe = PanelOLS(y, X_tv, entity_effects=True).fit(
    cov_type="clustered", cluster_entity=True
)

# theta and the variance components
print(re.theta.iloc[0, 0], re.variance_decomposition)

Code

library(haven)

wagepan <- read_dta(
  "https://empirical-economics.netlify.app/tutorials/datafiles/WAGEPAN.DTA"
)

# Breusch-Pagan LM test, H0: sigma^2_alpha = 0.
plmtest(pooled, type = "bp")

# Classical Hausman. Only regressors both estimators identify, so time-varying
# ones only, and default (non-robust) variances, as the statistic requires.
wage_tv <- lwage ~ union + exper + expersq + married

phtest(
  plm(wage_tv, wagepan, index = c("nr", "year"), model = "within"),
  plm(wage_tv, wagepan, index = c("nr", "year"), model = "random")
)

# Mundlak / correlated random effects: add each person's own means, then test
# them jointly. Any vcov may be used, clustered included.
wagepan_cre <- wagepan |>
  group_by(nr) |>
  mutate(across(c(union, exper, expersq, married), mean, .names = "mean_{.col}")) |>
  ungroup()

cre <- feols(
  lwage ~ union + exper + expersq + married + educ + black + hisp +
    mean_union + mean_exper + mean_expersq + mean_married,
  data = wagepan_cre, vcov = ~nr
)
wald(cre, "mean_")

Code

import numpy as np
import pandas as pd
from linearmodels.panel import RandomEffects, PanelOLS
from scipy import stats

df = pd.read_stata(
    "https://empirical-economics.netlify.app/tutorials/datafiles/WAGEPAN.DTA"
)
panel = df.set_index(["nr", "year"])
y = panel["lwage"]
tv = ["union", "exper", "expersq", "married"]

# Classical Hausman. linearmodels has no built-in version, so we build it:
# both models with default variances, or the difference need not be invertible.
fe_u = PanelOLS(y, panel[tv].assign(const=1.0), entity_effects=True).fit()
re_u = RandomEffects(y, panel[tv].assign(const=1.0)).fit()

d = fe_u.params[tv] - re_u.params[tv]
V = fe_u.cov.loc[tv, tv] - re_u.cov.loc[tv, tv]
H = float(d @ np.linalg.inv(V) @ d)
print(f"H = {H:.2f}, p = {stats.chi2.sf(H, len(tv)):.3g}")

# Mundlak: append each person's own means, then test them jointly.
means = df.groupby("nr")[tv].transform("mean").add_prefix("mean_")
cre_df = pd.concat([df, means], axis=1).set_index(["nr", "year"])
rhs = tv + ["educ", "black", "hisp"] + list(means.columns)

cre = RandomEffects(cre_df["lwage"], cre_df[rhs].assign(const=1.0)).fit(
    cov_type="clustered", cluster_entity=True
)
R = np.zeros((len(means.columns), len(cre.params)))
for i, c in enumerate(means.columns):
    R[i, list(cre.params.index).index(c)] = 1
print(cre.wald_test(R, np.zeros(len(means.columns))))

Takeaways

What did we learn?

Fixed effects cannot estimate a coefficient on a regressor with no within variation, and in our panel educ has exactly none.
Pooled OLS folds \(\alpha_i\) into the error, which makes the composite error serially correlated and the default standard errors too small.
Pooled OLS is consistent under contemporaneous exogeneity, unbiased under strict exogeneity. The two are not the same requirement.
Random effects quasi-demeans by \(\theta=1-\sqrt{\sigma^2_u/(\sigma^2_u+T\sigma^2_\alpha)}\); pooled OLS, RE and FE are one family with \(\theta=0\), \(0<\theta<1\) and \(\theta=1\).
Random effects can identify \(\gamma\) on a time-invariant \(z_i\), at the price of the assumption fixed effects were built to avoid.
Pooled OLS and random effects are inconsistent under exactly the same condition, so choosing between them is about efficiency, not credibility.
The Hausman test compares FE and RE; the Mundlak regression does the same thing with clustered standard errors and no efficiency assumption.
Non-rejection is not verification. Which estimator to trust is an argument about \(\alpha_i\), not a \(p\)-value.

Exit questions

Be able to answer these without looking back:

The within-person standard deviation of educ in our panel is exactly zero. What follows for a person-FE model, and why?
Pooled OLS and random effects become inconsistent under the same condition. What is it, and what then distinguishes them?
What is \(\theta\), and which estimators sit at \(\theta=0\) and \(\theta=1\)?
Our estimates of the union premium were 0.27, 0.18, 0.11 and 0.08. Put them in order and explain the ordering.
What does the classical Hausman statistic assume about random effects under \(H_0\), and why is that a problem here?
A Hausman test on some other dataset does not reject. May you conclude the RE assumption holds?

Answers to the exit questions

The zero. Demeaning gives \(z_i-\bar z_i=0\), so educ is perfectly collinear with the person fixed effect and drops out. Its return is not identified by within variation, because there is none.
The shared condition. \(\operatorname{Cov}(\alpha_i,x)\neq 0\) kills both. Under the assumption they are both consistent, and RE is the efficient one; the choice is about standard errors, not credibility.
\(\theta\). The fraction of each person’s mean that GLS subtracts, \(1-\sqrt{\sigma^2_u/(\sigma^2_u+T\sigma^2_\alpha)}\). \(\theta=0\) is pooled OLS, \(\theta=1\) is fixed effects, and random effects is in between (here 0.64).
The ordering. Between \(>\) pooled \(>\) RE \(>\) FE, monotone in how much between variation each uses. The between comparison contains the persistent differences between men, which inflate the apparent premium.
What Hausman assumes. That RE is efficient under \(H_0\), so the variance difference is positive. That needs homoskedastic, serially uncorrelated \(u_{it}\), which wage data violate; the Mundlak version drops the requirement.
Non-rejection. No. It means no violation was detected, which is different from there being none, and the test may simply lack power. The exogeneity of an unobservable cannot be verified from data.

Looking ahead

Every estimator in these two lectures removes confounding by discarding a comparison, and then asks you to believe an assumption about what is left.

From week 6 we stop adjusting comparisons and start designing them: difference-in-differences uses a control group to say what would have happened anyway, which is where the two-way fixed-effects model of Lecture 4 comes back with a warning attached.

Appendix

Source notes

Wage data: Vella and Verbeek (1998), “Whose Wages Do Unions Raise?”, Journal of Applied Econometrics 13(2), 163–183, distributed as wooldridge::wagepan.
The panel is balanced: 545 men observed in each of the 8 years 1980–1987.
Mundlak, Y. (1978), “On the Pooling of Time Series and Cross Section Data”, Econometrica 46(1), 69–85.
Hausman, J. A. (1978), “Specification Tests in Econometrics”, Econometrica 46(6), 1251–1271.

Where \(\theta\) comes from

Each of person \(i\)’s \(T\) composite errors is \(v_{it}=\alpha_i+u_{it}\), so

\[ \operatorname{Var}(v_{it})=\sigma^2_\alpha+\sigma^2_u, \qquad \operatorname{Cov}(v_{it},v_{is})=\sigma^2_\alpha\quad(t\neq s). \]

Every pair of periods for the same person shares the same \(\alpha_i\), and only that. GLS removes exactly the shared part by subtracting a fraction \(\theta\) of each person’s own mean, with

\[ \theta=1-\sqrt{\frac{\sigma^2_u}{\sigma^2_u+T\sigma^2_\alpha}}. \]

Read it through its two limits. If \(\sigma^2_\alpha=0\) nothing is shared, \(\theta=0\), and GLS is pooled OLS. As \(T\sigma^2_\alpha\) grows relative to \(\sigma^2_u\) the shared part dominates, \(\theta\to1\), and GLS subtracts the whole person mean, which is fixed effects.

In practice \(\sigma^2_u\) and \(\sigma^2_\alpha\) are unknown and estimated first, which is why the estimator is feasible GLS.

Empirical Economics

Outline

Course Overview

What do we do today?

What should you be able to do?

Two questions will guide us

Why fixed effects are not enough

The variable you care about may not move

What that zero costs us

Four ways to use a panel

Pooled OLS

Ignore the panel structure

Consistent under what, exactly?

The composite error is serially correlated

Clustering, revisited

Pooled OLS on the wage panel

Random effects

Averaging away what fixed effects throw out

The random-effects assumption

Quasi-demeaning: one estimator family

What \(\theta\) is in our data

Four estimates of one coefficient

What random effects buys, and what it costs

Standard errors for random effects

Choosing between them

Predict before you test

The Hausman test

The Hausman test on the wage panel

What the Hausman test cannot tell you

Correlated random effects

Correlated random effects on the wage panel

When the test rejects but you wanted \(\gamma\)

Pooled OLS or random effects?

The Breusch–Pagan LM test

The LM test on the wage panel

What dies when

Does the error process favour FE or FD?

A line of reasoning

Everything in one table

What applied papers actually do

Software

Estimating pooled OLS, RE, FE and between

Testing: Hausman and Mundlak

Takeaways

What did we learn?

Exit questions

Answers to the exit questions

Appendix

Source notes

Where \(\theta\) comes from

The End