Pooled OLS and RE can, but both require \(E(\alpha_i\mid x_{i1},\ldots,x_{iT},z_i)=0\): the individual effect must be uncorrelated with the regressors.
It subtracts a fraction \(\theta\) of each individual’s mean from every variable; \(\theta=0\) gives pooled OLS and \(\theta=1\) gives FE.
It tests \(H_0:\sigma_\alpha^2=0\); rejection says only that a persistent individual component exists for GLS to model, not that \(\alpha_i\) is uncorrelated with the regressors.
It compares the FE and RE estimates of the coefficients both estimators identify; the classical statistic additionally requires RE to be efficient under the null, so it is unreliable under heteroskedasticity or serial correlation.
It adds each individual’s means of the time-varying regressors, and a joint test that those coefficients are zero is a direct test of the RE restriction that tolerates person-clustered standard errors.
Solutions
Question 1: What variation identifies the union premium?
The variation calculation gives:
Variable
Between-person SD
Average within-person SD
educ
1.748
0.000
union
0.329
0.195
married
0.377
0.258
FE can identify the coefficients on union and married, because both change within at least some people. It cannot identify the coefficient on educ: for every person, \(\text{educ}_{it}-\overline{\text{educ}}_i=0\) in every year.
The key estimates are:
Estimator
union coefficient
educ coefficient
Between
0.2707
0.0946
Pooled OLS
0.1801
0.0994
Random effects
0.1074
0.1012
Fixed effects
0.0821
not identified
Thus, the union estimates are between \(>\) pooled OLS \(>\) RE \(>\) FE. The between estimator uses only cross-person differences. Pooled OLS mixes those differences with within-person changes. RE subtracts an estimated fraction of each person’s mean and therefore uses less between variation. FE subtracts the whole mean and uses none. Persistent differences between men who are more and less often unionized inflate the comparisons that retain more between variation.
The education coefficient disappears under FE because education has no within-person variation in this sample. RE can print a coefficient because it retains between-person variation. That does not make the estimate credible: it is valid only if the individual effect—containing factors such as ability—is uncorrelated with education and the other regressors. This is exactly the assumption we have reason to doubt.
For pooled OLS, the default standard error on union is 0.0171 and the person-clustered one is 0.0276. The coefficient remains 0.1801 in both cases. Clustering changes the estimated sampling uncertainty, not the OLS minimization problem. The default calculation incorrectly treats eight observations on the same man as independent even though they share \(\alpha_i\).
library(dplyr)library(haven)library(lmtest)library(plm)wagepan <-read_dta("https://empirical-economics.netlify.app/tutorials/datafiles/WAGEPAN.DTA")# Between and within variation.bind_rows(lapply(c("educ", "union", "married"), function(v) { by_person <- wagepan |>group_by(nr) |>summarise(person_mean =mean(.data[[v]]),within_sd =sd(.data[[v]]), .groups ="drop")data.frame(variable = v,between_sd =sd(by_person$person_mean),average_within_sd =mean(by_person$within_sd) )}))f_all <- lwage ~ union + educ + black + hisp + exper + expersq + marriedpooled <-plm(f_all, wagepan, index =c("nr", "year"), model ="pooling")re <-plm(f_all, wagepan, index =c("nr", "year"), model ="random")fe <-plm(f_all, wagepan, index =c("nr", "year"), model ="within")# On a balanced panel, OLS on the person means is the between estimator.wagepan_means <- wagepan |>group_by(nr) |>summarise(across(all_of(all.vars(f_all)[-1]), mean),lwage =mean(lwage), .groups ="drop")between <-lm(f_all, data = wagepan_means)# The coefficients.sapply(list(Between = between, Pooled = pooled, RE = re, FE = fe), coef)# The standard errors to report.V_between <- sandwich::vcovHC(between, type ="HC0")V_pooled <-vcovHC(pooled, method ="arellano", type ="HC0",cluster ="group")V_re <-vcovHC(re, method ="arellano", type ="HC0",cluster ="group")V_fe <-vcovHC(fe, method ="arellano", type ="HC0",cluster ="group")coeftest(between, V_between)coeftest(re, V_re)coeftest(fe, V_fe)coeftest(pooled) # default standard errorscoeftest(pooled, V_pooled) # person-clustered standard errors
RE therefore subtracts about 64% of each person’s mean. It lies substantially closer to FE than to pooled OLS, but it does not remove the between-person comparison completely.
For a time-invariant regressor, \(\bar z_i=z_i\), so quasi-demeaning gives
\[
z_i-\theta\bar z_i=(1-\theta)z_i.
\]
This retains variation when \(0<\theta<1\). At the FE endpoint, \(\theta=1\), the transformed regressor is zero and its coefficient is not identified.
If \(\sigma_\alpha^2\to0\), then \(\theta\to0\): there is no persistent component to remove, and RE becomes pooled OLS. If \(T\to\infty\) while \(\sigma_\alpha^2>0\), then \(\theta\to1\): quasi-demeaning approaches full demeaning and RE approaches FE.
A \(\theta\) close to one does not repair correlation between \(\alpha_i\) and a regressor. For every finite \(T\), RE retains some between variation. If that variation is endogenous, the RE estimator remains inconsistent; being numerically close to FE is not an identification argument.
Question 3: What do the specification tests actually establish?
The Breusch–Pagan test gives \(\chi^2(1)=3216.7\) and \(p<0.001\). Its null is \(H_0:\sigma_\alpha^2=0\). We reject: the same person’s errors share a persistent component, so RE’s GLS transformation has something to model and can improve efficiency relative to pooled OLS. The test does not examine \(\operatorname{Cov}(\alpha_i,x_{it})\), so it establishes the credibility of neither estimator.
The classical Hausman comparison gives \(\chi^2(4)=250.26\) and \(p<0.001\). educ, black and hisp must be excluded because FE cannot identify them; Hausman can compare only coefficients estimated by both models. Under
\[
H_0:E(\alpha_i\mid x_{i1},\ldots,x_{iT})=0,
\]
FE and RE are both consistent and should converge to the same slopes. The decisive rejection indicates a systematic difference and points to correlation between the individual effect and at least one regressor.
The classical covariance-difference formula relies on RE being efficient under \(H_0\). Cluster-robust covariance matrices do not preserve the required efficiency ordering, so their difference need not be positive definite and can even produce a negative statistic. Clustered inference belongs in the Mundlak test, not inside the classical Hausman formula.
In the Mundlak regression, the coefficients on the time-varying regressors equal the FE coefficients (to the displayed precision):
Regressor
FE
Mundlak
union
0.0821
0.0821
exper
0.1168
0.1168
expersq
-0.0043
-0.0043
married
0.0453
0.0453
The person means are jointly significant with person-clustered inference (\(p<0.001\)), so the ordinary RE restriction is rejected. FE gives a credible within-person estimate of the union coefficient under strict exogeneity. The return to education is not identified from within-person variation. Ordinary RE prints a number for it only by imposing the rejected orthogonality assumption, so this panel does not provide a credible answer without an additional research design.
If the level error is a random walk, \(u_{it}=u_{i,t-1}+e_{it}\), then \(\Delta u_{it}=e_{it}\) and the benchmark correlation is zero.
The interval \([-0.438,-0.354]\) contains neither \(-0.5\) nor zero. Thus, the level errors look like neither serially independent noise nor a random walk. The diagnostic concerns the relative efficiency of FE and FD under alternative error processes; it does not change their common identifying logic. Both remove \(\alpha_i\) and both require strict exogeneity of the idiosyncratic error in the static model.
The point estimates can differ because FE uses deviations from the full person mean whereas FD weights adjacent changes, and because a finite sample produces sampling variation. A large gap can also be a warning about a misspecified functional form, measurement error, or failure of strict exogeneity. It is not by itself proof that one estimator is consistent and the other is not.
Both models should use person-clustered standard errors because remaining disturbances can still be heteroskedastic and serially correlated within a person. The simple FE and FD transformations do not guarantee independent transformed errors.
Question 5: Make a model choice, not a test choice
Firms with stronger management or a more trusting workplace culture may both adopt remote work earlier and achieve higher productivity. Then remote-work adoption is correlated with \(\alpha_i\). Pooled OLS and ordinary RE retain the contaminated between-firm comparison and are inconsistent. FE and FD remove \(\alpha_i\) and remain credible under strict exogeneity.
The three tests answer different questions:
The LM rejection says \(\sigma_\alpha^2>0\), so there is a persistent firm component and RE may be more efficient than pooled OLS if their shared exogeneity assumption holds.
The classical Hausman non-rejection says that this particular test did not detect a difference. It does not prove orthogonality, may have low power, and relies on the classical efficiency assumptions.
The clustered Mundlak rejection directly detects correlation between the firm means of the regressors and the outcome, using inference appropriate for within-firm dependence. It is evidence against ordinary RE.
These statements are compatible: a firm effect can exist, the classical test can fail to detect its correlation with adoption, and the clustered Mundlak test can detect it.
For the remote-work coefficient, FE and FD both remove the firm effect and are more credible than pooled OLS or RE. The FD-residual correlation being close to zero is the random-walk benchmark, so it favours FD for efficiency; FE should still be reported as a robustness comparison. This efficiency choice is conditional on having already chosen a within-firm design for credibility.
Neither FE nor FD can identify the coefficient on FounderLed_i, because that variable has no within-firm changes. An RE coefficient is not an automatic substitute: it uses between-firm variation under the very orthogonality restriction rejected by the Mundlak test. Estimating a credible founder-led effect requires an additional source of identifying variation or a different research design.
The researcher should cluster by firm, the level at which policy adoption varies and within which errors may be dependent over time. Clustering can make standard errors valid under within-firm dependence. It never changes a coefficient and cannot repair omitted-variable bias.
Self-study
Question 6: Reading \(\theta\) off the variance components
1. Using \(\theta=1-\sqrt{\sigma_u^2/(\sigma_u^2+T\sigma_\alpha^2)}\):
Panel
\(T\hat\sigma_\alpha^2\)
\(\hat\sigma_u^2+T\hat\sigma_\alpha^2\)
\(\hat\theta\)
A
0.06
0.56
\(1-\sqrt{0.893}=0.055\)
B
4.50
4.65
\(1-\sqrt{0.032}=0.820\)
C
0.40
0.60
\(1-\sqrt{0.333}=0.423\)
2. Panel A has \(\hat\theta\approx0.06\), so RE removes almost nothing of each individual’s mean and sits essentially on top of pooled OLS. Panel B has \(\hat\theta\approx0.82\) and is the closest to FE. Panel C sits between the two. The pattern is intuitive: \(\theta\) rises with the share of the total variance attributable to the individual effect and with the number of periods.
3. No. Consistency is not continuous in \(\theta\). RE is consistent only under \(E(\alpha_i\mid x_{i1},\ldots,x_{iT},z_i)=0\). If that condition fails, RE remains inconsistent for every\(\theta<1\), because a fraction \(1-\theta\) of the contaminated between-individual variation is still used. A large \(\theta\) shrinks the asymptotic bias relative to pooled OLS, but only the complete removal at \(\theta=1\), which is FE, eliminates it.
4. The agreement is mechanical rather than evidential. With \(\hat\theta\approx0.06\) the RE transformation subtracts about six per cent of each person’s mean, so RE and pooled OLS are using almost exactly the same variation and are numerically bound to be close. Two estimators that share a comparison cannot be used to check whether that comparison is contaminated. The informative comparison is with FE, which uses only within-person variation, or better a Mundlak regression with person-clustered standard errors that tests the restriction directly.
Question 7: Audit the claims
1. Incorrect. Failing to reject is not evidence that the null is true. With little within-person variation the test has low power, and the classical statistic itself presumes that RE is efficient under the null, which fails under heteroskedasticity or serial correlation. A Mundlak test with clustered standard errors, together with an argument about what sits inside \(\alpha_i\), is the more informative route.
2. Overstated. FE is more robust because it allows \(\alpha_i\) to be correlated with the regressors, but it cannot identify the coefficient on any time-invariant regressor and it throws away the between-individual comparison. When a regressor barely moves within individuals, the FE estimate can be very imprecise. If the RE restriction is genuinely defensible, RE uses more of the information in the data.
3. Confused about what efficiency means. RE is more efficient than FE only when the RE assumptions hold. A smaller standard error around an inconsistent estimate is worse than a larger standard error around a consistent one. Efficiency is a conditional property, not a reason to prefer an estimator whose identifying assumption is in doubt, and once standard errors are clustered by individual the RE standard errors need not be smaller anyway.
4. Partly correct. FE and FD cannot estimate it: education has no within-person variation, so it is absorbed by \(\alpha_i\). Pooled OLS and RE do report a coefficient, but only under \(E(\alpha_i\mid x,z)=0\), which is exactly what is doubted when ability is inside \(\alpha_i\). Adding person means in a Mundlak regression does not rescue it either, since a time-invariant regressor has no deviation from its own mean. The honest statement is that this panel does not identify the return to education credibly, and that another source of variation or a different design would be required.
5. Incorrect. The LM test’s null is \(\sigma_\alpha^2=0\). Rejection says only that a persistent individual component exists, so pooled OLS is inefficient and RE has something to model. The FE-versus-RE decision turns on whether \(\alpha_i\) is correlated with the regressors, which the LM test does not examine.
Takeaways
What should you be able to do now?
Say which variation pooled OLS, the between estimator, RE, FE, and FD each use, and which coefficients each of them can identify.
Calculate \(\theta\) from the estimated variance components and read off where RE sits between pooled OLS and FE.
Estimate all four models in R or Python and compare them with standard errors clustered by individual.
Run the LM, Hausman, and Mundlak diagnostics and state precisely what each does and does not establish.
Justify an estimator from the economics of what may lie inside \(\alpha_i\), keeping credibility and efficiency separate.