Solutions Tutorial 2: The Linear Model I

Retrieval practice

  1. The population regression function \(E(y\mid x)=\beta_0+\beta_1x\) is the unknown relationship in the population; the sample regression function \(\hat y=\hat\beta_0+\hat\beta_1x\) is the line estimated from one sample.
  2. OLS minimizes the sum of squared residuals \(\sum_i\hat u_i^2\), which gives \(\hat\beta_1=\operatorname{Cov}(x,y)/\operatorname{Var}(x)\).
  3. In a log-log model the slope is an elasticity: a 1% change in \(x\) is associated with a \(\hat\beta_1\%\) change in \(y\).
  4. \(R^2\) measures the share of the sample variation in \(y\) accounted for by the fitted line; it says nothing about whether the estimate is causal.
  5. SLR.4, \(E(u\mid x)=0\), fails, and the sign of the bias is \(\operatorname{sign}(\beta_2)\times\operatorname{sign}(\delta_1)\).

Solutions

Housing prices

The regression model is: price = 300,000 + 1500 * sqmtr, with R² = 0.64.

(a) Interpret the intercept (300,000) and the slope coefficient (1,500) in plain English.

  • Intercept (300,000): This is the predicted price of a house with 0 square meters of interior surface. In this context, the intercept has no meaningful practical interpretation, as a house cannot have zero square meters. It is a statistical construct that helps position the regression line correctly in the data cloud.

  • Slope (1,500): One additional square meter of interior surface is associated with an estimated €1,500 higher sale price, on average. Without an exogeneity argument, this is not a causal effect of floor area.

(b) What does the R-squared value of 0.64 tell us about this model?

  • An R-squared of 0.64 means that the fitted line accounts for 64% of the sample variation in house prices. The remaining 36% is unaccounted for by this one-regressor model.

(c) If you were to re-estimate the model with price measured in thousands of euros (e.g., a 250,000 euro house becomes 250), what would the new equation be?

  • If we divide the dependent variable price by 1,000, we must also divide the entire right-hand side of the equation by 1,000 to maintain the equality. Let price_k be the price in thousands of euros. \[ \frac{\text{price}}{1000} = \frac{300,000}{1000} + \frac{1500}{1000} \times \text{sqmtr} \]
  • The new equation would be: price_k = 300 + 1.5 * sqmtr
  • The interpretation changes accordingly: The intercept is now 300 thousand euros, and each additional square meter increases the predicted price by 1.5 measured in the new units (thousands of euros).

Functional forms in words

(a) This is a log-log model. A 1% increase in price is associated with an estimated 0.85% decrease in sales, on average. The slope is the price elasticity of demand. Because its absolute value is below one, demand is price inelastic at the estimated relationship.

(b) This is a log-level model. One additional year of education is associated with an approximately \(100(0.08)=8\%\) higher wage, on average. The slope is a semi-elasticity, not an elasticity.

(c) This is a level-log model. A 1% increase in advertising is associated with an estimated \(12/100=0.12\) thousand euro increase in sales—€120—on average. The slope is a semi-elasticity, not an elasticity.

CEO compensation and firm sales

(a) Estimate a linear regression model and write down the estimated equation.

First, we load the data and estimate the model with fixest in R or pyfixest in Python.

library(fixest); library(tidyverse)
url <- 'https://empirical-economics.netlify.app/tutorials/datafiles/ceosal1.csv'
ceosal1 <- read_csv2(url)
model <- feols(log(salary) ~ log(sales), data = ceosal1)
summary(model)
## OLS estimation, Dep. Var.: log(salary)
## Observations: 209
## Standard-errors: IID 
##             Estimate Std. Error  t value   Pr(>|t|)    
## (Intercept) 4.821996   0.288340 16.72332  < 2.2e-16 ***
## log(sales)  0.256672   0.034517  7.43617 2.7034e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## RMSE: 0.501939   Adj. R2: 0.207005
import pandas as pd
import numpy as np
import pyfixest as pf

# Load the dataset from the URL
url = 'https://empirical-economics.netlify.app/tutorials/datafiles/ceosal1.csv'

ceosal1 = pd.read_csv(url, sep=";", decimal=',')

# Create the log-transformed variables
ceosal1['lsalary'] = np.log(ceosal1['salary'])
ceosal1['lsales'] = np.log(ceosal1['sales'])

# Estimate the OLS model
model = pf.feols('lsalary ~ lsales', data=ceosal1)

# Print the summary to see the coefficients
model.summary()

The estimated regression equation is: \[ \widehat{\text{lsalary}} = 4.8220 + 0.2567 \times \text{lsales} \]

(b) Provide a precise interpretation of the slope coefficient.

Since this is a log-log model, the slope coefficient is an elasticity. The interpretation is:

A 1% increase in firm sales is associated with an estimated 0.257% increase in CEO salary, on average. The association is positive but less than proportional: a 10% increase in sales is associated with approximately a 2.57% increase in salary. This estimate alone does not establish a causal effect of sales.

(c) Is the relationship statistically significant at the 1% level?

Yes, the relationship is statistically significant at the 1% level.

The two-sided p-value for log(sales) is approximately \(2.70\times10^{-12}\). Since this is below 0.01, we reject \(H_0:\beta_{\log(sales)}=0\) at the 1% level.

(d) What is the R-squared value of this regression?

The R-squared value from the summary is approximately 0.211.

This means that the fitted model accounts for approximately 21.1% of the sample variation in the natural log of CEO salaries.

Socioeconomic factors and fertility

(a) Estimate a linear regression model and write out the estimated equation.

First, we load the fertil2 dataset and estimate the simple linear model.

library(fixest); library(tidyverse)
url <- 'https://empirical-economics.netlify.app/tutorials/datafiles/fertil2.csv'
fertil2 <- read_csv2(url)
model <- feols(children ~ catholic, data=fertil2)
summary(model)
## OLS estimation, Dep. Var.: children
## Observations: 4,361
## Standard-errors: IID 
##              Estimate Std. Error  t value  Pr(>|t|)    
## (Intercept)  2.284875   0.035512 64.34054 < 2.2e-16 ***
## catholic    -0.166307   0.110922 -1.49931   0.13386    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## RMSE: 2.2212   Adj. R2: 2.861e-4
import pandas as pd
import pyfixest as pf

# Load the dataset from the URL
url = 'https://empirical-economics.netlify.app/tutorials/datafiles/fertil2.csv'
fertil2 = pd.read_csv(url, sep=';', decimal=',')

# Estimate the OLS model
# 'children' is the dependent variable, 'catholic' is the independent dummy variable
model_fertility = pf.feols('children ~ catholic', data=fertil2)

# Print the summary
model_fertility.summary()
## ###
## 
## Estimation:  OLS
## Dep. var.: children, Fixed effects: 0
## Inference:  iid
## Observations:  4361
## 
## | Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |   2.5% |   97.5% |
## |:--------------|-----------:|-------------:|----------:|-----------:|-------:|--------:|
## | Intercept     |      2.285 |        0.036 |    64.341 |      0.000 |  2.215 |   2.354 |
## | catholic      |     -0.166 |        0.111 |    -1.499 |      0.134 | -0.384 |   0.051 |
## ---
## RMSE: 2.221 R2: 0.001

The estimated regression equation is: \[ \widehat{\text{children}} = 2.284 - 0.166 \times \text{catholic} \]

(b) Provide a precise interpretation of the coefficient on the catholic variable.

The catholic variable is a dummy variable (taking a value of 1 if the woman is Catholic and 0 otherwise). The interpretation is as follows:

  • The intercept (2.284) represents the average number of children for the reference group (non-Catholic women).
  • The slope coefficient (-0.166) represents the estimated difference in the average number of children between Catholics and the rest.

Therefore, the interpretation is: Catholic women in this sample have an estimated 0.166 fewer children on average than non-Catholic women. This is an unadjusted association between two groups, not a causal effect of religion.

(c) Provide a summary of the model. What is the t-value corresponding to the catholic variable? What does that imply?

The full model summary is printed by the code above.

The t-value corresponding to the catholic variable is -1.49.

This t-value implies that the estimated coefficient (-0.166) is approximately 1.5 standard errors away from zero. Since this value is smaller than the typical critical value of ~1.96 (for a 5% significance level), it suggests that the difference in the number of children between Catholic and non-Catholic women is not statistically significant. The corresponding p-value (P>|t|) is 0.133, which is more than 0.05, confirming this conclusion. We do not reject the null hypothesis that there is no difference between the two groups.

The algebra of OLS

Consider the following small dataset of 4 observations for variables \(x\) and \(y\):

Observation (i) y x
1 2 1
2 7 2
3 6 4
4 9 5

(a) Calculate the sample means, \(\bar{x}\) and \(\bar{y}\).

  • Sample mean of y: \(\bar{y} = \frac{2 + 7 + 6 + 9}{4} = \frac{24}{4} = 6\)
  • Sample mean of x: \(\bar{x} = \frac{1 + 2 + 4 + 5}{4} = \frac{12}{4} = 3\)

(b) Using the OLS formula, calculate the slope estimator \(\hat{\beta}_1\).

First, we set up a table to calculate the numerator \(\sum (x_i - \bar{x})(y_i - \bar{y})\) and the denominator \(\sum (x_i - \bar{x})^2\).

\(y_i\) \(x_i\) \((y_i - \bar{y})\) \((x_i - \bar{x})\) \((x_i - \bar{x})(y_i - \bar{y})\) \((x_i - \bar{x})^2\)
2 1 \(2-6=-4\) \(1-3=-2\) \((-4) \times (-2) = 8\) \((-2)^2 = 4\)
7 2 \(7-6=1\) \(2-3=-1\) \(1 \times (-1) = -1\) \((-1)^2 = 1\)
6 4 \(6-6=0\) \(4-3=1\) \(0 \times 1 = 0\) \((1)^2 = 1\)
9 5 \(9-6=3\) \(5-3=2\) \(3 \times 2 = 6\) \((2)^2 = 4\)
Sum - - - 8 - 1 + 0 + 6 = 13 4 + 1 + 1 + 4 = 10

Now, we can calculate \(\hat{\beta}_1\): \[ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{13}{10} = 1.3 \]

(c) Using the formula, calculate the intercept estimator \(\hat{\beta}_0\).

\[ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} = 6 - (1.3 \times 3) = 6 - 3.9 = 2.1 \]

(d) Write down the Sample Regression Function (SRF). Then, calculate the four residuals and verify that their sum is zero.

The Sample Regression Function (SRF) is: \[ \hat{y} = 2.1 + 1.3x \]

Now we calculate the predicted values (\(\hat{y}_i\)) and the residuals (\(\hat{u}_i = y_i - \hat{y}_i\)) for each observation:

  • Observation 1: \(\hat{y}_1 = 2.1 + 1.3(1) = 3.4 \implies \hat{u}_1 = 2 - 3.4 = -1.4\)
  • Observation 2: \(\hat{y}_2 = 2.1 + 1.3(2) = 4.7 \implies \hat{u}_2 = 7 - 4.7 = 2.3\)
  • Observation 3: \(\hat{y}_3 = 2.1 + 1.3(4) = 7.3 \implies \hat{u}_3 = 6 - 7.3 = -1.3\)
  • Observation 4: \(\hat{y}_4 = 2.1 + 1.3(5) = 8.6 \implies \hat{u}_4 = 9 - 8.6 = 0.4\)

Finally, we verify that the sum of the residuals is zero: \[ \sum \hat{u}_i = (-1.4) + 2.3 + (-1.3) + 0.4 = 0 \] The sum is indeed zero, confirming this fundamental algebraic property of OLS.

From residuals to a \(t\)-test

(a) With \(n=27\), the regression estimates two parameters:

\[ SER=\sqrt{\frac{SSR}{n-2}} =\sqrt{\frac{300}{25}} =\sqrt{12}\approx3.464. \]

The typical residual is therefore about 3.464 units of \(y\).

(b)

\[ SE(\hat\beta_1) =\frac{SER}{\sqrt{\sum_i(x_i-\bar x)^2}} =\frac{\sqrt{12}}{\sqrt{48}} =0.5. \]

(c) The hypotheses are \(H_0:\beta_1=0.5\) and \(H_A:\beta_1\neq0.5\). Thus

\[ t=\frac{\hat\beta_1-\beta_{1,0}}{SE(\hat\beta_1)} =\frac{1.5-0.5}{0.5}=2. \]

The reference distribution has \(n-2=25\) degrees of freedom.

(d) Because \(|2|<2.06\), we fail to reject \(H_0\) at the 5% level. The estimate is not sufficiently far from 0.5, relative to its sampling uncertainty, to reject that population value with this test. The decision does not prove that \(\beta_1=0.5\). Nor does statistical significance—or its absence—establish whether the relationship is causal; that requires a credible exogeneity argument.

Error, residual, and a tempting mistake

(a) The error \(u_i=y_i-(\beta_0+\beta_1x_i)\) is the gap from the true population line. It is unobservable because \(\beta_0\) and \(\beta_1\) are unknown. The residual \(\hat u_i=y_i-(\hat\beta_0+\hat\beta_1x_i)\) is the gap from the fitted sample line and can be computed from the data.

(b) The first-order conditions for minimizing the sum of squared residuals are

\[ \sum_i\hat u_i=0,\qquad \sum_i x_i\hat u_i=0. \]

OLS chooses the coefficients to make these equations hold. They therefore hold for any dataset fitted with an intercept, whether the model is credible or hopeless.

(c) A zero sample correlation between \(x\) and \(\hat u\) is the second first-order condition, not evidence about the population error. Exogeneity instead requires the untestable condition \(E(u\mid x)=0\). Confusing \(\hat u\) with \(u\) turns a mechanical identity into a false credibility claim.

(d) Evaluate the fitted line at \(x=\bar x\):

\[ \hat y(\bar x) =\hat\beta_0+\hat\beta_1\bar x =(\bar y-\hat\beta_1\bar x)+\hat\beta_1\bar x =\bar y. \]

Thus the line passes through \((\bar x,\bar y)\). This too follows from how OLS chooses its coefficients and provides no evidence that the model is credible.

When does OLS recover the truth?

(a) The four simple-linear-regression assumptions are:

  1. SLR.1, linearity in parameters: \(y=\beta_0+\beta_1x+u\).
  2. SLR.2, random sampling from the population.
  3. SLR.3, variation in \(x\).
  4. SLR.4, zero conditional mean: \(E(u\mid x)=0\).

SLR.4 rules out a systematic relationship between \(x\) and unobserved determinants of \(y\). Under SLR.1–SLR.4, OLS is unbiased.

(b) Ability raises wages, so \(\beta_{ability}>0\). Ability and education are positively related, so \(\delta_{educ}>0\). Their product is positive: omitting ability gives an upward bias, and OLS overstates the return to education.

(c) Disadvantage lowers scores, so \(\beta_{disadvantage}<0\). Because disadvantaged pupils tend to be in smaller classes, disadvantage and class size are negatively related: \(\delta_{class\ size}<0\). The product is positive. This upward bias makes a negative class-size coefficient less negative and, if the bias is large enough, can push it above zero.

(d) A larger sample increases precision by making the sampling distribution tighter. It does not restore zero conditional mean, so neither omitted-variable bias disappears. Unbiasedness concerns where the sampling distribution is centred; precision concerns how spread out it is. More data can produce a very precise estimate centred on the wrong value.

Marginal effects

(a) The Quadratic Model: For \(y = \beta_0 + \beta_1 x + \beta_2 x^2 + u\), find \(\frac{dy}{dx}\).

  • To find the marginal effect of \(x\) on \(y\), we take the partial derivative of \(y\) with respect to \(x\): \[ \frac{\partial y}{\partial x} = \frac{\partial}{\partial x} (\beta_0 + \beta_1 x + \beta_2 x^2 + u) \] \[ \frac{\partial y}{\partial x} = 0 + \beta_1 + 2\beta_2 x + 0 \] \[ \frac{\partial y}{\partial x} = \beta_1 + 2\beta_2 x \]
  • This result shows that the marginal effect of a one-unit change in \(x\) on \(y\) is not constant; it depends on the current level of \(x\). For each value of \(x\), the slope of the relationship is different.

(b) The Level-Log Model: For \(y = \beta_0 + \beta_1 \log(x) + u\), show that a 1% change in \(x\) leads to an approximate change of \((\beta_1/100)\) units in \(y\).

  1. First, find the derivative of \(y\) with respect to \(x\): \[ \frac{dy}{dx} = \beta_1 \frac{1}{x} \]
  2. Rearrange the equation to find an expression for an infinitesimal change in \(y\), \(dy\): \[ dy = \beta_1 \frac{dx}{x} \]
  3. The term \(\frac{dx}{x}\) represents the proportional or percentage change in \(x\). For discrete changes, we can write this as an approximation: \[ \Delta y \approx \beta_1 \frac{\Delta x}{x} \]
  4. If we consider a 1% change in \(x\), then \(\frac{\Delta x}{x} = 0.01\).
  5. Substitute this value into the approximation: \[ \Delta y \approx \beta_1 (0.01) = \frac{\beta_1}{100} \]
  • Thus, a 1% change in \(x\) is associated with an approximate change in \(y\) of \((\beta_1/100)\) units.

Variance of the OLS estimator

The variance formula is \(\operatorname{Var}(\hat\beta_1\mid X)=\sigma^2/SST_x\). Greater precision means a smaller variance.

  1. Reduce noise. More uniform soil, water, sunlight, and measurement conditions can reduce \(\sigma^2\), the unexplained variation in yield.
  2. Spread out fertilizer doses. A wider range of \(x\) raises \(SST_x=\sum_i(x_i-\bar x)^2\). Observing plots far apart in dose is more informative about a slope than observing nearly identical doses.
  3. Use more plots. If the distribution of doses is maintained, adding observations raises \(SST_x\): there are more squared deviations in the denominator. Sample size therefore helps through the amount of variation in \(x\), not as a separate term in the formula.

Self-study

R-squared

Why is a high R-squared not necessarily the ultimate goal? What is often more important?

  • A high R-squared is not the ultimate goal because it only measures goodness-of-fit, not causal validity. A model can have a very high R-squared but still suffer from severe omitted variable bias, making its coefficients unreliable for policy decisions. For example, a model predicting crime rates using ice cream sales might have a high R-squared in the summer, but the relationship is spurious.
  • What is often more important is a credible, unbiased estimate of the coefficient that answers the causal question. That requires a defensible exogeneity argument and attention to threats such as omitted-variable bias, even when the resulting \(R^2\) is low.

OLS minimization

Why do we use the sum of squared residuals? Why not absolute values or just the sum?

  • Why not the sum of residuals? Minimizing \(\sum e_i\) is not a useful criterion. An infinite number of lines can make this sum equal to zero (any line passing through the point of means, \((\bar{x}, \bar{y})\)), so it does not yield a unique solution.

  • Why we use the sum of squared residuals:

    1. Treats Positive/Negative Errors Equally: Squaring makes all errors positive, so large positive errors and large negative errors are treated as equally “bad”.
    2. Penalizes Large Errors More: Squaring gives much more weight to large errors than to small ones (e.g., an error of 2 becomes 4, but an error of 10 becomes 100). This is often desirable, as it forces the line to fit the bulk of the data well by avoiding large deviations.
    3. Mathematical Convenience: The sum of squares is a smooth, differentiable function. Using calculus, we can easily derive a unique, closed-form analytical solution for the estimators \(\hat{\beta}_0\) and \(\hat{\beta}_1\). Minimizing the sum of absolute values (Least Absolute Deviations, or LAD) is computationally more complex and may not have a unique solution.

Polynomials

(a) Economic theory may predict diminishing marginal returns, a scatterplot may show curvature, or a linear model’s residuals may show a systematic U-shaped or inverted-U pattern. Each suggests that a constant slope is too restrictive.

(b) The marginal effect is

\[ \frac{\partial\widehat{wage}}{\partial experience} =2-0.08\,experience. \]

At 10 years it is \(2-0.08(10)=1.2\) wage units per additional year. At 30 years it is \(2-0.08(30)=-0.4\). The turning point solves \(2-0.08\,experience=0\), so it occurs at 25 years.

(c) The negative quadratic coefficient makes the fitted relationship concave. Wages initially rise with experience at a declining rate, peak at 25 years in this model, and then decline. The coefficient on experience alone is not the effect of experience except at \(experience=0\).

Takeaways

What should you be able to do now?

  1. Write the sentence that a slope coefficient supports, in the units of the data, for level-level, log-level, level-log, log-log, quadratic, and dummy specifications.
  2. Compute OLS estimates, residuals, the standard error of the regression, and a \(t\)-statistic by hand, and verify the algebraic properties of the fitted line.
  3. Estimate a simple regression in R or Python and report it as an equation together with the statistical evidence on its slope.
  4. State SLR.1–SLR.4 and use the omitted-variable-bias formula to sign the bias from a specific omitted determinant.
  5. Distinguish an association, a good in-sample fit, and a causal effect, and say what \(R^2\) does not establish.