Estimating Heterogeneous Effects

Data Analytics and Visualization with R
Session 8

Viktoriia Semenova

University of Mannheim
Spring 2023

Which of These Hypotheses Specify Conditional Relationsip?

  1. Women representatives are more likely than men representatives to focus on issues of relative importance to women.
  2. The stronger the partisan identification of the voter, the stronger is anger in response to threat of electoral loss.
  3. For candidates that the voter has seen on posters, appearance has a stronger effect on vote choice than for candidates that the voter has not seen on posters.
  4. Women are more likely to substantively represent women if they were elected in districts with larger magnitude.
  5. Government parties are more likely to engage in constructive legislative review than opposition parties.

Dataset: Student Evaluations

glimpse(evals)
Rows: 463
Columns: 9
$ x1         <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20…
$ eval       <dbl> 4.3, 4.5, 3.7, 4.3, 4.4, 4.2, 4.0, 3.4, 4.5, 3.9, 3.1, 4.0, 3.8, 3.4,…
$ beauty     <dbl> 5.000000, 3.000000, 3.333333, 3.166667, 7.333333, 5.500000, 4.166667,…
$ female     <dbl> 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, …
$ age        <dbl> 36, 59, 51, 40, 31, 62, 33, 51, 33, 47, 35, 37, 42, 49, 37, 45, 56, 4…
$ minority   <dbl> 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, …
$ nonenglish <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, …
$ lower      <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, …
$ course_id  <dbl> 3, 0, 4, 2, 0, 0, 4, 0, 0, 4, 4, 0, 0, 0, 9, 9, 10, 0, 0, 0, 9, 8, 10…

Baseline Model

term estimate std.error statistic p.value conf.low conf.high
(Intercept) 3.893 0.177 22.016 0.000 3.545 4.240
beauty 0.075 0.017 4.422 0.000 0.042 0.109
female -0.212 0.052 -4.066 0.000 -0.315 -0.110
age -0.002 0.003 -0.885 0.377 -0.008 0.003
nonenglish -0.334 0.104 -3.228 0.001 -0.538 -0.131

Baseline Model

Heterogenous Effects of Beauty

term estimate std.error statistic p.value conf.low conf.high
(Intercept) 3.751 0.193 19.443 0.000 3.372 4.130
beauty 0.103 0.023 4.501 0.000 0.058 0.148
female 0.056 0.157 0.356 0.722 -0.253 0.365
age -0.002 0.003 -0.712 0.477 -0.007 0.003
nonenglish -0.340 0.103 -3.289 0.001 -0.543 -0.137
beauty:female -0.060 0.033 -1.810 0.071 -0.124 0.005

Heterogenous Effects of Beauty

Heterogenous Effects: Model Specification

\[ \text{Evaluation}_i = \underbrace{\hat\beta_0}_{\text{Expected evaluation}\\\text{for males if}\\\text{beauty score = 0}} + \underbrace{\hat\beta_1}_{\text{Difference in}\\\text{evaluation between}\\\text{males and females}\\\text{if beauty score = 0}} \text{Female}_i + \underbrace{\hat\beta_2}_{\text{Effect of each}\\\text{additional point}\\\text{in score for males}} \text{Beauty}_i \\ + \underbrace{\hat\beta_3}_{\text{Difference in}\\\text{effect of beauty}\\\text{between males}\\\text{and females}} \text{Female}_i \times \text{Beauty}_i \]

  • \(\text{Female}_i\) and \(\text{Beauty}_i\): constitutive terms
  • \(\text{Female}_i \times \text{Beauty}_i\): interaction term

Heterogenous Effects: Model Specification

We are allowing the effects of beauty to vary across the subgroups:

Male instructors:

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times [\text{Female}_i = 0] + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times [\text{Female}_i = 0] \times \text{Beauty}_i \\ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times 0 + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times 0 \times \text{Beauty}_i \\ \text{Evaluation}_i = {\hat\beta_0} + {\hat\beta_2} \text{Beauty}_i \]

Female instructors:

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times [\text{Female}_i = 1] + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times [\text{Female}_i = 1] \times \text{Beauty}_i \\ \text{Evaluation}_i = (\hat\beta_0 + \hat\beta_1) + (\hat\beta_2 + \hat\beta_3 ) \times \text{Beauty}_i \]

Working with Models with Interaction Effects I

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times \text{Female}_i + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times \text{Female}_i \times \text{Beauty}_i \\ \]

Include all constitutive terms (i.e. those variables that compose the interaction), even if you expect that a specific constitutive term alone has no effect on \(Y\)

  • Omitting a constitutive term forces the model to set the value of the omitted coefficient to zero: e.g., if we estimated \(\text{Evaluation}_i = \hat\beta_0 + \hat\beta_2 [\text{Beauty}_i = 1] + \hat\beta_3 [\text{Female}_i = 1] \times \text{Beauty}_i\) instead of the original model, we assumed the \(\hat\beta_1 = 0\)
  • However, \(\hat\beta_1\) is not the average effect of Female for the entire sample but it only shows the effect of Female if Beauty = 0, a subgroup of the sample. It can be the case that on average, the effect of Female is still zero
  • \(\hat\beta_2\) captures the change in the intercepts of the regression lines between the categories of male vs. female instructors. If instead of group-specific intercept we only have one common intercept for both subgroups, the slopes may get distorted

Working with Models with Interactions Effects II

Do not interpret the coefficients on constitutive terms as unconditional marginal effects:

  • the coefficient on X only captures the effect of X on Y when Z is zero

  • the coefficient on Z only captures the effect of Z on Y when X is zero

  • E.g., it is incorrect to say that a positive and significant coefficient on \(X\) (or \(Z\)) means that an increase in \(X\) (or \(Z\)) is expected to lead to an increase in \(Y\)

Wrong: On average, beauty score is associated with an increase in teaching evaluations, holding all else constant.
Wrong: On average, female instructors receive 0.05 points more in teaching evaluations than males, holding all else constant.
Right: On average, beauty tends to have smaller effect on teaching evaluations among female instructors in comparison to male instructors, holding all else constant.

  • To know the average effect of \(X\) on \(Y\) we would have to know how frequently condition \(Z\) is present in our sample

Working with Models with Interactions Effects III

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times \text{Female}_i + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times \text{Female}_i \times \text{Beauty}_i \\ \]

Provide a substantively meaningful description of the marginal effects

  • For linear models without interactions, regression coefficients alone are the quantities of interest since the coefficients represent constant (marginal) effects of variables (i.e. the effects do not depend on values of other variables and can be straightforwardly interpreted)
  • For linear models with interactions, reporting only the coefficients means reporting marginal effect of \(X\) when the conditioning variable \(Z = 0\)
  • However, it is possible we never even observe real-world situations in which \(Z = 0\) (e.g., Beauty score ranges from 1 to 10, hence the effect of Female (\(\hat\beta_1\)) alone does not tell us anything meaningful, as we always need to have at least \(\hat\beta_1 + \hat\beta_3 \times 1\) to compare the differences even for the lowest possible value of beauty score)
  • While we can do some calculations for marginal effects based on the coefficients alone, we do not have enough information from these tables to also calculate uncertainty measures (e.g., standard errors) for these marginal effects

Working with Models with Interactions Effects IV

Do not drop interactions based on the insignificance of the interaction term coefficient

  • Insignificant interaction term coefficient \(\neq\) insignificant marginal effect of \(X\) on \(Y\) for substantively relevant values of the modifying variable \(Z\)

Estimate (SE)
Intercept (\(\beta_0\)) 3.622 (0.100)***
beauty (\(\beta_1\)) 0.105 (0.022)***
female (\(\beta_2\)) 0.061 (0.156)
beauty × female (\(\beta_3\)) -0.059 (0.033)

Note: * p < 0.05, ** p < 0.01, *** p < 0.001

E.g., based on an insignificant \(\beta_3\) we cannot say that there is no significant difference between male vs. female instructors because for certain values in beauty score, we may still observe significant differences between male and female instructors

Causal Interaction vs. Effect Modification

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times \text{Female}_i + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times \text{Female}_i \times \text{Beauty}_i \\ \]

Mathematically, interactions allow us to tell two stories:

  • the effect of Beauty on Teaching Evaluations differs between male and female instructors
  • the effect of Gender on Teaching Evaluations differs across values of beauty score

Interpret the interaction coefficients in line with your causal story and your selected control variables.

Alternative: Using Subsets

m3 <- lm(eval ~ beauty + age + nonenglish, 
         data = evals %>% filter(female == 1))
tidy(m3, conf.int = T, conf.level = 0.9) %>% 
  knitr::kable(digits = 3)
term estimate std.error statistic p.value conf.low conf.high
(Intercept) 4.253 0.259 16.392 0.000 3.824 4.682
beauty 0.034 0.025 1.376 0.171 -0.007 0.075
age -0.011 0.005 -2.409 0.017 -0.019 -0.003
nonenglish -0.117 0.160 -0.731 0.466 -0.382 0.148 
m4 <- lm(eval ~ beauty + age + nonenglish, 
         data = evals %>% filter(female == 0))
tidy(m4, conf.int = T, conf.level = 0.9) %>% 
  knitr::kable(digits = 3)
term estimate std.error statistic p.value conf.low conf.high
(Intercept) 3.499 0.230 15.215 0.000 3.119 3.878
beauty 0.113 0.023 4.885 0.000 0.075 0.152
age 0.002 0.003 0.664 0.507 -0.003 0.008
nonenglish -0.443 0.137 -3.237 0.001 -0.668 -0.217
  • Sample size: interactions rely on more data, hence smaller uncertainty
  • Comparisons are easier across effects with one model
  • Works straightforwardly when the moderator is categorical, continuous moderators require arbitrary decisions about cutoff points
  • Best: do both to make sure your results are robust

Intractions with Continous Variables

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times \text{Age}_i + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times \text{Age}_i \times \text{Beauty}_i \\ \]

  • Marginal effect of Beauty on Teaching evaluation: \(\hat\beta_2 + \hat\beta_3 \times \text{Age}_i\)

  • Marginal effect of Age on Teaching evaluation: \(\hat\beta_1 + \hat\beta_3 \times \text{Beauty}_i\)

What is the Marginal Effect of Beauty?

\[ \text{Evaluation}_i = \hat\beta_0 + \hat\beta_1 \times \text{Age}_i + \hat\beta_2 \text{Beauty}_i + \hat\beta_3 \times \text{Age}_i \times \text{Beauty}_i \\ + \hat\beta_4 \text{Female}_i + \hat\beta_5 \times \text{Female}_i \times \text{Beauty}_i \]

  1. \(\hat\beta_2 + \hat\beta_3 \times \text{Age}_i\)
  2. \(\hat\beta_2 + \hat\beta_3 \times \text{Age}_i + \hat\beta_5 \times \text{Female}_i\)
  3. \(\hat\beta_2 + \hat\beta_5 \times \text{Female}_i\)
  4. \(\hat\beta_3 \times \text{Age}_i + \hat\beta_5 \times \text{Female}_i\)

Plotting Interactions with Continuous Variables

Plotting Interactions with Continuous Variables

Plotting Interactions with Continuous Variables II

References

Brambor, Thomas, William Roberts Clark, and Matt Golder. 2006. “Understanding Interaction Models: Improving Empirical Analyses.” Political Analysis 14 (1). Cambridge University Press: 63–82. doi:10.1093/pan/mpi014.

Hamermesh, Daniel S., and Amy Parker. 2005. “Beauty in the classroom: Instructors’ pulchritude and putative pedagogical productivity.” Economics of Education Review 24 (4): 369-376.

Keele, Luke, and Randolph T. Stevenson. 2021. “Causal Interaction and Effect Modification: Same Model, Different Concepts.” Political Science Research and Methods 9 (3). Cambridge University Press: 641–49. doi:10.1017/psrm.2020.12.

🔗 r4da.live