What are mixed-effects models?

In a traditional general linear model (GLM), all of our data are independent (e.g., one data point per person). Statistically, we can write this as a linear model like: \[y_i=\beta_0+\beta_1(Time_i)+\epsilon_i\]

Each subject’s actual score (\(y_i\)) is the result of an intercept (\(\beta_0\)) and that constant is modified based on Time (the slope, \(\beta_1\) multiplied by the Time variable). The intercept and slope are collectively referred to as our statistical MODEL. Our model is not going to be perfect, however, so we need to include an error term (\(\epsilon_i\)). Good models will have small errors and thus be a better approximation of our DATA. As such, we can more generally say that: \[Data_i = Model_i + Error_i \]

Mixed-effect regressions are an extension of the general linear model, but they include random-effects in addition to the more traditional fixed-effects of our models. Theoretical definitions of these effects can pretty intense, but you can think about fixed-effects as variables where all of the levels we are interested are present in the data (e.g., our treatment and control groups are fixed. If someone is interested in other treatments, they need to run another experiment). Random-effects, in contrast, are those variables that reflect variables that we are interested in generalizing outside of a particular study (e.g., if we randomly sample students from different schools, both students and schools could be treated as random-effects, because we want to generalize the results to other students/schools whom we didn’t even measure).

In practice, I find it helpful to think about the fixed-effects as the variables about which we are testing hypotheses, and the random-effects allow us to account for statistical dependencies in our data, making sure we get the standard errors correct for the calculation of z-, t-, or F-statistics. For instance, consider the following situation:

These hypothetical data come from a cross-sectional study of younger and older adults. Both groups (hypothetically) walked in anxiety provoking conditions (let’s say we simulated a virtual alligator behind them) that initially led them to walk faster than they normally would. After repeated exposures however (4 trials), both groups started to walk slower.

It would be tempting to model these data using our traditional GLM where there was a fixed-effect of time. However, that would ignore the fact that time varies within each person and this violates one of our primary regression assumptions: that residuals are independent of each other.

To ensure that we have independent residuals, we need to account for the fact that we have multiple observations per person. The first step in this direction is to add a random-effect of subject: \[y_{ij}=\beta_0+U_{0j}+\beta_1(Time_{ij})+\epsilon_{ij}\] The random-effect of subject (\(U_j\)) allows each subject to have a separate intercept (\(\beta_0+U_{0j}\)) for each person. As such, we would refer to this model as a random-intercepts; fixed-slope model, because even though each subject has a unique intercept all subjects would have the same slope (\(\beta_1\)).

If we wanted to estimate a unique trajectory (i.e, slope) for each subject, then we we would need to add a random-slope to our model: \[y_{ij}=\beta_0+\beta_1(Time_{ij})+U_{0j}+U_{1j}(Time_{ij})+\epsilon_{ij}\] To show the effects specifically on the slopes and intercepts, this equation can be rewritten as: \[y_{ij}=(\beta_0+U_{0j})+(\beta_1+U_{1j})(Time_{ij})+\epsilon_{ij}\]

In this random-intercepts; random-slopes model, we estimate a unique trajectory for each person (\((\beta_1+U_{1j})(Time_{ij})\)). Visually, that model would look something like this:

## `geom_smooth()` using formula 'y ~ x'
## `geom_smooth()` using formula 'y ~ x'

Linear trajectories for each group are plotted as solid lines (older adults) and dashed lines (younger adults). The thick black lines represent the group level trajectories (\(\beta_0 +\beta_1(Time_{ij})\)) in each group. The estimated trajectories for each subject are color-coded based on the individual subjects. The intercepts for these lines are captured by the group-level intercept plus the individual distance from that intercept (\(\beta_0+U_{0j}\)). The slopes for these lines are captured by the group-level slope plus the individual distance from that slope (\((\beta_1+U_{1j})(Time_{ij})\)). Note that these random-effects (\(U\)’s) could be positive or negative, because they represent how a given participant deviates from the group. Thus, our mixed-effects MODEL is the combination of our fixed-effects (all of the \(\beta\)’s) and the random-effects (all of the \(U_j\)’s). However, \(DATA = MODEL + ERROR\) still applies, so we need to include a random-error term for each data point, \(ϵ_{ij}\).

In summary, we have the following terms to explain our DATA:

  1. The MODEL includes fixed effects and random effects.
  2. Fixed-Effects are the group-level \(\beta\)’s, these effects parallel the traditional main-effects and interactions that you have probably encountered in other statistical analyses.
  3. Random-Effects are the participant-level \(U_j\)’s that account for statistical dependencies in our data. (This is bit of a simplification, but you can think of not including the appropriate random-effects like running a between-subjects ANOVA when you should be running a repeated-measures ANOVA.)
  4. The ERRORS, or more specifically Random Errors, are the difference between our MODEL’s predictions and the actual DATA, \(\epsilon_{ij}\)’s.

But our model doesn’t look very linear?

Correct! In looking at the figures, it certainly doesn’t look like a straight line is the best description our data. There appear to be diminishing returns in the effect of trial; there is a large reduction in velocity from Trial 1 to Trial 2, but that reduction gets smaller to Trial 3 and to Trial 4. Mathematically, we could try explain this curvature using curvilinear model or a non-linear model.

A curvilinear model creates a curving line, but is linear in its parameters. The most common way this is accomplished is adding polynomials to our model (e.g., \(x, x^2, x^3\)). For instance, in the equation below, our model is linear its parameters (\(\beta\)s added to \(\beta\)s ), but by raising \(x\) to different powers and adding those factors together, we can make a curvilinear relationship between \(x\) and \(y\). \[y_i = \beta_0 + \beta_1(x_{1i}) +\beta_2(x^2_{2i})+\epsilon_i\]

In contrast, a non-linear model is not linear in its parameters. For instance, relationships that follow a power-function or an exponential-function (shown below) do not result from linear combinations (simple addition/subtraction) of the parameters and instead have more complex relationships. \[y_i = \alpha+\beta*e^{(-\gamma/x_i)}\] We can model non-linear relationships using mixed-effects regression, but that is more complicated topic that we will need to save for a later time. For now, let’s focus on what a curvilinear model might look like in our data:

Visually, this curvilinear model looks like it is providing a much better explanation our data, because there is a closer correspondence between our model estimates (the lines) and the real data (individual data points). Within each group, these lines would come from a mixed-effects model that looks like this: \[y_{ij}=(\beta_0+U_{0j})+(\beta_1+U_{1j})(Time_{ij})+(\beta_2+U_{2j})(Time^2_{ij})+\epsilon_{ij}\] The thick black lines correspond to the group-level estimates (\(\beta\)’s) and the thin lines correspond to the estimates for each individual participant (\((\beta+U)\)’s). It looks like our curvilinear model has explained a lot of the within-participant variability, because the difference between our estimates and the data (\(\epsilon\)’s) are very small. However, there does seem be a fair amount of variability between participants (\(U\)’s) that remains to be explained.

I hope this brief introduction gives you some sense of what mixed-effects regression is and what it can do. Mixed-effect regression is a very useful analytic tool when it comes to longitudinal data or in designs where the same participants are repeatedly exposed to different conditions (i.e., repeated measures/within-subject designs). Although mixed-effects regression is very useful in these study designs, the more commonly used method of analysis is repeated measures analysis of variance (RM ANOVA).

RM ANOVA is a perfectly valid method of analysis for a lot of study designs, but in many contexts, researchers use a RM ANOVA when a mixed-effect regression might be more appropriate or effective. Lohse, Shen, and Kozlowski (2020) provide a more detailed contrast of these two methods, but I have recreated some of the central arguments from that paper below.

Constrasting Mixed-Effects Regression and RM ANOVA:

Conclusions

I know that is a lot, but I hope it helps you find your footing with mixed-effect models. There are a whole host of topics that we haven’t covered yet, but we will dig into at least some of those topics in the next chapters. Mixed-effect regression is an incredibly flexible and powerful method for analyzing your data, but that flexibility comes at a cost. Analytic flexibility also means greater complexity and there are a lot of choices that an analyst must make that can have significant influence on the results. These modules will be no means be exhaustive, but I want to introduce you to:

Finally, I will provide a list of resources that I hope will be helpful on your journey deeper into mixed-effects models. These chapters provide a foundation, but fall far short of teaching you everything you need/want to know. Fortunately, there are some other great textbooks and online modules that can help you learn.

References

Lohse, K. R., Shen, J., & Kozlowski, A. J. (2020). Modeling Longitudinal Outcomes: A Contrast of Two Methods. Journal of Motor Learning and Development, 8(1), 145-165. doi: https://doi.org/10.1123/jmld.2019-0007