Multilevel regression with poststratification

Multilevel regression with poststratification (MRP) is a statistical technique used for correcting model estimates for known differences between a sample population (the population of the data one has), and a target population (a population one wishes to estimate for).

The poststratification refers to the process of adjusting the estimates, essentially a weighted average of estimates from all possible combinations of attributes (for example age and sex). Each combination is sometimes called a "cell". The multilevel regression is the use of a multilevel model to smooth noisy estimates in the cells with too little data by using overall or nearby averages.

One application is estimating preferences in sub-regions (e.g., states, individual constituencies) based on individual-level survey data gathered at other levels of aggregation (e.g., national surveys).

Individual seat polls can struggle to have a high enough sample size, while MRPs have such large sample sizes that even smaller sub-demographics (eg grouping by age, or cultural background) will have a high enough sample size, which can then be used to adjust seat forecasts. Since the mid-2010s, MRP has seen rapid adoption by commercial pollsters and academic election forecasters as a means of producing seat-by-seat or district-by-district estimates from a single large national survey, particularly in the United Kingdom and United States.

Mathematical formulation

Following the MRP model description, assume $Y$ represents single outcome measurement and the population mean value of $Y$, $\mu _{Y}$, is the target parameter of interest. In the underlying population, each individual, $i$, belongs to one of $j=1,2,\cdots ,J$ poststratification cells characterized by a unique set of covariates. The multilevel regression with poststratification model involves the following pair of steps:

MRP step 1 (multilevel regression): The multilevel regression model specifies a linear predictor for the mean $\mu _{Y}$, or the logit transform of the mean in the case of a binary outcome, in poststratification cell $j$,

$$g{\left({\mathrm {\mu } }_{j}\right)}=g{\left(E{\left[Y_{j{\lbrack i\rbrack }}\right]}\right)}={\mathrm {\beta } }_{0}+{\boldsymbol {X}}_{j}^{T}\mathbf {\beta } +\sum _{k=1}^{K}a_{l{\lbrack j\rbrack }}^{k},$$

where $Y_{j\lbrack i\rbrack }$ is the outcome measurement for respondent $i$ in cell $j$, $\beta _{0}$ is the fixed intercept, ${\boldsymbol {X}}_{j}$ is the unique covariate vector for cell $j$, ${\mathrm {\beta } }$ is a vector of regression coefficients (fixed effects), $a_{l{\lbrack j\rbrack }}^{k}$ is the varying coefficient (random effect), $l{\lbrack j\rbrack }$ maps the $j$ cell index to the corresponding category index $l$ of variable $k\in \{1,2,\cdots ,K\}$. All varying coefficients are exchangeable batches with independent normal prior distributions $a_{l}^{k}\sim \mathrm {N} \left(0,\mathrm {\sigma } _{k}^{2}\right),\ l\in \{1,\dots ,L_{k}\}$.

In practice, MRP implementations commonly extend the varying coefficients to include group-level predictors, which substantially improves estimation for geographic units with few respondents. For a state-level random effect, for example, the exchangeable prior can be replaced with $$a_{l}^{\text{state}}\sim \mathrm {N} \left(\gamma _{0}+\gamma _{1}Z_{l}+a_{m[l]}^{\text{region}},\ \sigma _{\text{state}}^{2}\right),$$ where $Z_{l}$ is a state-level covariate (such as prior-election vote share or median income) and $a_{m[l]}^{\text{region}}$ is a region-level random effect. This hierarchical structure allows the model to borrow strength not just from the overall mean but from states that resemble state $l$ on observed covariates.

MRP step 2: poststratification: The poststratification (PS) estimate for the population parameter of interest is ${\hat {\mu }}^{PS}={\frac {\sum _{j=1}^{J}N_{j}{\hat {\mu }}_{j}}{\sum _{i=1}^{J}N_{j}}}$ where ${\hat {\mu }}_{j}$ is the estimated outcome of interest for poststratification cell $j$ and $N_{j}$ is the size of the $j$-th poststratification cell in the population. Estimates at any subpopulation level $s$ are similarly derived $${\hat {\mu }}_{s}^{PS}={\frac {\sum _{j=1}^{J_{s}}N_{j}{\hat {\mu }}_{j}}{\sum _{i=1}^{J_{s}}N_{j}}}$$ where $J_{s}$ is the subset of all poststratification cells that comprise $s$.

Estimation and uncertainty: The MRP model is typically fit by fully Bayesian estimation using Markov chain Monte Carlo, often via Stan or its higher-level interfaces such as rstanarm and brms. Uncertainty in the poststratified estimate is obtained by applying the poststratification step separately to each posterior draw of the cell-level means ${\hat {\mu }}_{j}$, yielding a posterior distribution over ${\hat {\mu }}^{PS}$ from which credible intervals are constructed.

The technique and its advantages

The technique essentially involves using data from, for example, censuses relating to various types of people corresponding to different characteristics (e.g. age, race), in a first step to estimate the relationship between those types and individual preferences (i.e., multi-level regression of the dataset). This relationship is then used in a second step to estimate the sub-regional preference based on the number of people having each type or characteristic in that sub-region (a process known as "poststratification"). The multilevel component produces estimates for each cell that are a weighted average of the raw cell mean and the overall or group mean, a property known as partial pooling; the weight toward the pooled mean increases as the cell's sample size shrinks, stabilizing estimates in cells with few respondents without discarding the information they do contain. In this way the need to perform surveys at sub-regional level, which can be expensive and impractical in an area (e.g. a country) with many sub-regions (e.g. counties, ridings, or states), is avoided. It also avoids issues with consistency of survey when comparing different surveys performed in different areas. Additionally, it allows the estimating of preference within a specific locality based on a survey taken across a wider area that includes relatively few people from the locality in question, or where the sample may be highly unrepresentative.

History

The technique was originally developed by Gelman and T. Little in 1997, building upon ideas of Fay and Herriot and R. Little. It was subsequently expanded on by Park, Gelman, and Bafumi in 2004 and 2006. It was proposed for use in estimating US-state-level voter preference by Lax and Philips in 2009. Warshaw and Rodden subsequently proposed it for use in estimating district-level public opinion in 2012. Later, Wang et al. used survey data of Xbox users to predict the outcome of the 2012 US presidential election. The Xbox gamers were 65% 18- to 29-year-olds and 93% male, while the electorate as a whole was 19% 18- to 29-year-olds and 47% male. Even though the original data was highly biased, after multilevel regression with poststratification the authors were able to get estimates that agreed with those coming from polls using large amounts of random and representative data. Since then it has also been proposed for use in the field of epidemiology.

Use in United Kingdom elections

YouGov used the technique to successfully predict the overall outcome of the 2017 UK general election, correctly predicting the result in 93% of constituencies. In the 2019 election MRP was adopted by other pollsters including Survation. By the 2024 general election, MRP had become a standard method for seat-level forecasting in the UK, with published models from YouGov, Ipsos, Survation, More in Common, Savanta and Focaldata, among others.

Use in United States elections

In the United States, MRP has been used both in academic work on state- and district-level public opinion and in commercial election forecasting. The approach underpins parts of the methodology used by the Economist's presidential forecasting model, developed in collaboration with Andrew Gelman and colleagues, which combines state-level polls with demographic poststratification to produce state-by-state estimates. Democratic data firms such as Catalist have also used MRP-style methods to produce estimates of the composition of the electorate from voter files and survey data.

Limitations and extensions

MRP can be extended to estimating the change of opinion over time and when used to predict elections works best when used relatively close to the polling date, after nominations have closed. The accuracy of MRP estimates also depends on the availability and quality of the poststratification frame: if the cell population totals are drawn from an outdated census, or if important explanatory variables are missing from the frame, poststratification cannot fully correct for sample bias.

The method has also drawn critical assessments from some prominent election forecasters. At Harvard's 2018 Political Analytics Conference, FiveThirtyEight founder Nate Silver said that MRP "can be good, but it's overrated too", calling it "the Carmelo Anthony of election polling" and arguing that conventional methods already get forecasters most of the way toward an accurate answer.

Both the "multilevel regression" and "poststratification" ideas of MRP can be generalized. Multilevel regression can be replaced by nonparametric regression or regularized prediction, and poststratification can be generalized to allow for non-census variables, i.e. poststratification totals that are estimated rather than being known. Ghitza and Gelman (2013) extended MRP by allowing structured interactions between demographic and geographic variables — for instance age-by-state or ethnicity-by-state varying coefficients with their own multilevel priors — capturing heterogeneity in behaviour across subgroups that a purely additive specification would miss. For ordinal covariates such as age, education, or income, structured priors such as random walks or Gaussian processes over category indices further improve estimation by encoding the natural ordering of these variables rather than treating categories as exchangeable.