Polls and Preference Measurements: Challenges and Opportunities

We are now in the political season in the U.S. Forecasting the preferences of voters is in huge demand. Some polls appear to be acceptable, others not. So it may be worthwhile to step back and examine the basics of predictive science.

Predictive science consists of two fundamental components. First, specification of an appropriate model. Second, estimation of this model with data. A third important component is: inferences about counterfactuals i.e., asking and answering "what if" questions, and estimating causal effects.

However, we should be careful in posing the counterfactuals. When the counterfactuals posed are too far from the data at hand, the inferences drawn from the model and the empirical analyses are not robust or reliable. Essentially, such inferences are speculative, and quite often based on indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, and so this problem can be hard to detect. Often, scholars and forecasters are inadvertently drawing conclusions based more on modeling hypotheses than on their data. For some research questions, history contains insufficient information to be our guide.

Therefore, forecasters develop methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Scholars have used these methods to evaluate the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts.

So measuring preferences is complex -- it requires a good model, credible statistical estimation and thoughtful construction of counterfactuals.