A central question in contemporary populism research is to what extent people hold so-called populist attitudes. In light of this question, various scales have been proposed to measure populist attitudes (see here or Chapters 6 and 7 here for an overview). While the scales differ in various aspects, what they have in common is that, in one way or another, they draw on trailblazing measurement studies that were conducted in the USA or the Netherlands using a similar battery of survey items assumed to measure populist attitudes (see first six items in Table 1, below). At the moment of writing, the scale devised in the Dutch study remains perhaps the most influential battery of items to measure populist attitudes and examine relationships between populism and broader political behaviour. And while such studies are insightful and enrich the discipline, the integrity of their results depends on having an appropriate measurement of populist attitudes.
Survey items of the populist attitudes scale:
populism_1 | The politicians in Congress need to follow the will of the people.
populism_2 | The people, not the politicians, should make our most important policy decisions.
populism_3 | The political differences between the people and the elite are larger than the differences among the people.
populism_4 | I would rather be represented by an ordinary citizen than an experienced politician.
populism_5 | Politicians (elected officials) talk too much and take too little action.
populism_6 | What people call “compromise” in politics is really just selling out on one’s principles.
populism_7 | The particular interests of the political class negatively affect the welfare of the people
populism_8 | Politicians always end up agreeing when it comes to protecting their privileges.
itemsQuestion wordingpopulism_1The politicians in Congress need to follow the will of the people.populism_2The people, not the politicians, should make our most important policy decisions.populism_3The political differences between the people and the elite are larger than the differences among the people.populism_4I would rather be represented by an ordinary citizen than an experienced politician.populism_5Politicians (elected officials) talk too much and take too little action.populism_6What people call “compromise” in politics is really just selling out on one’s principles.populism_7The particular interests of the political class negatively affect the welfare of the peoplepopulism_8Politicians always end up agreeing when it comes to protecting their privileges.
In a recent article in Politics, we argue that the measurement and empirical evaluation of both the individual items in particular and the scale as a whole have yet to receive sufficient attention despite some efforts being made in other studies. With that crucial caveat in mind, we use a cross-national sample (9 countries, n = 18,368) and apply Item Response Theory (IRT) to the most commonly used populism scale in order to evaluate whether that scale (1) accurately extracts people’s populist attitudes from self-reported answers, and (2) provides a pan-European measurement for the European context. This exercise allows us to make three crucial observations that are relevant for (populism) scholars and beyond.
First, the IRT analysis of the traditional 6-item scale shows that all six items have difficulties in capturing high and extreme values of an aggregate populist attitudes instrument. Some of these items, like the ‘elite-procrastination’ item (item 5 in Table 1), perform notably better than others. While the brilliant study by Castanho Silva et al. (2019) confirms this is not uncommon across a wide variety of populism scales and samples, this observation presents a challenge for the findings and measurement of populist attitudes to the extent that most research in this area focuses precisely on the most -and least- populist. Hence, suggesting further scale development is required for accurate inferences.
With that in mind, we propose two additional items to formulate an 8-item scale (items 7 and 8 in Table 1). While our analysis shows the new items clearly outperform some of the traditional items, the addition of these new items gauging the perceived conflict between ‘the people’ and ‘the elite’ does not expand the extent to which we accurately capture the full range of the latent populist construct. For that reason, in a final step, we set out to fit a more parsimonious measurement through the refinement of the 8-item scale that can serve as the basis for future studies. By combining Occam’s razor and psychometric principles, we deduce a core measure comprised of 3 high-performing items covering the hypothesised theoretical concept and consisting of one item from the traditional 6-item scale (item 1 in Table 1) and the two novel items (items 7 and 8 in Table 1).
Overall, the intention of this exercise is not necessarily to provide a ‘minimal’ populist instrument that comprises no more than three items. Rather, we wish to highlight some of the areas for improvement of existing measures and propose a foundation (at least in a pan-European context) that researchers can build on and use as a baseline for further advancement of this field.
Text originally published at Politics Online. Drafted by Steven M. Van Hauwaert, Christian H. Schimpf and Flavio Azevedo and published on 12th July, 2019. Reproduced here for (self)-archiving purposes.