Poll Analysis: UVVM Volatility

Also available in Slovak, here

I have occasionally expressed concern about the recent numbers coming out of UVVM, particularly those for the Party of the Hungarian Coalition.  Something is wrong there. In the spirit of my heroes at http://www.fivethirtyeight.com, I decided to take a look at the data.  The chart below calculates the total month-to-month differences in party results (up, or down) and then measures these as a percentage of a party’s average support during that period (i.e. a party with an average support of 20 percentage points and average monthly volatility of 1 percentage points would be listed at 5%).  I have done the calculations for the overall average (all of the polls within a given month) and for UVVM and for the two electoral periods for which we have good data: 2002-2006 and 2006-present.

Party 2002-2006 2006-2008
  Overall UVVM Difference Overall UVVM Difference
Smer 7.0% 9.2% 2.2% 8.9% 7.8% -1.1%
SDKU 12.5% 16.4% 3.8% 13.2% 16.2% 3.1%
SNS 14.7% 18% 3.3% 12.1% 12.1% 0.0%
MK 8.0% 12.0% 4.0% 10.1% 16.1% 6.0%
HZDS 9.9% 12.4% 2.5% 12.2% 14.1% 2.0%
KDH 10.4% 16.1% 5.8% 10.2% 10.9% 0.7%
Average 10.4% 14% 3.6% 11.1% 12.9% 1.8%

What is not surprising here is that UVVM alone has much higher levels of volatility than the overall average which includes multiple polls that smooth out the monthly variability.  What is surprising here is the fact that the the difference in volatility of SMK between UVVM and the overall average grew (from 4% to 6%) even as the differences between UVVM and other surveys declined.  Between 2002 and 2006 the volatility of SMK was second to the bottom (after Smer) in both UVVM and overall averages.  Between 2006 and 2008, the SMK volatility stayed the second lowest in the overall averages but rose to highest in UVVM by a wide margin.

This is all particularly surprising since the overall volatility of the Hungarian Coalition’s electorate have remained remarkably stable over time, as the following table demonstrates:

Party (elections in sample) Volatility as a share of average party support Raw volatility
ANO (2002,2006) 103.2% 4.9%
SNS (1994,1998,2002,2006) 80.5% 5.9%
Smer (2002,2006) 73.6% 15.7%
KSS (1994,1998,2002,2006) 51.2% 2%
SDKU (2002,2006) 45.1% 7.5%
HZDS (1994,1998,2002,2006) 38.7% 8.7%
MK (1994,1998,2002,2006) 11.4% 1.2%
KDH (1994,2002,2006) 10.7% 0.9%

There is an easy explanation for this:  UVVM has had difficulty maintaining its Hungarian sample.  This is understandable–this is a difficult task–but it is important to keep this potential problem in mind rather than to assume that all polls actually reflect representative samples.  This should be a question of analysis rather than assumption.  It also raises questions about the representativeness of this sample (and that of other pollsters) for other parties.

I am deeply grateful to UVVM for all the support they have provided to me over time. This is a problem that needs attention but I am hopeful that the institute can return to its previous levels of excellence.