AVVM • Pozorblog

Almost a month ago, I reported doubts about the poll produced by the previously unknown pollster AVVM. Last week, AVVM issued a new poll, but though it only increased my doubts, I decided not to report on it because even too-credulous (or circulation focused) Pravda described the results as “strange” (cudny) and a search of other major news outlets produce other stories worth responding to and trying to correct. And yesterday an article in SME by Miroslav Kern (“Pre-election juggling with preferences“) dealt nicely with the issue in broader terms. I try not to do what the Slovak press is already doing well (and despite my occasional criticism, there is a lot that they do well) and so I figured there was nothing to write about here.

And then, in a search for results of another idiosyncratic poll (Presov University, on which more later) I found myself on the website of Novy cas.

I generally do not read Novy cas, as it does not have a particularly serious reputation, but I realize now that I should read it more often, for a variety of reasons. My first reaction to the Novy cas election website (after getting over my annoyance that it required me to download Microsoft Silverlight to even work) was “Wow.” Not only does the website look great (waving Slovak flag and all) but it had a “Your Ideal Quiz” (because of Silverlight I can’t actually link to it but the page it is on is here: http://volby.cas.sk/) that actually asked pretty good questions, showed the result of each answer on the suitability of particular parties and, when I tried to make answers point to a certain party, actually increased the score for that party. So far so good.

Novy cas also has lots of party information all in one place: tagged news stories, party list, party program (with an interesting mini-wordle that shows the most common word in the party’s program), and development of party preferences over time. This last, however, raised some questions: it listed only one poll per week, but polls in Slovakia are not done on a weekly basis so the polls must come from different sources and so cannot be easily compared. It did not list the sources, so I had to reconstruct it by comparing the data to my own poll database. The first of the four turned out to be FOCUS, the second Polis and the third and fourth…AVVM (a detail that Kern didn’t know or was too polite to mention). Let me restate that in stronger terms:

The self-proclaimed “most read daily newspaper on the internet” is (without telling anybody) basing its election infographics on the product of an untested polling firm with polls that are highly problematic.

Spot the Problematic Poll

How do we know that AVVM is problematic. In my previous post on AVVM’s first poll, I showed that the firm’s results were far from the median polling numbers of other firms or even from the overall range of those results. The second AVVM poll does not change this assessment in any meaningful way, as the following eight graphs show (AVVM is in yellow; the Presov University poll that I discuss below is in pink):

For the current government parties, AVVM’s second poll is not so out of line. In its first poll AVVM produced exceptionally low results for Smer (orange), though by the second poll this had moved up (and other polls had moved down) until it lay in a more normal range. Likewise AVVM is in the normal range for SNS and HZDS.

Among the Slovak opposition parties, AVVM produced fairly average results for SaS and KDH but quite high for SDKU.

It is for smaller parties that AVVM produces the oddest numbers (both in real and especially in percentage terms. Most-Hid results are fairly average but numbers for SMK are exeptionally low–lower even than the low results produced by Median.

Of course the biggest variation from the norm, and the one that attracted Pravda‘s “strange” headline is the party’s results for AZEN, the new party formed by HZDS defectors Zdenka Kramplova and Milan Urbani. I have not created a dashboard graph for this party because its numbers are actually so low that most polling firms do not report it. To compensate for the lack of full data, I’ve created an approximation graph which gives AZEN the benefit of the doubt (assuming that it just barely misses the threshold for reporting). Even giving it the benefit of the doubt, the results are striking:

AVVM results for AZEN are so far from the norm of the other four major firms (and from the Presov University poll as well) that they simply cannot be taken seriously. It may be that AVVM is right and the others are wrong, but nothing in my experience would suggest that to be the case.

Furthermore, the explanations for this divergence given to SME by AVVM director Martin Palásek are so bad as to disqualify the firm from any further consideration:

The first explanation: “the party is first on the alphabetical list of parties” and may therefore draw additional support (To prove I’m not making this up: “Konateľ AVVM Martin Palásek výsledok AZEN vysvetľuje najmä tým, že sú prví v abecede, a tak ich uvádzali aj na anketových lístkoch”.) If AVVM hasn’t figured out some way to control for this then either a) its polls do not deserved to be published anywhere or b) I will be able to win a significant share of Slovakia’s vote simply by registering a new party under the name “Aardvark Alliance”
The second explanation: Party chair Urbani “is among the most popular HZDS politicians” (“ponúkol tvrdenie, že Urbáni patril k populárnym politikom HZDS“). This is in some ways even worse as it reveals a willingness to follow conventional wisdom rather than the hard data which is the only currency of pollsters (except of course that they accept money from parties to do polls). In fact my I cannot find any evidence that Urbani has ever appeared among the lists of “most trusted” politicians conducted regularly by MVK, even though these sometimes contain as many as 30 names. So much for AVVM.

Spot the Ambitious (but Still Problematic) Poll

In the same general category as AVVM but with important specific differences are the polls conducted by university students at Presov University. These are fairly consistent with the other polls: just a bit high for Smer and SMK, a bit low for HZDS and SDKU and Most-Hid. For three parties the Presov numbers are further from the norm: low but moving in the right direction for SaS, high and moving in the wrong direction for SNS and shockingly high for Unia. For those who have forgotten, Unia is a merger of three economic and cultural liberal parties: Slobodne Forum (SF), which split from SDKU in the 2004, Civic Candidates (OK) which split from SDKU in 2008 (I think), and Liga -Civic Liberal Party (Liga-OLS) founded in 2008 by former officials of SDKU and ANO. Merging three parties with extremely low preferences into a single party and then adopting an entirely new name seems in retrospect to have been poor tactics as since its formation Unia has regularly polled less than the previous totals for its component parties. Except in the Presov University polls, as the graph below shows:

As with AVVM on AZEN, it is hard to take the Presov University numbers seriously on Unia since they are so far out of line even giving the party the benefit of the doubt on polls where its results are not reported.

It is hard to be as critical of this effort as it is of AVVM, first because it has not been used by a major newspaper without any effort to mention problems or discriminate among its strengths and weaknesses (though that is more the fault of Novy cas than AVVM itself), and second because from everything I can find out about it, it is a laudible educational effort that is open about its methodological limits, cites the geographical areas in which the poll-takers worked, and openly discusses its choice of question and the rationale for it. I am still not sure that I understand how they actually went about the questioning or whether their decision to use party leader names rather than party names is a good way to measure preferences, but at least they are trying something new and explaining why they are trying it. For that I give them great credit, even if I don’t feel like I can include their poll results in my overall average.

If a previously unknown astronomer announces the discovery of a major comet, major news outlets might be forgiven for not immediately running with the story. It could, of course, be another Hale-Bopp (or Deep Impact) or it could be dust on the lens. This is the problem the Slovak media faces with regard to a recent poll by a survey firm that has never before entered the party preference fray: AVVM, the Agency for Research on Public Opinion. Do media outlets pick up AVVM’s press release and run with it or do they wait for a second or third poll from this firm to see if it is doing good work. The answer is that they run with it (with the apparent exception, so far, of SME).

So what do we get from a poll by AVVM? Perhaps something, perhaps not. AVVM’s website is the simplest imaginable, an out-of-the box 4 page website (one of which is blank) with no logo or other distinguishing characteristics that would say “take me seriously.” On the other hand AVVM’s director Martin Palasek has worked previously (and maybe still does) for other firms. I will say this: AVVM certainly did a great job of getting out the news of its first poll. A quick search for the firm produces no records from any time before this week, but in the last two days the poll story has already been picked up by dozens of Slovakia’s news outlets. Before make any judgment about the firm itself, I will wait for first hand reports from those in the public opinion field in Slovakia who may know better.

If we cannot judge from the firm’s records, perhaps we can begin to judge the firm by its results. This is always difficult, of course, because we have no way of knowing that AVVM is not right and the others wrong. But to do this we at least need to put the AVVM numbers in context and array them against those of other firms whose accuracy we can judge (and whom we have judged in the past). The graphs below show individual polls and trendlines with the AVVM result in bright yellow. The conclusions:

The poll’s results are well within the expected range for every party except the two largest, Smer and SDKU. For these AVVM shows a strikingly low figure for Smer (about 10% below the normal range, though in the same direction as the trendline) and a rather high figure for SDKU (about 15% above the normal range, for which there is no ready trendline justification. Of course anomalies for only two out of eight parties is rather normal for polls, and without a track record we have no way of judging whether these are outliers for the firm itself.
The poll tends to show lower results than other polls for parties of the current coalition–its number for Smer is exceptionally low but its numbers for SNS and HZDS are on the low end of the range–and higher results for the Slovak parties of the opposition–its number for SDKU is exceptionally high and its number for KDH is on the high side of the range, while only its number for SaS is near the middle.
The AVVM poll differs from the major polls in showing Most-Hid well ahead of SMK. Other polls show both low (Median) or both high (Polis) or MKP-SMK above Most-Hid (FOCUS, MVK)
The AVVM poll also shows a surprisingly high result for the HZDS splinter AZEN (a result which I find unlikely) and for the revived SDL (again not likely to be translated into election results). This could account for the unexpectedly low levels of support for Smer, since these otherwise obscure parties lay in the same ideological space.

So who benefits from AVVM’s report this week (regardless of its accuracy)?

Most-Hid, which here (and nowhere else) appears to be the only viable Hungarian party. If nothing else, this is a valuable antidote to the upcoming Median poll which, if the past is any guide, will show Most-Hid below the threshold (because of Median’s unusual questioning methods).
Opponents of the current coalition. A poll showing Smer to be weak may help embolden the opposition. A continued weak poll showing for HZDS may add one more impression to an overall image that it will be unable to cross the threshold discourage its marginal voters from “wasting their vote.”
The leaders of AZEN and SDL. These parties have little chance but this must make them feel as if they are close to a breakthrough.

So what to make of AVVM’s poll overall? It’s not absurd, not far off, but there’s also no reason to pay it much heed just yet. With any luck the party will publish a June poll so that we can compare it to final results and make a better evaluation. Until then I’m not adding it to my averages but I will keep track of it and give the firm an individual analysis.