India Today Poll 98 [Making Sense of Opinion Polls]

How to tell a Rogue Poll

In psephology A "rogue" poll is the counterpart of yellow journalism. Whatever its motivations -- political propaganda, cost-cutting or simple ignorance -- there is a lot of this illegitimate variety in the Indian media. How do you tell a rogue poll from a genuine one? Here is a check-list of the questions an opinion poll must answer about itself before you take its conclusions seriously:
Sample size and spread: How many actual interviews were conducted? At how many locations? How well were these distributed all over the country?
Sampling method: How exactly was the sample selected? For election-related polls the best technique is random selection from the electoral rolls.
Sample profile: How closely does the eventual sample mirror the population profile? Does it have the same proportion of women, Dalits, Adivasis, OBCs, rural folk and illiterates?
Fieldwork: Were the interviews conducted face-to-face? When and where? Was secrecy ensured for the question regarding voting preferences? Were the respondents given a menu of answers to choose from? Or was it an unprompted response?
Margin of error: How wrong could its figures go? There is a statistical technique to measure it in percentage points. Every poll must report it.
Treatment of figures: Were the raw figures of the poll weighted or adjusted in any way? If they were, then how? What proportion of the respondents said "don't know" to a question?

In India, psephology is nothing short of black magic. About as difficult and esoteric as the term itself. By keeping their techniques under heavy wraps, at a safe distance from public scrutiny, the practitioners of this art haven't helped matters very much. No wonder, the awe it inspires is matched only by the dark suspicions it arouses. The most potent of its charms is an opinion poll. Either these are expected to perform miracles, or are believed to be nothing but motivated propaganda. The truth, as usual, lies somewhere in the middle.

It is a somewhat systematic form of doing what many of us do most of the time, namely gather what others think on a particular subject. Opinion polls try to collate this information, usually on public issues, by interviewing a sample of the population whose opinion we are interested in. The assumption is that their views mirror those of the entire population. The opinion could relate to any subject, from sports to foreign policy. One of the applications of this technique is to ascertain people's voting preferences before the election has taken place. In popular imagination, this special application has come to stand for opinion polls.

The first stage is the selection of a sample. Pollsters select locations that mirror the entire area. In a typical election survey, it means choosing sample constituencies and polling booths. The exercise is then extended to the selection of some voters who resemble the entire population. Usually all this is done on paper before the next stage of sending investigators to conduct face-to-face interviews with the selected persons. Their answers are then recorded. When asking sensitive questions, like on voting, good pollsters use dummy secret ballot to encourage honest response. All the responses are then collated and computed. The conclusions are drawn on the assumption that the opinion of the sample correctly reflected the opinion of the entire population.

Contrary to popular belief, pollsters do not sit down to make a political assessment of each seat on the basis of newspaper reports and their own impressions. Election forecasting is basically a numbers game involving two steps. First, the pollsters conduct an opinion poll to find out who people in different parts of the country intend to vote for. In the past decade Indian forecasters have usually carried out their polls in 50 to 100 parliamentary constituencies and have interviewed 5,000 to 10,000 potential voters to predict the outcome of national elections. This gives them an idea of the vote share for various parties in different states. But that is not enough to predict who gets how many seats. Under our electoral system, the percentage of votes has no direct relationship to the percentage of seats won by a party. The second step involves a translation of votes into seats through elaborate statistical techniques.

Everyone assumes that they do. But no one bothers to find out the facts. In the recent debate about banning opinion polls, no one remembered that after the 1996 elections, the ICSSR-CSDS-INDIA TODAY poll had tried to do so. Ordinary voters all over the country were asked if they had read or heard about opinion poll-based election forecasts. An overwhelming majority hadn't. The figure naturally varied with education and media exposure. Of those who had -- and they could be talking about all forms of forecast -- an overwhelming majority said that this knowlege made no difference to their voting decision. The remaining 4 per cent who said they were influenced by polls were asked to spell it out. It emerged that either they were unclear on this point or the polls only confirmed their original decision. In less than 1 per cent cases, the knowledge of poll-based forecast made people switch from what they thought was the losing party to the winning party. So, while everyone is right in assuming that there is a "bandwagon effect" of polls in India, they tend to grossly exaggerate its magnitude.

They do when either of the two steps goes wrong. You can go wrong because the opinion poll was poor in quality and failed to correctly estimate the vote share for different parties. You can also go wrong if the poll was perfect and you just made an error in translating seats into votes. Funnily enough, a correct forecast does not mean that you took both the steps correctly. Very often a dead-on forecast is the result of two errors which happened to cancel each other out. Remember your maths teacher in school who told you that a correct answer did not necessarily mean that you deserved full marks?

Because they are better indicators of popular mood than cocktail circles, press club gossip or hit-and-miss headlines. Unlike anyone from these circles, pollsters try to speak to ordinary voters to find out how they might vote. They may or may not succeed in their seats forecast, but they give a better idea than anyone else of the direction in which the wind is blowing. It was only when opinion poll-based forecasting began in the pages of India Today that election forecasts ceased to be exercises in wild speculation. Contrary to popular impression, the record of election forecasting in India in the past decade is as good as that of any democracy which follows the first-past-the-post system.

Your guide to Poll Jargon

Exit poll: A special kind of opinion poll which is conducted on election day at the exit gate of the polling booth. The basic purpose is to ask people who they voted for, and to quickly predict the election outcome before the actual counting.
Random sample: The ideal text-book technique of sampling which ensures that the sample is without any bias towards anyone. Random, incidentally, is not haphazard. It takes a lot of effort to select a random sample.
Quota sample: A technique of obtaining a representative sample by fixing in advance a quota for all the significant sections of the population. It is cheaper and less difficult to execute than the random sample, but compromises on quality.
Weightages: While analysing the findings, pollsters estimate the degree by which different sections of society may have been under- or over-represented in the sample. They can't change the sample now, but can correct possible distortions mathematically. That is, by reducing or enchancing the 'weight' of their answers in the final calculations.
Swing: It means a change in the percentage of votes for a party over two points in time. If the bjp's vote percentage increases from 20 to 24, it means a 4 per cent swing for it. Evolved by psephologist David Butler for the British elections, swing has acquired very simplistic meaning in the lexicon of the Indian pollsters. It hasn't lost its mystical charms though.

January 26, 1998