Exclusive Interview: Pew survey bias in non-US countries cannot be ruled out, says former chief statistician Pronab Sen

Former chief statistician of India Pronab Sen.

A recent survey by the US-based Pew research on June 29 said that 76 percent of Hindus, who say being Hindu is very important to being truly Indian, feel it is very important to stop Hindu women from marrying into another religion. By comparison, 52 percent of Hindus who place less importance on Hinduism’s role in Indian identity hold this view about religious intermarriage.

Moreover, Hindus in the Northern (69 percent) and Central (83 percent) parts of the country are much more likely than those in the South (42 percent) to strongly link Hindu identity with national identity. Together, the Northern and Central regions cover the country’s “Hindi belt,” where Hindi, one of the dozens of languages spoken in India, is most prevalent.  The Pew survey also said the vast majority of Hindus in these regions strongly link Indian identity with being able to speak Hindi.

The methodology

According to Pew website, the survey was conducted based on a face-to-face survey of 29,999 Indian adults fielded between late 2019 and early 2020 – before the COVID-19 pandemic and the survey took a closer look at religious identity, nationalism and tolerance in Indian society. The survey was conducted by local interviewers in 17 languages and covered nearly all of India’s states and union territories.

As a point of comparison, in 2016, Pew had released another survey which found that Narendra Modi remains the most popular Indian politician. That was among the main findings of a Pew Research Center survey conducted among 2,464 respondents in India from February 21 to March 10, 2017.

Moneycontrol sent an email to Pew on June 30 to understand how the research house selected the respondents and why the sample size is low. Responding to the queries, Pew Research Center said it has devoted all its "attention and resources for this project on India".

Pronab Sen, former chief statistician of India, in an exclusive interview to Moneycontrol on July 1, said low sample size results in higher error margin. Also, while Pew research has an unbiased image in the US, one cannot rule out the possibility of Pew surveys carrying some bias in other countries, Sen said.

Edited excerpts:

Do you think there is a serious issue with the surveys conducted from abroad with smaller samples?

I don’t know exactly how they do it. But look, one of the things about sampling theory is that sample size does not in itself lead to bias. But, if you have an insufficient sample size, then the quality of the estimate will be worse. If you work out the error, the error will be very large. So with a 2,400 sample, one cannot take a view. Even 30,000 is little. But that is not the point.

Could you explain?

The question is how have they (Pew) carried out the survey. So, the sampling procedure itself becomes very important. How have they selected that sample? What is the basis of speaking to the households or selecting the respondent? These are very important. Because when you choose your sample, from let us say, a telephone directory, you will get samples from that. But, the problem will of course be that you will be looking only at people who have telephones and would exclude everyone who doesn’t have a telephone. It will be completely non-representative.

What is an unbiased survey?

For a survey to be unbiased, it has to be truly random. There cannot be a factor which will systematically distort the sample selection. That is one issue. The 2,400 is ridiculously low, so the error of the estimate will be very large. It is not a robust estimate. 30,000 is a little better. But still it isn’t up to the mark.

What should be an ideal sample for Indian surveys?

Ideal of course is much larger. But if you want a sample which is only nationally representative, ie. if you don’t want to make any statements about states, then you will need about 60,000 to 70,000 minimum sample size. But, if you want to make any statement at the state level, the minimum needed is 1,20,000. If you want to do it at district levels, you will need a sample of about 5.5 lakh to 6 lakh.

That’s about the sample. What about the part of conducting the survey?

If there is a face-to-face survey, you are likely to get better answers than if you conduct a survey telephonically. Because then you can actually show the person your credentials. If you do telephonically, if the issue is a sensitive one, it is quite likely that the person may not actually tell you what he or she truly believes but what he or she thinks you want to hear. You think of it, you will feel vulnerable if somebody calls you to say they are doing a survey. I don’t know who this person is and for what purpose the survey is being done. My fear is you know exactly who I am. You are the one calling me. But I don’t know who you are. So I may not actually truly reveal what I actually believe.

What is your opinion about the Pew surveys on India?

The question is whether it is truly representative. As far as Pew is concerned, their reputation in the US is that they don’t have any specific bias. They don’t usually do surveys because they are paid by a particular interested party. But, at the end of the day, Pew is a commercial organisation. In other countries, whether they do paid for surveys--is possible. You will not know.

So you are saying that one cannot fully rule out some bias in Pew surveys in countries other than the US. Is that what you saying?

Yes. One cannot rule out that.

What will be the impact of such surveys on the socio-political system in India?

That depends on how seriously this is taken.

I’m not talking about the relatively better-informed audience…

Let me be also very clear. It has nothing to do with better informed or less informed. Except for professionals, nobody actually understands surveys. Generally, everyone tends to go with the data, implicitly believing that the survey is correct. Most people do this. Now, the question is who is picking up the story. For instance, you are a journalist and you are doing this. So, if you are writing a story saying Pew said this, your readers will take that as truth.

Which is why there is a need for clarity on the survey itself…

Until we understand the full methodology (and whether the sample is truly representative), we just don’t know. If I don’t know the methodology, I would not put any credibility to it.

Can you explain how the surveys should be done in the right way?

Selecting a specific set of people who will respond itself is tricky. I can tell you what is being done by most survey organisations around the world. We first look at either the village or an urban ward. We do the random sampling from the census data. You then have the full history of villages or wards. We choose a random sample of these areas. Then within the ward what is usually done, NSS (national sample survey) for instance, is go down to the village offices which will list all the houses, however many there are. Then from the houses we randomly pick up the number of houses. If you do it in any other way and if use any existing list, the chances are that it may not be truly representative. I gave you the example of the telephone directory. From Airtel or Vodafone, if you take the subscriber list, and if you choose randomly from that, the choice may be random but the lists are not random. Because you are excluding a very large section of people.