Pew Research: Devoted "all our attention and resources” for India religion survey

(Representative image)

In an interview to Moneycontrol on July 1, former India Chief Statistician Pronab Sen raised certain questions on various surveys conducted by US-based Pew Research on India and said for any statistical surveys to be valid, it needs to be truly representative of the population being surveyed and the sample needs to have minimum size.

Commenting on two surveys conducted by US-based Pew research on religion in India and a previous one conducted in 2017, Sen said both samples are inadequate.

"For any national survey in India, there needs to be a sample size of minimum 70,000. For a survey that talks about states, sample size needs to be 1,20,000 and for district level surveys, sample needs to be 5.5 lakh to 6 lakh," Sen said.

The recent Pew research survey on religion in India showed that 76 percent of Hindus, who say being Hindu is very important to being truly Indian, feel it is very important to stop Hindu women from marrying into another religion. By comparison, 52 percent of Hindus who place less importance on Hinduism’s role in Indian identity hold this view about religious intermarriage.

Moreover, Hindus in the Northern (69 percent) and Central (83 percent) parts of the country are much more likely than those in the South (42 percent) to strongly link Hindu identity with national identity. Together, the Northern and Central regions cover the country’s “Hindi belt,” where Hindi, one of the dozens of languages spoken in India, is most prevalent.  The Pew survey also said the vast majority of Hindus in these regions strongly link Indian identity with being able to speak Hindi.

Pew interviewed 30,0000 people for this survey. In 2016-17, Pew had released another survey which found that Narendra Modi remains the most popular Indian politician. That was among the main findings of a Pew Research Center survey conducted among 2,464 respondents in India from February 21 to March 10, 2017.

Sen said "The 2,464 is ridiculously low, so the error of the estimate will be very large. It is not a robust estimate. 30,000 is a little better. But still it isn’t up to the mark."

Exclusive Interview: Pew survey bias in non-US countries cannot be ruled out, says former chief statistician Pronab Sen

Responding to Pronab Sen's comments and a detailed email questionnaire sent by Moneycontrol, Jonathan Evans, Research associate at Pew Research Center, who was the primary project manager for the India religion survey, gave the following responses.

You can find a detailed, in-depth write-up about the survey’s methodology towards the end of the 200+ page report, you can go to it directly by going here: https://www.pewforum.org/2021/06/29/appendix-a-methodology-12/

The two surveys are part of two different research projects. The 2017 survey was conducted by the Global Attitudes team as part of a larger, annual global survey on political attitudes. For more details, see here.

The new survey on religion India was conducted by the Religion and Public Life team as part of a larger effort by Pew Research Center to understand religious change and its impact on societies around the world. The Center previously has conducted religion-focused surveys across sub-Saharan Africa, in the Middle East-North Africa region and many other countries with large Muslim populations, in Latin America, in Israel, in Central and Eastern Europe, in Western Europe, and in the United States.

Typically, we don’t conduct single-country studies outside the United States. We feel that the best contribution we can make to the existing body of international public opinion research is with multi-national, comparative studies.

That said, India is unique in several different ways. For one, it is home to the vast majority (more than 90%) of the world’s Hindus. It’s the second-most populous country in the world, and within its massive population there are stark regional and linguistic differences and tremendous religious diversity. In order to capture a careful look at religion in India, including sufficient samples of several religious minority groups, we decided to devote all our attention and resources for this project on India.

We chose survey respondents through a probability-based design. First, we sampled districts from most states and union territories across India. Within these districts, we then sampled subdistricts. Districts and subdistricts were chosen probabilistically – meaning all of them had a chance of being selected – but we gave certain areas higher chances of selection if they had more hard-to-reach religious groups based on data from the 2011 Indian census. This allowed us to identify areas more likely to be home to religious minority groups. (During data analysis, these locations were weighted to offset their increased likelihood of selection.) Finally, within subdistricts, we sampled villages and census enumeration blocks (CEBs) with a probability of selection based on their total populations.

Field teams traveled to the roughly 3,000 selected villages and CEBs across 26 states and three union territories in India to conduct interviews. At each location, the teams used a systematic method to select the houses to contact. Then, with cooperation from a household member, the interviewer created a list of all the adults living in the household. Finally, one eligible household member was randomly selected to complete the survey.

This survey was conducted face-to-face.

The survey includes respondents from 29 states and union territories. The only states where we were unable to conduct fieldwork were Manipur and Sikkim due to the first wave of COVID-19 stopping fieldwork.

Pew Research Center is located in the United States, so for this project we worked with local Indian survey companies and their interviewing teams to conduct interviews. Pew Research Center staff were in frequent contact with these local firms to manage the project.

Pew Research Center conducts all if its studies independently. We do not have clients, no one can commission a Pew Research Center study. We are charitably funded (also see question 9) and we conduct our research because we see data as a public good. We want to generate a foundation of facts that enriches the public dialogue and supports sound decision-making.

It may be interesting for you to know that we consider ourselves a “fact tank”, not a think tank, because we do not take policy positions and we don’t make recommendations based on our findings.

Using a survey of about 30,000 respondents to know the attitudes and behaviors of India’s more than 750 million adults is possible through 1) probability-based sampling and 2) weighting.

1) Probability-based sampling is when people in the country (both those who do and do not participate) have a known, calculatable non-zero chance of being selected to participate in the survey. For this survey, roughly 98% of adults in the country had a known chance of being selected to participate. The only adults who were excluded were those living in places that posed a safety risk to interviewers (such as the Kashmir Valley) and those that were too remote or generally inaccessible (such as living in the Andaman and Nicobar Islands).

In this survey, out of all the households selected to participate, 86% of interviews with randomly selected individuals were completed. However, we know that some people are more likely than others to participate in surveys.

2) To weight the survey data, we first adjusted for the different probabilities of being chosen to participate in the survey. We then accounted for different willingness levels to participate through an iterative process which aligned our sample to look like the national population in regard to age, gender, education, urbanicity and geographic location using India’s most recent census data (2011).

As with any survey, there is a margin of error. In this study, our overall Indian respondents have a margin of error of 1.7 percentage points. We can say with 95% confidence that the error attributable to collecting data from some, rather than all, adults in India is plus or minus 1.7 percentage points. This means that in 95 out of 100 samples of the same size and following our same sampling and weighting procedures, the results would vary by no more than plus or minus this margin of error.

The Center’s most recent study of religion in India, as well as its previous Global Attitudes surveys, were not commissioned by another party. The Center does not have clients or advertisers. No one can commission a Pew Research Center study.

Pew Research Center is a subsidiary of The Pew Charitable Trusts, our primary funder. We partner strategically with philanthropists and institutional funders who share our commitment to impartial research and data that drive discussion.

The Pew Charitable Trusts is an independent nonprofit organization — the sole beneficiary of seven individual trusts established between 1948 and 1979 by two sons and two daughters of Sun Oil Company founder Joseph N. Pew and his wife, Mary Anderson Pew.

This particular study was funded by The Pew Charitable Trusts and the John Templeton Foundation. It is part of the Pew-Templeton Global Religious Futures project, a larger effort by Pew Research Center to understand religious change and its impact on societies around the world.

The Center previously has conducted religion-focused surveys across sub-Saharan Africa; the Middle East-North Africa region and many other countries with large Muslim populations; Latin America; Israel; Central and Eastern Europe; Western Europe; and the United States.

Pew Research Center conducts its research independently. Neither the John Templeton Foundation nor any other funder is involved in the collection or reporting of our survey data. The John Templeton Foundation did not have access to the survey’s results until our report was written and ready for publication. For more details about how we are funded, please see here