Monday, November 30, 2020
Home Editor's picks Qualitatively garbage: The bogus methodology employed by BBC in their research on 'Fake News'

Qualitatively garbage: The bogus methodology employed by BBC in their research on ‘Fake News’

The most egregious assault on academic rigour was the fact that the researchers took only 7 days to conclude their field immersion.

In an earlier report, OpIndia.com has comprehensively debunked the BBC research on ‘Fake News’. The BBC has tried to brush the criticisms under the garb of ‘Qualitative Research’ and ‘Ethnographic Research’, therefore, it is imperative that we take a moment to understand what these terms actually mean and what such research entails.

For those unaware, Qualitative Research focuses on subjective experience and it’s objective is not to determine objective truth which can be generalized across all sections of the population. It employs techniques such as interviews and on-field observations to develop a greater understanding of a particular social phenomenon. It seeks to answer the questions of ‘why’ and ‘how’ to understand a particular social phenomenon. One of the ontological principles of qualitative research is that there are multiple realities which can be accessed through subjective experiences.

Limitations of Qualitative Research

One of the fundamental positions of Qualitative research is that there are multiple realities and people perceive reality in their own unique way. Thus, the focus is not on discovering an objective reality that holds true across all situations but to explore relationships within a unique setting and based upon the purpose of research and its results, develop hypotheses which could then be tested using Quantitative methods. Qualitative methods cannot be generalized to any significant extent, much less, an entire country.

Even an editor at the BBC claimed that since it isn’t possible to interview 1.2 billion people, ‘there will have to be a generalization’. Anyone who knows anything about Qualitative research knows that generalization is not the objective of such research and the results obtained from it cannot be generalized across all situations to any significant extent. It appears that the BBC isn’t even aware of the purpose and objective of the Qualitative research.

There are some crippling limitations of Qualitative research. It depends a lot on the personal skills and biases of the researcher. Since such research employs methods such as interviews and case studies, a research may choose to focus on one aspect due to his or her own individual biases even though another detail in the study is more significant.

Under such circumstances, it was imperative that researchers in their team included people of diverse political affiliations if they wanted their research to be more authentic. However, the research makes no mention of the political affiliations of the researchers themselves.

Flaws in Research Methodology

Flaws in the using the ‘localised version of left-right orientation questionnaire

The research states that it used a “localised version of the 10 point left-right orientation questionnaire developed by John Curtice and Caroline Bryson” to determine the political affiliation of their subjects to ensure an equal representation. But the reliability and validity of the localized version is not mentioned. And it’s unclear to what extent such a test would be effective in a country like India considering the ‘Left-Right’ paradigm is not applicable in our country. For instance, if the test indicated that a person was likely to vote for Shiv Sena, would he be classified as pro-BJP or anti-BJP?

The researchers state that using the localized version of the test “allowed us to recruit for political affiliation without asking questions about people’s voting behaviours or support for particular political entities.” The researchers need to justify their reasoning here as India has a multi-party political system where the left-right paradigm is not applicable by giving more details about the localized version of the test that was used. Binaries can work for countries like the USA where they have to choose between the Republicans or the Democrats, but in India, that prism gives a skewed result and thus, the validity of methodology employed has to be provided with respect to India.

Flawed definition of ‘Fake News’

The problems with the research begin with the operational definition of ‘Fake News’ itself. The research defines Fake News as “information, however, conveyed, in whichever format, on whatever platform, which is not fully supported by factual evidence”. The researchers appear to be unaware that people across the political spectrum differ on what constitutes as factual evidence and that naturally pervades even the academia whose standards of factual evidence they claim to have followed.

However, if ‘information not fully supported by factual evidence’ is the core definition, then will every message declaring the greatness of Gods be dubbed ‘Fake News’? More realistically, will messages that proudly proclaim “Diversity is our strength” be treated as ‘Fake News’ because there’s evidence which suggests that diversity is definitely not our strength and homogeneous societies tend to function a lot better in almost every respect.

If the aforementioned message is regarded as ‘Fake News’, then every single liberal is guilty of being the purveyor of Fake News. Why haven’t they been named in the list of people who spread Fake News? The fact that OpIndia.com, Swati Goel and R. Jagannathan were mentioned as purveyors of ‘Fake News’, it only goes to show that their definition of ‘Fake News’ and ‘factual evidence’ aren’t reliable at all. And what about that news which was believed to be true at one point of time but was proven as false after a couple of weeks, or maybe even a couple of days, with the revelation of new data? Were those clubbed as Fake News as well considering the study was completed within 3 months?

Flaws in selection and nature of interviews, sample selection and generalisation

The follies in the research do not stop there. There are multiple mistakes to show that the researchers themselves were not very serious about their research. For one, The research was limited to major cities alone. Smaller towns and cities did not have any representation, let alone the rural population. Secondly, 40 subjects were distributed across 10 cities, that is 4 subjects on an average in each city. With such an infinitesimally small sample size, that the researchers hoped to learn anything at all from their study only shows that the entire exercise was merely to provide them with a convenient excuse to peddle their agenda.

Under the best of circumstances, the results of Qualitative research can be generalized only across immediate homogeneous settings and here they have generalized the opinions of an average of four people across 10 cities to the entire nation which just magically happens to be one of the most heterogeneous countries in the entire world.

Moreover, the researchers state that they screened their subjects only for political affiliation and economic background. Any student of social science is aware that the educational qualifications of people play an important role in determining their behaviour in a social setting.

In the ‘Data and Methodology Appendix’ section, the researchers do not mention anywhere that they took the educational qualifications of an individual into account. If the BJP supporters who were part of the study were not very highly educated and the anti-BJP subjects were, then it is natural that their study will come to very dubious conclusions. Moreover, the researchers do not state whether they took into account the mental health of the subjects or any personal history of mental illness or in their family which could be a very important factor as mentally unhealthy people may be more inclined to believe and share ‘Fake News’.

Even the economic background of individuals isn’t mentioned as a criterion in the ‘Data and Methodology Appendix’ section although it is said in Methodology section that subjects came from a ‘mix’ of economic backgrounds. If they indeed came from a mix, then what are the different categories of economic backgrounds the subjects were categorized under? It is not mentioned. Was it ensured that the four groups of 10 subjects each had an equal representation of supporters from various political parties from the various categories of economic background? It was not mentioned.

The researchers haven’t even mentioned the sampling technique they have used. In Qualitative Research, random sampling is not employed. Usually, the purposive sampling technique is used to ensure that researchers have the desired subjects necessary to fulfil the objective of their study. The researchers state that they ensured there was equal representation of various political affiliations among their subjects. But with the multitude of political parties that we have, how did they ensure such a thing and how many supporters of each political party was part of the research exactly?

It should also be borne in mind that it’s true that ethnographic studies generally study a very small population and the BBC is not wrong when they say it explains the small number of people they studied. But they compromise on their ability to generalize across different situations with their small sample. And when the sample is so small, researchers need to have an extra robust sampling technique if they wish to generalize their findings at all. Therefore, what measures were taken to ensure that researcher bias did not creep into sampling? Were any measures taken at all? Unfortunately, the research provides no answer to these questions.

Call for Quantitative research

The conclusion segment of the research states, “Given that central to our observations is the idea that people’s socio-political identities seem to influence their consumption and sharing of fake news quite a bit, and that national pride and nationalistic feelings seem to affect their critical thinking, fake news will need to be handled as a social problem. We would urge future researchers to quantitatively assess – perhaps through regression analysis techniques – just how important socio-political identities are in fake news consumption relative to other factors.”

Thus, it is clear from the conclusion that researchers believe certain aspects of people identities ‘seem’ to felicitate sharing of fake news and call for quantitative research into the matter. Journalists at the BBC, on the other hand, take the leap to “Nationalism a driving force behind fake news in India, research shows” without mentioning that the researchers themselves call for more Quantitative research to determine the extent to which socio-political identities play a part.

Ethnographic or Semi-ethnographic?

The research employs words like ‘ethnographic’ and ‘semi-ethnographic’ liberally. On page number 18, the research states that they used ‘semi-ethnographic approaches’ to describe their interviews while in the ‘Data and Methodology Appendix’, they mention ‘ethnographic in-depth interviews’. Which one was it? Ethnographic or semi-ethnographic? An editor at the BBC also claimed that it was ethnographic research but how could it be if it adopted semi-ethnographic approaches? To the laymen, it may appear trivial but you cannot have that ambiguity in your research if you want it to be taken seriously.

The most egregious assault on academic rigour was the fact that the researchers took only 7 days to conclude their field immersion. In Ethnographic research, the researcher attempts to study people in their own natural setting. This requires the researchers to get to know more about the day to day life of their subjects and the subjects, too, need to get comfortable around the researchers to behave as they would naturally so that the researchers can observe their natural behaviour. Is 7 days enough for such an endeavor? Ethnographic research is unique for the long duration it takes to complete the collection of data. Generally, it takes months if not a whole year and on certain occasions, it takes years to complete the collection of data.

Of course, due to practicality, the time frame is being decreased due to various constraints but 7 days although not unheard of is much too less if you are to make such fantastical claims. One could call it semi-ethnographic (as they did call it in the methodology section) perhaps but it certainly cannot be said that the findings will offer results that could be generalized on such a high scale. It could still be excusable if they had mentioned that field immersion for the ‘ethnographic’ research was completed in 7 days, as research methods are subject to various constraints, but they should have mentioned and highlighted that fact if they were honest about their research.

One of the most fundamental criticisms of ethnographic research is that researcher bias plays too much of a significant role in the conclusions that are drawn. The bias of the research was amply revealed when they used portals like AltNews as their source for Fake News. Therefore, whatever findings the BBC observed from their study can be said to hold true only for those set of 40 people at its worst and maybe, just may, for other groups which bear a close resemblance to that particular group of 40 people. The Qualitative study does not reveal a single thing about the phenomenon of Fake News beyond that which could be generalized for other groups.

The crisis in social science

BBC’s research also points towards the crisis within social sciences. There have been numerous revelations regarding how corruptible social science research is. There is a widespread replication crisis where researchers have failed to reproduce the findings of previous studies and experiments.

On plenty of occasions, especially in high profiled ones, researchers have admitted to falsifying data to produce desired results. The BBC’s research may well be one of them. The million other problems with the research, including how their algorithm worked, has already been elaborated by my editor and therefore, I shall not elaborate on them here. BBC also mentioned that their algorithm selected for portals which shared even a single fake news. If that was indeed the case, why don’t all mainstream media channels including the BBC feature in the list? In conclusion, the BBC research is an assault on academic rigour and Ethnographic research.

  Support Us  

Whether NDTV or 'The Wire', they never have to worry about funds. In name of saving democracy, they get money from various sources. We need your support to fight them. Please contribute whatever you can afford

K Bhattacharjee
Black Coffee Enthusiast. Post Graduate in Psychology. Bengali.

Related Articles

Trending now

Farmer thanks PM Modi for mentioning him during Mann Ki Baat, narrates how the passage of Farm Bills helped him

PM Modi's Mann Ki Baat speech came just as the issue has been raked up once again as ‘farmers’ from Punjab and Haryana have marched towards Delhi to protest against the farm bills.

SFJ ads appear on YouTube offering $10m support for farmers, in return, wants Sikhs to join ‘Khalistan movement’: Details

Amid ongoing farmers’ protests in Punjab, Haryana, and Delhi, it is evident that Khalistani supporters have hijacked the protests and now, they are running ads on YouTube

A year after quitting Congress over ‘petty politics’, Urmila set to join Shiv Sena, to be nominated as MLC

Urmila Matondkar is expected to be nominated as MLC by the Shiv Sena led Maharashtra government

Indian man proposes to his Australian girlfriend during Ind vs Aus cricket match, here is what happened next

While India's performance has been below par, netizens found happiness in the unusual proposal that became the highlight of the match.

NIA refutes claims about ‘activist’ Stan Swamy’s straw and sipper seized by agency. Read details

NIA stated that when Stan Swamy's personal search was undertaken in presence of two independent witnesses no such straw or sipper was found on him as claimed by him.

France, Germany, UK and all other countries we should boycott for using water cannon and tear gas

As a newly woke nation, we also have a moral responsibility to oppose other countries that carry out atrocities such as using water cannons on protesters.

Recently Popular

Wife of late Wajid Khan writes on her sufferings in an inter-faith marriage, says anti-conversion law should be nationalised

Wife of late music composer Wajid Khan wrote how her husband and his family had been forcing her to convert to Islam after their marriage

Who is the man in farmer protest video, lying about ‘govt snatching farmland’: Profile of an actor and Khalistan supporter

Khalistan supporter Punjabi actor Deep Sidhu has been seen spreading the lie that govt will snatch farmer's land using the new farm laws

Actor Deep Sandhu disappoints Barkha Dutt after she provides platform to ‘distance himself’ from OpIndia report that he supports Khalistan

Barkha Dutt lamented after actor Deep Sidhu refused to condemn Jarnail Singh Bhindranwale and to call him terrorist on her YouTube channel

The truth about the viral picture, shared by Rahul Gandhi, where police can be seen ‘beating up’ an old farmer

Congress leaders share cunningly edited version of a video to allege brute force is being used by security officials to quell the 'farmer' protest

Odisha: Largest ever vigilance raid in state unearths massive wealth gathered by IFS officer, fancy cars, ultra-luxurious lifestyle

Pathak's family reportedly owns a lavish apartment in Bhubaneswar with Italian marble flooring, ivory furnitures and other luxuries. The son is engaged to an MLA's daughter.

Netizens call to Boycott tax evasion accused Amrapali Jewels after it pitches Swara Bhasker as its brand ambassador, Amrapali deletes tweet

The official Twitter account of Amrapali Jewels had posted a tweet with a picture of actor Swara Bhasker endorsing its products

Farmer thanks PM Modi for mentioning him during Mann Ki Baat, narrates how the passage of Farm Bills helped him

PM Modi's Mann Ki Baat speech came just as the issue has been raked up once again as ‘farmers’ from Punjab and Haryana have marched towards Delhi to protest against the farm bills.

Rebellion inside Rahul’s Congress grows, Anand Sharma defies party line, hails PM Modi for visit to Coronavirus vaccine hubs

A day after Congress criticised PM Modi's visit to Coronavirus vaccine hubs, senior Congress leader Anand Sharma has defied the party line

From support for zamindars to ‘protest will help overthrow Modi govt’: A saga of clueless people at farmer protest

A series of interview of protestors by the team of Know the Nation reveals how many of these protestors are completely clueless about why they are there

SFJ ads appear on YouTube offering $10m support for farmers, in return, wants Sikhs to join ‘Khalistan movement’: Details

Amid ongoing farmers’ protests in Punjab, Haryana, and Delhi, it is evident that Khalistani supporters have hijacked the protests and now, they are running ads on YouTube

President of farmer organisation says they will block all five entry points to Delhi because Burari park is an ‘open jail’: Details

The farmers protesting against the Modi government over new farm laws have rejected Union Home Minister Amit Shah’s proposal for a discussion on their grievances.

A year after quitting Congress over ‘petty politics’, Urmila set to join Shiv Sena, to be nominated as MLC

Urmila Matondkar is expected to be nominated as MLC by the Shiv Sena led Maharashtra government

‘Haven’t you seen how they cry, do ‘haye tauba’: Here is what Amit Shah said, responding to Owaisi’s comment about illegal Rohingyas

Amit Shah said when he takes action against Rohingyas and Bangladeshis back, leaders like Owaisi stand against him in Lok Sabha.

Indian man proposes to his Australian girlfriend during Ind vs Aus cricket match, here is what happened next

While India's performance has been below par, netizens found happiness in the unusual proposal that became the highlight of the match.

NIA refutes claims about ‘activist’ Stan Swamy’s straw and sipper seized by agency. Read details

NIA stated that when Stan Swamy's personal search was undertaken in presence of two independent witnesses no such straw or sipper was found on him as claimed by him.

France, Germany, UK and all other countries we should boycott for using water cannon and tear gas

As a newly woke nation, we also have a moral responsibility to oppose other countries that carry out atrocities such as using water cannons on protesters.

Connect with us

245,563FansLike
490,955FollowersFollow
20,200SubscribersSubscribe