Do You Think It’s Biased? Best Practices and Recommendations on Assessing Media Bias Perception
Takeaways:
- Gathered 824 questions to assess media bias.
- Grouped them into six factors.
- Semantically reduced them to 48 question contents.
- Tested them in a survey with 663 participants.
- Finally present the most relevant 21 questions.
A questionnaire about media bias – our recommendations
It’s about time – This is the first blog post of our newly created Media Bias Group blog! Here, we will introduce all the topics around media bias and our research. My name is Christoph, and I must say, I feel great being the first one to write here. But anyway, for today, for the first one, I selected a very fundamental and, nevertheless, maximally overlooked topic. Buckle up, because we’re about to dive into the depths of human perception, biases, and a little bit of our own existential crisis!
Let’s start with an important question: How do you know whether someone perceived anything as biased? Anyhow did they perceive it in general? Saying it more biased: Why does your significant other insist on watching that terrible reality TV show, while you know deep down it’s a hot mess of bias?
We face similar questions in our day-to-day research work, trying to understand: How good is our system? Our classifier? Our website? Does it help users to get a more balanced language understanding? Or is there no effect? Or does it even make it worse? To understand how information receivers perceive content (or misinformation, fake news, media bias, racial bias, gender bias, or others), we need to know how we ask them reliably about their perception. But what questions do you need to ask to determine if an article is biased? To our surprise, no one has ever addressed this before, and no standardized questionnaire or standard set of questions is available. Instead, thousands of surveys and research projects design their own questions, without considering the influence of any questions on such a complex topic.
Therefore, we started closing this gap: We collected almost a thousand question sets, analyzed them, reduced and summarized similar ones (while keeping the process transparent), and then tested which ones measure media bias perception most reliably. We have the final questions ready for you to use on THIS LINK. Below, we will explain everything about the process, the paper (which you can find HERE), and our next steps to making the questionnaire more diverse, reliable, and covering slightly different topics.
Let’s get started!
The Journey
Literature Search
Our Journey started by searching what questions were already used in surveys and research projects. To find these we conducted a systematic search on PsychInfo and Google Scholar with the search term “Perception of Media Bias”. This gave us 405 promising papers (going through them was a pleasure…). For those, we scanned the headlines and abstracts and searched for keywords connected to media bias and its perception. After carefully filtering the papers, we reduced the number to 107, for which we retrieved the full text. Unfortunately, we had to remove an additional 46 because there was no or no significant relevant connection to the perception of media bias.
Finally, we ended up with 74 studies that have made it to the next step, in which individual questions are retrieved!
Item Collection
While precisely reading those papers, we extracted every piece of information that was relevant to build a questionnaire. For each information found, we grouped items that contained the relevant information needed in the further process. With this, we found 824 items which contain the following information:
- The original source
- How to answer the question (e.g. yes/no, scales, …)
- What type of bias is asked for? (e.g. Journalist bias, Issue-related bias, …)
- If given the original wording in the study otherwise a reconstruction of it
- supplementary material if provided by the authors
Since nobody wants to answer 824 questions in a single survey, we reduced the number of questions. To have a guideline on what items we want to keep, we formulated four main criteria for the selection. 1) is that the items must be related to media bias. 2) The items should cover different aspects of media bias to gain insights into all of them. 3) The questions measure the media bias on the article level and not e.g. on the sentence or word level. The final main criterion 4) is that the questions should be usable for the Visual Analog Scale (VAS).
With these criteria, we started a three iterations reduction and filtering process.
Let’s start with the first phase, the Categorization!
Categorization
In the first phase, we began by organizing the items into general categories, such as:
- Political Background
- Demographics
- Perception of Media Bias
- Influence of Media Bias
Since this categorization doesn’t reduce the number of questions we have, we chose to proceed with only the questions related to “Perception of Media Bias” and “Influence of Media Bias” in our progress, as these directly address the topic of media bias. The remaining items were revisited later to gather relevant background information, such as demographics or political background, resulting in a list of 419 potential items.
Reduction
To further reduce the number of items, we initially grouped them into five bias measurement categories: Cause, Existence, Direction, Strength, and Influence.
Subsequently, we grouped semantically and topically similar items to create constructs that encompassed as many items as possible without omitting any relevant aspects. This process resulted in a total of 141 items, comprising 42 constructs and 99 general items without specific constructs.
Selection
Since 141 items are still impractical, we further organized them based on their content and selected items that addressed each content aspect in a final phase. Within this process, some questions had to be excluded for the following reasons:
- Unsuitability for a visual analog scale response format
- Lack of specificity in identifying bias in an article
- Absence of original wording for items, with no possibility to create adequate items without it
- Specificity limited to the original study’s issue
- Single usage in the studies
- Inclusion of semantic differentials used by at least two authors in the questionnaire.
The entire process resulted in a final questionnaire of just 48 items. This questionnaire comprises 25 items with varying response formats, 17 semantic differentials, and 6 feeling ratings. To ensure coverage of third-person perception, we included 3 items twice: one set asking about the article’s impact on the participant directly and another set inquiring about its impact on others.
In order to maintain the questionnaire’s generality, we employed the term “another person.” Additionally, we included five placeholder items, which were subsequently replaced with article-specific information.
Let’s briefly recap our progress. Through this entire process, we have managed to reduce our initial set of 824 questions to just 48 questions. These questions are exclusively focused on either the “perception of media influence” or the “influence of media influence.” They have been categorized for measuring bias and are designed to be answered using a VAS.
With this compact questionnaire, we can finally go a step further and test the questions. As mentioned above, with just an example being shown in the next figure; the final results of our selection process are shown on:
Validation
Having crafted our questionnaire, we had to ask ourselves: Did we select the right questions? Do they cover all the necessary aspects of media bias? Could we further reduce the number to focus on the most insightful ones?
To address these crucial questions, we conducted a survey.
Survey
In October 2020, we initiated this process. Within our survey, every participant was tasked with reading a randomly selected article from a pool of 190 articles. For each article, we presented a set of our 48 questions, spread across five pages, utilizing a VAS that ranged from -10 to 10, with responses restricted to whole numbers only. Both the order of the pages and the arrangement of items were randomized, and we thoughtfully included an attention check item to ensure data quality.
In addition to responding to these questions, participants also answered additional questions regarding general media bias and provided demographic information. As a mark of our commitment to ethical research, at the conclusion of the study, we offered participants the option to grant permission for their data to be used for scientific purposes.
The Participants
Our survey was completed by a total of 827 of the 940 recruited U.S. participants. Unfortunately, we later had to exclude some participants due to missing or untrustworthy data. The final sample consisted of 663 participants, with a gender distribution of 53.5% women, 44.8% men, and 1.7% other. The participants’ ages ranged from 18 to 80 years, with an average age of 33.86. The highest level of education varied among participants, with 35.7% having a bachelor’s degree and 18.3% having graduate work. On average, participants reported spending 2.95 hours per day viewing or reading the news.
Article Selection
The selected articles aimed to provide a balanced representation of politically left and right perspectives. We carefully handpicked a total of 190 articles from a wide array of topics, media sources, authors, and writing styles. As a result, our selected articles encompassed a spectrum from unbiased reporting to highly biased commentary, addressing a diverse range of issues, including Coronavirus, Elections, Economy, Racism, Gun Control, Abortion, and Immigration.
Furthermore, in our selection process, we took care to ensure the inclusion of both controversial and less controversial topics. We sourced the articles from Allsides and alternative news outlets, with the additional inclusion of some extreme articles. Notably, the political ideology ratings of these articles spanned from very liberal to very conservative, providing a comprehensive representation of political viewpoints.
Results
When the survey was finally completed, we were able to look at the results. Each of the articles we selected was rated between 1 and 5 times, with an average of 3.49. With the new information we gained, we were finally able to test how well our questions measured the perception of media bias in an article and whether we could reduce the number of questions further.
So we started this process with an Exploratory factor analysis.
Exploratory factor analysis
Since 48 questions are still a substantial number for a single survey, one of the primary goals of this survey was to identify which questions provide the most valuable information. Consequently, armed with this knowledge, we can further streamline our questionnaire by including only those questions that yield the greatest information gains.
So, how did we approach this task? We employed an exploratory factor analysis (EFA) with maximum likelihood estimators. EFA is a statistical method that we use to investigate the underlying structure within a larger initial set of variables (our questions) and to condense them into a more concise set of summary variables.
In our case, we began with our existing 48 questions, aiming to discover patterns or groups of items that share common characteristics and are interconnected. These patterns, referred to as factors, aid in comprehending the relationships between the items and account for their similarities. In the EFA, we calculated the average ratings for the questions and utilized these average ratings to analyze the data. To measure the agreement among different raters for each question, we employed a statistical technique known as Intraclass Correlation (ICC).
To determine the appropriate grouping of our questions, we utilized Velicer’s MAP criterion, which resulted in the identification of 6 factors.
Following the completion of the EFA, we identified the following factors that group questions based on shared characteristics. We assigned descriptive names to these factors:
- Factuality
- Influence
- Topic Affirmation
- Negative Emotion
- Bias
- Political Ideology
So, what does this factorization do for us now? Since we have now grouped questions according to the factors they measure, we can reduce the number of questions in a certain group to a smaller set that yields similar or the same information gains. We selected our final items by measuring the factor loadings of each question and applied a threshold of 0.7 for each group. However, in the case of Factuality, we utilized a threshold of 0.95 to avoid including too many items
With this, we finally ended up with our final 21 questions, distributed like:
- Factuality: 5 questions
- Influence: 2 questions
- Topic Affirmation: 3 questions
- Negative Emotion: 3 questions
- Bias: 6 questions
- Political Ideology: 2 questions
A full overview of the results of this EFA can be found in our paper.
Summary and Future Directions
After this extensive journey, we have arrived at our final set of 21 questions. Let’s briefly recap our progress.
We began with an initial pool of 824 questions sourced from 74 studies, undertaking a systematic approach to construct our questionnaire. This involved stages like categorization, reduction, and final selection, resulting in a refined set of 48 questions.
Subsequently, we assessed the questionnaire’s effectiveness by engaging 663 participants to rate 190 different articles, leading to further refinements. Leveraging the collected data, we conducted an Exploratory Factor Analysis (EFA), which allowed us to condense the questionnaire to its final form of 21 questions.
However, this doesn’t mark the end of our quest for the ultimate Bias Perception questionnaire. While we successfully reduced the questions through the EFA, we encountered limitations in conducting cross-validation due to our limited sample size. Consequently, we embarked on a follow-up study to further test the effectiveness of our questions.
Stay tuned for our next blog post as our journey continues!
