Resources & Publications

All members of the network can share their recent work on media bias here.
Resources
Most recent models are published on Huggingface
[Benchmark, GitHub] MBIB – the first Media Bias Identification Benchmark Task and Dataset Collection
[Dataset, GitHub] BABE – Bias Annotations By Experts
[Scale/Questionnaire to measure bias perception] Do You Think It’s Biased? How To Ask For The Perception Of Media Bias (A set of tested questions to assess media bias perception to be used in any bias-related research)
[Dataset, Zenodo] MBIC -A Media Bias Annotation Dataset Including Annotator Characteristics
Publications
2025
Hinterreiter, Smi; Wessel, Martin; Schliski, Fabian; Echizen, Isao; Latoschik, Marc Erich; Spinde, Timo
NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback Proceedings Article Forthcoming
In: Proceedings of the International AAAI Conference on Web and Social Media (ICWSM'25), AAAI, Copenhagen, Denmark, Forthcoming, (Conditionally accepted for publication).
Abstract | Links | BibTeX | Tags: crowdsourcing, HITL, linguistic bias, media bias, news bias
@inproceedings{Hinterreiter2025NewsUnfold,
title = {NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback},
author = {Smi Hinterreiter and Martin Wessel and Fabian Schliski and Isao Echizen and Marc Erich Latoschik and Timo Spinde},
url = {https://media-bias-research.org/wp-content/uploads/2024/07/Preprint_ICWSM_25_NewsUnfold.pdf},
year = {2025},
date = {2025-06-01},
urldate = {2025-06-01},
booktitle = {Proceedings of the International AAAI Conference on Web and Social Media (ICWSM'25)},
volume = {19},
publisher = {AAAI},
address = {Copenhagen, Denmark},
abstract = {Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31},
note = {Conditionally accepted for publication},
keywords = {crowdsourcing, HITL, linguistic bias, media bias, news bias},
pubstate = {forthcoming},
tppubtype = {inproceedings}
}
2024
Hinterreiter, Smi; Spinde, Timo; Oberdörfer, Sebastian; Echizen, Isao; Latoschik, Marc Erich
News Ninja: Gamified Annotation of Linguistic Bias in Online News Journal Article Forthcoming
In: Proc. ACM Hum.-Comput. Interact., vol. 8, no. CHI PLAY, Forthcoming, (Publisher: Association for Computing Machinery. Conditionally accepted for publication).
Abstract | Links | BibTeX | Tags: crowdsourcing, Game With A Purpose, linguistic bias, media bias, news bias
@article{Hinterreiter2024News,
title = {News Ninja: Gamified Annotation of Linguistic Bias in Online News},
author = {Smi Hinterreiter and Timo Spinde and Sebastian Oberdörfer and Isao Echizen and Marc Erich Latoschik},
url = {https://media-bias-research.org/wp-content/uploads/2024/07/Preprint_News_Ninja.pdf},
doi = {10.1145/3677092},
year = {2024},
date = {2024-10-14},
urldate = {2024-10-14},
journal = {Proc. ACM Hum.-Comput. Interact.},
volume = {8},
number = {CHI PLAY},
abstract = {Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.},
note = {Publisher: Association for Computing Machinery.
Conditionally accepted for publication},
keywords = {crowdsourcing, Game With A Purpose, linguistic bias, media bias, news bias},
pubstate = {forthcoming},
tppubtype = {article}
}
2023
Spinde, Timo; Richter, Elisabeth; Wessel, Martin; Kulshrestha, Juhi; Donnay, Karsten
What do Twitter comments tell about news article bias? Assessing the impact of news article bias on its perception on Twitter Journal Article
In: Online Social Networks and Media, vol. 37-38, pp. 100264, 2023, ISSN: 2468-6964.
Abstract | Links | BibTeX | Tags: Hate speech detection, media bias, Sentiment analysis, Transfer learning
@article{SPINDE2023100264,
title = {What do Twitter comments tell about news article bias? Assessing the impact of news article bias on its perception on Twitter},
author = {Timo Spinde and Elisabeth Richter and Martin Wessel and Juhi Kulshrestha and Karsten Donnay},
url = {https://www.sciencedirect.com/science/article/pii/S246869642300023X},
doi = {https://doi.org/10.1016/j.osnem.2023.100264},
issn = {2468-6964},
year = {2023},
date = {2023-01-01},
journal = {Online Social Networks and Media},
volume = {37-38},
pages = {100264},
abstract = {News stories circulating online, especially on social media platforms, are nowadays a primary source of information. Given the nature of social media, news no longer are just news, but they are embedded in the conversations of users interacting with them. This is particularly relevant for inaccurate information or even outright misinformation because user interaction has a crucial impact on whether information is uncritically disseminated or not. Biased coverage has been shown to affect personal decision-making. Still, it remains an open question whether users are aware of the biased reporting they encounter and how they react to it. The latter is particularly relevant given that user reactions help contextualize reporting for other users and can thus help mitigate but may also exacerbate the impact of biased media coverage. This paper approaches the question from a measurement point of view, examining whether reactions to news articles on Twitter can serve as bias indicators, i.e., whether how users comment on a given article relates to its actual level of bias. We first give an overview of research on media bias before discussing key concepts related to how individuals engage with online content, focusing on the sentiment (or valance) of comments and on outright hate speech. We then present the first dataset connecting reliable human-made media bias classifications of news articles with the reactions these articles received on Twitter. We call our dataset BAT - Bias And Twitter. BAT covers 2,800 (bias-rated) news articles from 255 English-speaking news outlets. Additionally, BAT includes 175,807 comments and retweets referring to the articles. Based on BAT, we conduct a multi-feature analysis to identify comment characteristics and analyze whether Twitter reactions correlate with an article’s bias. First, we fine-tune and apply two XLNet-based classifiers for hate speech detection and sentiment analysis. Second, we relate the results of the classifiers to the article bias annotations within a multi-level regression. The results show that Twitter reactions to an article indicate its bias, and vice-versa. With a regression coefficient of 0.703 (p<0.01), we specifically present evidence that Twitter reactions to biased articles are significantly more hateful. Our analysis shows that the news outlet’s individual stance reinforces the hate-bias relationship. In future work, we will extend the dataset and analysis, including additional concepts related to media bias.},
keywords = {Hate speech detection, media bias, Sentiment analysis, Transfer learning},
pubstate = {published},
tppubtype = {article}
}
2022
Zhukova, Anastasia; Hamborg, Felix; Donnay, Karsten; Gipp, Bela
XCoref: Cross-Document Coreference Resolution in the Wild Proceedings Article
In: Information for a Better World: Shaping the Global Future: 17th International Conference, IConference 2022, Virtual Event, February 28 – March 4, 2022, Proceedings, Part I, pp. 272–291, Springer-Verlag, Berlin, Heidelberg, 2022, ISBN: 978-3-030-96956-1.
Abstract | Links | BibTeX | Tags: Cross-document coreference resolution, media bias, news analysis
@inproceedings{zhukova2022,
title = {XCoref: Cross-Document Coreference Resolution in the Wild},
author = {Anastasia Zhukova and Felix Hamborg and Karsten Donnay and Bela Gipp},
url = {https://doi.org/10.1007/978-3-030-96957-8_25},
doi = {10.1007/978-3-030-96957-8_25},
isbn = {978-3-030-96956-1},
year = {2022},
date = {2022-01-01},
booktitle = {Information for a Better World: Shaping the Global Future: 17th International Conference, IConference 2022, Virtual Event, February 28 – March 4, 2022, Proceedings, Part I},
pages = {272–291},
publisher = {Springer-Verlag},
address = {Berlin, Heidelberg},
abstract = {Datasets and methods for cross-document coreference resolution (CDCR) focus on events or entities with strict coreference relations. They lack, however, annotating and resolving coreference mentions with more abstract or loose relations that may occur when news articles report about controversial and polarized events. Bridging and loose coreference relations trigger associations that may expose news readers to bias by word choice and labeling. For example, coreferential mentions of “direct talks between U.S. President Donald Trump and Kim” such as “an extraordinary meeting following months of heated rhetoric” or “great chance to solve a world problem” form a more positive perception of this event. A step towards bringing awareness of bias by word choice and labeling is the reliable resolution of coreferences with high lexical diversity. We propose an unsupervised method named XCoref, which is a CDCR method that capably resolves not only previously prevalent entities, such as persons, e.g., “Donald Trump,” but also abstractly defined concepts, such as groups of persons, “caravan of immigrants,” events and actions, e.g., “marching to the U.S. border.” In an extensive evaluation, we compare the proposed XCoref to a state-of-the-art CDCR method and a previous method TCA that resolves such complex coreference relations and find that XCoref outperforms these methods. Outperforming an established CDCR model shows that the new CDCR models need to be evaluated on semantically complex mentions with more loose coreference relations to indicate their applicability of models to resolve mentions in the “wild” of political news articles.},
keywords = {Cross-document coreference resolution, media bias, news analysis},
pubstate = {published},
tppubtype = {inproceedings}
}
2021
Spinde, Timo
An Interdisciplinary Approach for the Automated Detection and Visualization of Media Bias in News Articles Proceedings Article
In: 2021 IEEE International Conference on Data Mining Workshops (ICDMW), 2021.
Links | BibTeX | Tags: media bias, news analysis, slanted coverage, text retrieval
@inproceedings{spinde2021g,
title = {An Interdisciplinary Approach for the Automated Detection and Visualization of Media Bias in News Articles},
author = {Timo Spinde},
url = {https://media-bias-research.org/wp-content/uploads/2021/09/Spinde2021g.pdf},
doi = {10.1109/ICDMW53433.2021.00144},
year = {2021},
date = {2021-09-30},
urldate = {2021-09-30},
booktitle = {2021 IEEE International Conference on Data Mining Workshops (ICDMW)},
keywords = {media bias, news analysis, slanted coverage, text retrieval},
pubstate = {published},
tppubtype = {inproceedings}
}
Spinde, Timo; Rudnitckaia, Lada; Mitrović, Jelena; Hamborg, Felix; Granitzer, Michael; Gipp, Bela; Donnay, Karsten
Automated identification of bias inducing words in news articles using linguistic and context-oriented features Journal Article
In: Information Processing & Management, vol. 58, no. 3, pp. 102505, 2021, ISSN: 0306-4573.
Abstract | Links | BibTeX | Tags: bias data set, context analysis, feature engineering, media bias, news analysis, text analysis
@article{SPINDE2021102505,
title = {Automated identification of bias inducing words in news articles using linguistic and context-oriented features},
author = {Timo Spinde and Lada Rudnitckaia and Jelena Mitrović and Felix Hamborg and Michael Granitzer and Bela Gipp and Karsten Donnay},
url = {https://www.sciencedirect.com/science/article/pii/S0306457321000157/pdfft?md5=64e81212b3bfa861d01a6fe3d5b979c3&pid=1-s2.0-S0306457321000157-main.pdf},
doi = {https://doi.org/10.1016/j.ipm.2021.102505},
issn = {0306-4573},
year = {2021},
date = {2021-01-01},
journal = {Information Processing & Management},
volume = {58},
number = {3},
pages = {102505},
abstract = {Media has a substantial impact on public perception of events, and, accordingly, the way media presents events can potentially alter the beliefs and views of the public. One of the ways in which bias in news articles can be introduced is by altering word choice. Such a form of bias is very challenging to identify automatically due to the high context-dependence and the lack of a large-scale gold-standard data set. In this paper, we present a prototypical yet robust and diverse data set for media bias research. It consists of 1,700 statements representing various media bias instances and contains labels for media bias identification on the word and sentence level. In contrast to existing research, our data incorporate background information on the participants’ demographics, political ideology, and their opinion about media in general. Based on our data, we also present a way to detect bias-inducing words in news articles automatically. Our approach is feature-oriented, which provides a strong descriptive and explanatory power compared to deep learning techniques. We identify and engineer various linguistic, lexical, and syntactic features that can potentially be media bias indicators. Our resource collection is the most complete within the media bias research area to the best of our knowledge. We evaluate all of our features in various combinations and retrieve their possible importance both for future research and for the task in general. We also evaluate various possible Machine Learning approaches with all of our features. XGBoost, a decision tree implementation, yields the best results. Our approach achieves an F1-score of 0.43, a precision of 0.29, a recall of 0.77, and a ROC AUC of 0.79, which outperforms current media bias detection methods based on features. We propose future improvements, discuss the perspectives of the feature-based approach and a combination of neural networks and deep learning with our current system.},
keywords = {bias data set, context analysis, feature engineering, media bias, news analysis, text analysis},
pubstate = {published},
tppubtype = {article}
}
Zhukova, Anastasia; Hamborg, Felix; Donnay, Karsten; Gipp, Bela
Concept Identification of Directly and Indirectly Related Mentions Referring to Groups of Persons Proceedings Article
In: Diversity, Divergence, Dialogue: 16th International Conference, IConference 2021, Beijing, China, March 17–31, 2021, Proceedings, Part I, pp. 514–526, Springer-Verlag, Beijing, China, 2021, ISBN: 978-3-030-71291-4.
Abstract | Links | BibTeX | Tags: Coreference resolution, media bias, news analysis
@inproceedings{zhukova2021,
title = {Concept Identification of Directly and Indirectly Related Mentions Referring to Groups of Persons},
author = {Anastasia Zhukova and Felix Hamborg and Karsten Donnay and Bela Gipp},
url = {https://doi.org/10.1007/978-3-030-71292-1_40},
doi = {10.1007/978-3-030-71292-1_40},
isbn = {978-3-030-71291-4},
year = {2021},
date = {2021-01-01},
booktitle = {Diversity, Divergence, Dialogue: 16th International Conference, IConference 2021, Beijing, China, March 17–31, 2021, Proceedings, Part I},
pages = {514–526},
publisher = {Springer-Verlag},
address = {Beijing, China},
abstract = {Unsupervised concept identification through clustering, i.e., identification of semantically related words and phrases, is a common approach to identify contextual primitives employed in various use cases, e.g., text dimension reduction, i.e., replace words with the concepts to reduce the vocabulary size, summarization, and named entity resolution. We demonstrate the first results of an unsupervised approach for the identification of groups of persons as actors extracted from a set of related articles. Specifically, the approach clusters mentions of groups of persons that act as non-named entity actors in the texts, e.g., “migrant families” = “asylum-seekers.” Compared to our baseline, the approach keeps the mentions of the geopolitical entities separated, e.g., “Iran leaders” ≠ “European leaders,” and clusters (in)directly related mentions with diverse wording, e.g., “American officials” = “Trump Administration.”},
keywords = {Coreference resolution, media bias, news analysis},
pubstate = {published},
tppubtype = {inproceedings}
}
2020
Spinde, Timo; Hamborg, Felix; Gipp, Bela
An Integrated Approach to Detect Media Bias in German News Articles Proceedings Article
In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pp. 505–506, Association for Computing Machinery, Virtual Event, China, 2020, ISBN: 9781450375856.
Abstract | Links | BibTeX | Tags: content analysis, frame analysis, media bias, news bias, news slant
@inproceedings{10.1145/3383583.3398585,
title = {An Integrated Approach to Detect Media Bias in German News Articles},
author = {Timo Spinde and Felix Hamborg and Bela Gipp},
url = {https://doi.org/10.1145/3383583.3398585},
doi = {10.1145/3383583.3398585},
isbn = {9781450375856},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020},
pages = {505–506},
publisher = {Association for Computing Machinery},
address = {Virtual Event, China},
series = {JCDL '20},
abstract = {Media bias may often affect individuals' opinions on reported topics. Many existing methods that aim to identify such bias forms employ individual, specialized techniques and focus only on English texts. We propose to combine the state-of-the-art in order to further improve the performance in bias identification. Our prototype consists of three analysis components to identify media bias words in German news articles. We use an IDF-based component, a component utilizing a topic-dependent bias dictionary created using word embeddings, and an extensive dictionary of German emotional terms compiled from multiple sources. Finally, we discuss two not yet implemented analysis components that use machine learning and network analysis to identify media bias. All dictionary-based analysis components are experimentally extended with the use of general word embeddings. We also show the results of a user study.},
keywords = {content analysis, frame analysis, media bias, news bias, news slant},
pubstate = {published},
tppubtype = {inproceedings}
}
Garz, Marcel; Sörensen, Jil; Stone, Daniel F.
Partisan selective engagement: Evidence from Facebook Journal Article
In: Journal of Economic Behavior & Organization, vol. 177, pp. 91-108, 2020, ISSN: 0167-2681.
Abstract | Links | BibTeX | Tags: Filter bubble, media bias, Polarization, Political immunity, Social media
@article{GARZ202091,
title = {Partisan selective engagement: Evidence from Facebook},
author = {Marcel Garz and Jil Sörensen and Daniel F. Stone},
url = {https://www.sciencedirect.com/science/article/pii/S0167268120302079},
doi = {https://doi.org/10.1016/j.jebo.2020.06.016},
issn = {0167-2681},
year = {2020},
date = {2020-01-01},
journal = {Journal of Economic Behavior & Organization},
volume = {177},
pages = {91-108},
abstract = {This study investigates the effects of variation in “congeniality” of news on Facebook user engagement (likes, shares, and comments). We compile an original data set of Facebook posts by 84 German news outlets on politicians that were investigated for criminal offenses from January 2012 to June 2017. We also construct an index of each outlet's media slant by comparing the language of the outlet with that of the main political parties, which allows us to measure the congeniality of the posts. We find that user engagement with congenial posts is higher than with uncongenial ones, especially in terms of likes. The within-outlet, within-topic design allows us to infer that the greater engagement with congenial news is likely driven by psychological and social factors, rather than a desire for accurate or otherwise instrumental information.},
keywords = {Filter bubble, media bias, Polarization, Political immunity, Social media},
pubstate = {published},
tppubtype = {article}
}
Garz, Marcel; Sood, Gaurav; Stone, Daniel F.; Wallace, Justin
The supply of media slant across outlets and demand for slant within outlets: Evidence from US presidential campaign news Journal Article
In: European Journal of Political Economy, vol. 63, pp. 101877, 2020, ISSN: 0176-2680.
Abstract | Links | BibTeX | Tags: Horse race news, media bias, Media slant, Motivated beliefs, Polarization, Selective exposure
@article{GARZ2020101877,
title = {The supply of media slant across outlets and demand for slant within outlets: Evidence from US presidential campaign news},
author = {Marcel Garz and Gaurav Sood and Daniel F. Stone and Justin Wallace},
url = {https://www.sciencedirect.com/science/article/pii/S0176268020300252},
doi = {https://doi.org/10.1016/j.ejpoleco.2020.101877},
issn = {0176-2680},
year = {2020},
date = {2020-01-01},
journal = {European Journal of Political Economy},
volume = {63},
pages = {101877},
abstract = {We conduct across-outlet and within-outlet (and within-topic) analyses of “congenially” slanted news. We study “horse race” news (news on candidates' chances in an upcoming election) from six major online outlets for the 2012 and 2016 US presidential campaigns. We find robust evidence that horse race headlines were slanted congenially with respect to the preferences of the outlets' typical readers. However, evidence of congenial slant in the timing and frequency of horse race stories is weaker. We also find limited evidence of greater within-outlet demand for headlines most congenial to outlets' typical readers, and somewhat stronger evidence of greater demand for relatively uncongenial headlines. We discuss how various aspects of our results are consistent with each of the major mechanisms driving slant studied in the theoretical literature, and may help explain when each mechanism is more likely to come into play. In particular, readers may be more likely to click on uncongenial headlines due to inferring that these stories are particularly informative when they stand in contrast to an outlet's typically congenial slant.},
keywords = {Horse race news, media bias, Media slant, Motivated beliefs, Polarization, Selective exposure},
pubstate = {published},
tppubtype = {article}
}