Resources & Publications
Resources
Most recent models are published on Huggingface
[Benchmark, GitHub] MBIB – the first Media Bias Identification Benchmark Task and Dataset Collection
[Dataset, GitHub] BABE – Bias Annotations By Experts
[Scale/Questionnaire to measure bias perception] Do You Think It’s Biased? How To Ask For The Perception Of Media Bias (A set of tested questions to assess media bias perception to be used in any bias-related research)
[Dataset, Zenodo] MBIC -A Media Bias Annotation Dataset Including Annotator Characteristics
Publications
2025
Hinterreiter, Smi; Wessel, Martin; Schliski, Fabian; Echizen, Isao; Latoschik, Marc Erich; Spinde, Timo
NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback Proceedings Article Forthcoming
In: Proceedings of the International AAAI Conference on Web and Social Media (ICWSM'25), AAAI, Copenhagen, Denmark, Forthcoming, (Conditionally accepted for publication).
Abstract | Links | BibTeX | Tags: crowdsourcing, HITL, linguistic bias, media bias, news bias
@inproceedings{Hinterreiter2025NewsUnfold,
title = {NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback},
author = {Smi Hinterreiter and Martin Wessel and Fabian Schliski and Isao Echizen and Marc Erich Latoschik and Timo Spinde},
url = {https://media-bias-research.org/wp-content/uploads/2024/07/Preprint_ICWSM_25_NewsUnfold.pdf},
year = {2025},
date = {2025-06-01},
urldate = {2025-06-01},
booktitle = {Proceedings of the International AAAI Conference on Web and Social Media (ICWSM'25)},
volume = {19},
publisher = {AAAI},
address = {Copenhagen, Denmark},
abstract = {Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31},
note = {Conditionally accepted for publication},
keywords = {crowdsourcing, HITL, linguistic bias, media bias, news bias},
pubstate = {forthcoming},
tppubtype = {inproceedings}
}
Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31
2024
Hinterreiter, Smi; Spinde, Timo; Oberdörfer, Sebastian; Echizen, Isao; Latoschik, Marc Erich
News Ninja: Gamified Annotation of Linguistic Bias in Online News Journal Article Forthcoming
In: Proc. ACM Hum.-Comput. Interact., vol. 8, no. CHI PLAY, Forthcoming, (Publisher: Association for Computing Machinery. Conditionally accepted for publication).
Abstract | Links | BibTeX | Tags: crowdsourcing, Game With A Purpose, linguistic bias, media bias, news bias
@article{Hinterreiter2024News,
title = {News Ninja: Gamified Annotation of Linguistic Bias in Online News},
author = {Smi Hinterreiter and Timo Spinde and Sebastian Oberdörfer and Isao Echizen and Marc Erich Latoschik},
url = {https://media-bias-research.org/wp-content/uploads/2024/07/Preprint_News_Ninja.pdf},
doi = {10.1145/3677092},
year = {2024},
date = {2024-10-14},
urldate = {2024-10-14},
journal = {Proc. ACM Hum.-Comput. Interact.},
volume = {8},
number = {CHI PLAY},
abstract = {Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.},
note = {Publisher: Association for Computing Machinery.
Conditionally accepted for publication},
keywords = {crowdsourcing, Game With A Purpose, linguistic bias, media bias, news bias},
pubstate = {forthcoming},
tppubtype = {article}
}
Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.
2020
Spinde, Timo; Hamborg, Felix; Gipp, Bela
An Integrated Approach to Detect Media Bias in German News Articles Proceedings Article
In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pp. 505–506, Association for Computing Machinery, Virtual Event, China, 2020, ISBN: 9781450375856.
Abstract | Links | BibTeX | Tags: content analysis, frame analysis, media bias, news bias, news slant
@inproceedings{10.1145/3383583.3398585,
title = {An Integrated Approach to Detect Media Bias in German News Articles},
author = {Timo Spinde and Felix Hamborg and Bela Gipp},
url = {https://doi.org/10.1145/3383583.3398585},
doi = {10.1145/3383583.3398585},
isbn = {9781450375856},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020},
pages = {505–506},
publisher = {Association for Computing Machinery},
address = {Virtual Event, China},
series = {JCDL '20},
abstract = {Media bias may often affect individuals' opinions on reported topics. Many existing methods that aim to identify such bias forms employ individual, specialized techniques and focus only on English texts. We propose to combine the state-of-the-art in order to further improve the performance in bias identification. Our prototype consists of three analysis components to identify media bias words in German news articles. We use an IDF-based component, a component utilizing a topic-dependent bias dictionary created using word embeddings, and an extensive dictionary of German emotional terms compiled from multiple sources. Finally, we discuss two not yet implemented analysis components that use machine learning and network analysis to identify media bias. All dictionary-based analysis components are experimentally extended with the use of general word embeddings. We also show the results of a user study.},
keywords = {content analysis, frame analysis, media bias, news bias, news slant},
pubstate = {published},
tppubtype = {inproceedings}
}
Media bias may often affect individuals' opinions on reported topics. Many existing methods that aim to identify such bias forms employ individual, specialized techniques and focus only on English texts. We propose to combine the state-of-the-art in order to further improve the performance in bias identification. Our prototype consists of three analysis components to identify media bias words in German news articles. We use an IDF-based component, a component utilizing a topic-dependent bias dictionary created using word embeddings, and an extensive dictionary of German emotional terms compiled from multiple sources. Finally, we discuss two not yet implemented analysis components that use machine learning and network analysis to identify media bias. All dictionary-based analysis components are experimentally extended with the use of general word embeddings. We also show the results of a user study.
Spinde, Timo; Hamborg, Felix; Donnay, Karsten; Becerra, Angelica; Gipp, Bela
Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias Proceedings Article
In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pp. 389–392, Association for Computing Machinery, Virtual Event, China, 2020, ISBN: 9781450375856.
Abstract | Links | BibTeX | Tags: bias visualization, news bias, news slant, perception of news
@inproceedings{10.1145/3383583.3398619,
title = {Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias},
author = {Timo Spinde and Felix Hamborg and Karsten Donnay and Angelica Becerra and Bela Gipp},
url = {https://doi.org/10.1145/3383583.3398619},
doi = {10.1145/3383583.3398619},
isbn = {9781450375856},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020},
pages = {389–392},
publisher = {Association for Computing Machinery},
address = {Virtual Event, China},
series = {JCDL '20},
abstract = {Traditional media outlets are known to report political news in a biased way, potentially affecting the political beliefs of the audience and even altering their voting behaviors. Many researchers focus on automatically detecting and identifying media bias in the news, but only very few studies exist that systematically analyze how theses biases can be best visualized and communicated. We create three manually annotated datasets and test varying visualization strategies. The results show no strong effects of becoming aware of the bias of the treatment groups compared to the control group, although a visualization of hand-annotated bias communicated bias in-stances more effectively than a framing visualization. Showing participants an overview page, which opposes different viewpoints on the same topic, does not yield differences in respondents' bias perception. Using a multilevel model, we find that perceived journalist bias is significantly related to perceived political extremeness and impartiality of the article.},
keywords = {bias visualization, news bias, news slant, perception of news},
pubstate = {published},
tppubtype = {inproceedings}
}
Traditional media outlets are known to report political news in a biased way, potentially affecting the political beliefs of the audience and even altering their voting behaviors. Many researchers focus on automatically detecting and identifying media bias in the news, but only very few studies exist that systematically analyze how theses biases can be best visualized and communicated. We create three manually annotated datasets and test varying visualization strategies. The results show no strong effects of becoming aware of the bias of the treatment groups compared to the control group, although a visualization of hand-annotated bias communicated bias in-stances more effectively than a framing visualization. Showing participants an overview page, which opposes different viewpoints on the same topic, does not yield differences in respondents' bias perception. Using a multilevel model, we find that perceived journalist bias is significantly related to perceived political extremeness and impartiality of the article.
Hamborg, Felix; Zhukova, Anastasia; Gipp, Bela
Automated Identification of Media Bias by Word Choice and Labeling in News Articles Proceedings Article
In: Proceedings of the 18th Joint Conference on Digital Libraries, pp. 196–205, IEEE Press, Champaign, Illinois, 2020, ISBN: 9781728115474.
Abstract | Links | BibTeX | Tags: automated content analysis, automated frame analysis, CAQDAS, CAS, emotions, entity perception, news bias, news slant, NLP
@inproceedings{hamborg2019a,
title = {Automated Identification of Media Bias by Word Choice and Labeling in News Articles},
author = {Felix Hamborg and Anastasia Zhukova and Bela Gipp},
url = {https://doi.org/10.1109/JCDL.2019.00036},
doi = {10.1109/JCDL.2019.00036},
isbn = {9781728115474},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings of the 18th Joint Conference on Digital Libraries},
pages = {196–205},
publisher = {IEEE Press},
address = {Champaign, Illinois},
series = {JCDL '19},
abstract = {Media bias can strongly impact the individual and public perception of news events. One difficult-to-detect, yet powerful form of slanted news coverage is bias by word choice and labeling (WCL). Bias by WCL can occur when journalists refer to the same concept, yet use different terms, which results in different sentiments being sparked in the readers, such as the terms "economic migrants" vs. "refugees." We present an automated approach to identify bias by WCL that employs models and manual analysis approaches from the social sciences, a research domain in which media bias has been studied for decades. This paper makes three contributions. First, we present NewsWCL50, the first open evaluation dataset for the identification of bias by WCL consisting of 8,656 manual annotations in 50 news articles. Second, we propose a method capable of extracting instances of bias by WCL while outperforming state-of-the-art methods, such as coreference resolution, which currently cannot resolve very broadly defined or abstract coreferences used by journalists. We evaluate our method on the NewsWCL50 dataset, achieving an F1=45.7% compared to F1=29.8% achieved by the best performing state-of-the-art technique. Lastly, we present a prototype demonstrating the effectiveness of our approach in finding frames caused by bias by WCL.},
keywords = {automated content analysis, automated frame analysis, CAQDAS, CAS, emotions, entity perception, news bias, news slant, NLP},
pubstate = {published},
tppubtype = {inproceedings}
}
Media bias can strongly impact the individual and public perception of news events. One difficult-to-detect, yet powerful form of slanted news coverage is bias by word choice and labeling (WCL). Bias by WCL can occur when journalists refer to the same concept, yet use different terms, which results in different sentiments being sparked in the readers, such as the terms "economic migrants" vs. "refugees." We present an automated approach to identify bias by WCL that employs models and manual analysis approaches from the social sciences, a research domain in which media bias has been studied for decades. This paper makes three contributions. First, we present NewsWCL50, the first open evaluation dataset for the identification of bias by WCL consisting of 8,656 manual annotations in 50 news articles. Second, we propose a method capable of extracting instances of bias by WCL while outperforming state-of-the-art methods, such as coreference resolution, which currently cannot resolve very broadly defined or abstract coreferences used by journalists. We evaluate our method on the NewsWCL50 dataset, achieving an F1=45.7% compared to F1=29.8% achieved by the best performing state-of-the-art technique. Lastly, we present a prototype demonstrating the effectiveness of our approach in finding frames caused by bias by WCL.
