LLM

Journalism

Bias

Qwen

Deepseek

BiasExpert: Teaching AI to Spot Bias More Efficiently

Emergent Methods

13 min readJul 14, 2025

In our previous exploration of bias in LLMs, we discovered something notable: when we asked leading LLMs to evaluate bias in controversial topics, they showed remarkable consistency in identifying it, even in each other’s outputs. But that raised an important question: if these models are so good at recognizing bias, could we harness that ability to create a specialized tool for detecting bias in news articles?

That’s exactly what we set out to do.

But our goal wasn’t to just detect bias. We wanted to transfer that skill, including how it detects bias, from the big LLMs to a smaller model. Why? To create an interpretable and trustworthy bias detection tool that can be used by anyone. This way, the model can help readers parse the ever-increasing flow of news content, and it can work as a key component in a future multi-agent system that could assist newsrooms in various aspects of content creation and quality assurance.

The Challenge of Defining Bias: Building Our Taxonomy

Before we could teach AI models to detect bias, we faced a fundamental challenge: bias itself is notoriously difficult to define. What one reader perceives as balanced reporting, another might see as clearly slanted. This subjectivity has long plagued both human editors and automated systems attempting to identify problematic content.

To address this challenge, we developed a comprehensive taxonomy that integrates multiple frameworks for bias classification. Our approach draws from several sources including AllSides, academic research on systematic bias detection, and frameworks for identifying bias patterns in news content. By synthesizing these different approaches, we created a foundation that captures the full spectrum of bias manifestations in news content.

The resulting taxonomy encompasses 18 distinct bias types, ranging from obvious political slant to subtle linguistic choices that can influence reader perception (see Appendix: Complete Bias Taxonomy for detailed definitions). These include demographic biases (political, gender, cultural), structural presentation issues (statement bias, opinion-as-fact), and more nuanced problems like source selection bias and mind reading. Each bias type can be classified at four intensity levels: None, Low, Moderate, or High, allowing for granular analysis of how bias manifests across different contexts.

This systematic approach was crucial for ensuring our AI models could identify bias consistently. Rather than relying on subjective human judgment alone, we created a standardized framework that multiple models could apply uniformly, leading to the consensus-based validation that makes our approach so reliable.

Key sources for our taxonomy:

The Power of Consensus: When Four AI Minds Are Better Than One

Building on our previous findings, we decided to put four reasoning-capable LLMs to work as our “bias detection committee”: Claude 3.7 Sonnet, DeepSeek R1, o3-mini, and Gemini 2.5 Pro Preview. But this time, instead of having them evaluate each other, we gave them another crucial task: analyze over 2,300 news articles to identify 18 different types of bias.

Think of it like having four expert journalists review the same article and compare their notes. Each model received identical instructions to spot everything from political bias and loaded language to more subtle issues like source selection bias and opinion presented as fact. For each bias type, they rated the intensity on a four-point scale: None, Low, Moderate, or High.

To measure how well our “expert committee” agreed with each other, we developed a scoring system that rewards consensus. When two models give identical ratings, they get perfect agreement (1.0). When they’re close but not identical, like one saying “Low” and another saying “Moderate”, they still get partial credit (0.75). When both detect bias but are further apart, like one saying “Low” and another saying “High”, they get an intermediate score (0.50). But if one model sees bias where another sees none, they get zero agreement. This approach ensures we only trust bias classifications that multiple models can confirm.

Model agreement scoring system. — Our scoring system for assessing model agreement.

The results were revealing. These models achieved strong agreement scores, as shown in our pairwise analysis:

Claude and Gemini 2.5 showed the highest consensus at 82.7%. What’s particularly noteworthy is how consistently they agreed across different types of bias. The figure below shows the comprehensive agreement analysis across all bias categories, revealing that demographic biases (political, gender, cultural) showed the strongest agreement, while more nuanced linguistic biases had more variation — exactly what you’d expect from such a complex task.

But here’s where it gets interesting: each model showed distinct “personalities” in their bias detection. Claude and Gemini were more sensitive to political and opinion-based biases, while DeepSeek and o3-mini took more conservative approaches. The next figure illustrates these pairwise agreement patterns across all bias categories, showing how well each model pair agreed on specific types of bias detection.

Choosing Our Baseline: Why Claude Led the Committee

While Gemini 2.5 actually achieved the highest overall coherence score, we selected Claude 3.7 as our baseline model for a practical reason: transparency.

Model performance in terms of overall coherence (average agreement with all other models).

Our methodology required access to the detailed reasoning process that led to each bias classification, not just the final answer. At the time of our research, only Claude 3.7 and DeepSeek-R1 provided access to their internal reasoning through their APIs, while Gemini 2.5 and o3-mini kept their thinking processes hidden. Since Claude also demonstrated strong coherence (80.7%) and showed the highest agreement with Gemini, it became the natural choice for training our model to articulate bias detection decisions transparently.

This decision proved crucial because our goal wasn’t just to detect bias, but to teach a smaller model to explain how it detected bias, making the entire process interpretable and trustworthy.

Quality Over Quantity: Building a Better Dataset

Not all AI-generated analysis is created equal, this is something we learned early on. Out of our original 2,320 articles, we used a rigorous two-step filtering process that kept only 1,220 articles, about 53% of what we started with.

Why throw away nearly half our data? Because requiring multiple models to agree on their bias detection improves dataset quality by reducing false positives and ensuring more reliable classifications. Our validation rule was straightforward but strict: whenever Claude flagged any type of bias in an article, at least one other model had to confirm that assessment. If Claude found no bias, at least one other model had to agree with that assessment too. This multi-model validation approach helps filter out individual model quirks and hallucinations, leaving us with bias classifications that represent genuine consensus rather than single-model opinions.

The next figure shows how each model’s sensitivity varied across different bias categories. This analysis revealed that while models generally agreed on the presence of bias, they had different thresholds for detection, some were more conservative, others more liberal in their assessments.

This “confirmation requirement” helped us avoid the AI equivalent of false alarms and ensured that every piece of training data represented genuine consensus among expert-level models. The filtering was demanding, but it paid off with a dataset where every bias classification had been validated by multiple models working independently.

Teaching a Smaller Model to Think Like the Experts

Here comes the compelling part: we took all this high-quality, consensus-validated data and used it to train a much smaller model, Qwen3-4B, to think like our expert committee. This approach builds on recent academic research around the “Less-Is-More” principle in AI reasoning.

Previous studies have demonstrated good results for developing model alignment or adding reasoning capabilities using small but high-quality curated datasets. While Qwen3–4B already has reasoning output capabilities, our idea was to improve this output to align more closely with larger models and achieve better agreement in final results. Our results show that this strategy worked effectively.

The next figure illustrates our complete methodology, showing how we move from the initial dataset through consensus validation to the final fine-tuned model that can perform bias detection with transparency and efficiency.

The key was in the training approach. Instead of just teaching the model to give answers, we trained it to show its work, just like Claude does. Every bias detection comes with a full explanation: what linguistic patterns it spotted, which framing choices raised red flags, and why it reached its conclusions.

Think of it as the difference between a student who memorizes answers and one who truly understands the reasoning process. Our BiasExpert model doesn’t just detect bias, it explains exactly how it arrived at its conclusions.

Results That Matter

The numbers tell a clear story. Here’s how our BiasExpert model performed compared to baseline models:

Our BiasExpert model achieved an agreement score of 84.6% with Claude, significantly outperforming both the vanilla Qwen3/4B model (75.1%) and even the much larger Qwen3/32B model (80.0%) that has 8x more parameters. This demonstrates that targeted fine-tuning with high-quality consensus data can enable smaller models to surpass both their un-tuned counterparts and significantly larger general-purpose models. But beyond the numbers, what really matters is reliability and transparency.

The model generates detailed explanations averaging about 8,000 words, approaching Claude’s reasoning depth (9,344 words) while being significantly more efficient than the original model. This shows that our fine-tuned model successfully learned to replicate the comprehensive reasoning process that makes Claude’s bias analysis so effective.

Model Output Example

The BiasExpert model uses a standard prompt template (complete example available on Hugging Face) combined with the article as input. The model processes the information and returns:

Reasoning Process: Detailed analysis enclosed within <think></think> tags, showing how the model evaluates the content.
Structured Output: A final JSON with a summary of the analysis for the full article and a detailed analysis for each type of bias, including the level result (None, Low, Moderate, High), the reasoning for this result, and also suggestions when applicable.

Below is an example with parts of the model’s output, which includes the analysis for all 18 types of bias:

{
    "bias_summary": "This article exhibits moderate statement bias, slant, and negativity bias. The reporting frames economic uncertainty in a somewhat negative light, using loaded language like 'paralyzed' for oil markets and 'browbeating' for Trump's actions. The article emphasizes potential risks and uncertainties while giving less attention to positive economic developments, creating an overall impression of economic instability. It presents some analysts' interpretations as facts rather than opinions, and quotes primarily economists with similar perspectives. While the article provides relevant economic data, its framing, word choices, and selective emphasis guide readers toward viewing the economic situation more negatively than a balanced presentation would.",
    "bias_analysis": {
        "source_selection": {
            "level": "Low",
            "reasoning": "While the article quotes multiple economists and analysts, they all share similar perspectives on economic risks. There's no inclusion of economists or analysts who might offer more optimistic or different interpretations of the economic data, which creates a slight source selection bias.",
            "suggestions": [
                {
                    "description": "Include a wider range of expert perspectives",
                    "reasoning": "Adding voices with different interpretations of the economic data would provide readers with a more comprehensive understanding of the situation."
                }
            ]
        },
        "omission_of_source_attribution": {
            "level": "Low",
            "reasoning": "The article mostly attributes statements to specific economists and analysts, but there are a few instances where attribution is vague or unclear. For example, the statement 'some economists are breathing a sigh of relief' doesn't specify which economists or how many.",
            "suggestions": [
                {
                    "description": "Specify which economists are expressing relief and how many",
                    "reasoning": "Adding specific attribution would increase transparency and credibility of the reporting."
                }
            ]
        },
        "spin": {
            "level": "Moderate",
            "reasoning": "The article uses rhetoric techniques to create a narrative of economic uncertainty and risk. The framing of events like the rate cut and trade negotiations suggests that the economy is in a precarious state, with potential for further downturns. Language choices like 'paralyzed,' 'mixed bag,' and references to 'plenty of pitfalls ahead' create a particular impression that goes beyond the basic facts.",
            "suggestions": [
                {
                    "description": "Use more neutral framing and language",
                    "reasoning": "A more neutral presentation would allow readers to form their own judgments about the economic situation based on the facts presented."
                }
            ]
        }
   }
}

What This Means for Readers and Newsrooms

This research addresses real needs in today’s media landscape. For news consumers, the benefits are clear: in an era where we’re increasingly consuming news through AI-powered summaries and algorithmic feeds, having tools that can surface subtle bias becomes crucial. It’s not about determining what’s “true” or “false”, it’s about understanding how information is being shaped and presented.

For journalists and newsrooms, the applications are equally valuable. The model provides a systematic way to examine reporting practices, identify potential blind spots, and ensure more balanced coverage. It’s like having an experienced editor who can spot subtle language choices or framing decisions that might inadvertently introduce bias. This isn’t about censoring or constraining journalism, it’s about empowering reporters with tools to strengthen their craft.

Our model doesn’t just flag bias; it explains what specific language choices, source selections, or framing techniques might be influencing how a story is perceived. This transparency helps readers develop their own critical evaluation skills while providing newsrooms with actionable insights to improve their reporting standards.

The Path Forward: Structured Analysis for AI-Powered Journalism

What stands out about this research is that it demonstrates how sophisticated AI capabilities can be made more accessible and transparent. Instead of requiring massive computational resources, our approach shows that well-designed smaller models can deliver expert-level analysis while remaining interpretable and efficient.

The BiasExpert model is designed to work with structured prompts and produces standardized JSON outputs, making it perfect for integration into larger automated systems. This structured approach isn’t accidental; it’s part of our broader vision for AI-powered journalism tools. We’re developing this model as a key component in a future multi-agent system that could assist newsrooms in various aspects of content creation and quality assurance.

Imagine a newsroom workflow where AI agents can help with fact-checking, bias detection, source verification, and style consistency, all working together to support journalists rather than replace them. The BiasExpert model’s structured outputs make it an ideal building block for such collaborative AI systems, where each agent has a specialized role but can communicate seamlessly with others.

The model represents a step toward more democratic access to bias detection tools, something that could benefit individual readers, newsrooms, and researchers alike. By making the reasoning process transparent and the outputs machine-readable, we’re not just detecting bias; we’re creating infrastructure for the next generation of journalism tools.

Ready to try it yourself? The BiasExpert model is available on Hugging Face, making it accessible to researchers, developers, and anyone interested in exploring bias detection in news content. Whether you’re a journalist looking to examine your own work, a researcher studying media bias, or a developer building journalism tools, the structured approach makes integration straightforward.

Research paper: Access our detailed paper on the research behind the creation of this model.

As we continue to navigate an increasingly complex information landscape, having transparent, reliable tools for bias detection isn’t just helpful, it’s essential for maintaining an informed democracy. And with structured, interoperable AI tools, we can build systems that support rather than supplant human expertise in journalism.

Want to stay updated on our latest research in AI and media analysis? Sign up for a free AskNews account to get our newsletters, join our Discord, or follow us on LinkedIn!

Appendix: Complete Bias Taxonomy

Our BiasExpert model analyzes 18 distinct types of bias. Below are simplified definitions with basic examples for each type. This comprehensive taxonomy ensures systematic and consistent bias detection across all news content.

Political: Favors or criticizes a specific political viewpoint. Example: “The radical left continues to sabotage the economy.”
Gender: Reinforces stereotypes or prejudices based on gender. Example: “The female engineer surprisingly solved the problem.”
Cultural/Ethnicity: Unfairly portrays or generalizes ethnic or cultural groups. Example: “Immigrants are taking away local jobs.”
Age: Unfairly stereotypes or discriminates based on age. Example: “Older employees rarely adapt to new technology.”
Religion: Unfairly stereotypes or discriminates based on religion. Example: “Muslim neighborhoods are often hotspots of radicalism.”
Disability: Portrays individuals with disabilities in negative, stereotypical, or dehumanizing ways. Example: “This facility is for retarded individuals.”
Statement Bias: Uses loaded language or presents one side as the only perspective. Example: “Gender-affirming care” vs. “sex reassignment procedure.”
Unsubstantiated Claims: Makes claims without supporting evidence or uses flawed reasoning. Example: “The senator’s absence clearly shows he doesn’t care.”
Slant (Omission): Highlights one angle while ignoring other perspectives through selective information.
Source Selection: Chooses sources that support the story rather than providing accurate accounts.
Missing Attribution: Doesn’t back up claims with specific sources. Example: “According to a source” or “critics say.”
Spin: Creates “memorable stories” using loaded language or selective facts for dramatic effect.
Sensationalism: Exaggerates information to create emotional reactions. Example: “Bloodbath at the debate stage!”
Negativity Bias: Emphasizes negative news or frames events negatively. Example: “The country is collapsing under failed leadership.”
Subjective Adjectives: Uses qualifying adjectives that suggest how readers should interpret issues. Example: “The disturbing trend continues.”
Ad Hominem: Makes unfair accusations or attacks character instead of addressing arguments. Example: “He’s a clown with no credibility.”
Mind Reading: Assumes knowledge of what others think without explicit evidence. Example: “She clearly intended to undermine the election.”
Opinion-as-Fact: Presents subjective statements as factual information in objective reporting. Example: “This policy proves the government doesn’t care.”