8 Times AI Bias Caused Real-World Harm

Artificial intelligence?(AI) bias is where AI systems inadvertently reflect prejudices from their training data or design.

Why should you care? Because AI is already creeping into our lives, from healthcare to financial services, and the world is still grappling with how we regulate its impact.

Biased data can have disastrous consequences, and here are just a few real-life examples of why tackling machine bias is already headline news.

Table of Contents Table of Contents

1. Key Takeaways
2. 8 Times AI Bias Caused Real-World Harm
3. The Bottom Line

Show Full Guide

Table of Contents

1. Key Takeaways
2. 8 Times AI Bias Caused Real-World Harm
3. The Bottom Line

8 Times AI Bias Caused Real-World Harm

8. Microsoft Chatbot Becomes Racist and Sexist on Twitter

Thanks to generative AI, Microsoft has reclaimed its position as the most valuable public company in the world. But the tech giant’s relationship with AI has been far from a smooth journey. In 2016, Microsoft ventured into AI-driven social media with “Tay,” an AI-driven chatbot that would learn through the art of conversation with humans. What could possibly go wrong?

Tay was designed to mimic and engage with millennials and evolve through user interaction. But it took Twitter users less than 24 hours to corrupt the innocent bot. Tay quickly devolved into a channel for offensive content, echoing racist and lewd remarks fed to it by users. This incident exposed the complexities and risks of programming AI to interact in unrestricted human environments.

Tay’s rapid shift from a friendly digital persona to a source of controversial racist and sexist statements led to its shutdown, raising questions about Microsoft’s preparedness for handling the unpredictable nature of AI-enhanced social media.

Eight years on, the big question remains: How do we ensure that AI doesn’t inherit humanity’s worst traits, especially when tech giants like Microsoft could overlook crucial preventative measures?

7. Falsely Arrested by AI: Man Wrongfully Detained Due to Biased Algorithm

Robert Williams was working at a Detroit auto shop in January 2020 when he received a call from police ordering his arrest. Shortly after, he would learn that AI had falsely identified him in a robbery case, highlighting the dangers of biased algorithms in law enforcement. The old mantra of if you’ve got nothing to hide, you’ve got nothing to fear is far from the truth when you find yourself accused of committing a crime by an AI algorithm.

The wrongful arrest of Robert Williams due to a ‘racist algorithm’ in facial recognition software raised severe concerns about the hidden dangers of AI in law enforcement when biases can have profound real-life consequences. It also highlights the broader challenge of ensuring that AI systems do not simply mirror societal prejudices but are developed and used to promote equity and justice.

6. The Dark Side of Predictive Policing and Pre-Crime Algorithms

The Pasco County Sheriff’s Office in Florida hit the headlines for all the wrong reasons after adopting an AI-driven method to law enforcement, eerily echoing the pre-crime concept made famous in Philip K. Dick’s “Minority Report.”

This predictive policing strategy utilizes AI algorithms to analyze big data, including arrest histories, and identify individuals most likely to commit future crimes.

Critics argued that it led to unnecessary harassment and amplified existing biases, particularly against minorities, potentially replicating and intensifying systemic racism within law enforcement. The method was criticized for creating feedback loops, where specific neighborhoods were targeted repeatedly, exacerbating the problem rather than solving it.

Although the controversial intelligence program run by pre-algorithms was eventually discontinued, it remains a timely reminder of why we must avoid making similar mistakes in the future.

5. iTutorGroup Forced to Pay $365K for Age Discrimination

Imagine applying for your dream job — only to be automatically rejected by a computer program because of your age.

That’s the harsh reality some job seekers faced at iTutorGroup, thanks to what looked to be a biased AI hiring tool. This tool, meant to speed up hiring, allegedly ended up discriminating against older applicants, tossing aside resumes of women over 55 and men over 60. Thankfully, a tech-savvy and determined applicant exposed this ageism by cleverly reapplying with a younger fake age and getting accepted.

This shocking incident resulted in a hefty $365,000 settlement with the Equal Employment Opportunity Commission (EEOC), a stark reminder that even fancy AI tools can be flawed and perpetuate biases. That’s why the EEOC is now focusing on “Algorithmic Fairness,” ensuring that AI used in hiring and other vital areas doesn’t discriminate.

So, what can we learn from this? For employers, carefully examining your AI tools for hidden biases is crucial, as we discussed in our interview with IBM Fellow?Francesca Rossi last week.

Remember, AI is a tool, not a magic wand that replaces human judgment and empathy. Age shouldn’t be a factor unless necessary and legal, so avoid collecting age data during recruitment. Finally, transparency is critical. Be open about how you use AI in hiring and have clear procedures for applicants to raise concerns.

4. Real Estate Giant Loses Millions on Homes It Can’t Sell

Remember when Zillow promised to revolutionize home buying with AI-powered “Zestimates” and instant cash offers? After a spectacular tumble, Zillow revealed it was shutting down its “Zillow Offers”, leaving a $304 million pile of unsold houses and 2,000 jobs.

What went wrong? While the Zestimate could analyze massive datasets and offer seemingly intelligent valuations, it couldn’t predict the wild swings in housing prices fueled by the pandemic and changing buyer preferences.

But Zillow’s story isn’t just about bad bets. It’s a cautionary tale about the limitations of AI. While algorithms can process information faster than humans, they lack the human touch.

Zillow’s experience doesn’t mean AI is useless in real estate. However, it emphasizes that AI needs careful human guidance and shouldn’t replace crucial human judgment and empathy.

The real estate market, at its core, is still about people, emotions, and communities, and that’s something AI might struggle to replicate anytime soon.

3. Why You Can’t Trust Your AI Lawyer

In a striking example of the pitfalls of AI in legal practice, a New York lawyer, Steven A. Schwartz, faced a courtroom debacle after relying on ChatGPT for legal research. The AI program fabricated judicial decisions and citations, which Schwartz included in a court filing for a case against Avianca Airlines. This incident led to a hearing for potential sanctions and highlights the dangers of uncritically accepting AI-generated content in professional settings.

The incident raised significant ethical concerns about using AI in legal work, especially when verifying the accuracy and authenticity of the information.

2. The AI Against Women in Tech Hiring

Amazon’s attempt to revolutionize hiring with AI backfired spectacularly. Their algorithm, trained on a mountain of primarily male resumes, was accused of favoring men. Despite efforts to remove the AI bias, the Guardian alleged the tool discriminated against resumes from women. This unfortunate case, which led to the tool’s shutdown in 2019, highlights the difficulty of creating fair and neutral AI, especially when tackling complex human characteristics like gender.

It serves as another stark warning for other companies exploring AI in hiring: constant vigilance and ongoing improvement are crucial to prevent AI from amplifying societal biases.

1. AI Fabricates Sexual Harassment Claim Against Law Professor

ChatGPT chillingly fabricated a sexual harassment scandal by falsely accusing a law professor. The professor initially received an email claiming ChatGPT listed him as a legal scholar accused of sexual harassment, claiming he made inappropriate comments and touched a student during a class trip to Alaska.

However, upon investigation, it was revealed that no such article existed in the Washington Post (cited as the source). The professor had never taken a class trip to Alaska or been accused of harassing a student.

The ease with which AI could generate convincing lies is enough to send shockwaves across any community. But it did raise awareness of the limitations of AI technology and why we need to be very careful in believing the more terrifying hallucinations that could ruin an individual’s life.

The Bottom Line

As we’ve seen, AI bias can have real-world consequences, impacting everything from employment to legal proceedings. While the technology shows immense potential, its capacity for harm underscores the crucial question: how can we ensure the responsible development and implementation of AI to safeguard individuals and society?

Regular analysis, error monitoring, expert consultations, and tools like Google’s What-if Tool or IBM’s AI Fairness 360 are all crucial in detecting and correcting AI bias. Are we prepared to meet this challenge and build a future where AI serves, rather than endangers, humanity? Only time will tell.