African American English (AAE) influences LLMs towards discrimination

Bias has always been a problem in AI, but a new study shows that it’s covertly integrated into language models with potentially catastrophic consequences.

In what has already been heralded as a landmark study, a team of researchers, including Valentin Hofman, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King, documented how large language models (LLMs) discriminate against African American English (AAE).

In short, the study tests how different spelling and dialects affect LLMs’ behavior. It probes whether certain dialects and word usage influence an LLM’s behavior, focusing on bias and discrimination. 

We know that LLM outputs are highly sensitive to the input. Even small deviations in spelling and style can influence outputs. But does this mean certain inputs – e.g., those typed in AAE – produce biased outputs? If so, what are the possible consequences? 

To answer these questions, the researchers analyzed the prejudices held by a total of 12 LLMs against AAE, revealing biases that match or exceed those typically held by humans. The study is available on ArXiv.

The researchers then applied their findings to societal domains such as employment and criminal justice, where AI decision-making is becoming more common. 

Hofmann described the study methodology on X: “We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English (SAE) texts in prompts that ask for properties of the speakers who have uttered the texts, and compare the model predictions for the two types of input.” 

This method allows the team to directly compare the responses of LLMs to AAE versus SAE inputs, unmasking the covert biases that would otherwise remain obscured.

The study’s findings are unsettling. Hofmann notes, “We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement.” 

This suggests that the biases present in LLMs are not merely reflections of contemporary stereotypes but are more aligned with prejudices that many believed society had moved beyond.

One of the most concerning aspects of the study is the specific linguistic triggers of bias. 

Hofmann elaborates, “What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, such as the use of ‘finna’ as a future marker.”

This indicates that the prejudice is not just against the use of AAE in general but is tied to the distinct linguistic elements that characterize the dialect.

The potential for harm

The potential for harm from such biases is immense. Previous studies have already demonstrated how AI systems tend to fail women, darker-skinned individuals, and other marginalized groups. 

Before the last few years, many AI systems were trained almost exclusively on unrepresentative datasets that contained predominantly white males. Some, like MIT’s Tiny Images, made in 2008, were withdrawn due to severe issues. 

One influential 2018 study, Gender Shades, analyzed hundreds of ML algorithms and found that error rates for darker-skinned women were up to 34% greater than for lighter-skinned males, whereas others revealed stark risks, including high rates of skin cancer misdiagnosis. 

Building on this research, Hofman’s team delved into hypothetical scenarios where LLMs make decisions about individuals based on their speech patterns. 

He shares, “Focusing on the areas of employment and criminality, we find that the potential for harm is massive.” 

Specifically, LLMs were found to assign less prestigious jobs and suggest harsher criminal judgments against speakers of AAE. These biases demonstrate the real-world consequences of AI prejudice.

Hofmann warns, “Our results point to two risks: that users mistake decreasing levels of overt prejudice for a sign that racism in LLMs has been solved when LLMs are in fact reaching increasing levels of covert prejudice.” 

What’s more is that erasing these problems is technically challenging. The study says, “We show that existing methods for alleviating racial bias in language models such as human feedback training do not mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level.”

It’s feasible to think these biases apply to other dialects or possibly even entire languages, too. More research is needed to clarify LLM performance across different linguistic inputs, cultural differences in how people use the models, and so on. It’s a potentially limitless arena of research.

The study concludes with a call to action for the AI research community and society at large. 

Addressing these biases becomes paramount as AI systems become increasingly embedded in various aspects of life, from hiring practices to legal judgments.

It’s an extremely challenging problem. 

The post African American English (AAE) influences LLMs towards discrimination appeared first on DailyAI.

Unlock the power of our talent network. Partner with QAT Global for your staffing needs and experience the difference of having a dedicated team of experts supporting your enterprise’s growth.

Explore Articles from QAT Global