WinoIdentity: An intersectional fairness benchmark for LLMs

This software project accompanies the research paper, Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution.

Overview

This work focuses on intersectional bias in large language models (LLMs), particularly when they're used in applications like hiring. While LLMs are powerful, it's widely accepted that AI can worsen societal biases. In this work, we extend single-axis fairness evaluations to examine intersectional bias, recognizing that when multiple axes of discrimination intersect, they create distinct patterns of disadvantage. We create a new benchmark called WinoIdentity by augmenting the WinoBias dataset [1] with 25 demographic markers across 10 attributes, including age, nationality, and race, intersected with binary gender, yielding 247,104 prompts to evaluate 50 distinct bias patterns.

Dataset Construction

Intersectional Demographic Attributes: We identify 10 attributes that affect social dynamics, drawing from "The Wheel of Power and Privilege." From these, 25 demographic groups commonly found in the US are selected. These groups are categorized as either hegemonic/privileged (historically holding power) or disadvantaged (historically marginalized), as shown in Table 1.

Augmenting WinoBias: We design three augmentations:

• Referent Augmentation (R-Aug): A demographic marker (e.g., "young") is added to the person or occupation being referred to. This checks if knowing more about the subject changes how the model assigns a pronoun. Using example in Figure 1, it investigates "how likely is [pronoun] the nurse, given that the nurse is young?”
• Non-Referent Augmentation (NR-Aug): A demographic marker is added to a different person or occupation in the sentence. This examines if information about someone else in the context affects the model's pronoun assignment for the main subject. Using example in Figure 1, it assesses "how likely is [pronoun] the nurse, given that the physician is young?”
• Contrastive Augmentation (C-Aug): Markers are added to both the referent and non-referent occupations, often using contrasting demographic markers (e.g., "young" for one and "old" for the other). This helps evaluate bias when comparing two different demographic attributes. Using example in Figure 1, it explores "how likely is [pronoun] the nurse, given that the nurse is young and the physician is old?”

Prompt construction: For each original WinoBias sentence, a specific augmentation type and a chosen demographic attribute (like age or race) are used to generate a set of new prompts, as illustrated in Figure 1. This process combines demographic markers with feminine and masculine pronouns to evaluate intersectional biases (biases that arise from the overlap of multiple demographic identities). In total, the WinoIdentity dataset comprises 247,104 unique sentences.

Data Format

Data Folder Structure

WinoIdentity dataset is stored as 3 .csv files, one for each augmentation type:

• intersectional_referent_occupation.csv -- Contains 79,200 WinoIdentity sentences that contain intersectionally augmented referent occupations. In other words, for every original sentence in WinoBias, there are 25 intersectionally augmented WinoIdentity sentences in this split.
• intersectional_nonreferent_occupation.csv -- Contains 79,200 WinoIdentity sentences that contain intersectionally augmented non-referent occupations. In other words, for every original sentence in WinoBias, there are 25 intersectionally augmented WinoIdentity sentences in this split.
• intersectional_both_occupations.csv -- Contains 88,704 WinoIdentity sentences that contain intersectionally augmented non-referent occupations. In other words, for every sentence in WinoBias, there are 28 intersectionally augmented WinoIdentity sentences in this split.

File format

Each file contains identical columns:

• Column 1, Prompt: contains intersectionally augmented WinoIdentity sentence.
• Column 2, pronoun: pronoun present in column 1 sentence; there is only 1 pronoun per sentence.
• Column 3, referent_occ: The occupation to which the pronoun in column 2 refers to. These referent values are taken directly from the WinoBias dataset. This column contains a single referent value, even if the sentence is known to be ambiguous, identical to WinoBias.
• Column 4, occs_in_samples: List of two occupations present in each sentence, of which one occupation will be female dominant and the other male dominant according to Bureau of Labor Stats.
• Column 5, stereotype_label: anti-stereotypical or pro-stereotypical label for each sentence, adopted from WinoBias. There should be equal number of anti- and pro- stereotypical sentences in each file.
• Column 6, winobias_task_type: type1 (ambiguous) or type2 (non-ambiguous) label for each sentence, adopted from WinoBias. There should be equal number of type 1 and 2 sentences in each file.
• Column 7, augmentation_to: intersectional_referent_occ.csv identifies intersectionally augmented referent sentences using the label referent, intersectional_nonreferent_occ.csv identifies intersectionally augmented non-referent sentences using the label non-referent, and intersectional_both_occs.csv identifies that both occupations are intersectionally augmented sing the label both.
• Column 8, demographic_identifier: The demographic identifier(s) in each sentence using which the sentence has been intersectionally augmented.

References

[1] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K. W. (2018). Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.