Our code, data, and methodology can be found in our Github repo.

Adversarial Datasets

We've compiled some of the most commonly used red-teaming datasets and evaluated their attack success rate against various open-sourced models:

• Advbench: Dataset of adversarial behaviors ranging from profanity, discrimination, to violance, formulated as instructions from Universal and Transferable Adversarial Attacks on Aligned Language Models. [Github]
• AART: A collection of generated adverasrial behaviors created through AI-assisted recipes with a wide range of diverse cultural, geographic, and application settings, taken from AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications. [Github]
• Beavertails: An extensive collection of datasets specifically developed to support research on safety alignment in large language models, taken from A Human-Preference Dataset for Improving Safety Alignment of large Language Models. [Github]
• Do Not Answer (DNA): An open-source dataset to evaluate LLMs' safety mechanism at a low cost. The dataset is curated and filtered to consist only of prompts to which responsible language models do not answer, taken from Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. [Github]
• RedEval - HarmfulQA: Dataset of harmful questions covering 10 topics and ~10 subtopics each, ranging cultural studies to ancient history. Subset of RedEval from Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. [Github]
• RedEval - DangerousQA: Dataset of harmful questions covering racist, stereotypical, sexist, illegal, toxic, and harmful content. Subset of RedEval from Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. [Github]
• Student-Teacher Prompting (STP): Dataset of harmful prompts and formats that successully broke Vicuna-13B. Formats are specified in the paper Student-Teacher Prompting for Red Teaming to Improve Guardrails. [Github]
• SAP: Generated attacks created through in-context learning to mimic human speech. Taken from Attack Prompt Generation for Red Teaming and Defending Large Language Models. [Github]

Adversarial Content

Using OpenAI's 14 Risk Policies, we've classified each prompt from our collected datasets into their respective content violation policy and measured the resulting model performance.