2025年生成式AI红队百次测试经验白皮书-微软（英）

Lessons from red teaming 100 generative AI products Authored by: Microsoft AI Red TeamAuthorsBlake Bullwinkel, Amanda Minnich, Shiven Chawla, Gary Lopez, Martin Pouliot, Whitney Maxwell, Joris de Gruyter, Katherine Pratt, Saphir Qi, Nina Chikanov, Roman Lutz, Raja Sekhar Rao Dheekonda, Bolor-Erdene Jagdagdorj, Eugenia Kim, Justin Song, Keegan Hines, Daniel Jones, Giorgio Severi, Richard Lundeen, Sam Vaughan, Victoria Westerhoff, Pete Bryan, Ram Shankar Siva Kumar, Yonatan Zunger, Chang Kawaguchi, Mark Russinovich2Lessons from red teaming 100 generative AI productsTable of contents304Abstract07Red teaming operations09Case study #1 Jailbreaking a vision language model to generate hazardous content12Lesson 4 Automation can help cover more of the risk landscape05Introduction08Lesson 1 Understand what the system can do and where it is applied10Lesson 3 AI red teaming is not safety benchmarking12Lesson 5 The human element of AI red teaming is crucial05AI threat model ontology 08Lesson 2 You don’t have to compute gradients to break an AI system 11Case study #2 Assessing how an LLM could be used to automate scams13Case study #3 Evaluating how a chatbot responds to a user in distressLessons from red teaming 100 generative AI products14Case study #4 Probing a text-to-image generator for gender bias14Lesson 6 Responsible AI harms are pervasive but difficult to measure15Lesson 7 LLMs amplify existing security risks and introduce new ones 16Case study #5 SSRF in a video-processing GenAI application17Lesson 8 The work of securing AI systems will never be complete18ConclusionAbstractIn recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned:1. Understand what the system can do and where it is applied 2. You don’t have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking4. Automation can help cover more of the risk landscape5. The human element of AI red teaming is crucial6. Responsible AI harms are pervasive but difficult to measure 7. Large language models (LLMs) amplify existing security risks and introduce new ones8. The work of securing AI systems will never be completeBy sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.4Lessons from red teaming 100 generative AI products5IntroductionAs generative AI (GenAI) systems are adopted across an increasing number of domains, AI red teaming has emerged as a central practice for assessi