《OpenAI+o1大模型》英文技术报告

OpenAI o1 System CardOpenAISept 12, 20241IntroductionThe o1 model series is trained with large-scale reinforcement learning to reason using chain ofthought. These advanced reasoning capabilities provide new avenues for improving the safety androbustness of our models. In particular, our models can reason about our safety policies in contextwhen responding to potentially unsafe prompts. This leads to state-of-the-art performance oncertain benchmarks for risks such as generating illicit advice, choosing stereotyped responses,and succumbing to known jailbreaks. Training models to incorporate a chain of thought beforeanswering has the potential to unlock substantial benefits, while also increasing potential risks thatstem from heightened intelligence. Our results underscore the need for building robust alignmentmethods, extensively stress-testing their efficacy, and maintaining meticulous risk managementprotocols. This report outlines the safety work carried out for the OpenAI o1-preview and OpenAIo1-mini models, including safety evaluations, external red teaming, and Preparedness Frameworkevaluations.2Model data and trainingThe o1 large language model family is trained with reinforcement learning to perform complexreasoning. o1 thinks before it answers—it can produce a long chain of thought before respondingto the user. OpenAI o1-preview is the early version of this model, while OpenAI o1-mini isa faster version of this model that is particularly effective at coding. Through training, themodels learn to refine their thinking process, try different strategies, and recognize their mistakes.Reasoning allows o1 models to follow specific guidelines and model policies we’ve set, ensuringthey act in line with our safety expectations. This means they are better at providing helpfulanswers and resisting attempts to bypass safety rules, to avoid producing unsafe or inappropriatecontent. o1-preview is state-of-the-art (SOTA) on various evaluations spanning coding, math,and known jailbreaks benchmarks [1, 2, 3, 4].The two models were pre-trained on diverse datasets, including a mix of publicly available data,proprietary data accessed through partnerships, and custom datasets developed in-house, whichcollectively contribute to the models’ robust reasoning and conversational capabilities.Select Public Data: Both models were trained on a variety of publicly available datasets,including web data and open-source datasets. Key components include reasoning data andscientific literature. This ensures that the models are well-versed in both general knowledgeand technical topics, enhancing their ability to perform complex reasoning tasks.1Proprietary Data from Data Partnerships: To further enhance the capabilities of o1-previewand o1-mini, we formed partnerships to access high-value non-public datasets. These propri-etary data sources include paywalled content, specialized archives, and other domain-specificdatasets that provide deeper insights into industry-specific kno