DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Abstract

Key Aspects

• Introduction of DeepSeek-R1-Zero and DeepSeek-R1: The abstract introduces two new reasoning models: DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero is trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT), while DeepSeek-R1 incorporates multi-stage training and cold-start data before RL to address the limitations of DeepSeek-R1-Zero, such as poor readability and language mixing.
• Reinforcement Learning Approach: The core methodological approach is based on reinforcement learning (RL). DeepSeek-R1-Zero is trained purely through RL, demonstrating that significant reasoning capabilities can emerge without initial supervised fine-tuning. This is a novel contribution to the field, suggesting new avenues for developing reasoning in large language models (LLMs).
• Emergent Reasoning Behaviors: Through RL, DeepSeek-R1-Zero naturally develops powerful and intriguing reasoning behaviors. The paper highlights that the model exhibits behaviors such as self-verification and reflection, indicating a level of autonomous reasoning development not explicitly programmed but emerging from the RL process.
• Addressing Limitations with DeepSeek-R1: To address the issues of poor readability and language mixing encountered in DeepSeek-R1-Zero, DeepSeek-R1 is introduced. This model uses multi-stage training and incorporates cold-start data before the RL phase, leading to improved performance and more coherent outputs.
• Performance and Benchmarking: DeepSeek-R1 achieves performance comparable to OpenAI-01-1217 on reasoning tasks. The abstract mentions specific benchmarks where the model performs well, indicating its strong competitive standing in the field of reasoning models.
• Open-Source Contribution: To support the research community, the authors open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1. This is a significant contribution, providing valuable resources for further research and development in the field.
• Distillation to Smaller Models: The abstract mentions that the six dense models are distilled from DeepSeek-R1 based on Qwen and Llama. This indicates an exploration of knowledge distillation techniques to transfer the reasoning capabilities of the larger model to smaller, more efficient models.

Strengths

• Clear Introduction of Models

  The abstract clearly introduces the two main models, DeepSeek-R1-Zero and DeepSeek-R1, and differentiates between their training approaches.

  "We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1." (Page 1)
• Highlighting Novelty

  The abstract effectively highlights the novelty of using pure reinforcement learning without supervised fine-tuning for developing reasoning capabilities.

  "DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without super-vised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities." (Page 1)
• Concise Performance Summary

  The abstract provides a concise summary of the performance of DeepSeek-R1, mentioning its comparability to OpenAI-01-1217.

  "DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks." (Page 1)
• Open-Source Commitment

  The abstract clearly states the intention to open-source the models, which is a significant strength for fostering research collaboration.

  "To support the research community, we open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models" (Page 1)

Suggestions for Improvement

• Clarify Performance Metrics

  This medium-impact improvement would enhance the reader's understanding of the model's performance by providing specific quantitative results. The Abstract section particularly needs this detail as it forms the first impression of the model's capabilities.

  "DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks." (Page 1)

  Implementation: Include specific performance metrics such as accuracy percentages or benchmark scores. For example, "DeepSeek-R1 achieves 79.8% accuracy on the AIME 2024 benchmark, comparable to OpenAI-01-1217."

• Expand on Distillation Results

  This medium-impact improvement would provide a clearer picture of the effectiveness of the distillation process. The Abstract section particularly needs this detail as it highlights a key contribution of the paper.

  "six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama." (Page 1)

  Implementation: Briefly mention the performance of the distilled models. For example, "The distilled models, including a 7B variant, demonstrate strong performance, with the 7B model achieving X% accuracy on benchmark Y."

• Elaborate on Training Data

  This low-impact improvement would provide more context on the nature of the training data used for DeepSeek-R1. The Abstract section particularly needs this detail as it sets the stage for understanding the model's training process.

  "we introduce DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL." (Page 1)

  Implementation: Briefly describe the type and source of the cold-start data. For example, "DeepSeek-R1 incorporates cold-start data from diverse reasoning tasks, including mathematical and logical problems, before undergoing RL."

Non-Text Elements

Figure 1 | Benchmark performance of DeepSeek-R1.

Figure/Table Image (Page 1)

First Reference in Text

We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without super-vised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities. Through RL, DeepSeek-R1-Zero naturally emerges with numerous powerful and intriguing reasoning behaviors. However, it encounters challenges such as poor readability, and language mixing. To address these issues and further enhance reasoning performance, we introduce DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL. DeepSeek-R1 achieves performance comparable to OpenAI-01-1217 on reasoning tasks. To support the research community, we open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama. Figure 1 | Benchmark performance of DeepSeek-R1.

Description

• Overview of the graph: The figure is a bar graph. It compares the performance of a new artificial intelligence model called "DeepSeek-R1" against several other existing AI models. Performance is measured using accuracy, which is a way to quantify how often the AI gets the right answer, or a percentile, which ranks the AI's performance relative to others. Think of percentile like a ranking in a race - a higher percentile means the AI performed better compared to others.
• Models compared: The graph specifically looks at six different AI models: DeepSeek-R1, OpenAI-01-1217, DeepSeek-R1-32B, OpenAI-01-mini, DeepSeek-V3, and an unnamed model with poor performance. OpenAI models are developed by the company OpenAI. The "-01-1217" and "-01-mini" are likely version numbers or identifiers for different versions of their models. DeepSeek-R1 is the main model being introduced, and DeepSeek-R1-32B and DeepSeek-V3 are likely variations or related models from the same research team. The "32B" probably refers to the model's size, with "B" standing for billions of parameters, which are a measure of the model's complexity and capacity to learn. More parameters often mean a more powerful model.
• Benchmarks used: The AI models are tested on five different benchmarks, which are standardized tests designed to evaluate different aspects of an AI's capabilities. The benchmarks used here are: AIME 2024 (Pass@1), Codeforces (Percentile), GPQA Diamond (Pass@1), MATH-500 (Pass@1), MMLU (Pass@1), and SWE-bench Verified (Resolved). "Pass@1" is a specific metric that likely means the percentage of times the AI gets the correct answer on its first try. Each benchmark likely tests a different skill, such as mathematical reasoning (AIME 2024, MATH-500), coding ability (Codeforces), or general knowledge (MMLU).
• Visual representation: Each model's performance on each benchmark is represented by a colored bar. The height of the bar corresponds to the model's score (accuracy or percentile). Taller bars mean better performance. Different colors are used for each model to make it easy to compare them across the different benchmarks.

Scientific Validity

• Relevance to Abstract: The figure directly supports the claim made in the abstract that DeepSeek-R1 achieves performance comparable to OpenAI-01-1217 on reasoning tasks. It provides a visual comparison of DeepSeek-R1's performance against this and other models across various benchmarks.
• Benchmark Selection: The selection of benchmarks appears appropriate for evaluating reasoning capabilities, covering areas like mathematics, coding, and general knowledge. However, the rationale for choosing these specific benchmarks over others is not explicitly stated in the abstract or caption.
• Metric Clarity: While the caption mentions "Pass@1" and "Percentile," a more detailed explanation of these metrics within the figure or caption would enhance scientific rigor. It is assumed that the reader is familiar with these metrics in the context of AI evaluation.
• Model Identification: The models are clearly identified, but more context on the less prominent models (DeepSeek-R1-32B, DeepSeek-V3) would be beneficial. Their relationship to DeepSeek-R1 and their significance in the comparison could be clarified.

Communication

• Clarity of Presentation: The bar graph format is effective for comparing performance across multiple models and benchmarks. The use of color-coding is helpful for distinguishing between models.
• Caption Conciseness: The caption is concise but could be more informative. It states what the figure shows but doesn't provide much context or interpretation.
• Accessibility to Non-Experts: While the figure is visually clear, a non-expert reader might struggle to understand the significance of the benchmarks and metrics without further explanation. The caption could be expanded to provide a brief overview of what each benchmark assesses.
• Visual Appeal: The figure is visually appealing and easy to read. The bars are clearly labeled, and the color scheme is distinct.
• Axis labels: The y-axis is labeled as "Accuracy/Percentile (%)", which is appropriate. However, it may be beneficial to add a label to the x-axis, such as "Benchmarks" or "Evaluation Tasks"

Introduction

Key Aspects

• Context and Motivation: The introduction establishes the context by highlighting the rapid advancements in Large Language Models (LLMs) and their progression towards Artificial General Intelligence (AGI). It emphasizes the growing importance of post-training techniques in enhancing reasoning, aligning with values, and adapting to user preferences, especially in comparison to the resource-intensive pre-training phase. This sets the stage for the paper's focus on improving reasoning capabilities through reinforcement learning.
• Problem Statement: The introduction identifies a key challenge in the field: the difficulty of achieving effective test-time scaling for reasoning tasks. While inference-time scaling methods like Chain-of-Thought have shown improvements, achieving general reasoning performance comparable to state-of-the-art models like OpenAI's o1 series remains an open problem. This establishes the research gap that the paper aims to address.
• Approach: Pure Reinforcement Learning: The paper introduces its core approach: using pure reinforcement learning (RL) to improve language model reasoning capabilities. The novelty lies in exploring the potential of LLMs to develop reasoning without any supervised data, focusing on self-evolution through a pure RL process. DeepSeek-V3-Base is used as the base model, and GRPO is employed as the RL framework.
• Introduction of DeepSeek-R1-Zero: DeepSeek-R1-Zero is introduced as a model trained purely through RL, without any supervised fine-tuning. It demonstrates significant performance improvements on reasoning benchmarks, such as the AIME 2024, where it achieves a pass@1 score increase from 15.6% to 71.0%, matching OpenAI-o1-0912's performance. This model serves as a proof of concept for the effectiveness of the pure RL approach.
• Addressing Limitations: DeepSeek-R1: The introduction acknowledges the limitations of DeepSeek-R1-Zero, such as poor readability and language mixing. To address these, DeepSeek-R1 is introduced, which incorporates a small amount of cold-start data and a multi-stage training pipeline. This includes fine-tuning with cold-start data, reasoning-oriented RL, rejection sampling to create new SFT data, and a final RL phase considering all scenarios. DeepSeek-R1 achieves performance on par with OpenAI-o1-1217.
• Distillation to Smaller Models: The paper further explores distilling the reasoning capabilities of DeepSeek-R1 into smaller, more efficient dense models. Using Qwen2.5-32B as a base, direct distillation outperforms applying RL, demonstrating the importance of reasoning patterns discovered by larger models. The authors open-source the distilled Qwen and Llama series, with the 14B model outperforming QwQ-32B-Preview and the 32B and 70B models setting new records among dense models on reasoning benchmarks.
• Contributions: The introduction outlines the main contributions of the paper, including the novel application of RL without supervised fine-tuning, the development of the DeepSeek-R1 pipeline, the demonstration of effective distillation to smaller models, and the open-sourcing of the distilled models. These contributions aim to advance the field of reasoning in LLMs and provide valuable resources for the research community.
• Summary of Evaluation Results: The introduction provides a brief overview of the evaluation results, highlighting DeepSeek-R1's strong performance on reasoning tasks, coding-related tasks, and knowledge benchmarks. It mentions specific achievements, such as surpassing OpenAI-o1-1217 on AIME 2024 and MATH-500, achieving expert-level performance on Codeforces, and outperforming DeepSeek-V3 on MMLU, MMLU-Pro, and GPQA Diamond. These results demonstrate the effectiveness of the proposed approach and the competitive standing of DeepSeek-R1.

Strengths

• Clear Problem Definition

  The introduction effectively defines the problem of limited reasoning capabilities in current LLMs and the need for improved test-time scaling, establishing a clear motivation for the research.

  "However, the challenge of effective test-time scaling remains an open question for the research community." (Page 3)
• Well-Structured Overview

  The introduction provides a well-structured overview of the paper's approach, including the use of pure RL, the development of DeepSeek-R1-Zero and DeepSeek-R1, and the exploration of distillation techniques.

  "In this paper, we take the first step toward improving language model reasoning capabilities using pure reinforcement learning (RL)." (Page 3)
• Strong Justification for Approach

  The introduction strongly justifies the use of pure RL by highlighting its potential to enable self-evolution of reasoning capabilities without supervised data, a novel and significant contribution to the field.

  "Our goal is to explore the potential of LLMs to develop reasoning capabilities without any supervised data, focusing on their self-evolution through a pure RL process." (Page 3)
• Clear Presentation of Contributions

  The introduction clearly presents the main contributions of the paper in a dedicated subsection, making it easy for readers to understand the significance of the work.

  "1.1. Contributions" (Page 4)
• Concise Summary of Results

  The introduction provides a concise summary of the evaluation results, highlighting key achievements and demonstrating the effectiveness of the proposed models.

  "1.2. Summary of Evaluation Results" (Page 4)

Suggestions for Improvement

• Expand on Cold-Start Data

  This medium-impact improvement would provide more context on the nature and collection process of the cold-start data used for DeepSeek-R1. The Introduction section particularly needs this detail as it sets the stage for understanding the model's training process and how it differs from DeepSeek-R1-Zero.

  "we begin by collecting thousands of cold-start data to fine-tune the DeepSeek-V3-Base model." (Page 3)

  Implementation: Briefly describe the type, source, and collection method of the cold-start data. For example, "To address the limitations of DeepSeek-R1-Zero, we introduce DeepSeek-R1, which incorporates a small amount of carefully curated cold-start data. This data consists of thousands of examples of long Chain-of-Thought reasoning, collected through a combination of few-shot prompting, expert annotation, and refinement of DeepSeek-R1-Zero outputs. The data focuses on diverse reasoning tasks, including mathematical, logical, and scientific problems, and is formatted to ensure readability and coherence."

• Clarify Distillation Process

  This medium-impact improvement would enhance the reader's understanding of the distillation process and its significance. The Introduction section particularly needs this detail as it highlights a key contribution of the paper and sets the stage for the later discussion on distillation.

  "We further explore distillation from DeepSeek-R1 to smaller dense models." (Page 3)

  Implementation: Briefly explain the key aspects of the distillation process. For example, "We further explore distillation as a method to transfer the reasoning capabilities of DeepSeek-R1 to smaller, more efficient dense models. This process involves using the outputs of DeepSeek-R1 as training data for smaller models based on Qwen and Llama architectures. Specifically, we generate 800,000 training samples from DeepSeek-R1, covering a wide range of reasoning tasks and incorporating both reasoning processes and final summaries. By fine-tuning the smaller models on this data, we aim to impart the reasoning patterns and strategies learned by the larger model."

• Provide More Context on GRPO

  This low-impact improvement would provide readers with a better understanding of the chosen RL framework and its advantages. The Introduction section particularly needs this detail as it introduces a key methodological component of the paper.

  "we use DeepSeek-V3-Base as the base model and employ GRPO (Shao et al., 2024) as the RL framework" (Page 3)

  Implementation: Briefly describe GRPO and its key features. For example, "To improve model performance in reasoning, we employ GRPO (Group Relative Policy Optimization) as the RL framework. GRPO is a recent advancement in RL that optimizes the policy model by estimating the baseline from group scores, rather than relying on a separate critic model. This approach significantly reduces the training costs of RL while maintaining effectiveness, making it particularly suitable for large-scale training of LLMs."

Approach

Key Aspects

• Overview of the Approach: This section outlines a novel approach to enhance the reasoning capabilities of large language models (LLMs) through reinforcement learning (RL). It introduces a two-pronged strategy: first, applying RL directly to a base model without any supervised fine-tuning (SFT), resulting in DeepSeek-R1-Zero; and second, using a small amount of SFT data as a 'cold start' before applying RL, leading to DeepSeek-R1. Additionally, it explores distilling the reasoning capabilities learned by DeepSeek-R1 into smaller, more efficient dense models. The core idea is to leverage RL to enable the model to autonomously develop and refine its reasoning processes, moving beyond the limitations of traditional supervised learning approaches.
• DeepSeek-R1-Zero: Pure Reinforcement Learning: This subsection details the development of DeepSeek-R1-Zero, a model trained purely through RL without any supervised data. It employs Group Relative Policy Optimization (GRPO), an efficient RL algorithm that forgoes a separate critic model. The reward system is rule-based, focusing on accuracy and format, with a specific template guiding the model to produce a reasoning process followed by a final answer. This approach allows for the observation of the model's natural evolution in reasoning, leading to significant performance improvements on benchmarks like AIME 2024.
• Reinforcement Learning Algorithm: GRPO: The paper utilizes Group Relative Policy Optimization (GRPO) as its core RL algorithm. GRPO is a technique that optimizes the policy model by estimating the baseline from group scores instead of using a separate critic model, which is typically the same size as the policy model. This approach significantly reduces training costs. Specifically, for each question, GRPO samples a group of outputs from the old policy and optimizes the policy model by maximizing a specific objective function, incorporating advantage and a KL divergence penalty term. The advantage is computed using the rewards of the outputs within each group, providing a measure of relative performance.
• Reward Modeling: The reward modeling in this study is entirely rule-based, comprising two main components: accuracy rewards and format rewards. Accuracy rewards assess the correctness of the model's response, using methods like rule-based verification for math problems or a compiler for LeetCode problems. Format rewards enforce a specific structure, requiring the model to enclose its thinking process within '<think>' and '</think>' tags. The authors deliberately avoid using neural reward models to prevent reward hacking and simplify the training pipeline.
• Training Template for DeepSeek-R1-Zero: A specific training template is designed to guide DeepSeek-R1-Zero. This template, shown in Table 1, structures the interaction as a conversation between a User and an Assistant. The Assistant is required to first provide a reasoning process within '<think>' and '</think>' tags, followed by the final answer within '<answer>' and '</answer>' tags. This template is intentionally kept simple, focusing on structural format rather than content-specific biases, to allow for the observation of the model's natural progression during RL.
• Performance and Self-Evolution of DeepSeek-R1-Zero: DeepSeek-R1-Zero demonstrates significant performance improvements through the RL process, as evidenced by its performance on the AIME 2024 benchmark. The model's pass@1 score increases from 15.6% to 71.0%, comparable to OpenAI-o1-0912. The self-evolution process is characterized by an increase in 'thinking time,' represented by the generation of longer reasoning sequences. This leads to the emergence of sophisticated behaviors like reflection and exploration of alternative approaches, ultimately enhancing the model's reasoning capabilities.
• DeepSeek-R1: Reinforcement Learning with Cold Start: This subsection introduces DeepSeek-R1, which addresses the limitations of DeepSeek-R1-Zero, such as poor readability and language mixing. DeepSeek-R1 incorporates a small amount of 'cold start' data, consisting of long Chain-of-Thought (CoT) examples, to fine-tune the base model before applying RL. The training pipeline involves four stages: 1) fine-tuning with cold-start data, 2) reasoning-oriented RL, 3) rejection sampling to create new SFT data, and 4) a final RL phase considering all scenarios. This approach aims to improve both reasoning performance and the model's general capabilities.
• Distillation to Smaller Models: The paper explores distilling the reasoning capabilities of DeepSeek-R1 into smaller, more efficient dense models. This is achieved by fine-tuning open-source models like Qwen and Llama using the training data generated by DeepSeek-R1. The results demonstrate that this straightforward distillation method significantly enhances the reasoning abilities of smaller models, with the distilled models outperforming other models on various benchmarks. This highlights the effectiveness of transferring the reasoning patterns learned by larger models to smaller ones.

Strengths

• Clear Explanation of Methodology

  The section provides a clear and detailed explanation of the reinforcement learning methodology used, including the GRPO algorithm, reward modeling, and training template.

  "In order to save the training costs of RL, we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which foregoes the critic model that is typically the same size as the policy model, and estimates the baseline from group scores instead." (Page 5)
• Well-Defined Training Template

  The training template for DeepSeek-R1-Zero is well-defined and provides a clear structure for the model's learning process.

  "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. User: prompt. Assistant:" (Page 6)
• Comprehensive Description of DeepSeek-R1 Pipeline

  The section provides a comprehensive description of the four-stage pipeline used to train DeepSeek-R1, including the rationale behind each stage.

  "To address these questions, we design a pipeline to train DeepSeek-R1. The pipeline consists of four stages, outlined as follows." (Page 9)
• Justification for Design Choices

  The authors provide clear justifications for their design choices, such as the use of a rule-based reward system and the avoidance of neural reward models.

  "We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process" (Page 6)

Suggestions for Improvement

• Elaborate on Hyperparameter Selection

  This medium-impact improvement would provide greater transparency and reproducibility to the study. While the Approach section mentions the existence of hyperparameters, it lacks detail on their specific values and the process for selecting them, which is crucial for understanding the model's training dynamics.

  "where 𝜀 and 𝛽 are hyper-parameters, and 𝐴𝑖 is the advantage, computed using a group of rewards {𝑟1, 𝑟2, . . . , 𝑟𝐺} corresponding to the outputs within each group:" (Page 5)

  Implementation: Provide a table or appendix listing the key hyperparameters used in the GRPO algorithm, such as epsilon and beta, along with their chosen values. Briefly describe the method used for selecting these values, such as grid search, Bayesian optimization, or prior experience. For example, "The hyperparameter epsilon, controlling the clipping in the GRPO objective, was set to 0.2, while beta, the coefficient for the KL divergence penalty, was set to 0.01. These values were chosen based on a preliminary grid search over a range of values, selecting the combination that yielded the best performance on a validation set."

• Clarify Language Consistency Reward

  This medium-impact improvement would enhance the reader's understanding of a key component of the training process for DeepSeek-R1. The Approach section introduces the concept of a language consistency reward but lacks sufficient detail on its calculation and implementation, which is important for understanding how language mixing is mitigated.

  "To mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT." (Page 10)

  Implementation: Provide a more detailed description of how the language consistency reward is calculated. For example, "The language consistency reward is calculated as the proportion of words in the generated Chain-of-Thought that belong to the target language, determined using a pre-trained language identification model. Specifically, we tokenize the CoT, identify the language of each token, and compute the ratio of target language tokens to the total number of tokens. This ratio is then scaled and incorporated into the overall reward signal." Additionally, specify how this reward is combined with the accuracy reward.

• Expand on Cold-Start Data Collection

  This medium-impact improvement would provide more context and transparency regarding the creation of the cold-start data used for DeepSeek-R1. While the Approach section mentions the collection of cold-start data, it lacks specific details on the data sources, annotation process, and quality control measures, which are crucial for understanding the foundation of DeepSeek-R1's training.

  "To collect such data, we have explored several approaches: using few-shot prompting with a long CoT as an example, directly prompting models to generate detailed answers with reflection and verification, gathering DeepSeek-R1-Zero outputs in a readable format, and refining the results through post-processing by human annotators." (Page 9)

  Implementation: Provide more details on the sources of the cold-start data, such as the specific prompts or tasks used for few-shot prompting and the criteria for selecting outputs from DeepSeek-R1-Zero. Describe the role of human annotators in refining the data, including their expertise and the guidelines they followed. For example, "The cold-start data was collected from three primary sources: 1) few-shot prompting of a pre-trained language model with examples of long CoT reasoning, 2) selected outputs from DeepSeek-R1-Zero that demonstrated coherent reasoning but required formatting improvements, and 3) expert-written examples of reasoning processes for complex tasks. Human annotators, consisting of graduate students in computer science and mathematics, were tasked with refining the collected data to ensure readability, logical coherence, and adherence to the specified format. Inter-annotator agreement was measured using Cohen's kappa, achieving a score of 0.85, indicating substantial agreement."

Non-Text Elements

Table 1 | Template for DeepSeek-R1-Zero. prompt will be replaced with the...

Full Caption

Table 1 | Template for DeepSeek-R1-Zero. prompt will be replaced with the specific reasoning question during training.

Figure/Table Image (Page 6)

First Reference in Text

To train DeepSeek-R1-Zero, we begin by designing a straightforward template that guides the base model to adhere to our specified instructions. As depicted in Table 1, this template requires DeepSeek-R1-Zero to first produce a reasoning process, followed by the final answer.

Description

• Purpose of the table: This table shows a template, which is like a pre-formatted structure, used for training an AI model called DeepSeek-R1-Zero. Think of it as giving the AI a specific set of instructions to follow when answering questions.
• Structure of the template: The template outlines a conversation between a "User" and an "Assistant" (the AI). It's designed to guide the AI to first think through the problem step-by-step and then provide the answer. The thinking process is enclosed in special tags called "<think>" and "</think>", and the answer is enclosed in "<answer>" and "</answer>". These tags act like markers, telling the AI where the reasoning part starts and ends, and where the final answer starts and ends.
• Role of "prompt": The word "prompt" in the template will be replaced with an actual question that the AI needs to answer during its training. This is how the AI learns to solve different types of problems - by being given many different prompts (questions) and following the template to produce a reasoned answer.
• Training process: During training, which is like the learning phase for the AI, the model is given many examples of questions and is expected to follow this template. It learns to generate text that represents its "thinking" process within the <think> tags and then to produce a final answer within the <answer> tags. This structured approach helps the AI learn to reason through problems in a logical manner before arriving at a solution.

Scientific Validity

• Methodological clarity: The template provides a clear and structured approach for training the model to produce reasoning processes followed by answers. This method aligns with the goal of eliciting explicit reasoning from the model.
• Reproducibility: The template is well-defined, enhancing the reproducibility of the training process. Other researchers can use this template to train their models and potentially achieve similar results.
• Potential limitations: The template's strictness might limit the model's flexibility in exploring different reasoning styles. The model may become overly reliant on this specific format and struggle with prompts that deviate from it.

Communication

• Table clarity: The table is straightforward and easy to understand. The format of the conversation between the User and Assistant is clearly presented.
• Caption effectiveness: The caption is concise and accurately describes the table's content. The clarification that "prompt" will be replaced during training is helpful.
• Use of tags: The use of tags (<think> and <answer>) is clearly explained in the reference text, making it easy to understand their purpose in the template.
• Contextual understanding: The reference text provides sufficient context for understanding the purpose and use of the template within the overall training process.

Table 2 | Comparison of DeepSeek-R1-Zero and OpenAI o1 models on...

Full Caption

Table 2 | Comparison of DeepSeek-R1-Zero and OpenAI o1 models on reasoning-related benchmarks.

Figure/Table Image (Page 7)

First Reference in Text

Table 2 provides a comparative analysis between DeepSeek-R1-Zero and OpenAI's o1-0912 models across a variety of reasoning-related benchmarks.

Description

• Purpose of the table: This table compares the performance of an AI model called DeepSeek-R1-Zero with two other AI models developed by OpenAI. These models are being tested on their ability to reason, which means their ability to think logically and solve problems.
• Models being compared: The table compares three models: DeepSeek-R1-Zero, OpenAI-o1-mini, and OpenAI-o1-0912. The names "OpenAI-o1-mini" and "OpenAI-o1-0912" suggest these are different versions or variations of models developed by the company OpenAI. The "mini" likely suggests a smaller or less complex version compared to "0912".
• Benchmarks used: The models are tested on different "benchmarks," which are like standardized tests for AI. These benchmarks have names like AIME 2024, MATH-500, GPQA Diamond, LiveCode Bench, and CodeForces. Each benchmark likely tests a different aspect of reasoning ability, such as math problem-solving or coding skills. Think of these benchmarks as different exams an AI has to take to show its proficiency.
• Metrics used: The table uses metrics like "pass@1," "cons@64," and "rating" to measure performance. "pass@1" likely means the percentage of times the AI gets the correct answer on the first try. "cons@64" probably involves getting a consensus answer from multiple attempts (64 in this case). "Rating" is a score that reflects the AI's overall performance on a particular benchmark, similar to how players are rated in a game like chess.

Scientific Validity

• Appropriateness of Comparison: Comparing DeepSeek-R1-Zero with OpenAI models is relevant, given OpenAI's prominence in the field. The selection of o1-0912 as a specific point of comparison is justified by its reported strong performance on reasoning tasks.
• Benchmark Relevance: The chosen benchmarks (AIME 2024, MATH-500, GPQA Diamond, LiveCode Bench, CodeForces) are appropriate for evaluating reasoning capabilities, covering areas like mathematics, coding, and problem-solving. However, a more detailed justification for selecting these specific benchmarks would strengthen the paper.
• Metric Validity: The metrics (pass@1, cons@64, rating) are relevant to the evaluation of reasoning performance. However, the table caption could benefit from a more explicit definition of these metrics and their significance in the context of AI evaluation.
• Statistical Significance: The table presents numerical results but lacks information on statistical significance. Reporting confidence intervals or p-values would enhance the scientific rigor of the comparison.

Communication

• Table Structure: The table is well-structured, with clear row and column labels. The layout facilitates easy comparison between the models across different benchmarks.
• Caption Accuracy: The caption accurately describes the table's content. However, it could be more informative by briefly explaining the significance of the comparison and the benchmarks used.
• Accessibility to Non-Experts: While the table is technically sound, a non-expert reader might find it challenging to fully grasp the significance of the results without a deeper understanding of the benchmarks and metrics. Expanding the caption or providing a brief explanation in the main text could improve accessibility.
• Model Abbreviation: The use of "o1" as an abbreviation for OpenAI is not immediately clear and could be confusing. Using the full name "OpenAI" would improve clarity.

Figure 2 | AIME accuracy of DeepSeek-R1-Zero during training. For each...

Full Caption

Figure 2 | AIME accuracy of DeepSeek-R1-Zero during training. For each question, we sample 16 responses and calculate the overall average accuracy to ensure a stable evaluation.

Figure/Table Image (Page 7)

First Reference in Text

Figure 2 depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process.

Description

• Type of graph: This is a line graph that shows how the accuracy of an AI model, called DeepSeek-R1-Zero, changes over time as it undergoes a process called "training." Think of training as the learning phase for the AI, where it gets better at a task by practicing.
• What is being measured: The graph measures "accuracy," which is how often the AI gets the right answer on a test called AIME. AIME stands for American Invitational Mathematics Examination, which is a challenging math competition. So, this graph is showing how good the AI is at solving math problems from this specific exam.
• X-axis (Steps): The horizontal axis, or the x-axis, represents the "steps" of the training process. Imagine these steps as practice sessions for the AI. As the AI goes through more steps, it gets more practice and, hopefully, gets better at the task.
• Y-axis (Accuracy): The vertical axis, or the y-axis, represents the "accuracy" of the AI. This is a measure of how well the AI is performing on the AIME test, likely expressed as a percentage of correct answers. A higher point on the y-axis means higher accuracy, or that the AI is getting more answers right.
• Data points and lines: The graph has multiple lines, each representing a different aspect of the AI's performance. There is a blue line for "r1-zero-pass@1", a red line for "r1-zero-cons@16", and two dashed lines representing the performance of another AI model called "o1-0912" with "pass@1" and "cons@64". "pass@1" means the AI is given one chance to answer a question, while "cons@16" or "cons@64" likely mean the AI gets to try multiple times (16 or 64) and the best or most common answer is chosen. The lines connect data points, which represent the AI's accuracy at different stages of training. By looking at the lines, you can see if the AI's accuracy is improving, staying the same, or getting worse over time.
• Sampling method: The caption mentions that for each question, they "sample 16 responses." This means that instead of just asking the AI a question once, they ask it the same question 16 different times and then calculate the average accuracy across those responses. This is done to get a more stable and reliable measure of the AI's performance, as it reduces the chance that a single lucky or unlucky answer will skew the results.

Scientific Validity

• Relevance to training process: The figure is directly relevant to understanding the training process of DeepSeek-R1-Zero, as it illustrates the model's performance trajectory on a key benchmark (AIME 2024) over the course of reinforcement learning (RL) training.
• Appropriateness of benchmark: AIME 2024 is a suitable benchmark for evaluating mathematical reasoning abilities. Using a standardized test like AIME allows for objective comparison with other models and provides a clear measure of the model's problem-solving capabilities in this domain.
• Sampling methodology: The caption states that 16 responses are sampled for each question to calculate average accuracy. This approach helps to mitigate the effects of randomness in the model's responses and provides a more stable evaluation. However, the rationale for choosing 16 samples could be further elaborated.
• Comparison with baseline: The inclusion of a baseline (OpenAI's o1-0912 model) allows for a direct comparison of DeepSeek-R1-Zero's performance against an established model. This provides context for evaluating the effectiveness of the training process.
• Clarity of metrics: The figure uses "pass@1" and "cons@16/64" as metrics. While these are mentioned, a more detailed explanation of their calculation and significance would enhance the scientific rigor. Providing definitions within the caption or figure itself would be beneficial.

Communication

• Clarity of trend visualization: The line graph effectively visualizes the performance trend of DeepSeek-R1-Zero during training. The upward trend in accuracy is readily apparent, indicating that the model is learning and improving over time.
• Caption informativeness: The caption is informative, explaining the purpose of the figure, the sampling method, and the benchmark used. However, it could be further improved by defining the metrics ("pass@1" and "cons@16/64") more explicitly.
• Axis labeling: The y-axis is labeled as "Accuracy," which is appropriate. The x-axis is labeled as "Steps," which is understandable in the context of training. However, adding units or further clarifying what constitutes a "step" could enhance clarity.
• Legend clarity: The legend distinguishes between different lines using different colors and labels. It would be beneficial to provide a brief explanation of "r1-zero" in the legend or caption to clarify that it refers to DeepSeek-R1-Zero.
• Visual distinction of lines: Using solid lines for DeepSeek-R1-Zero and dashed lines for the baseline (o1-0912) is a good visual choice, making it easy to distinguish between the model being studied and the comparison model.

Figure 3 | The average response length of DeepSeek-R1-Zero on the training set...

Full Caption

Figure 3 | The average response length of DeepSeek-R1-Zero on the training set during the RL process. DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time.

Figure/Table Image (Page 8)

First Reference in Text

As depicted in Figure 3, the thinking time of DeepSeek-R1-Zero shows consistent improve-

Description

• Type of graph: This is a line graph, which is a type of chart that displays information as a series of data points connected by straight line segments. It's used here to show how a quantity changes over time.
• What is being measured: The graph measures the "average response length" of an AI model named DeepSeek-R1-Zero. "Response length" likely refers to the number of words or characters the AI uses when answering a question or solving a problem. Think of it as how much the AI "writes" or "says" when it's working on a task.
• Training set: The "training set" refers to the set of data that is used to teach the AI. It's like the practice material the AI uses to learn and improve its abilities. In this case, it seems like the AI is learning to solve reasoning tasks, which are problems that require logical thinking.
• RL process: RL stands for Reinforcement Learning. This is a type of machine learning where the AI learns by trial and error, receiving rewards for correct actions and penalties for incorrect ones. It's like teaching a dog a trick by giving it treats when it does something right. Here, the "RL process" refers to the period during which the AI is being trained using this method.
• X-axis (Steps): The horizontal axis, or x-axis, represents "steps." These are likely individual stages or iterations in the training process. As the AI goes through more steps, it has more opportunities to learn and refine its responses.
• Y-axis (Average length per response): The vertical axis, or y-axis, represents the "average length per response." This is a measure of how long the AI's responses are, on average. A higher value on the y-axis means the AI is producing longer responses.
• Trend shown: The graph shows an upward trend, meaning the average length of the AI's responses increases as it goes through more training steps. The caption suggests that this is because the AI is learning to "think" more, or to spend more time processing information, before giving an answer. The shaded area around the line likely represents some measure of variability or uncertainty in the average response length.

Scientific Validity

• Correlation between response length and reasoning: The figure suggests a correlation between increased response length and improved reasoning ability, as implied by the caption "DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time." However, the paper does not provide a direct measure of "thinking time." Response length is used as a proxy, which is a reasonable but indirect measure. Further analysis or justification for this correlation would strengthen the scientific validity.
• Measurement of response length: The caption mentions "average response length," but the specific units (e.g., words, tokens, characters) are not specified in the caption or the visible part of the figure. Providing this information would enhance clarity and reproducibility.
• Statistical significance of the trend: The upward trend in response length is visually apparent, but the figure lacks information about the statistical significance of this trend. Reporting confidence intervals or p-values would provide a more rigorous assessment of the observed increase.
• Causality vs correlation: While the figure suggests that the model learns to solve reasoning tasks with more "thinking time" (as measured by response length), it's important to note that correlation does not necessarily imply causation. Other factors during the RL process could contribute to both increased response length and improved reasoning performance.

Communication

• Clarity of trend visualization: The line graph effectively visualizes the trend of increasing average response length during the RL process. The upward trend is easily discernible, supporting the claim that the model is producing longer responses over time.
• Caption informativeness: The caption is informative, explaining the purpose of the figure and the observed trend. It also introduces the concept of "thinking time" as an interpretation of the increased response length.
• Axis labeling: The y-axis is labeled "Average length per response," which is appropriate but could be made more precise by specifying the unit of measurement (e.g., words, tokens). The x-axis is labeled "Steps," which is understandable in the context of training, but providing more detail on what constitutes a "step" would enhance clarity.
• Shaded area around the line: The shaded area around the line likely represents the standard deviation or confidence interval of the average response length. However, this is not explicitly stated in the caption. Clarifying the meaning of the shaded area would improve the figure's interpretability.
• Visual appeal and simplicity: The figure is visually simple and uncluttered, making it easy to focus on the main trend. The use of a single line and a clear color scheme contributes to its visual appeal.

Table 3 | An interesting “aha moment" of an intermediate version of...

Full Caption

Table 3 | An interesting “aha moment" of an intermediate version of DeepSeek-R1-Zero. The model learns to rethink using an anthropomorphic tone. This is also an aha moment for us, allowing us to witness the power and beauty of reinforcement learning.

Figure/Table Image (Page 9)

First Reference in Text

This moment, as illustrated in Table 3, occurs in an intermediate version of the model.

Description

• Content of the table: This table shows an example of an AI model called DeepSeek-R1-Zero at a certain stage in its development, where it demonstrates an "aha moment." An "aha moment" is like a sudden realization or a breakthrough in understanding. In this case, it's the AI showing an unexpected ability to rethink its approach to a problem.
• The math problem: The AI is given a math problem to solve: finding the sum of the real solutions of the equation √(a - √(a + x)) = x, where a > 1. This is an algebra problem that involves square roots and requires some manipulation to solve.
• Initial approach: The AI starts by attempting to solve the equation step-by-step, using algebraic manipulations like squaring both sides and rearranging terms. It's like the AI is showing its work on a math test.
• The "aha moment": After some steps, the AI says, "Wait, wait. Wait. That's an aha moment I can flag here." This is the interesting part. The AI seems to realize that it might need to reconsider its approach. It then says, "Let's reevaluate this step-by-step to identify if the correct sum can be ..." and starts to review its work.
• Anthropomorphic tone: The caption mentions that the model uses an "anthropomorphic tone." This means the AI is using language that makes it sound like a human. Phrases like "Wait, wait. Wait." and "That's an aha moment I can flag here" are things a person might say when they're thinking through a problem. It's as if the AI is talking to itself, just like a human might do.
• Significance for researchers: The caption also states that this is an "aha moment" for the researchers as well. They see this as a demonstration of the power of reinforcement learning, which is the method they're using to train the AI. Reinforcement learning is a way of teaching AI by rewarding it for correct actions. The fact that the AI is showing this kind of self-reflection or rethinking ability is a significant and exciting result for them.

Scientific Validity

• Subjectivity of "aha moment": The concept of an "aha moment" is inherently subjective. While the example provided is intriguing, it's difficult to objectively quantify or measure this phenomenon. The interpretation of the model's response as an "aha moment" relies on the researchers' judgment.
• Anecdotal evidence: The table presents a single example, which is essentially anecdotal evidence. While interesting, a single instance does not provide strong scientific evidence of a general capability for rethinking. A more rigorous analysis would involve multiple examples and a systematic evaluation of the model's ability to reconsider its approach across different problems and contexts.
• Anthropomorphism and scientific rigor: The use of anthropomorphic language ("aha moment," "rethink") can be helpful for conveying the significance of the observation, but it's important to maintain scientific rigor. The paper should clearly distinguish between the model's actual behavior (generating a specific sequence of text) and the researchers' interpretation of that behavior.
• Reproducibility: The table provides a specific example, but it's not clear how often the model exhibits this "rethinking" behavior. Providing information on the frequency or conditions under which this behavior occurs would enhance the reproducibility of the observation.

Communication

• Intriguing example: The table presents a captivating example that is likely to pique the reader's interest. The "aha moment" and the model's human-like language are attention-grabbing.
• Caption clarity: The caption effectively conveys the main point of the table, highlighting the "aha moment" and its significance for both the model and the researchers.
• Table layout: The table is well-formatted and easy to read. The use of bold text for the "aha moment" statement helps to emphasize it.
• Contextual understanding: The reference text and the caption provide sufficient context for understanding the significance of the example within the broader research.
• Potential for misinterpretation: The use of anthropomorphic language, while engaging, could potentially lead to misinterpretation. Some readers might overattribute human-like qualities or consciousness to the model based on this example.

Experiment

Key Aspects

• Evaluation Benchmarks and Metrics: The Experiment section details the evaluation of DeepSeek-R1 and its distilled models across a wide range of benchmarks, including MMLU, MMLU-Redux, MMLU-Pro, C-Eval, CMMLU, IFEval, FRAMES, GPQA Diamond, SimpleQA, C-SimpleQA, SWE-Bench Verified, Aider, LiveCodeBench, Codeforces, CNMO 2024, and AIME 2024. Performance is measured using metrics such as Pass@1, consensus@64, accuracy, F1 score, and Elo rating. These benchmarks cover various domains, including general knowledge, coding, mathematics, and long-context understanding, providing a comprehensive assessment of the models' capabilities.
• Evaluation Setup and Methodology: The evaluation setup involves setting the maximum generation length to 32,768 tokens and using a sampling temperature of 0.6 and a top-p value of 0.95 for generating responses. The authors default to pass@k evaluation and report pass@1 using a non-zero temperature to address issues with greedy decoding, which can lead to higher repetition rates and variability. For AIME 2024, consensus (majority vote) results are also reported using 64 samples. The evaluation prompts follow the setup in DeepSeek-V3, with modifications for zero-shot settings where necessary. The CoT (Chain of Thought) in few-shot may negatively affect DeepSeek-R1 performance.
• Baselines for Comparison: The models are evaluated against several strong baselines, including DeepSeek-V3, Claude-Sonnet-3.5-1022, GPT-4o-0513, OpenAI-01-mini, and OpenAI-01-1217. For distilled models, the open-source model QwQ-32B-Preview is also used as a baseline. The performance of OpenAI-01-1217 is based on official reports due to access challenges in mainland China. These baselines provide a context for understanding the performance of DeepSeek-R1 and its distilled versions relative to other state-of-the-art models.
• Performance of DeepSeek-R1: DeepSeek-R1 demonstrates strong performance across various benchmarks. On AIME 2024, it achieves a Pass@1 score of 79.8%, surpassing OpenAI-01-1217. On MATH-500, it attains a score of 97.3%, performing on par with OpenAI-01-1217. In coding-related tasks, it achieves expert-level performance on Codeforces (96.3% percentile) and performs slightly better than DeepSeek-V3 on engineering-related tasks. On knowledge benchmarks like MMLU, MMLU-Pro, and GPQA Diamond, it significantly outperforms DeepSeek-V3. These results highlight the effectiveness of the reinforcement learning approach in enhancing the model's reasoning and general capabilities.
• Evaluation of Distilled Models: The evaluation of distilled models shows that simply distilling DeepSeek-R1's outputs into smaller models significantly enhances their reasoning abilities. For example, DeepSeek-R1-Distill-Qwen-7B outperforms non-reasoning models like GPT-4o-0513 across the board, and DeepSeek-R1-Distill-Qwen-14B surpasses QwQ-32B-Preview on all evaluation metrics. DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B significantly exceed OpenAI-01-mini on most benchmarks. These findings demonstrate the strong potential of distillation in transferring reasoning capabilities from larger to smaller models.
• Comparison of Distillation and Reinforcement Learning: The section compares the performance of models trained through distillation versus those trained through large-scale reinforcement learning. Experiments with Qwen-32B-Base show that while large-scale RL can achieve performance on par with QwQ-32B-Preview, distillation from DeepSeek-R1 yields significantly better results. This suggests that distilling more powerful models into smaller ones is more effective and economical than relying solely on large-scale RL for smaller models. However, it also indicates that advancing beyond current intelligence boundaries may require more powerful base models and larger-scale reinforcement learning.

Strengths

• Comprehensive Evaluation Suite

  The paper employs a wide array of benchmarks covering diverse domains, providing a thorough assessment of the models' capabilities.

  "Benchmarks We evaluate models on MMLU, MMLU-Pro, C-Eval, and CMMLU, IFEval, FRAMES, GPQA Diamond, SimpleQA, C-SimpleQA, SWE-Bench Verified," (Page 11)
• Clear Evaluation Methodology

  The authors clearly describe the evaluation setup, including the use of pass@k evaluation and the rationale behind it, enhancing transparency.

  "Therefore, we default to pass@k evaluation and report pass@1 using a non-zero temperature." (Page 12)
• Strong Baselines for Comparison

  The models are compared against several strong baselines, including state-of-the-art models like GPT-4o and OpenAI-01-1217, providing a robust performance context.

  "We conduct comprehensive evaluations against several strong baselines, including DeepSeek-V3, Claude-Sonnet-3.5-1022, GPT-40-0513, OpenAI-01-mini, and OpenAI-01-1217." (Page 12)
• Detailed Performance Analysis

  The paper provides a detailed analysis of DeepSeek-R1's performance across different tasks, highlighting its strengths in reasoning, coding, and knowledge-based tasks.

  "On math tasks, DeepSeek-R1 demonstrates performance on par with OpenAI-01-1217, surpassing other models by a large margin." (Page 14)
• Effective Distillation Results

  The evaluation of distilled models demonstrates the effectiveness of distillation in transferring reasoning capabilities, showcasing significant performance improvements.

  "simply distilling DeepSeek-R1's outputs enables the efficient DeepSeek-R1-7B to outperform non-reasoning models like GPT-40-0513 across the board." (Page 14)

Suggestions for Improvement

• Provide More Details on Hyperparameter Tuning

  This medium-impact improvement would enhance the reproducibility and transparency of the study. The Experiment section mentions the use of specific hyperparameters but lacks detail on the process for selecting these values and their potential impact on model performance, which is crucial for understanding the robustness of the results.

  "Specifically, we use a sampling temperature of 0.6 and a top-p value of 0.95 to generate k responses" (Page 12)

  Implementation: Provide a table or appendix listing the key hyperparameters used in the evaluation, such as the range of temperatures and top-p values explored during sampling. Briefly describe the method used for selecting these values, such as grid search or Bayesian optimization, and report the criteria used for determining the optimal settings. For example, "We conducted a grid search over temperature values ranging from 0.2 to 1.0 in increments of 0.2 and top-p values ranging from 0.8 to 0.95 in increments of 0.05. The optimal values of 0.6 for temperature and 0.95 for top-p were selected based on the best average Pass@1 performance on a held-out validation set."

• Clarify Handling of Invalid or Refused Outputs

  This medium-impact improvement would enhance the clarity and completeness of the evaluation methodology. The Experiment section mentions that DeepSeek-R1 tends to refuse answering certain queries after safety RL, but it does not fully explain how these refused or invalid outputs are handled in the calculation of performance metrics, which is important for accurately interpreting the results.

  "However, DeepSeek-R1 performs worse than DeepSeek-V3 on the Chinese SimpleQA benchmark, primarily due to its tendency to refuse answering certain queries after safety RL." (Page 13)

  Implementation: Clearly state how refused or invalid outputs are handled in the calculation of each performance metric. For example, "When calculating Pass@k, if a model refuses to answer or produces an invalid output for a given question, that response is treated as incorrect (p_i = 0). For metrics like accuracy and F1 score, questions with refused or invalid outputs are excluded from the calculation. We also report the percentage of questions for which the model refused to provide an answer or produced an invalid output for each benchmark." Additionally, provide the percentage of refused/invalid outputs for each benchmark in the results tables.

• Expand Discussion on Limitations of Evaluation

  This low-impact improvement would provide a more balanced and critical assessment of the evaluation results. While the Experiment section presents a comprehensive evaluation, it could benefit from a more explicit discussion of the limitations of the chosen benchmarks and evaluation setup, which is important for contextualizing the findings.

  "Evaluation Prompts Following the setup in DeepSeek-V3, standard benchmarks such as MMLU, DROP, GPQA Diamond, and SimpleQA are evaluated using prompts from the simple-evals framework." (Page 12)

  Implementation: Add a subsection to the Discussion section specifically addressing the limitations of the evaluation. Discuss potential biases in the benchmarks, such as a focus on specific types of reasoning or a lack of diversity in the tasks. Address any limitations of the evaluation setup, such as the use of fixed prompts or the potential for the evaluation metrics to not fully capture all aspects of reasoning ability. For example, "While our evaluation covers a wide range of benchmarks, it is important to acknowledge that each benchmark has its own limitations and may not fully capture the complexities of real-world reasoning. Additionally, our use of fixed prompts for evaluation may not reflect the model's performance when used interactively with varied prompts. Future work should explore more dynamic and interactive evaluation methodologies."

Non-Text Elements

Table 4 | Comparison between DeepSeek-R1 and other representative models.

Figure/Table Image (Page 13)

First Reference in Text

Table 4 | Comparison between DeepSeek-R1 and other representative models.

Description

• Purpose of the table: This table compares the performance of an AI model called DeepSeek-R1 with several other AI models. It's like a scorecard showing how well each model does on a variety of tests.
• Models being compared: The table compares six different AI models: Claude-3.5-Sonnet-1022, GPT-4o-0513, DeepSeek-V3, OpenAI-o1-mini, OpenAI-o1-1217, and DeepSeek-R1. These models are developed by different research groups or companies. For example, "GPT-4o-0513" and "OpenAI" models are from OpenAI, while "Claude" is from Anthropic. DeepSeek-R1 is the main model being studied in this paper.
• Benchmarks and metrics: The models are tested on various benchmarks, which are like standardized tests for AI. Each benchmark has a name, such as "MMLU," "GPQA Diamond," or "Codeforces." These benchmarks assess different capabilities of the AI, like general knowledge, problem-solving, or coding skills. The results are reported using different metrics, such as "Pass@1," "EM" (which likely stands for Exact Match), "F1," "Correct," "Acc." (Accuracy), "LC-winrate," and "Rating." These metrics are different ways of measuring how well the AI performed on each test. For example, "Pass@1" might mean the percentage of questions the AI answered correctly on the first try, while "Rating" could be a score similar to those used in chess.
• Categories of benchmarks: The benchmarks are grouped into categories: English, Code, Math, and Chinese. This helps to organize the results and see how the models perform in different domains. For instance, the "English" category includes tests of language understanding and reasoning in English, while the "Code" category focuses on coding abilities.
• Model architecture and parameters: The table also provides information about the "Architecture," "# Activated Params," and "# Total Params" of the models. "Architecture" refers to the underlying structure or design of the AI model. "MoE" stands for Mixture of Experts. "Params" stands for parameters, which are like the internal settings or variables that the AI learns during training. "Activated Params" might refer to the number of parameters that are actually used during a specific task, while "Total Params" is the total number of parameters the model has. Generally, models with more parameters tend to be more powerful but also require more computational resources.

Scientific Validity

• Representativeness of models: The selection of models for comparison appears to be representative of the state-of-the-art, including models from major players like OpenAI and Anthropic. This allows for a meaningful evaluation of DeepSeek-R1's performance relative to its peers.
• Benchmark selection: The table includes a wide range of benchmarks covering various domains (English, Code, Math, Chinese), which provides a comprehensive assessment of the models' capabilities. However, a more detailed rationale for choosing these specific benchmarks could further strengthen the paper.
• Metric appropriateness: The metrics used are generally appropriate for the respective benchmarks. However, providing more explicit definitions of each metric within the table or in the accompanying text would enhance clarity and reproducibility.
• Statistical significance: The table presents numerical results but lacks information on statistical significance. Reporting confidence intervals or p-values would provide a more rigorous evaluation of the observed differences in performance.
• Model details: The inclusion of model architecture and parameter details is valuable for understanding the scale and complexity of the models being compared. However, further clarification of "Activated Params" would be beneficial.

Communication

• Table organization: The table is well-organized, with clear row and column labels. The grouping of benchmarks by category facilitates easy navigation and comparison.
• Caption clarity: The caption is concise and accurately describes the table's content. However, it could be more informative by briefly highlighting the significance of the comparison or the main findings.
• Metric abbreviations: While some metric abbreviations are relatively common (e.g., Acc., EM), others are less clear (e.g., LC-winrate). Providing a key or expanding the abbreviations within the table would improve readability.
• Benchmark details: While the benchmark names are provided, a brief description of each benchmark's focus or the type of task it involves would enhance the table's understandability, especially for readers unfamiliar with these specific benchmarks.
• Visual appeal: The table is visually clean and uncluttered. However, the use of boldface or shading to highlight key results (e.g., the best performance on each benchmark) could further improve its visual impact.