Superhuman performance of a large language model on the reasoning tasks of a physician

Abstract

Key Aspects

• Context of LLM evaluation in medical tasks: The abstract introduces the study's focus on evaluating the performance of large language models (LLMs) in medical reasoning tasks. Traditionally, LLMs in medicine have been evaluated using multiple-choice question benchmarks. These benchmarks, while common, are often limited because they are highly constrained and don't fully capture the complexity of real-world clinical scenarios. Think of it like training a chess-playing AI solely on simple puzzles; it might excel at those, but not necessarily at a full game. The abstract points out that clinical reasoning, which involves critical thinking to gather and synthesize clinical data for diagnosis and management, is a more appropriate benchmark. This is like testing the chess AI in a real game to see if it can truly strategize and adapt. The authors are thus moving away from simple tests to more complex evaluations.
• Study Objective and Experimental Design: The abstract outlines the study's objective: to evaluate OpenAI's 01-preview model, a specific LLM designed to enhance runtime through 'chain of thought' processes. This means the model is intended to 'think through' the problem step-by-step before providing an answer, much like a human expert would. The study characterizes the performance of this model using five different experiments. These experiments cover a range of medical reasoning domains: differential diagnosis generation (coming up with a list of possible diagnoses), displaying diagnostic reasoning (explaining the thought process), triage differential diagnosis (prioritizing diagnoses in urgent cases), probabilistic reasoning (assessing the likelihood of different diagnoses), and management reasoning (deciding on treatment plans). These tasks are adjudicated by physician experts using validated psychometrics, which ensures a rigorous and reliable evaluation. It's like having a panel of expert chess players assess the AI's performance in various strategic scenarios.
• Key Findings and Implications: The abstract highlights the study's key findings. The primary outcome was a comparison of the 01-preview model's performance with previous experiments that included historical human controls and benchmarks of earlier LLMs. The study found significant improvements in differential diagnosis generation and the quality of diagnostic and management reasoning. This indicates that the 01-preview model excels in tasks that require complex critical thinking, such as diagnosis and treatment planning. However, the study also found no improvements in probabilistic reasoning or triage differential diagnosis. This suggests that the model is not uniformly superior across all types of medical reasoning tasks. It's like finding that the chess AI is excellent at attacking and strategizing but not as good at calculating probabilities or prioritizing moves under time pressure. The abstract concludes by noting that new, robust benchmarks are needed for scalable evaluation of LLM capabilities compared to human physicians, along with trials in real clinical settings. This suggests that the current benchmarks are not sufficient to fully capture the complexity of medical decision-making.

Strengths

Suggestions for Improvement

• Include a summary of the study's limitations.

  This high-impact improvement would enhance the abstract's clarity and completeness, making it more informative and impactful. The abstract currently lacks a clear statement of the study's limitations, which is crucial for proper interpretation of the findings. Including limitations in the abstract ensures that readers are aware of potential caveats from the outset. Addressing this would strengthen the paper by providing a balanced perspective on the study's contributions and limitations, which is essential for maintaining scientific rigor and transparency. This would also prevent over-interpretation of the results and encourage a more nuanced understanding of the study's implications. Ultimately, explicitly stating the study's limitations in the abstract would improve the overall quality and credibility of the research.

  "This study highlights 01-preview's ability to perform strongly on tasks that require complex critical thinking such as diagnosis and management" (Page 2)

  Implementation: Add a sentence or two to the end of the abstract summarizing the main limitations of the study. For example, 'Limitations of this study include the focus on internal medicine cases, the potential for verbosity bias, and the lack of human-computer interaction analysis.'

• Specify the exact version of the 01-preview model.

  This medium-impact improvement would enhance the abstract's clarity and contextual understanding by explicitly stating the specific version of the model used. The abstract currently mentions the '01-preview model' but does not provide the exact version, which is important for reproducibility and future comparisons. Adding the specific version will ensure that other researchers can accurately replicate the study and understand the precise model being evaluated. This will strengthen the paper by increasing transparency and enabling more precise comparisons with other studies. It also provides essential information for readers to assess the study's findings and implications. Ultimately, stating the exact model version enhances the scientific rigor and reproducibility of the work.

  "We sought to evaluate OpenAl's 01-preview model, a model developed to increase run-time via chain of thought processes prior to generating a response." (Page 2)

  Implementation: Include the specific version of the 01-preview model used in the study. For example, change the sentence to read: 'We sought to evaluate OpenAl's 01-preview model (o1-preview-2024-09-12), a model developed to increase run-time via chain of thought processes prior to generating a response.'

• Explicitly state the primary outcome measure used.

  This medium-impact improvement would enhance the abstract's clarity and completeness by explicitly stating the primary outcome measure. The abstract mentions 'comparison of the o1-preview output' but does not clearly define what metric was used for this comparison. Stating the primary outcome measure will ensure that readers immediately understand how the model's performance was evaluated. This will strengthen the paper by making the abstract more informative and accessible, allowing readers to quickly grasp the study's main findings. It also provides essential context for interpreting the results and assessing the study's implications. Ultimately, specifying the primary outcome measure will improve the overall quality and clarity of the abstract.

  "Our primary outcome was comparison of the o1-preview output to identical prior experiments that have historical human controls and benchmarks of previous LLMs." (Page 2)

  Implementation: Add a phrase or sentence that explicitly states the primary outcome measure. For example, change the sentence to read: 'Our primary outcome was comparison of the o1-preview output to identical prior experiments that have historical human controls and benchmarks of previous LLMs, measured by the percentage of correct diagnoses and management decisions.'

Introduction

Key Aspects

• Context of AI in Medical Diagnostics: The introduction begins by establishing the context of artificial intelligence (AI) in medicine, specifically focusing on diagnostic support tools. These tools have evolved from early computational methods like regression modeling and rule-based systems to more advanced natural language processing (NLP) techniques. Think of it like the evolution of calculators: starting from simple abacuses, then moving to mechanical calculators, and finally to sophisticated electronic calculators. The introduction highlights that recent advances in large language models (LLMs) have surpassed the performance of these earlier methods, similar to how modern computers outperform older calculators. This sets the stage for the study's focus on evaluating the capabilities of a specific LLM in medical reasoning.
• Limitations of Multiple-Choice Benchmarks: The introduction then discusses the limitations of traditional multiple-choice question benchmarks for evaluating LLMs in medicine. These benchmarks, while commonly used, are often highly constrained and do not accurately reflect the complexity of real-world clinical scenarios. It's like assessing a chef's skills by only asking them to identify ingredients in a multiple-choice quiz, rather than having them prepare a complex dish. The introduction argues that clinical reasoning, which involves critical thinking to gather and synthesize clinical data for diagnosis and management, is a more appropriate benchmark for evaluating LLMs. This is like testing the chef by having them create a meal from scratch, requiring them to use their skills and knowledge in a realistic setting. The authors are thus making a case for moving beyond simple tests to more complex evaluations.
• Introduction of the 01-preview Model: The introduction highlights the recent release of OpenAI's 01-preview model, which is designed to enhance runtime through 'chain of thought' (CoT) processes. CoT prompting is a technique that encourages the model to break down complex problems into a series of intermediate steps, similar to how a human expert would approach a problem. This is like a chess player explaining their thought process step-by-step before making a move, rather than simply providing the final move. The introduction also notes that this model has demonstrated superior ability in solving complex problems across various domains, including informatics, mathematics, engineering, and medical question-answering. This sets the stage for the study's evaluation of the 01-preview model in a more comprehensive clinical setting.
• Importance of Multi-Step Reasoning: The introduction emphasizes the importance of multi-step reasoning in medical tasks, which involves constant adjustments based on new data, iterative refinement of diagnoses and management plans, and consequential treatment decisions under uncertainty. This type of reasoning is crucial in clinical practice and is not well-captured by simple multiple-choice questions. It's like navigating a complex maze, where each step requires careful consideration of the current situation and potential future paths. The introduction also notes that prior LLMs have shown promise in outperforming clinicians in routine and complex diagnostic scenarios, which further justifies the need for rigorous evaluation of these models. This provides the rationale for the study's focus on evaluating the 01-preview model's abilities across several medical reasoning domains.
• Study Approach and Evaluation Metrics: The introduction concludes by outlining the study's approach: a series of experiments to evaluate the 01-preview system's abilities across several medical reasoning domains. These domains include differential diagnosis generation (creating a list of possible diagnoses), presentation of reasoning (explaining the thought process), probabilistic reasoning (assessing the likelihood of different diagnoses), and management reasoning (deciding on treatment plans). The introduction also mentions that physician experts assessed the quality of LLM outputs using validated psychometrics, which ensures a rigorous and reliable evaluation. This is like having a panel of expert judges evaluate the chef's dishes using a standardized scoring system. The introduction also notes that the study compared the performance of 01-preview to the responses of hundreds of physicians and prior LLMs, providing a clear context for the study's findings.

Strengths

Suggestions for Improvement

• Explicitly state the study's research question.

  This high-impact improvement would enhance the introduction's clarity and contextual understanding by explicitly stating the study's main research question. The introduction currently provides background on the limitations of multiple-choice benchmarks and the promise of LLMs in clinical reasoning, but it does not explicitly articulate the central question the study aims to answer. Adding a clear research question would provide a focal point for the reader and set the stage for the study's methodology and findings. This is essential for the introduction as it provides the reader with a clear understanding of what the study is trying to achieve. A well-defined research question ensures that the study's purpose is immediately apparent, guiding the reader's understanding of the subsequent sections. Ultimately, stating the research question would significantly improve the introduction's clarity and purpose, making it more effective in setting up the study.

  "Given the importance of multi-step reasoning in medical tasks, we performed a series of experiments to evaluate the 01-preview system's abilities" (Page 2)

  Implementation: Add a sentence or two at the end of the introduction that clearly states the research question. For example, 'Therefore, this study aims to evaluate the performance of the 01-preview model across five medical reasoning domains, comparing its performance to human physicians and prior LLMs.'

• Briefly mention the study's limitations.

  This medium-impact improvement would enhance the introduction's completeness by briefly mentioning the limitations of the study. The introduction currently focuses on the rationale for the study and the need for better benchmarks, but it does not provide any indication of the study's limitations. Including a brief mention of the limitations would provide a balanced perspective and prepare the reader for a more nuanced interpretation of the results. This is crucial in the introduction to give the reader an idea of the study's scope and the potential caveats to the findings. Addressing this would strengthen the paper by demonstrating the authors' awareness of the study's limitations and promoting a more critical evaluation of the results. Ultimately, briefly mentioning the study's limitations in the introduction would improve the paper's credibility and transparency.

  "In reality, clinical practice requires real-time complex multi-step reasoning" (Page 2)

  Implementation: Add a sentence or two at the end of the introduction that briefly mentions the main limitations of the study. For example, 'While this study provides valuable insights, it is limited by its focus on internal medicine cases and the absence of human-computer interaction analysis.'

• Explicitly connect benchmark limitations to the focus on complex reasoning.

  This medium-impact improvement would enhance the introduction's flow and logical progression by explicitly connecting the study's focus on complex reasoning to the limitations of existing benchmarks. The introduction discusses the limitations of multiple-choice benchmarks and the importance of complex reasoning, but it does not explicitly link these two concepts. By making this connection explicit, the authors would clarify why the study focuses on complex reasoning tasks. This connection is important in the introduction as it establishes the logical flow of the argument and justifies the study's approach. Addressing this would strengthen the paper by making the rationale for the study more transparent and coherent. Ultimately, explicitly connecting the limitations of benchmarks to the focus on complex reasoning would improve the introduction's clarity and logical structure.

  "However, existing multiple choice evaluations do not represent the breadth or complexity of clinical decision-making" (Page 2)

  Implementation: Add a sentence that explicitly connects the limitations of multiple-choice benchmarks to the study's focus on complex reasoning. For example, 'Given that multiple-choice benchmarks do not capture the complexity of clinical decision-making, this study focuses on evaluating the model's performance in complex reasoning tasks.'

Results

Key Aspects

• Differential Diagnosis Generation on NEJM CPCs: The Results section begins by evaluating the o1-preview model's ability to generate differential diagnoses using the New England Journal of Medicine (NEJM) Clinicopathological Conferences (CPCs). These CPCs are like medical mystery cases, where doctors try to figure out the correct diagnosis based on patient information. The model's performance is assessed by comparing its generated list of possible diagnoses with the actual diagnosis. Two physicians evaluated the quality of the model's differential diagnoses, and their agreement was measured using Cohen's kappa, a statistical measure that quantifies inter-rater reliability. A kappa of 0.66 indicates substantial agreement, suggesting the model's diagnoses were consistently evaluated by the physicians. The model included the correct diagnosis in 78.3% of cases, with a 95% confidence interval of 70.7% to 84.8%. This means that if we were to repeat this experiment many times, we'd expect the model to include the correct diagnosis in 70.7% to 84.8% of cases 95% of the time. The first diagnosis suggested by the model was correct in 52% of cases, indicating that while the model often includes the correct diagnosis, it's not always the top suggestion. This is like a detective identifying the correct suspect but not always putting them first on the list.
• Diagnostic Test Selection on NEJM CPCs: The Results section further examines the o1-preview model's ability to select the next diagnostic test in the NEJM CPCs. This is like evaluating a doctor's ability to choose the right tests to confirm a diagnosis. Two physicians scored the model's suggested test plan, and their agreement was measured using Cohen's kappa. Although the proportion of agreements was high (86%), the kappa was low (0.28) due to severe class imbalance, which means that the majority of the responses fell into one category, making it difficult to measure agreement. In 87.5% of cases, the model selected the correct test to order. In 11% of cases, the chosen testing plan was judged to be helpful, and in 1.5% of cases, it was deemed unhelpful. This shows that the model is generally good at choosing the right tests, but there is room for improvement. It's like a chess AI being good at choosing the right move most of the time, but occasionally making a less optimal choice.
• Presentation of Reasoning in NEJM Healer Cases: The Results section then assesses the o1-preview model's presentation of reasoning using 20 clinical medical cases from the NEJM Healer curriculum. These cases are virtual patient encounters designed to assess clinical reasoning. The quality of the model's reasoning was evaluated using the Revised-IDEA (R-IDEA) score, a validated 10-point scale that measures four core domains of documenting clinical reasoning. Two physicians scored the model's reasoning, and their agreement was high (99%), with a kappa of 0.89, indicating very strong agreement. The model achieved a perfect R-IDEA score in 78 out of 80 cases, which is a remarkable performance. This suggests that the model is not only good at making diagnoses but also at explaining its reasoning, similar to a skilled debater who can clearly articulate their arguments. This is a key aspect of clinical practice, where explaining the thought process is crucial for patient care and communication with other physicians.
• Management Reasoning in Grey Matters Cases: The Results section also evaluates the o1-preview model's performance on 'Grey Matters Management Cases,' which are based on real clinical scenarios. The model's responses were scored by two physicians, with substantial agreement (kappa = 0.71). The median score for the model was 86%, compared to 42% for GPT-4, 41% for physicians with access to GPT-4, and 34% for physicians with conventional resources. Using a mixed-effects model, the o1-preview model scored significantly higher than all other groups. This demonstrates that the model excels in tasks requiring complex critical thinking, such as deciding on the next steps in patient management. It's like a project manager who can not only identify the problems but also create an effective plan to solve them. This is a crucial aspect of medical practice, where effective management of patients is essential for good outcomes.
• Diagnostic Reasoning in Landmark Cases: The Results section further examines the o1-preview model's performance on 'Landmark Diagnostic Cases,' which are based on a landmark study of computer-based diagnostic systems. The model's responses were scored by two physicians, with moderate agreement (kappa = 0.42). The median score for the model was 97%, compared to 92% for GPT-4, 76% for physicians with access to GPT-4, and 74% for physicians with conventional resources. Using a mixed-effects model, the model performed comparably to GPT-4 and better than physicians. This shows that the model can perform well in complex diagnostic scenarios. This is similar to a chess AI being able to perform at a high level, but not significantly better than other top AI players. It's important to note that these cases were not publicly released to prevent memorization, which is crucial for maintaining the validity of the evaluation.
• Probabilistic Reasoning in Diagnostic Cases: Finally, the Results section evaluates the o1-preview model's performance on 'Diagnostic Probabilistic Reasoning Cases,' which involve estimating the probability of a disease before and after a test. The model performs similarly to GPT-4 in estimating pre-test and post-test probabilities. The exception is the stress test for coronary artery disease, where the model's density is closer to the reference range than both models and humans. This indicates that the model's performance on probabilistic reasoning tasks is not superior to previous models, and there is room for improvement. It's like a weather forecasting AI being good at predicting rain, but not better than other existing models. This is an important area for future research, as probabilistic reasoning is a key component of clinical decision-making.

Strengths

• Use of robust statistical measures

  The Results section effectively uses statistical measures like Cohen's kappa, confidence intervals, and p-values to quantify the agreement between physician raters and to compare the performance of the o1-preview model with other models and human controls. This rigorous approach provides a strong foundation for the study's findings and enhances the credibility of the results.

  "There was substantial agreement between the two physicians evaluating the quality of o1-preview’s differential diagnosis (agreement on 120/143 cases [84%], κ=0.66)." (Page 3)
• Clear presentation of results with figures and tables

  The Results section presents data clearly through figures and tables, which aids in understanding the findings. The figures visually compare the model's performance with other benchmarks, and the tables provide detailed examples of the model's outputs. This enhances the accessibility of the results and allows readers to easily interpret the data.

  "o1-preview included the correct diagnosis in its differential in 78.3% of cases (95% CI, 70.7% to 84.8%) (Figure 1)." (Page 3)
• Comprehensive evaluation across multiple reasoning tasks

  The Results section includes a variety of clinical reasoning tasks, including differential diagnosis generation, diagnostic test selection, presentation of reasoning, and probabilistic reasoning. This broad approach provides a comprehensive assessment of the model's capabilities across different aspects of medical decision-making, increasing the generalizability of the findings.

  "We next evaluated the ability of o1-preview to select the next diagnostic test in the NEJM CPCs." (Page 3)

Suggestions for Improvement

• Explicitly state metrics for each experiment

  This medium-impact improvement would enhance the clarity and completeness of the Results section by explicitly stating the specific metrics used for each experiment. While the section presents various results, it does not always clearly define the exact metrics used for each comparison. For example, while the section mentions 'agreement' for differential diagnoses, it does not specify how 'agreement' was measured for the 'Grey Matters Management Cases'. Explicitly stating the metrics would ensure that readers understand how the model's performance was evaluated in each experiment. This is crucial for the Results section as it provides the necessary context for interpreting the findings and enhances the reproducibility of the study. This would strengthen the paper by increasing transparency and enabling more precise comparisons with other studies. Ultimately, explicitly stating the metrics used for each experiment would improve the overall quality and clarity of the Results section.

  "The median score for the o1-preview per case was 86% (IQR, 82%-87%) (Figure 5A)" (Page 4)

  Implementation: For each experiment, include a sentence that clearly states the metric used for evaluation. For example, when presenting the results for the 'Grey Matters Management Cases', add a phrase like 'The primary outcome was the percentage of total points obtained by o1-preview, measured using a 100-point scale'.

• Mention specific limitations for each experiment

  This medium-impact improvement would enhance the Results section by explicitly mentioning any limitations associated with each experiment. While the Discussion section addresses general limitations, the Results section would benefit from brief, specific limitations for each experiment. For example, the kappa value for diagnostic test selection was low, which should be noted in the results. Adding these specific limitations would provide a balanced perspective on the study's findings and prevent over-interpretation of the results. This is important in the Results section to give the reader an idea of the scope and potential caveats of the findings. Addressing this would strengthen the paper by demonstrating the authors' awareness of the limitations and promoting a more critical evaluation of the results. Ultimately, briefly mentioning the limitations within the Results section would improve the paper's credibility and transparency.

  "The proportion of agreements was high, but the kappa was low due to severe class imbalance." (Page 3)

  Implementation: For each experiment, include a sentence or phrase that briefly mentions any specific limitations. For example, when presenting the results for the diagnostic test selection, add a phrase like 'While the proportion of agreements was high, the kappa was low due to severe class imbalance'.

• Provide clinical context for the findings

  This medium-impact improvement would enhance the Results section by providing more context for the clinical significance of the findings. While the section presents the model's performance on various tasks, it does not always explain the clinical relevance of these results. For example, the section mentions that the model achieved a perfect R-IDEA score in 78/80 cases, but it does not discuss the implications of this result for clinical practice. Adding this context would help readers understand the practical importance of the study's findings. This is crucial in the Results section as it bridges the gap between statistical results and real-world applications. Addressing this would strengthen the paper by making the results more meaningful and relevant to the clinical audience. Ultimately, adding clinical context to the findings would improve the overall impact and practical value of the study.

  "For 78/80 of the cases, o1-preview achieved a perfect R-IDEA score." (Page 3)

  Implementation: For each experiment, include a sentence or two that discusses the clinical significance of the results. For example, after presenting the R-IDEA scores, add a phrase like 'This suggests that the model is highly capable of documenting clinical reasoning, which is a crucial aspect of medical practice'.

Non-Text Elements

Discussion

Key Aspects

• Summary of Study Objective and Main Findings: The Discussion section opens by summarizing the study's main objective: evaluating the o1-preview model's medical reasoning abilities across five different experiments. The experiments covered a range of tasks, including differential diagnosis generation, presentation of reasoning, probabilistic reasoning, and management reasoning. The authors compared the model's performance to historical controls, including human baselines and the GPT-4 model. This is like a coach reviewing a player's performance across different drills to see where they excel and where they need improvement. The key finding is that the o1-preview model showed significant gains in performance for most tasks, similar to what has been observed in non-medical studies. This sets the stage for a deeper dive into the specific strengths and weaknesses of the model.
• Specific Strengths and Weaknesses of the Model: The Discussion section then elaborates on the specific areas where the o1-preview model demonstrated superior performance. It highlights that the model surpassed both GPT-4 and previous non-LLM differential generators in differential diagnosis generation, as well as the human baseline. The model also showed similar gains in the display of reasoning and management reasoning compared to prior studies. This is like a chef not only making a delicious dish but also explaining the steps and techniques involved, showcasing both skill and understanding. However, the authors also note that there were no improvements in probabilistic reasoning or critical diagnosis identification over GPT-4, although probabilistic reasoning was still superior to the human baseline. This nuanced finding suggests that while the model excels in tasks requiring critical thinking, it struggles with tasks involving abstraction, similar to a chess AI being great at strategy but not at calculating probabilities.
• Implications for Clinical Practice and Research: The Discussion section then transitions to discussing the implications of these findings for the science and practice of clinical medicine. The authors point out that the rapid improvement in LLMs has major implications for the field. The study demonstrates consistent and superhuman performance on many human-adjudicated medical reasoning tasks. This is like a new technology that is not only faster but also more accurate than previous methods. The authors also address the concerns about applying AI in clinical decision support, noting that while it may be viewed as high-risk, it also has the potential to mitigate the human and financial costs of diagnostic error and delay. This is like a safety feature in a car that, while complex, can prevent accidents and save lives. This section underscores the importance of further research and trials to safely integrate AI into clinical practice.
• Need for New Benchmarks and Evaluation Strategies: The Discussion section further emphasizes the need for new benchmarks to assess how AI models should be integrated into clinical reasoning workflows. The authors argue that multiple-choice question benchmarks are not realistic proxies for high-stakes medical reasoning. The NEJM CPCs, used in this study, are more challenging, and the o1-preview model is able to produce high-quality differentials in almost 90% of cases. This is like moving from simple puzzles to real-world scenarios to truly test the AI's capabilities. The authors also note that current metrics for clinical management reasoning are laborious and not scalable, indicating a need for additional challenging and realistic evaluations, including adding modalities and enriching existing benchmarks. This is like a call for better testing methods to accurately assess the performance of AI in complex medical tasks.
• Limitations of the Study: The Discussion section acknowledges several limitations of the study. First, the o1-preview model tends towards verbosity, which may have led to higher scoring in some experiments. Second, the study only reflects model performance and does not include human-computer interaction analysis, which is crucial for developing clinical decision support tools. Third, the study examined only five aspects of clinical reasoning, while there are many other tasks that could be studied. Fourth, the study focused on internal medicine cases, which is not representative of broader medical practice. This is like a scientist acknowledging the limitations of their experiment and suggesting ways to improve future studies. These limitations highlight the need for further research to address these gaps and ensure the safe and effective implementation of AI in clinical settings.
• Conclusion and Call for Future Action: The Discussion section concludes by reiterating the o1-preview model's superhuman performance in differential diagnosis, diagnostic clinical reasoning, and management reasoning, surpassing prior models and human physicians. The authors emphasize the urgent need for better and more meaningful evaluation strategies, given the rapid pace of improvement in automated systems. They also suggest that the model's performance on challenging diagnostic problems indicates opportunities to support clinicians in real-world settings. The section ends with a call for clinical trials and workforce retraining with integrated AI systems to confirm the potential of such systems to boost clinical practice and patient outcomes. This is like a final summary of the key takeaways and a call to action for the future of AI in medicine.

Strengths

• Clear summary of key findings

  The Discussion section effectively summarizes the key findings, highlighting the superior performance of the o1-preview model in differential diagnosis, diagnostic reasoning, and management reasoning. This clear summary provides a strong foundation for the subsequent discussion of implications and limitations.

  "We evaluated the medical reasoning abilities of the 01-preview model across five diverse experiments" (Page 5)
• Connection to clinical practice

  The Discussion section appropriately connects the study's findings to broader implications for clinical practice, such as the potential for AI to mitigate diagnostic errors and delays. This demonstrates the practical relevance of the research and its potential impact on healthcare.

  "This rapid pace of improvement in LLMs has major implications for science and practice of clinical medicine." (Page 5)
• Acknowledgment of limitations

  The Discussion section acknowledges the limitations of the study, including the model's verbosity, the lack of human-computer interaction analysis, and the focus on internal medicine cases. This demonstrates a balanced and critical perspective on the research.

  "Our study has several limitations. First, 01-preview tends towards verbosity" (Page 5)

Suggestions for Improvement

• Explicitly state the main conclusions

  This high-impact improvement would enhance the Discussion section by explicitly stating the main conclusions of the study in a concise manner. While the section discusses various implications and limitations, it does not clearly articulate the core conclusions that the authors want the reader to take away. Adding a concluding paragraph that summarizes the main findings and their significance would provide a strong closing statement. This is crucial for the Discussion section as it reinforces the study's key contributions and provides a clear sense of closure. This would strengthen the paper by ensuring that the main messages are clearly conveyed and that the reader is left with a clear understanding of the study's overall impact. Ultimately, explicitly stating the main conclusions would significantly improve the Discussion section's effectiveness.

  "In conclusion, o1-preview demonstrates superhuman performance in differential diagnosis, diagnostic clinical reasoning, and management reasoning" (Page 6)

  Implementation: Add a concluding paragraph that explicitly states the main conclusions of the study. For example, 'In conclusion, this study demonstrates that the o1-preview model exhibits superior performance in complex medical reasoning tasks, particularly in differential diagnosis and management. These findings suggest the potential for AI to improve clinical practice, but further research is needed to address limitations and ensure safe implementation.'

• Discuss the lack of improvement in probabilistic reasoning

  This medium-impact improvement would enhance the Discussion section by explicitly addressing the lack of improvement in probabilistic reasoning. While the section notes that no improvements were observed in probabilistic reasoning, it does not fully explore the potential reasons for this finding. Discussing why the model did not improve in this area would provide a more comprehensive analysis of the results. This is important in the Discussion section as it provides a more nuanced understanding of the model's capabilities and limitations. Addressing this would strengthen the paper by demonstrating a thorough exploration of all the findings and encouraging future research in this area. Ultimately, explicitly discussing the lack of improvement in probabilistic reasoning would improve the overall quality and depth of the Discussion section.

  "We did not see improvements in either probabilistic reasoning or critical diagnosis identification over GPT-4" (Page 5)

  Implementation: Add a paragraph that explores the potential reasons for the lack of improvement in probabilistic reasoning. For example, 'The lack of improvement in probabilistic reasoning tasks may be due to the model's difficulty in handling abstract concepts or the limitations of the current probabilistic reasoning benchmarks. Further research is needed to investigate these factors and develop more robust evaluation methods.'

• Provide specific recommendations for future research

  This medium-impact improvement would enhance the Discussion section by providing more specific recommendations for future research. While the section mentions the need for new benchmarks and trials in real-world settings, it does not provide specific directions for future studies. Suggesting concrete research avenues would make the discussion more actionable and impactful. This is important in the Discussion section as it provides a clear path for future research and encourages the scientific community to build upon the current findings. Addressing this would strengthen the paper by demonstrating a forward-thinking approach and highlighting the potential for further advancements in the field. Ultimately, providing more specific recommendations for future research would improve the overall value and impact of the Discussion section.

  "These findings suggest the need for trials to evaluate these technologies in real-world patient care settings" (Page 5)

  Implementation: Add a paragraph that provides specific recommendations for future research. For example, 'Future research should focus on developing new benchmarks that incorporate human-computer interaction, exploring the model's performance in diverse clinical settings, and investigating methods to improve its probabilistic reasoning capabilities. Additionally, clinical trials are needed to evaluate the impact of these models on patient outcomes.'

Methods

Key Aspects

• Model Identification and Access: The Methods section begins by identifying the specific large language model (LLM) used in the study: the o1-preview model, accessed via OpenAI's Application Programming Interface (API). This is akin to specifying the exact type of microscope used in a biology experiment. The specific version of the model, 'o1-preview-2024-09-12', is crucial for reproducibility, much like noting the precise settings on the microscope. This level of detail ensures that other researchers can replicate the study and verify the findings. Without this information, the study's results would be difficult to interpret and build upon.
• NEJM CPC Case Selection: The Methods section then details the selection process for the New England Journal of Medicine (NEJM) Clinicopathological Conference (CPC) cases. The authors chose all 143 diagnostic cases from 2021 to September 2024 that included a 'Differential Diagnosis' section. This is like selecting a specific set of test tubes from a larger collection, ensuring that each test tube meets specific criteria. The section also notes that 70 of these cases, published between 2021 and 2022, were previously evaluated in a prior study of GPT-4. This information is essential for understanding the dataset used in the study and for contextualizing the results in comparison to previous research. It is akin to noting the specific batch number of chemicals used in an experiment, which is crucial for interpreting the results.
• Prompting Strategy for NEJM CPCs: For the NEJM CPC cases, the Methods section explains how the model was prompted. The authors adapted the prompt from a prior study of GPT-4 for differential diagnosis prediction. This is like using a specific recipe in a cooking experiment, where slight variations in the recipe can lead to different results. After the initial differential diagnosis, the model was queried with "What diagnostic tests would you order next given this differential?" This two-step prompting approach is important because it allows the model to first generate a list of possible diagnoses and then decide on the next steps, mimicking the real-world clinical reasoning process. This is like asking a chef to first plan a menu and then decide which ingredients to buy, reflecting a step-by-step decision-making process.
• Primary Outcomes and Scoring for NEJM CPCs: The Methods section clearly defines the primary outcomes for the NEJM CPC cases: differential diagnosis quality and the quality of the suggested testing plan. This is like defining the specific measurements that will be taken in an experiment, such as the volume of liquid or the temperature of a solution. The quality of the differential diagnoses was rated independently by two attending internal medicine physicians using the Bond Score, a previously developed scoring system. Bond Scores range from zero to five, where five represents a differential list that contains the exact target diagnosis, and zero represents a list with no suggestions close to the target diagnosis. This is like using a standardized ruler to measure the length of an object, ensuring consistency and reliability. The quality of the testing plan was scored using a Likert scale from zero to two, comparing the suggested plan to the actual diagnostics performed in the case. This is like using a calibrated thermometer to measure temperature, ensuring that the measurements are accurate and consistent.
• Interrater Agreement and Statistical Analysis for NEJM CPCs: The Methods section describes the statistical analysis used to assess interrater agreement for the NEJM CPC cases. A linear-weighted Cohen's kappa was computed to assess agreement between the two physicians rating the differential diagnoses and diagnostic test selections. Cohen's kappa is a statistical measure that quantifies the level of agreement between two raters, accounting for the possibility of agreement occurring by chance. The linear weighting is used to give more credit for agreement on adjacent categories (e.g. a score of 4 vs 5) than on categories that are further apart (e.g. 0 vs 5). This is like using a specialized statistical tool to measure the consistency of two different people's measurements, ensuring that the results are not due to random chance. Discordant scores were reconciled through discussion, which is like resolving disagreements in measurements by re-examining the data, ensuring that the final scores are as accurate as possible.
• Addressing Memorization and Statistical Comparisons: The Methods section addresses the potential for memorization, given that the o1-preview model has a pretraining end date of October 2023. To assess this, the authors conducted a sensitivity analysis, examining the model's performance before and after this cutoff date. This is like performing a control experiment to rule out potential confounding factors. The performance of other LLMs and differential diagnosis generators on NEJM CPCs from previous studies are included for comparison, which is like comparing results to established benchmarks. The comparison of o1-preview to a historical control of GPT-4 was performed using a McNemar's test, which is a statistical test used to compare paired proportions. This is like using a specific statistical tool to compare two different methods, ensuring that the results are statistically valid.
• NEJM Healer Diagnostic Cases and R-IDEA Scoring: The Methods section details the use of 20 NEJM Healer cases, separated into four sections representing sequential stages of clinical data acquisition. This is like breaking down a complex problem into smaller, more manageable parts. The primary outcome for these cases was the quality of clinical reasoning documentation, measured by the R-IDEA score. The R-IDEA is a validated 10-point scale for evaluating four core domains of documenting clinical reasoning. This is like using a standardized rubric to assess the quality of a written report, ensuring that the evaluation is consistent and reliable. Two attending internal medicine physicians rated the cases, and a linear-weighted Cohen's kappa was used to assess interrater agreement. Discordant scores were reconciled by a third internal medicine physician, which is like having a third judge resolve disagreements, ensuring the reliability of the results.
• Identification of "Cannot-Miss" Diagnoses: The Methods section explains the secondary outcome for the NEJM Healer cases: the identification of "cannot-miss" diagnoses. These are diagnoses that, if missed, could have severe consequences for the patient. The authors used a list of "cannot-miss" diagnoses defined in a prior study, focusing on the initial triage presentation data. This is like prioritizing critical tasks in a complex project, ensuring that the most important aspects are addressed first. The proportion of "cannot-miss" diagnoses included by each model was compared using pairwise t-tests with Holm-Bonferroni correction, a statistical method used to account for multiple comparisons. This is like using a specific statistical tool to compare different methods, ensuring that the results are statistically valid and not due to chance.
• Grey Matters Management Cases Methodology: The Methods section describes the methodology for the 'Grey Matters Management Cases,' which involved five clinical vignettes. The cases were presented to the o1-preview model, and the responses were graded by two attending internal medicine physicians using rubrics generated by a combination of generalists and subspecialists. This is like using a panel of experts to evaluate a complex task, ensuring that the evaluation is comprehensive and accurate. The scoring of all rubrics was normalized on a 100-point scale. A linear-weighted Cohen's kappa was computed to assess interrater agreement, and discordant scores were reconciled by a third internal medicine physician. The primary outcome was the percentage of total points obtained by the model for each case. This is like using a standardized scoring system to measure performance, ensuring consistency and comparability.
• Statistical Analysis for Grey Matters Cases: The Methods section explains that the same methodology used in prior studies evaluating GPT-4 for management and diagnostic reasoning was used for the 'Grey Matters Management Cases.' A linear mixed-effects model was used to compare the total percentage points scored by o1-preview to historical controls. This is like using a consistent experimental setup across different studies, ensuring that the results are comparable. The group was used as the fixed effect, and random intercepts were included for the case number and the interaction of trial number and individual. This is like controlling for various factors that might influence the results, ensuring that the findings are accurate and reliable. P-values were computed for each fixed effect using Satterthwaite's approximation for degrees of freedom, a statistical method used to account for the variability in the data. This is like using a specific statistical tool to analyze data, ensuring that the results are statistically valid.
• Landmark Diagnostic Cases Methodology: The Methods section outlines the methodology for the 'Landmark Diagnostic Cases,' which involved six clinical vignettes adapted from a landmark study. The model was asked to respond with the top three differential diagnoses, three factors that favor or oppose each diagnosis, the final most likely diagnosis, and three next diagnostic steps. This is like asking a detective to not only identify the suspects but also to explain the evidence for and against each suspect, and to propose the next steps in the investigation. Each case was scored across four categories: initial diagnoses, supporting factors, opposing factors, and final diagnosis, as well as the next steps. The scoring rubrics were normalized on a 100-point scale. Two internal medicine attending physicians graded each of the responses, and a linear-weighted Cohen's kappa was computed to assess interrater agreement. Scoring discrepancies were reconciled by a third internal medicine physician. The primary outcome was the percentage of total points obtained by o1-preview for each of the six cases. This is like using a detailed scoring rubric to evaluate a complex task, ensuring that the evaluation is thorough and consistent.
• Statistical Analysis for Landmark Cases: The Methods section explains that the same statistical method used for the 'Grey Matters Management Cases' was used for the 'Landmark Diagnostic Cases.' This is like using the same analysis method to compare different sets of data, ensuring that the results are consistent and comparable. The total percentage points scored by o1-preview were compared to historical controls using a linear mixed effects model. This is like using a specific statistical tool to compare different groups, ensuring that the results are statistically valid. The analysis was performed in R version 4.4.2, which is like noting the specific software used to perform the analysis, ensuring that the results are reproducible.
• Diagnostic Probabilistic Reasoning Cases Methodology: The Methods section describes the methodology for the 'Diagnostic Probabilistic Reasoning Cases,' which involved posing the same questions from a prior study on primary care topics. The o1-preview model was given case vignettes and asked to estimate the probability of a disease before and after a test. This is like asking a weather forecasting model to predict the probability of rain before and after receiving new data, testing its ability to update its predictions based on new information. One hundred model outputs were generated for each of the five cases. The previous study used a default temperature of 1 for GPT-4, which is a parameter that modulates the diversity of the output. The same default temperature of 1 was used for o1-preview. This is like using the same settings on a machine, ensuring that the results are comparable.
• Statistical Analysis for Probabilistic Reasoning Cases: The Methods section explains that Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) were computed to compare the model's predictions to reference probabilities collected from a previous literature review. MAE is a measure of the average magnitude of errors in a set of predictions, and MAPE is a measure of the average percentage error. This is like using specific statistical tools to measure the accuracy of a model's predictions, ensuring that the results are quantifiable and comparable. The analysis was performed in R version 4.4.2, which is like noting the specific software used to perform the analysis, ensuring that the results are reproducible.

Strengths

• Clear identification of the model and access method

  The Methods section clearly identifies the specific model used (o1-preview-2024-09-12) and how it was accessed through OpenAI's API. This is crucial for reproducibility and allows other researchers to understand exactly which model was evaluated.

  "The 01-preview model (“o1-preview-2024-09-12”) was accessed through OpenAl's Application Programming Interface (API)." (Page 6)
• Detailed description of case selection process

  The Methods section provides a detailed description of the case selection process for the NEJM CPCs, including the specific years of publication and the criteria for inclusion (cases with a "Differential Diagnosis" section). This level of detail is important for understanding the dataset used in the study.

  "We selected all 143 diagnostic cases from 2021 to September 2024 (cases including the section “Differential Diagnosis”)." (Page 6)
• Clear definition of primary outcomes

  The Methods section clearly defines the primary outcomes for each experiment, including differential diagnosis quality, quality of the suggested testing plan, and the R-IDEA score for clinical reasoning documentation. This helps to clarify what the study was measuring and how the model's performance was evaluated.

  "Our primary outcomes were differential diagnosis quality and the quality of the suggested testing plan." (Page 6)
• Thorough description of scoring systems and interrater reliability

  The Methods section thoroughly describes the scoring systems used, such as the Bond Score for differential diagnoses and the Likert scale for testing plans. It also specifies the use of linear-weighted Cohen's kappa for interrater agreement and how discordant scores were resolved. This level of detail enhances the rigor and transparency of the study.

  "Differential diagnoses were rated independently by two attending internal medicine physicians (Z.K., A.R), using a previously-developed scoring system called the Bond Score." (Page 6)

Suggestions for Improvement

• Include specific prompts used for each experiment

  This high-impact improvement would enhance the Methods section by explicitly stating the specific prompts used for each experiment. While the section mentions adapting prompts from previous studies, it does not provide the exact wording of these prompts, which is essential for reproducibility. Including the prompts would allow other researchers to replicate the study precisely and understand the context in which the model was evaluated. This is crucial for the Methods section as it details the core experimental procedures. This would strengthen the paper by increasing transparency and enabling more precise comparisons with other studies. Ultimately, providing the specific prompts would significantly improve the study's scientific contribution by ensuring its findings can be properly contextualized and replicated.

  "For differential diagnosis prediction, we adapted the prompt from the prior study of GPT-4 (Supplement 1A)." (Page 6)

  Implementation: Include the exact prompts used for each experiment, either in the main text or in the supplementary materials. For example, for the NEJM CPCs, include the exact prompt used to query the model after the differential diagnosis prediction ('What diagnostic tests would you order next given this differential?').

• Explain rationale for using linear-weighted Cohen's kappa

  This medium-impact improvement would enhance the Methods section by explicitly stating the rationale for using a linear-weighted Cohen's kappa instead of a standard Cohen's kappa. While the section mentions using a linear-weighted kappa, it does not explain why this specific measure was chosen. Providing this rationale would clarify the methodological choices and ensure that readers understand the statistical approach. This is important in the Methods section as it provides context for the statistical analyses. This would strengthen the paper by demonstrating the authors' awareness of the nuances of interrater reliability measures and promoting a more critical evaluation of the results. Ultimately, explaining the rationale for using a linear-weighted kappa would improve the overall quality and transparency of the Methods section.

  "For both the differential diagnosis and diagnostic test selections, a linear-weighted Cohen's kappa was computed to assess interrater agreement" (Page 7)

  Implementation: Add a sentence or two explaining why a linear-weighted Cohen's kappa was used. For example, 'A linear-weighted Cohen's kappa was used to assess interrater agreement because it accounts for the degree of disagreement, which is important when scoring systems have ordinal categories.'

• Specify criteria for "cannot-miss" diagnoses

  This medium-impact improvement would enhance the Methods section by explicitly stating the criteria used to define "cannot-miss" diagnoses. While the section mentions using a list of "cannot-miss" diagnoses defined in a prior study, it does not specify what these criteria were. Providing these criteria would clarify the study's methodology and ensure that readers understand how these diagnoses were identified. This is crucial for the Methods section as it describes the key elements of the study's design. This would strengthen the paper by increasing transparency and enabling more precise comparisons with other studies. Ultimately, specifying the criteria for "cannot-miss" diagnoses would improve the overall quality and clarity of the Methods section.

  "For each case, using only the initial triage presentation data, we used a list of “cannot-miss” defined in a prior study" (Page 7)

  Implementation: Include the criteria used to define "cannot-miss" diagnoses, either in the main text or in the supplementary materials. For example, 'Cannot-miss diagnoses were defined as conditions that, if missed, could lead to significant patient harm or death.'

• Provide more detail on statistical analysis for probabilistic reasoning

  This medium-impact improvement would enhance the Methods section by providing more detail on the statistical analysis used for the probabilistic reasoning cases. While the section mentions using Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE), it does not fully explain how these metrics were applied in this context. Providing more detail would help readers understand the statistical methods and ensure the reproducibility of the study. This is important in the Methods section as it provides context for the statistical analyses. This would strengthen the paper by demonstrating a thorough understanding of the statistical methods and promoting a more critical evaluation of the results. Ultimately, providing more detail on the statistical analysis for probabilistic reasoning would improve the overall quality and transparency of the Methods section.

  "Mean absolute error (MAE) and mean absolute percentage error were computed to compare predictions to the reference probabilities" (Page 9)

  Implementation: Add a sentence or two explaining how MAE and MAPE were computed in the context of the probabilistic reasoning cases. For example, 'MAE and MAPE were computed by comparing the model's predicted probabilities to the reference probabilities for each case, averaging the absolute errors across all cases'.