GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial

Abstract

Key Aspects

• Primary Research Objective: The primary objective of this study was to determine if the use of a large language model (LLM), specifically GPT-4, could improve physician performance in management reasoning tasks. This is significant because while LLMs have shown promise in diagnostic reasoning, their effectiveness in the more complex domain of management reasoning, which involves treatment decisions, testing strategies, and risk management, was previously unknown. The study aims to address a critical gap in the understanding of LLM capabilities in clinical decision-making.
• Study Design: The study employed a prospective, randomized, controlled trial design. Practicing physicians were randomly assigned to one of two groups: one using GPT-4 in addition to conventional resources, and the other using conventional resources alone. This rigorous methodology allows for a direct comparison of the impact of LLM assistance on physician performance, minimizing bias and strengthening the validity of the findings.
• Clinical Vignettes: The study utilized five expert-developed clinical vignettes based on real, de-identified patient encounters. Information was presented sequentially to mimic the dynamic nature of clinical environments. This approach enhances the ecological validity of the study, making the findings more relevant to real-world clinical practice.
• Outcome Measures: The primary outcome measured was the difference in total score between the two groups, assessed using expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and the time spent by physicians on each case. This comprehensive assessment approach allows for a detailed evaluation of the impact of LLM assistance on various aspects of physician performance.
• Main Findings: The key finding was that physicians using GPT-4 scored significantly higher than those using conventional resources alone, with a mean difference of 6.5% (95% CI = 2.7 to 10.2, P < 0.001). However, LLM users also spent significantly more time per case. Notably, there was no significant difference between the performance of LLM-augmented physicians and the LLM used independently. These findings demonstrate the potential of LLMs to improve physician management reasoning, but also highlight the increased time investment associated with their use.

Strengths

• Clear and Concise Summary

  The abstract clearly states the research question, the study design (randomized controlled trial), the intervention (GPT-4 assistance), the control group (conventional resources), the primary outcome (difference in total score), and the main findings, including statistical significance.

  "This prospective, randomized, controlled trial assessed whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources... Physicians using the LLM scored significantly higher compared to those using conventional resources (mean difference = 6.5%, 95% confidence interval (CI) = 2.7 to 10.2, P < 0.001)." (Page 1)
• Quantitative Data Support

  The abstract provides key statistical data (mean difference, confidence interval, p-value) for the primary outcome, lending quantitative support to the findings.

  "Physicians using the LLM scored significantly higher compared to those using conventional resources (mean difference = 6.5%, 95% confidence interval (CI) = 2.7 to 10.2, P < 0.001)." (Page 1)
• Inclusion of Secondary Outcomes

  The abstract concisely mentions secondary outcomes, including time spent per case, and provides relevant statistical data for these as well.

  "Secondary outcomes included domain-specific scores and time spent per case. ... LLM users spent more time per case (mean difference = 119.3 s, 95% CI = 17.4 to 221.2, P = 0.02)." (Page 1)
• Trial Registration

  The abstract includes the ClinicalTrials.gov registration number, promoting transparency and reproducibility.

  "ClinicalTrials.gov registration: NCT06208423." (Page 1)

Suggestions for Improvement

• Include Participant Demographics

  This high-impact improvement would enhance the abstract's clarity and completeness by providing a brief overview of the participant characteristics. The abstract is the first point of contact for most readers, and quickly conveying the type of physicians involved is crucial for contextualizing the results. It helps readers assess the generalizability and relevance of the findings to their own practice or research.

  "From November 2023 to April 2024, 92 practicing physicians were randomized to use either GPT-4 plus conventional resources or conventional resources alone to answer five expert-developed clinical vignettes in a simulated setting." (Page 1)

  Implementation: Add a sentence summarizing the key demographics of the physician participants, such as their specialties and years of experience. For example: 'Participants were 92 practicing physicians (primarily internal medicine, with an average of 7.6 years of experience) randomized to...'

• Clarify Direction of Time Difference

  This medium-impact improvement would strengthen the abstract by explicitly stating the direction of the effect on time spent per case. The abstract currently mentions a statistically significant difference, but it's not immediately clear whether LLM users spent more or less time. Explicitly stating this enhances clarity and avoids potential misinterpretations. This information belongs in the abstract as it is a key finding that contextualizes the primary outcome.

  "LLM users spent more time per case (mean difference = 119.3 s, 95% CI = 17.4 to 221.2, P = 0.02)." (Page 1)

  Implementation: Modify the sentence about time spent to clearly indicate that LLM users spent *more* time per case. For example, change to: 'LLM users spent significantly more time per case...'

• Add Trend Direction for LLM Alone vs. Conventional Resources

  This low-impact improvement would provide a more complete and nuanced understanding of the study's findings within the abstract. While the abstract mentions that LLM-augmented physicians performed similarly to the LLM alone, adding the direction of the trend compared to conventional resources adds important context. It allows readers to quickly grasp the relative performance of all three groups (physicians + LLM, physicians + conventional resources, and LLM alone).

  "There was no significant difference between LLM-augmented physicians and LLM alone (−0.9%, 95% CI = −9.0 to 7.2, P = 0.8)." (Page 1)

  Implementation: Add a phrase indicating the direction of the trend when comparing LLM alone to conventional resources. For example: 'There was no significant difference between LLM-augmented physicians and LLM alone (−0.9%, 95% CI = −9.0 to 7.2, P = 0.8), although the LLM alone trended towards higher scores than the conventional resources group.'

Introduction

Key Aspects

• LLMs in Diagnostic Reasoning: The introduction establishes the context of the study by highlighting the existing capabilities of Large Language Models (LLMs) in diagnostic reasoning. It cites previous research showing that LLMs can outperform both AI models and human physicians in tasks such as constructing differential diagnoses and explaining reasoning. This sets the stage for exploring LLMs' potential in a different, more complex area: management reasoning.
• Diagnostic vs. Management Reasoning: A core distinction is drawn between diagnostic reasoning and management reasoning. Diagnostic reasoning is characterized as a classification task, often with a single correct answer. Management reasoning, on the other hand, is presented as a more complex process involving decision-making around treatment, testing, patient preferences, social determinants of health, and cost-conscious care, all while managing risk. This highlights the novelty of the research focus.
• Management Reasoning in Clinical Context: The introduction situates management reasoning within the broader context of clinical reasoning. It acknowledges the overlap between diagnostic and management reasoning but emphasizes the distinct challenges of management reasoning, which often involves weighing trade-offs between risky courses of action and may not have a single "right" answer.
• Theoretical Framework of Management Reasoning: The introduction provides a brief overview of the theoretical underpinnings of management reasoning, referencing concepts like shared decision-making, dynamic relationships, and competing priorities. It also mentions the use of "management scripts" as cognitive heuristics, highlighting their potential susceptibility to fallibilities.
• Historical Context of Reasoning Research: The introduction positions the study within the existing body of research by noting the long history of diagnostic reasoning research and the comparatively recent emergence of management reasoning as a focus of study. This helps to establish the novelty and relevance of the current investigation.
• Study Design Overview: The introduction briefly describes the study's design as a prospective, randomized, controlled trial. This provides a high-level overview of the methodology used to investigate the research question.
• Research Question: The central research question is clearly stated: to assess whether physicians using an LLM perform better than those using standard resources on complex clinical management questions. This provides a concise and focused statement of the study's primary objective.
• Comparison to LLM Alone: The study includes a comparison of physician answers to the output of the LLM alone, without human interaction. This adds another dimension to the investigation, exploring the potential of the LLM as an independent tool.
• Use of Clinical Vignettes: The introduction highlights the use of clinical vignettes derived from real, de-identified patient encounters, with information revealed sequentially. This design choice aims to mirror the complex nature of clinical progression and enhance the study's ecological validity.

Strengths

• Clear Research Gap

  The introduction clearly establishes the research gap by highlighting the known capabilities of LLMs in diagnostic reasoning and contrasting it with the unknown impact on management reasoning. This sets the stage for the study's central question.

  "While large language models (LLMs) have shown promise in diagnostic reasoning, their impact on management reasoning, which involves balancing treatment decisions and testing strategies while managing risk, is unknown." (Page 1)
• Distinction Between Diagnostic and Management Reasoning

  The introduction effectively differentiates between diagnostic and management reasoning, providing context for the study's focus. It highlights the complexity of management reasoning, involving multiple factors and trade-offs, unlike diagnostic reasoning, which often has a single correct answer.

  "Unlike diagnostic reasoning, which can be thought of as a classification task with often a single right answer, management reasoning may have no right answers and involves weighing trade-offs between inherently risky courses of action..." (Page 1)
• Concise Study Overview

  The introduction provides a concise overview of the study design, mentioning the prospective, randomized, controlled trial approach and the use of clinical vignettes derived from real patient encounters. This gives readers a quick understanding of the study's methodology.

  "We designed a prospective, randomized, controlled trial to assess whether physicians using an LLM performed better than physicians using standard resources on a series of complex clinical management questions...All cases were derived from real, de-identified patient encounters." (Page 1)
• Contextualization within Existing Literature

  The introduction connects the research to existing literature on clinical reasoning, mentioning the history of diagnostic reasoning research and the more recent focus on management reasoning. This contextualizes the study within the broader field.

  "The study of diagnostic reasoning has a century-long history with many metacognitive frameworks and assessment methods, while management reasoning processes are a comparatively recent area of study9–11." (Page 1)

Suggestions for Improvement

• Explicitly Justify Focus on Physician Performance

  This medium-impact improvement would strengthen the introduction by providing a more explicit justification for focusing on *physician* performance. While the ultimate goal is likely improved patient care, the introduction could briefly explain why studying physician performance is a crucial intermediate step. This is important because it clarifies the study's direct focus and links it to the broader goal of patient well-being. It also helps readers understand the chain of reasoning: better physician performance is expected to lead to better patient outcomes.

  "This prospective, randomized, controlled trial assessed whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources." (Page 1)

  Implementation: Add a sentence or phrase explaining the link between physician performance and patient outcomes. For example: 'By focusing on physician performance, a key determinant of care quality, this study aims to contribute to the broader goal of improving patient outcomes.'

• Mention Specific LLM

  This low-impact improvement would add a brief mention of the specific LLM used (GPT-4) in the introduction. While this is mentioned in the abstract, including it in the introduction provides immediate context for readers. It clarifies the technology being evaluated and helps readers familiar with different LLMs understand the scope of the study.

  "While large language models (LLMs) have shown promise in diagnostic reasoning, their impact on management reasoning, which involves balancing treatment decisions and testing strategies while managing risk, is unknown." (Page 1)

  Implementation: Add "(GPT-4)" after the first mention of LLM in the introduction. For instance, 'While large language models (LLMs) (GPT-4) have shown promise...'

• Hint at Potential Benefits Before Research Question

  This low-impact improvement would enhance the introduction by briefly mentioning the potential benefits of LLMs in management reasoning *before* stating the research question. This creates a more compelling narrative by first suggesting the potential and then highlighting the need for research to confirm it. It frames the study as addressing a promising but unproven area.

  "This prospective, randomized, controlled trial assessed whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources." (Page 1)

  Implementation: Add a sentence or phrase before the statement of the research question, hinting at the potential benefits. For example: 'Given the potential of LLMs to assist with complex decision-making, this study investigated whether...' or 'If LLMs can indeed improve management reasoning, this could have significant implications for clinical practice. Therefore, this study...'

Results

Key Aspects

• Participant Enrollment and Demographics: The study enrolled 92 physicians, evenly randomized into two groups: one using an LLM (GPT-4) in addition to conventional resources, and the other using conventional resources alone. The participants were primarily attending physicians (73%) specializing in internal medicine (74%), with a mean of 7.6 years in practice. A minority (24%) self-described as frequent LLM users.
• Case Scoring and Inter-rater Reliability: A total of 400 cases were scored, derived from the 92 physicians: 176 from the LLM group, 199 from the conventional resources group, and 25 from the LLM alone. Three graders assessed the cases, achieving substantial agreement (82% agreement, pooled kappa statistic of 0.80).
• Primary Outcome: Total Score: The primary outcome, the total score on the cases, was significantly higher for physicians using the LLM compared to those using conventional resources (43.0% vs. 35.7%, difference = 6.5%, 95% CI = 2.7% to 10.2%, P < 0.001). This indicates a statistically significant improvement in performance with LLM assistance.
• Comparison with LLM Alone: The LLM alone performed comparably to physicians using the LLM (43.7% vs. 43.0%, difference = 0.9%, 95% CI = -7.2% to 9.0%, P = 0.80), and trended towards higher scores than physicians using conventional resources (43.7% vs. 35.7%, difference = 7.3%, 95% CI = -0.7% to 15.4%, P = 0.074). This suggests the potential for LLMs to be used independently in some clinical scenarios.
• Performance in Question Subgroups: Physicians using the LLM scored significantly higher in questions testing management decisions (40.5% vs. 33.4%, P = 0.001), diagnostic decisions (56.8% vs. 45.8%, P = 0.009), and context-specific questions (42.4% vs. 34.9%, P = 0.002). No significant difference was found in factual recall or general management knowledge, although the trends were similar.
• Time Spent per Case: Physicians using the LLM spent significantly more time per case (801.5s vs. 690.2s, difference = 119.3s, 95% CI = 17.4 to 221.2, P = 0.022). However, even after adjusting for time spent, the LLM group still showed a significant increase in score.
• Response Length Analysis: A post-hoc sensitivity analysis adjusting for response length showed an attenuated but still positive effect for the LLM group (3.7 percentage points higher, 95% CI = 0.7 to 6.7, P = 0.02). This suggests that the improved performance was not solely due to longer responses.
• Potential Harm Analysis: Analysis of potential harm revealed similar patterns between the LLM-assisted and conventional resources groups, suggesting that LLM use did not increase the likelihood or severity of harmful responses in this study setting.

Strengths

• Clear Presentation of Participant Demographics

  The Results section clearly presents the participant demographics, including career stage, specialty, years in training, and prior experience with GPT. This provides essential context for interpreting the study findings and assessing their generalizability.

  "Participants were randomized evenly between the LLM and conventional resources groups (Fig. 1); 73% (67 of 92) were attending physicians while 27% (25 of 92) were residents (Table 1). Seventy-four percent (68 of 92) specialized in internal medicine, 20% (18 of 92) emergency medicine and 6.5% (6 of 92) family medicine. The mean time in practice of all physicians was 7.6 years while the median was 5.8 years (interquartile range (IQR) = 3.0 to 9.0 years). Only 24% (22 of 92) self-described as frequent users of LLMs; 20.8% (19 of 92) had either used it only once or never used it." (Page 2)
• Detailed Reporting of Primary Outcome

  The section reports the primary outcome (total score) with appropriate statistical detail, including the difference between groups, confidence interval, and p-value. This allows for a clear understanding of the magnitude and significance of the effect.

  "Physicians randomized to use the LLM performed better than the control group (43.0% compared to 35.7%, difference = 6.5%, 95% confidence interval (CI) = 2.7% to 10.2%, P < 0.001) (Table 2 and Fig. 2)." (Page 2)
• Inclusion of Secondary Outcomes

  The Results section includes secondary outcomes, such as performance in different question domains (management, diagnostic, specific, general, factual) and time spent per case. This provides a more nuanced understanding of the LLM's impact.

  "The physicians using the LLM scored better than those using conventional resources alone in questions explicitly testing management decisions (40.5% versus 33.4%, difference = 6.1%, 95% CI = 2.5% to 9.7%, P = 0.001), questions testing diagnostic decisions (56.8% versus 45.8%, difference = 12.1%, 95% CI = 3.1% to 21%, P = 0.009) and context-specific questions (42.4% versus 34.9%, difference = 6.2%, 95% CI = 2.4% to 9.9%, P = 0.002)." (Page 2)
• Comparison with LLM Alone

  The section presents a comparison between the performance of physicians using the LLM, physicians using conventional resources, and the LLM alone. This provides valuable insight into the potential of the LLM as an independent tool.

  "The LLM alone scored comparably to humans using the LLM (43.7% versus 43.0%, difference = 0.9%, 95% CI = −7.2% to 9.0%, P = 0.80), while trending toward scoring higher than humans using conventional resources (43.7% versus 35.7%, difference = 7.3%, 95% CI = −0.7% to 15.4%, P = 0.074) (Fig. 3)." (Page 2)
• Reporting of Inter-rater Reliability

  The section reports inter-rater reliability statistics (pooled kappa) for the scoring of cases, indicating substantial agreement between graders. This enhances the credibility of the scoring process.

  "Three graders agreed on the scoring of 328 of 400 cases (82%), with a pooled kappa statistic (κ) of 0.80, reflecting substantial agreement between graders..." (Page 2)
• Inclusion of Sensitivity Analyses

  The section includes post-hoc sensitivity analyses adjusting for time spent and response length, addressing potential confounding factors and strengthening the main findings.

  "We performed an additional post hoc sensitivity analysis adjusting for time spent on each case (Extended Data Table 1), which showed a 5.4 percentage point (95% CI = 1.7 to 9.0, P = 0.004) increase in score per case even after adjustment for time spent on the case...To address the potential influence of response length on scores, we conducted an additional post hoc sensitivity analysis adjusting our primary analysis for the character count of responses (Supplementary Item 3)." (Page 2)
• Consideration of Potential Harm

  The Results section briefly addresses the potential for harm, reporting similar patterns of likelihood and severity of harm between the two groups. This is an important consideration for the ethical implementation of LLMs.

  "Analysis of potential harm revealed similar patterns between groups (Supplementary Item 6)." (Page 2)

Suggestions for Improvement

• Clarify Number of Cases Per Physician

  This low-impact improvement would enhance the clarity of the Results section. While the text mentions "400 cases were scored in total," and later refers to n values for different groups and outcomes, it's not immediately obvious how these numbers relate to the 92 physicians. Explicitly stating the number of cases completed *per physician* (on average or the range) would improve clarity. This aligns with the Results section's purpose of transparently reporting the data.

  "From these 92 physicians, 400 cases were scored in total, 176 from the group of physicians using the LLM, 199 from physicians using only conventional resources and 25 from the LLM alone." (Page 2)

  Implementation: Add a sentence clarifying the average or range of cases completed per physician. For example: 'The 92 physicians completed a total of 400 cases, with an average of X cases per physician (range Y-Z).'

• Include Raw Numbers for Past GPT Experience in Table 1

  This medium-impact improvement would provide a more complete picture of the participant demographics and their experience with LLMs. While Table 1 provides percentages for past GPT experience, it doesn't explicitly state the *number* of physicians in each category. Adding the raw numbers (n values) alongside the percentages would give a clearer sense of the distribution. The Results section is where this level of detail is expected, providing context for the study's findings.

  "Past GPT experience I use it frequently (weekly or more) 22 (24%) 11 (24%) 11 (24%)" (Page 3)

  Implementation: In Table 1, add the raw number (n) for each category of past GPT experience, in addition to the percentages. For example: 'I use it frequently (weekly or more) 22 (24%) 11 (24%) 11 (24%)'

• Explain Varying 'n' Values in Table 2

  This low-impact improvement would enhance the clarity and readability of Table 2. Currently, the table presents the number of cases (n) for each outcome, but these numbers vary. Providing a brief explanation for *why* the 'n' values differ across outcomes would improve reader understanding. This is important for transparency and aligns with the Results section's role in presenting the data clearly.

  "Table 2 | Comparisons of the primary and secondary outcomes for physicians with LLM and with conventional resources only (scores standardized to 0–100)" (Page 3)

  Implementation: Add a footnote to Table 2 explaining the varying 'n' values. For example: 'The 'n' values vary across outcomes due to some physicians not completing all questions within each case, or not all questions being applicable to every case.'

• Provide Specific Percentages for Harm Analysis

  This medium-impact improvement would enhance the Results section by providing more specific information about the potential for harm. While the section mentions similar patterns between groups, it lacks detail. Reporting the actual percentages for *each* category of likelihood and severity of harm would provide a more complete and nuanced understanding of this important aspect. This belongs in the Results section as it is a direct finding of the study.

  "Analysis of potential harm revealed similar patterns between groups (Supplementary Item 6)." (Page 2)

  Implementation: Include the specific percentages for each category of likelihood (medium, high) and severity (mild-to-moderate, severe) of harm for both groups. For example: 'In the LLM-assisted group, 8.5% and 4.2% of physician responses carried medium and high likelihood of harm, respectively, compared to 11.4% and 2.9% in the conventional resources group. Regarding harm severity, mild-to-moderate harm was observed in 4.0% of LLM-assisted responses compared to 5.3% in the conventional resources group. Severe harm ratings were nearly identical between groups (LLM = 7.7%; conventional = 7.5%).'

Non-Text Elements

Fig. 1| Study flow diagram. The study included 92 practicing attending...

Full Caption

Fig. 1| Study flow diagram. The study included 92 practicing attending physicians and residents with training in internal medicine, family medicine or emergency medicine. Five expert-developed cases were presented, with scoring rubrics created using a Delphi process. Physicians were randomized to use either GPT-4 via ChatGPT plus in addition to conventional resources (for example, UpToDate, Google), or conventional resources alone. The primary outcome was the difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case.

Figure/Table Image (Page 2)

First Reference in Text

Participants were randomized evenly between the LLM and conventional resources groups (Fig. 1); 73% (67 of 92) were attending physicians while 27% (25 of 92) were residents (Table 1).

Description

• The study included 92 physicians with various training.: The study flow diagram, often called a CONSORT diagram, visually explains how participants moved through the research study. Ninety-two practicing physicians and residents—meaning doctors still in training—were involved. These doctors specialized in internal medicine (adult healthcare), family medicine (healthcare for all ages), or emergency medicine (immediate medical care). The doctors reviewed five cases, prepared by experts. The expert's scoring was based on rubrics, or detailed scoring guides, developed using a 'Delphi process'. A Delphi process is a way to get a group of experts to agree on something by having them answer questions and then giving them anonymous feedback until they reach a consensus. The doctors were then randomly put into one of two groups: one group used GPT-4 (a large language model) via ChatGPT and regular resources like UpToDate (an online medical resource) and Google; the other group used only the regular resources. The main thing the researchers were measuring was the difference in scores between the groups, based on the expert-developed scoring guides. They also looked at other things, like scores on specific areas of the cases and how much time each group spent on each case.
• Physicians were randomized evenly between the LLM and conventional resources groups.: The physicians were randomized to either GPT-4 with conventional resources or conventional resources alone. Randomization is a process of randomly assigning participants to different groups in a study, such as a treatment group or a control group. This helps to ensure that the groups are as similar as possible at the beginning of the study, so that any differences in outcomes can be attributed to the intervention being studied. This is often done by using a computer-generated random number sequence to assign participants to groups.
• 73% (67 of 92) were attending physicians while 27% (25 of 92) were residents: The reference text from the Results section notes that 73% (67 out of 92) of the participants were attending physicians, meaning fully qualified doctors, while the remaining 27% (25 out of 92) were residents, or doctors still undergoing training. These percentages are used to describe the study population.

Scientific Validity

• The study flow diagram is a standard element in RCTs.: The study flow diagram is a standard element in RCTs, adhering to CONSORT guidelines, and is essential for transparency. The mention of the Delphi process for rubric development strengthens the validity by highlighting a rigorous approach to outcome assessment.
• Even randomization minimizes selection bias.: The even randomization, as stated in the reference text and implied by the flow diagram, is crucial for minimizing selection bias. The percentages of attending physicians and residents are descriptive and relevant for characterizing the study population.

Communication

• The caption provides a good overview of the study.: The caption effectively summarizes the key aspects of the study design and outcomes, providing a concise overview for readers to quickly understand the figure's content. It clearly states the number of participants, their backgrounds, the intervention, and the primary and secondary outcomes.
• The caption could be more explicit about the purpose of a CONSORT diagram.: While the caption is informative, it could benefit from a more explicit mention of the CONSORT diagram's purpose, which is to visually represent the flow of participants through each stage of the randomized controlled trial.

Table 1 | Participant characteristics according to randomized group

Figure/Table Image (Page 3)

First Reference in Text

Participants were randomized evenly between the LLM and conventional resources groups (Fig. 1); 73% (67 of 92) were attending physicians while 27% (25 of 92) were residents (Table 1).

Description

• Table 1 describes participant characteristics split by randomized group.: Table 1 is all about describing the people who took part in the study. It shows key details about them, split into two groups based on how they were randomly assigned: one group used GPT-4 along with regular resources, and the other used only regular resources. The table lists things like what stage of their career they're at (attending physician or resident), their medical specialty (internal medicine, emergency medicine, or family medicine), how many years they've been in medical training, and their past experience with using GPT-like tools. It also includes a number called the 'standardized mean difference' or SMD. The SMD is a way to measure how different the two groups are for each of these characteristics. An SMD close to zero means the groups are very similar, which is what you want when you randomly assign people to groups in a study.
• The table provides numbers, percentages, means and standard deviations.: The table provides specific numbers and percentages for each characteristic within each group. For instance, it shows that in the overall study population (n=92), 73% were attending physicians. This is broken down to 74% in the GPT-4 group and 72% in the conventional resources group. The table also gives the mean (average) and standard deviation (a measure of spread) for continuous variables like years in medical training.

Scientific Validity

• Describing participant characteristics is essential for assessing comparability.: The inclusion of participant characteristics is essential for assessing the comparability of the randomized groups at baseline. This is a standard practice in RCTs to ensure that any observed differences in outcomes are attributable to the intervention rather than pre-existing differences between the groups.
• The use of SMD is appropriate for assessing balance.: The use of the standardized mean difference (SMD) is appropriate for assessing the balance between the groups. While there is no universally agreed-upon threshold, an SMD less than 0.2 is generally considered a small and acceptable difference, suggesting adequate randomization.

Communication

• The table is well-organized and facilitates comparison.: The table's clear labeling and organization facilitate easy comparison of baseline characteristics between the two randomized groups. The inclusion of the standardized mean difference (SMD) provides a quick assessment of any imbalances.
• A definition of SMD would improve understanding for some readers.: While the table is generally clear, providing a brief definition of SMD in the caption or a footnote would enhance understanding for readers unfamiliar with this statistic.

Table 2 | Comparisons of the primary and secondary outcomes for physicians with...

Full Caption

Table 2 | Comparisons of the primary and secondary outcomes for physicians with LLM and with conventional resources only (scores standardized to 0-100)

Figure/Table Image (Page 3)

First Reference in Text

Physicians randomized to use the LLM performed better than the control group (43.0% compared to 35.7%, difference = 6.5%, 95% con- fidence interval (CI) = 2.7% to 10.2%, P < 0.001) (Table 2 and Fig. 2).

Description

• Table 2 compares outcomes between LLM and conventional resources groups.: Table 2 presents a head-to-head comparison of how well doctors performed on the cases, depending on whether they had access to the GPT-4 language model or only conventional resources. The table focuses on both the main (primary) outcome and other related (secondary) outcomes. To make the scores easier to understand and compare, they've been converted to a scale from 0 to 100. The table shows two main numbers for each outcome: the average score (mean) and the middle score (median). The average can be skewed by very high or low scores, while the median represents the true middle value. The table also includes the interquartile range (IQR), which is the range between the 25th and 75th percentile, representing the spread of the middle 50% of the data. Finally, it provides the difference between the two groups, a confidence interval (CI), and a P-value. The confidence interval is a range of values that we are fairly sure contains the true population mean. A p-value is a measure of the probability that an observed difference could have occurred just by random chance, if there really was no difference.
• LLM group had an average total score of 43.0 vs 35.7 for conventional resources (p<0.001).: The table shows that the average total score for physicians using LLM was 43.0 (out of 100), while for those using conventional resources it was 35.7. The difference between these averages was 6.5, with a 95% confidence interval ranging from 2.7 to 10.2, and a p-value less than 0.001. This suggests that the LLM group performed significantly better than the conventional resources group.

Scientific Validity

• The table provides comprehensive statistical information.: The table provides essential statistical information for evaluating the study's findings. The inclusion of means, medians, interquartile ranges, differences, confidence intervals, and p-values allows for a comprehensive assessment of the magnitude and statistical significance of the observed effects.
• The use of a generalized mixed-effects model is appropriate.: The use of a generalized mixed-effects model, as mentioned in the table footnote, is appropriate for accounting for the potential correlation between cases for a given participant. This approach strengthens the validity of the statistical analysis.

Communication

• Score standardization enhances interpretability.: The standardization of scores to a 0-100 scale enhances interpretability and allows for easier comparison across different outcome measures. The inclusion of both means and medians provides a more complete picture of the data distribution, as means can be influenced by outliers.
• Clarity could be improved by specifying the level of confidence for CIs in the table header.: While the table is generally well-organized, the presentation of confidence intervals (CI) could be improved. Specifying the level of confidence (95% CI) in the column header, rather than only in the reference text, would provide clarity and avoid ambiguity.

Fig. 2 | Comparison of the primary outcome for physicians with LLM and with...

Full Caption

Fig. 2 | Comparison of the primary outcome for physicians with LLM and with conventional resources only (total score standardized to 0-100). Ninety-two physicians (46 randomized to the LLM group and 46 randomized to conventional resources) completed 375 cases (178 in the LLM group, 197 in the conventional resources group). The center of the box plot represents the median, with the boundaries representing the first and third quartiles. The whiskers represent the furthest data points from the center within 1.5 times the IQR.

Figure/Table Image (Page 4)

First Reference in Text

Physicians randomized to use the LLM performed better than the control group (43.0% compared to 35.7%, difference = 6.5%, 95% con- fidence interval (CI) = 2.7% to 10.2%, P < 0.001) (Table 2 and Fig. 2).

Description

• Figure 2 uses a box plot to compare total scores between the two groups.: Figure 2 uses a box plot to show how the total scores compare between the two groups of doctors. The scores have already been standardized to a 0-100 scale, making it easier to compare. A box plot is a way of summarizing data using five key numbers: the median (the middle value), the first quartile (the value below which 25% of the data falls), the third quartile (the value below which 75% of the data falls), and the minimum and maximum values. The box itself shows the range between the first and third quartiles, representing the middle 50% of the data. The line inside the box shows the median. The 'whiskers' extend out from the box to the furthest data points that are within 1.5 times the interquartile range (IQR) from the box. Any points beyond the whiskers are considered outliers.
• The caption specifies the number of physicians and cases in each group.: The caption specifies that 92 physicians participated in the study, with 46 in each group. The LLM group completed 178 cases, while the conventional resources group completed 197 cases. The box plot visually represents the distribution of these scores for each group.

Scientific Validity

• The use of a box plot is appropriate for visualizing the distribution of scores.: The use of a box plot is appropriate for visualizing the distribution of scores and comparing the central tendency and spread between the two groups. The caption clearly defines the elements of the box plot, ensuring accurate interpretation.
• The reference to Table 2 provides complementary statistical information.: The reference to Table 2 provides the statistical information (mean, confidence interval, p-value) that complements the visual representation in the box plot. This combination of visual and numerical data enhances the rigor and interpretability of the results.

Communication

• The box plot effectively visualizes the distribution of scores.: The box plot effectively visualizes the distribution of total scores for each group, making it easy to compare the median, quartiles, and range of scores. The clear labeling and concise caption contribute to the figure's understandability.
• Adding a visual indicator of statistical significance could enhance the figure's impact.: While the box plot is informative, adding a visual indicator of statistical significance (e.g., an asterisk) could further enhance its impact. Also, reporting the exact p-value within the figure or caption would provide more precise information than simply stating 'P < 0.001' in the reference text.

Fig. 3 | Comparison of the primary outcome according to GPT alone versus...

Full Caption

Fig. 3 | Comparison of the primary outcome according to GPT alone versus physician with GPT-4 and with conventional resources only (total score standardized to 0-100). The GPT-alone arm represents the model being prompted by the study team to complete the five cases, with the models prompted five times for each case for a total of 25 observations. The physicians with GPT-4 group included 46 participants that completed 178 cases, while the physician with conventional resources group included 46 participants that completed 197 cases. The center of the box plot represents the median, with the boundaries representing the first and third quartiles. The whiskers represent the furthest data points from the center within 1.5 times the IQR.

Figure/Table Image (Page 4)

First Reference in Text

While trending toward scoring higher than humans using conventional resources (43.7% versus 35.7%, difference = 7.3%, 95% CI = -0.7% to 15.4%, P = 0.074) (Fig. 3).

Description

• Figure 3 compares primary outcomes across three groups using a box plot.: Figure 3 presents a comparison of the primary outcome (total score standardized to 0-100) across three groups: GPT-4 used alone, physicians using GPT-4, and physicians using conventional resources. It uses a box plot to visualize the distribution of scores for each group. The box plot shows the median (the middle value), the first and third quartiles (the 25th and 75th percentiles), and the range of the data (with whiskers extending to 1.5 times the interquartile range).
• The caption explains the methodology for the GPT-alone arm.: The caption explains that the 'GPT-alone arm' represents the model being prompted by the study team to complete the five cases. Each case was prompted five times, resulting in a total of 25 observations. The physicians with GPT-4 group included 46 participants who completed 178 cases, and the physician with conventional resources group included 46 participants who completed 197 cases.

Scientific Validity

• The inclusion of a GPT-alone arm is a strength of the study.: The inclusion of a GPT-alone arm is a strength of the study, as it allows for a direct comparison of the model's performance against human physicians. This provides valuable information about the potential for standalone AI applications in this domain.
• The small sample size of the GPT-alone arm should be considered.: The relatively small number of observations in the GPT-alone arm (25) compared to the physician groups (178 and 197) should be considered when interpreting the results. This difference in sample size may limit the statistical power to detect significant differences.
• The p-value of 0.074 indicates a trend, but is not statistically significant.: The p-value of 0.074 indicates a trend towards higher scores for GPT alone compared to physicians using conventional resources, but this difference is not statistically significant at the conventional alpha level of 0.05. Therefore, caution is warranted when interpreting this result.

Communication

• The figure effectively compares GPT alone against human physicians.: The figure effectively compares the performance of GPT alone against human physicians, both with and without GPT assistance. The inclusion of the GPT-alone arm provides valuable insight into the potential of the model as a standalone tool.
• The caption clearly explains the methodology for the GPT-alone arm.: The caption clearly explains the methodology for the GPT-alone arm, including the number of prompts and observations. This transparency is crucial for understanding the context of the GPT-alone results.
• The p-value of 0.074 should be interpreted cautiously.: The p-value of 0.074 is marginal and should be interpreted cautiously. While the text notes a 'trend', it's important to avoid overstating the significance of this result.