Assessing the Clinical Utility of Multimodal Large Language Models in the Diagnosis and Management of Pigmented Choroidal Lesions

• Quantification of Inter-Grader Agreement: This table quantifies the level of agreement between two expert human graders who were asked to evaluate the same set of medical images based on six different prompts or tasks. The 'Percentage Agreement' column shows the raw proportion of cases where both experts gave the identical answer, with values ranging from 81.2% to 85.4%, indicating a high degree of consistency.
• Cohen's Kappa (κ) Statistic: The table includes Cohen's Kappa (κ), a statistical measure used to assess the reliability of agreement between two raters. Unlike simple percentage agreement, Kappa accounts for the possibility that agreement could occur by chance. A Kappa value of 0 indicates agreement is equivalent to chance, while 1 indicates perfect agreement. The values in the table range from 0.634 to 0.707, which are generally interpreted as representing 'substantial agreement', reinforcing that the expert graders were highly consistent in their judgments beyond random chance.

• ✅ Use of an appropriate statistical measure: The inclusion of Cohen's Kappa (κ) alongside percentage agreement is a methodological strength. Kappa is the standard and appropriate metric for assessing inter-rater reliability for categorical ratings, as it corrects for chance agreement. This provides a more rigorous and scientifically sound measure of consistency than percentage agreement alone.
• ✅ Establishes a crucial human performance benchmark: This analysis is essential for the study's overall validity. By demonstrating substantial agreement between expert human graders, you establish a reliable 'gold standard' for performance. This benchmark is critical for contextualizing and evaluating the performance of the multimodal large language models (MLLMs) presented later in the study.
• 💡 Lack of prompt descriptions: The table lists prompts by number (1-6) but does not describe the nature of each task. While this information is likely available elsewhere in the manuscript, the table would be more informative and self-contained if it included a brief description for each prompt (e.g., 'Multiple-choice diagnosis', 'Binary classification'). This would allow readers to see if agreement levels varied based on task complexity or type.

• ✅ Clear and concise presentation: The table is well-structured, with a simple three-column layout and clear headings. This minimalist design makes the data easy to read and interpret, allowing the reader to quickly grasp the key takeaway: the human graders showed high agreement.
• 💡 Placement within the manuscript: The table is referenced in the 'Methods' section, but it presents results. Standard reporting convention would place a table summarizing data outcomes in the 'Results' section. Moving this table to the Results section would improve the manuscript's logical flow, separating the description of the methodology from the presentation of its outcomes.

• Presentation of Diagnostic Performance Metrics: This table details the performance of two individual human experts ('Grader 1' and 'Grader 2') across six different diagnostic tasks, or 'prompts'. Performance is measured using three standard metrics: Accuracy, Sensitivity, and Specificity. 'Accuracy' represents the overall proportion of correct diagnoses. 'Sensitivity' measures how well the expert correctly identified the presence of a condition (e.g., melanoma), acting like a 'disease-detector'. 'Specificity' measures how well the expert correctly identified the absence of that condition (e.g., a harmless nevus), acting as a 'healthy-case detector'.
• High Performance Levels of Human Experts: The data shows that both human graders performed at a very high level across all tasks. Accuracy ranged from 78.7% to 93.8%. Sensitivity was also high, ranging from 78.6% to 96.2%, indicating a strong ability to detect disease. Specificity was similarly excellent, ranging from 78.9% to 95.2%, showing a strong ability to rule out disease when it was not present. Each performance value is accompanied by a 95% confidence interval (e.g., 0.812 (0.667, 0.896)), which provides a range of plausible values for the true performance, giving an indication of the estimate's precision.

• ✅ Comprehensive and appropriate metrics: The use of accuracy, sensitivity, and specificity is the standard and most appropriate method for evaluating the performance of a diagnostic test or classifier. This provides a much more complete picture than accuracy alone, distinguishing between the ability to correctly identify positive and negative cases, which is clinically crucial.
• ✅ Inclusion of confidence intervals: Reporting 95% confidence intervals for each point estimate is a significant methodological strength. It transparently communicates the statistical uncertainty and precision of the performance measures, allowing for a more robust interpretation of the results and comparison between graders and prompts.
• ✅ Establishes a robust human performance benchmark: This table is critical for the study's objective as it quantifies expert-level performance. This data serves as the 'gold standard' or benchmark against which the performance of the AI models is later compared, providing essential context for evaluating the clinical utility of the MLLMs.

• ✅ Clear and logical structure: The table is well-organized and easy to read. It logically groups data by prompt, then by grader, with separate columns for each key performance metric. This structure allows for easy comparison between the two graders and across the different tasks.
• 💡 Lack of context for prompts: The prompts are identified only by number (1-6). The table would be significantly more self-contained and informative if a brief description of each prompt's task was included (e.g., 'Prompt 1 - Multiple-choice diagnosis'). This would allow readers to interpret performance variations across tasks without needing to search for this information in the main text.
• 💡 Potential for misplacement in the manuscript: The element is referenced in the 'Results' section of the paper, but the provided section is 'Methods'. While this might be an artifact of the prompt, tables presenting study outcomes, such as this one, should be located in the 'Results' section to maintain a logical flow from methodology to findings.

• Comparative Performance of AI Models: This table presents the performance of three different AI systems—ChatGPT, Gemini, and Perplexity—on six different tasks (prompts) related to diagnosing eye lesions. Their performance is measured using three key metrics: 'Accuracy' (the overall percentage of correct answers), 'Sensitivity' (the ability to correctly identify cases with the disease, like a good smoke detector), and 'Specificity' (the ability to correctly identify cases without the disease, avoiding false alarms). Each reported value is accompanied by a 95% confidence interval, which indicates the range of plausible true values for that metric.
• Performance Varies by Model and Task: The results show a clear difference in performance. Gemini consistently performed best, especially on Prompt 6 (recommending treatment with clinical information), where it achieved the highest accuracy of 0.708 (or 70.8%). In contrast, ChatGPT's performance was generally lower, with an accuracy of just 0.314 (31.4%) on the same task. This suggests that the choice of AI model and the specific nature of the task significantly impact the outcome.
• Critical Failure on Rule-Based Diagnosis: A notable finding is the performance on Prompt 4, which required the AIs to apply strict clinical guidelines (COMS criteria). For this task, both Gemini and Perplexity had a sensitivity of 0.000, meaning they failed to identify a single case of melanoma. They achieved perfect specificity (1.000), meaning they never misidentified a healthy case, but their inability to detect the disease under these rules is a critical failure.

• ✅ Comprehensive and appropriate metrics: The use of accuracy, sensitivity, and specificity provides a robust and multifaceted evaluation of the MLLMs' diagnostic capabilities. This is far superior to relying on accuracy alone, as it differentiates between the models' ability to identify true positives and true negatives, which is essential for clinical applicability.
• ✅ Inclusion of confidence intervals: Reporting 95% confidence intervals for all performance metrics is a major strength. It provides a measure of statistical precision for each estimate, transparently conveying the uncertainty associated with the sample size and allowing for more nuanced comparisons between models.
• ✅ Strong experimental design: The stepwise progression of prompts (e.g., diagnosis without clinical info vs. with clinical info) is an excellent design choice. It allows for a direct assessment of how additional data impacts model performance, which is a key question for understanding their reasoning capabilities. The results from Prompt 4, testing adherence to strict guidelines, provide particularly powerful evidence of the models' current limitations.
• 💡 Wide confidence intervals suggest limited statistical power: While the inclusion of confidence intervals is a strength, their width (e.g., Gemini's accuracy on Prompt 6 is 0.708 with a CI of 0.562 to 0.812) indicates considerable uncertainty in the point estimates. This is an inherent limitation of the sample size (n=48) and should be explicitly acknowledged when interpreting the magnitude of performance differences between models.

• ✅ Logical and clear table structure: The table is well-organized, grouping results first by prompt and then by model. This hierarchical structure makes it straightforward for readers to compare the performance of the three MLLMs on any given task and to track a single model's performance across different tasks.
• 💡 Lack of visual aids for comparison: The table is dense with numerical data, which can make it difficult to quickly identify the key findings. To improve scannability, consider using bold text to highlight the best-performing model for each metric within each prompt. This simple visual cue would help guide the reader's attention to important patterns, such as Gemini's general superiority.
• 💡 Table is not self-contained: The prompts are only identified by number, requiring the reader to refer back to the main text to understand the specific task being evaluated. The table's utility would be greatly enhanced by adding a brief description of each prompt, either in a separate column or as a footnote to the table.
• 💡 Potential misplacement in manuscript: The table presents key study results, but its section is listed as 'Methods'. To align with standard scientific reporting structure, this table should be placed in the 'Results' section, ensuring a clear distinction between the description of the study's design and the presentation of its findings.

• Error Analysis of AI Models: This table breaks down the specific mistakes made by three AI models (ChatGPT, Gemini, Perplexity) when asked to perform a multiple-choice diagnosis of eye lesions. It shows not just how often they were wrong ('Misclassification, %'), but also what incorrect diagnosis they chose. This helps to reveal if the AIs have specific biases in their errors.
• ChatGPT's Tendency to Over-diagnose Malignancy: The data shows a significant bias for ChatGPT. When presented with a choroidal nevus, which is a benign, mole-like spot in the back of the eye, ChatGPT misdiagnosed it 93.1% of the time. Critically, of these errors, nearly half (48.1%) were misclassifications as melanoma, which is a dangerous form of eye cancer. This indicates a strong tendency to mistake a harmless condition for a malignant one.
• Gemini's Tendency to Under-diagnose Malignancy: Gemini displayed the opposite bias. When shown cases of actual melanoma (cancer), it failed to identify it correctly 100% of the time. The vast majority of these errors (84.2%) were misclassifications as a benign choroidal nevus. This reveals a dangerous tendency to mistake a malignant condition for a harmless one.
• Perplexity's High Error Rate: Perplexity showed high rates of misclassification for both conditions. It incorrectly diagnosed benign nevi 62.1% of the time and malignant melanomas 68.4% of the time. Its errors were more varied, but it also showed a tendency to mislabel melanoma as a benign nevus (53.8% of its melanoma errors).

• ✅ Provides crucial insights beyond accuracy metrics: This misclassification analysis is a major strength of the study. By moving beyond simple accuracy scores and detailing the specific nature of the errors, you provide invaluable insight into the diagnostic biases and failure modes of each MLLM. This level of analysis is essential for evaluating the clinical safety and utility of these models.
• ✅ Data strongly supports the conclusion of systematic bias: The results clearly demonstrate that the errors are not random. The distinct and opposing patterns—ChatGPT's over-diagnosis of malignancy and Gemini's under-diagnosis—are stark. This provides strong evidence for systematic biases within the models' decision-making processes, which is a significant finding.
• 💡 Ambiguity in the 'Misclassification, %' column: The column header 'Misclassification, %' could be interpreted in multiple ways. It appears to represent the error rate for that specific ground truth category (e.g., 100% of melanoma cases were misclassified by Gemini). Clarifying the header to 'Error Rate (%)' or 'Misclassification Rate for Ground Truth (%)' would remove any ambiguity for the reader.

• ✅ Effective structure for error analysis: The table is well-structured, clearly separating the overall misclassification rate from the detailed breakdown of error types. This allows the reader to first grasp the magnitude of the errors and then investigate their nature, which is a logical and effective way to present this complex data.
• 💡 Overuse of abbreviations hinders readability: The use of abbreviations for all diagnostic classes (AMD, CH, CN, etc.) in the column headers forces the reader to constantly consult the table footer. This significantly slows down interpretation. Given the available space, spelling out the full condition names (e.g., 'Choroidal Nevus', 'Melanoma') would make the table much more self-contained and user-friendly.
• 💡 Lack of visual hierarchy: The table presents many numbers, but the most critical findings—the specific biases—do not stand out visually. Consider using bold text or cell shading to highlight the most frequent misclassification for each ground truth (e.g., bolding the '48.1' for ChatGPT/CN and the '84.2' for Gemini/Melanoma). This would help guide the reader's eye to the key takeaways.

• Analysis of AI Treatment Recommendation Errors: This table details the specific mistakes made by three AI models (ChatGPT, Gemini, Perplexity) when asked to recommend a course of treatment for eye lesions. The analysis focuses on cases where the AI's recommendation was incorrect, showing not just how often it was wrong but also which incorrect treatment it chose. The correct treatments, or 'Ground Truth', fall into three categories: 'Enucleation' (surgical removal of the eye), 'Observation' (a 'watch and wait' approach), and 'Radiotherapy' (using radiation to treat the lesion).
• Systematic Bias Towards Recommending Radiotherapy: A striking pattern emerged across all three AI models: a strong tendency to default to recommending radiotherapy, especially when it was the wrong answer. For example, in cases where the correct action was 'Observation', both ChatGPT and Perplexity incorrectly recommended radiotherapy 100% of the time. Similarly, for cases requiring 'Enucleation', ChatGPT and Gemini incorrectly recommended radiotherapy in 100% of their errors. This shows the AIs are not making random mistakes but are systematically biased towards a specific treatment option.
• High Rates of Critical Misclassifications: The table highlights clinically significant errors. For instance, ChatGPT misclassified 100% of cases that should have been observed and 100% of cases that required eye removal, defaulting to radiotherapy in every single instance. This represents a high potential for both unnecessary, aggressive treatment (overtreatment) and failure to recommend the definitive surgical procedure (undertreatment).

• ✅ Crucial analysis for clinical safety assessment: This misclassification analysis is vital for evaluating the real-world safety of these MLLMs. By detailing the nature of the errors (e.g., recommending aggressive treatment for cases needing observation), the table provides far more insight into potential patient harm than a simple accuracy score would. This is a methodologically sound and clinically relevant approach.
• ✅ Data provides strong evidence for systematic bias: The data overwhelmingly supports the claim made in the reference text. The consistent defaulting to 'Radiotherapy' across different models and for different ground truths is not a random occurrence but a clear, systematic failure mode. This is a significant and robust finding regarding the current state of these models for medical decision-making.
• 💡 Suggests a potential 'middle-ground' bias: The strong preference for radiotherapy could be interpreted as the models selecting a 'middle-ground' option between the less invasive 'Observation' and the highly invasive 'Enucleation'. While the analysis is sound, a discussion of this potential cognitive bias—a tendency to avoid extremes—could add a valuable interpretive layer to the results, suggesting a specific area for future model refinement.

• ✅ Effective hierarchical structure: The table is well-organized, grouping the data first by MLLM and then by the correct treatment ('Ground Truth'). This structure makes it easy for the reader to assess each model's error patterns individually and compare them.
• 💡 Ambiguity in nested percentages: The columns under 'Misclassification Class, %' represent the percentage breakdown of the errors, not of the total cases. For example, for ChatGPT/Observation, it misclassified 86.2% of cases, and of those errors, 100% were 'Radiotherapy'. This nested percentage can be confusing. To improve clarity, the sub-header could be changed to '% of Misclassified Cases' or a footnote could be added to explain that these columns characterize the distribution of incorrect answers.
• 💡 Placement in manuscript: This table presents key findings from the study's experiments. According to standard scientific reporting conventions, it should be located in the 'Results' section rather than the 'Methods' section to maintain a clear and logical separation between the description of the experimental setup and the presentation of its outcomes.

• Analysis of AI Treatment Recommendations with Clinical Data: This table shows the specific errors made by three AI models (ChatGPT, Gemini, Perplexity) when asked to recommend a treatment for eye lesions, but this time they were also provided with relevant patient clinical information (Prompt 6). The analysis breaks down the incorrect recommendations for three potential correct actions: 'Enucleation' (surgical removal of the eye), 'Observation' (a 'watch and wait' strategy), and 'Radiotherapy'. This allows for a direct comparison with Table 5 to see if clinical data improved the AIs' performance.
• Gemini's Significant Improvement: The addition of clinical data led to a dramatic improvement for the Gemini model. Its misclassification rate for cases requiring 'Observation' dropped from 62.1% (in Table 5) to just 10.3%. For cases requiring 'Enucleation', its error rate fell from 100% to 50%. This suggests Gemini was able to effectively integrate the textual clinical data with the images to make more accurate recommendations.
• ChatGPT's Persistent Bias: In contrast to Gemini, ChatGPT's performance did not improve with the addition of clinical data. It continued to misclassify 100% of cases requiring 'Enucleation' and a very high percentage (89.7%) of cases requiring 'Observation'. In virtually all of these errors, its recommendation was 'Radiotherapy', indicating that the clinical information did not correct its strong, pre-existing bias.
• Perplexity's Moderate Improvement: Perplexity showed some improvement but was not as successful as Gemini. Its misclassification rate for 'Observation' cases decreased from 75.9% to 27.6%, and for 'Enucleation' cases, it dropped from 100% to 50%. While an improvement, it still made incorrect recommendations at a significant rate.

• ✅ Powerful experimental design isolates the impact of clinical data: By presenting the same cases with and without clinical information (Prompt 5 vs. Prompt 6), the study design effectively isolates the variable of interest: the models' ability to integrate multimodal data. This direct comparison provides strong evidence for the differential capabilities of the MLLMs, which is a significant finding.
• ✅ Reveals critical differences in model architecture or training: The starkly different outcomes—Gemini's marked improvement versus ChatGPT's stagnation—strongly suggest fundamental differences in how these models process and weigh combined image and text inputs. This analysis provides valuable data for understanding the current state of multimodal AI in medicine.
• 💡 Lacks formal statistical comparison between prompts: While the improvement for Gemini and Perplexity is visually apparent when comparing Table 5 and Table 6, the analysis would be more rigorous if it included a formal statistical test (e.g., McNemar's test) to confirm that the change in error rates between the two conditions is statistically significant. This would add quantitative support to the qualitative observations.

• ✅ Consistent structure facilitates comparison: The table maintains the same clear and logical format as Table 5. This consistency is an excellent design choice, as it makes it very easy for the reader to directly compare the results across the two prompts and quickly identify changes in performance.
• 💡 Ambiguous percentage reporting: The columns under 'Misclassification Class, %' detail the percentage breakdown of the errors, not the overall case count. This could be misinterpreted. To prevent confusion, the sub-header should be clarified to something like '% of Misclassified Cases' or a footnote explaining the calculation should be added.
• 💡 Missed opportunity for visual emphasis: The key story of this table is the change in performance from Table 5. This could have been visually emphasized to aid rapid comprehension. For example, using color-coding (e.g., green for improved metrics, red for stagnant ones) or adding small arrows (↑ or ↓) next to the misclassification percentages to indicate the direction of change from the previous prompt would have made the main findings instantly recognizable.

Table 7. Significant Differences for Accuracy and Sensitivity across Various Prompts

• Pairwise Performance Comparison Matrix: This table is a matrix that summarizes the results of statistical tests comparing the performance of five different 'graders' (three AI models: ChatGPT, Gemini, Perplexity; and two human experts: Grader 1, Grader 2) against each other. Each cell in the table answers the question: 'Was the grader in this row significantly worse than the grader in this column?'
• Decoding the Table's Structure and Content: The table is split into two halves. The shaded upper-right half shows comparisons for 'Accuracy' (overall correctness), while the unshaded lower-left half shows comparisons for 'Sensitivity' (the ability to correctly identify a condition when it's present). If a cell contains a prompt number (e.g., 'Prompt 6'), it means the row grader's performance was statistically worse than the column grader's on that specific task. A cell with 'None' indicates no significant difference was found.
• Key Findings on AI vs. Human Performance: The table clearly shows that the human graders consistently outperformed the AI models. For example, the row for 'ChatGPT' shows prompt numbers in almost every column, indicating it was significantly worse than nearly every other grader on most tasks for both accuracy and sensitivity. In contrast, the rows for the human graders have many 'None' entries when compared to each other, showing their performance was statistically similar. Gemini appears to be the best-performing AI, as its row contains fewer entries indicating poor performance compared to ChatGPT.

• ✅ Rigorous statistical methodology: Presenting the results of pairwise statistical comparisons, presumably corrected for multiple comparisons as mentioned in the methods, is a methodologically rigorous approach. It moves beyond simply observing differences in mean performance scores (from Tables 2 and 3) to determining whether those differences are statistically meaningful, which strengthens the study's conclusions.
• 💡 Lacks detail on statistical tests: The table effectively summarizes the outcomes but lacks transparency about the underlying statistics. It does not specify the statistical test used, the p-values, or the effect sizes for the significant differences. While the goal is simplification, this omission prevents a full critical appraisal of the statistical evidence. A supplementary table with these details would enhance the study's rigor.
• 💡 Directionality of comparison is not explicit: The most significant limitation is that the table does not explicitly state the direction of the significant difference (i.e., which grader was better or worse). This crucial information must be inferred by the reader from the narrative text, which undermines the table's value as a standalone element. The analysis is sound, but its reporting lacks the necessary clarity to be fully self-contained.

• ✅ Highly efficient format for complex data: Using a matrix to display all pairwise comparisons is an extremely space-efficient and effective method. It condenses a large number of individual statistical results into a single, comprehensive visual, avoiding the need for lengthy and repetitive text.
• 💡 Poorly self-contained due to lack of a key: The table is nearly impossible to interpret without external context from the main text. There is no key or footnote to explain what the cell entries mean (e.g., 'Cell entries indicate prompts for which the row grader performed significantly worse than the column grader'). This is a major communication failure and should be rectified by adding a clear explanatory note to the table.
• 💡 Awkward labeling and visual design: The diagonal label splitting 'Accuracy' and 'Sensitivity' is unconventional and not immediately clear. A better design would be to use explicit super-headers above the table or create two separate, smaller tables. Furthermore, the distinction between the shaded and unshaded regions is subtle and could be missed by a casual reader.
• 💡 Caption is imprecise: The caption, 'Significant Differences...', is neutral, whereas the table exclusively reports directional outcomes (i.e., one grader being worse than another). A more precise caption, such as 'Summary of Pairwise Comparisons Identifying Significantly Poorer Performance', would more accurately reflect the table's content.