Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

Table 1: Taxonomy of set-based evaluation metrics. Each row represents a different approach to handling error costs, and each column represents a different approach to handling class balance.

• Table Purpose and Organizing Dimensions: Table 1 organizes different ways to measure how well a classification model performs, specifically for tasks where the outcome is one of two options (binary classification). It structures these methods based on two main considerations: how the measurement deals with 'error costs' and how it handles 'class balance'. 'Error costs' refer to the idea that making one type of mistake (e.g., saying someone is sick when they are healthy) might be more or less serious than making the other type of mistake (e.g., saying someone is healthy when they are sick). 'Class balance' refers to whether the dataset has an equal number of examples for each outcome (e.g., equal numbers of sick and healthy patients) or an unequal number (e.g., many more healthy patients than sick ones).
• Row Categories: Handling Error Costs: The table has three rows, each representing a different approach to incorporating error costs into the evaluation: "Accuracy" (which typically treats all errors equally), "Weighted Accuracy" (which assigns different weights to correctly classifying each class, implicitly handling asymmetric error costs), and "Net Benefit" (a metric often used in medical decision making that quantifies the benefit of using a model considering the relative costs of true positives, false positives, etc.).
• Column Categories: Handling Class Balance: The table has three columns, each representing a different approach to handling class balance: "Empirical" (likely meaning evaluation on the observed class distribution of the dataset), "Balanced" (evaluation as if the classes were balanced, e.g., 50/50), and "Prior-Adjusted Maximum" (evaluation that considers a target or deployment class prevalence, potentially different from the training data, and optimizes for it).
• Specific Metrics Listed: The cells of the table contain acronyms for specific evaluation metrics that fit the intersection of the row and column categories. For example: - Under "Empirical" and "Accuracy", the metric is "Accuracy". - Under "Balanced" and "Accuracy", the metric is "BA" (Balanced Accuracy). - Under "Prior-Adjusted Maximum" and "Accuracy", the metric is "PAMA" (Prior-Adjusted Maximum Accuracy). Similar acronyms fill the table: WA (Weighted Accuracy), BWA (Balanced Weighted Accuracy), PAMWA (Prior-Adjusted Maximum Weighted Accuracy), NB (Net Benefit), BNB (Balanced Net Benefit), and PAMNB (Prior-Adjusted Maximum Net Benefit).
• Equivalence Note for Balanced Metrics: A note below the table clarifies that when the "Balanced" approach to class balance is used (the middle column), the metrics for "Weighted Accuracy" (BWA) and "Net Benefit" (BNB) become equivalent. This means that under balanced conditions, these two ways of accounting for error costs yield the same or a proportionally scaled result.

• ✅ Logical Taxonomy Structure: The proposed taxonomy provides a structured and logical way to categorize set-based evaluation metrics along two critical dimensions relevant to clinical machine learning: sensitivity to error costs and robustness to class imbalance/label shift. This is a useful conceptual framework.
• ✅ Relevance of Included Metrics: The metrics listed (Accuracy, BA, PAMA, WA, BWA, PAMWA, NB, BNB, PAMNB) are generally recognized or are plausible extensions of existing metrics within the machine learning and medical decision-making literature. The categorization appears consistent with the typical properties of these metrics.
• ✅ Valid Distinction of Class Balance Approaches: The distinction between empirical, balanced, and prior-adjusted maximum approaches to class balance is a valid and important one, reflecting different assumptions and goals in model evaluation.
• ✅ Insightful Equivalence Note: The note about the equivalence of Weighted Accuracy and Net Benefit under balanced conditions is an insightful observation that might not be immediately obvious, adding to the utility of the table.
• 💡 Scope of "Set-Based" Metrics: While the taxonomy is useful, it is focused on "set-based" metrics. The paper also discusses calibration and proper scoring rules. The table doesn't explicitly incorporate metrics that directly measure calibration (like ECE, Brier score variants that might fit here) or sharpness, which are central themes of the paper. This is not a flaw of the table itself for its stated purpose, but its scope should be understood in the context of the paper's broader aims. Perhaps the term "set-based" is meant to distinguish these from scoring rules applied to probabilistic outputs directly.
• 💡 Dependence on External Definitions for Novel Metrics: The definitions or derivations of the "Prior-Adjusted Maximum" versions of these metrics (PAMA, PAMWA, PAMNB) are critical for their correct application and interpretation. The table itself doesn't provide this but relies on the text (e.g., Definition 3.3 for PAMA, Definition 3.4 for PAMNB). The validity of these specific novel metrics depends on those definitions being sound and well-justified.

• ✅ Clear Structure: The tabular format is an effective way to organize and present a taxonomy of metrics based on two distinct dimensions (handling error costs and class balance).
• ✅ Informative Caption: The caption clearly explains the organizing principle of the table: rows for error cost approaches and columns for class balance approaches. This helps the reader understand how to navigate the table.
• 💡 Acronym Usage: The use of acronyms (BA, WA, BWA, NB, BNB, PAMA, PAMWA, PAMNB) is concise but requires the reader to either know them or for them to be defined in the text. For a table aiming to be somewhat self-contained, brief expansions or references to where they are defined would be beneficial. Suggestion: Consider adding a footnote or a brief parenthetical expansion for less common acronyms, or ensure they are defined immediately preceding or following the table's first mention.
• ✅ Header Clarity: The column headers ("Empirical", "Balanced", "Prior-Adjusted Maximum") and row headers ("Accuracy", "Weighted Accuracy", "Net Benefit") are reasonably clear, though "Empirical" as a contrast to "Balanced" might benefit from a slightly more descriptive term if space permits (e.g., "Standard/Unadjusted").
• ✅ Useful Interpretive Note: The note "Note that when balanced, the second and third rows are equivalent" is an important piece of information for correctly interpreting the table. Placing it directly below the table is appropriate.

• Table Purpose: Defining Value Function for Accuracy: Table 2 defines a 'value function' for the classification metric 'Accuracy'. A value function in this context assigns a numerical score (value) to each possible outcome of a prediction. The table is structured as a 2x2 grid, representing the four possible scenarios in a binary (two-outcome) classification problem.
• Table Structure: True States vs. Predicted Actions: The columns represent the true state of an individual: 'y=1' indicates the individual truly has the condition (e.g., Syphilis), and 'y=0' indicates they do not (e.g., No Syphilis). The rows represent the model's prediction or the action taken based on it: 'ŷ=1' means the model predicts the condition is present (e.g., Treat for Syphilis), and 'ŷ=0' means the model predicts the condition is absent (e.g., Don't treat).
• Cell Values: Scores for Prediction Outcomes: The cells of the table show the value assigned to each combination: - If the true state is 'y=1' (has Syphilis) and the prediction is 'ŷ=1' (Treat), the value is 1. This is a True Positive (correctly identifying a sick person). - If the true state is 'y=0' (No Syphilis) and the prediction is 'ŷ=1' (Treat), the value is 0. This is a False Positive (incorrectly diagnosing a healthy person as sick). - If the true state is 'y=1' (has Syphilis) and the prediction is 'ŷ=0' (Don't treat), the value is 0. This is a False Negative (incorrectly diagnosing a sick person as healthy). - If the true state is 'y=0' (No Syphilis) and the prediction is 'ŷ=0' (Don't treat), the value is 1. This is a True Negative (correctly identifying a healthy person).
• Relation to Standard Accuracy: This value function directly corresponds to the standard definition of accuracy, where correct classifications (True Positives and True Negatives) are given a value of 1, and incorrect classifications (False Positives and False Negatives) are given a value of 0. Accuracy is then calculated as the sum of these values divided by the total number of instances.

• ✅ Correct Representation of Accuracy's Value Function: The table accurately represents the standard value function (or utility/loss function where loss = 1 - value) underlying the calculation of simple accuracy. It assigns a value of 1 to correct classifications (TP, TN) and 0 to incorrect ones (FP, FN).
• ✅ Appropriateness for Basic Accuracy: This representation is fundamental and appropriate for defining the most basic form of accuracy, which treats all types of errors (FP and FN) as equally costly and all types of correct classifications as equally beneficial.
• ✅ Highlights Limitations of Standard Accuracy: The table implicitly highlights the limitations of standard accuracy that the paper aims to address: it does not differentiate between the costs of false positives and false negatives, nor does it inherently account for class imbalance. This makes it a good starting point for discussing more nuanced metrics.
• ✅ Generalizability Beyond Example: The use of a specific clinical example ("Syphilis") is illustrative but does not change the fundamental mathematical definition of the value function for accuracy, which is general.

• ✅ Clear Structure: The table uses a standard 2x2 contingency table format, which is familiar and easy to understand for representing outcomes of a binary classification.
• ✅ Clear and Intuitive Labeling: The labels for true states (columns: "y=1 (Syphilis)", "y=0 (No Syphilis)") and predicted actions/labels (rows: "ŷ=1 (Treat)", "ŷ=0 (Don't treat)") are clear and use a concrete example (Syphilis) which aids intuition.
• ✅ Unambiguous Values: The numerical values (1 for correct classification, 0 for incorrect) are standard for a simple accuracy value function and are unambiguous.
• 💡 Notation V1/2(y, ŷ): The notation V1/2(y, ŷ) in the top-left cell, representing the value function, is specific. While V(y, ŷ) is common, the subscript '1/2' might require clarification if it has a specific meaning beyond this context (e.g., related to balanced classes or equal costs, which is implicit in standard accuracy). Suggestion: If '1/2' has a specific implication beyond standard accuracy (e.g., tied to balanced scenarios or specific normalization), briefly note it. Otherwise, V(y, ŷ) would suffice.

• Table Purpose: Value Function for Balanced Accuracy: Table 3 specifies a 'value function' used in the calculation of 'Balanced Accuracy'. A value function assigns a numerical score to each possible outcome of a classification. This table is for binary classification, where there are two possible true states and two possible predictions.
• Table Structure: True States vs. Predicted Actions: The table is a 2x2 grid. The columns represent the true actual condition of an individual: 'y=1' (e.g., has Syphilis) and 'y=0' (e.g., No Syphilis). The rows represent the model's prediction or the resulting action: 'ŷ=1' (e.g., Treat for Syphilis) and 'ŷ=0' (e.g., Don't treat).
• Cell Values and Role of π₀: The values in the cells are defined in terms of π₀ (pi-naught), which represents the empirical class prevalence of the positive class (y=1) in the dataset. That is, π₀ is the proportion of actual positive cases in the data being evaluated. - True Positive (y=1, ŷ=1): Correctly identifying a positive case. Value = 1 / (2π₀). - False Positive (y=0, ŷ=1): Incorrectly identifying a negative case as positive. Value = 0. - False Negative (y=1, ŷ=0): Incorrectly identifying a positive case as negative. Value = 0. - True Negative (y=0, ŷ=0): Correctly identifying a negative case. Value = 1 / (2(1-π₀)).
• Mechanism of Balancing: This value function achieves 'balance' by weighting the contribution of correct predictions for each class by the inverse of that class's prevalence (scaled by 2). For instance, if the positive class is rare (small π₀), a correct positive prediction gets a higher weight (1/(2π₀)). This ensures that the performance on both the majority and minority classes contributes equally to the overall Balanced Accuracy score, which is typically calculated as the average of sensitivity (true positive rate) and specificity (true negative rate). Summing these values over all instances and averaging appropriately yields the Balanced Accuracy.

• ✅ Correct Mathematical Formulation: The table correctly represents the per-instance contributions to Balanced Accuracy. When these values are summed over all instances in a dataset with positive class prevalence π₀, and then averaged (divided by the total number of instances N), the result is equivalent to (Sensitivity + Specificity) / 2.
• ✅ Appropriateness for Balanced Accuracy: This value function is appropriate for defining Balanced Accuracy, a metric specifically designed to address class imbalance issues by giving equal importance to performance on each class, regardless of their relative frequencies in the data.
• ✅ Implicit Error Cost Handling: By assigning values of 0 to misclassifications (False Positives and False Negatives), this specific value function still implies that all errors are equally undesirable in terms of direct penalty, similar to standard accuracy. The balancing comes from up-weighting correct classifications of the minority class, rather than differential penalization of error types (which would be handled by metrics like Weighted Accuracy or Net Benefit).
• ✅ Consistency with Paper's Definitions: The formulation is consistent with the paper's Definition B.2 of Balanced Accuracy, where these cell values correspond to the product of the basic 0/1 accuracy value function (V1/2 from Table 2) and the importance sampling weights (W(π₀ → 1/2; y)) used to reweight the original dataset to a perfectly balanced one.

• ✅ Clear Structure: The 2x2 contingency table format is standard and easy to interpret for binary classification outcomes.
• ✅ Intuitive Labeling: The labels for true states (e.g., "y=1 (Syphilis)") and predicted actions (e.g., "ŷ=1 (Treat)") are clear and the use of a concrete example aids understanding.
• ✅ Explicit Weighting: The mathematical expressions in the cells, involving π₀, clearly show how the weighting is applied to achieve balance.
• 💡 Definition of π₀: The term π₀ is used in the cell values. While it's a standard notation for empirical class prevalence in this paper (defined on page 3), briefly defining it in a footnote to the table or ensuring its definition is very proximal in the text would enhance the table's self-containedness for readers who jump to it. Suggestion: Add a footnote: "π₀ represents the empirical prevalence of the positive class (y=1) in the dataset."
• ✅ Appropriate Header Notation: The table header V(y, ŷ) is simpler and appropriate here.

• Table Purpose: Value Function for Shifted Accuracy: Table 4 defines a 'value function' for a metric called 'Shifted Accuracy'. In classification, a value function assigns a numerical score to each possible outcome (e.g., correctly identifying a sick person, incorrectly identifying a healthy person). 'Shifted Accuracy' is a type of accuracy measurement that is adjusted to reflect a scenario where the prevalence (or frequency) of the condition in the real-world deployment setting (target prevalence, denoted by π) is different from its prevalence in the dataset used for evaluation (empirical prevalence, denoted by π₀).
• Table Structure: True States vs. Predicted Actions: The table is a 2x2 grid. The columns represent the true condition of an individual: 'y=1' (e.g., has Syphilis) and 'y=0' (e.g., No Syphilis). The rows represent the model's prediction or action: 'ŷ=1' (e.g., Treat for Syphilis) and 'ŷ=0' (e.g., Don't treat).
• Cell Values and Prevalence Adjustment Factors: The values in the cells represent the score assigned to each outcome, adjusted for the shift in prevalence: - True Positive (y=1, ŷ=1): Correctly identifying a positive case. The value is π/π₀. This means a correct positive identification is weighted by the ratio of the target prevalence to the empirical prevalence. - False Positive (y=0, ŷ=1): Incorrectly identifying a negative case as positive. The value is 0. - False Negative (y=1, ŷ=0): Incorrectly identifying a positive case as negative. The value is 0. - True Negative (y=0, ŷ=0): Correctly identifying a negative case. The value is (1-π)/(1-π₀). This means a correct negative identification is weighted by the ratio of the target negative class prevalence to the empirical negative class prevalence.
• Mechanism of Shifting/Re-weighting: This value function essentially re-weights the standard accuracy outcomes. If the target prevalence of the positive class (π) is higher than in the evaluation data (π₀), then True Positives are given more weight. Conversely, if the target prevalence of the negative class (1-π) is higher than in the evaluation data (1-π₀), True Negatives are given more weight. Misclassifications (False Positives and False Negatives) are still assigned a value of 0.

• ✅ Correct Formulation for Label Shift Adjustment: The table correctly represents the value function for an accuracy metric adjusted for label shift using importance weighting. The factors π/π₀ and (1-π)/(1-π₀) are standard importance weights for re-weighting class-conditional expectations when moving from a source distribution with prevalence π₀ to a target distribution with prevalence π, assuming P(X|Y) is invariant.
• ✅ Appropriateness for Label Shift Evaluation: This approach is scientifically valid and appropriate for evaluating classifier performance under known or assumed label shift conditions, providing a more relevant estimate of performance in a target deployment environment.
• ✅ Consistency with PAMA Definition: This value function, when applied to a dataset, directly leads to the calculation of Prior-Adjusted Maximum Accuracy (PAMA) as defined in Definition B.3 (and Definition 3.3) of the paper, where the standard 0/1 accuracy values are multiplied by these importance weights.
• 💡 Error Cost Handling Unchanged from Standard Accuracy: It's important to note that this 'Shifted Accuracy' still assigns a value of 0 to both types of errors (False Positives and False Negatives), meaning it doesn't inherently differentiate between the costs of these errors, similar to standard accuracy. The adjustment is purely for the shift in class prevalences, not for asymmetric error costs.

• ✅ Clear Structure: The 2x2 contingency table is a standard and clear way to present this type of value function.
• ✅ Intuitive Labeling: The use of a concrete example (Syphilis) for true states and predicted actions aids in understanding the context.
• ✅ Explicit Re-weighting Factors: The mathematical expressions in the cells, involving π and π₀, clearly demonstrate how the re-weighting is applied to adjust for the shift in class prevalence.
• 💡 Definition of π and π₀: The symbols π (target class prevalence) and π₀ (empirical class prevalence) are crucial. While defined in the text (π on page 3, π₀ on page 3), adding a brief footnote to the table defining them would improve its self-containedness. Suggestion: Add a footnote: "π represents the target/deployment class prevalence of y=1; π₀ represents the empirical class prevalence of y=1 in the evaluation dataset."

• Table Purpose: Value Function for Balanced Weighted Accuracy: Table 5 outlines a 'value function' for a metric called 'Balanced Weighted Accuracy'. In classification, a value function assigns a numerical score to each possible outcome. This table applies to binary classification (two possible outcomes) and shows how scores are assigned when considering both the balance of classes in the dataset and the differing costs or importance of various correct/incorrect predictions.
• Table Structure: True States vs. Predicted Actions: The table is structured as a 2x2 grid. The columns represent the true actual condition: 'y=1' (e.g., the patient has Syphilis) and 'y=0' (e.g., the patient does not have Syphilis). The rows represent the model's prediction or the action taken based on it: 'ŷ=1' (e.g., predict/treat for Syphilis) and 'ŷ=0' (e.g., predict/don't treat for Syphilis).
• Cell Values and Key Parameters (π₀, c): The values in the cells are defined using π₀ (pi-naught), which is the observed proportion of positive cases (y=1) in the evaluation dataset, and 'c', a parameter between 0 and 1 related to the relative costs or importance of outcomes. - True Positive (y=1, ŷ=1): Correctly identifying a positive case. The value assigned is (1-c)/π₀. - False Positive (y=0, ŷ=1): Incorrectly predicting positive when it's negative. The value is 0. - False Negative (y=1, ŷ=0): Incorrectly predicting negative when it's positive. The value is 0. - True Negative (y=0, ŷ=0): Correctly identifying a negative case. The value assigned is c/(1-π₀).
• Mechanism of Balancing and Weighting: This value function achieves 'balancing' by dividing by the class prevalences (π₀ for positives, 1-π₀ for negatives). This means that correctly identifying a case from a rare class gets a higher raw value, helping to ensure performance on minority classes isn't overlooked. It achieves 'weighting' through the 'c' parameter: (1-c) acts as a weight for the value of correctly identifying positives, and 'c' acts as a weight for the value of correctly identifying negatives. If, for example, correctly identifying positives is considered more important, '1-c' would be larger than 'c'. Misclassifications (False Positives and False Negatives) are assigned a value of 0 in this specific formulation.

• ✅ Correct Combination of Balancing and Cost-Weighting Principles: The formulation presented in the table correctly combines principles of class re-weighting (using inverse prevalence: 1/π₀ and 1/(1-π₀)) for balancing, and cost-based weighting (using 1-c and c) for correct classifications. This is a scientifically valid approach to constructing a value function for a metric that considers both aspects.
• ✅ Appropriateness for Balanced Weighted Accuracy: This value function is appropriate for defining a 'Balanced Weighted Accuracy' where the contributions of True Positives and True Negatives are adjusted both for their class sizes and for their relative importance as determined by the cost parameter 'c'. When averaged over the dataset Dπο, the sum of these values per instance yields (1-c)TPR + c*TNR, which is a standard form for weighted accuracy (where TPR is True Positive Rate, TNR is True Negative Rate).
• 💡 Relationship to Paper's BWA Definition (Scaling Factor): The paper's Definition B.4 for BWA is Σ_{x,y∈D1/2} (1 – c)^y c^(1-y) V1/2(Y, K(s(x), c)), which evaluates to 0.5 [(1-c)TPR + cTNR]. The value function in Table 5, when averaged over the original dataset Dπο, yields a sum of (1-c)TPR + c*TNR. Therefore, the value function in Table 5 defines a quantity that is twice the BWA as formulated in Definition B.4. This scaling factor of 2 is a nuance but does not invalidate the relative contributions defined by the value function.
• ✅ Logical Interpretation of Cost Parameter 'c': The interpretation of 'c' as a cost parameter is crucial. If 'c' is defined such that (1-c) reflects the importance or value of a True Positive and 'c' reflects the importance or value of a True Negative (consistent with contexts where c might be Cost(FP)/(Cost(FP)+Cost(FN))), then the weighting scheme is logical.
• ✅ Error Handling Approach: Assigning a value of 0 to both False Positives and False Negatives means that this specific value function does not directly penalize errors differently; rather, it values correct classifications differently based on class and cost, and achieves balance. This is a valid approach, distinct from directly assigning different negative values (losses) to different error types.

• ✅ Clear Structure: The 2x2 contingency table format is a standard and clear way to present the value function for different classification outcomes.
• ✅ Intuitive Labeling: The use of a concrete example (Syphilis) for true states (y=1, y=0) and predicted actions (ŷ=1, ŷ=0) aids in understanding the context of the classification task.
• ✅ Explicit Weighting Factors: The mathematical expressions in the cells clearly show how both cost-weighting (via 'c') and class balance adjustment (via π₀) are incorporated into the value of correct classifications.
• 💡 Definition of π₀ and 'c': The symbols π₀ (empirical positive class prevalence) and 'c' (cost/threshold parameter) are crucial for understanding the table. While likely defined elsewhere in the paper, adding a brief footnote to the table defining them would improve its self-containedness and immediate interpretability. Suggestion: Add a footnote: "π₀ represents the empirical prevalence of the positive class (y=1). 'c' is a cost parameter, where (1-c) can be seen as the relative importance/value of correctly identifying a positive case, and 'c' as the relative importance/value of correctly identifying a negative case."

• Table Purpose: Value Function for Weighted Accuracy: Table 6 defines a 'value function' for a metric called 'Weighted Accuracy'. A value function in classification assigns a numerical score to each possible outcome (e.g., correctly identifying a sick person). This table is for binary classification and specifies how scores are assigned when considering the differing importance or costs associated with correctly classifying positive versus negative cases, and also normalizing the metric.
• Table Structure: True States vs. Predicted Actions: The table is a 2x2 grid. Columns represent the true actual condition: 'y=1' (e.g., has Syphilis) and 'y=0' (e.g., No Syphilis). Rows represent the model's prediction or action: 'ŷ=1' (e.g., Treat for Syphilis) and 'ŷ=0' (e.g., Don't treat).
• Cell Values and Key Parameters (c, π₀, D): The values in the cells are defined using 'c' (a cost parameter, 0 < c < 1) and π₀ (pi-naught, the observed proportion of positive cases (y=1) in the evaluation dataset). The denominator in each cell, D = (1-c)π₀ + c(1-π₀), serves as a normalization factor. - True Positive (y=1, ŷ=1): Correctly identifying a positive case. Value = (1-c) / D. - False Positive (y=0, ŷ=1): Incorrectly predicting positive when it's negative. Value = 0. - False Negative (y=1, ŷ=0): Incorrectly predicting negative when it's positive. Value = 0. - True Negative (y=0, ŷ=0): Correctly identifying a negative case. Value = c / D.
• Mechanism of Weighting and Normalization: This value function achieves 'weighting' through the 'c' parameter: (1-c) acts as a weight for the value of correctly identifying positives, and 'c' acts as a weight for the value of correctly identifying negatives. For instance, if correctly identifying positives is deemed more important, '1-c' would be larger than 'c' (meaning 'c' is small). Misclassifications (False Positives and False Negatives) are assigned a value of 0. The denominator 'D' normalizes the metric so that a perfect classifier (one that gets everything right according to these weights) achieves a score of 1, regardless of the class balance π₀ or the cost 'c'. The text mentions this normalization: "normalized so that a perfect classifier achieves a score of 1 regardless of class balance."

• ✅ Correct Formulation for Normalized Weighted Accuracy: The table correctly represents a value function for Weighted Accuracy where the weights for true positives and true negatives are (1-c) and c respectively, and the result is normalized by the expected score of a perfect classifier under these weights and the empirical class distribution π₀. This normalization ensures the metric ranges from 0 to 1.
• ✅ Appropriateness for Weighted and Normalized Evaluation: This formulation is appropriate for evaluating classifiers when different types of correct classifications have different utilities or when aiming for a score that is comparable across different class balances due to the normalization.
• ✅ Correct Normalization Factor: The normalization factor D = (1-c)π₀ + c(1-π₀) is precisely the expected value achieved by a perfect classifier given the weights (1-c) for true positives and c for true negatives, and the empirical class distribution (π₀ for positives, 1-π₀ for negatives). Dividing by D ensures the maximum score is 1.
• ✅ Consistency with Paper's Definition of WA: This value function is consistent with the definition of Weighted Accuracy (WA) in Definition B.5 of the paper, where the sum of these per-instance values gives the overall WA score.
• ✅ Error Handling Focus: By assigning a value of 0 to both False Positives and False Negatives, this value function focuses on the weighted sum of correct classifications rather than differentially penalizing specific error types. The cost 'c' influences the relative value of a True Positive versus a True Negative.

• ✅ Clear Structure: The 2x2 contingency table format is a standard and clear way to present the value function.
• ✅ Intuitive Labeling: The use of a concrete example (Syphilis) for true states and predicted actions aids in understanding.
• ✅ Explicit Weighting and Normalization: The mathematical expressions in the cells clearly show how cost-weighting (via 'c') and normalization (via the denominator involving π₀ and 'c') are applied.
• 💡 Definition of π₀ and 'c': The symbols π₀ (empirical positive class prevalence) and 'c' (cost parameter) are central. While likely defined elsewhere, a brief footnote defining them would enhance the table's self-containedness. Suggestion: Add a footnote: "π₀ represents the empirical prevalence of the positive class (y=1). 'c' is a cost parameter, where (1-c) can be seen as the relative cost/importance of a false negative error, and 'c' as the relative cost/importance of a false positive error, in some formulations. Here, it weights the value of true positives and true negatives."

• Table Purpose: Value Function for Net Benefit: Table 7 defines a 'value function' for a metric called 'Net Benefit'. A value function in classification assigns a numerical score (benefit or cost) to each possible outcome. This table is for binary classification (two possible outcomes) and shows how these scores are assigned.
• Table Structure: True States vs. Predicted Actions: The table is a 2x2 grid. The columns represent the true actual condition of an individual: 'y=1' (e.g., has Syphilis) and 'y=0' (e.g., No Syphilis). The rows represent the model's prediction or the action taken: 'ŷ=1' (e.g., Treat for Syphilis) and 'ŷ=0' (e.g., Don't treat).
• Cell Values and Role of Cost Parameter 'c': The values in the cells represent the net benefit associated with each outcome, defined in terms of a parameter 'c' (typically a cost ratio or threshold, 0 < c < 1): - True Positive (y=1, ŷ=1): Correctly identifying a positive case. The value (benefit) is 1. - False Positive (y=0, ŷ=1): Incorrectly predicting positive when it's negative. The value (which is a cost or negative benefit) is -c/(1-c). This term represents the cost of a false positive relative to the benefit of a true negative (or, more directly, if 'c' is the probability threshold, this is the odds of a false positive being treated as equivalent to the loss of a true negative's benefit). - False Negative (y=1, ŷ=0): Incorrectly predicting negative when it's positive. The value (cost/negative benefit) is 0. In this specific formulation of Net Benefit focusing on the benefit of true negatives (as stated on page 5), the cost of a false negative is implicitly handled by the missed benefit of a true positive (which is 1). - True Negative (y=0, ŷ=0): Correctly identifying a negative case. The value (benefit) is 0. This formulation, as described on page 5,
• Specific Formulation Context: This specific formulation of Net Benefit, as stated in the paper (page 5),

• ✅ Consistency with Paper's Specific Net Benefit Definition: The table represents a specific formulation of Net Benefit. Standard Decision Curve Analysis (DCA) defines Net Benefit as (TP - FP (pt/(1-pt)))/N, where pt is the probability threshold. This table's values (1 for TP, -c/(1-c) for FP, 0 for FN, 0 for TN) are consistent with a definition where Net Benefit = TP_value P(TP) + FP_value P(FP) + TN_value P(TN) + FN_value P(FN). If c = pt, then -c/(1-c) = -pt/(1-pt). The paper's Definition 3.2 focuses on the benefit of true negatives, which is a variation. The table here is V(y,ŷ) for the term (c/(1-c))^(1-y) 1(y=k(s(x),τ)) in Definition 3.2. For y=1 (positive), the exponent is 0, so value is 1 if predicted positive (TP), 0 if predicted negative (FN). For y=0 (negative), the exponent is 1. If predicted positive (FP), value is -c/(1-c). If predicted negative (TN), value is 0. This is consistent with a net benefit calculation where true positives yield a benefit of 1, false positives incur a cost of c/(1-c), and both false negatives and true negatives yield zero direct benefit/cost in this particular additive formulation focusing on benefit over a default of 'treat none' and comparing to 'treat all'. The paper states on page 5: "We use a variation of net benefit that focuses on the benefit of true negatives rather than the costs of false positives in order to be more directly comparable to accuracy." This table then seems to be a component of that variation, rather than the more common Net Benefit. The values in Table 7 are: TP=1, FP=0, FN=0, TN=c/(1-c). This is different from the description above. Let's re-evaluate based on the table's actual content. TP=1, FP=0, FN=0, TN = c/(1-c). This formulation is unusual for 'Net Benefit' which typically penalizes FPs. This seems to be a 'Weighted Accuracy' focused on TPs and TNs, where TNs are weighted by c/(1-c). The text for Definition 3.2 states Net Benefit(D,s,τ,c) = (1/|D|) Σ (c/(1-c))^(1-y) * 1(y=κ(s(x),τ)). If y=1, (c/(1-c))^0 = 1. So TP contributes 1. If y=0, (c/(1-c))^1 = c/(1-c). So TN contributes c/(1-c). All errors (FP, FN) contribute 0 because 1(y=κ(s(x),τ)) is 0 for them. This matches the table values.
• ✅ Appropriateness for Defined Variation: This specific formulation (TP=1, TN=c/(1-c), errors=0) is indeed a variation. It measures a weighted sum of correct classifications, where the weight of a True Negative is c/(1-c) relative to a True Positive's weight of 1. This is appropriate if the goal is to define a benefit metric that scales the value of true negatives based on the cost parameter 'c'.
• ✅ Logical Use of Cost Parameter 'c': The parameter 'c' is typically related to a decision threshold or a cost ratio. If 'c' is the threshold probability for action, then c/(1-c) represents the odds. The value function correctly uses 'c' to modulate the benefit of a true negative relative to a true positive.
• 💡 Non-Standard Formulation (Acknowledged by Authors): This definition of Net Benefit is explicitly stated as a variation by the authors to be "more directly comparable to accuracy" by focusing on positive contributions. While different from some standard DCA formulations that might show negative values for FPs, it is internally consistent with the authors' Definition 3.2.

• ✅ Clear Structure: The 2x2 contingency table format is a standard and clear way to present the value function for different classification outcomes.
• ✅ Intuitive Labeling: The use of a concrete example (Syphilis) for true states and predicted actions aids in understanding the context.
• ✅ Clear Value Definitions: The values in the cells (1, 0, and c/(1-c)) clearly define the benefit/cost structure for this version of Net Benefit.
• 💡 Definition of 'c': The parameter 'c' (cost ratio) is central to this table. While likely defined elsewhere in the paper (e.g., page 5, Definition 3.2), a brief footnote defining 'c' or its interpretation in this context (e.g., related to the trade-off between false positives and false negatives, or the threshold for decision making) would enhance the table's self-containedness. Suggestion: Add a footnote: "'c' represents a cost parameter or decision threshold, where c/(1-c) is the cost ratio of a false positive to a true negative's benefit (or relative cost of a false positive to the benefit of avoiding it)."

• Table Purpose: Value Function for PAMWA: Table 8 specifies a 'value function' for a metric called 'Prior-Adjusted Maximum Weighted Accuracy' (PAMWA). This value function assigns a numerical score to each of the four possible outcomes in a binary classification task (True Positive, False Positive, False Negative, True Negative). The metric aims to evaluate model performance while simultaneously accounting for (1) a shift in class prevalence from an evaluation dataset (with positive class prevalence π₀) to a target deployment setting (with positive class prevalence π), (2) differing costs or importance of correctly classifying positives versus negatives (via a cost parameter 'c'), and (3) ensuring balanced consideration of performance on both classes.
• Table Structure and Parameters: The table is a 2x2 grid where columns represent the true class (1 for positive, 0 for negative) and rows would represent the predicted class (though row labels are missing, they are implicitly ŷ=1 and ŷ=0). The cell entries define the value for each outcome based on π, π₀, and 'c'. Let D = (1-c)π + c(1-π) be a normalization factor.
• Cell Values for Outcomes: The values assigned are: - True Positive (y=1, predicted ŷ=1): Value = (1-c)π / (2π₀ D). This rewards correctly identifying a positive case, weighted by (1-c), adjusted for the target prevalence π, normalized by D, and further adjusted by the empirical prevalence π₀ with a balancing factor of 1/2. - False Positive (y=0, predicted ŷ=1): Value = 0. - False Negative (y=1, predicted ŷ=0): Value = 0. - True Negative (y=0, predicted ŷ=0): Value = c(1-π) / (2(1-π₀) D). This rewards correctly identifying a negative case, weighted by 'c', adjusted for the target prevalence of negatives (1-π), normalized by D, and further adjusted by the empirical prevalence of negatives (1-π₀) with a balancing factor of 1/2.
• Combined Adjustments in the Value Function: This value function essentially combines several adjustments: cost-weighting ((1-c) for TPs, c for TNs, applied to target prevalences), label shift adjustment (implicit in using π and π₀), a balancing factor (the '2' in the denominator, common in balanced accuracy type metrics), and normalization (division by D). The goal is to create a sophisticated metric that reflects performance under specific target conditions and cost assumptions.

• ✅ Sound Combination of Methodological Principles: The formulation combines importance weighting for label shift (from π₀ to π), cost-weighting for differential utility of true positives vs. true negatives (via 'c'), and a balancing component (the factor of 1/2 and division by source class prevalence π₀ or 1-π₀). This multi-faceted approach is scientifically sound for constructing a comprehensive evaluation metric tailored to specific deployment conditions and cost structures.
• ✅ Correct Normalization Factor for Weighted Accuracy Component: The normalization factor D = (1-c)π + c(1-π) correctly represents the expected weighted sum of correct classifications for a perfect classifier under the target prevalence π and cost weights (1-c, c). Dividing by D ensures that the weighted accuracy component is scaled to a maximum of 1 before other adjustments.
• ✅ Consistency with PAMWA Metric Goal: This value function, when applied to instances from the empirical dataset Dπο and summed, should yield the PAMWA score as intended by Definition B.7. The structure of the terms ( (Weighted value in target) * (Balancing/Importance weight from source) / (Target Normalization) ) is consistent with deriving such a prior-adjusted, weighted, and balanced metric.
• ✅ Error Handling Approach (Focus on Positive Contributions): Assigning a value of 0 to both False Positives and False Negatives means that, like other accuracy-based metrics in this paper, errors are not directly penalized with negative values in this specific value function; rather, the focus is on the positively valued contributions of correct classifications, adjusted in multiple ways.
• 💡 Sensitivity to Parameter Specification (π, c): The complexity of the metric, while comprehensive, relies heavily on the accurate specification of π (target prevalence) and 'c' (cost parameter). Misspecification of these parameters could lead to misleading evaluations. The validity of the PAMWA score in practice depends on the reliability of these inputs.

• ✅ Clear Structure: The 2x2 contingency table format is standard for defining value functions in binary classification, making the structure familiar.
• ✅ Comprehensive Scope: The table attempts to show the combined effects of prior adjustment (for label shift), cost-weighting, and balancing. However, the resulting expressions are quite complex.
• 💡 Parameter and Normalization Factor Definition: The cell values involve multiple parameters (π, π₀, c) and a derived normalization factor D (which is (1-c)π + c(1-π)). For clarity, it would be beneficial to explicitly define D in a footnote or directly within the table if space allowed. Suggestion: Add a footnote: "π is the target positive class prevalence, π₀ is the empirical positive class prevalence, c is the cost parameter. D = (1-c)π + c(1-π) is a normalization factor based on the target distribution and costs."
• 💡 Incomplete Row/Column Labeling: The column headers 'V', '1', '0' are terse. While '1' and '0' likely refer to true positive (y=1) and true negative (y=0) states, expanding 'V' to 'V(y,ŷ)' or similar, and explicitly labeling columns as 'y=1' and 'y=0' (as in other tables) would improve clarity. The row labels for predictions (ŷ=1, ŷ=0) are missing entirely. Suggestion: Add row labels for predictions (e.g., ŷ=1, ŷ=0) and ensure column labels clearly denote true states.
• 💡 Cross-referencing to Text: Given the complexity of the expressions, ensuring these are cross-referenced clearly with their derivation or definition (Definition B.7 for PAMWA) in the main text is crucial for reader comprehension.

• Table Purpose: Value Function for PAMNB: Table 9 defines a 'value function' for a metric called 'Prior-Adjusted Maximum Net Benefit' (PAMNB). In the context of evaluating classification models, a value function assigns a numerical score (representing benefit or cost) to each of the four possible outcomes of a binary prediction: True Positive, False Positive, False Negative, and True Negative.
• Table Structure: True States vs. Predicted Actions: The table is laid out as a 2x2 grid. The columns represent the true actual condition of an individual: 'y=1' (e.g., the individual has Syphilis) and 'y=0' (e.g., the individual does not have Syphilis). The rows represent the model's prediction or the action taken based on that prediction: 'ŷ=1' (e.g., predict/treat for Syphilis) and 'ŷ=0' (e.g., predict/don't treat for Syphilis).
• Cell Values and Key Parameters (π, π₀, c): The values within the cells of the table define the score for each outcome, based on three key parameters: π (pi, the target or deployment prevalence of the positive class), π₀ (pi-naught, the empirical or observed prevalence of the positive class in the evaluation dataset), and 'c' (a cost parameter or decision threshold, typically between 0 and 1). - True Positive (y=1, ŷ=1): Correctly identifying a positive case. The value assigned is π/π₀. - False Positive (y=0, ŷ=1): Incorrectly predicting positive when the true state is negative. The value assigned is 0. - False Negative (y=1, ŷ=0): Incorrectly predicting negative when the true state is positive. The value assigned is 0. - True Negative (y=0, ŷ=0): Correctly identifying a negative case. The value assigned is (c/(1-c)) * ((1-π)/(1-π₀)).
• Mechanism: Combining Label Shift Adjustment and Cost-Sensitivity for True Negatives: This value function for PAMNB combines adjustments for label shift (the π/π₀ and (1-π)/(1-π₀) terms, which re-weight outcomes based on changes in class prevalence between evaluation and target settings) with a cost-sensitive valuation of True Negatives (the c/(1-c) term). Specifically, a True Positive is valued based on the ratio of target to empirical positive prevalence. A True Negative's value is scaled by the odds c/(1-c) and then by the ratio of target to empirical negative prevalence. Both types of errors (False Positives and False Negatives) are assigned a value of 0 in this particular formulation.

• ✅ Correct Formulation for PAMNB: The table correctly represents the value function for Prior-Adjusted Maximum Net Benefit (PAMNB) as implied by Definition 3.4 in the paper. The terms π/π₀ and (1-π)/(1-π₀) are standard importance weights for label shift correction. The term c/(1-c) introduces cost-sensitivity, specifically to the valuation of True Negatives.
• ✅ Appropriateness for Target Application: This value function is appropriate for scenarios where one needs to evaluate a classifier's net benefit under a specific target class prevalence (π) different from the evaluation set's prevalence (π₀), and where the benefit of a True Negative is scaled by a cost-related factor c/(1-c) relative to the (label-shift-adjusted) benefit of a True Positive.
• ✅ Alignment with Paper's Goals: The formulation is consistent with the authors' stated aim of developing metrics that are sensitive to clinical deployment conditions, including distributional shifts (label shift) and asymmetric error costs (handled here by valuing TNs differently based on 'c').
• ✅ Specific Approach to Error Valuation: Assigning a value of 0 to both False Positives and False Negatives means that this particular Net Benefit formulation focuses on the positive contributions of correct classifications, adjusted for label shift and cost-weighting of TNs, rather than directly assigning negative values (penalties) for errors. This is a specific choice in defining 'benefit'.
• 💡 Dependence on Accurate Parameter Specification (π, c): The practical utility and interpretation of the PAMNB score derived from this value function will heavily depend on the accurate estimation or specification of the target prevalence π and the cost parameter 'c'. Sensitivity analyses for these parameters would be important in real-world applications.

• ✅ Clear Structure: The 2x2 contingency table format is a standard and clear method for presenting value functions in binary classification contexts.
• ✅ Intuitive Labeling: The use of a concrete example (Syphilis) for defining true states (y=1, y=0) and predicted actions (ŷ=1, ŷ=0) helps in making the table's context more intuitive.
• ✅ Explicit Parameter Usage: The mathematical expressions in the cells explicitly show how adjustments for target prevalence (π), empirical prevalence (π₀), and cost/threshold parameter (c) are incorporated into the value function.
• 💡 Definition of Parameters (π, π₀, c): The parameters π (target positive class prevalence), π₀ (empirical positive class prevalence), and 'c' (cost parameter/threshold) are fundamental to understanding the table. While these are likely defined in the main text of the paper, adding a brief footnote to the table that defines these symbols would significantly improve its self-containedness and immediate comprehensibility for readers. Suggestion: Include a footnote such as: "π represents the target/deployment positive class prevalence; π₀ is the empirical positive class prevalence in the evaluation dataset; 'c' is a cost parameter or decision threshold."

(C) African American patients (orange) have noticeably better AUC- ROC than Caucasian patients (blue). However, we can decompose the difference in accuracy into (A) a difference in mechanism of prediction at equal class balances (i.e. same in-hospital mortality) and (B) a difference in the class balance at which accuracy is evaluated for the two groups.

• Graph Axes and Range: The graph displays "Accuracy" on its vertical axis, ranging from approximately 0.86 to 1.00, plotted against "Positive Class Fraction" on its horizontal axis, which spans from 0.01 to 0.10. The "Positive Class Fraction" likely represents the proportion of instances classified as positive by the model at varying decision thresholds, or the prevalence of the positive class in different data segments.
• Comparative Curves: Two distinct curves are shown: an orange line representing data for African American patients and a blue line for Caucasian patients. Across the depicted range of positive class fractions, the orange line is generally positioned above the blue line, suggesting higher accuracy for the model's predictions concerning African American patients compared to Caucasian patients.
• Overall AUC-ROC Finding (C): The caption indicates an overall finding (labeled C) that African American patients exhibit a noticeably better AUC-ROC. AUC-ROC, or Area Under the Receiver Operating Characteristic curve, is a common metric evaluating a model's ability to distinguish between positive and negative classes across all decision thresholds; a higher value (closer to 1.0) signifies better discriminatory power.
• Decomposition Component (A): Mechanism Difference: The graph visually decomposes the observed accuracy difference. Component (A), described as "a difference in mechanism of prediction at equal class balances (i.e. same in-hospital mortality)," is marked on the graph as a vertical distance between the orange and blue curves at a specific positive class fraction (around 0.04). This highlights that at a comparable positive class fraction, the model achieves higher accuracy for African American patients.
• Decomposition Component (B): Class Balance Difference: Component (B), termed "a difference in the class balance at which accuracy is evaluated for the two groups," is depicted as a horizontal segment between points on the two curves. This suggests that the groups might be evaluated under different prevailing class balances, or that a similar level of accuracy is achieved at different positive class fractions for the two groups.

• ✅ Decomposition Approach: The attempt to decompose overall performance differences (like AUC-ROC or accuracy) into more granular components related to prediction mechanisms and class balance effects across subgroups is a valuable analytical approach, potentially offering deeper insights than a single aggregate metric.
• 💡 Link Between Plotted Accuracy and AUC-ROC: The caption states that point (C) refers to African American patients having "noticeably better AUC-ROC," yet the graph plots "Accuracy" against "Positive Class Fraction." The precise mathematical and conceptual linkage between this specific accuracy plot and the overall AUC-ROC claim is not explicitly detailed. While accuracy at various thresholds forms the basis of an ROC curve, the figure itself is not an ROC curve. This ambiguity could affect the interpretation of how the decomposition directly explains AUC-ROC differences.
• 💡 Operationalization of "Positive Class Fraction": The x-axis term "Positive Class Fraction" and its operationalization in this context (e.g., is it based on true prevalence, predicted prevalence, or tied to specific threshold settings?) requires clearer definition to fully assess the validity of the decomposition, particularly for component (B) related to "class balance at which accuracy is evaluated."
• 💡 Rigor of Decomposition Definitions: The conceptual distinction and empirical representation of component (A) "difference in mechanism of prediction" (shown as a vertical gap) and (B) "difference in class balance" (shown as a horizontal segment) need rigorous definition. For instance, it's not immediately clear how the visual representation of 'A' isolates only the "mechanism of prediction" independent of other factors, or how 'B' precisely quantifies the impact of differing class balances from this plot.
• 💡 Lack of Statistical Significance Assessment: The reference text and caption claim a "noticeably better" AUC-ROC and show higher accuracy curves. However, the graph lacks error bars, confidence intervals, or any reported statistical tests. Without these, it is not possible to determine if the observed differences between the groups are statistically significant or could be due to chance, especially if sample sizes for the subgroups differ or are small.
• 💡 Justification for X-axis Range: The analysis is presented for a specific range of "Positive Class Fraction" (0.01 to 0.10). The rationale for selecting this particular range and its relevance to the clinical context (e.g., typical prevalence rates for in-hospital mortality, or clinically meaningful threshold ranges) should be provided to support the generalizability and applicability of the findings.
• 💡 Contextual Details for Standalone Interpretation: The paper mentions this analysis is on the eICU dataset for in-hospital mortality predictions using APACHE IV scores (page 11). While this context is available elsewhere, for the element to stand alone or be fully assessed, key details about the dataset, patient population, and specific model being evaluated would ideally be briefly summarized with the figure itself or in its immediate caption.

• ✅ Color Usage: The use of distinct colors (orange for African American patients, blue for Caucasian patients) effectively differentiates the two groups being compared.
• ✅ Direct Labeling: The direct labeling of components (A and B) on the graph helps connect visual features to the explanations provided in the caption, aiding interpretation.
• 💡 Accuracy vs. AUC-ROC Clarity: The primary source of confusion is the relationship between the plotted "Accuracy" (y-axis) and the "AUC-ROC" metric prominently mentioned in component (C) of the caption and the reference text. The graph itself is not an ROC curve. If this plot is intended to illustrate factors contributing to the AUC-ROC difference, this connection needs to be explicitly and clearly articulated. Suggestion: Clarify if this graph represents accuracy values at different decision thresholds (where "Positive Class Fraction" might relate to predicted positive rates or prevalence in evaluated segments) which then contribute to the overall AUC-ROC, or if it's a related but distinct accuracy analysis.
• 💡 Caption Structure: The caption is quite dense and combines the main finding (C) with the decomposition (A, B). Suggestion: Improve readability by stating the main AUC-ROC finding (C) first, then clearly explaining how the plotted accuracy differences, visualized as (A) and (B), contribute to understanding this overall performance difference between the groups.
• 💡 X-axis Scale Clarity: The x-axis, labeled "Positive Class Fraction," shows tick marks at 0.01 and 0.10. The spacing of intermediate, unlabeled ticks suggests a non-linear scale, possibly logarithmic. This should be explicitly stated if it is a log scale to avoid misinterpretation of the rate of change or distribution. Suggestion: Clearly indicate the scale of the x-axis (e.g., "Positive Class Fraction (log scale)").
• 💡 Explicit Legend: While the colors are explained in the caption, adding an explicit legend directly on the graph (e.g., "African American", "Caucasian") would enhance immediate comprehension without requiring the reader to refer back to the text. Suggestion: Include an embedded legend in the graph.
• 💡 Context of Parenthetical Remark: The caption for point (A) includes "(i.e. same in-hospital mortality)". This parenthetical remark implies that "Positive Class Fraction" might be linked to actual mortality rates (prevalence) or that the comparison is made at points of equivalent outcome prevalence. This connection should be made clearer in the main explanation of the axis or the decomposition. Suggestion: Elaborate on the meaning of "Positive Class Fraction" and how it relates to "in-hospital mortality" in the context of this graph.

(C) African American patients (orange) have noticeably better AUC- ROC than Caucasian patients (blue). However, we can plot the accuracy of a perfectly recalibrated model (dashed lines), and then de- compose the average accuracy using the calibration-sharpness frame- work [80, 28].

• Graph Axes and Plotted Metric: The graph presents "Accuracy" on the vertical axis (ranging approximately from 0.86 to 1.00) against "Positive Class Fraction" on the horizontal axis (from 0.01 to 0.10). "Positive Class Fraction" likely refers to the proportion of instances identified as positive, possibly varying with decision thresholds or representing different data segments by prevalence.
• Plotted Lines: Original vs. Recalibrated Models: Four lines are plotted: Solid orange and blue lines represent the accuracy of the original predictive model for African American and Caucasian patients, respectively. Dashed orange and blue lines show the accuracy of a "perfectly recalibrated" version of the model for the same respective groups. A recalibrated model is one whose predicted probabilities are adjusted to better match observed frequencies.
• Overall Performance (C) and AUC Labels: The caption and an annotation (C) indicate that African American patients (orange) have a noticeably better overall AUC-ROC than Caucasian patients (blue). AUC-ROC (Area Under the Receiver Operating Characteristic curve) is a measure of a model's ability to distinguish between classes; a higher value is better. On the graph, labels "AUC" point to the orange curves (near accuracy ~0.94-0.95) and blue curves (near accuracy ~0.91-0.92), presumably indicating these overall AUC-ROC values.
• Miscalibration Loss (A and B): The graph visually decomposes accuracy based on a calibration-sharpness framework. The gap labeled 'A' represents the miscalibration loss for African American patients (the difference between the solid orange line and the higher dashed orange line). This gap 'A' is visibly larger than the gap 'B', which represents the miscalibration loss for Caucasian patients (difference between the solid blue line and the slightly higher dashed blue line).
• Model Sharpness: The "sharpness" of the model for each group can be inferred from the height of the dashed (recalibrated model) lines. The dashed orange line (recalibrated model for African Americans) is generally higher than the dashed blue line (recalibrated model for Caucasians), suggesting the model has the potential for sharper (more confident and accurate when perfectly calibrated) predictions for African American patients.
• Comparative Accuracy of Original Models: Overall, the solid orange line (original model, African Americans) is mostly above the solid blue line (original model, Caucasians), indicating higher observed accuracy for African Americans across the shown range of positive class fractions, despite a larger miscalibration loss ('A').

• ✅ Decomposition Framework: The approach of decomposing model performance into calibration and sharpness components is a methodologically sound way to gain deeper insights beyond aggregate metrics like AUC-ROC or overall accuracy. Plotting the performance of a "perfectly recalibrated model" provides a clear theoretical baseline for evaluating calibration loss.
• 💡 Link Between Accuracy Decomposition and AUC-ROC Claim: The caption states that point (C) refers to AUC-ROC differences, and the graph includes "AUC" labels. However, the y-axis plots "Accuracy." While accuracy across thresholds underlies AUC-ROC, the direct quantitative link showing how the decomposition of this specific accuracy plot explains the differences in overall AUC-ROC values is not explicitly established by the figure and its immediate caption. This requires careful articulation.
• ✅ Support for Calibration/Sharpness Claims (with external text): The interpretation of "sharpness" (from the height of dashed lines) and "miscalibration loss" (gaps A and B) is crucial. The text accompanying this figure (page 11, paragraph 2) clarifies these interpretations, aligning with the visual representation. The figure, in conjunction with this text, effectively supports the claim of higher sharpness but worse calibration for the African American patient group with this model.
• 💡 Definition of "Positive Class Fraction": The term "Positive Class Fraction" on the x-axis needs a precise operational definition within this context (e.g., is it related to true class prevalence, predicted positive rates at varying thresholds, or something else?). This definition is critical for interpreting the accuracy variations and the meaning of the decomposition across this axis.
• 💡 Absence of Statistical Significance Assessment: The figure lacks error bars, confidence intervals, or any reported statistical tests for the plotted accuracy curves or the derived AUC values and decomposition components (A, B, sharpness differences). Without these, it's impossible to assess whether the observed differences are statistically significant or could be attributable to sampling variability, especially if subgroup sample sizes are disparate or small.
• 💡 Clarity on "Perfectly Recalibrated Model": The meaning of "perfectly recalibrated model" should ideally be briefly clarified (e.g., a model whose predicted probabilities match observed frequencies across all score ranges). The method for achieving this recalibration (e.g., isotonic regression, Platt scaling) could also be mentioned if it impacts the interpretation of the dashed lines.
• 💡 Justification for X-axis Range: The choice of the x-axis range (0.01 to 0.10 for Positive Class Fraction) should be justified in terms of its relevance to the clinical problem (e.g., typical prevalence of in-hospital mortality, or range of clinically relevant decision thresholds).

• ✅ Color and Line Style: The use of distinct colors (orange for African American patients, blue for Caucasian patients) and line styles (solid for original model, dashed for recalibrated model) effectively differentiates the groups and model states.
• ✅ Visual Annotations (A, B): The annotations 'A' and 'B' on the graph visually highlight the differences between the original and recalibrated model accuracies for the two patient groups, aiding in understanding the concept of miscalibration loss.
• 💡 Accuracy vs. AUC-ROC Clarity: The graph plots "Accuracy" on the y-axis, while the primary finding (C) and the overlaid "AUC" labels refer to "AUC-ROC." This can be confusing. The caption mentions decomposing "average accuracy," but the relationship to the AUC-ROC values needs to be explicitly bridged. Suggestion: Clarify in the caption how the plotted accuracy decomposition (into calibration loss via A/B and sharpness via dashed lines) helps explain or contextualize the observed AUC-ROC differences (C).
• 💡 Placement of "AUC" Labels: The labels "AUC" placed directly on the graph near the curves, presumably representing the overall AUC-ROC values for each group, are somewhat unconventional on an accuracy plot. Suggestion: State these AUC-ROC values explicitly in the caption or a legend entry for better clarity and to avoid potential misinterpretation as specific accuracy points.
• 💡 X-axis Definition and Scale: The x-axis, labeled "Positive Class Fraction," ranges from 0.01 to 0.10. Its specific meaning (e.g., true prevalence, predicted positive rate at varying thresholds) and the nature of its scale (linear or logarithmic, as suggested by tick spacing) should be clearly defined. Suggestion: Specify the scale of the x-axis and provide a brief explanation of what "Positive Class Fraction" represents in this context.
• 💡 Explicit Legend: While the groups are color-coded, adding an explicit legend within the graph (e.g., "African American (Original)", "African American (Recalibrated)", etc.) would enhance immediate comprehension and make the figure more self-contained.
• 💡 Caption Structure and Flow: The caption is dense and introduces multiple concepts (AUC-ROC difference, recalibrated models, calibration-sharpness framework). Suggestion: Structure the caption to first state the overall AUC-ROC finding (C), then clearly introduce the concept of recalibrated models, and finally explain how the visual elements (A, B, and dashed lines) represent the calibration and sharpness components of the decomposition.

Figure 1: Causal diagrams: (1) shows differences in performance are based both on label shift and differences in mechanism between subgroups (2) if we can intervene on Y to hold it constant, then we can measure differences in performance based purely on differing performance of the model between subgroups.

• Overall Structure: Two Causal Diagrams: Figure 1 presents two conceptual diagrams, labeled (1) and (2), which are types of causal diagrams. Causal diagrams use nodes (represented by letters D, Y, X, K) to denote variables or factors, and arrows to show presumed causal influences between them. The direction of the arrow indicates the direction of influence.
• Diagram (1): Combined Effects: Diagram (1) illustrates a scenario where a factor 'D' (likely representing different data distributions or subgroups) influences both 'Y' (perhaps the true outcome or label) and 'X' (likely the input features for a model). 'Y' also influences 'X'. Finally, 'X' influences 'K' (possibly the model's prediction or performance). According to the caption, this diagram represents a situation where performance differences arise from both 'label shift' (changes in the distribution of Y, influenced by D) and differences in 'mechanism' (how X is generated or how X relates to K, also potentially influenced by D via its effect on X).
• Diagram (2): Effect of Intervention on Y: Diagram (2) shows a modified scenario. Here, 'D' still influences 'Y', but the influence of 'D' directly on 'X' is absent. The relationship Y -> X and X -> K remains. The key difference is the introduction of a 'do(Y)' operator applied to 'Y'. The 'do-operator' is a concept from causal inference signifying an intervention where the variable 'Y' is forced to take on a specific value, or its distribution is set, independently of its usual causes (like D, in terms of its direct path to X). The caption states this diagram represents measuring performance differences based purely on the model's differing performance between subgroups when 'Y' is held constant through intervention.
• Comparative Purpose: The comparison between diagram (1) and (2) is intended to conceptually separate the sources of performance differences: diagram (1) shows the combined effects, while diagram (2) isolates the effect of the model's behavior (X -> K) under a controlled 'Y', thereby removing the confounding influence of 'D' on 'X' that isn't mediated through 'Y'.

• ✅ Appropriateness of Causal Diagrams: The use of directed acyclic graphs (DAGs) as causal diagrams is a standard and appropriate method for representing assumed causal relationships and for reasoning about concepts like confounding, intervention, and decomposition of effects. They provide a formal language for the concepts discussed.
• ✅ Representation of Combined Effects: Diagram (1) effectively illustrates a scenario where a common cause 'D' (subgroup/distribution) can influence both the label distribution 'Y' (contributing to label shift) and the feature distribution 'X' or the feature-outcome relationship (contributing to mechanism differences affecting 'K'). The path D -> Y -> X -> K and D -> X -> K shows these combined influences.
• ✅ Representation of Intervention: Diagram (2) correctly uses the 'do(Y)' notation to represent an intervention on 'Y'. This intervention, by definition, removes incoming arrows to 'Y' from its natural causes if 'Y' were being set exogenously, or more relevantly here, it isolates the downstream effects from Y to K, effectively allowing comparison of X->K across subgroups 'D' under a common distribution of 'Y'. This is a valid way to conceptualize isolating performance differences not attributable to label shift.
• 💡 Abstract Nature and Dependence on Variable Definitions: The diagrams are abstract and rely on the reader understanding the mapping of D, Y, X, K to specific concepts in the authors' model evaluation framework. While the caption provides high-level interpretation, the scientific validity of the conclusions drawn from these specific causal structures would depend on the precise definitions of these variables and the justification for the depicted arrows (or their absence) in the context of the problem being studied (e.g., model fairness, robustness to distribution shift). The diagrams themselves are plausible causal models for the described scenarios.
• 💡 Specificity of 'Mechanism Differences': The term 'mechanism between subgroups' in the caption for diagram (1) is broad. The diagram shows D influencing X. This could represent differences in P(X|D) or P(X|Y,D). The diagram D->X is a direct path, while D->Y->X is an indirect path. Clarity on what specific 'mechanism' (e.g., feature distribution P(X|D), or conditional feature distribution P(X|Y,D), or model P(K|X,D)) is being referred to would strengthen the link between the diagram and the concept.
• ✅ Qualitative Conceptual Framework: The diagrams are qualitative. They set up a conceptual framework but do not, by themselves, provide quantitative evidence or test specific hypotheses. Their validity lies in their ability to correctly frame the problem for subsequent quantitative analysis, which seems to be their intended role here.

• ✅ Simplicity of Notation: The use of simple, standard notation for nodes (letters) and directed arrows makes the causal relationships easy to follow at a basic level.
• ✅ Clear Distinction of Diagrams: Numbering the diagrams (1) and (2) clearly distinguishes between the two scenarios being presented, which is helpful for comparative understanding.
• ✅ Informative Caption: The caption clearly explains the purpose of each diagram, linking them to the concepts of label shift, mechanism differences, and intervention.
• 💡 Undefined Nodes: The meaning of the nodes D, Y, X, and K is not defined within the figure or its immediate caption. While likely defined elsewhere in the text (e.g., D for dataset/distribution, Y for true label/outcome, X for features, K for model performance/prediction), their absence here reduces the figure's self-containedness. Suggestion: Add a brief legend or key defining D, Y, X, and K directly with the figure.
• 💡 Clarity of 'do(Y)' Notation: The 'do(Y)' notation in diagram (2) is specific to causal inference and might not be immediately understood by all readers. Suggestion: While the caption explains its implication (intervening on Y), a brief parenthetical explanation of 'do(Y)' (e.g., "setting Y to a fixed value") could improve accessibility for a broader audience.
• 💡 Visual Emphasis of Intervention Effect: The visual distinction between D influencing Y and X in diagram (1) versus only Y in diagram (2) after the 'do' operation could be made more salient. The core change is the removal of the D->X arrow implicitly by the intervention on Y. Suggestion: Consider subtly highlighting the arrows that are present/absent between the two diagrams to emphasize the effect of the intervention, perhaps with a brief note.
• 💡 Figure Size and Legibility: The diagrams are quite small. Suggestion: Ensure the figure is rendered at a sufficient size in the final publication so that all labels and arrows are clearly legible without strain.

Figure 2: The label shift effect is quite dramatic on the public subsample of EICU. In fact a great deal of this results from small sample size.