Paper Review: Think Twice Before Trusting: Mitigating Over-Trust in LLM Self-Detection

Overview

This research paper addresses the issue of hallucination in Large Language Models (LLMs) and proposes a new self-detection paradigm called "Think Twice Before Trusting" (T3) to mitigate over-trust in incorrect LLM-generated answers. The T3 framework considers a broader answer space, prompting the LLM to reflect on and justify multiple candidate answers before calibrating confidence in the target answer. Extensive experiments across various tasks and datasets demonstrate the effectiveness of this approach in improving self-detection and reducing over-trust.

Key Findings

• The T3 framework consistently outperforms existing self-detection methods in terms of AUROC, PRAUC, and ECE across various datasets and tasks, including Sentiment Analysis, Natural Language Inference, and Commonsense Question Answering.
• Ablation studies confirm the importance of each component of the T3 framework, including reflection and justification, joint confidence calibration, and order shuffling of justifications.
• T3 demonstrates robustness across different target answers, LLMs (GPT-3.5, GLM-4, Gemini), and parameter settings, indicating its generalizability and flexibility.
• The T3 framework shows potential for selective prediction scenarios, where abstaining from answers with low detection scores leads to improved accuracy on the remaining instances.
• Case studies illustrate how T3 helps LLMs identify and reduce over-trust in incorrect answers by considering justifications from different answer perspectives and reflecting uncertainty when appropriate.

Strengths

• The paper clearly identifies the over-trust issue in LLM self-detection and proposes a novel paradigm to address it, considering a broader answer space and leveraging LLM's reflection and justification capabilities.
• The proposed T3 framework is well-defined, with clear steps and justifications for its design choices. The framework is also flexible and can be integrated with existing self-detection methods.
• The experimental evaluation is comprehensive, covering various tasks, datasets, and LLMs. The inclusion of ablation studies and robustness analysis further strengthens the validity of the findings.
• The paper provides detailed implementation details, including prompts and hyperparameters, facilitating reproducibility and future research.
• The paper acknowledges limitations and ethical considerations, promoting transparency and responsible development of LLM self-detection techniques.

Areas for Improvement

• While the paper focuses on black-box API LLMs, exploring the applicability of the T3 framework to white-box LLMs could broaden its impact and address potential limitations.
• Further investigation into the practical utility of self-detection for enhancing task accuracy and enabling LLM self-correction would strengthen the practical implications of the research.
• A more in-depth analysis of the generated justifications, including potential biases and limitations, would provide a deeper understanding of the LLM's reasoning process and inform future improvements.

Significant Elements

• Figure 4: Box plots visualizing the bias mitigation effect of T3, showing reduced detection score overlaps between correct and incorrect instances across various datasets.
• Table 2: Comprehensive results table comparing T3 with various self-detection methods across different datasets and tasks, showcasing its superior performance in terms of AUROC, PRAUC, and ECE.

Conclusion

This research presents a significant advancement in addressing the over-trust issue in LLM self-detection. The proposed T3 framework, based on a comprehensive answer evaluation paradigm, effectively mitigates over-trust and improves self-detection performance across various tasks and datasets. While limitations and future research directions are acknowledged, the findings contribute valuable insights and a promising approach for enhancing the reliability and trustworthiness of LLM outputs. Further exploration of this approach could lead to more robust and reliable LLM applications in various domains.

Summary

The abstract introduces the issue of hallucination in Large Language Models (LLMs), where LLMs generate incorrect or nonsensical outputs. It proposes a new self-detection paradigm to address this, moving beyond simply evaluating LLM-generated answers. This paradigm, called "Think Twice before Trusting" (T3), involves considering a broader answer space and comparing the trustability of multiple candidate answers. The framework instructs the LLM to reflect on and justify each candidate answer, then aggregates these justifications to evaluate the target answer. The abstract claims that this approach effectively mitigates over-trust in incorrect LLM answers and shows promising results across various tasks and datasets.

Strengths

• The abstract clearly states the problem of LLM hallucination and the limitations of existing self-detection approaches.

  'However, existing self-detection approaches only retrospectively evaluate answers generated by LLM, typically leading to the over-trust in incorrectly generated answers.'p. 1

• The abstract concisely describes the proposed T3 framework and its two-step process of reflection and justification followed by joint confidence calibration.

  'Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each candidate answer, and then aggregates the justifications for comprehensive target answer evaluation.'p. 1

• The abstract highlights the potential benefits of the T3 framework in mitigating over-trust and improving self-detection.

  'It thoroughly compares the trustability of multiple candidate answers to mitigate the over-trust in LLM-generated incorrect answers.'p. 1

Suggestions for Improvement

• While the abstract mentions "extensive experiments," it would be beneficial to briefly state the specific tasks and datasets used to demonstrate the effectiveness of the framework.

  'Extensive experiments on six datasets spanning three tasks demonstrate the effectiveness of the proposed framework.'p. 1

• The abstract could briefly mention the key findings or quantitative results to further strengthen the claim of the framework's effectiveness.

Summary

The Introduction section emphasizes the issue of hallucination in Large Language Models (LLMs), where LLMs generate incorrect or nonsensical outputs. It highlights self-detection as a promising approach to evaluate output trustability and identify incorrect outputs. The focus is on black-box API LLMs due to their performance and the challenges posed by limited output information. The section briefly introduces two existing self-detection paradigms: confidence calibration and self-evaluation. However, it points out a significant drawback of both paradigms: a tendency to over-trust incorrect LLM-generated answers. The section argues that this over-trust stems from evaluating only LLM-generated answers, neglecting a broader answer space. It proposes a new paradigm that considers a comprehensive answer space beyond LLM generations to mitigate this over-trust issue.

Strengths

• The section effectively introduces the problem of LLM hallucination and its impact on trustability.

  'Large Language Model (LLM) typically suffers from the hallucination issue, (Zhang et al., 2023c; Li et al., 2023a; Golovneva et al., 2022; Bang et al., 2023), which significantly undermines the trustability of LLM's outputs.'p. 1

• It clearly explains the concept of self-detection and its relevance for evaluating LLM output trustability.

  'Given a question, self-detection aims to leverage LLM's own ability to evaluate the trustability of its generated answers, without relying on external knowledge sources or specifically trained detection models.'p. 1

• The section explicitly states the focus on black-box API LLMs and the rationale behind this choice.

  'This paper investigates self-detection methods tailored for black-box API LLMs due to their excellent performance and the inherent challenge posed by limited output information (Achiam et al., 2023; OpenAI, 2024).'p. 1

• It identifies a key limitation of existing self-detection paradigms: over-trust in incorrect LLM answers.

  'However, both self-detection paradigms have shown a significant drawback: an inclination towards over-trusting the incorrect answers generated by LLM (Si et al., 2022; Xiong et al., 2023; Jiang et al., 2023; Kadavath et al., 2022).'p. 1

• The section proposes a new paradigm that considers a broader answer space to address the over-trust issue.

  'An ideal self-detection paradigm should consider a more comprehensive answer space beyond LLM's generations.'p. 2

Suggestions for Improvement

• While the Introduction mentions two existing paradigms, it would be beneficial to provide a more detailed explanation of how these paradigms work, including their specific approaches and limitations.

  'Previous studies in self-detection can be broadly categorized into two paradigms (cf. Figure 2). The first paradigm is confidence calibration, aiming to estimate LLM's confidence on the generated answer to align with the actual answer accuracy via multi-answer sampling and aggregation (Xiong et al., 2023; Tian et al., 2023b; Si et al., 2022; Jiang et al., 2023). The second one is self-evaluation, which directly examines the compatibility of question and answer by designing various prompt strategies (Miao et al., 2023; Kadavath et al., 2022; Weng et al., 2023).'p. 1

• The Introduction could benefit from a more explicit discussion of the challenges associated with evaluating trustability in black-box API LLMs. This would strengthen the rationale for the proposed new paradigm.

  'This paper investigates self-detection methods tailored for black-box API LLMs due to their excellent performance and the inherent challenge posed by limited output information (Achiam et al., 2023; OpenAI, 2024).'p. 1

Visual Elements Analysis

Summary

The Problem Formulation section formally defines the task of self-detection for Large Language Models (LLMs). It explains that given a question (q) and a prompt (p), the LLM generates an answer (a), referred to as the target answer. Self-detection then aims to evaluate the trustability of this target answer using the LLM's own capabilities, typically producing a detection score (c). The section further elaborates on two existing paradigms for self-detection: Confidence Calibration and Self-Evaluation. Confidence Calibration focuses on estimating the LLM's confidence in the generated answer, aligning it with the actual answer accuracy. This involves sampling multiple answers and aggregating their probabilities. Self-Evaluation, on the other hand, directly examines the compatibility between the question and the generated answer using various prompt strategies. However, the section highlights a significant limitation of both paradigms: a tendency to over-trust incorrect LLM-generated answers. This over-trust stems from evaluating only the LLM-generated answers, neglecting a broader answer space. The section concludes by proposing a new paradigm, Comprehensive Answer Evaluation, which considers multiple candidate answers within the answer space to mitigate this over-trust issue. This paradigm involves evaluating the trustability of each question-answer pair and aggregating these evaluations to enhance self-detection of the target answer.

Strengths

• The section provides a clear and concise definition of the LLM self-detection task, including the input (question and prompt), output (target answer), and goal (evaluating trustability).

  'Given the input comprising of question q combined with prompt p, which consists of an instruction and optional in-context examples, LLM can generate the answer a (Brown et al., 2020), denoted as the target answer. Thereafter, self-detection aims to evaluate the trustability of a by LLM’s own ability, generally in the form of a detection score c ∈ R 1 .'p. 2

• It effectively explains the two existing self-detection paradigms, Confidence Calibration and Self-Evaluation, outlining their respective approaches and goals.

  'Confidence calibration aims to estimate LLM’s level of certainty on the answer a, e.g., estimating the LLM output probability of a, where the obtained confidence score as the detection score c aims to calibrate with the actual answer accuracy. Self-evaluation methods concatenate q and a and leverage various designed prompts to instruct LLM in self-evaluating the correctness of a from different perspectives.'p. 3

• The section clearly identifies the over-trust issue as a key limitation of existing paradigms, providing a rationale for the proposed new paradigm.

  'A notable limitation of the existing two paradigms is that their evaluation merely involves LLM-generated answers ai , in which LLM may exhibit over-trust. We argue that such biased over-trust could be alleviated if LLM had thoroughly compared the trustability of more candidate answers of q beyond LLM-generated answers.'p. 3

Suggestions for Improvement

• While the section mentions the potential of counterfactual questions for mitigating over-trust, it would be beneficial to provide a more detailed explanation of how counterfactual questions are generated and how their evaluation contributes to self-detection.

  'We aim to utilize the label difference between q and q̄ to identify unreliable LLM-generated answer for q and adjust its detection score.'p. 4

• The section could benefit from a more explicit discussion of the challenges associated with obtaining a comprehensive answer space for different question-answering settings. While it mentions multi-choice questions as an example, it briefly suggests answer retrieval or model prediction for other settings without elaborating on their feasibility and limitations.

  'We consider the multi-choice question answering setting where a comprehensive answer space is provided. 2 If other answers in q’s answer space had a strong tendency to be correct, the high detection score for LLM-generated incorrect a could be diminished, reducing the over-trust issue.'p. 3

Visual Elements Analysis

Summary

This section introduces the "Think Twice Before Trusting" (T3) framework, designed to address the over-trust issue in LLM self-detection. It highlights two key considerations for implementing the proposed comprehensive answer evaluation paradigm: resisting LLM's inherent bias to accurately evaluate each question-answer pair and effectively aggregating these evaluations for the target answer's self-detection. The framework consists of two steps: 1) Reflection and Justification: The LLM is instructed to reflect on the trustability of each candidate answer and provide justifications for its potential correctness. This step aims to mitigate LLM's bias by forcing it to seek evidence supporting the rationality of each answer. 2) Joint Confidence Calibration: After obtaining justifications for each answer, the framework integrates them with a confidence calibration method, specifically the Top-K verbalized method. This involves generating a set of potential answers and their probabilities, considering the provided justifications. The section emphasizes that the T3 framework can be seamlessly integrated with existing approaches, such as prompt ensemble and Hybrid methods, for superior self-detection.

Strengths

• The section clearly identifies the two key considerations for implementing the comprehensive answer evaluation paradigm, providing a solid foundation for the proposed framework.

  'Implementing the proposed paradigm involves two key considerations. First, given the potential bias of LLM over-trust in the generated answer a, it is essential to develop strategies to resist this bias and thoroughly evaluate the trustability of each answer aqi . Secondly, it is crucial to derive strategies to effectively combine these evaluations for effective self-detection of a.'p. 4

• The section provides a detailed explanation of the two steps involved in the T3 framework, outlining their respective goals and mechanisms.

  'Step 1: Reflection and Justification. We first instruct LLM to reflect on the trustability of each answer aqi and force LLM to seek justification for aqi as the correct answer of q, as defined by Eq. 8. Step 2: Joint Confidence Calibration. After obtaining the justification ei for each aqi , we choose to integrate these ei with a confidence calibration method, the Top-K verbalized (cf. Eq. 5) to derive the confidence of answer a as the detection score.'p. 4

• The section highlights the flexibility of the T3 framework in integrating with existing self-detection approaches, suggesting its potential for broader application and improvement.

  'Notably, the T3 framework can be combined with existing approaches, such as prompt ensemble (Jiang et al., 2023), and Hybrid method which adjust the detection score based on the difference with other methods (Xiong et al., 2023).'p. 5

Suggestions for Improvement

• While the section mentions the rationale for choosing the Top-K verbalized method for confidence calibration, it would be beneficial to provide a more in-depth discussion of its advantages and limitations compared to other calibration methods. This would strengthen the justification for its selection within the T3 framework.

  'We choose this method due to its capability to generate a set of K potential answers and their respective probabilities efficiently in a single response, where we set K as the number of answers N .'p. 5

• The section could benefit from a more explicit explanation of how the order shuffling of justifications in the prompt pv mitigates position bias and improves self-detection. While it mentions the sensitivity to order, a clearer explanation of the mechanism would enhance understanding.

  'Moreover, we find that the detection scores are sensitive to the order of justification in pv , thus we shuffle the order of ei in pv and compute the averaged score.'p. 5

Visual Elements Analysis

Summary

The Related Work section provides a concise overview of previous research in areas relevant to the paper's focus on LLM self-detection. It covers three main areas: 1) Confidence Calibration of LLM: This part discusses prior work on calibrating LLM confidence, including methods utilizing output token probability and those suitable for black-box API LLMs. It distinguishes the paper's focus on black-box API LLMs from research on white-box LLMs. 2) Self-Evaluation of LLM: This part summarizes research on LLM self-evaluation, particularly in specific domains like code generation and fact-checking. It highlights the limitations of existing general methods and contrasts the paper's approach with those focusing solely on LLM-generated answers. 3) Application of LLM Self-Detection: This part briefly outlines various applications of LLM self-detection, such as knowledge retrieval, guided output decoding, and selective generation. It emphasizes the broad potential of self-detection in addressing LLM hallucination and improving output reliability.

Strengths

• The section effectively categorizes related work into three distinct areas, providing a structured overview of the research landscape.

• It clearly distinguishes the paper's focus on black-box API LLMs from research on white-box LLMs, justifying the relevance of the chosen approach.

  'Our research is orthogonal to them, since we focus on black-box API LLM itself.'p. 5

• The section highlights the limitations of existing self-evaluation methods, particularly those focusing solely on LLM-generated answers, emphasizing the novelty of the proposed comprehensive answer evaluation paradigm.

  'Feng et al. (2024) also performs answer reflection and employs model collaboration, yet they still focus on answers generated by LLM.'p. 5

• It briefly but effectively outlines various applications of LLM self-detection, demonstrating the potential impact and relevance of the research.

  'The outcome of self-detection can be applied in many ways to avoid hallucination and erroneous outputs, such as identifying potentially hallucinated generation for knowledge retrieval and verification (Zhao et al., 2023a), guided output decoding (Xie et al., 2023), identifying ambiguous questions (Hou et al., 2023), selective generation (Ren et al., 2023a; Zablotskaia et al., 2023), and LLM self-improve (Huang et al., 2023a).'p. 5

Suggestions for Improvement

• While the section mentions various methods within each area, it would be beneficial to provide a more critical analysis of their strengths and weaknesses. This would help readers understand the rationale for the proposed approach and its potential advantages over existing methods.

• The section could benefit from a more explicit discussion of the challenges associated with self-detection in black-box API LLMs. This would further strengthen the motivation for the proposed framework and its focus on comprehensive answer evaluation.

Summary

The Experiments section details the setup, compared methods, evaluation metrics, and results of the proposed "Think Twice Before Trusting" (T3) framework for LLM self-detection. It covers experiments on six datasets across three tasks (SA, NLI, CQA) using three different LLMs (GPT-3.5, GLM-4, Gemini). The section presents results in terms of AUROC, PRAUC, and ECE, demonstrating the effectiveness of T3 in improving self-detection and mitigating over-trust in incorrect answers. It also includes ablation studies to validate the framework's design choices and analysis of T3's robustness across different target answers, LLMs, and parameter settings. The section highlights the potential of T3 in selective prediction scenarios and discusses its limitations, particularly its focus on black-box API LLMs and the need for further exploration of its utility in enhancing task accuracy and enabling self-correction.

Strengths

• The section provides a clear and comprehensive description of the experimental setup, including the datasets, LLMs, compared methods, and evaluation metrics.

  'We conduct experiments on six datasets across three tasks. IMDB (Maas et al., 2011) and Flipkart (Vaghani and Thummar, 2023) for SA, SNLI (Bowman et al., 2015) and HANS (McCoy et al., 2019) for NLI, CommonsenseQA (Talmor et al., 2019) and PIQA (Bisk et al., 2020) for commonsense question answering (CQA). For LLMs, we utilize GPT-3.5 (gpt-3.5-turbo-1106) from OpenAI3 , GLM-4 (Du et al., 2022) from ZhipuAI4 , and Gemini (gemini-1.0-pro-001) from Google5 . Dataset statistics and LLM hyperparameters are listed in Appendices A.1 and A.2.'p. 6

• It presents a thorough comparison with various existing self-detection methods, demonstrating the superior performance of the proposed T3 framework.

  'Table 2 shows the performance of the compared methods on GPT-3.5. We can observe the followings. 1) T 3 outperforms all compared methods in AUROC and PRAUC on all datasets except HANS and PIQA, and in ECE on all datasets except SNLI, demonstrating its effectiveness.'p. 6

• The section includes ablation studies to validate the design choices of the T3 framework, providing evidence for the effectiveness of reflection and justification, joint confidence calibration, and order shuffling.

  'From Table 3, we can observe that: 1) w/ CoT expl largely underperforms T 3 on all three tasks, demonstrating the rationality of pushing LLM to reflect and justify from each answer\'s perspective.'p. 7

• It analyzes the robustness of T3 across different target answers, LLMs, and parameter settings, demonstrating its generalizability and flexibility.

  'Analysis on the Robustness of T 3 . We evaluate the robustness of T 3 from three aspects: different target answers, different LLMs, and parameter sensitivity. In addition, we examine prompt sensitivity of pe and pv in Appendix C.'p. 8

Suggestions for Improvement

• The use of a line graph in Figure 5 is appropriate for showing the trend of accuracy improvement as the percentage of abstained instances increases. It effectively visualizes the continuous nature of the relationship and allows for easy identification of the point at which accuracy plateaus or starts to decline. The connected data points clearly indicate the progression of accuracy values along the x-axis.

• While the section mentions the potential of T3 in selective prediction, it would be beneficial to provide more concrete examples or use cases where this capability could be particularly valuable. This would further demonstrate the practical implications of the research.

  'To show the utility of the detection score, we conduct experiments in selective prediction. The idea of selective prediction is to abstain the LLM-generated answers with low detection score to maintain better accuracy of the remaining instances.'p. 7

• The section could benefit from a more in-depth discussion of the limitations of T3, particularly its dependence on the LLM's ability to generate meaningful justifications and the potential for prompt engineering biases. Addressing these limitations would strengthen the overall impact of the research.

  'Our work has several limitations. Firstly, our research scope is limited to the self-detection for black-box API LLM. While our framework is suitable for many state-of-the-art LLMs in this form, it might not be optimal for white-box LLMs, which offer access to token probabilities, thus limiting its broader applicability.'p. 9

Visual Elements Analysis

Summary

The Conclusion section summarizes the paper's key contributions in addressing the over-trust issue in self-detection for black-box API LLMs. It reiterates the limitations of existing self-detection paradigms that focus solely on LLM-generated answers, leading to over-trust in incorrect outputs. The section highlights the proposed comprehensive answer evaluation paradigm, which considers a broader answer space and compares the trustability of multiple candidate answers. It emphasizes the effectiveness of the "Think Twice Before Trusting" (T3) framework in implementing this paradigm by instructing the LLM to reflect on and justify each candidate answer before calibrating confidence. The section concludes by acknowledging the limitations of the current work, particularly its focus on black-box API LLMs and the need for further exploration of self-detection's utility in enhancing task accuracy and enabling self-correction. It also suggests future research directions, including combining T3 with more methods and exploring its applicability to white-box LLMs.

Strengths

• The Conclusion effectively summarizes the key contributions of the paper, highlighting the proposed paradigm and framework for mitigating over-trust in LLM self-detection.

  'In this paper, we tackled the over-trust issue of self-detection on black-box API LLMs. We categorized existing methods into two paradigms and pointed out their limitation of merely evaluating on LLM-generated answer with potential LLM over-trust. We proposed a novel paradigm to address this limitation by comprehensively evaluating the trustability of multiple candidate answers in the answer space.'p. 8

• It clearly reiterates the limitations of existing self-detection approaches and the rationale for considering a broader answer space.

  'We categorized existing methods into two paradigms and pointed out their limitation of merely evaluating on LLM-generated answer with potential LLM over-trust.'p. 8

• The Conclusion acknowledges the limitations of the current work, particularly its focus on black-box API LLMs and the need for further exploration of self-detection's utility in improving task accuracy and enabling self-correction.

  'Our work has several limitations. Firstly, our research scope is limited to the self-detection for black-box API LLM. While our framework is suitable for many state-of-the-art LLMs in this form, it might not be optimal for white-box LLMs, which offer access to token probabilities, thus limiting its broader applicability. Secondly, the utility of self-detection is not primarily studies in this work. Although we demonstrate the utility of detection scores in selective prediction scenarios, the challenge still lies in leveraging them to enhance task accuracy or enable LLM self-correction, calling for further exploration.'p. 9

Suggestions for Improvement

• While the Conclusion mentions exploring T3's applicability to white-box LLMs, it would be beneficial to briefly discuss the potential challenges and opportunities associated with this extension. This would provide a more nuanced perspective on future research directions.

  'In future work, we will explore the combination of T3 with more methods, and its utility in white-box LLMs.'p. 8

• The Conclusion could benefit from a more explicit discussion of the potential impact of the research on the broader field of LLM development and deployment. This would highlight the significance of addressing the over-trust issue and the potential benefits of the proposed approach for improving the reliability and trustworthiness of LLMs.

Summary

The Limitations section acknowledges the restricted scope of the research, focusing solely on self-detection for black-box API LLMs. It recognizes that the framework might not be ideal for white-box LLMs, which provide access to token probabilities, limiting its broader applicability. The section also highlights the need for further investigation into the practical utility of self-detection in enhancing task accuracy or enabling LLM self-correction. It admits that the current work lacks consideration for prompt optimization in self-detection, suggesting it as an area for future research.

Strengths

• The section clearly states the limitation of focusing solely on black-box API LLMs and acknowledges the potential for better suitability with white-box LLMs.

  'Firstly, our research scope is limited to the self-detection for black-box API LLM. While our framework is suitable for many state-of-the-art LLMs in this form, it might not be optimal for white-box LLMs, which offer access to token probabilities, thus limiting its broader applicability.'p. 9

• The section identifies the need for further research on leveraging self-detection to improve task accuracy or enable LLM self-correction, recognizing the gap between current capabilities and potential applications.

  'Secondly, the utility of self-detection is not primarily studies in this work. Although we demonstrate the utility of detection scores in selective prediction scenarios, the challenge still lies in leveraging them to enhance task accuracy or enable LLM self-correction, calling for further exploration.'p. 9

• The section acknowledges the lack of consideration for prompt optimization in the current framework, suggesting it as a potential area for improvement in future research.

  'Lastly, our framework lacks consideration in prompt optimization for self-detection, an area where future self-detection methods are expected to consider.'p. 9

Suggestions for Improvement

• While the section mentions the limitations, it would be beneficial to elaborate on potential strategies or directions for addressing them. For example, it could briefly discuss how the framework could be adapted for white-box LLMs or how prompt optimization could be incorporated.

Summary

The Ethics Statement section raises three ethical concerns related to the research. Firstly, it acknowledges the limited evaluation of the framework's applicability to languages other than English, as the experiments primarily used English datasets. Secondly, it recognizes the ethical implications of focusing on black-box API LLMs, advocating for the use of open-sourced LLMs to enhance reproducibility. Lastly, it cautions against the potential for self-detection mechanisms to mislead users into blindly trusting LLMs, potentially leading to the acceptance of untrustable answers and causing harm.

Strengths

• The section explicitly acknowledges the limitation of evaluating the framework primarily on English datasets and the need for broader language applicability.

  'First, our experimental results are mainly obtained in English datasets, where the applicability on other languages are not comprehensively evaluated.'p. 9

• The section raises the ethical concern of focusing on black-box API LLMs and advocates for the use of open-sourced LLMs to promote reproducibility.

  'Secondly, our research scope is black-box API LLMs, where open-sourced LLMs are more advocated for its reproducibility.'p. 9

• The section highlights the potential risk of users blindly trusting LLMs due to self-detection mechanisms, leading to the acceptance of untrustable answers and potential harm.

  'Finally, the self-detection of LLM may mislead people to blindly trust LLM and easily accept untrustable answers, causing potential harms.'p. 9

Suggestions for Improvement

• While the section identifies the ethical concerns, it would be beneficial to propose potential mitigation strategies or guidelines for addressing these concerns. For example, it could suggest specific steps for evaluating the framework on other languages, promoting the use of open-sourced LLMs, or educating users about the limitations of self-detection.

Summary

The References section lists the bibliographic details of all the sources cited throughout the paper. It includes a variety of research papers, conference proceedings, and online resources related to Large Language Models (LLMs), self-detection, confidence calibration, self-evaluation, and various applications of these techniques in natural language processing tasks.

Strengths

• The References section provides a comprehensive list of sources, covering a wide range of relevant topics related to the paper's focus.

Suggestions for Improvement

• The references are correctly formatted and consistently presented, following standard academic conventions. There are no significant suggestions for improvement in terms of content or organization.

Summary

Appendix A provides supplementary information related to the experiments conducted in the paper. It includes details about the datasets used, the hyperparameters for the LLMs, and additional implementation details for the compared methods. Specifically, it presents: 1) Dataset Detail: This part describes the datasets used in the experiments, including the number of training data samples, the number and examples of candidate answers for each dataset, and the rationale for dataset selection. 2) LLM Hyperparameters: This part lists the hyperparameters used for the different LLMs (GPT-3.5, GLM-4, Gemini) in the experiments, including the maximum token limit, temperature settings, and sampling strategies. 3) Details for compared methods: This part provides the specific prompts used for the compared self-detection methods, outlining the instructions and input formats for each method and dataset. 4) Additional Implementation Detail: This part elaborates on the implementation details of the T3 framework and the compared methods, including the setting for the number of candidate answers, the shuffling strategy for justification order, and the prompt ensemble approach used in CAPE. It also compares the number of API calls for different methods, highlighting the reasonable cost of T3.

Strengths

• The appendix provides comprehensive details about the datasets used in the experiments, including the number of training data samples and examples of candidate answers.

  'Due to the cost limitation, we randomly sample 300 training data for each dataset in our experiments. For IMDB and SNLI datasets, we use the same randomly sampled 300 data sets as the CAD SA and NLI in the preliminary experiments. We will release the dataset splits. Table 6 shows the number and examples of candidate answers for each dataset.'p. 12

• It clearly lists the hyperparameters used for the different LLMs, ensuring reproducibility of the experiments.

  'For all LLMs, we set the maximum token as 200. For GPT-3.5 and Gemini, if sampling a single response (N = 1), we set the temperature as 0, and other hyperparameters as default. If sampling multiple responses, we sample N = 30 (N = 5 for Gemini due to API call limitation) responses with temperature as 1, which is only for Self-cons, CoT-cons, and P(True).'p. 12

• The appendix provides the specific prompts used for the compared self-detection methods, allowing for a clear understanding of their implementation.

  'The prompts for compared methods are shown below, where [instruction] denotes the task instruction with the task input, and [instruction_only] denotes the instruction without task input.'p. 13

• It includes a comparison of the number of API calls for different methods, demonstrating the reasonable cost of the proposed T3 framework.

  'The number of API calls for different methods are shown in Table 7. We can observe that compared with other methods T3 does not incur large increase in number of calls. In our experiments, the maximum value of N is 5. Considering its effectiveness, the cost of T3 is reasonable.'p. 14

Suggestions for Improvement

• While the appendix mentions the rationale for choosing specific datasets, it would be beneficial to provide a more detailed discussion of their characteristics and potential limitations. This would help readers understand the generalizability of the findings.

  'Due to the cost limitation, we randomly sample 300 training data for each dataset in our experiments.'p. 12

• The appendix could benefit from a more explicit justification for the chosen LLM hyperparameters. While it mentions that they were not carefully tuned, explaining the rationale behind the initial settings would enhance transparency.

  'Note that these LLM hyperparameters are not carefully tuned.'p. 12

• While the appendix provides the prompts for the compared methods, it would be helpful to include examples of actual LLM outputs generated using these prompts. This would further clarify their implementation and potential biases.

Visual Elements Analysis

Summary

Appendix B provides implementation details for the preliminary experiments conducted to validate the concept of using counterfactual questions for mitigating over-trust in LLM self-detection. It describes the dataset used (CAD), the specific prompts employed, the LLM used (GPT-3.5), and the rationale for choosing the Top-K verbalized method as the basis for the counterfactual strategy. The appendix also mentions the random sampling of 300 instances from the training set and the random selection of one counterfactual question for each original question with multiple counterfactuals.

Strengths

• The appendix provides specific details about the dataset (CAD) used for the preliminary experiments, including the tasks (SA and NLI) and the source datasets (IMDB and SNLI).

  'We experiment with the CAD dataset (Kaushik et al., 2019), which contains human-annotated original and counterfactual data pairs for sentiment analysis (SA) and natural language inference (NLI) tasks.'p. 4

• It clearly states the LLM used (GPT-3.5) and refers to Appendix A.1 for the LLM hyperparameters, ensuring reproducibility.

  'The LLM is GPT-3.5 (gpt-3.5-1106). See Section A.1 for LLM hyperparameters.'p. 14

• The appendix explains the rationale for choosing the Top-K verbalized method as the basis for the counterfactual strategy, highlighting its better calibration compared to Self-cons.

  'The w/ cf is based on Top-K Verb, which is better calibrated than Self-cons.'p. 14

Suggestions for Improvement

• While the appendix mentions random sampling of 300 instances, it would be beneficial to provide a justification for this sample size and discuss its potential impact on the reliability of the results.

  'For the preliminary experiments, we randomly sample 300 instances from the training set of CAD SA and NLI, respectively.'p. 14

• The appendix could benefit from a more detailed explanation of how the counterfactual questions were generated or selected. While it mentions random selection for questions with multiple counterfactuals, it doesn't elaborate on the criteria or process used.

  'For those original questions with more than one counterfactual questions, we randomly select one counterfactual question for experiment.'p. 14

• The appendix briefly mentions that the prompts can be viewed in Section A.3. It would be helpful to provide the actual prompts used for the counterfactual strategy within Appendix B for easier reference and comparison.

  'The prompts can be viewed in Section A.3.'p. 14

Summary

Appendix C focuses on evaluating the prompt sensitivity of the pe and pv prompts used in the T3 framework. It examines the impact of rephrasing these prompts on the framework's performance, specifically in terms of AUROC scores. The analysis involves rephrasing each prompt three times using ChatGPT and calculating the average and standard deviation of AUROC scores across these variations. The results suggest that while prompt rephrasing has a mild effect on T3's performance, the HANS dataset exhibits higher sensitivity compared to Flipkart and PIQA. Additionally, the analysis indicates that changes in the pe prompt have a larger impact on detection performance than changes in the pv prompt, potentially due to the wider range of variations in justifications generated by pe compared to the outputs of pv.

Strengths

• The appendix directly addresses the potential concern of prompt sensitivity by systematically evaluating the impact of prompt rephrasing on T3's performance.

  'We examine the prompt sensitivity of pe and pv by rephrasing each of them three times with ChatGPT6 and compute the average and standard deviation of AUROC, as shown in Table 8.'p. 14

• It provides a clear methodology for assessing prompt sensitivity, involving rephrasing each prompt three times and calculating the average and standard deviation of AUROC scores.

  'We examine the prompt sensitivity of pe and pv by rephrasing each of them three times with ChatGPT6 and compute the average and standard deviation of AUROC, as shown in Table 8.'p. 14

• The appendix identifies the HANS dataset as being more sensitive to prompt rephrasing compared to Flipkart and PIQA, providing insights into dataset-specific variations in prompt sensitivity.

  'Across the three datasets, HANS is the most sensitive to prompt rephrasing, potentially related to its lower AUROC performance.'p. 14

• It highlights the observation that changes in the pe prompt have a larger impact on detection performance than changes in the pv prompt, offering a potential explanation for this difference based on the nature of the outputs generated by each prompt.

  'The change of pe has larger impact on the detection performance than pv . This is probably because the justifications generated by pe have a larger space of variation than the outputs of pv , i.e., guesses and probabilities.'p. 14

Suggestions for Improvement

• While the appendix rephrases each prompt three times, it would be beneficial to increase the number of prompt variations to assess a wider range of wording choices and potential biases.

  'We examine the prompt sensitivity of pe and pv by rephrasing each of them three times with ChatGPT6 and compute the average and standard deviation of AUROC, as shown in Table 8.'p. 14

• The analysis could be strengthened by performing statistical significance testing to determine if the observed differences in AUROC scores across different prompt variations are statistically significant.

• The appendix could benefit from a more detailed discussion of the specific rephrasing techniques used and the rationale behind choosing them. This would enhance transparency and allow for better understanding of the variations introduced.

Visual Elements Analysis

Summary

Appendix D presents a performance comparison of the T3 framework on the Gemini language model using three datasets: Flipkart, PIQA, and CommonsenseQA. The results, presented in Table 9, show that while T3 performs well on PIQA and CommonsenseQA, it doesn't outperform all compared methods on Flipkart. The appendix attributes this discrepancy to Gemini's tendency to prioritize answer prediction over following the instructions for reflection and justification. It concludes that the effectiveness of T3 depends on the LLM's ability to adhere to the instructions for generating justifications from different answer perspectives.

Strengths

• The appendix provides a performance comparison of T3 on a different LLM (Gemini), expanding the evaluation beyond GPT-3.5 and GLM-4.

  'In addition to GPT-3.5 and GLM-4, we show the results of Gemini on three datasets.'p. 15

• It identifies a potential limitation of T3's effectiveness, highlighting its dependence on the LLM's ability to follow instructions for reflection and justification.

  'Therefore, the effectiveness of T3 depends on the ability of the specific LLM in following the instructions in Table 1.'p. 15

Suggestions for Improvement

• While the appendix mentions Gemini's tendency to prioritize prediction over reflection, it would be beneficial to provide examples of Gemini's outputs to illustrate this behavior. This would provide more concrete evidence for the observed discrepancy in performance.

  'Instead, it tends to perform answer prediction and followed by an explanation on its predicted answer.'p. 15

Visual Elements Analysis

Summary

Appendix E presents two case studies from the PIQA dataset to illustrate the effectiveness of the T3 framework in mitigating over-trust in LLM self-detection. The first case study involves a question about repairing a torn shirt, where the LLM initially assigns a high confidence score (0.7) to an incorrect answer. However, after applying T3 and generating justifications for both answer choices, the detection score for the incorrect answer is lowered to 0.45. The second case study focuses on a question about keeping a couch fur-free, where the LLM initially exhibits high confidence (0.7) in an incorrect answer. After applying T3, the detection score is adjusted to 0.5, reflecting the LLM's uncertainty about the correct answer. These case studies demonstrate how T3 can help LLMs identify and reduce over-trust in incorrect answers by considering justifications from different answer perspectives.

Strengths

• The appendix provides concrete examples of how T3 works in practice, using actual questions and answers from the PIQA dataset.

• It clearly shows how the generation of justifications for each answer choice can lead to adjustments in the detection score, reflecting the LLM's revised confidence.

Suggestions for Improvement

• While the appendix presents two case studies, it would be beneficial to include more examples to demonstrate the generalizability of T3's effectiveness across different question types and answer choices.

• The appendix could benefit from a more detailed analysis of the generated justifications, exploring the specific reasoning patterns or evidence used by the LLM to support its confidence adjustments.

Visual Elements Analysis