People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

Figure 1: A human expert's annotations of an article generated by OpenAI's 01-PRO with humanization. The expert provides a judgment on whether the text is written by a human or AI, a confidence score, and an explanation (including both free-form text and highlighted spans) of their decision.

• Overview of Figure 1: Figure 1 shows an example of how a human expert evaluated and marked up a piece of text. This text is a news article that was produced by a Large Language Model, specifically OpenAI's 01-PRO, which is a type of artificial intelligence designed to generate human-like text. The term "humanization" here likely refers to techniques used to make the AI-generated text appear more like it was written by a human. The expert's job is to determine whether the article was written by a human or a machine, and to explain their reasoning.
• Content of the Article: The article discusses a pilot in Alaska who drops turkeys to rural homes for Thanksgiving. This scenario sets a real-world context for evaluating the AI's ability to generate text that could be mistaken for human writing in a practical, everyday situation.
• Expert's Annotation Components: The expert provides three pieces of information: a judgment, a confidence score, and an explanation. The "judgment" is a binary classification, meaning the expert decides if the text is "AI-generated" or "human-written." The "confidence score" is a rating on a scale of 1 to 5, where 1 means the expert is not very sure about their judgment, and 5 means they are very sure. The "explanation" is a written justification for their decision, and it can include highlighting parts of the text that influenced their judgment. Highlighting parts of the text means visually marking specific words or phrases, like using a yellow marker on a printed page, to show which parts were particularly important in making their decision.
• Expert's Decision and Rationale: In this specific example, the expert has labeled the text as "AI-generated" with a confidence score of 4 out of 5, indicating a high level of certainty. The expert's explanation points out that some of the quotes in the article feel realistic, but others are presented in a way that seems unnatural. For instance, the expert notes that the phrase "He shrugged as if that were the most ordinary idea, then laughed" could be simplified. They also suggest that the article could provide more factual details about the challenges faced by people in Alaska and the limited transportation options, indicating that the AI-generated text may lack the depth and detail typically found in human-written articles on similar topics. The expert also mentions that the article gets "sentimental and corny at times," which could be another clue that it was generated by an AI, as AI models might overuse emotional language or clichés in an attempt to mimic human writing.

• Clarity of Methodology: The figure and its caption, along with the reference text, provide a reasonably clear overview of the methodology used for human annotation. The task given to the annotators is well-defined, and the criteria for their judgments are outlined. However, the scientific validity would be enhanced by providing more details about the selection criteria for "expert" annotators and the specific instructions given to them beyond the general task description.
• Subjectivity in Annotation: The annotation process inherently involves subjective judgment, which can introduce bias. The reliance on human judgment for determining whether a text is AI-generated or human-written is a potential source of variability. The use of a confidence score helps to quantify this subjectivity to some extent, but the explanation provided by the annotator is crucial for understanding the basis of their judgment. The scientific validity of the study could be improved by having multiple annotators evaluate the same texts and measuring the inter-rater reliability to assess the consistency of these subjective judgments.
• Representation of AI-Generated Text: The figure represents an example of text generated by OpenAI's 01-PRO with humanization. It is important to note that this is just one example, and the quality and characteristics of AI-generated text can vary widely depending on the model and the specific techniques used for humanization. The validity of the study's findings would be strengthened by including a diverse set of examples from different AI models and humanization methods.

• Visual Presentation: The figure is visually well-organized, with clear sections for the article text, the annotator's decision, confidence score, and explanation. The use of color-coding and highlighting helps to distinguish between different parts of the annotation. However, the specific colors used for highlighting are not explained in the caption or the figure itself, which could be a minor point of confusion for the reader.
• Clarity of Explanation: The expert's explanation is concise and provides specific examples from the text to support their judgment. The language used is accessible and avoids overly technical jargon, making it understandable to a reader who may not be familiar with AI-generated text analysis. The explanation effectively communicates the thought process behind the expert's decision.
• Effectiveness in Demonstrating Annotation Process: The figure serves as an effective illustration of the annotation process described in the text. It demonstrates how an expert evaluates an article, makes a judgment, assigns a confidence score, and provides a rationale. This helps the reader to understand the methodology used in the study and provides a concrete example of the kind of data collected from human annotators.
• Potential for Misinterpretation: While the figure is generally clear, there is a potential for misinterpretation regarding the term "humanization." A reader unfamiliar with the concept might assume it refers to a process of making the text more humane or ethical, rather than more human-like in style. A brief definition or clarification of this term within the caption or the figure itself could improve understanding.

Table 5: Survey of Annotators, specifically their backgrounds relating to LLM usage and field of work. Note that expert Annotator #1 was one of the original 5 annotators (along with the 4 non-experts) and remained an annotator for all expert trials.

• Purpose of the Table: This table provides information about the people who participated in the study as annotators, specifically the "expert" annotators. Annotators are the people who labeled the text as either human-written or AI-generated. The table focuses on their backgrounds, particularly their experience with using Large Language Models (LLMs) - a type of AI that can generate text - and their professional fields. This is like a brief introduction to the people involved in the study, similar to how a news article might introduce the key people involved in a story.
• Content of the Table: The table likely includes information such as the annotators' level of education, their native language (which the reference text indicates is English), their nationality (US, UK, and South Africa), their profession, and their experience with using LLMs. For example, it might show that one annotator has a Master's degree, is from the UK, works as an editor, and uses LLMs daily. By providing this information, the researchers are giving us a better understanding of who these "expert" annotators are and what kind of expertise they bring to the task.
• Expert vs. Non-expert: The caption highlights that one of the expert annotators, "Annotator #1," was also part of the initial group of five annotators, which included four non-experts. This suggests that this particular expert was involved in the study from the beginning and participated in all the "expert trials," meaning they have a lot of experience with the task. The distinction between "expert" and "non-expert" is based on their self-reported usage of LLMs, as established earlier in the paper.
• Relevance of Background Information: The annotators' backgrounds are important because they can influence their ability to detect AI-generated text. For example, someone who works as a professional writer or editor and frequently uses LLMs might be better at spotting the nuances of AI-generated text compared to someone with less experience in these areas. By providing information about the annotators' backgrounds, the researchers are adding context to the study's findings and helping us understand the qualifications of the people making the judgments.

• Selection of Expert Annotators: The validity of using "expert" annotators depends on the criteria used for defining expertise. The reference text suggests that expertise is based on being a native English speaker, working in a relevant field (editors, writers, proofreaders), and having extensive experience using AI assistants. These are reasonable criteria, but the specific thresholds for "extensive experience" should be clearly defined. Additionally, while self-reported expertise is a practical approach, it would be beneficial to include some objective measure of their ability to detect AI-generated text, perhaps through a pre-test or calibration task.
• Sample Size and Diversity: The reference text mentions five expert annotators. While this is a relatively small sample size, it's important to consider the trade-off between the number of annotators and the depth of their expertise. Having a small group of highly qualified experts might be more valuable than a larger group with less specialized knowledge. However, the authors should acknowledge the limitations of a small sample size in terms of generalizability. The diversity of the annotators' backgrounds (US, UK, South Africa) is a strength, as it reduces the potential for cultural or linguistic biases in the annotations.
• Potential for Bias: The annotators' backgrounds could potentially introduce biases. For example, editors or writers might be more attuned to certain stylistic features or errors that are common in AI-generated text. The authors should discuss the potential for such biases and how they might have mitigated them (e.g., through clear guidelines for annotation, training, or inter-rater reliability checks).
• Transparency of Information: Providing a detailed survey of the annotators' backgrounds enhances the transparency of the study. This allows readers to better understand the qualifications of the experts and evaluate the potential impact of their backgrounds on the findings. The authors should ensure that the table includes all relevant information about the annotators' demographics, education, profession, and LLM usage, while also respecting their privacy.

• Caption Clarity: The caption is clear and concisely explains the purpose of the table, which is to provide information about the annotators' backgrounds in relation to LLM usage and their field of work. It also highlights the unique status of Annotator #1, which is helpful for understanding the continuity of expertise across the study.
• Table Organization and Content: Assuming a standard tabular format with clear labels for rows (annotators) and columns (background characteristics), the table should be easy to read and understand. The specific characteristics included in the table (e.g., education, profession, LLM usage) should be clearly defined and relevant to the study's focus on detecting AI-generated text.
• Reference Text Support: The reference text provides additional context about the expert annotators, including their native language and professional backgrounds. This information complements the table and helps to paint a more complete picture of the experts' qualifications.
• Potential Improvements: The communication effectiveness could be improved by including a brief explanation in the caption of *why* the annotators' backgrounds are important in the context of this study. Additionally, the table itself could include a column summarizing each annotator's overall level of expertise or experience with AI-generated text, perhaps based on a composite score derived from their background characteristics. This would provide a more direct measure of their qualifications for the task.

Figure 4: Guidelines provided to the annotators for the annotation task. The annotators were also provided additional examples and guidance during the data collection process.

• Purpose of the Figure: This figure displays the instructions given to the people who participated in the study as "annotators." Annotators were responsible for reading articles and deciding whether they were written by a human or an AI. The guidelines are like a rule book or a training manual for the annotators, explaining how to perform the task. Think of it like the instructions you'd get before playing a new board game - they tell you what you need to do, what the goal is, and any special rules you need to follow.
• Content of the Guidelines: The guidelines likely explain the task in detail, including what criteria to use when deciding if a text is human or AI-generated, how to label or mark the text, and how to record their decisions. The caption mentions that annotators were also given "additional examples and guidance." This means they probably saw examples of human-written and AI-generated text, along with explanations of why they were classified that way. It is like showing someone examples of correctly solved math problems before asking them to solve new ones. The guidelines likely also include instructions for using the annotation interface and details about the rating scale used for indicating confidence in their judgments.
• Importance of Guidelines: Providing clear and comprehensive guidelines is crucial for ensuring that the annotators understand the task and perform it consistently. This is important for the quality and reliability of the data collected in the study. If the annotators don't understand the instructions or apply them differently, it can introduce errors or inconsistencies into the data, making it harder to draw valid conclusions. The guidelines help to standardize the annotation process, making it more like a controlled experiment.
• Consent and Ethics: The reference text mentions that annotators were also required to sign a consent form. This is an important ethical requirement in research involving human participants. The consent form ensures that the annotators understand the purpose of the study, what they will be asked to do, any potential risks or benefits, and that their participation is voluntary. By reading the guidelines and signing the consent form, the annotators are indicating that they understand the task and agree to participate.

• Standardization of Annotation Task: Providing guidelines to annotators is a crucial step in ensuring the standardization and consistency of the annotation task. By providing clear instructions and examples, the researchers are attempting to minimize variability in how different annotators approach the task and interpret the criteria for distinguishing between human-written and AI-generated text. This standardization is essential for the scientific validity of the study, as it helps to ensure that the data collected is reliable and comparable across annotators.
• Clarity and Completeness of Guidelines: The scientific validity of the annotation process depends on the clarity and completeness of the guidelines. The guidelines should clearly define the task, provide specific criteria for distinguishing between human and AI-generated text, and address any potential ambiguities or edge cases. The authors should demonstrate that the guidelines were sufficiently detailed and unambiguous to ensure that annotators understood the task and applied the criteria consistently. The mention of "additional examples and guidance" suggests an effort to further clarify the task, which is a positive aspect.
• Potential for Subjectivity: Despite the use of guidelines, the task of distinguishing between human and AI-generated text inherently involves subjective judgment. Different annotators might interpret the criteria differently or have varying levels of sensitivity to certain linguistic features. The authors should acknowledge this potential for subjectivity and discuss how they attempted to minimize its impact (e.g., through training, pilot testing, or inter-rater reliability checks).
• Ethical Considerations: The reference to a consent form demonstrates attention to ethical considerations, which is essential when conducting research involving human participants. The authors should provide more details about the consent process and the information provided to annotators, ensuring that participants were fully informed about the study's purpose, procedures, and their rights before agreeing to participate.

• Caption Clarity: The caption clearly states the purpose of the figure: to present the "Guidelines provided to the annotators for the annotation task." It also mentions that additional examples and guidance were provided, which is important information about the annotation process. The caption is concise and easy to understand.
• Visual Presentation: The provided image of the guidelines is well-structured and easy to read. It uses headings, bullet points, and numbered lists to organize the information, making it easy to follow. The language is clear and direct, and the instructions are broken down into manageable steps. The use of bolding and different font sizes helps to highlight important information and distinguish between different sections of the guidelines.
• Reference Text Support: The reference text provides important context by mentioning the consent form, emphasizing the ethical considerations involved in the study. It also reinforces the importance of the guidelines in the annotation process.
• Potential Improvements: While the guidelines are generally well-written, the communication effectiveness could be further improved by providing a brief rationale in the figure caption for *why* guidelines are important in this study (e.g., to ensure consistency and reliability of annotations). Additionally, while the figure mentions examples, it could be beneficial to include a few key examples directly in the figure or in an appendix to illustrate the application of the guidelines. The specific criteria used for distinguishing between human and AI-generated text could also be made more explicit, either in the figure or in an accompanying table. Finally, the instructions could be made more concise by removing redundant phrases like "You do this in 2 ways" and simplifying some of the explanations.

Figure 5: Consent form which the annotators were asked to sign via GoogleForms before collecting the data.

• Purpose of the Figure: This figure shows the consent form that the annotators (the people participating in the study) were required to sign before they could start labeling articles as human-written or AI-generated. A consent form is a document that explains the study's purpose, procedures, risks, and benefits to potential participants. It's a way of ensuring that people understand what they're agreeing to before they participate in a research study. It is a standard ethical requirement in research involving human subjects.
• Content of the Consent Form: The consent form likely includes information about the study's goals, what the annotators will be asked to do (e.g., read articles, label them as human or AI, provide explanations), how much time it will take, any potential risks or benefits of participating, how their data will be used and protected, and their rights as participants (e.g., the right to withdraw from the study at any time). It also includes a statement that participation is voluntary and that they are over 18 years of age. The form in the figure appears to use a question-and-answer format to convey this information, which can make it easier for participants to understand.
• Use of Google Forms: The caption mentions that the consent form was administered via Google Forms. This is a common online tool for creating surveys and forms. Using an online platform like Google Forms makes it easy for the researchers to collect and manage consent from participants remotely. It also ensures that all participants receive the same information and that their responses are recorded electronically.
• Importance of Informed Consent: Obtaining informed consent from participants is a fundamental ethical principle in research involving human subjects. It means that participants must be fully informed about the study and voluntarily agree to participate, without any coercion or undue influence. By signing the consent form, the annotators are indicating that they have read and understood the information provided and that they agree to participate in the study under the stated conditions. This helps to protect the rights and welfare of the participants.

• Ethical Requirement: Obtaining informed consent is a standard ethical requirement for research involving human participants. It is essential for ensuring that participants are treated ethically and that their rights are protected. By including the consent form in the paper, the authors are demonstrating their commitment to ethical research practices.
• Completeness of Information: The scientific validity of the consent process depends on the completeness and clarity of the information provided in the consent form. The form should adequately explain the study's purpose, procedures, risks, benefits, data handling practices, and participant rights in a way that is understandable to the target audience. The provided image suggests that the form covers these key elements, but the full text would need to be reviewed to assess its completeness.
• Voluntary Participation: The consent form should emphasize that participation is voluntary and that participants can withdraw at any time without penalty. This is crucial for ensuring that participants are not coerced into participating and that they have the autonomy to make their own decisions about involvement in the study. The provided form clearly states this.
• Documentation and Record-Keeping: Using Google Forms to collect consent provides a clear record of participant agreement. This is important for documentation and accountability. The authors should describe how they stored and managed the consent forms to ensure confidentiality and data security.

• Caption Clarity: The caption clearly states the purpose of the figure: to present the consent form used in the study. It also specifies that the form was administered via Google Forms, which is helpful information about the data collection process.
• Visual Presentation: The provided image of the consent form is well-structured and easy to read. It uses clear headings, bullet points, and a question-and-answer format to organize the information. The language is generally straightforward and accessible, although there are a few instances where the phrasing could be simplified for a broader audience. The form is visually appealing and uses formatting effectively to highlight important information.
• Reference Text Support: The reference text reinforces the importance of the consent form by stating that all annotators were required to sign it before participating in the labeling task. This highlights the ethical considerations involved in the study.
• Potential Improvements: While the consent form is generally well-designed, there are a few areas where the communication effectiveness could be improved. Some of the language could be simplified to make it more accessible to a lay audience (e.g., "a risk of breach of confidentiality" could be rephrased as "a risk that your personal information could be revealed"). Additionally, the form could explicitly state that participants should be 18 or older *before* describing the compensation. The form could also benefit from more explicit statements about data privacy and security, assuring participants that their data will be handled responsibly. Finally, while the form mentions that participation will "help inform if people can detect whether text they are reading is written by another human or by AI models," it could elaborate on the potential societal benefits of this research (e.g., helping to address concerns about misinformation or the misuse of AI).

Table 15: Prompt Template for Story Generation, where STORY PROMPT is the writing prompt from r/WritingPrompts that the human-written story was written about and WORD COUNT is the length of the story to generate.

• Purpose of the Table: This table presents the template used for generating AI-written stories in the study. The template is a set of instructions, or a "prompt," given to an AI model to produce a story. The prompt is designed to mimic the kind of creative writing prompts found on the website Reddit, in a section called r/WritingPrompts. This subreddit is a place where people share ideas and write stories based on prompts. Essentially, the researchers are using a similar approach to get the AI to generate stories for their experiment.
• Content of the Table: The table shows the structure of the prompt, which includes placeholders for two key pieces of information: "STORY PROMPT" and "WORD COUNT." The "STORY PROMPT" is the actual writing prompt taken from r/WritingPrompts. It is the creative idea or scenario that the story should be based on. The "WORD COUNT" specifies the desired length of the generated story. For example, a prompt might be "STORY PROMPT: A detective discovers a hidden room in a library" and "WORD COUNT: 500". This would instruct the AI to write a 500-word story based on that prompt.
• Connection to Human-Written Stories: The caption explains that the "STORY PROMPT" used in the template is the same prompt that a human writer used to write a story on r/WritingPrompts. This means that the researchers are creating AI-generated stories that are directly comparable to human-written stories based on the same prompts. This is important for the study because it allows the researchers to compare human and AI writing under similar conditions, making it easier to isolate the differences between them.
• Generation of AI Stories: The reference text states that the researchers collected 30 stories from r/WritingPrompts and generated corresponding AI-generated stories using the prompt in Table 15. This means that for each human-written story, they used the same prompt to generate an AI version. The AI is essentially being asked to write a story on the same topic and with the same length constraint as a human writer. This allows for a direct comparison between human and AI writing abilities.

• Controlled Comparison: Using the same writing prompts for both human-written and AI-generated stories is a scientifically sound approach for creating a controlled comparison. By holding the prompt constant, the researchers can isolate the effects of the writer (human vs. AI) on the resulting story. This allows for a more direct assessment of the differences between human and AI writing capabilities.
• Relevance of r/WritingPrompts: Using prompts from r/WritingPrompts is a reasonable choice, as it provides a source of diverse and creative writing prompts that are likely to be representative of the kinds of prompts that humans might use to inspire their own writing. However, the authors should acknowledge that r/WritingPrompts is a specific online community with its own norms and conventions, which might not be fully representative of all types of creative writing.
• Word Count as a Constraint: Specifying the desired word count in the prompt is important for controlling the length of the generated stories. This helps to ensure that the AI-generated stories are comparable in length to the human-written stories from r/WritingPrompts. However, the authors should discuss how they determined the appropriate word count for each prompt and whether they considered the natural variation in story length that might occur among human writers.
• Dependence on AI Model Capabilities: The quality and characteristics of the AI-generated stories will depend on the capabilities of the specific AI model used. The authors should clearly state which AI model they used for story generation and discuss any known limitations of the model that might affect the results. They should also consider the potential impact of different model parameters or settings on the generated stories.

• Caption Clarity: The caption clearly explains the purpose of the table, which is to present the "Prompt Template for Story Generation." It also defines the placeholders "STORY PROMPT" and "WORD COUNT," which is helpful for understanding how the template is used. The connection to r/WritingPrompts is clearly stated.
• Table Organization and Content: The provided image of the table is very simple and easy to understand. It presents the prompt template in a clear and concise manner. The use of placeholders is intuitive, and the instructions within the prompt are straightforward.
• Reference Text Support: The reference text provides important context by explaining that 30 stories were collected from r/WritingPrompts and that corresponding AI-generated stories were created using the prompt in Table 15. This helps readers understand how the prompt template was used in the study.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* this prompt template is being used and how it relates to the overall goals of the study. Additionally, while the table is clear, it could benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Generating AI-Generated Stories Based on r/WritingPrompts Prompts." The table could also include an example of a filled-in prompt to further illustrate how it is used in practice. Finally, it would be helpful to clarify whether the prompt shown in the table is the *exact* prompt used for all story generations or if it was modified slightly for each prompt based on the specific content of the "STORY PROMPT" and "WORD COUNT" placeholders.

Table 16: Story performance of initial 5 annotators, which included 4 nonexpert annotators and expert annotator # 1, who was used in all article experiments.

• Purpose of the Table: This table shows how well a group of five people performed when they tried to distinguish between stories written by humans and stories generated by AI. This was an initial test to see how good people are at this task before doing the main experiments. It is like a practice round before the actual competition. The table compares the performance of four "nonexpert" annotators and one "expert" annotator, who was involved in all the experiments in the study. The term "story performance" likely refers to their accuracy in identifying whether a story was written by a human or an AI.
• Content of the Table: The table likely shows the True Positive Rate (TPR) and False Positive Rate (FPR) for each of the five annotators, and possibly their average scores. The TPR measures how often they correctly identified AI-generated stories as AI-generated. The FPR measures how often they incorrectly identified human-written stories as AI-generated. The reference text mentions that the nonexperts had an average TPR of 62.5%, which suggests that they were able to correctly identify AI-generated stories a little better than chance (which would be 50%). The table probably also includes the expert's individual performance metrics, allowing for a comparison between the expert and the nonexperts.
• Expert vs. Nonexpert: The distinction between "expert" and "nonexpert" is based on their experience with AI and writing, as established earlier in the paper. The expert annotator is likely someone who has a lot of experience with AI language models and is skilled at detecting AI-generated text. The nonexperts, on the other hand, have less experience with AI. Comparing their performance helps to understand how much of a difference expertise makes in this task. The fact that the expert was used in all article experiments suggests that their performance in this initial test was considered reliable enough to use them throughout the study.
• Initial Test Before Main Experiments: This table presents the results of an initial test, or a pilot study, conducted before the main experiments. Pilot studies are often used to refine the experimental design, test the procedures, and get a preliminary sense of the results. In this case, the initial test with the five annotators likely served to evaluate the difficulty of the task, assess the performance of the expert, and potentially identify any issues with the instructions or materials before proceeding with the larger study. The results of this initial test may have informed decisions about the selection of annotators or the design of subsequent experiments.

• Small Sample Size: The initial test involved only five annotators, which is a very small sample size. This limits the generalizability of the findings and makes it difficult to draw strong conclusions about the performance of experts vs. nonexperts. The authors should acknowledge this limitation and interpret the results with caution. A larger sample size would be needed to make more robust claims about the differences between these groups.
• Selection of Expert: The validity of the comparison between experts and nonexperts depends on how the expert was selected and whether their expertise is truly representative of experts in the field. The authors should provide more details about the criteria used to identify and select the expert annotator. They should also discuss any potential biases that might have been introduced by using only one expert.
• Task Difficulty: The performance of the annotators depends on the difficulty of the task, which in turn depends on the quality of the AI-generated stories. If the AI-generated stories were relatively easy to detect, even nonexperts might perform well. Conversely, if the stories were very difficult to detect, even the expert might struggle. The authors should discuss the difficulty of the task and provide more information about the characteristics of the stories used in this initial test.
• Potential Learning Effect: Since expert annotator #1 was used in all subsequent experiments, there's a potential for a learning effect. This annotator might have improved their performance over time simply due to repeated exposure to the task. The authors should consider this potential learning effect when interpreting the results and discuss whether it might have influenced the expert's performance in later experiments.

• Caption Clarity: The caption clearly states the purpose of the table: to present the "Story performance of initial 5 annotators." It also specifies the composition of the group (4 nonexperts and 1 expert) and highlights the expert's involvement in all article experiments. The caption is concise and informative.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. It clearly labels each annotator and presents their TPR and FPR scores. The table also includes an "Average" row, which is helpful for summarizing the overall performance of the nonexperts. The use of bolding for the "Average" row makes it stand out. The table effectively communicates the performance data in a clear and concise manner.
• Reference Text Support: The reference text provides additional context by stating the average TPR of the nonexperts (62.5%), which is consistent with the information presented in the table. This helps to reinforce the finding that nonexperts perform slightly better than chance but still relatively poorly.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* this initial test was conducted and how it relates to the overall goals of the study. Additionally, while the table is generally clear, it could benefit from a more descriptive title that reflects its purpose, such as "Initial Test of Annotator Performance on Story Detection Task." The table could also include a column for the overall accuracy of each annotator, in addition to TPR and FPR, to provide a more comprehensive measure of performance. Finally, explaining why the expert was chosen to be part of all article experiments would be helpful.

Figure 6: Interface for annotators, with an example annotation from Annotator #4 with a humanized article from §2.5. This is the same article displayed in Figure 1. An annotator can highlight texts, make their decision, put confidence, and write an explanation. This AI-generated article was based off of In Alaska, a pilot drops turkeys to rural homes for Thanksgiving, written by Mark Thiessen & Becky Bohrer, was originally published by Associated Press on November 28, 2024.

• Purpose of the Figure: This figure shows the interface, or the screen, that the annotators used to perform the task of labeling articles as human-written or AI-generated. It's like a screenshot of the software used for the study. The figure also includes an example of an annotation made by one of the annotators (Annotator #4) on an article that was "humanized," meaning it was generated by AI but modified to appear more human-like. This particular article is the same one shown in Figure 1, but here we see it within the annotation interface.
• Content of the Interface: The interface allows annotators to perform several actions. They can highlight text within the article, which is like using a highlighter pen on a printed page. They can then make their decision on whether the article was written by a human or an AI. They also provide a confidence score, indicating how sure they are about their decision, and write an explanation justifying their choice. The interface likely includes buttons or menus for these actions, as well as a way to move between articles. The provided image shows the article text, with some parts highlighted, and a section at the bottom for the annotator to choose "Human-Generated" or "Machine-Generated", rate their confidence, and write their explanation.
• Example Annotation: The caption mentions that the example annotation is from Annotator #4 on a "humanized" article. This means that the AI-generated text has been modified to make it more difficult to detect. The example likely shows which parts of the text the annotator highlighted as clues, their decision (human or AI), their confidence level, and their written explanation. By looking at the example, one can get a better sense of how the annotators performed the task and what kind of reasoning they used.
• Connection to Other Figures: The caption notes that this is the same article displayed in Figure 1, but without the title. The reference text clarifies that, unlike in Figure 1, the annotators did not see the titles of the articles when performing the annotation task. This was likely done to prevent any potential bias based on the title. The caption also provides information about the original human-written article that the AI-generated article was based on, including the title, authors, and publication. This allows for a comparison between the original and the AI-generated version.

• Standardization of Annotation Process: Using a dedicated interface for the annotation task helps to standardize the process across all annotators. This ensures that everyone is performing the task under the same conditions and using the same tools. Standardization is important for the scientific validity of the study because it reduces variability in the data that might be caused by differences in the annotation process.
• Blinding to Article Titles: The reference text highlights that annotators did not see the article titles during the annotation process, unlike in Figure 1. This is a crucial methodological detail, as it demonstrates an attempt to minimize potential bias. If annotators knew the titles, they might be influenced by their prior knowledge or assumptions about the publication or topic, which could affect their judgments. By blinding the annotators to the titles, the researchers are trying to ensure that the judgments are based solely on the content and style of the text.
• Control over Annotation Environment: The interface likely provides a controlled environment for the annotation task. This means that the researchers can control what information is presented to the annotators and how they interact with the text. This level of control is important for ensuring the consistency and reliability of the data collected.
• Potential for Interface Bias: While the interface helps to standardize the annotation process, it's also possible that the design of the interface itself could introduce biases. For example, the size of the text area, the placement of buttons, or the ease of highlighting could all potentially influence the annotators' behavior. The authors should consider these potential biases and discuss how they attempted to minimize them.

• Caption Clarity: The caption is generally clear and provides a good overview of the figure's content and purpose. It explains that the figure shows the interface for annotators, includes an example annotation, and relates it to Figure 1. It also provides information about the original article on which the AI-generated article was based. However, the caption could be improved by explicitly stating that the annotators did not see the titles during the task, as this is a key difference from Figure 1.
• Visual Presentation: The provided image of the interface is well-designed and easy to understand. The layout is clear, and the different elements of the interface (article text, highlighting, decision buttons, confidence rating, explanation box) are clearly separated. The use of color-coding for "Human-Generated Text" and "Machine-Generated Text" is intuitive and visually helpful. The example annotation from Annotator #4 is clearly visible and provides a good illustration of how the interface is used in practice.
• Reference Text Support: The reference text provides crucial information about the difference between Figure 1 and Figure 6 regarding the visibility of article titles. This helps to clarify the methodological choice made by the researchers and highlights the importance of blinding in this context.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* it's important to show the interface and how it relates to the overall goals of the study. Additionally, the figure itself could include labels for the different elements of the interface (e.g., "Article Text," "Highlighting Tool," "Decision Buttons," "Confidence Rating," "Explanation Box") to further enhance clarity. While the example annotation is helpful, it might be beneficial to include a separate figure or panel that shows the interface without any annotations, to give a clearer view of the basic layout. The connection between the AI-generated text and the original article could also be made more explicit, perhaps by including a side-by-side comparison of a passage from each.

Table 1: Mean and standard deviation (subcripted) of article length in words across experiments, computed by splitting on whitespaces.

• Purpose of the Table: This table shows the average length of articles used in the experiments, measured in words. It also shows how much the length varies. Imagine you have a group of articles; this table tells you the average number of words in those articles and how spread out those numbers are. This is important because the researchers want to make sure that the length of the articles doesn't unintentionally influence the results of their study. For example, if AI-generated articles were consistently much longer or shorter than human-written ones, it could make it easier for people to guess correctly based on length alone, rather than the actual writing style.
• Content of the Table: The table presents two main statistics for article length: the mean and the standard deviation. The "mean" is simply the average length of the articles, calculated by adding up the number of words in each article and dividing by the number of articles. The "standard deviation" is a measure of how much the lengths of the articles vary around the mean. A smaller standard deviation means that most articles are close to the average length, while a larger standard deviation means that the lengths are more spread out. The standard deviation is shown as a subscript number next to the mean. The table shows these statistics for both human-written and AI-generated articles across different experiments, which are labeled as Exp # with numbers 1 through 5, each representing a different experimental condition or setup. The models used to generate the AI articles are also listed, such as GPT-40, CLAUDE-3.5-SONNET, and 01-PRO.
• Method of Counting Words: The table specifies that the number of words was counted by "splitting on whitespaces." This means that the researchers counted the words by looking for spaces between them. For example, the sentence "The cat sat on the mat" would be counted as six words because there are five spaces. This is a common and straightforward way to count words in a text.

• Control for Confounding Variable: Controlling for article length is crucial in this study. If the lengths of human-written and AI-generated articles were significantly different, it could introduce a confounding variable, meaning that differences in detection accuracy could be attributed to length rather than the actual writing quality. By reporting the mean and standard deviation of article lengths, the authors demonstrate that they have considered this potential issue.
• Appropriateness of Statistical Measures: The use of mean and standard deviation is appropriate for summarizing the distribution of article lengths. These are standard statistical measures for describing the central tendency and variability of a dataset. The subscript notation for standard deviation is a common and accepted way to present this information concisely.
• Transparency of Methodology: The caption clearly states that the word count was computed by splitting on whitespaces. This level of detail enhances the transparency and reproducibility of the study. However, it's worth noting that different methods of counting words (e.g., using different tokenizers) could yield slightly different results. The authors acknowledge this by referencing further details in section §B.1.5.
• Completeness of Data: The table appears to present data for all five experiments mentioned in the paper, for both human-written and AI-generated articles. However, without seeing the actual table, it's difficult to assess whether there are any missing data points or inconsistencies. Assuming the table is complete, it provides a comprehensive overview of article lengths across the different experimental conditions.

• Clarity of Caption: The caption is generally clear and informative. It explains what the table shows (mean and standard deviation of article length), the units of measurement (words), and the method used for counting words (splitting on whitespaces). The use of the term "subscripted" to describe the standard deviation notation is technically correct and concise.
• Use of Technical Terminology: The caption uses some technical terms, such as "mean," "standard deviation," and "splitting on whitespaces." While these terms are relatively common in scientific contexts, they might not be immediately familiar to a lay audience. However, given that this is a scientific paper, the use of such terminology is appropriate and expected.
• Organization and Readability: Without the actual table, it is hard to fully assess. Assuming a standard tabular format, with clear labels for rows and columns. The table should be easy to read and understand. The use of separate rows for each experiment and columns for human-written and AI-generated articles would facilitate comparisons.
• Potential for Improvement: The communication effectiveness could be further improved by adding a brief explanation of why controlling for article length is important in the context of this study. This would help readers who may not be familiar with experimental design to understand the significance of the information presented in the table.

Table 2: On average, nonexperts perform similar to random chance at detecting AI-generated text, while experts are highly accurate.

• Main Idea Conveyed: This table summarizes the ability of two groups of people, "experts" and "nonexperts," to tell the difference between text written by humans and text written by AI. The main takeaway is that nonexperts are not very good at this task—they perform about as well as if they were just guessing randomly. On the other hand, experts are very good at telling the difference.
• Definition of Expert and Nonexpert: The reference text explains that "nonexperts" are people who either don't use Large Language Models (LLMs) at all or only use them for tasks that don't involve writing. LLMs are a type of artificial intelligence that can generate human-like text, such as ChatGPT. "Experts" are not explicitly defined in this excerpt, but we can infer that they are people who use LLMs frequently for writing-related tasks, based on the contrast with nonexperts and the findings in the paper.
• Performance Metrics: The table likely presents data on how well each group performed in the task. The reference text mentions two metrics: True Positive Rate (TPR) and False Positive Rate (FPR). TPR measures how often the annotators correctly identified AI-generated text as AI-generated. FPR measures how often they incorrectly identified human-written text as AI-generated. For the nonexperts, the average TPR is 56.7%, and the average FPR is 52.5%. These values are close to 50%, which is what you would expect if someone were just guessing randomly, like flipping a coin to make their decision.
• Random Chance Baseline: The caption mentions "random chance." In this context, "random chance" means the level of performance you would expect if someone were guessing without any real ability to distinguish between human and AI-generated text. If there are only two options (human or AI), random guessing would lead to correct answers about 50% of the time. The fact that nonexperts perform similarly to random chance suggests that they don't have any special ability to detect AI-generated text.

• Clarity of Definitions: The distinction between "experts" and "nonexperts" is based on self-reported usage of LLMs. While this is a reasonable starting point, the validity of the findings depends on the accuracy of these self-reports and the assumption that LLM usage directly correlates with the ability to detect AI-generated text. The paper should provide a more detailed operational definition of these groups and ideally include some objective measure of expertise.
• Appropriateness of Metrics: TPR and FPR are appropriate metrics for evaluating the performance of a binary classification task (human vs. AI). However, it would be beneficial to also report other metrics like precision, recall, F1-score, and Area Under the Receiver Operating Characteristic Curve (AUROC) to provide a more comprehensive assessment of performance. These metrics can offer a more nuanced view of the trade-offs between different types of errors.
• Statistical Significance: The caption states that nonexperts perform "similar to random chance." However, it's important to determine whether this similarity is statistically significant. The authors should conduct appropriate statistical tests (e.g., t-tests or chi-squared tests) to compare the performance of experts and nonexperts and report the p-values. This would help to establish whether the observed differences are likely due to chance or reflect a real difference in ability.
• Sample Size and Generalizability: The reference text mentions four nonexpert annotators. The scientific validity of the findings would be strengthened by having a larger sample size for both groups. A larger sample would increase the statistical power of the study and make the results more generalizable to a wider population. The authors should discuss the limitations related to sample size and the potential impact on the generalizability of their findings.

• Caption Clarity: The caption is concise and effectively communicates the main finding of the table: nonexperts perform at chance level, while experts are highly accurate. The use of the phrase "random chance" is intuitive and helps to convey the idea that nonexperts are essentially guessing.
• Table Content Clarity: Assuming the table presents the TPR, FPR, and potentially other relevant metrics for both groups, it should be relatively straightforward to interpret. Clear labels and organization are crucial for effective communication. However, without seeing the actual table, it's difficult to fully assess its clarity.
• Use of Jargon: The caption and reference text use some technical terms like "TPR" and "FPR." While these terms are defined in the reference text, they might not be familiar to all readers. In a scientific paper, it's generally acceptable to use such terminology, but the authors should ensure that they are clearly defined when first introduced.
• Potential Improvements: The communication effectiveness could be improved by including a brief explanation of the importance of distinguishing between experts and nonexperts in the caption. Additionally, providing a visual representation of the data, such as a bar graph comparing the performance of the two groups, could make the findings even more accessible to a wider audience.

Table 3: Performance of expert humans (top), existing automatic detectors (middle), and our prompt-based detectors (bottom), where each cell displays TPR (FPR). Colors indicate performance bins where darkest teal is best, orange is middling, and purple is worst. We further mark closed-source (7) and open-weights (S) detectors. The majority vote of expert humans ties Pangram Humanizers for highest overall TPR (99.3) without any false positives, while substantially outperforming all other detectors. While the majority vote is extremely reliable, individual annotator performance varies, especially on 01-PRO articles with and without humanization. Prompt-based detectors are unable to match the performance of either expert humans or closed-source detectors.

• Overall Purpose: This table compares the performance of different methods for detecting AI-generated text. It evaluates three types of detectors: expert humans, existing automatic detectors (software designed to detect AI text), and the researchers' own "prompt-based detectors," which likely involve using carefully designed prompts to query a language model in a way that helps reveal whether it generated a given text. The goal is to see how well each method can distinguish between human-written and AI-generated text.
• Performance Metrics: The table uses two main metrics to measure performance: True Positive Rate (TPR) and False Positive Rate (FPR). TPR represents the percentage of AI-generated texts that were correctly identified as AI-generated. FPR represents the percentage of human-written texts that were incorrectly identified as AI-generated. Each cell in the table shows the TPR and FPR for a specific detector and a specific type of AI-generated text, with the format TPR (FPR). For example, a cell showing 90 (5) would mean that the detector correctly identified 90% of AI-generated texts but also incorrectly flagged 5% of human-written texts as AI-generated.
• Color Coding: The table uses a color scheme to make it easier to compare performance levels. Darkest teal represents the best performance (high TPR and low FPR), orange represents middling performance, and purple represents the worst performance. This allows readers to quickly grasp the overall performance landscape without having to scrutinize each individual number.
• Types of Detectors: The table distinguishes between closed-source and open-weights detectors. "Closed-source" means that the internal workings of the detector are not publicly available—like a secret recipe. "Open-weights" means that the model's parameters (the values that determine how it operates) are publicly available. The caption uses symbols (7 and S) to mark these two types of detectors, although without seeing the table, it's unclear what specific symbols are used.
• Key Findings: The caption highlights several key findings. First, the majority vote of expert humans (meaning the answer given by at least 3 out of 5 experts) is very accurate, achieving a TPR of 99.3% with no false positives. This is as good as the best automatic detector, "Pangram Humanizers." Second, individual expert performance varies, especially when dealing with text generated by the 01-PRO model, which is likely a more advanced AI. Third, the researchers' prompt-based detectors are not as good as either expert humans or the best closed-source detectors.

• Comparison of Different Methods: Comparing the performance of expert humans, existing automatic detectors, and the authors' own prompt-based detectors is a scientifically sound approach. It allows for a direct evaluation of the effectiveness of different detection methods and provides a benchmark for the performance of the new prompt-based approach. This comparison is crucial for understanding the current state of AI-text detection and the potential of different methodologies.
• Use of Majority Vote: Using the majority vote of five expert annotators is a reasonable method for aggregating individual judgments and reducing the impact of individual biases or errors. This approach enhances the reliability of the human expert performance measure. However, it's important to also report the individual annotator performance to provide a more complete picture of the variability within the expert group, which the caption does acknowledge.
• Metrics Appropriateness: TPR and FPR are standard metrics for evaluating the performance of binary classification tasks. However, as mentioned before, reporting additional metrics like precision, recall, F1-score, and AUROC would provide a more comprehensive and nuanced evaluation of the detectors' performance. Different metrics can highlight different aspects of performance and potential trade-offs.
• Generalizability: The caption mentions that performance varies on 01-PRO articles with and without humanization. This highlights the importance of evaluating detectors on a diverse range of AI-generated text, as different models and generation techniques can produce text with varying characteristics. The generalizability of the findings would be strengthened by testing on an even wider range of AI models and text types.
• Prompt-Based Detector Limitations: The caption acknowledges that the prompt-based detectors are unable to match the performance of expert humans or closed-source detectors. This is a valuable finding, as it highlights the limitations of the proposed approach and suggests areas for future improvement. The scientific validity would be enhanced by a more detailed analysis of the specific weaknesses of the prompt-based detectors and potential reasons for their lower performance.

• Caption Clarity: The caption is quite long and dense, but it does provide a substantial amount of information about the table's content and key findings. The use of color coding and symbols is explained, although the specific symbols are not shown. The main findings regarding the performance of expert humans and the limitations of prompt-based detectors are clearly stated.
• Table Organization: Assuming a standard tabular format, with clear labels for rows (detectors) and columns (different types of AI-generated text or different experiments), the table should be relatively easy to understand. The grouping of detectors into three categories (expert humans, existing automatic detectors, and prompt-based detectors) is logical and facilitates comparisons.
• Use of Terminology: The caption uses some technical terms like "TPR," "FPR," "closed-source," and "open-weights." While these terms are appropriate for a scientific audience, they might not be immediately familiar to all readers. The caption does provide brief explanations of these terms, which is helpful.
• Potential Improvements: The communication effectiveness could be improved by simplifying the caption slightly and focusing on the most important findings. Additionally, providing a visual representation of the performance differences, such as a bar graph or a ROC curve, could make the results more immediately accessible and impactful. Including a brief explanation of why the 01-PRO model poses a particular challenge for detection would also be beneficial.

Figure 2: Expert confidence in their decisions drops when judging humanized articles generated by 01-PRO.

• Main Idea of the Figure: This figure shows how sure experts are about their judgments when they are trying to determine if a piece of text was written by a human or an AI. Specifically, it suggests that experts are less confident when the AI-generated text has been "humanized," which means it has been modified to make it look more like human writing. The AI model used to generate this humanized text is called 01-PRO.
• Confidence Scale: The experts' confidence is measured on a scale of 1 to 5. A score of 5 means they are very confident in their judgment, while a score of 1 means they are not confident at all. The reference text indicates that the average confidence for humanized 01-PRO articles was 4.21, which is lower than the average confidence scores for text used in Experiments 1, 2, and 3 (4.39, 4.38, and 4.48, respectively).
• What the Figure Likely Shows: Although we don't see the actual figure, it likely presents a visual comparison of these confidence scores. It might be a bar graph, for example, with different bars representing the average confidence for each experiment or type of text. The bar for humanized 01-PRO articles would likely be shorter than the bars for the other experiments, visually demonstrating the drop in confidence.
• Implication of Reduced Confidence: The reduced confidence when judging humanized 01-PRO articles suggests that this type of AI-generated text is more difficult for experts to distinguish from human-written text. This implies that the "humanization" techniques used are somewhat effective in making the AI output appear more human-like, at least to some extent, as it introduces uncertainty even for experts.

• Subjectivity of Confidence: Confidence scores are subjective and can be influenced by various factors, including individual differences in self-assessment and prior experience. However, using a quantitative scale allows for a more objective comparison than relying solely on qualitative descriptions of confidence. The validity of using confidence as a measure depends on the assumption that experts can accurately assess their own certainty.
• Statistical Significance: The caption states that confidence "drops" when judging humanized 01-PRO articles, and the reference text provides specific average confidence scores. However, it's crucial to determine whether this drop is statistically significant. The authors should perform appropriate statistical tests (e.g., t-tests) to compare the confidence scores across different experiments and report the p-values. This would help to establish whether the observed differences are likely due to chance or reflect a real effect of humanization.
• Operational Definition of Humanization: The scientific validity of the findings depends on how well "humanization" is defined and implemented. The paper should provide a clear and detailed description of the specific techniques used to humanize the 01-PRO articles. This would allow other researchers to replicate the study and verify the findings. Without a clear definition, it's difficult to assess the validity of the comparison between humanized and non-humanized text.
• Relationship to Performance: While the figure focuses on confidence, it's important to consider the relationship between confidence and actual performance (e.g., TPR and FPR). Are experts less confident because they are making more mistakes, or are they simply more cautious when judging humanized text? The paper should explore this relationship to provide a more complete understanding of the impact of humanization on detection difficulty.

• Caption Clarity: The caption clearly states the main finding: expert confidence decreases when judging humanized 01-PRO articles. The use of the term "humanized" is consistent with the rest of the paper, although it might require further explanation for readers unfamiliar with the concept.
• Reference Text Support: The reference text provides specific numerical values for average confidence scores, which supports the claim made in the caption. It also directs the reader to Figure 2 for more details, indicating that the figure likely provides a visual representation of these data.
• Visual Representation: Without seeing the actual figure, it's difficult to fully assess its communication effectiveness. However, a well-designed graph (e.g., a bar graph or line graph) could effectively communicate the differences in confidence scores across experiments. Clear labels, a meaningful y-axis scale, and potentially error bars to represent the variability in confidence scores would enhance the figure's clarity.
• Potential Improvements: The communication effectiveness could be improved by briefly explaining in the caption why a drop in confidence is significant in the context of this study. Additionally, the figure itself could include a brief explanation of the 1-5 confidence scale and potentially a visual indicator of statistical significance (e.g., asterisks to mark significant differences between groups).

Table 6: List of publications included in HUMAN DETECTORS. The section is provided as listed as the section of the publication website where the article was published. All articles were taken from publications that wrote using American English.

• Purpose of the Table: This table lists the sources of the articles used in the study. The articles are the texts that the expert annotators are trying to determine were written by humans or AI. The name "HUMAN DETECTORS" likely refers to the dataset of articles that the researchers put together for this study. Think of it like a bibliography or a list of ingredients used in a recipe. It tells you where the materials for the study came from.
• Content of the Table: The table likely lists the names of the publications where the articles were originally published, such as newspapers or magazines. For each publication, it also specifies the "section" where the article appeared on the publication's website. For example, an article might be listed as coming from the "Science" section of the "New York Times" website. This is important because different sections might have different writing styles or focus on different topics.
• Use of American English: The caption specifies that all the articles are from publications that use American English. This is important because language can vary between different regions and dialects. By focusing on American English, the researchers are controlling for potential variations in language that could affect the results. It's like making sure all the ingredients in a recipe are measured using the same units, like cups instead of ounces.
• Connection to Corpus and Figure 7: The reference text mentions that the articles listed in Table 6 are part of the study's "corpus." A corpus, in this context, is a collection of texts used for research. It's like a library of texts that the researchers are studying. The reference text also mentions that Figure 7 shows the "section distribution," which likely means a chart or graph that shows how many articles were taken from each section of the publications. For example, it might show that 20% of the articles came from "Science" sections, 15% from "Technology" sections, and so on.

• Representativeness of the Corpus: The scientific validity of the study's findings depends on how well the articles in the corpus represent the broader population of human-written and AI-generated text. The authors should provide a clear rationale for their selection of publications and sections. They should discuss whether the chosen publications are representative of the types of writing they are interested in studying and whether the sections within those publications are appropriate for their research question. For example, are they focusing on news articles, opinion pieces, scientific articles, or a mix? Are certain types of publications or sections overrepresented or underrepresented?
• Potential Biases in Selection: The selection of publications and sections could introduce biases into the study. For example, if the researchers only selected articles from publications known for high-quality writing, it might be easier for annotators to distinguish between human and AI-generated text compared to a more diverse set of publications. Similarly, if certain sections (e.g., "Science") tend to have more specialized vocabulary or complex sentence structures, it might be easier to identify AI-generated text in those sections. The authors should discuss potential biases in their selection criteria and how they might affect the results.
• Reproducibility: Providing a detailed list of the publications and sections used in the study enhances the reproducibility of the research. Other researchers could potentially use the same list to replicate the study or conduct further analyses. However, the authors should also provide the specific criteria they used for selecting articles from each publication and section (e.g., date range, length, topic) to ensure that other researchers can accurately reconstruct the corpus.
• Focus on American English: Limiting the articles to those written in American English is a reasonable decision, as it controls for potential variations in language that could confound the results. However, it's important to acknowledge that this limits the generalizability of the findings to other varieties of English. The authors should discuss the implications of this choice for the scope of their conclusions.

• Caption Clarity: The caption is relatively clear and explains the purpose of the table, which is to list the publications from which the articles were sourced. It also clarifies that the "section" refers to the section of the publication's website where each article was found and that all articles are from publications using American English.
• Table Organization and Content: Assuming a standard tabular format with clear labels for rows (publications) and columns (sections), the table should be easy to understand. The specific publications and sections included in the table should be clearly identified and relevant to the study's focus on detecting AI-generated text. The table should also be organized in a logical manner, perhaps alphabetically by publication name or grouped by type of publication (e.g., newspapers, magazines, online sources).
• Reference Text Support: The reference text provides additional context by explaining that Table 6 lists all publications in the corpus and that Figure 7 shows the section distribution. This helps readers understand the relationship between the table and the broader study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* it's important to list the publications and sections used in the study. Additionally, the table itself could include a column indicating the number of articles taken from each publication and section, providing a more quantitative overview of the corpus composition. The footnote 22, mentioned in the reference text should also be included for completeness.

• Purpose of the Figure: This figure shows the breakdown of where the articles used in the study came from, specifically from which sections of the publications they were published in. For example, if the study used articles from a newspaper, this figure would show how many articles came from the "Science" section, how many from the "Business" section, how many from "Sports," and so on. The phrase "across all trials" suggests that the data is combined across the different experimental conditions or tests conducted in the study.
• Type of Visualization: While the caption doesn't explicitly state what type of visualization is used, it's likely a bar graph or a pie chart. These are common ways to represent the distribution of data across different categories. In a bar graph, each section (e.g., "Science," "Business") would have a bar, and the height of the bar would represent the number or percentage of articles from that section. In a pie chart, each section would be represented by a slice of the pie, with the size of the slice corresponding to its proportion of the total articles.
• Connection to Table 6: The reference text indicates that Figure 7 is related to Table 6, which lists all the publications included in the study's corpus (the collection of texts used for analysis). Figure 7 essentially provides a visual summary of the "section" information from Table 6, showing the distribution of articles across different sections in a graphical format. This makes it easier to see which sections contributed the most articles to the study.
• Importance of Section Distribution: Understanding the section distribution is important because different sections of a publication might have different writing styles, levels of formality, or subject matter. For example, articles from a "Science" section might be more technical and fact-based than articles from an "Opinion" section. By showing the section distribution, the researchers are providing information about the diversity and representativeness of the articles used in the study. This can help readers understand the context of the findings and potential limitations to generalizability. For example, if all articles are from the science section, the results may not apply as well to articles from the sports section.

• Representativeness of the Sample: The scientific validity of the study's findings depends, in part, on how well the articles used represent the broader population of human-written and AI-generated text. The section distribution provides some information about the diversity of the articles, but it's important to consider whether the chosen sections are representative of the types of writing the researchers are interested in studying. The authors should provide a rationale for their selection of sections and discuss any potential biases that might arise from this selection.
• Potential Confounding Variables: The section from which an article is taken could be a confounding variable. For example, if articles from certain sections are easier or harder for annotators to classify correctly, this could affect the overall results. By presenting the section distribution, the authors are acknowledging this potential confounder and allowing readers to consider its possible impact. Ideally, the authors should also analyze the performance of annotators separately for different sections to see if there are any significant differences.
• Transparency and Reproducibility: Providing a clear breakdown of the section distribution enhances the transparency of the study. It allows other researchers to understand the composition of the corpus and potentially replicate the study using a similar set of articles. This contributes to the overall rigor and reproducibility of the research.

• Caption Clarity: The caption is concise and clearly states the purpose of the figure, which is to show the "section distribution of articles across all trials." However, it could be improved by briefly defining what "section distribution" means for readers who may be unfamiliar with the term.
• Visual Clarity: Without seeing the actual figure, it's difficult to fully assess its visual clarity. However, assuming it's a well-designed bar graph or pie chart, it should be relatively easy to understand. Clear labels for each section, a meaningful scale for the y-axis (in a bar graph) or percentages (in a pie chart), and a legend (if necessary) are crucial for effective communication. The use of color should be consistent with other figures in the paper and should be chosen to ensure that the figure is accessible to individuals with color vision deficiencies.
• Connection to Table 6: The reference text clearly links Figure 7 to Table 6, indicating that the figure provides a visual representation of the section information from the table. This helps readers understand the relationship between the two elements and how they contribute to the overall description of the corpus.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* the section distribution is important in the context of this study. Additionally, the figure itself could include a brief summary of the key takeaways from the distribution, such as highlighting the sections that contributed the most articles or any notable imbalances in the distribution. The specific type of chart (e.g., bar graph, pie chart) could also be mentioned in the caption to aid in visualization for the reader.

• Purpose of the Figure: This figure displays how the lengths of the articles used in the study vary across the different experiments. In this study, "experiments" are like different rounds of testing. The length of an article is measured by its word count, which is simply the number of words in the article. It's like counting how many words are in a sentence, but on a larger scale. The figure shows the distribution of these word counts, which means it shows how many articles are short, how many are medium-length, and how many are long, for each experiment.
• Type of Visualization: The figure is described as showing a "distribution," so it's likely a histogram, a box plot, or a violin plot. These are all graphical ways to show how a set of values (in this case, word counts) is spread out. A histogram groups the data into bins (e.g., 0-100 words, 101-200 words, etc.) and shows how many articles fall into each bin. A box plot shows the median (middle value), quartiles (values that divide the data into quarters), and potential outliers (extreme values). A violin plot is similar to a box plot but also shows the density of the data at different values, like a smoothed histogram. The specific type of plot used will affect how the information is presented visually.
• Focus on Word Count: The figure specifically focuses on word count as the measure of article length. This is a common and straightforward way to quantify the size of a text. By showing the distribution of word counts, the researchers are providing information about the variability in article length within each experiment. This is important because they want to make sure that differences in length aren't influencing the results. For example, if all the AI-generated articles were much longer than the human-written articles, it might be easier for annotators to guess correctly based on length alone, rather than the actual writing style.
• Per Experiment Breakdown: The figure shows the word count distribution "per experiment," meaning that the data is presented separately for each experimental condition or trial. This allows for comparisons between experiments, to see if the distribution of article lengths is consistent across different conditions. For example, the researchers might want to check if the articles used in Experiment 1 are similar in length to those used in Experiment 2, or if one experiment used a wider range of article lengths than another.

• Control for Confounding Variable: Article length could be a confounding variable in this study, as it might influence the ability of annotators to distinguish between human-written and AI-generated text. By presenting the distribution of word counts per experiment, the authors are demonstrating that they have considered this potential confounder and are providing data to assess its impact. This enhances the scientific validity of the study by allowing readers to evaluate whether differences in article length might have affected the results.
• Appropriateness of Visualization: The choice of a histogram, box plot, or violin plot to visualize the distribution of word counts is appropriate. These types of plots are commonly used to display the distribution of a continuous variable and can reveal important features of the data, such as its central tendency, spread, and skewness. The specific choice of plot might depend on the particular aspects of the distribution that the authors want to emphasize.
• Statistical Analysis: While the figure provides a visual representation of the word count distribution, it's important to also perform statistical analyses to determine whether there are significant differences in article length between experiments or between human-written and AI-generated articles within each experiment. The authors should report relevant statistical measures (e.g., mean, standard deviation, range) and conduct appropriate statistical tests (e.g., t-tests, ANOVA) to compare the distributions.
• Transparency and Reproducibility: Presenting the word count distribution per experiment enhances the transparency of the study by providing detailed information about the articles used. This allows other researchers to better understand the characteristics of the corpus and potentially replicate the study using articles with similar length distributions. To further enhance reproducibility, the authors should also provide the raw data or summary statistics for each experiment.

• Caption Clarity: The caption is concise and clearly states the purpose of the figure, which is to show the "Word Count Distribution Per Experiment." However, it could be improved by briefly explaining *why* this distribution is important in the context of the study (i.e., to control for article length as a potential confounding variable).
• Visual Clarity: Without seeing the actual figure, it's difficult to fully assess its visual clarity. However, assuming it's a well-designed histogram, box plot, or violin plot, it should be relatively easy to understand. Clear axis labels (e.g., "Word Count," "Experiment"), a meaningful scale, and a legend (if necessary) are crucial for effective communication. The use of different colors or patterns to distinguish between experiments can also enhance clarity. The specific design choices will affect how effectively the figure communicates the distribution of word counts.
• Potential Improvements: The communication effectiveness could be improved by adding a brief interpretation of the key findings from the figure to the caption. For example, the caption could mention whether the distributions are similar across experiments or if there are any notable differences. Additionally, the figure itself could include visual indicators of statistical significance (e.g., asterisks to mark significant differences between groups) if statistical tests were performed. The figure could also benefit from including a visual representation of the mean or median word count for each experiment, to facilitate comparisons.
• Use of Color to Distinguish Human and AI Articles: The provided image uses a teal color for the Human-written articles and a dark green color for the AI-generated articles. This helps visually distinguish between these two categories in the boxplots. Using distinct colors is a good practice for enhancing clarity, especially when comparing two different groups. The colors are different enough to be easily distinguished but not so contrasting that they become visually jarring. This choice contributes positively to the overall communication effectiveness of the figure.

• Purpose of the Table: This table provides information about the length of the articles used in the study, specifically focusing on the number of "tokens" and "words" in each article. It breaks down this information by the source of the articles, which are the different publications they came from (like newspapers or magazines). "Tokens" are the basic units of text that a computer uses for analysis - they can be words, parts of words, or even punctuation marks. Counting both tokens and words gives a more detailed picture of article length. Think of it like measuring the length of a train both by the number of carriages and the total length in meters - both measurements are useful, but they tell you slightly different things.
• Content of the Table: The table likely shows the average number of tokens and words for articles from each publication. It probably also includes measures of variability, like the standard deviation, which tells you how much the lengths of the articles vary around the average. For example, it might show that articles from one publication have an average of 500 words with a standard deviation of 50, while articles from another publication have an average of 800 words with a standard deviation of 100. This means that the articles from the second publication are generally longer and have more variation in their length.
• Importance of Article Length: The length of the articles is important because it could potentially influence the results of the study. If the AI-generated articles were consistently much longer or shorter than the human-written articles, it might make it easier for the annotators to guess correctly based on length alone, rather than the actual quality of the writing. By providing statistics on article length, the researchers are showing that they have considered this potential issue and are providing data to help readers evaluate whether length might have played a role in the findings. It is like ensuring that runners in a race all start at the same point on the track, to make sure that the race is fair.
• Breakdown by Source: The table breaks down the information by the source of the articles, meaning the specific publication they came from. This is important because different publications might have different writing styles or standards for article length. For example, a scientific journal might have longer and more complex articles than a tabloid newspaper. By showing the statistics for each publication separately, the researchers are providing a more nuanced picture of the articles used in the study and allowing readers to see if there are any systematic differences in length between publications.

• Control for Confounding Variable: As mentioned in previous analyses, article length could be a confounding variable in this study. By providing statistics on the number of tokens and words per article, broken down by publication, the authors are demonstrating that they have considered this potential issue. This allows readers to assess whether differences in article length might have influenced the annotators' ability to distinguish between human-written and AI-generated text.
• Appropriateness of Measures: Reporting both the number of tokens and the number of words is a good practice, as it provides a more complete picture of article length. Tokens can capture nuances that might be missed by simply counting words, such as the use of compound words or hyphenated phrases. The specific tokenization method used should be clearly defined in the methods section of the paper.
• Statistical Measures: The table likely includes measures of central tendency (e.g., mean) and variability (e.g., standard deviation) for both tokens and words. These are appropriate statistical measures for summarizing the distribution of article lengths. The authors should also consider reporting other relevant statistics, such as the range (minimum and maximum values) or percentiles, to provide a more detailed description of the distributions.
• Comparison Between Sources: The breakdown by publication allows for comparisons between different sources of articles. This is important for assessing the diversity of the corpus and identifying any potential biases related to the selection of publications. The authors should discuss any notable differences in article length between publications and consider their potential implications for the study's findings.

• Caption Clarity: The caption is concise and clearly states the purpose of the table, which is to present the number of tokens and words across articles, broken down by source. The use of the term "tokens" might be unfamiliar to some readers, but the caption does implicitly link it to word count, which is a more familiar concept.
• Table Organization and Content: Assuming a standard tabular format with clear labels for rows (publications) and columns (number of tokens, number of words, and potentially other statistics), the table should be relatively easy to understand. The specific publications included in the table should be clearly identified and relevant to the study's focus on detecting AI-generated text. The table should also be organized in a logical manner, perhaps alphabetically by publication name or grouped by type of publication.
• Reference Text Support: The reference text provides minimal additional context, simply stating that the table provides statistics for articles by publication. However, it does reinforce the connection between the table and the broader study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* it's important to report the number of tokens and words by source. Additionally, the table itself could include a column indicating the total number of articles from each publication, providing a clearer sense of the contribution of each source to the overall corpus. A brief definition of "tokens" in the caption or as a footnote to the table would also enhance clarity for readers unfamiliar with the term. Finally, although the caption mentions "by source", the provided image of the table has a title that says "by publication". This should be made consistent.

• Purpose of the Table: This table shows the specific instructions, called a "prompt template," that the researchers used to generate paraphrased versions of articles for Experiment 2 in their study. A prompt is like a set of instructions given to an AI model to get it to produce a specific kind of output. In this case, the prompt is designed to get the AI to rewrite an existing article in a different way, while keeping the main meaning the same. This process is called paraphrasing. Think of it like asking someone to explain the same idea using different words. The table shows the exact wording of the instructions given to the AI model.
• Content of the Table: The table likely shows the different parts of the prompt template, which includes placeholders for specific information about the article being paraphrased. The reference text mentions that the researchers provided the models with the article's title, subtitle, approximate length, publication, and section. So, the prompt template probably has slots for each of these pieces of information. For example, it might have a section that says: "Title: [Insert Title Here], Subtitle: [Insert Subtitle Here], Length: [Insert Length Here]". The researchers would then fill in these slots with the actual information from the article they want to paraphrase. The prompt also likely includes instructions on how the AI should paraphrase the text, such as "Rewrite the following article using different words but keeping the same meaning." The table also has separate prompts for article generation and paraphrasing.
• Experiment 2 Focus: The caption specifies that this prompt template was used for Experiment 2, which involved paraphrasing. This suggests that Experiment 2 focused on testing whether humans could detect AI-generated text when the AI was specifically instructed to paraphrase existing articles. This is different from simply generating text from scratch, as paraphrasing requires the AI to understand and rephrase existing content.
• Use of API and Models: The reference text mentions that closed-source models were prompted using the provider's API. An API (Application Programming Interface) is a way for computer programs to interact with each other. In this case, it's a way for the researchers to send instructions to the AI models and receive their output. "Closed-source" means that the internal workings of these models are not publicly available. The fact that they used the provider's API suggests that they were working with commercially available AI models, rather than models they developed themselves.

• Reproducibility: Providing the exact prompt template used for paraphrasing is crucial for the reproducibility of the study. Other researchers can use the same template to generate their own paraphrased articles and attempt to replicate the findings of Experiment 2. This enhances the transparency and scientific rigor of the research.
• Control over Paraphrasing Process: By using a specific prompt template, the researchers have a degree of control over the paraphrasing process. They can specify the desired length, style, and other characteristics of the paraphrased text. However, the specific instructions included in the prompt will significantly influence the output of the AI model. The authors should carefully consider the wording of the prompt to ensure that it elicits the desired type of paraphrasing and does not introduce any unintended biases.
• Validity of Paraphrasing Task: The validity of using paraphrased articles to test human detection of AI-generated text depends on how well the paraphrasing task reflects real-world scenarios. The authors should discuss the rationale for using paraphrasing and its relevance to the broader question of AI detection. They should also acknowledge any limitations of using paraphrased text compared to text generated entirely from scratch.
• Dependence on Model Capabilities: The quality of the paraphrased text will depend on the capabilities of the specific AI models used. Different models may have different strengths and weaknesses when it comes to paraphrasing. The authors should discuss the specific models used in Experiment 2 and any known limitations of those models that might affect the results. The reference text mentions using both closed-source models and, presumably, open-source or other models for comparison, which is a good practice for assessing the generalizability of the findings across different types of models.

• Caption Clarity: The caption is relatively clear and concisely states the purpose of the table, which is to present the "Prompt Template for Experiment 2 Paraphrasing." However, it could be improved by briefly explaining what a prompt template is and why it's important in the context of the study.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. It clearly separates the "Article Generation Prompt" from the "Paraphrase Prompt," and further divides the paraphrase prompt into "Initial Sentence Only" and a general prompt. Each section of the prompt is clearly labeled, and the placeholders for specific information (e.g., YOUR TITLE, YOUR SECTION) are easily identifiable. The instructions within each prompt are concise and relatively straightforward.
• Reference Text Support: The reference text provides important context about how the prompt template was used, including the types of information provided to the models (title, subtitle, length, etc.) and the use of APIs for closed-source models. This helps readers understand the broader methodology behind the generation of paraphrased articles.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of how the prompt template was used in Experiment 2. For example, the caption could mention that the template was used to generate paraphrased versions of articles from the corpus, which were then evaluated by human annotators. Additionally, the table itself could include a brief example of a filled-in prompt to further illustrate how the template is used in practice. While the table is generally clear, adding a sentence or two explaining the rationale behind the specific instructions in the prompt (e.g., why the model is asked to "make it seem like a human wrote the article") could enhance understanding for readers unfamiliar with AI prompting techniques.

• Purpose of the Table: This table provides the specific instructions, or "prompt template," used to tell an AI model how to paraphrase the first sentence of an article in Experiment 2. Paraphrasing means rewriting something using different words while keeping the original meaning. In this study, the researchers are using paraphrasing to create versions of articles that might be harder for humans to distinguish from human-written text. This table shows the exact wording of the prompt given to the AI for paraphrasing the initial sentence, while another table (Table 10) shows the prompt for the rest of the sentences.
• Content of the Table: The table displays the "Prompt Template," which is like a fill-in-the-blank instruction sheet for the AI. It has placeholders for specific information about the article, such as "YOUR SECTION" and "YOUR PUBLICATION," which would be replaced with the actual section and publication of the article being paraphrased. The prompt instructs the AI to "Paraphrase the given sentence," to "Only return the paraphrased sentence," and to "Make it seem like a human wrote the article and that it is from the [specified section] of [specified publication]." This tells the AI to not only change the wording but also to mimic a human writing style appropriate for the given source.
• Focus on Initial Sentence: The table specifically focuses on the prompt for the *initial* sentence of the article. This suggests that the researchers might be treating the first sentence differently than the rest of the text. There might be several reasons for this. For example, the first sentence often sets the tone and introduces the main topic, so it might require special handling to ensure the paraphrased version flows well and accurately reflects the original article's intent. Or, it might be more difficult to paraphrase just a single sentence without any prior context.
• Connection to Other Tables: The reference text mentions that Table 10 provides the prompt for the remaining sentences. This means that the researchers are using a slightly different approach for paraphrasing the first sentence compared to the rest of the article. By looking at both tables, one can understand the complete instructions given to the AI for paraphrasing the entire article. It is like having one set of instructions for the first step of a recipe and another set of instructions for all the following steps.

• Reproducibility and Transparency: Providing the exact prompt template used for paraphrasing the initial sentence is crucial for the reproducibility and transparency of the study. Other researchers can use the same template to replicate the paraphrasing process and verify the findings. This level of detail allows for a clear understanding of how the paraphrased articles were generated.
• Rationale for Separate Treatment of Initial Sentence: The scientific validity of treating the initial sentence differently depends on the rationale behind this decision. The authors should provide a clear explanation for why they used a separate prompt for the first sentence. Is it because the first sentence plays a unique role in the article? Is it more challenging to paraphrase in isolation? Or are there other reasons related to the specific AI model or the paraphrasing task? Without a clear justification, it's difficult to assess the validity of this approach.
• Impact on Paraphrasing Quality: The specific wording of the prompt will significantly influence the quality and characteristics of the paraphrased text. The instructions to "make it seem like a human wrote the article" and to consider the specified section and publication are designed to elicit more natural and context-appropriate paraphrases. However, the effectiveness of these instructions depends on the capabilities of the AI model being used. The authors should discuss the potential limitations of the prompt and the AI model in achieving high-quality paraphrases.
• Consistency with Overall Methodology: The use of a separate prompt for the initial sentence should be consistent with the overall methodology of Experiment 2 and the broader goals of the study. The authors should clearly explain how this approach fits into their experimental design and how it contributes to answering their research questions about human detection of AI-generated text. The specific parameters of the prompt should also be justified in relation to the overall goals.

• Caption Clarity: The caption is concise and clearly states the purpose of the table: to present the "Prompt Template for Experiment 2 Paraphrasing." It correctly identifies that the table is specifically for paraphrasing, which is an important detail. It could be improved by briefly mentioning that this table focuses on the initial sentence, while Table 10 focuses on the rest.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the placeholders for specific information are easily identifiable. The instructions within the prompt are concise and relatively straightforward, although they might require some familiarity with AI prompting techniques to fully grasp.
• Reference Text Support: The reference text effectively connects Table 9 to Table 10, indicating that Table 9 focuses on the initial sentence while Table 10 covers the rest of the sentences. This helps readers understand the relationship between the two tables and the overall approach to paraphrasing.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* a separate prompt is used for the initial sentence. Additionally, while the table is generally clear, adding a sentence or two explaining the rationale behind the specific instructions in the prompt (e.g., why the model is asked to consider the section and publication) could enhance understanding for readers unfamiliar with AI prompting techniques. It may also be helpful to clarify that this prompt is specifically for generating the *paraphrased* version of the articles, as the table also contains the prompt for generating the original articles.

• Purpose of the Table: This table shows the specific instructions, or "prompt template," that the researchers used to tell an AI model how to paraphrase sentences from an article *after* the first sentence. Remember, paraphrasing means rewriting something using different words but keeping the original meaning. In this study, Experiment 2 focuses on using paraphrased text to see if humans can still detect that it was generated by an AI. This table complements Table 9, which showed the prompt for paraphrasing just the first sentence. Together, they provide the complete instructions for paraphrasing an entire article.
• Content of the Table: The table displays the "Prompt Template," which acts like a fill-in-the-blank instruction sheet for the AI. It includes placeholders for specific information about the article and the part that has already been paraphrased. These placeholders are marked with things like "YOUR SECTION," "YOUR PUBLICATION," and "YOUR PARAPHRASED ARTICLE SO FAR." The researchers would replace these with the actual section and publication of the article, and the portion of the article that has already been paraphrased. The prompt instructs the AI to "Paraphrase the given sentence," to "Only return the paraphrased sentence," and to "Make it seem like a human wrote the article and that it is from the [specified section] of [specified publication]." It also tells the AI to consider the "Previous context" which is the part of the article that has already been paraphrased. This helps the AI maintain consistency and flow throughout the entire paraphrased article.
• Focus on Subsequent Sentences: Unlike Table 9, which focused on the initial sentence, this table provides the prompt for paraphrasing all sentences *after* the first one. This suggests that the researchers are using a slightly different approach for the bulk of the text compared to the very beginning. By providing the "Previous context" to the AI, the researchers are trying to ensure that each newly paraphrased sentence fits well with what has already been rewritten. It's like building a wall brick by brick, where each new brick needs to align with the ones already in place.
• Connection to Table 9: The reference text directly links Table 10 to Table 9, emphasizing that they work together to provide the complete instructions for paraphrasing. Table 9 handles the first sentence, and Table 10 handles the rest. By looking at both tables, one can understand the full process the researchers used to generate the paraphrased articles used in their experiment. This is important for ensuring that the study can be replicated by others.

• Reproducibility and Transparency: Providing the exact prompt template used for paraphrasing is crucial for the reproducibility and transparency of the study. Other researchers can use this template, along with the one in Table 9, to replicate the paraphrasing process and potentially verify the findings. This level of detail allows for a clear understanding of how the paraphrased articles were generated, which is essential for evaluating the scientific rigor of the experiment.
• Rationale for Separate Prompts: Using different prompts for the initial sentence (Table 9) and subsequent sentences (Table 10) suggests a deliberate methodological choice. The authors should provide a clear rationale for this decision. It's possible that the initial sentence requires special treatment to ensure a natural and coherent introduction, while subsequent sentences can be paraphrased more effectively by considering the preceding context. The validity of this approach depends on whether this distinction is justified and whether it improves the overall quality of the paraphrased text.
• Importance of Context: The inclusion of "Previous context" in the prompt highlights the importance of maintaining coherence and flow during paraphrasing. By providing the AI with the already paraphrased portion of the article, the researchers are attempting to generate more natural-sounding and contextually appropriate paraphrases. The scientific validity of this approach depends on how effectively the AI model can utilize this context to improve the quality of the paraphrased sentences. The authors should discuss the capabilities and limitations of the chosen AI model in this regard.
• Consistency with Experimental Goals: The prompt template should be consistent with the overall goals of Experiment 2 and the broader aims of the study. The instructions to "make it seem like a human wrote the article" and to consider the specified section and publication are designed to elicit more realistic and context-appropriate paraphrases. The authors should clearly explain how this approach contributes to their investigation of human detection of AI-generated text and how it helps to address their research questions.

• Caption Clarity: The caption is concise and clearly states the purpose of the table: to present the "Prompt Template for Experiment 2 Paraphrasing." It correctly identifies that the table is specifically for paraphrasing. It could be improved by briefly mentioning that this table focuses on sentences *after* the initial sentence, while Table 9 focuses on the first sentence.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the placeholders for specific information are easily identifiable. The instructions within the prompt are concise and relatively straightforward, although they might require some familiarity with AI prompting techniques to fully grasp.
• Reference Text Support: The reference text effectively connects Table 10 to Table 9, indicating that Table 9 focuses on the initial sentence while Table 10 covers the rest of the sentences. This helps readers understand the relationship between the two tables and the overall approach to paraphrasing.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of *why* a separate prompt is used for subsequent sentences compared to the initial sentence. Additionally, while the table is generally clear, adding a sentence or two explaining the rationale behind the specific instructions in the prompt (e.g., why the model is asked to consider the "Previous context") could enhance understanding for readers unfamiliar with AI prompting techniques. It may also be helpful to clarify that this prompt is specifically for generating the *paraphrased* version of the articles, as the table also contains the prompt for generating the original articles.

Table 11: A truncated version of the AI Text Detection Guide. The full guide is located at https://github.com/ jenna-russell/human_detectors.

• Purpose of the Table: This table presents a shortened version of a guide designed to help people identify text that has been generated by AI. The guide is called the "AI Text Detection Guide," and it's based on insights from experts who are good at telling the difference between human-written and AI-generated text. Think of it like a cheat sheet or a set of tips and tricks to help someone spot an AI imposter. The full, unabridged version of this guide is available online at a specific GitHub repository, which is a place where people can store and share documents and code.
• Content of the Guide: The guide is organized into different sections, each focusing on a specific aspect of writing, such as vocabulary, grammar, tone, introductions, and conclusions. For each section, the guide provides explanations and examples of how AI writing typically differs from human writing in that area. For instance, it might point out that AI tends to overuse certain words or phrases (vocabulary), make fewer grammatical errors (grammar), or have a consistently formal or neutral tone. By studying these differences, someone can learn to spot patterns that might indicate a text was generated by an AI.
• Development of the Guide: The reference text explains that the guide was created by consulting with experts in AI text detection. These experts were paid for their time and asked to provide a list of "clues" they look for when trying to determine if a text is AI-generated. The researchers then took these clues and organized them into a unified guidebook. This process is like interviewing a group of experienced detectives to find out what they look for when solving a case and then compiling their insights into a training manual for other detectives.
• Truncated Version: The caption specifies that this table shows a "truncated version" of the guide, which means it's a shortened or abridged version. Some parts of the full guide have been left out, probably to save space or to focus on the most important clues. It's like presenting the key takeaways from a longer document, rather than the entire thing. However, the full guide is available online for those who want to see all the details.

• Expert-Driven Approach: The methodology of consulting experts to develop the AI Text Detection Guide is a strength of this approach. By leveraging the knowledge and experience of individuals who are skilled at identifying AI-generated text, the researchers are grounding the guide in real-world expertise. This expert-driven approach increases the likelihood that the guide will capture relevant and diagnostic clues for distinguishing between human and AI writing.
• Subjectivity and Potential Biases: While relying on expert input is valuable, it's important to acknowledge the potential for subjectivity and bias in the experts' judgments. The specific clues identified by the experts might reflect their individual experiences and may not be universally applicable to all types of AI-generated text. The authors should discuss the potential for such biases and how they attempted to mitigate them during the development of the guide.
• Organization and Structure of the Guide: The reference text indicates that the guide is organized into sections based on different aspects of writing (e.g., vocabulary, grammar, tone). This is a logical and systematic way to structure the guide, as it allows for a focused examination of different linguistic features. The scientific validity of the guide's organization depends on whether these categories are comprehensive, mutually exclusive, and relevant to the task of AI detection. The authors should provide a clear rationale for their choice of categories.
• Generalizability: The effectiveness of the guide likely depends on the specific types of AI-generated text considered. The guide might be more effective for detecting text generated by models similar to those that the experts are familiar with. The authors should discuss the potential limitations of the guide's generalizability to different AI models, writing styles, and domains. They should also consider the possibility that the guide might become less effective over time as AI models evolve and become better at mimicking human writing.
• Validation of the Guide: The scientific validity of the guide would be greatly enhanced by empirical validation. The authors should test the effectiveness of the guide by having a separate group of individuals (either experts or non-experts) use it to detect AI-generated text and then comparing their performance to a control group that does not use the guide. This would provide evidence for the guide's utility and help to identify areas where it could be improved.

• Caption Clarity: The caption clearly states that the table presents a "truncated version" of the AI Text Detection Guide and provides a link to the full version online. This is helpful for readers who want to access the complete guide. However, the caption could be improved by briefly explaining the purpose of the guide and how it was developed.
• Table Content and Organization: The provided image of the table is well-organized and easy to read. It presents a truncated version of the guide, with clear headings for each section (e.g., "## Vocabulary / Word Choice Patterns," "## Grammar"). The use of bullet points and concise explanations makes the information accessible. However, without seeing the full guide, it's difficult to assess the comprehensiveness of the truncated version. The table should ideally provide a representative sample of the different types of clues and explanations included in the full guide.
• Reference Text Support: The reference text provides valuable context about the development of the guide, including the involvement of paid experts and the organization of clues into different sections. This helps readers understand the origin and structure of the guide.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of how the guide is intended to be used and its potential benefits for detecting AI-generated text. Additionally, the table itself could include a brief introductory paragraph providing an overview of the guide's purpose and structure. While the table is generally clear, some of the explanations could be made more concise and accessible to a wider audience. Providing more concrete examples within the table could also enhance understanding. Finally, the specific criteria used for truncating the guide should be clarified (e.g., were certain sections prioritized over others?).

Table 12: All 'AI' Vocabulary our expert annotators noted, as listed in the Detector Guide. See the full detection guide prompt in Table 11.

• Purpose of the Table: This table lists specific words that expert annotators have identified as being frequently used by AI when generating text. These words are considered "AI vocabulary" because they are perceived as being more characteristic of AI writing than human writing. The table serves as a reference for words that might signal a text was generated by an AI, like a red flag or a tell-tale sign. It is essentially a list of vocabulary to be on the lookout for when trying to detect AI-generated text.
• Content of the Table: The table contains a comprehensive list of words that are considered typical of AI writing, as identified by the expert annotators. These words are grouped by parts of speech: nouns, verbs, adjectives, and adverbs. There is also a category for "phrases". For example, under "nouns" you might find words like "tapestry" or "realm," while under "verbs" you might find words like "delve" or "embark." The phrases section might include expressions like "it's crucial to" or "in a world where." These are all words and phrases that, according to the experts, tend to appear more often in AI-generated text than in human-written text.
• Connection to the Detector Guide: The table is directly related to the AI Text Detection Guide mentioned in previous tables (specifically Table 11). The reference text clarifies that this table details the "AI vocab" found in that guide. This means that the words listed in Table 12 are part of the clues that the experts provided for identifying AI-generated text. The guide likely explains why these specific words are considered indicative of AI writing, while this table provides the complete list.
• Use of Quotation Marks: The reference text uses quotation marks around "AI vocab," which suggests that this is a term of art used by the researchers or the experts. It implies that this vocabulary is not necessarily a scientifically defined category but rather a collection of words identified through expert observation as being characteristic of AI writing. The quotation marks also acknowledge that these words might not *always* indicate AI-generated text, but rather that they are more likely to appear in such text.

• Subjectivity of "AI Vocab": The identification of "AI vocab" is inherently subjective and based on the expert annotators' perceptions and experiences. While these experts likely have a good understanding of the nuances of AI-generated text, their judgments might be influenced by individual biases or the specific types of AI models they have encountered. The scientific validity of this list depends on the extent to which these experts' judgments are representative of a broader consensus and whether the identified words are truly diagnostic of AI authorship across different models and contexts.
• Empirical Validation: The authors should provide empirical evidence to support the claim that the listed words are indeed more frequent in AI-generated text compared to human-written text. This could involve comparing the frequency of these words in a large corpus of AI-generated text and a comparable corpus of human-written text. Statistical tests should be performed to determine whether the observed differences are statistically significant.
• Context Dependence: The diagnostic value of specific words might vary depending on the context in which they are used. Some words on the list might be perfectly appropriate in certain types of human writing, while others might be more consistently indicative of AI authorship. The authors should discuss the potential for context dependence and acknowledge that the presence of these words alone is not definitive proof of AI generation.
• Dynamic Nature of AI Writing: AI language models are constantly evolving, and their writing styles are likely to change over time. The "AI vocab" identified in this study might be specific to the models and training data available at the time of the research. The authors should acknowledge the dynamic nature of AI writing and the possibility that the list might become less accurate as AI models become more sophisticated.
• Potential for Misuse: The publication of a list of "AI vocab" could potentially be misused by individuals seeking to evade AI detection. By avoiding the words on the list, they might be able to make their AI-generated text appear more human-like. The authors should discuss the potential for such misuse and consider the ethical implications of making this information publicly available. They might also consider including a disclaimer that the presence or absence of these words alone is not a definitive indicator of AI authorship.

• Caption Clarity: The caption clearly states that the table lists "'AI' Vocabulary" noted by expert annotators and refers to the Detector Guide in Table 11. The use of quotation marks around "AI" is consistent with the reference text and indicates that this is a specific term used in the study. However, the caption could be improved by briefly explaining *why* this vocabulary is considered indicative of AI-generated text.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. It clearly groups the words by part of speech (nouns, verbs, adjectives, adverbs, phrases), which makes sense given that different parts of speech often play different roles in a sentence. The words themselves are presented in a straightforward list format. The table is easy to read and navigate.
• Reference Text Support: The reference text provides helpful context by explicitly linking Table 12 to the "AI vocab" mentioned in the detection guide (Table 11). This clarifies the relationship between the two tables and reinforces the idea that this list of words is an integral part of the guide.
• Potential Improvements: The communication effectiveness could be improved by adding a brief introductory paragraph to the table explaining the rationale behind identifying "AI vocab" and how this list was compiled. Additionally, while the table is generally clear, providing examples of how these words are used differently in AI-generated vs. human-written text could further enhance understanding. The table could also benefit from a more descriptive title that reflects its purpose, such as "Vocabulary Frequently Used by AI Models, as Identified by Expert Annotators." Finally, it would be helpful to explicitly state whether the listed words are exhaustive or just a representative sample of the "AI vocab" identified by the experts.

Table 13: Prompt used for evader. The first insert into the prompt is filled by Table 11. The second insert is filled by examples of human and machine-generated articles.

• Purpose of the Table: This table shows the specific instructions, or "prompt," given to an AI model to make it try to evade detection - that is, to generate text that is difficult for humans to distinguish from human-written text. The prompt is designed to challenge the AI to produce text that appears as though it was written by a human, essentially acting as an "evader" trying to avoid being caught. Think of it like a set of instructions for a game where the AI is trying to blend in with humans by mimicking their writing style as closely as possible.
• Content of the Table: The table displays the "evader prompt template," which includes placeholders for specific information. The caption indicates that the first "insert" is filled with the content of Table 11, which is the AI Text Detection Guide. This means that the AI is given the very guidelines that humans are using to try to detect AI-generated text. The second insert is filled with examples of both human-written and machine-generated articles, providing the AI with examples of different writing styles. The prompt then instructs the AI to write an article that follows the guidelines, includes the provided examples, and incorporates information from the detection guide, all while making it appear as if it was written by a human. The prompt also specifies that the article should be written for a particular publication and that no reader, even those with access to the detection guide, should be able to detect that it was written by AI.
• Use of Inserts: The use of "inserts" from other tables (Table 11 and examples of articles) is a key aspect of this prompt. By providing the AI with the detection guide and examples of different writing styles, the researchers are essentially giving it the tools to learn how to evade detection. This is like giving a would-be imposter a detailed manual on how to act like the person they're trying to imitate, along with examples of that person's behavior. The AI can then use this information to try to generate text that avoids the characteristics that typically give away AI-generated content.
• Goal of the Evader Prompt: The ultimate goal of the evader prompt is to generate AI text that is indistinguishable from human-written text, even to experts who are actively trying to detect it. This represents a challenging test for the AI, as it has to not only generate coherent and plausible text but also avoid the specific pitfalls that are known to betray AI authorship. The success of the evader prompt in generating such text would have significant implications for the field of AI detection, suggesting that current methods might not be sufficient to reliably identify AI-generated content.

• Challenging the AI Model: Using an "evader" prompt to challenge the AI model is a scientifically valid approach for testing the robustness of AI detection methods. By attempting to generate text that can evade detection, the researchers are essentially stress-testing their detection methods and identifying potential weaknesses. This approach is analogous to testing the security of a computer system by trying to hack into it.
• Dependence on Detection Guide and Examples: The effectiveness of the evader prompt depends heavily on the quality and comprehensiveness of the AI Text Detection Guide (Table 11) and the examples of human and machine-generated articles provided. If the guide is incomplete or inaccurate, or if the examples are not representative, the AI might not learn to evade detection effectively. The authors should carefully consider the limitations of the guide and the examples and discuss their potential impact on the results.
• Iterative Process: The reference text suggests that the evader prompt is the result of an iterative process, where the researchers refined the prompt based on previous results. This is a good practice, as it allows for continuous improvement and adaptation. The authors should provide more details about this iterative process, including how many iterations were performed, what criteria were used to evaluate the effectiveness of each iteration, and how the prompt was modified based on the results.
• Generalizability: The ability of the AI to evade detection using this specific prompt might be limited to the specific model and training data used in the study. Different AI models might respond differently to the prompt, and the effectiveness of the evasion might also depend on the specific characteristics of the human-written and AI-generated examples provided. The authors should discuss the potential limitations of the generalizability of their findings to other models and contexts.

• Caption Clarity: The caption clearly states that the table presents the "Prompt used for evader" and explains how the two inserts are filled (with the content of Table 11 and examples of articles). This provides a good overview of the table's purpose and content. However, the caption could be improved by briefly explaining *why* an evader prompt is used in the study (i.e., to test the robustness of AI detection methods).
• Table Organization and Content: The provided image of the table is well-organized and relatively easy to understand. The prompt is clearly presented, and the placeholders for specific information (e.g., publication name) are easily identifiable. The instructions within the prompt are detailed and provide specific guidance to the AI model. However, the table could benefit from a clearer indication of where the inserts from Table 11 and the article examples should be placed within the prompt.
• Reference Text Support: The reference text provides helpful context by explaining that the evader prompt is the result of an iterative process and that it includes instructions, examples, the detection guide, and the initial article prompt. This helps readers understand the complexity of the prompt and its development.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the overall goal of using an evader prompt in the study. Additionally, while the table is generally clear, providing a concrete example of a filled-in prompt (with the inserts in place) could further enhance understanding. The table could also benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Generating AI Text to Evade Detection." Finally, explicitly stating that the prompt is designed to generate *new* articles, as opposed to paraphrasing existing ones (as in Tables 9 and 10), would further clarify the purpose of this specific prompt.

Table 14: The one article the majority of annotators did not detect correctly, generated from 01-PRO as part of Experiment 4 (see §2.4). "New Telescope Could Potentially Identify Planet X" was originally written by Emilie Le Beau Lucchesi in Discover Magazine on Nov. 6th, 2024

• Purpose of the Table: This table presents a specific article that most of the expert annotators in the study failed to correctly identify as AI-generated. It's like a "rogue" example that managed to fool the majority of the experts. This table highlights an instance where the AI was particularly successful at mimicking human writing, making it difficult for the human detectors to spot it as an AI creation.
• Content of the Table: The table displays the full text of the article that the majority of annotators misclassified. The caption provides additional context, indicating that the article was generated by the 01-PRO AI model as part of Experiment 4. It also mentions the title of the original human-written article ("New Telescope Could Potentially Identify Planet X"), the author (Emilie Le Beau Lucchesi), the publication (Discover Magazine), and the publication date (Nov. 6th, 2024). This information suggests that the AI-generated article was likely based on or inspired by a real article, although the caption states the real one was from the future.
• Significance of the Misclassified Article: The fact that a majority of annotators failed to correctly identify this particular article as AI-generated is significant because it demonstrates that the 01-PRO model, especially when used to generate text based on an existing article, is capable of producing text that can pass as human-written even to experts. This highlights the increasing sophistication of AI language models and the challenges involved in detecting AI-generated content. It suggests there may be limitations to human ability to discern AI text from human text.
• Connection to Experiment 4: The caption specifies that this article was generated as part of Experiment 4, which likely involved testing the annotators' ability to detect AI-generated text under specific conditions or using a particular type of AI model (01-PRO). The reference to section §2.4 suggests that more details about Experiment 4 can be found in that section of the paper. By examining the specifics of Experiment 4, one might gain insights into why this particular article was more difficult to detect.

• Focus on a Single Case: Presenting a single article that was misclassified by the majority of annotators is a valid approach for illustrating a specific challenge or limitation of human detection. However, it's important to recognize that this is just one example, and it may not be representative of all articles that were difficult to detect. The authors should discuss the characteristics of this particular article and why it might have been particularly challenging for annotators.
• Comparison to Original Article: The caption mentions that the AI-generated article was based on an original human-written article. Comparing the AI-generated text to the original article could provide valuable insights into the specific changes made by the AI and why these changes might have made the text more difficult to detect. The authors should consider including a detailed comparison of the two articles, highlighting the key differences and similarities.
• Analysis of Annotator Explanations: The authors should analyze the explanations provided by the annotators for their judgments on this particular article. This could reveal specific clues that misled the annotators or aspects of the AI-generated text that were particularly convincing. Examining the explanations of both the annotators who correctly identified the article as AI-generated and those who did not could provide further insights into the factors that contribute to successful or unsuccessful detection.
• Generalizability: While this table highlights a specific instance where AI-generated text evaded detection, it's important to consider the broader implications for the generalizability of the study's findings. The authors should discuss whether this case is an outlier or if it represents a more general trend. They should also consider whether the characteristics of this particular article (e.g., topic, style, length) might have contributed to its difficulty and whether similar results would be expected for other types of articles.

• Caption Clarity: The caption is generally clear and provides important context about the article, including its origin (01-PRO, Experiment 4), the title of the original article, the author, publication, and date. However, it could be improved by explicitly stating that the table presents the *full text* of the misclassified article. Additionally, the caption could briefly explain *why* this particular article is significant (i.e., because it fooled the majority of annotators).
• Table Content and Presentation: The provided image of the table presents the full text of the misclassified article in a clear and readable format. The text is well-formatted and easy to follow. The table effectively communicates the content of the article, allowing readers to examine the specific text that was difficult for annotators to detect. The inclusion of the original article's title, author, and publication information in the caption provides helpful context.
• Reference to Experiment 4: The caption's reference to Experiment 4 and section §2.4 helps to connect the table to the broader study design and methodology. This allows readers to easily locate more information about the specific experimental conditions under which this article was generated and evaluated.
• Potential Improvements: The communication effectiveness could be improved by adding a brief introductory paragraph to the table that explains the significance of this particular article and why it is being presented in detail. Additionally, the table could include annotations or highlights to draw attention to specific passages or features of the text that might have contributed to its difficulty for annotators. A more descriptive title for the table, such as "Full Text of the AI-Generated Article Most Frequently Misclassified by Annotators," could also enhance clarity. The caption is also slightly contradictory, stating that the article was "originally written by" and then giving the name of a human author, but also saying it was "generated from 01-PRO". This should be clarified to state that it was *based on* or *paraphrased from* an article written by that author.

Table 4: Taxonomy of clues used by experts to explain their detection decisions. For each category, we report the frequency of explanations that mention that category (regardless of if the annotator was correct) and provide examples of explanations for both human-written and AI-generated articles. While vocabulary and sentence structure form the most frequent clues, more complex phenomena like originality, clarity, formality, and factuality are also distinguishing features.

• Purpose of the Table: This table classifies the different types of clues that expert annotators used when deciding whether a piece of text was written by a human or an AI. Think of it like this: the experts are detectives, and they're looking for clues in the text. This table is a list of all the different types of clues they found, organized into categories. It's like a detective's handbook that lists different types of evidence, like fingerprints, DNA, or witness testimonies. Here, instead of those, the 'evidence' types are things like the specific words used (vocabulary), how sentences are put together (sentence structure), and so on.
• Content of the Table: For each category of clue, the table shows two main things: how often that type of clue was mentioned by the experts in their explanations, and examples of what the experts said about that clue. The frequency is like a measure of how important that clue is—a clue type that's mentioned a lot is probably more useful for detecting AI-generated text. The examples are actual quotes from the experts, showing how they described the clues they found. Importantly, the table includes examples from explanations for both human-written and AI-generated articles. This is like showing examples of fingerprints found at a crime scene and fingerprints found in a suspect's home, to illustrate the difference.
• Categories of Clues: The caption mentions some specific categories of clues: "vocabulary," "sentence structure," "originality," "clarity," "formality," and "factuality." "Vocabulary" refers to the specific words used in the text. "Sentence structure" refers to how sentences are constructed, their length, and complexity. "Originality" likely refers to whether the ideas and their expression seem novel or creative. "Clarity" probably relates to how easy the text is to understand. "Formality" describes the tone of the writing, whether it's casual or more formal. "Factuality" likely refers to whether the information presented is accurate and supported by evidence. The caption suggests that while vocabulary and sentence structure are the most common clues, these more complex aspects also play an important role.
• Schema Development: The reference text mentions that the table is based on a "schema" developed by the authors. A schema, in this context, is like a classification system or a set of rules for organizing information. The authors created this schema by carefully reading the experts' explanations and identifying the different types of clues they mentioned. They then grouped these clues into categories to create the taxonomy presented in the table. The fact that they used GPT-40, an AI language model, to help with this process suggests they might have used it to analyze the explanations and suggest categories or group similar explanations together, which they then refined manually.

• Subjectivity of Taxonomy: Developing a taxonomy of clues involves subjective judgment in defining and categorizing the different types of clues. The authors acknowledge that they developed the schema after "careful manual analysis," which suggests a qualitative approach. The scientific validity of the taxonomy would be strengthened by providing a more detailed description of the development process, including the specific criteria used for categorizing clues and potentially having multiple researchers independently develop taxonomies and then compare them to assess inter-rater reliability.
• Use of GPT-40 for Coding: Using GPT-40 to code the explanations into the schema is an interesting approach that leverages the capabilities of large language models for text analysis. However, it's important to recognize that GPT-40 is not a perfect tool and may introduce its own biases or errors into the coding process. The authors should describe how they used GPT-40 (e.g., specific prompts used, any fine-tuning or training involved) and how they validated its output. For example, did they manually check a subset of the GPT-40 coded explanations to ensure accuracy?
• Frequency as a Measure of Importance: Reporting the frequency of explanations that mention each category is a reasonable way to quantify the relative importance of different clues. However, it's important to note that frequency alone does not necessarily equate to importance or diagnostic value. Some clues might be mentioned less frequently but be highly reliable indicators of AI or human authorship. The authors should consider weighting the clues based on their accuracy or using other metrics to assess their diagnostic value.
• Generalizability of Taxonomy: The taxonomy is developed based on the explanations provided by the expert annotators in this specific study. The generalizability of the taxonomy to other contexts or types of AI-generated text would need to be further investigated. It's possible that different AI models or different types of writing tasks might lead to the use of different clues by annotators.

• Caption Clarity: The caption is relatively clear and provides a good overview of the table's purpose and content. It explains the concept of a taxonomy, the type of information reported for each category, and highlights the main findings regarding the frequency of different clue types. The use of the term "taxonomy" might be unfamiliar to some readers, but the caption provides a sufficient explanation in the context of the study.
• Table Organization: Assuming a standard tabular format, with clear labels for rows (categories of clues) and columns (frequency, examples for human-written, examples for AI-generated), the table should be relatively easy to understand. The organization by category facilitates comparisons and allows readers to quickly grasp the different types of clues used by experts.
• Examples: Providing examples of explanations for both human-written and AI-generated articles is crucial for illustrating the differences between the two and helping readers understand how the clues are manifested in actual text. The effectiveness of the communication depends on the quality and representativeness of the chosen examples. The caption suggests that the examples are illustrative, but without seeing the table, it's difficult to fully assess their clarity and effectiveness.
• Potential Improvements: The communication effectiveness could be improved by providing a brief rationale for why a fine-grained analysis of expert explanations is important in the context of the study. Additionally, the table itself could include a brief definition of each category of clue to further enhance clarity. Visual aids, such as highlighting or color-coding the examples to emphasize specific clues within the text, could also improve understanding.

Figure 3: (Top) A heatmap displaying the frequency with which annotators mentioned specific categories in their explanations when they were correct. Interestingly, vocabulary becomes a less frequent clue for 01-PRO-generated articles, especially with humanization. (Bottom) Same as above, except only computed over explanations when experts were incorrect. Formality is a big source of misdirection for 01-PRO articles, while fixating on sentence structure can lead experts to false positives. Details of each category can be found in Table 4.

• Overall Description: This figure uses heatmaps to show how often expert annotators used different types of clues when trying to identify whether a text was written by a human or an AI. A heatmap is a visual way of representing data where different colors correspond to different values, like a weather map where different colors represent different temperatures. In this case, the colors represent how frequently each type of clue was mentioned in the experts' explanations. The figure has two parts: a top part and a bottom part.
• Top Heatmap: The top heatmap shows the frequency of clue usage when the experts were *correct* in their judgments. Imagine the experts are detectives solving cases. This heatmap shows, for the cases they solved correctly, what types of evidence (clues) they used most often. The caption mentions an interesting finding: when the AI-generated text was produced by the 01-PRO model, especially when it was "humanized" (modified to look more human-like), the experts relied less on vocabulary as a clue. This suggests that 01-PRO is good at mimicking human-like vocabulary, making it a less reliable indicator of AI authorship.
• Bottom Heatmap: The bottom heatmap is similar, but it shows the frequency of clue usage when the experts were *incorrect* in their judgments. This is like showing what evidence the detectives focused on in cases they didn't solve correctly. The caption points out that when experts were misled by 01-PRO articles, they often focused on "formality" as a clue. This suggests that the way 01-PRO uses formal language can sometimes trick even experts into thinking it's human-written. It also mentions that focusing too much on sentence structure can lead to "false positives," which means incorrectly identifying a human-written text as AI-generated.
• Categories of Clues: The heatmaps likely show the same categories of clues as described in Table 4 (e.g., vocabulary, sentence structure, originality, clarity, etc.). Each category is probably represented by a row or column in the heatmap, and the color intensity in each cell indicates the frequency with which that clue was mentioned. The caption refers the reader to Table 4 for more details on each category.

• Comparison of Correct and Incorrect Judgments: Comparing the clue usage patterns between correct and incorrect judgments is a scientifically sound approach. It allows for the identification of clues that are reliably associated with accurate detection (top heatmap) and clues that may be misleading or less effective (bottom heatmap). This comparison can provide valuable insights into the strengths and weaknesses of expert human detection strategies.
• Focus on 01-PRO: Highlighting the differences observed with 01-PRO-generated articles is important, as it suggests that this particular AI model poses a greater challenge for human detection, especially when humanization techniques are applied. This finding has implications for the development of more robust AI detection methods and underscores the need to understand the specific characteristics of different AI models.
• Dependence on Taxonomy: The validity of the analysis presented in Figure 3 depends heavily on the quality and reliability of the taxonomy of clues defined in Table 4. If the categories in the taxonomy are not well-defined, are overlapping, or do not accurately capture the nuances of expert decision-making, then the conclusions drawn from the heatmaps may be questionable. The authors should ensure that the taxonomy is robust and validated.
• Quantitative Nature of Heatmaps: Heatmaps provide a visual representation of quantitative data (frequency of clue usage). However, the specific numerical values underlying the color scale are not mentioned in the caption. To ensure scientific rigor, the authors should clearly indicate the frequency ranges corresponding to each color in the heatmap. Additionally, they should report the total number of explanations analyzed for each heatmap (correct and incorrect) to provide context for the frequency data.

• Caption Clarity: The caption is relatively clear and provides a good overview of the figure's content and purpose. It explains the distinction between the top and bottom heatmaps, highlights key findings related to 01-PRO and humanization, and points out the connection to Table 4. However, the caption could be improved by briefly defining what a heatmap is for readers who may be unfamiliar with this type of visualization.
• Visual Effectiveness of Heatmaps: Heatmaps are generally an effective way to visualize frequency data and identify patterns. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of different clue categories. However, the specific color scheme used (darkest teal, orange, purple) should be carefully chosen to ensure that the differences in color intensity are easily perceptible and that the colors are distinguishable for individuals with color vision deficiencies.
• Explanation of Key Findings: The caption effectively highlights the key findings related to vocabulary usage with 01-PRO and the role of formality and sentence structure in misdirection. These findings are clearly stated and provide valuable insights into the challenges of detecting AI-generated text, particularly from more advanced models like 01-PRO.
• Potential Improvements: The communication effectiveness could be improved by including a more explicit statement about the overall goal of the figure, which is to understand what clues are most helpful (and potentially misleading) for experts when detecting AI-generated text. Additionally, the figure itself could benefit from clearer axis labels and a color scale legend that explicitly indicates the frequency ranges corresponding to each color. Providing the total number of explanations analyzed for each heatmap would also enhance clarity.

• Purpose of the Figure: This figure shows how often Annotator 1, one of the expert annotators in the study, mentioned different categories of clues in their explanations. When the annotators were deciding whether a text was written by a human or an AI, they also wrote explanations for their decisions, describing the clues they used. This figure breaks down those explanations by category, showing which types of clues this particular annotator focused on. It is like a breakdown of how one detective approaches solving cases, showing which types of evidence they tend to focus on.
• Content of the Figure: The figure likely presents a visual representation of the frequency with which Annotator 1 mentioned different categories of clues, such as "vocabulary," "grammar," "tone," etc. These categories are based on the taxonomy developed by the researchers (as described in Table 4). The figure might be a bar graph, where each bar represents a category, and the height of the bar indicates the frequency with which that category was mentioned. Alternatively, it could be a heatmap or another type of visualization that shows the relative frequency of each category. The specific categories and their frequencies are shown in the provided image of the figure.
• Focus on Individual Annotator: This figure focuses specifically on Annotator 1's explanations, providing a detailed look at their individual approach to the detection task. The reference text indicates that there are separate figures for each of the five expert annotators (Figures 10-13). By analyzing each annotator's preferred clues, the researchers can gain insights into the different strategies used by experts and potentially identify the most effective clues for detecting AI-generated text.
• Connection to Other Figures: The reference text explains that Figures 9-13 each show the category mention frequencies for a different annotator. This suggests that the figures are part of a series that allows for a comparison of the individual annotators' approaches. By examining all five figures, one can get a sense of the variability among experts and potentially identify any common patterns or unique strategies.

• Individual Differences in Detection Strategies: Analyzing the frequency of category mentions for each annotator separately is a scientifically sound approach for investigating individual differences in detection strategies. This allows the researchers to explore the variability among experts and potentially identify different approaches that are effective for detecting AI-generated text. The validity of this approach depends on the assumption that the categories in the taxonomy are meaningful and capture the relevant aspects of the annotators' decision-making processes.
• Dependence on Taxonomy: The analysis presented in Figure 9 is based on the taxonomy of clues developed by the researchers (Table 4). The validity of the findings depends on the quality and comprehensiveness of this taxonomy. If the categories are not well-defined or do not accurately capture the nuances of the annotators' explanations, then the frequency counts might not be meaningful. The authors should ensure that the taxonomy is robust and validated.
• Quantitative Analysis of Qualitative Data: The figure presents a quantitative analysis (frequency counts) of qualitative data (annotator explanations). This approach can provide valuable insights into the patterns and trends in the annotators' decision-making processes. However, it's important to recognize that reducing complex qualitative explanations to simple frequency counts might result in some loss of information or oversimplification. The authors should be cautious in their interpretation of the frequency data and consider the potential limitations of this approach.
• Comparison Across Annotators: The reference text suggests that Figures 9-13 allow for a comparison of the different annotators' approaches. This comparison is valuable for understanding the range of strategies used by experts and identifying any common patterns or unique approaches. The validity of this comparison depends on the consistency of the annotation and coding process across annotators. The authors should ensure that all annotators received the same instructions and that the coding of their explanations into categories was performed reliably.

• Caption Clarity: The caption clearly states the purpose of the figure: to show the "Frequency of Categories Mentioned in Explanations" for Annotator 1. It is concise and to the point. However, it could be improved by briefly explaining *why* this frequency analysis is important in the context of the study.
• Visual Presentation: The provided image of the figure is a heatmap, which is an effective way to visualize frequency data across different categories and text sources. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of each category. The figure is well-organized and easy to read, with clear labels for the categories and text sources. The specific color scheme used is visually appealing and helps to distinguish between different frequency levels.
• Reference Text Support: The reference text provides important context by explaining that Figures 9-13 each show the category mention frequencies for a different annotator. This helps readers understand that the figures are part of a series and allows for comparisons between experts.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of how the frequency data was calculated and what the different colors in the heatmap represent. Additionally, the figure itself could benefit from a more descriptive title that reflects its purpose, such as "Frequency of Clue Categories Mentioned in Annotator 1's Explanations." The figure could also include a color scale legend to further clarify the meaning of the different colors. While the figure is generally clear, adding a brief summary of the key findings or patterns observed in Annotator 1's explanations could enhance its impact. For example, the caption or a separate text box could highlight the categories most frequently mentioned by this annotator or any notable differences in their approach compared to other annotators. The x-axis label could be made more concise, such as "Text Source" instead of "Test Source". Finally, since the table refers to frequencies, it would be helpful to clarify that these are percentages in the caption or axis labels.

• Purpose of the Figure: Similar to Figure 9, this figure shows how often a specific expert annotator, Annotator 2 in this case, mentioned different categories of clues in their explanations when deciding whether a text was written by a human or an AI. It's like looking at the specific working style of a detective, and seeing what kind of evidence they tend to focus on when solving a case. Each annotator has their own preferred set of clues, and this figure breaks down the preferences of Annotator 2.
• Content of the Figure: The figure likely uses a heatmap, just like Figure 9, to visually represent the frequency with which Annotator 2 mentioned different categories of clues. These categories are the same as those defined in Table 4 and used in Figure 9 (e.g., "vocabulary," "grammar," "tone"). The color intensity in each cell of the heatmap corresponds to the frequency with which that category was mentioned. For example, a darker color might indicate that Annotator 2 frequently mentioned vocabulary as a clue, while a lighter color might indicate that they rarely mentioned originality. The provided image shows a heatmap with the categories on the y-axis, different text sources on the x-axis, and the color of each cell representing the frequency of that category being mentioned for that text source.
• Focus on Individual Annotator: This figure focuses specifically on Annotator 2's explanations, providing a detailed look at their individual approach to the detection task. The reference text indicates that this is one of a series of figures (Figures 9-13), each presenting the category mention frequencies for a different annotator. By analyzing each annotator's preferred clues, the researchers can gain insights into the different strategies used by experts.
• Comparison with Other Annotators: By comparing Figure 10 with Figures 9 and 11-13, one can see how Annotator 2's approach differs from or is similar to the other experts. This comparison can help to identify common patterns among experts, as well as unique strategies used by individual annotators. For example, it might reveal that some annotators rely heavily on vocabulary clues, while others focus more on sentence structure or tone.

• Individual Differences Analysis: Analyzing individual differences in expert detection strategies is a valuable contribution to the study. By examining the frequency with which each annotator mentions different clue categories, the researchers can gain insights into the cognitive processes involved in detecting AI-generated text. The validity of this analysis depends on the assumption that the annotators' explanations accurately reflect their decision-making processes.
• Reliability of Coding: The analysis relies on coding the annotators' explanations into predefined categories. The scientific validity of the findings depends on the reliability of this coding process. The authors should describe the procedures used to code the explanations and report inter-coder reliability statistics to demonstrate that the coding was performed consistently and accurately. This would help ensure that the frequency counts are not simply an artifact of subjective coding decisions.
• Consistency of Annotator Behavior: The reference text suggests that each expert had clues they favored "throughout all experiments." This implies a degree of consistency in each annotator's behavior across different tasks and datasets. The authors should provide evidence to support this claim, perhaps by analyzing the stability of category mention frequencies across different experiments or subsets of the data. This would strengthen the argument that the observed patterns reflect genuine individual differences in detection strategies.
• Connection to Performance: While the figure shows the frequency of category mentions, it does not directly link these frequencies to the annotator's performance (e.g., accuracy, TPR, FPR). The authors should investigate the relationship between the preferred clues and the annotator's success in detecting AI-generated text. For example, do annotators who frequently mention certain categories perform better than those who do not? This would help to identify the most effective clues and strategies for detection.

• Caption Clarity: The caption is clear and concise, stating that the figure shows "Annotator 2 Frequency of Categories Mentioned in Explanations." This accurately describes the content of the figure. However, similar to the caption for Figure 9, it could be improved by briefly explaining *why* this frequency analysis is important in the context of the study.
• Visual Presentation: The provided image of the figure is a heatmap, which is an effective way to visualize frequency data across different categories and text sources. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of each category. The figure is well-organized and easy to read, with clear labels for the categories and text sources. The specific color scheme used is visually appealing and helps to distinguish between different frequency levels.
• Reference Text Support: The reference text provides important context by explaining that Figures 9-13 each show the category mention frequencies for a different annotator. This helps readers understand that the figures are part of a series and allows for comparisons between experts. The reference text also reinforces the idea that each expert has their own favored clues.
• Potential Improvements: Similar to Figure 9, the communication effectiveness could be improved by adding a brief explanation in the caption of how the frequency data was calculated and what the different colors in the heatmap represent. Additionally, the figure itself could benefit from a more descriptive title that reflects its purpose, such as "Frequency of Clue Categories Mentioned in Annotator 2's Explanations." The figure could also include a color scale legend to further clarify the meaning of the different colors. While the figure is generally clear, adding a brief summary of the key findings or patterns observed in Annotator 2's explanations could enhance its impact. For example, the caption or a separate text box could highlight the categories most frequently mentioned by this annotator or any notable differences in their approach compared to other annotators. The x-axis label could be made more concise, such as "Text Source" instead of "Test Source". Finally, since the table refers to frequencies, it would be helpful to clarify that these are percentages in the caption or axis labels.

• Purpose of the Figure: This figure is very similar to Figures 9 and 10, but it focuses on Annotator 3 instead of Annotators 1 or 2. It shows how often Annotator 3 mentioned different categories of clues in their explanations when deciding whether a text was written by a human or an AI. Each annotator has their own preferred set of clues, and this figure reveals the specific preferences of Annotator 3. It's like analyzing the specific techniques used by a third detective to solve their cases, showing what kind of evidence they tend to rely on.
• Content of the Figure: Like Figures 9 and 10, this figure likely uses a heatmap to visually represent the frequency with which Annotator 3 mentioned different categories of clues. The categories are the same as those defined in Table 4 and used in the previous figures (e.g., "vocabulary," "grammar," "tone"). The color intensity in each cell of the heatmap corresponds to the frequency of that category being mentioned. The provided image shows a heatmap with the categories on the y-axis, different text sources on the x-axis, and the color of each cell representing the frequency.
• Focus on Individual Annotator: This figure focuses solely on Annotator 3's explanations, providing a detailed look at their individual approach to the detection task. The reference text indicates that this is one of a series of figures (Figures 9-13), each presenting the category mention frequencies for a different annotator. By analyzing each annotator's preferred clues, the researchers can gain insights into the different strategies used by experts.
• Comparison with Other Annotators: By comparing Figure 11 with Figures 9, 10, 12, and 13, one can see how Annotator 3's approach differs from or is similar to the other experts. This comparison can help to identify common patterns among experts, as well as unique strategies used by individual annotators. For example, it might reveal that Annotator 3 relies more heavily on formality clues compared to other annotators. This comparison helps us understand the variety of expert approaches to the task.

• Individual Differences Analysis: Continuing the analysis of individual differences in expert detection strategies is valuable. Examining the frequency with which each annotator mentions different clue categories allows researchers to explore the variability among experts and potentially identify different, but effective, approaches for detecting AI-generated text. The validity of this approach still depends on the assumption that the annotators' explanations accurately reflect their decision-making processes.
• Reliability of Coding: The analysis depends on the accurate coding of the annotators' explanations into predefined categories. The scientific validity of the findings depends on the reliability of this coding process. The authors should describe the procedures used to code the explanations and report inter-coder reliability statistics to demonstrate that the coding was performed consistently and accurately.
• Consistency of Annotator Behavior: The reference text suggests that each expert had clues they favored "throughout all experiments." This implies a degree of consistency in each annotator's behavior across different tasks and datasets. The authors should provide evidence to support this claim, perhaps by analyzing the stability of category mention frequencies across different experiments or subsets of the data.
• Connection to Performance: While the figure shows the frequency of category mentions, it does not directly link these frequencies to the annotator's performance (e.g., accuracy, TPR, FPR). The authors should investigate the relationship between the preferred clues and the annotator's success in detecting AI-generated text. This would help to identify the most effective clues and strategies for detection and further validate the usefulness of the identified categories.

• Caption Clarity: The caption is clear and concise, stating that the figure shows "Annotator 3 Frequency of Categories Mentioned in Explanations." This accurately describes the content of the figure. However, similar to the captions for Figures 9 and 10, it could be improved by briefly explaining *why* this frequency analysis is important in the context of the study.
• Visual Presentation: The provided image of the figure is a heatmap, which effectively visualizes frequency data across different categories and text sources. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of each category. The figure is well-organized and easy to read, with clear labels for the categories and text sources. The color scheme is visually appealing and helps to distinguish between different frequency levels.
• Reference Text Support: The reference text provides important context by explaining that Figures 9-13 each show the category mention frequencies for a different annotator. This helps readers understand that the figures are part of a series and allows for comparisons between experts.
• Potential Improvements: Similar to Figures 9 and 10, the communication effectiveness could be improved by adding a brief explanation in the caption of how the frequency data was calculated and what the different colors in the heatmap represent. Additionally, the figure itself could benefit from a more descriptive title that reflects its purpose, such as "Frequency of Clue Categories Mentioned in Annotator 3's Explanations." The figure could also include a color scale legend to further clarify the meaning of the different colors. While the figure is generally clear, adding a brief summary of the key findings or patterns observed in Annotator 3's explanations could enhance its impact. The x-axis label could be made more concise, such as "Text Source". Finally, since the table refers to frequencies, it would be helpful to clarify that these are percentages in the caption or axis labels.

• Purpose of the Figure: This figure continues the analysis of individual expert annotators' approaches to detecting AI-generated text, now focusing on Annotator 4. Like the preceding figures (9, 10, and 11), it shows how frequently this specific annotator mentioned different categories of clues in their explanations. It's like examining the investigative methods of a fourth detective, revealing their preferred types of evidence when determining if a text is human-written or AI-generated.
• Content of the Figure: The figure likely employs a heatmap similar to the ones used in Figures 9, 10, and 11. The heatmap visually represents the frequency with which Annotator 4 mentioned different categories of clues (e.g., "vocabulary," "grammar," "tone," etc.), with color intensity corresponding to frequency. The provided image confirms this, showing a heatmap with categories on the y-axis, different text sources on the x-axis, and the color of each cell representing the frequency.
• Focus on Individual Annotator: This figure is dedicated to presenting Annotator 4's individual approach to the detection task. The reference text indicates that this is part of a series (Figures 9-13), each dedicated to a different annotator. By analyzing each expert's preferred clues, the researchers can gain a deeper understanding of the diverse strategies employed by experienced individuals in this task.
• Comparison with Other Annotators: By comparing Figure 12 with Figures 9, 10, 11, and 13, one can analyze how Annotator 4's approach differs from or resembles that of the other experts. This comparison helps identify both common patterns among experts and unique individual strategies. For example, it might show that Annotator 4 relies more heavily on "originality" clues compared to the others, who might focus more on "vocabulary" or "sentence structure."

• Individual Differences Analysis: Continuing the analysis of individual differences in expert detection strategies remains a valuable aspect of the study. Examining the frequency of category mentions for each annotator provides insights into the cognitive processes involved in detecting AI-generated text. The validity of this approach rests on the assumption that the annotators' explanations accurately reflect their decision-making processes.
• Reliability of Coding: The analysis depends on the accurate coding of the annotators' explanations into predefined categories. The scientific validity of the findings hinges on the reliability of this coding process. The authors should describe the procedures used to code the explanations and report inter-coder reliability statistics to demonstrate consistency and accuracy.
• Consistency of Annotator Behavior: The reference text suggests that each expert had clues they favored "throughout all experiments," implying consistency in each annotator's behavior. The authors should provide evidence to support this claim, perhaps by analyzing the stability of category mention frequencies across different experiments or subsets of the data.
• Connection to Performance: While the figure shows the frequency of category mentions, it does not directly link these frequencies to the annotator's performance (e.g., accuracy, TPR, FPR). The authors should investigate the relationship between the preferred clues and the annotator's success in detecting AI-generated text to identify the most effective clues and strategies.

• Caption Clarity: The caption is clear and concise, stating that the figure shows "Annotator 4 Frequency of Categories Mentioned in Explanations." This accurately describes the content of the figure. However, similar to previous figures, it could be improved by briefly explaining *why* this frequency analysis is important in the context of the study.
• Visual Presentation: The provided image of the figure is a heatmap, effectively visualizing frequency data across different categories and text sources. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of each category. The figure is well-organized and easy to read, with clear labels for the categories and text sources. The color scheme is visually appealing and helps to distinguish between different frequency levels.
• Reference Text Support: The reference text provides important context by explaining that Figures 9-13 each show the category mention frequencies for a different annotator. This helps readers understand that the figures are part of a series and allows for comparisons between experts.
• Potential Improvements: Similar to previous figures, the communication effectiveness could be improved by adding a brief explanation in the caption of how the frequency data was calculated and what the different colors in the heatmap represent. The figure itself could benefit from a more descriptive title, such as "Frequency of Clue Categories Mentioned in Annotator 4's Explanations." A color scale legend could further clarify the meaning of the colors. A brief summary of key findings or patterns in Annotator 4's explanations could enhance the figure's impact. The x-axis label could be made more concise, and it would be helpful to clarify that the frequencies are percentages.

• Purpose of the Figure: This figure concludes the series of individual annotator analyses, now focusing on Annotator 5. Like Figures 9-12, it shows how often this specific annotator mentioned different categories of clues in their explanations when deciding whether a text was written by a human or an AI. It's like examining the investigative methods of the fifth and final detective, revealing their preferred types of evidence when determining the nature of a text. Each annotator has a unique approach, and this figure breaks down the preferences of Annotator 5.
• Content of the Figure: The figure likely uses a heatmap, consistent with Figures 9-12, to visually represent the frequency with which Annotator 5 mentioned different categories of clues. The categories are the same as those defined in Table 4 and used in the previous figures (e.g., "vocabulary," "grammar," "tone"). The color intensity in each cell of the heatmap corresponds to the frequency, with darker colors likely indicating higher frequency. The provided image confirms this, showing a heatmap with categories on the y-axis, different text sources on the x-axis, and the cell color representing the frequency.
• Focus on Individual Annotator: This figure is dedicated to presenting Annotator 5's individual approach to the detection task. The reference text indicates that this is the last of a series (Figures 9-13), each dedicated to a different annotator. By analyzing each expert's preferred clues, the researchers can gain a deeper understanding of the diverse strategies employed by experienced individuals in this task.
• Comparison with Other Annotators: By comparing Figure 13 with Figures 9-12, one can analyze how Annotator 5's approach differs from or resembles that of the other experts. This comparison helps identify both common patterns among experts and unique individual strategies. For example, it might reveal that Annotator 5 relies more heavily on "quotes" compared to the others, who might focus more on "vocabulary" or "sentence structure." This allows for a comprehensive view of expert approaches.

• Individual Differences Analysis: Continuing the analysis of individual differences in expert detection strategies is valuable. Examining the frequency of category mentions for each annotator provides insights into the cognitive processes involved in detecting AI-generated text. The validity of this approach rests on the assumption that the annotators' explanations accurately reflect their decision-making processes.
• Reliability of Coding: The analysis depends on the accurate coding of the annotators' explanations into predefined categories. The scientific validity of the findings hinges on the reliability of this coding process. The authors should describe the procedures used to code the explanations and report inter-coder reliability statistics to demonstrate consistency and accuracy.
• Consistency of Annotator Behavior: The reference text suggests that each expert had clues they favored "throughout all experiments," implying consistency in each annotator's behavior. The authors should provide evidence to support this claim, perhaps by analyzing the stability of category mention frequencies across different experiments or subsets of the data.
• Connection to Performance: While the figure shows the frequency of category mentions, it does not directly link these frequencies to the annotator's performance (e.g., accuracy, TPR, FPR). The authors should investigate the relationship between the preferred clues and the annotator's success in detecting AI-generated text to identify the most effective clues and strategies.

• Caption Clarity: The caption is clear and concise, stating that the figure shows "Annotator 5 Frequency of Categories Mentioned in Explanations." This accurately describes the content of the figure. However, similar to previous figures, it could be improved by briefly explaining *why* this frequency analysis is important in the context of the study.
• Visual Presentation: The provided image of the figure is a heatmap, effectively visualizing frequency data across different categories and text sources. The use of color gradients allows for a quick and intuitive understanding of the relative frequency of each category. The figure is well-organized and easy to read, with clear labels for the categories and text sources. The color scheme is visually appealing and helps to distinguish between different frequency levels.
• Reference Text Support: The reference text provides important context by explaining that Figures 9-13 each show the category mention frequencies for a different annotator. This helps readers understand that the figures are part of a series and allows for comparisons between experts.
• Potential Improvements: Similar to previous figures, the communication effectiveness could be improved by adding a brief explanation in the caption of how the frequency data was calculated and what the different colors in the heatmap represent. The figure itself could benefit from a more descriptive title, such as "Frequency of Clue Categories Mentioned in Annotator 5's Explanations." A color scale legend could further clarify the meaning of the colors. A brief summary of key findings or patterns in Annotator 5's explanations could enhance the figure's impact. The x-axis label could be made more concise, and it would be helpful to clarify that the frequencies are percentages.

Table 17: Truncated prompt used for comment analysis. The first insert is filled by a list of definitions of the categories found in Table 4 and the second insert is filled by an annotator explanation. A full version of the guide can be found at https://github.com/jenna-russell/human_detectors.

• Purpose of the Table: This table presents a shortened, or "truncated," version of a prompt that was used for analyzing the comments made by expert annotators. The prompt is essentially a set of instructions given to an AI model, in this case, likely a large language model like GPT-40, to get it to perform a specific task. The task here is "comment analysis," which likely involves categorizing or extracting information from the explanations provided by the annotators about why they classified a text as human or AI-generated. The table shows the structure and wording of the prompt used for this analysis.
• Content of the Table: The table displays a template for a prompt, which includes placeholders or "inserts" that are filled with specific information. The caption indicates that the first insert is a list of definitions of the categories found in Table 4. These categories are the different types of clues that annotators used to detect AI-generated text (e.g., vocabulary, grammar, tone). The second insert is an actual explanation written by an annotator. The prompt likely instructs the AI to analyze the annotator's explanation based on the provided category definitions. The provided image of the table shows an example of such a prompt, with sections for defining the categories and providing an example explanation.
• Use of Inserts: The use of inserts from other tables (Table 4 and an annotator's explanation) is a way of providing the AI with the necessary context and information to perform the comment analysis. By giving the AI the category definitions and an example explanation, the researchers are essentially teaching it how to categorize the comments. This is like providing a set of guidelines and an example before asking someone to sort a collection of items into different categories.
• Connection to Other Tables and Resources: The caption refers to two other tables: Table 4, which contains the definitions of the categories, and Table 11, which is the AI Text Detection Guide. It also provides a link to a GitHub repository where the full version of the guide can be found. This indicates that the prompt used for comment analysis is part of a larger framework or methodology that involves multiple components. The table serves as a bridge between the expert-generated explanations, the predefined categories, and the automated analysis performed by the AI.

• Automated Comment Analysis: Using a large language model to analyze the annotators' comments is a scientifically valid approach for extracting structured information from unstructured text data. This method can help to automate the process of categorizing and analyzing the explanations, potentially saving time and effort compared to manual coding. However, the accuracy and reliability of the analysis depend on the quality of the prompt and the capabilities of the AI model being used.
• Dependence on Category Definitions: The effectiveness of the comment analysis depends heavily on the clarity and comprehensiveness of the category definitions provided in Table 4. If the categories are not well-defined or do not adequately capture the nuances of the annotators' explanations, the AI model might miscategorize or misinterpret the comments. The authors should ensure that the category definitions are robust and validated.
• Prompt Engineering: The specific wording and structure of the prompt are crucial for obtaining accurate and meaningful results from the AI model. The authors should describe the process they used to develop and refine the prompt, including any pilot testing or iterations. They should also discuss the potential limitations of the prompt and how they attempted to address them. Providing the full prompt in the table is a good practice for transparency and reproducibility.
• Validation of AI Analysis: The authors should describe how they validated the output of the AI-based comment analysis. This could involve comparing the AI's categorizations to those of human coders for a subset of the explanations and calculating inter-rater reliability. Alternatively, they could manually review a sample of the AI's output to assess its accuracy and identify any systematic errors or biases. This is crucial to ensure the results are scientifically valid.

• Caption Clarity: The caption clearly states that the table presents a "truncated prompt used for comment analysis" and explains how the two inserts are filled. It also provides a link to the full version of the guide. However, it could be improved by briefly explaining *why* this prompt is used and how it relates to the overall goals of the study.
• Table Organization and Content: The provided image of the table is well-organized and relatively easy to understand. It clearly separates the instructions for the prompt from the example explanation. The use of placeholders for the inserts is intuitive. However, the table could benefit from a more explicit statement about the purpose of the prompt and the specific task the AI is being asked to perform.
• Reference to Other Tables: The caption clearly links Table 17 to Tables 4 and 11, indicating that the prompt relies on information from these other tables. This helps readers understand the connection between different elements of the study and how they fit together.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the overall goal of the comment analysis and how it contributes to the study's findings. Additionally, while the table is generally clear, providing a concrete example of a filled-in prompt (with the inserts in place) could further enhance understanding. The table could also benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for AI-Based Analysis of Annotator Explanations." Finally, the specific criteria used for truncating the prompt should be clarified (e.g., were certain sections prioritized or omitted?). The instructions could also be made more concise in some areas.

• Purpose of the Table: This table shows the specific instructions, or "prompt template," used to create a "zero-shot detector." The zero-shot detector is an AI model that's being tested to see if it can identify whether a piece of text was written by a human or an AI without any prior training or examples. "Zero-shot" means that the model has not been specifically trained to perform this task. It's like asking someone to identify a new animal they've never seen before, based only on a general description. The prompt template provides the instructions that are given to the AI model to set it up for this task.
• Content of the Table: The table displays the "Prompt Template," which includes placeholders for the text that the AI is supposed to analyze. The prompt tells the AI that it's being given a "candidate text" and that its task is to determine whether the text was written by a human or generated by an AI. It instructs the AI to answer "HUMAN-WRITTEN" if it thinks the text was likely written by a human and "AI-GENERATED" if it thinks the text was likely generated by an AI. The prompt also specifies the format for the AI's answer, which should be "<answer> YOUR ANSWER </answer>." The provided image shows the exact wording of this prompt template.
• Zero-Shot Detection: The term "zero-shot" refers to a type of machine learning where a model is asked to perform a task that it has not been explicitly trained on. In this case, the AI model is being asked to detect AI-generated text without having been given any examples of human-written or AI-generated text beforehand. This is a challenging task because the model has to rely on its general knowledge of language and its understanding of the differences between human and AI writing styles to make its decision. It is like asking someone to identify a new language they have never heard before, based only on their general knowledge of how languages work.
• Baseline Performance: The reference text mentions that this prompt is used to find a "baseline performance" of detector models. This means that the researchers are using the zero-shot detector to establish a starting point or a benchmark for comparison with other, potentially more sophisticated, detection methods. By evaluating the performance of the zero-shot detector, they can determine how well an AI model can perform this task without any specific training and then compare it to the performance of models that have been trained or fine-tuned for this task. This allows them to measure the improvement gained from using more advanced techniques.

• Establishing a Baseline: Establishing a baseline performance using a zero-shot detector is a scientifically valid approach for evaluating the effectiveness of other detection methods. It provides a reference point for comparison and helps to determine whether more complex or resource-intensive methods are actually necessary to achieve satisfactory performance. By comparing the performance of the zero-shot detector to that of trained models or human experts, the researchers can quantify the added value of those approaches.
• Prompt Design: The specific wording and structure of the prompt are crucial for obtaining meaningful results. The prompt should be clear, unambiguous, and provide sufficient instructions for the AI model to understand the task. The provided prompt is relatively straightforward, but its effectiveness might depend on the specific capabilities of the AI model being used. The authors should discuss their rationale for choosing this particular prompt and consider potential variations or improvements.
• Dependence on Model Capabilities: The performance of the zero-shot detector will depend on the inherent capabilities of the underlying AI model. Different models might have different levels of understanding of language and different abilities to distinguish between human and AI writing styles. The authors should specify the AI model used for the zero-shot detector and discuss any known limitations of the model that might affect the results. They should also consider testing the prompt with multiple models to assess the generalizability of the findings.
• Zero-Shot Assumption: The validity of the zero-shot approach depends on the assumption that the AI model has not been inadvertently exposed to similar detection tasks during its pre-training or fine-tuning. If the model has encountered similar tasks before, it might not be a true zero-shot setting. The authors should discuss the potential for such inadvertent training and its implications for the interpretation of the results.

• Caption Clarity: The caption clearly states that the table presents the "Prompt Template for the zero-shot detector set up." This accurately describes the content of the table. However, it could be improved by briefly explaining what a "zero-shot detector" is and why establishing a baseline performance is important in the context of the study.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the instructions are concise and straightforward. The use of placeholders for the candidate text is intuitive. The table effectively communicates the prompt template in a clear and accessible manner.
• Reference Text Support: The reference text provides helpful context by explaining that the prompt is used to find a baseline performance of detector models. It also clarifies that the model should determine if the candidate text is human-written or AI-generated. This helps readers understand the purpose of the prompt and how it fits into the overall study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the concept of "zero-shot" learning for readers who might be unfamiliar with the term. Additionally, the table itself could benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Zero-Shot Detection of AI-Generated Text." While the prompt is generally clear, it could be further improved by explicitly instructing the AI to consider the style and content of the text when making its decision, rather than just providing the answer. Finally, including an example of a filled-in prompt with a candidate text could further enhance understanding.

• Purpose of the Table: This table presents the "prompt template" used to create a "zero-shot + CoT detector." This is a type of AI detector that tries to determine if a text was written by a human or an AI. The "zero-shot" part means that the AI has not been specifically trained on this task beforehand. "CoT" stands for "Chain of Thought," which is a technique that encourages the AI to explain its reasoning step-by-step, like a human would. The prompt template provides the instructions given to the AI model to set it up for this task, similar to providing instructions for a game.
• Content of the Table: The table shows the structure of the prompt, which includes placeholders for the text that the AI is supposed to analyze ("Candidate Text"). The prompt instructs the AI to determine if the text is "Human-written" or "AI-generated." Importantly, it also asks the AI to provide an "explanation" of its decision, describing the features of the text that suggest human or AI authorship. The prompt specifies the format for the AI's response, including a description section and an answer section. The provided image shows the exact wording of this prompt template, with separate sections for the question, description, and answer.
• Zero-Shot + CoT: The "zero-shot + CoT" approach combines two ideas. "Zero-shot" means the AI is attempting a task without prior training on that specific task. "Chain of Thought" (CoT) prompting encourages the AI to explain its reasoning process step-by-step. By combining these, the researchers are trying to get the AI to not only make a judgment about the text's authorship but also to articulate the reasoning behind its decision, similar to how a human expert might explain their thought process. This is like asking someone not just to guess the answer but also to explain how they arrived at that answer.
• Comparison to Table 18: This table builds upon the prompt template presented in Table 18, which was for a simpler zero-shot detector. The key difference is the addition of the "Chain of Thought" component, which requires the AI to provide an explanation for its decision. This allows the researchers to gain insights into the AI's reasoning process, not just its final answer. By comparing the performance of the zero-shot detector (Table 18) with the zero-shot + CoT detector (Table 19), the researchers can assess the benefits of using the CoT approach.

• Chain of Thought Rationale: Using a Chain of Thought (CoT) approach is a scientifically sound method for probing the reasoning abilities of large language models. By requiring the AI to articulate its decision-making process, the researchers can gain a deeper understanding of how the model arrives at its conclusions. This is important for assessing the validity of the model's judgments and identifying potential biases or limitations in its reasoning.
• Prompt Design: The specific wording and structure of the prompt are crucial for eliciting meaningful and accurate responses from the AI model. The provided prompt is relatively clear and well-structured, but its effectiveness might depend on the specific capabilities of the AI model being used. The authors should discuss their rationale for choosing this particular prompt and consider potential variations or improvements. They should also ensure the prompt does not contain leading information.
• Evaluation of Explanations: The scientific validity of the approach depends on how the researchers evaluate the quality and relevance of the AI-generated explanations. They should describe the criteria used to assess the explanations and whether they involved human evaluation or automated metrics. This is important for determining whether the AI is truly mimicking human expert reasoning or simply generating plausible-sounding but ultimately unhelpful explanations.
• Comparison to Human Experts: The section title, "Can LLMs be prompted to mimic human expert detectors?", suggests that the researchers are aiming to compare the performance of the prompted LLM to that of human experts. The authors should clearly define the criteria for successful mimicry and describe how they will compare the AI's explanations and decisions to those of human experts. This comparison is crucial for evaluating the potential of LLMs to perform this task at a human level.

• Caption Clarity: The caption clearly states that the table presents the "Prompt Template for the Zero-shot + CoT detector setup." This accurately describes the content of the table. However, it could be improved by briefly explaining what "zero-shot" and "CoT" mean and why this approach is being used in the study.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the different sections (question, description, answer) are well-defined. The instructions within the prompt are concise and relatively straightforward, although they might require some familiarity with AI prompting techniques to fully grasp.
• Reference Text Support: The reference text provides helpful context by explaining that the prompt is used to determine if the candidate text is human-written or AI-generated and to generate an explanation. This helps readers understand the purpose of the prompt and how it fits into the overall study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the concepts of "zero-shot" and "Chain of Thought" for readers who might be unfamiliar with these terms. Additionally, the table itself could benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Zero-Shot + Chain of Thought Detection of AI-Generated Text." While the prompt is generally clear, it could be further improved by providing more specific guidance on the types of features or clues the AI should focus on in its explanation (e.g., vocabulary, style, structure). Finally, including an example of a filled-in prompt with a candidate text and a sample AI-generated response could further enhance understanding.

• Purpose of the Table: This table presents the "prompt template" used to create a "zero-shot + guide detector." This is a type of AI model that is being tested to see if it can identify whether a piece of text was written by a human or an AI. The "zero-shot" part means the AI hasn't been specifically trained on this task before. The "guide" part refers to the AI Text Detection Guide developed earlier in the study (Table 11). This detector is given the detection guide as part of its instructions, unlike the previous detectors. It's like giving the AI a set of guidelines or a cheat sheet to help it make its decision.
• Content of the Table: The table shows the structure of the prompt, which includes placeholders for the "DETECTION GUIDE" and the "Candidate Text." The prompt instructs the AI to determine if the candidate text is "Human-written" or "AI-generated" *based on the information in the provided guide*. It also specifies the format for the AI's answer. The provided image shows the exact wording of this prompt template. The AI is being asked to use the guidelines to analyze the text and make a judgment, similar to how a human expert might use a rubric to evaluate something.
• Zero-Shot + Guide: The "zero-shot + guide" approach combines the idea of zero-shot learning (performing a task without specific training) with providing the AI with explicit guidelines or instructions. This is different from the previous prompts (Tables 18 and 19) where the AI was not given any specific guidance. By providing the AI with the detection guide, the researchers are essentially giving it a tool to help it mimic the reasoning process of a human expert who has access to such a guide. This is like giving someone a set of criteria to evaluate something they've never seen before.
• Comparison to Previous Tables: This table builds upon the prompt templates presented in Tables 18 and 19. In Table 18, the AI was given a simple zero-shot prompt without any guidance. In Table 19, the AI was prompted to use a Chain of Thought (CoT) approach to explain its reasoning. Here, in Table 20, the AI is given the detection guide as an additional resource. By comparing the performance of these different detectors, the researchers can assess the effectiveness of providing the AI with explicit guidelines compared to relying solely on its inherent capabilities or prompting it to explain its reasoning.

• Incorporating Expert Knowledge: Providing the AI with the detection guide is a scientifically sound approach for incorporating expert knowledge into the detection process. The guide, which was developed based on expert input, represents a distilled set of criteria for distinguishing between human and AI-generated text. By giving the AI access to this knowledge, the researchers are essentially equipping it with a tool that human experts find useful. The validity of this approach depends on the quality and comprehensiveness of the detection guide itself.
• Prompt Design: The specific wording and structure of the prompt are crucial for obtaining meaningful results. The provided prompt is relatively clear and well-structured, but its effectiveness might depend on the specific capabilities of the AI model being used. The authors should discuss their rationale for choosing this particular prompt and consider potential variations or improvements. The prompt should also ensure that the AI is using the guide actively and not simply ignoring it.
• Dependence on Guide Quality: The performance of the zero-shot + guide detector will heavily depend on the quality and relevance of the AI Text Detection Guide. If the guide is incomplete, inaccurate, or not well-suited to the specific characteristics of the AI-generated text being tested, the detector's performance might be compromised. The authors should thoroughly validate the guide and discuss its potential limitations.
• Comparison to Other Approaches: The scientific validity of this approach is best assessed by comparing its performance to other detection methods, such as the zero-shot detector (Table 18) and the zero-shot + CoT detector (Table 19). This comparison will help to determine whether providing the AI with the guide actually improves its ability to detect AI-generated text. The authors should also compare the performance of this detector to that of human experts to evaluate its potential for practical application.

• Caption Clarity: The caption clearly states that the table presents the "Prompt Template for the Zero-shot + Guide detector setup." This accurately describes the content of the table. However, it could be improved by briefly explaining what a "zero-shot + guide" detector is and why this approach is being used in the study.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the placeholders for the detection guide and candidate text are easily identifiable. The instructions within the prompt are concise and straightforward, although they might require some familiarity with AI prompting techniques to fully grasp.
• Reference Text Support: The reference text provides helpful context by explaining that the prompt is used to determine if the candidate text is human-written or AI-generated. This helps readers understand the purpose of the prompt and how it fits into the overall study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the concept of "zero-shot + guide" for readers who might be unfamiliar with the term. Additionally, the table itself could benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Zero-Shot Detection of AI-Generated Text Using the AI Text Detection Guide." While the prompt is generally clear, it could be further improved by providing more specific guidance on how the AI should use the detection guide in its analysis. Finally, including an example of a filled-in prompt with the detection guide and a candidate text could further enhance understanding.

• Purpose of the Table: This table presents the "prompt template" used to create a "zero-shot + CoT + guide detector." This is the most complex type of AI detector tested in the study. "Zero-shot" means the AI hasn't been trained on this specific task. "CoT" stands for "Chain of Thought," meaning the AI is asked to explain its reasoning. "Guide" refers to the AI Text Detection Guide developed earlier in the study. This detector combines all three approaches: it's a zero-shot detector that is also provided with the detection guide and asked to explain its reasoning step-by-step. It's like giving someone a set of guidelines and asking them to not only make a decision but also to explain their thought process using those guidelines, all without prior practice.
• Content of the Table: The table shows the structure of the prompt, which includes placeholders for the "DETECTION GUIDE" and the "Candidate Text." The prompt instructs the AI to determine if the candidate text is "Human-written" or "AI-generated" based on the provided guide. It also asks the AI to provide an explanation, or a "chain of thought," describing the features of the text that exemplify either human or AI writing. Finally, it specifies the format for the AI's answer, including a description and a final answer. The provided image shows the exact wording of this prompt template.
• Zero-Shot + CoT + Guide: This approach combines three elements: zero-shot learning (no prior training on the task), Chain of Thought prompting (explaining the reasoning), and providing the AI with the detection guide. The researchers are essentially trying to mimic the process a human expert might use, where they have a set of guidelines, they analyze the text, and they explain their reasoning before making a decision. This is the most sophisticated prompt used in the study, combining all the techniques explored previously.
• Comparison to Previous Tables: This table builds upon the prompt templates presented in Tables 18, 19, and 20. Table 18 used a simple zero-shot prompt. Table 19 added the CoT component. Table 20 introduced the detection guide. Table 21 combines all three approaches. By comparing the performance of these different detectors, the researchers can assess the effectiveness of each technique (zero-shot, CoT, and guide) individually and in combination. This allows them to determine which approach is most successful in mimicking human expert detection.

• Comprehensive Approach: Combining zero-shot learning, Chain of Thought prompting, and providing the detection guide is a scientifically sound approach for attempting to mimic human expert detection. It represents a comprehensive strategy that leverages different techniques for improving the performance and interpretability of AI models. The validity of this approach depends on the quality of the detection guide, the effectiveness of the CoT prompting, and the capabilities of the AI model being used.
• Prompt Design: The specific wording and structure of the prompt are crucial for obtaining meaningful results. The provided prompt is relatively clear and well-structured, but its effectiveness might depend on the specific capabilities of the AI model being used. The authors should discuss their rationale for choosing this particular prompt and consider potential variations or improvements. They should also ensure the prompt does not contain leading information that might bias the AI's response.
• Evaluation of Explanations: The scientific validity of the approach depends on how the researchers evaluate the quality and relevance of the AI-generated explanations. They should describe the criteria used to assess the explanations and whether they involved human evaluation or automated metrics. This is important for determining whether the AI is truly mimicking human expert reasoning or simply generating plausible-sounding but ultimately unhelpful explanations.
• Comparison to Human Experts and Other Detectors: The authors should compare the performance of the zero-shot + CoT + guide detector to that of human experts and the other detectors developed in the study (Tables 18, 19, and 20). This comparison is crucial for evaluating the potential of LLMs to perform this task at a human level and for determining the added value of each component of the prompt (zero-shot, CoT, guide). The authors should also discuss the limitations of this approach and potential areas for future improvement.

• Caption Clarity: The caption clearly states that the table presents the "Prompt Template for the Zero-shot + CoT + Guide detector setup." This accurately describes the content of the table. However, it could be improved by briefly explaining what "zero-shot," "CoT," and "guide" mean and why this combined approach is being used in the study.
• Table Organization and Content: The provided image of the table is well-organized and easy to understand. The prompt is clearly presented, and the different sections (question, description, answer) are well-defined. The instructions within the prompt are concise and relatively straightforward. The use of placeholders for the detection guide and candidate text is intuitive.
• Reference Text Support: The reference text provides helpful context by explaining that the prompt is used to determine if the candidate text is human-written or AI-generated and to generate an explanation. This helps readers understand the purpose of the prompt and how it fits into the overall study design.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the concepts of "zero-shot," "CoT," and "guide" for readers who might be unfamiliar with these terms. Additionally, the table itself could benefit from a more descriptive title that reflects its purpose, such as "Prompt Template for Zero-Shot Detection of AI-Generated Text with Chain of Thought and Detection Guide." While the prompt is generally clear, it could be further improved by providing more specific guidance on how the AI should use the detection guide in its analysis and explanation. Including an example of a filled-in prompt with the detection guide, a candidate text, and a sample AI-generated response could further enhance understanding.

Table 22: Each cell displays TPR (FPR), with TPR in normal text and FPR in smaller parentheses. Colors indicate performance bins: TPR is darkest teal (100) at best, medium teal (90-99), and burnt orange (89-70). Scores 69 and below are in purple (<70). FPR is darkest teal (0) at best, medium teal (1-5), burnt orange (6-10), and purple (>10) at worst. No percentage signs appear in the cells, but the numeric values represent percentages (e.g., "90" means 90%). We further mark closed-source ( ) and open-weights ( ) detectors.

• Purpose of the Table: This table likely presents the performance results of different AI models or methods used to detect whether a text was written by a human or an AI. These models or methods are referred to as "detectors." The table compares these detectors based on two key metrics: True Positive Rate (TPR) and False Positive Rate (FPR). It's like evaluating the accuracy of different diagnostic tests in medicine, where you want to know how well they can correctly identify both the presence and absence of a disease.
• Content of the Table: Each cell in the table shows the performance of a specific detector on a specific task or dataset. The performance is represented by two numbers: TPR and FPR. The TPR, shown in normal text, indicates the percentage of AI-generated texts that were correctly identified as AI-generated. The FPR, shown in smaller parentheses, indicates the percentage of human-written texts that were incorrectly identified as AI-generated. For example, a cell might show "85 (8)," which means the detector correctly identified 85% of the AI-generated texts (TPR) but also incorrectly flagged 8% of the human-written texts as AI-generated (FPR).
• Color Coding: The table uses a color-coding scheme to visually represent the performance levels. For TPR, darker teal indicates better performance (higher percentages), with the darkest teal representing a perfect score of 100. Burnt orange represents middling performance, and purple represents the worst performance (below 70). For FPR, it's the opposite: darker teal indicates better performance (lower percentages), with the darkest teal representing a perfect score of 0. Burnt orange represents middling performance, and purple represents the worst performance (above 10). This color scheme makes it easy to quickly identify the best and worst performing detectors.
• Closed-Source vs. Open-Weights: The caption mentions that the table further marks detectors as either "closed-source" or "open-weights." "Closed-source" means that the internal workings of the detector are not publicly available - it's like a secret recipe. "Open-weights" means that the model's parameters (the values that determine how it operates) are publicly available. This distinction is important because it relates to the transparency and reproducibility of the research. The specific symbols used to mark closed-source and open-weights detectors are not provided in the text.
• Numeric Values as Percentages: Although the cells in the table don't include percentage signs, the numeric values are meant to be interpreted as percentages. For example, a TPR of "90" means 90%, and an FPR of "5" means 5%. This is a common convention in reporting performance metrics, but it's important to clarify this in the caption to avoid any confusion. It's like saying a score of 0.9 on a test is equivalent to 90%.

• Choice of Metrics: Using TPR and FPR to evaluate the performance of detectors is a scientifically sound approach. These are standard metrics for assessing the accuracy of binary classification tasks (in this case, classifying text as human-written or AI-generated). However, it's important to consider other metrics as well, such as precision, recall, F1-score, and Area Under the Receiver Operating Characteristic Curve (AUROC), to provide a more comprehensive evaluation of performance. Different metrics can highlight different aspects of a detector's strengths and weaknesses.
• Thresholds for Model Usage: The reference text mentions that some automatic detectors did not provide suggested thresholds for model usage. This is an important consideration because the performance of a detector can vary depending on the threshold used to classify a text as AI-generated or human-written. The authors should describe how they determined the thresholds used for these detectors and discuss the potential impact of different threshold choices on the results. They should also consider using threshold-independent metrics like AUROC to compare the detectors.
• Statistical Significance: When comparing the performance of different detectors, it's important to determine whether the observed differences are statistically significant. The authors should conduct appropriate statistical tests (e.g., t-tests, ANOVA) to compare the TPR and FPR values across different detectors and report the p-values. This would help to establish whether the observed differences are likely due to chance or reflect real differences in performance.
• Generalizability: The performance of the detectors might vary depending on the specific characteristics of the AI-generated text being evaluated (e.g., the type of AI model used, the topic, the length). The authors should discuss the potential limitations of the generalizability of their findings to different types of AI-generated text and different detection tasks. They should also consider evaluating the detectors on a diverse range of datasets to assess their robustness.

• Caption Clarity: The caption is generally clear and provides a good explanation of the information presented in the table. It defines the metrics used (TPR and FPR), explains the color-coding scheme, and clarifies that the numeric values represent percentages. However, it could be improved by briefly explaining *why* these metrics are important in the context of the study and by providing a more descriptive title for the table that reflects its purpose (e.g., "Performance of Different Detectors on AI-Generated Text Detection Task").
• Table Organization and Content: Without seeing the actual table, it's difficult to fully assess its organization and content. However, assuming a standard tabular format with clear labels for rows (detectors) and columns (different types of AI-generated text or experiments), the table should be relatively easy to understand. The use of color-coding, as described in the caption, should further enhance the readability of the table. The specific detectors included in the table should be clearly identified and relevant to the study's focus on detecting AI-generated text.
• Explanation of Color Coding: The caption provides a detailed explanation of the color-coding scheme used in the table, which is helpful for interpreting the results. The use of different colors for different performance ranges allows for a quick visual assessment of the detectors' performance. However, the specific color choices should be carefully considered to ensure that they are easily distinguishable and accessible to individuals with color vision deficiencies.
• Potential Improvements: The communication effectiveness could be improved by adding a brief explanation in the caption of the significance of TPR and FPR in the context of detecting AI-generated text. Additionally, the table itself could include a brief legend or key that summarizes the color-coding scheme and the meaning of the symbols used to mark closed-source and open-weights detectors. While the caption mentions these symbols, they are not defined, which could lead to confusion. The table could also benefit from including a column or row that summarizes the overall performance of each detector across all experiments or datasets, to provide a more holistic view of their performance. Finally, although the caption states that the table presents TPR and FPR, it would be helpful to also include a column for accuracy to provide a more complete picture of the detectors' performance. The symbols used for closed-source and open-weights are not provided in the text and should be included in the caption or a footnote.