Evaluating Latent Knowledge of Public Tabular Datasets in Large Language Models

Abstract

Key Aspects

• 🗺️ The Confound of Dataset Contamination: This aspect introduces the central problem investigated: Large Language Models (LLMs) are increasingly evaluated on their ability to reason with structured, tabular data, but these assessments are often compromised by dataset contamination. This issue arises because popular benchmark datasets (e.g., Adult Income, Titanic) are likely present in the massive web-scraped corpora used to train LLMs. The authors posit that this overlap makes it difficult to distinguish between a model's genuine reasoning capabilities and its ability to simply recall memorized information, thereby inflating performance metrics and threatening the validity of research findings.
• 🔬 Probing the Role of Semantic Cues: The paper's methodology centers on a series of controlled probing experiments designed to isolate the mechanism behind contamination effects. The core hypothesis is that an LLM's prior knowledge is activated primarily by "strong semantic cues" within a dataset. These cues refer to human-interpretable elements like meaningful column names ('age', 'occupation') or value categories ('male', 'female') that allow the model to leverage its vast linguistic understanding. The experiments systematically test this by comparing model performance on original datasets versus versions where these semantic cues have been removed or randomized.
• 🔑 Performance Dependency on Semantics: The primary finding of the study is that an LLM's apparent competence on tabular tasks is critically dependent on the presence of semantic information. The experiments reveal that above-chance performance emerges almost exclusively for datasets rich with semantic cues. When these cues are neutralized through obfuscation or randomization, the models' performance drops dramatically to levels near random guessing. This result strongly suggests that the models are not reasoning from the raw numerical structure but are instead relying on memorized associations tied to the data's semantic meaning.
• 🎯 A Call for Contamination-Aware Evaluation: The paper concludes by discussing the significant implications of its findings for the field of LLM evaluation. The results indicate that what is often presented as tabular reasoning may, in large part, be a reflection of memorization facilitated by semantic leakage from training data. Consequently, the authors advocate for the development and adoption of new, more robust evaluation protocols. They propose that future assessments must incorporate strategies to disentangle authentic reasoning from latent knowledge to produce more accurate and reliable measures of LLM capabilities.

Strengths

• ✅ Clear and Logical Structure

  The abstract is exceptionally well-structured, presenting a clear and logical progression from the core problem to the proposed solution. It effectively establishes the context (contamination confound), outlines the method (probing experiments), states the key finding (dependence on semantic cues), and concludes with the broader implications, providing a comprehensive and easy-to-follow summary of the entire paper.

  "In this work, we investigate whether LLMs exhibit prior knowledge... Through a series of controlled probing experiments, we reveal that... These findings suggest that... We discuss implications for evaluation protocols..." (Page 1)
• ✅ Precise Framing of the Central Hypothesis

  The abstract excels at precisely defining the central mechanism under investigation: "strong semantic cues." By immediately specifying what this means (e.g., meaningful column names), it moves beyond a general discussion of contamination to offer a specific, testable hypothesis. This clarity effectively frames the paper's unique contribution and focuses the reader's attention on the key variable being manipulated.

  "...contamination effects emerge exclusively for datasets containing strong semantic cues—for instance, meaningful column names or interpretable value categories." (Page 1)

Suggestions for Improvement

• 💡 Quantify the Performance Decline for Greater Impact

  The statement that performance "sharply declines to near-random levels" is impactful, but the abstract could be strengthened by adding a quantitative example. Including specific accuracy figures (e.g., a drop from >70% to ~20%) would provide a more concrete and powerful illustration of the effect's magnitude right from the start. This is a high-impact change that aligns with the convention of empirical abstracts to feature key numerical results, making the findings more memorable and compelling.

  "In contrast, when such cues are removed or randomized, performance sharply declines to near-random levels." (Page 1)

  Implementation: Revise the sentence to include a parenthetical with example figures drawn from the paper's results. For instance: "In contrast, when such cues are removed or randomized, performance sharply declines to near-random levels (e.g., from over 70% to below 20% accuracy)."

• 💡 Explicitly Name the Proposed Strategies

  The final sentence mentions proposing "strategies to disentangle semantic leakage" but remains abstract. Specifying the primary strategy explored in the paper—namely, the use of semantically obfuscated or synthetic datasets for evaluation—would make the paper's contribution more concrete. This low-effort, high-impact clarification would provide readers with a clearer understanding of the paper's proposed solution without requiring them to read further.

  "We discuss implications for evaluation protocols and propose strategies to disentangle semantic leakage from authentic reasoning ability in future LLM assessments." (Page 1)

  Implementation: Refine the final sentence to be more specific. For example, change "propose strategies to disentangle semantic leakage from authentic reasoning ability" to "propose strategies, such as benchmarking on semantically obfuscated datasets, to disentangle semantic leakage from authentic reasoning ability."

1 Introduction

Key Aspects

• 🧠 Reasoning vs. Recollection: This aspect establishes the paper's broad context, highlighting a fundamental tension in Large Language Models (LLMs). While their training on vast text corpora enables impressive generalization, it also obscures whether their performance stems from genuine reasoning or simple recollection of information seen during training. This ambiguity creates an 'alarmingly blurred' boundary, questioning the true nature of LLM knowledge and capabilities when applied to tasks like classification and regression.
• 📊 The Problem of Tabular Contamination: The introduction narrows its focus to the critical issue of dataset contamination, specifically within common tabular benchmarks like 'Adult Income' and 'Titanic'. The authors argue that because these datasets are ubiquitous in online tutorials, academic papers, and competitions, they are almost certainly part of LLM pretraining corpora. Consequently, a model's apparent ability to 'reason over data' might be an illusion, reflecting instead memorized statistical patterns or schema-level cues absorbed from its training data.
• 🎯 Motivation and Research Approach: The motivation for this work is rooted in the increasing reliance on LLMs for data science tasks. The paper posits that if performance evaluations are inflated by prior data exposure, it undermines both the reproducibility and the validity of claims about emergent reasoning abilities in these models. To address this, the authors propose an approach of 'systematic probing and controlled testing' designed to disentangle true reasoning from latent recall, hypothesizing that contamination effects are primarily driven by strong semantic cues within the data.
• 🔑 Contribution and Call for New Protocols: The paper frames its ultimate contribution as advancing the understanding of how large-scale pretraining shapes the apparent intelligence of LLMs. By demonstrating the significant impact of semantic data leakage, the analysis culminates in a call to action for the research community. The authors urge the adoption of 'contamination-aware evaluation protocols' to ensure that future benchmarks can more reliably and validly assess the genuine reasoning capabilities of LLMs.

Strengths

• ✅ Effective 'Funnel' Structure

  The introduction is structured as a highly effective logical funnel. It begins with the broad, conceptual challenge of distinguishing reasoning from recollection in LLMs and progressively narrows the focus to the specific, testable problem of semantic contamination in tabular benchmarks. This structure masterfully builds the case for the research, ensuring the reader understands both the wider context and the precise issue being investigated.

  "As LLMs increasingly intersect with machine learning research... the boundary between reasoning and recollection becomes alarmingly blurred. In this work, we investigate a subtle yet critical issue: dataset contamination in LLMs, with a focus on widely used tabular benchmarks..." (Page 1)
• ✅ Compelling Justification of Significance

  The authors clearly articulate the 'so what?' of their research by directly linking the problem of data contamination to the core scientific principles of reproducibility and validity. This framing elevates the work from a niche technical investigation to a crucial inquiry into the integrity of LLM evaluation, making a strong case for its importance and timeliness.

  "If these models have prior exposure to benchmark datasets, their performance evaluations risk being inflated, undermining both reproducibility and the validity of claims about emergent reasoning." (Page 1)

Suggestions for Improvement

• 💡 Concretize 'Semantic Cues' Upfront

  The term 'strong semantic cues' is central to the paper's thesis but is introduced without immediate definition, leaving its meaning open to interpretation until later. Providing a concrete example when the term first appears would significantly enhance clarity. This is a low-effort, high-impact change that would immediately ground the paper's core mechanism for the reader, mirroring the effective strategy used in the abstract.

  "...revealing that contamination effects primarily emerge in datasets containing strong semantic cues—those where variable names or value labels align closely with natural language semantics." (Page 1)

  Implementation: In the third paragraph, amend the sentence to include a parenthetical explanation. For instance, change '...revealing that contamination effects primarily emerge in datasets containing strong semantic cues...' to '...revealing that contamination effects primarily emerge in datasets containing strong semantic cues—those where variable names or value labels align closely with natural language semantics.'

• 💡 Foreshadow the Experimental Contrast

  The introduction states the use of 'systematic probing and controlled testing' but misses an opportunity to briefly foreshadow the core experimental design—the comparison between original and semantically obfuscated datasets. Adding this detail would provide a clearer methodological roadmap and strengthen the narrative link between the stated problem and the proposed investigation. This medium-impact suggestion would better prepare the reader for the methodology section without revealing results.

  "Through systematic probing and controlled testing, we aim to disentangle genuine reasoning from latent recall..." (Page 1)

  Implementation: Expand the sentence in the third paragraph to hint at the experimental setup. For example, revise 'Through systematic probing and controlled testing, we aim to disentangle genuine reasoning from latent recall...' to 'Through systematic probing and controlled testing, which involves comparing performance on original datasets against semantically obfuscated versions, we aim to disentangle genuine reasoning from latent recall...'

2 Related Work

Key Aspects

• 📚 Historical Context of Contamination: This aspect establishes the historical foundation of data contamination as a significant concern in LLM research. It traces the issue back to early models like GPT-3, where primitive mitigation strategies such as n-gram overlap filtering were first proposed. The section highlights subsequent work that confirmed the pervasiveness of this problem, showing that large web-crawled corpora inherently contain verbatim text from standard evaluation benchmarks, rendering simple heuristic filters insufficient.
• 🛠️ Evolution of Detection and Mitigation: The review progresses to cover the evolution of methods for quantifying, detecting, and mitigating contamination. It discusses empirical studies that formalized the distinction between memorization and exploitation of leaked data, demonstrating how even limited duplication can inflate performance metrics. The section also introduces more sophisticated, recent techniques, such as instruction-based probing to directly query models for known instances and methods that leverage model confidence as a signal of prior exposure, reflecting a shift towards more robust detection and correction.
• 🌐 Contamination as a Cross-Modal Risk: This aspect broadens the scope of the contamination problem beyond the textual domain, establishing it as a fundamental risk inherent to large-scale pretraining across different data modalities. The paper cites research demonstrating similar leakage phenomena in both code generation, where programming tasks are duplicated in training corpora, and in vision-language models, which can benefit from prior exposure to test images. This contextualizes data contamination not as an NLP-specific issue but as a systemic challenge in modern AI.
• 🎯 The Research Gap in Tabular Data: This is the pivotal aspect of the section, where the authors identify and articulate the specific research gap their work aims to fill. It argues that despite the centrality of tabular data in classical machine learning, it has been largely overlooked in the contamination discourse. The paper posits that popular benchmarks like 'Adult Income' and 'Titanic' are especially vulnerable due to their long-term, unchanged public availability. It concludes by positioning its contribution as the first systematic analysis to address this gap, specifically by distinguishing between syntactic (exact record) and semantic (schema-level) leakage.

Strengths

• ✅ Comprehensive and Logical Funneling

  The section is structured as a highly effective logical funnel, skillfully guiding the reader from the broad, established problem of data contamination in LLMs to the specific, underexplored niche of tabular data. It begins with foundational work, covers the evolution of detection techniques and cross-modal issues, and culminates in a clear and compelling statement of the research gap. This structure masterfully builds the justification for the paper's contribution.

  "The problem of data contamination... has become a central concern in large language model (LLM) research. ... However, tabular data remain largely overlooked in this discourse." (Page 1)
• ✅ Explicit and Persuasive Gap Statement

  The final paragraph serves as an exemplary 'gap statement,' clearly articulating why the present study is both necessary and novel. It effectively synthesizes the preceding review to highlight that while contamination is a known issue, its specific manifestation in tabular data is under-investigated. The authors persuasively argue that existing warnings lack systematic analysis, thereby carving out a distinct and valuable research space.

  "Recent discussions... have warned that pretrained tabular and forecasting models may inadvertently include these datasets, but systematic analyses are scarce. Our work addresses this gap..." (Page 2)

Suggestions for Improvement

• 💡 Preemptively Clarify Key Terminology

  This is a medium-impact suggestion. The final paragraph introduces the crucial distinction between 'syntactic' and 'semantic' leakage, which is central to the paper's contribution. While these terms are formally defined in Section 3.1, providing a brief, parenthetical definition here would immediately clarify the specific nature of the identified research gap for the reader. This would enhance the narrative flow and make the transition to the problem definition section smoother and more intuitive.

  "...distinguishing between syntactic (exact record overlap) and semantic (schema- or feature-level) leakage, and assessing how such contamination biases LLM-based reasoning evaluations." (Page 2)

  Implementation: In the final sentence of the section, expand on the terms when they are first introduced. For example, revise '...distinguishing between syntactic (exact record overlap) and semantic (schema- or feature-level) leakage...' to '...distinguishing between syntactic leakage, which involves exact record overlap, and semantic leakage, which concerns schema- or feature-level associations...'.

• 💡 Bridge the Gap to the Central Hypothesis

  This is a high-impact suggestion. The section successfully identifies the research gap in tabular data contamination but misses an opportunity to explicitly link it to the paper's core hypothesis about the role of 'semantic cues' (as stated in the Abstract and Introduction). Mentioning that this gap is especially critical because tabular benchmarks often contain rich semantic information would create a more seamless narrative bridge from the existing literature to the specific investigation this paper undertakes, reinforcing the novelty and importance of the work from the outset.

  "However, tabular data remain largely overlooked in this discourse. Despite their centrality in classical machine learning, benchmarks such as Adult Income, or Titanic, are often publicly available in unchanged form for years, making them especially susceptible to leakage..." (Page 2)

  Implementation: In the final paragraph, add a clause that connects the susceptibility of tabular data to the concept of semantic cues. For instance, after '...making them especially susceptible to leakage through web-scraped pretraining data,' add a phrase like 'a risk amplified by the strong semantic cues embedded in their schemas which this paper investigates.'

3 Background and Problem Definition

Key Aspects

• ⚖️ Defining Syntactic vs. Semantic Contamination: This aspect formally defines the paper's central methodological concern: dataset contamination. The authors establish a critical taxonomy by distinguishing between two primary forms. 'Syntactic contamination' refers to the verbatim inclusion of dataset entries in the training corpus, leading to direct memorization. In contrast, 'semantic contamination' is a more subtle form arising from exposure to descriptions, summaries, or conceptual patterns related to the data, which induces a 'spurious familiarity' without direct textual overlap. This distinction is foundational to the paper's experimental design and interpretation of results.
• 📊 The Tabular Data Paradox: The paper justifies its focus on tabular data by highlighting a unique paradox this data modality presents for LLMs. While structured, schema-driven data is not inherently linguistic and might seem less susceptible to inclusion in text-based pretraining corpora, popular benchmarks like Titanic are ubiquitous in online tutorials and teaching materials. Furthermore, their schemas contain strong 'semantic cues' (e.g., meaningful feature names like 'age' or 'occupation'), making tabular data an ideal and controlled testbed for isolating and studying the mechanisms of contamination.
• 📉 Threats to Scientific Validity: This aspect outlines the severe consequences of data contamination for the empirical evaluation of LLMs, framing it as a direct threat to scientific validity. The authors argue that when a model performs well on a benchmark it has already seen, the results do not reflect genuine reasoning or generalization but rather the retrieval of memorized patterns. This fundamentally undermines claims about core AI capabilities like zero-shot learning and causal reasoning, thereby obscuring the critical boundary between acquired knowledge and true inferential ability.
• ❓ Framing the Inquiry: The section culminates by translating the abstract problem into two precise and testable research questions that guide the subsequent investigation. The first question (RQ1) probes the extent of direct, verbatim recall by asking if LLMs remember specific data entries. The second, more central question (RQ2), investigates the primary mechanism of contamination, seeking to determine whether it occurs through semantic cues (feature meanings) or exact syntactic memorization. These questions provide a clear and direct structure for the experimental methodology.

Strengths

• ✅ Precise and Well-Exemplified Definitions

  The section excels at establishing a clear conceptual framework by providing precise, distinct definitions for 'syntactic' and 'semantic' contamination. The inclusion of concrete, intuitive examples (e.g., a summary of high earners in the Adult dataset) makes these abstract concepts immediately understandable and grounds the paper's core argument in tangible scenarios, which is crucial for the problem definition.

  "Semantic contamination arises when the model has been exposed not to the exact data but to descriptions, summaries, or derived representations (e.g., “people earning over 50K in the Adult dataset are often men in tech jobs”)." (Page 2)
• ✅ Strong Logical Progression

  The section is structured with exemplary clarity, moving logically from a general definition of the problem (3.1) to its specific manifestation in the chosen domain (3.2), its scientific consequences (3.3), and finally to the precise questions that will guide the investigation (3.4). This progression (What is it? -> Why this domain? -> Why does it matter? -> What will we ask?) effectively builds the rationale for the study and prepares the reader for the methodology.

  "Dataset contamination poses a severe threat to validity in the empirical evaluation of LLMs... To systematically investigate this issue, we frame our study around the following research questions:" (Page 2)

Suggestions for Improvement

• 💡 Explicitly State the Central Hypothesis

  This is a high-impact suggestion. While the research questions are clearly articulated, the section would be strengthened by explicitly stating the study's central hypothesis after them. Based on the paper's framing, the hypothesis is that contamination is primarily semantic (addressing RQ2). Stating this directly in the problem definition section would provide a stronger argumentative anchor, clarify the authors' expectations, and create a more direct logical link between the problem definition and the experimental results, which is a conventional and effective practice in empirical papers.

  "RQ2: Does contamination primarily occur through semantic cues (feature and label meanings) or exact memorisation of data entries?" (Page 2)

  Implementation: After listing RQ1 and RQ2, add a concluding sentence or brief paragraph to Section 3.4 stating the hypothesis. For example: "Based on the widespread availability of dataset descriptions and the semantic richness of their schemas, we hypothesize that contamination occurs primarily through semantic channels (RQ2) rather than through exact memorization of entries (RQ1)."

4 Methodology

Key Aspects

• 🎯 Experimental Objective: This aspect outlines the core objective of the experimental framework: to systematically disentangle two distinct types of data contamination in LLMs. The methodology is explicitly designed to differentiate between performance gains arising from syntactic contamination (direct memorization of data records or value patterns) and those from semantic contamination (reliance on learned associations tied to meaningful feature names and domain context). This foundational goal directly addresses the paper's central research questions and shapes the entire experimental design.
• 🛠️ Contamination Probing Tasks: The methodology operationalizes its objective through two complementary assessment tasks designed to probe for contamination at different levels. The 'Completion Task' involves masking a portion of a record's attributes and asking the model to select the true value from a list of candidates, testing for memorized inter-feature associations. The 'Existence Task' presents the model with one genuine and four perturbed versions of a record and asks it to identify the original, directly testing for verbatim memorization of entire data instances.
• 🧪 Controlled Dataset Variants: To isolate the specific mechanisms of contamination, the methodology introduces two crucial controlled modifications of the benchmark datasets. The 'Like (Synthetic Marginal) Dataset' preserves the statistical properties of individual features but eliminates inter-feature dependencies, thereby neutralizing syntactic memorization of record structures. The 'Obfuscated Dataset' systematically replaces all meaningful categorical labels and feature names with abstract symbols, which disrupts domain-level semantics and allows the researchers to measure the model's reliance on semantic contamination.

Strengths

• ✅ Excellent Operationalization of Concepts

  The methodology excels at translating the abstract theoretical concepts of 'syntactic' and 'semantic' contamination into a concrete, measurable, and falsifiable experimental design. The dual-task structure (Completion vs. Existence) and the controlled dataset variants (Obfuscated vs. Synthetic) create a robust framework that effectively isolates the variables of interest, representing a high standard of scientific rigor.

  "Our primary goal was to disentangle cases where model performance arises from memorisation of known records or column, i.e., value patterns (syntactic contamination) versus reliance on learned associations tied to domain semantics or feature names (semantic contamination)." (Page 2)
• ✅ Clear and Logical Experimental Design

  The section is structured with exemplary clarity, presenting a logical progression that is easy for the reader to follow. It begins by stating the overarching goal, then details the specific tasks designed to probe for contamination, and finally explains the controlled dataset variants used to distinguish between contamination types. This clear, step-by-step exposition enhances the transparency and reproducibility of the study.

  "To distinguish contamination from genuine reasoning, we further introduced two controlled modifications of each dataset:" (Page 3)

Suggestions for Improvement

• 💡 Justify the 20% Masking/Perturbation Rate

  This is a medium-impact suggestion to enhance methodological rigor. The paper specifies a 20% rate for both masking attributes in the Completion Task and perturbing them in the Existence Task but does not provide a rationale for this specific value. Justifying this parameter choice—for instance, by explaining how it balances task difficulty with the need to retain sufficient record context for the model—would strengthen the methodology's transparency and help readers assess the robustness of the experimental design.

  "For each record, we randomly masked 20% of the available attributes, ensuring a balanced mix of numerical and categorical features." (Page 2)

  Implementation: Add a brief sentence or clause after the parameter is introduced. For example: "For each record, we randomly masked 20% of the available attributes, a rate chosen to pose a significant challenge to the model while preserving enough contextual information for potential completion."

• 💡 Provide More Detail on Feature Selection for Masking

  This is a low-impact suggestion aimed at improving reproducibility. The methodology states that masked variables were selected from the 'top four features with the highest entropy (categorical) or variance (numerical)'. While this provides a clear principle, it leaves some ambiguity. Clarifying whether the 20% of attributes were sampled exclusively from this pool of four, or if this pool was just a priority, would remove this ambiguity and allow for more precise replication of the experiment.

  "Masked variables were selected among the top four features with the highest entropy (categorical) or variance (numerical)." (Page 3)

  Implementation: Refine the sentence to clarify the sampling process. For example, change 'Masked variables were selected among the top four features...' to 'The attributes to be masked were randomly sampled from a pool consisting of the four features with the highest entropy (for categorical) or variance (for numerical).'

5 Experiments

Key Aspects

• 📊 Experimental Setup and Datasets: This aspect details the experimental configuration, which is designed to systematically investigate data contamination. The authors selected a diverse set of benchmark tabular datasets, categorizing them as either 'semantic' (containing human-interpretable fields like 'occupation' or 'fare') or 'non-semantic' (containing abstract or purely numerical features). The study evaluates three families of open-weight LLMs (llama_8B, mistral_7B, and multiple qwen variants) in a zero-shot setting, applying the 'Completion' and 'Existence' tasks defined in the methodology to each dataset.
• 🔑 Core Finding: Semantic, Not Syntactic, Contamination: The central empirical finding of the paper is that evidence of contamination emerges exclusively through semantic channels. Models achieved performance substantially above the random baseline only on the 'Completion' task and only for the most widely known semantic datasets (Adult, Titanic, Credit). Conversely, the 'Existence' task, which directly tests for verbatim memorization of data instances, was consistently failed across all models and datasets. This stark contrast provides strong evidence that the observed contamination is not syntactic (direct recall) but semantic, driven by the models' familiarity with the meaning of the data's features.
• 🧠 Conclusion: Remembering Meaning, Not Data: This aspect encapsulates the main conclusion drawn from the experimental results: the LLMs tested do not remember specific data entries but rather the meaning and conceptual relationships within the data. The models leverage contextual and conceptual knowledge acquired during pretraining to infer plausible values when dataset semantics are familiar, a phenomenon termed 'semantic familiarity'. This critical distinction implies that while direct data leakage at the instance level is not evident, performance on popular benchmarks can still be artificially inflated, necessitating the use of less disseminated datasets for valid evaluations of true reasoning ability.

Strengths

• ✅ Clear and Direct Interpretation of Results

  The section excels at presenting the results with exceptional clarity, directly and unambiguously answering the research questions posed earlier in the paper. The authors effectively synthesize the outcomes of the various tasks and dataset variants into a coherent and compelling narrative, culminating in a powerful summary statement that distinguishes between remembering 'data' and remembering 'meaning'.

  "Overall, our findings provide clear answers to the two research questions. Regarding RQ1, the systematic failure of all models in the existence task indicates that LLMs do not recall specific entries... Concerning RQ2, the strong and consistent performance... demonstrates that contamination primarily occurs through semantic channels..." (Page 5)
• ✅ Strong Validation of Methodological Design

  The experimental results provide a strong validation for the methodological design outlined in Section 4. The clear divergence in performance between the 'Completion' and 'Existence' tasks, as well as between semantic and non-semantic datasets, demonstrates that the experimental framework was successful in isolating and measuring the distinct phenomena of semantic and syntactic contamination.

  "By contrast, the existence task, which directly tests for the memorisation of specific records, was consistently failed across all models and datasets. This systematic failure indicates that none of the examined LLMs... exhibit clear syntactic contamination or verbatim memorisation of data instances." (Page 3)

Suggestions for Improvement

• 💡 Quantify Key Performance Metrics in the Narrative

  This is a medium-impact suggestion to enhance the immediacy and force of the findings. The text states that models achieved accuracies 'substantially above the random-guess baseline' but relies on the reader consulting Table 1 to grasp the magnitude. Integrating key quantitative results directly into the narrative would make the Results section more self-contained and impactful, immediately conveying the strength of the observed effect without breaking the reading flow.

  "In these cases, all evaluated models achieved accuracies substantially above the random-guess baseline (≈ 20%), suggesting that the models likely possess semantic-level familiarity with these datasets." (Page 3)

  Implementation: Revise the sentence to include a parenthetical with the peak accuracy observed. For example: '...all evaluated models achieved accuracies substantially above the random-guess baseline (≈ 20%), with some models reaching over 70% accuracy (e.g., 78% for qwen_32BQ on the 'credit' dataset), suggesting that the models likely possess semantic-level familiarity...'

• 💡 Elaborate on the 'Emergent Reasoning' Hypothesis

  This is a high-impact suggestion. The paper identifies an 'intriguing exception' where models show non-trivial accuracy on the obfuscated 'credit' dataset and hypothesizes this is due to 'emergent reasoning' on latent statistical regularities. This is a potentially significant finding that deviates from the main narrative but is mentioned only briefly. Expanding on this point within the Experiments section, even with a sentence or two of additional speculation on what specific structural properties of the data might enable this, would add considerable depth and nuance to the interpretation of the results.

  "An intriguing exception arises in the credit dataset, where some models maintain non-trivial accuracy even in the obfuscated variant. We hypothesise that this effect reflects the models’ ability to exploit latent statistical regularities within the tabular structure..." (Page 5)

  Implementation: After introducing the hypothesis, add a sentence speculating on the cause. For instance: 'We hypothesise that this effect reflects the models’ ability to exploit latent statistical regularities... an instance of... emergent reasoning abilities, potentially driven by strong correlations between its numerical features that persist even after semantic obfuscation.'

Non-Text Elements

Table 1: Summary of contamination results across datasets and models. AC...

Full Caption

Table 1: Summary of contamination results across datasets and models. AC denotes accuracy in the Completion task, while AE denotes accuracy in the Existence task. Bold values indicate performances that are statistically significant above the random-guess baseline (a = 0.001).

Figure/Table Image (Page 4)

First Reference in Text

As summarised in Table 1, our experiments reveal a clear divide between semantic and non-semantic datasets in terms of potential data contamination.

Description

• Experimental Tasks and Metrics: The table measures the performance of several large language models on two tasks. The 'Completion task' (AC) tests if a model can correctly fill in missing values in a row of data, with accuracy measured against four incorrect options (a random guess would be 20% accurate). The 'Existence task' (AE) tests if a model can identify an authentic row of data from among four perturbed, or slightly altered, versions. This helps distinguish between understanding relationships in data versus rote memorization of exact rows.
• Dataset Categories and Variants: The experiments use two main types of datasets. 'Semantic Datasets' like 'adult' and 'titanic' use human-readable column names (e.g., 'age', 'occupation') that provide contextual clues. 'Non-semantic Datasets' like 'gamma' use abstract numerical data without obvious real-world meaning. Each dataset is also tested in different forms: 'real' (the original data), 'like' (synthetic data that mimics statistical properties but not exact rows), and 'obf' (obfuscated, where meaningful names are replaced with generic codes like 'c01').
• Performance on Semantic Datasets: For well-known semantic datasets like 'adult' and 'credit', the models show high accuracy on the Completion task (AC), far exceeding the 20% random-guess baseline. For instance, the 'qwen_32BQ' model achieves 73% accuracy on the 'adult' dataset and 78% on the 'credit' dataset. However, their performance on the Existence task (AE) for these same datasets is very low (e.g., 36% for 'adult'), suggesting the models have learned relationships between features but have not memorized the exact data rows.
• Performance on Non-semantic and Obfuscated Datasets: When models are tested on non-semantic datasets ('gamma', 'synthetic'), their accuracy on the Completion task drops to levels near random guessing (around 18-25%). Similarly, when a semantic dataset like 'adult' is obfuscated (stripping it of meaningful names), the 'qwen_32BQ' model's accuracy plummets from 73% to 13%. This indicates that the models' high performance relies heavily on the semantic meaning embedded in the feature names and values, not on an inherent ability to reason about abstract tabular data.

Scientific Validity

• ✅ Strong Experimental Controls: The experimental design is robust. The inclusion of non-semantic datasets as a negative control, and 'obfuscated' and 'like' variants as controls for different types of information leakage, allows for a clear disentanglement of semantic contamination from syntactic memorization or structural pattern learning. This strengthens the conclusion that semantic cues are the primary driver of inflated performance.
• ✅ Clear Support for Central Claims: The data presented in the table provides compelling evidence for the paper's main arguments. The stark difference in Completion Accuracy (AC) between semantic and non-semantic datasets, combined with the consistently low Existence Accuracy (AE) across all conditions, strongly supports the conclusion that contamination is semantic in nature and does not involve verbatim memorization of instances.
• 💡 Lack of Reported Variance: The table reports point estimates for accuracy scores from what appears to be a single experimental run. Without measures of variance, such as standard deviations or confidence intervals derived from multiple runs with different random seeds, it is difficult to assess the stability and statistical certainty of these results. Performance could be subject to noise from the random sampling of instances and features.
• 💡 Ambiguity of the Random-Guess Baseline: The caption notes that bold values are significant above a 'random-guess baseline' but does not explicitly state its value. The text implies a 1-in-5 choice task, making the baseline 0.20. Stating this value directly in the caption (e.g., '...baseline of 0.20 (a = 0.001)') would make the table more self-contained and interpretation more immediate for the reader.

Communication

• ✅ Effective Hierarchical Structure: The table is well-organized with a clear hierarchy: dataset type (Semantic/Non-semantic), specific dataset, variant (real/like/obf), and metric (AC/AE). This structure makes it easy for readers to make critical comparisons, such as contrasting 'real' vs. 'obf' for a given dataset, which is central to the paper's argument.
• ✅ Good Use of Text Formatting: Defining the abbreviations 'AC' and 'AE' in the caption is good practice and essential for comprehension. Furthermore, using bold font to highlight statistically significant results allows the reader to quickly identify the key outcomes without needing to mentally compare each value to the baseline.
• 💡 High Information Density Could Be Complemented Visually: The table is very dense, presenting 160 distinct results. While comprehensive, this makes it challenging to quickly grasp the main trends. A supplementary figure, such as a bar plot comparing AC scores for 'real', 'like', and 'obf' variants on the 'adult' and 'gamma' datasets, would provide a more intuitive and immediate visual summary of the core findings.
• 💡 Inconsistent Numerical Alignment: The numbers in the columns are not consistently aligned by the decimal point. This minor formatting issue slightly hinders readability and the ease of scanning down columns to compare values. Aligning all numerical values to the decimal point would improve the table's professionalism and visual clarity.

6 Limitations and Future Work

Key Aspects

• 🗺️ Delineating the Study's Scope: The authors transparently acknowledge the primary constraints on their findings' generalizability. The first limitation is the scale of the models tested, which were capped at 32 billion parameters; they posit that larger models (e.g., >70B) might exhibit different contamination behaviors due to enhanced memorization capacity. The second limitation is the restricted scope and diversity of the datasets used, highlighting the need for a broader set of benchmarks to more effectively distinguish genuine reasoning from mere dataset familiarity.
• 🧭 Charting Future Research Directions: This aspect outlines a multi-dimensional plan for future research that directly extends from the identified limitations. The primary proposed direction is to evaluate larger-scale LLMs to investigate how contamination patterns scale with model size and data exposure. A second, more novel direction involves developing an empirical framework to categorize datasets based on their observed level of contamination, creating a formal distinction between 'contaminated' and 'clean' benchmarks for tabular data.
• 🎯 The Goal of Standardized Benchmarking: The section culminates by stating the ultimate ambition of this research trajectory: to establish standardized, contamination-aware benchmarks for evaluating LLMs on tabular data. By understanding whether contamination merely impacts recall or also biases higher-order reasoning and generalization, this work aims to contribute to a more rigorous and reliable evaluation ecosystem. This goal positions the paper not just as an isolated study but as a foundational step toward improving the scientific validity of future LLM research in this domain.

Strengths

• ✅ Transparent and Honest Self-Critique

  The section demonstrates strong academic integrity by clearly and proactively identifying the key limitations of the study. Rather than overstating the conclusions, the authors provide a balanced perspective on the generalizability of their findings, which enhances the credibility and trustworthiness of the research.

  "Our study provides an initial yet informative exploration of contamination effects in LLMs... Nonetheless, several limitations constrain the generalizability of our conclusions." (Page 5)
• ✅ Logical and Coherent Research Agenda

  The proposed future work is not a disparate list of ideas but a tightly-coupled and logical extension of the identified limitations. The plan to evaluate larger models and expand the dataset scope directly addresses the constraints mentioned, creating a clear and compelling roadmap for subsequent research that builds directly on the current paper's foundation.

  "Future work will aim to extend these analyses along multiple dimensions. A primary direction involves evaluating larger LLMs to test whether contamination patterns scale with model size or data exposure." (Page 5)

Suggestions for Improvement

• 💡 Specify the Nature of Proposed Datasets

  This is a medium-impact suggestion. The text proposes expanding the diversity of benchmarks, particularly 'non-semantic datasets,' but this remains abstract. Specifying the types of non-semantic datasets envisioned (e.g., from scientific domains like genomics, physics, or finance where features are often numerical and lack common-sense linguistic cues) would make the future work plan more concrete and guide other researchers looking to build on this work.

  "Expanding both the number and diversity of benchmarks, particularly by incorporating more widely used and non-semantic datasets, would enable a clearer separation between genuine reasoning ability and dataset familiarity effects." (Page 5)

  Implementation: In the first paragraph, after mentioning the need for more non-semantic datasets, add a parenthetical with examples. For instance: '...by incorporating more widely used and non-semantic datasets (e.g., from domains such as genomics or high-energy physics where features represent sensor readings or abstract measurements), would enable a clearer separation...'

• 💡 Propose a Metric for Contamination Level

  This is a high-impact suggestion. The paper proposes a key future step: categorizing datasets by their 'empirically observed contamination level.' However, it does not suggest how this level might be quantified. Briefly proposing a potential metric or method—such as developing a 'Contamination Score' based on the performance gap between the original and obfuscated versions in the Completion Task—would significantly strengthen the methodological foresight of the future work section and make the proposed research agenda more tangible.

  "Moreover, we plan to categorize datasets according to their empirically observed contamination level." (Page 5)

  Implementation: In the second paragraph, after the sentence about categorizing datasets, add a sentence outlining a potential approach. For example: 'Moreover, we plan to categorize datasets according to their empirically observed contamination level. This could be operationalized by defining a 'Contamination Score' for each dataset, calculated as the performance difference between its original and obfuscated variants on our probing tasks.'

7 Conclusions

Key Aspects

• 🔑 Empirical Confirmation of Semantic Contamination: The conclusion crystallizes the study's primary empirical contribution: providing clear evidence that well-known tabular benchmarks, such as Adult Income and Titanic, are subject to data contamination when evaluated with LLMs up to 32B parameters. This restates the central finding, grounding the paper's entire argument in a concrete, demonstrated phenomenon. It serves as the foundational claim upon which the subsequent interpretations and implications are built.
• 🧠 The Nature of Contamination: Meaning over Memorization: This aspect clarifies the specific mechanism of the observed contamination, defining it as predominantly semantic rather than syntactic. The authors emphasize that models are not directly memorizing individual data entries but are instead leveraging prior knowledge about the meaning of features and their contextual relationships. This distinction is the paper's core theoretical insight, shifting the focus from simple data leakage to the more nuanced issue of how pretraining shapes a model's conceptual understanding.
• 🛠️ A Call for Methodological Reform: Moving beyond the specific findings, this aspect articulates the paper's broader contribution to the field. The work is positioned as a catalyst for change, aiming to raise awareness about the subtleties of contamination in structured data tasks. By providing a 'methodological foundation,' the study advocates for the development of more transparent, contamination-aware evaluation pipelines to ensure that future research is fairer, more reproducible, and accurately measures genuine reasoning abilities.

Strengths

• ✅ Effective Synthesis of Core Contributions

  The conclusion is exceptionally well-crafted to serve its primary function: synthesizing the paper's main takeaways. It concisely restates the key empirical finding (semantic contamination exists), the central insight (it's semantic, not syntactic), and the broader contribution (a call for methodological reform), providing a clear, powerful, and self-contained summary of the entire work.

  "Our study provides empirical evidence of data contamination... The contamination observed is predominantly semantic... our work aims to raise awareness and provide a methodological foundation..." (Page 5)
• ✅ Forward-Looking and Impact-Oriented

  The conclusion successfully frames the research not as an end in itself, but as a foundational step toward improving the field. By emphasizing the goals of promoting 'fairer and more reproducible future research' and providing a 'methodological foundation,' the authors effectively articulate the long-term value and impact of their work, strengthening its overall significance.

  "...our work aims to raise awareness and provide a methodological foundation for more transparent and contamination-aware evaluation pipelines... promoting fairer and more reproducible future research." (Page 5)

Suggestions for Improvement

• 💡 Explicitly Reiterate the Null Finding on Syntactic Contamination

  This is a low-impact suggestion for enhancing completeness. The conclusion correctly emphasizes the primary finding of semantic contamination but misses an opportunity to explicitly restate the equally important null finding: the lack of evidence for direct, syntactic memorization. Including this contrast would make the conclusion a more comprehensive summary of how the paper answered both of its core research questions (RQ1 and RQ2), reinforcing the central dichotomy that drives the paper's argument.

  "...indicating that models leverage prior knowledge about feature meanings and contextual associations rather than directly memorising tabular entries." (Page 5)

  Implementation: After stating that contamination is semantic, add a clause that contrasts this with the lack of syntactic evidence. For example, revise the sentence to: 'The contamination observed is predominantly semantic... rather than directly memorising tabular entries, for which our experiments found no evidence.'

• 💡 Concretize the 'Methodological Foundation' with an Example

  This is a medium-impact suggestion. The conclusion calls for 'contamination-aware evaluation pipelines' and offers a 'methodological foundation,' but these phrases remain abstract. The paper's key practical implication, articulated at the end of Section 5, is the need to use less-disseminated or synthetic datasets. Briefly referencing this specific strategy in the conclusion would make the call to action more concrete and actionable for readers, providing a tangible example of what a 'contamination-aware' approach entails.

  "...our work aims to raise awareness and provide a methodological foundation for more transparent and contamination-aware evaluation pipelines of LLMs in tabular reasoning and prediction tasks..." (Page 5)

  Implementation: In the final sentence, add a clause that gives an example of a contamination-aware practice. For instance: '...provide a methodological foundation for more transparent and contamination-aware evaluation pipelines, such as by prioritizing evaluation on less-disseminated datasets, promoting fairer and more reproducible future research.'