Exploration of Sparse Autoencoder Feature Structure: Multi-Scale Analysis

Figure 1. Parallelogram and trapezoid structure is revealed (left) when using LDA to project out distractor dimensions, tightening up clusters of pairwise Gemma-2-2b activation differences (right).

• Overall purpose and structure of the figure: This figure aims to show how applying Linear Discriminant Analysis (LDA), a method for finding the best linear combination of features to separate different groups, reveals hidden "crystal structures" within the feature space of a language model called Gemma-2-2b. Imagine the model's "thoughts" as points in a high-dimensional space. Sometimes, these points form geometric shapes like parallelograms or trapezoids, where each point represents a word or concept. These shapes, called "crystals," capture relationships between concepts (e.g., "man" is to "woman" as "king" is to "queen"). However, irrelevant information, like word length, can obscure these shapes. LDA helps by projecting the points onto a lower-dimensional space where these irrelevant "distractor dimensions" are minimized, making the "crystals" more apparent. The left side likely shows the effect of LDA, while the right side visualizes the improved clusters of points representing differences between word activations, now forming tighter, clearer shapes.
• Visualization method and data representation: The figure uses a 2D scatter plot to visualize the "crystal structures." Each point represents a pairwise difference between the activations of two words or concepts in the language model. The position of the point is determined by the values of the first two principal components after LDA is applied. Color-coding is used to distinguish different clusters, which presumably correspond to different types of semantic relationships (e.g., country-capital, singular-plural). The right side likely shows these clusters after LDA has been applied, demonstrating how removing distractor dimensions tightens the clusters and makes the parallelogram/trapezoid shapes more apparent.

• Justification for using LDA and quantifying improvement: Using LDA to remove distractor dimensions is a valid approach, especially when dealing with high-dimensional data and known confounders. The authors' claim that this reveals underlying structure is plausible, but requires further justification. They should quantify the improvement in "crystal structure" using metrics like variance explained within clusters or the degree to which the shapes resemble ideal parallelograms/trapezoids. Simply stating "dramatic improvement" is subjective and not scientifically rigorous. The authors should also justify their choice of LDA parameters and demonstrate that the results are robust to different parameter settings.
• Validity of "crystal structures" as semantic representations: The authors' focus on "crystal structures" as indicators of semantic relationships is interesting, but needs further validation. They should provide evidence that these geometric shapes genuinely reflect meaningful semantic connections, beyond anecdotal examples. A more systematic analysis of the words/concepts within each cluster is needed to establish the validity of this approach. They should also consider alternative methods for dimensionality reduction and clustering, and compare their performance to LDA.

• Clarity and interpretability of visualization: The figure effectively communicates the main idea of crystal structure improvement through LDA. The visualization of clusters before and after LDA application clearly demonstrates the impact of distractor dimensions. The color-coding helps differentiate clusters, and the choice of a 2D projection makes the results easily interpretable. Labeling the axes with PC1 and PC2 is standard practice and helps orient the reader. Including examples like (Austria, Vienna, Switzerland, Bern) adds a concrete illustration of the concept. However, directly labeling the clusters with the transformations they represent (e.g., "country-capital") would enhance clarity and avoid relying solely on color-coding. Additionally, a brief note in the caption about the specific LDA implementation or parameter choices would improve reproducibility.
• Informativeness of caption: The caption clearly explains the figure's purpose, highlighting the effect of LDA on revealing parallelogram and trapezoid structures. The reference to "distractor dimensions" provides context, and the mention of "Gemma-2-2b activation differences" specifies the data source. However, the caption could be more informative by briefly mentioning the meaning of "crystal structure" in this context. For example, adding a phrase like "...revealing parallelogram and trapezoid structures indicative of semantic relations between concepts (crystal structure)..." would improve understanding for a broader audience.

Figure 11. Silhouette Score, a measure of clustering quality, as a function of reduced dimension in LDA.

• Purpose of the figure and explanation of silhouette score and LDA: This figure assesses the quality of clustering achieved after applying Linear Discriminant Analysis (LDA), a dimensionality reduction technique. Imagine you have a bunch of points representing features, and you want to group them into clusters based on their similarity. LDA helps by projecting these points onto a lower-dimensional space, making it easier to separate the clusters. This figure uses the silhouette score, a metric that measures how similar each point is to its own cluster compared to other clusters, to evaluate the quality of clustering after LDA is applied. A higher silhouette score indicates better-defined clusters.
• Explanation of axes and different lines: The x-axis represents the number of dimensions after LDA is applied. The y-axis represents the silhouette score, which typically ranges from -1 to 1. Values closer to 1 indicate better clustering. The plot shows how the silhouette score changes as the number of dimensions is reduced. Separate lines are shown for different layers of the language model, indicating how the clustering quality varies across layers.

• Methodology for silhouette score calculation and choice of LDA: Using the silhouette score to assess clustering quality is a standard practice. However, the authors should clarify the specific clustering algorithm used to compute the silhouette score. What distance metric was employed? These details are crucial for reproducibility. They should also justify the choice of LDA as the dimensionality reduction technique and consider alternative methods, such as PCA or t-SNE.
• Significance of the trend and implications for crystal structure identification: The authors should explain the significance of the observed trend. Why does the silhouette score vary with reduced dimension and across layers? How does this relate to the identification of "crystal structures"? They should also investigate the optimal number of dimensions for clustering based on the silhouette score and discuss the implications of their findings for the broader research question.

• Clarity and informativeness of the plot: The plot effectively communicates the trend of silhouette score with reduced dimension. The use of separate lines for different layers is helpful. However, the plot could be improved by adding a clear indication of the peak silhouette score for each layer, perhaps with a marker or label. The x-axis label could be more descriptive (e.g., "Reduced Dimension"). The caption could also benefit from a brief explanation of what the silhouette score represents and its range.
• Context and purpose in caption: The caption clearly states what the plot shows: silhouette score as a function of reduced dimension in LDA. However, it lacks context. What is the purpose of analyzing the silhouette score? How does it relate to the research question? Adding a brief explanation, such as "...to assess the impact of dimensionality reduction on crystal structure identification...",

Figure 2. Features in the SAE point cloud identified that tend to fire together within documents are seen to also be geometrically co-located in functional "lobes", here down-projected to 2D with t-SNE with point size proportional to feature frequency.

• Purpose and visualization method: This figure visualizes the spatial organization of features learned by a Sparse Autoencoder (SAE) from a large language model. Think of the SAE as a system that learns to represent the meaning of text by activating different combinations of features. Each feature can be thought of as a point in a high-dimensional space. This figure explores whether features that frequently activate together when processing the same document are also close to each other in this space. To visualize this high-dimensional space, the authors use t-SNE, a technique that projects the points onto a 2D plane while trying to preserve their relative distances. The size of each point represents how often that feature is activated across a dataset of documents. The figure shows these points grouped into "lobes," suggesting a spatial organization where functionally related features cluster together.
• Key terms: point cloud, firing together, lobes: The "point cloud" refers to the set of all features learned by the SAE, each represented as a point in a high-dimensional space. "Firing together" means that these features are frequently activated simultaneously when the SAE processes the same document. The figure investigates whether this functional co-activation corresponds to spatial proximity in the feature space. The "lobes" are visually distinct clusters of points in the 2D t-SNE projection, suggesting that functionally related features tend to be located near each other.

• Quantifying spatial localization and statistical significance: While visually suggestive, the claim of spatial localization needs more rigorous support. The authors should quantify the degree of clustering using appropriate metrics, such as silhouette scores or modularity measures. They should also compare the observed clustering to a null model, such as randomly shuffled feature assignments, to demonstrate statistical significance. Relying solely on visual inspection is not sufficient to establish the presence of meaningful structure.
• Justification for co-occurrence and sensitivity analysis: The choice of co-occurrence as a measure of functional similarity should be justified. Do features that co-occur within a document necessarily represent related concepts? The authors should explore alternative measures of functional similarity, such as semantic similarity based on word embeddings, and compare the resulting spatial organization. They should also investigate the sensitivity of the results to the chosen document size and context window for co-occurrence calculation.

• t-SNE limitations and labeling: The figure's use of t-SNE for dimensionality reduction is appropriate for visualization, but its limitations should be acknowledged. t-SNE can distort distances and create artificial clusters. The authors should clarify that the observed spatial localization is suggestive and requires further investigation with more robust methods. Labeling the lobes directly on the plot, rather than relying solely on the caption, would improve clarity. A brief explanation of the color scheme in the caption would also be beneficial.
• Clarity and precision of language: The caption clearly conveys the main finding: features that activate together are spatially clustered. The use of "lobes" as an analogy to brain regions is effective in conveying the idea of functional modularity. However, the caption could be more specific about the meaning of "fire together." Does it refer to co-occurrence within the same document, sentence, or some other unit of text? Clarifying this would enhance the precision of the message.

Figure 3. Top Left: Adjusted mutual information between spatial clusters and functional (cooccurrence-based) clusters.

• Purpose of the graph and meaning of AMI: This graph, specifically the top-left panel, shows how well two different ways of grouping features of a language model agree with each other. One way, creating "spatial clusters," groups features based on their geometric locations in a high-dimensional space, like points clustered on a map. The other way, creating "functional clusters," groups features based on how often they activate together when processing a document. Imagine if words like "king" and "queen" often appeared in the same texts, they would be in the same functional cluster. The graph uses a metric called Adjusted Mutual Information (AMI) to measure the agreement between these two groupings. A higher AMI means better agreement, suggesting that features close in space also tend to function similarly.
• Explanation of axes and different lines: The x-axis represents the number of clusters created in either the spatial or functional grouping (needs clarification). The y-axis represents the AMI score, which ranges from 0 to 1, with higher values indicating better agreement. Different lines on the graph correspond to different methods of calculating functional clusters based on co-occurrence, such as the Phi coefficient, which measures the correlation between two features' activations.

• Methodology for spatial clustering: Using AMI to compare clusterings is a standard and valid approach. However, the authors should clarify how the spatial clusters were generated. What clustering algorithm was used? What distance metric was employed in the high-dimensional space? These details are crucial for reproducibility and interpretation of the results.
• Statistical significance and alternative metrics: The claim that the Phi coefficient "wins" needs further justification. Is the difference in AMI scores statistically significant? The authors should perform statistical tests to compare the different co-occurrence measures and report the p-values. They should also consider other evaluation metrics beyond AMI, such as the Fowlkes-Mallows index or the Rand index, to provide a more comprehensive assessment.

• Clarity and completeness of caption: The caption clearly states what the graph depicts: adjusted mutual information between two types of clusters. However, it lacks context. It should specify what the 'spatial' and 'functional' clusters represent. For instance, are the spatial clusters based on feature vector geometry and functional clusters based on co-occurrence within documents? Adding this information would make the caption self-sufficient.
• Clarity of axis labels and data presentation: The plot itself is clear, with well-labeled axes. The use of different colors for different co-occurrence measures is helpful. However, the x-axis label "Number of Clusters" could be more informative. Does it refer to the number of spatial clusters, functional clusters, or both? Clarifying this would prevent ambiguity. Also, indicating the peak AMI value for the Phi coefficient directly on the graph would enhance readability.

Figure 4. Fraction of contexts in which each lobe had the highest proportion of activating features.

• Overall purpose and visualization method: This figure shows how different "lobes" of features in a language model are activated across various types of documents. Imagine the model's "brain" is divided into specialized regions ("lobes"), each responsible for understanding different aspects of language. This figure explores which lobes are most active when processing different kinds of text, like scientific papers, code, or news articles. The figure uses a heatmap, where each row represents a document type (e.g., Wikipedia articles, code from GitHub) and each column represents a lobe. The color intensity in each cell indicates how often that lobe was the most active one when processing a chunk of text from that document type. Brighter colors mean higher activation.
• Explanation of lobes and contexts: The "lobes" are groups of features that tend to activate together, identified using a method based on the Phi coefficient, a measure of correlation. "Contexts" refer to short segments of text within a document. For each context, the figure determines which lobe has the highest proportion of its features activated. The heatmap then aggregates these results across many contexts within each document type, showing the overall preference of each lobe for different types of text.

• Justification for context length and alternative methods: The methodology of identifying lobes based on the Phi coefficient and then analyzing their activation across document types is sound. However, the choice of 256-token contexts should be justified. Is this length optimal for capturing the relevant semantic information? The authors should explore different context lengths and assess their impact on the results. They should also consider alternative methods for identifying functional modules, such as clustering based on semantic similarity, and compare them to the co-occurrence-based approach.
• Statistical significance of lobe specialization: The authors should quantify the statistical significance of the observed associations between lobes and document types. For example, they could compare the observed activation patterns to a null model where lobes are randomly assigned to features. This would help determine whether the observed specialization of lobes is statistically significant or simply due to chance.

• Clarity and informativeness of visualization: The heatmap effectively visualizes the association between lobes and document types. The color intensity clearly represents the activation strength, making it easy to identify which lobes are dominant in each document type. However, the labels for document types could be more descriptive. For instance, instead of "Pile-CC," specifying "Common Crawl" would be more informative. Similarly, clarifying abbreviations like "USPTO" would improve accessibility. Adding a colorbar to explicitly map colors to activation fractions would further enhance clarity.
• Completeness and context in caption: The caption is concise but lacks detail. It states what the heatmap represents but doesn't explain what "contexts" are or how they relate to documents. Adding a brief explanation, such as "...fraction of 256-token blocks within each document type..." would improve understanding. The caption should also mention the use of the Phi coefficient for lobe computation, as this is crucial for interpreting the results.

Figure 5. Comparison of the lobe partitions of the SAE point cloud discovered with different affinity measures, with the same t-SNE projection as Figure 2.

• Purpose of the figure and meaning of affinity measures: This figure compares different ways of grouping features in a language model's representation space into "lobes." Imagine the model's "brain" is divided into specialized regions, and we're trying to find the best way to define these regions. Each subplot in the figure shows a different way to group the features, using different methods for measuring how similar features are to each other. These methods, called "affinity measures," determine which features are grouped together into lobes. The figure uses t-SNE, a technique to visualize high-dimensional data in 2D, to show the spatial organization of these lobes. The same t-SNE projection is used for all subplots, allowing for direct comparison of the different grouping methods.
• Explanation of SAE point cloud, lobe partitions, and affinity measures: The "SAE point cloud" refers to the set of all features learned by a Sparse Autoencoder (SAE), each represented as a point in a high-dimensional space. "Lobe partitions" are different ways of dividing this point cloud into clusters ("lobes") based on how similar the features are. The figure explores how different affinity measures, which quantify feature similarity, affect the resulting lobe structure. The different affinity measures likely include variations on how often features activate together when processing the same text.

• Quantifying differences between partitions and justifying affinity measure choices: Comparing different affinity measures is a valid approach for understanding the robustness of the lobe structure. However, the authors should quantify the differences between the partitions using appropriate metrics, such as variation of information or the Rand index. Visual comparison alone is not sufficient to draw strong conclusions. They should also justify the choice of the specific affinity measures used and explain why these measures are relevant to the research question.
• Addressing t-SNE limitations and exploring alternative methods: While using the same t-SNE projection facilitates comparison, the authors should acknowledge the limitations of t-SNE. As t-SNE can distort distances and create artificial clusters, the observed spatial organization should be interpreted with caution. The authors should consider using additional dimensionality reduction techniques, such as UMAP or PCA, to verify the robustness of the lobe structure.

• Clarity and consistency of visualization: Using the same t-SNE projection across all subplots facilitates direct comparison of the different affinity measures. The consistent color scheme and layout make it easy to see the similarities and differences between the resulting lobe partitions. However, labeling each subplot directly with the corresponding affinity measure would improve clarity and avoid the need to constantly refer back to the caption. Additionally, a brief explanation of the color scheme (e.g., what do different colors represent?) would be beneficial.
• Informativeness and context in caption: The caption clearly explains the figure's purpose: comparing lobe partitions based on different affinity measures. The reference to "the same t-SNE projection as Figure 2" provides helpful context. However, the caption could be more informative by briefly mentioning what types of affinity measures are being compared (e.g., "...different co-occurrence-based affinity measures...").

Figure 10. Plot of the first principal component in the difference space as a function of last token length difference in Gemma-2-2b layer 0.

• Purpose of the figure and explanation of PCA: This figure investigates the factors that influence the first principal component of the feature representations learned by a language model. Imagine each word represented as a point in a high-dimensional space. Principal Component Analysis (PCA) finds the directions of greatest variation in this space. The first principal component is the direction along which the data varies the most. This figure specifically examines how the first principal component relates to the difference in length between the last tokens of two words. It visualizes this relationship for the initial layer (Layer 0) of the Gemma-2-2b language model.
• Explanation of plot and difference space: The plot shows the first principal component value on the y-axis and the difference in the last token length between two words on the x-axis. Each point in the scatter plot represents a pair of words. The "difference space" likely refers to the space of pairwise differences between word representations. The figure aims to demonstrate that the first principal component is strongly influenced by the difference in the length of the last tokens.

• Quantifying the relationship and exploring other layers/positions: Analyzing the relationship between the first principal component and word length is a valid approach for understanding the factors that influence feature representations. However, the authors should quantify the strength of this relationship using a correlation coefficient or regression analysis. Simply stating that the first principal component "encodes mainly" length difference is not sufficient. They should also investigate whether this relationship holds for other layers of the model and other token positions besides the last token.
• Relevance to research question and controlling for word length: The authors should explain why this finding is relevant to the broader research question. Does the influence of word length have implications for the interpretation of the "brain scale" structure? They should also consider controlling for word length in their analysis of functional lobes to isolate the effects of semantic similarity.

• Clarity and informativeness of the plot: The scatter plot clearly visualizes the relationship between the first principal component and the last token length difference. The linear trend is apparent, supporting the authors' claim. However, adding a regression line to the plot would quantify the strength of the relationship and improve the visual representation. Including axis labels with units (e.g., "PC1 Value", "Length Difference (characters)") would further enhance clarity. The caption could also benefit from a brief explanation of what "difference space" refers to.
• Context and significance in caption: The caption clearly describes the plot's content. However, it lacks context. Why is this relationship important? How does it relate to the broader research question? Adding a brief explanation, such as "...demonstrating that word length is a significant factor in the first principal component...",

Figure 6. 3D Point Cloud visualizations of top PCA components for the Gemma2-2b layer 12 SAE features.

• Purpose of visualization and explanation of PCA: This figure shows a 3D representation of how features learned by a Sparse Autoencoder (SAE) are distributed. Imagine each feature as a point in a very high-dimensional space. Since we can't visualize such a high-dimensional space directly, the authors use Principal Component Analysis (PCA), a method for reducing dimensionality, to project these points onto a 3D space. PCA finds the directions of greatest variance in the data, and the figure shows the points along the three most important directions. This visualization helps explore the overall "shape" of the feature distribution and whether it resembles a simple cloud or has more complex structure.
• Explanation of point cloud, Gemma2-2b, and SAE features: The "point cloud" refers to the set of all features learned by the SAE, each represented as a point. "Gemma2-2b layer 12" specifies the particular language model and layer being analyzed. "SAE features" are the learned representations of concepts or words. The 3D visualization shows these features projected onto the first three principal components, which capture the most significant variations in the data. The color of each point represents its "magnitude," which could be related to the activation strength or frequency of the feature.

• Quantifying deviation from isotropy and statistical testing: Visualizing the point cloud after PCA is a reasonable first step in exploring its structure. However, it's crucial to go beyond visual inspection and quantify the deviation from isotropy. The authors should calculate the eigenvalues of the covariance matrix and compare them to the expected distribution for an isotropic Gaussian. They should also perform statistical tests, such as Bartlett's test of sphericity, to assess the significance of the deviation.
• Justification for using top three components and exploring alternatives: The choice of the top three principal components should be justified. Do these components capture a sufficient amount of variance in the data? The authors should report the explained variance ratio for each component. They should also consider visualizing the point cloud along other combinations of principal components or using alternative dimensionality reduction techniques to ensure that the observed structure is not an artifact of the chosen projection.

• Clarity and interpretability of 3D visualization: Visualizing the point cloud in 3D helps illustrate the non-isotropic nature, but it's difficult to discern fine-grained details. While the color-coding based on point magnitude adds some information, it's unclear what insights are gained from it. Labeling the axes with the corresponding principal components (e.g., PC1, PC2, PC3) would improve clarity. Additionally, providing a brief explanation in the caption about what the point cloud represents (e.g., "Each point represents an SAE feature") would enhance understanding.
• Connecting visualization to research question: The caption clearly states what is being visualized: a 3D point cloud of SAE features projected onto the top principal components. However, it lacks context. What is the purpose of this visualization? What is the significance of examining the point cloud structure? Connecting the visualization to the research question (e.g., "...to assess the isotropy of the feature distribution...") would make the caption more informative.

Figure 7. Eigenvalues of the point cloud are seen to decay as an an approximate power law (left), whose slope depends on layer (right) and is strongly inconsistent with sampling from an isotropic Gaussian distribution.

• Purpose of the figure and meaning of eigenvalues: This figure analyzes the distribution of features learned by a language model by examining the eigenvalues of the covariance matrix of the feature vectors. Imagine each feature as a point in a high-dimensional space. The covariance matrix describes how these features vary together. Its eigenvalues represent the extent of variation along different directions in this space. If the features were distributed like a simple spherical cloud (isotropic Gaussian), the eigenvalues would decay smoothly. However, the figure shows that the eigenvalues decay approximately as a power law, indicating a different, non-isotropic distribution. The figure also shows how the steepness of this power-law decay changes across different layers of the language model.
• Explanation of plots and layer dependence: The left plot shows the eigenvalues plotted against their rank on a log-log scale. A straight line on a log-log plot indicates a power law. The right plot shows how the slope of this power law, which represents the steepness of the decay, changes with the layer of the language model. The layers refer to different processing stages within the model. The figure also shows the "effective point cloud volume," which is a measure of the spread of the feature distribution, and how it varies across layers.

• Quantifying goodness of fit and considering alternative distributions: Analyzing the eigenvalue spectrum is a valid approach for assessing the isotropy of the feature distribution. The comparison to an isotropic Gaussian provides a meaningful baseline. However, the authors should quantify the goodness of fit for the power-law approximation. They should also consider alternative distributions, such as heavy-tailed distributions, and compare their fit to the observed eigenvalue decay. Simply stating "strongly inconsistent" is not sufficient; statistical tests should be performed to assess the significance of the deviation from the Gaussian distribution.
• Investigating layer dependence and practical implications: The observation of a layer-dependent slope is interesting and warrants further investigation. The authors should explore the reasons behind this variation. Is it related to the specific architecture of the language model or the nature of the features learned at different layers? They should also analyze the relationship between the eigenvalue spectrum and the performance of the model on different tasks to assess the practical implications of the observed non-isotropy.

• Clarity and informativeness of plots: Presenting the eigenvalue decay and layer-dependent slope in separate plots is a good choice. The log-log scale on the left plot is appropriate for visualizing power-law behavior. However, the right plot could be improved. Instead of plotting just the slopes, showing the actual power-law fits for different layers would be more informative. Adding a legend to the left plot to identify the different layers would also enhance clarity. The right plot's y-axis label could be more descriptive (e.g., "Power-law exponent") and indicate clearly what is being plotted. Finally, the caption could benefit from a brief explanation of what the "effective point cloud volume" represents.
• Precision and context in caption: The caption clearly states the main findings: power-law decay of eigenvalues and layer-dependent slope. The reference to the isotropic Gaussian distribution provides context. However, the caption could be more precise by specifying what "layer" refers to (e.g., "...layer of the language model..."). Also, briefly mentioning the implication of the power-law decay (e.g., "...suggesting a non-isotropic distribution of features...") would enhance the message.

Figure 8. Estimated clustering entropy across layers with 95% confidence intervals.

• Purpose of the figure and meaning of clustering entropy: This figure measures how clustered the features learned by a language model are at different stages of processing. Imagine the features as points in a high-dimensional space. If the points are randomly scattered, the entropy is high. If the points form tight clusters, the entropy is low. This figure calculates the "clustering entropy," which quantifies the degree of clustering, for each layer of the language model. The layers represent different processing stages, from the initial input to the final output. The figure shows how the clustering entropy changes across these layers.
• Explanation of axes and confidence intervals: The x-axis represents the layer of the language model. The y-axis represents the clustering entropy. The plot shows the estimated clustering entropy for each layer, along with 95% confidence intervals. These intervals indicate the range within which the true clustering entropy is likely to fall, given the estimated value. The plot helps visualize how the degree of feature clustering varies across different processing stages in the model.

• Methodology for clustering entropy calculation: Using clustering entropy to quantify the degree of clustering is a valid approach. However, the authors should clearly define how the clustering entropy is calculated. What clustering algorithm is used? What distance metric is employed? These details are crucial for reproducibility. They should also justify the choice of the k-NN method for entropy estimation and consider alternative methods.
• Investigating the trend and its implications: The observed trend of reduced clustering entropy in middle layers is interesting, but requires further investigation. The authors should explore the reasons behind this trend. Is it related to the specific architecture of the language model or the nature of the features learned at different layers? They should also analyze the relationship between clustering entropy and the performance of the model on different tasks to understand the practical implications of the observed trend.

• Clarity and informativeness of the plot: The plot clearly shows the trend of clustering entropy across layers. The inclusion of confidence intervals is crucial for assessing the reliability of the estimates. However, the y-axis label could be more informative. While "Clustering Entropy" is technically correct, briefly explaining what it represents (e.g., "Measure of feature clustering") would make the plot more accessible. Adding a horizontal line representing the entropy of a uniform distribution would provide a useful visual reference point.
• Context and significance in caption: The caption is concise and accurately describes the plot. However, it could be improved by providing more context. What is the significance of the observed trend? How does it relate to the research question? Adding a brief explanation, such as "...indicating reduced clustering in middle layers...",

Figure 9. Histogram, over all features, of Phi coefficient with k-th nearest cosine similarity neighbor.

• Purpose of the figure and explanation of Phi coefficient and cosine similarity: This figure explores the relationship between two ways of measuring the similarity between features learned by a language model. One way uses the Phi coefficient, which measures how often two features activate together when processing the same text. The other way uses cosine similarity, which measures the angle between the feature vectors in a high-dimensional space. Imagine each feature as a point in this space; points close together have high cosine similarity. The figure creates histograms of the Phi coefficient between each feature and its k-th nearest neighbor based on cosine similarity. This helps understand if features that are close in space also tend to activate together.
• Explanation of histograms and k-th nearest neighbor: The histograms show the distribution of Phi coefficient values for different values of k, where k represents the number of nearest neighbors considered. For example, k=1 corresponds to the nearest neighbor, k=2 corresponds to the second nearest neighbor, and so on. The x-axis represents the Phi coefficient, and the y-axis represents the frequency of features with that Phi coefficient value. The "random" baseline represents the expected distribution if there were no relationship between co-occurrence and cosine similarity.

• Methodology for random baseline and statistical testing: Analyzing the relationship between co-occurrence and cosine similarity is a valid approach for understanding the structure of feature representations. The use of a "random" baseline is helpful for assessing the significance of the observed distributions. However, the authors should clarify how the "random" baseline is generated. Is it based on randomly shuffled features or a theoretical distribution? They should also quantify the difference between the observed distributions and the random baseline using statistical tests, such as the Kolmogorov-Smirnov test.
• Justification for Phi coefficient and sensitivity analysis: The choice of the Phi coefficient as a measure of co-occurrence should be justified. Are there alternative measures that might be more appropriate? The authors should also explore the sensitivity of the results to the choice of k. Does the relationship between co-occurrence and cosine similarity change significantly with different values of k? They should also consider using other distance metrics besides cosine similarity.

• Clarity and informativeness of histograms: Presenting separate histograms for Layer 0 and Layer 12 is helpful for comparing the distributions. The inclusion of a "random" baseline provides a useful reference point. However, the plot could be improved by clarifying the meaning of k. Adding a brief explanation in the caption or labels (e.g., "k = number of nearest neighbors") would enhance understanding. Also, labeling the peaks of the distributions with the corresponding k values would improve readability. The x-axis label could be more descriptive (e.g., "Phi Coefficient with k-th Nearest Neighbor").
• Context and purpose in caption: The caption clearly states what the histograms represent. However, it lacks context. What is the purpose of analyzing the Phi coefficient distribution? How does it relate to the research question? Adding a brief explanation, such as "...to assess the relationship between feature co-occurrence and cosine similarity...",

Figure 12. Smoothed PCA scores for each SAE feature of the layer 12, width 16k, Lo = 176 Gemma Scope 2b SAE, sorted by frequency.

• Purpose of the figure and explanation of PCA scores: This figure analyzes how the features learned by a Sparse Autoencoder (SAE) are distributed across the principal components of the activation space. Imagine each feature as a point in a high-dimensional space. Principal Component Analysis (PCA) finds the directions of greatest variance in this space. Each feature can be assigned a "PCA score," which represents its weighted position along these principal components. This figure plots the smoothed PCA scores for each feature in a specific layer of the Gemma Scope 2b SAE model. The features are sorted by their frequency of activation, meaning how often they are "turned on" when processing text.
• Explanation of axes, smoothing, and the observed dip: The x-axis represents the index of the SAE feature, sorted by decreasing frequency. The y-axis represents the smoothed PCA score. The plot shows how the PCA scores vary across features. Different lines correspond to different smoothing windows, which are used to average the scores and reveal the underlying trend. The figure highlights a "dip" in PCA scores towards the end, suggesting that less frequent features have different PCA score characteristics.

• Justification for smoothing windows and feature sorting, and comparison to baseline: Analyzing PCA scores can provide insights into the structure of feature representations. The use of smoothing is appropriate for visualizing trends in noisy data. However, the authors should justify the choice of smoothing windows. Are the observed patterns robust to different window sizes? They should also explain the significance of sorting features by frequency. Does this sorting reveal any meaningful patterns? They should also compare the observed distribution of PCA scores to a random baseline or a theoretical distribution.
• Significance of the observed dip and its implications for galaxy scale structure: The authors should explain the implications of the observed dip in PCA scores. What does it mean for the less frequent features to have different PCA score characteristics? How does this relate to the "galaxy scale" structure of the point cloud? They should also investigate whether this dip is specific to layer 12 or if it occurs in other layers as well. They should also explore the relationship between PCA scores and the performance of the model on different tasks.

• Clarity and informativeness of the plot: The plot clearly shows the trend of smoothed PCA scores across features. The inclusion of different smoothing windows helps visualize the underlying pattern. However, the plot could be improved by clarifying the meaning of "PCA score." The formula provided in the caption is helpful, but a brief intuitive explanation would enhance understanding for a broader audience. Also, the x-axis label could be more informative (e.g., "SAE Feature Index (Sorted by Frequency)"). The caption could also benefit from a brief explanation of why the features are sorted by frequency and the significance of the observed dip.
• Context and purpose in caption: The caption provides detailed information about the specific SAE model and parameters. However, it lacks context. What is the purpose of analyzing PCA scores? How does it relate to the research question? Adding a brief explanation, such as "...to investigate the distribution of SAE features across principal components...",