AI Geometric Thinking: From Tiny Crystals to Whole Galaxies

AI’s Geometric Worlds

Artificial Intelligence (AI) doesn’t just learn words – it builds entire geometric worlds. It turns out, AI thinks in shapes – from tiny crystals to whole galaxies. Large Language Models (LLMs) internal concepts form geometric patterns similar to the human brain’s organization. LLMs organize knowledge in a three-tiered architecture: atomic, neural, and cosmic scales.

Understanding Sparse Autoencoders (SAE)

The original problem with Sparse Autoencoders (SAE) was that while they discovered interpretable features in LLMs, we didn’t understand how these features were organized in the high-dimensional space and what patterns they formed. This paper provides a solution by analyzing the SAE feature space at three distinct scales:

Atomic scale: Studied parallelogram/trapezoid structures (like man:woman::king:queen).
Brain scale: Identified functional ‘lobes’ where similar features cluster.
Galaxy scale: Examined large-scale point cloud structure.

Key Insights

Here are some key insights from the analysis:

SAE features form crystal-like structures at small scales, but these are hidden by irrelevant dimensions.
Features that fire together in documents are geometrically co-located, forming brain-like functional lobes.
Middle layers show steeper power-law slopes (-0.47) compared to early/late layers (-0.24/-0.25).
Distinct lobes were found for code/math, short messages/dialogue, and scientific papers.

Results

The results of the study are significant:

Linear Discriminant Analysis (LDA) significantly improved parallelogram/trapezoid structure quality.
The Phi coefficient showed the best correspondence between functional and geometric clustering.
Middle layers demonstrated reduced clustering entropy, suggesting more concentrated feature representation.
Statistical significance: 954σ for mutual information, 74σ for logistic regression tests.