Geometric Memory III: Resonant Optimization, Consensus Distillation, and Evolutionary Training Paradigms

AbstractPhil March 2026

---

Abstract

We present a geometric optimization framework that replaces weight decay with manifold-aware gradient filtering, achieving +15% accuracy over standard cross-entropy on a 30-class shape classification benchmark. The optimizer combines three mechanisms: tangential projection constraining gradients to the hypersphere surface, separation preservation preventing cluster collapse, and a differentiable Cayley-Menger CV loss maintaining pentachoron volume regularity at micro weights (1e-3). We demonstrate that Adam with geometric filtering consistently outperforms AdamW — weight decay is uniform damping that destroys the geometric harmonic the autograd creates, establishing that the geometry IS the regularization. We introduce dual-teacher Procrustes consensus distillation, where two independently-trained models are GPA-aligned and their geometric center is distilled into a student that exceeds both teachers (0.761 vs 0.699/0.649) while still accelerating at epoch 30. We extend this to multi-generational geometric evolution — a training paradigm where each generation inherits consensus-derived anchor coordinates from its ancestors and introduces fresh variation through new founders and data perturbation. Over 5 generations with data-diverse training, final models achieve 0.830 accuracy from founders averaging 0.664, with the consensus manifold converging toward CV=0.2 naturally. We establish that the architecture ceiling is data-limited, not optimizer-limited: 5× data raised the ceiling from 0.78 to 0.83, and distilled models still outperform raw baselines at equal data volume. We report the discovery of resonant dynamics in geometric training — constructive interference compounding across epochs rather than dissipating, a phenomenon incompatible with standard optimization theory.

---

1. Introduction

Parts I and II established geometric memory as a practical system: frozen encoders extended with memory banks, regularized by pentachoron CV, aligned through Procrustes rotation. That work focused on the architecture — what to build. This work addresses the optimizer — how to train.

Standard optimizers treat all gradient components equally. AdamW applies uniform weight decay regardless of manifold structure. SGD with momentum accumulates velocity without regard for the hypersphere constraint. These optimizers were designed for unconstrained parameter spaces. Geometric embedding spaces are constrained — embeddings live on hyperspheres, anchors define coordinate systems, pentachoron volumes encode manifold regularity. An optimizer that respects these constraints should outperform one that ignores them.

We ask three questions:

1. Can gradient filtering replace weight decay? We demonstrate that tangential projection + separation preservation + micro CV loss provide equivalent or superior regularization to weight decay, without the uniform damping that destroys geometric structure.

2. Can weak models produce strong students? We show that Procrustes consensus between two models averaging 0.664 accuracy produces offspring at 0.761 — the geometric center of independently-trained models is more accurate than either model alone.

3. Does multi-generational training compound? We demonstrate that iterating the consensus-distillation process across generations with data diversity produces monotonically improving models, with each generation inheriting the geometric agreement of its ancestors.

---

2. The Geometric Autograd

2.1 Architecture

The optimizer operates at two boundaries in the computation graph: the embedding output and the anchor parameters. At each boundary, gradient filtering removes destructive components while preserving constructive signal.

Embedding backward (tangential projection + separation): Given gradient g at embedding e on the unit hypersphere, decompose g into tangential and radial components relative to e. The tangential component slides the embedding along the manifold surface. The radial component pushes it off the surface. The optimizer passes the tangential component fully and attenuates the radial component by factor (1 - tang_strength). Additionally, if the gradient would move the embedding toward its nearest anchor (collapse), the collapse component is attenuated by sep_strength.

Anchor backward (drift guard): Each anchor's gradient is independently projected tangential to the hypersphere at that anchor's position. Anchors can slide along the surface but never drift off it. The drift_strength parameter scales how much gradient passes through, preserving the Xavier initialization's near-orthogonality while allowing adaptation.

Forward losses (differentiable, proper gradient flow):

All forward losses are fully differentiable. Gradients flow naturally through torch.stack, torch.sqrt, torch.linalg.det. No manual gradient injection. The critical architectural decision: CV regulation is a forward loss, never backward surgery. Injecting equidistance corrections in backward collapsed the embedding space to random chance (3.3% accuracy on 30 classes) because the injected gradients dominated Adam's momentum and learning rate schedule. A loss term works WITH the optimizer; injection works AGAINST it.

2.2 The CV Loss

The differentiable CV loss uses the production Cayley-Menger determinant:

pts → pairwise distances² → bordered matrix →
det × (-1)^V / (2^(V-1) × (V-1)!²) → volume² →
sqrt(relu + ε) → stack samples → std/mean → |cv - target|

The generic formula handles any vertex count V, unlike hardcoded divisors (e.g., /9216 for V=5) that appeared in early prototypes. The target is 0.2, empirically validated across 17+ models from the Procrustes analysis survey (Part I) as the universal CV band for pretrained embedding spaces.

The weight MUST be micro — 0.001 or below. At higher weights, the CV loss dominates cross-entropy and optimizes for geometric regularity at the expense of discriminative capacity. At 0.001, the presence of the constraint matters more than its magnitude. Over millions of sample-touches, the micro correction compounds into manifold-level regularization without taxing per-batch classification performance.

2.3 Validated Parameters

Extensive sweeps established the optimal configuration:

2.4 Adam vs AdamW: The Geometry IS the Regularization

The most significant finding: Adam with geometric autograd consistently outperforms AdamW.

Weight decay applies uniform damping to all parameters — constellation anchors, projection matrices, patchwork MLPs — regardless of their geometric role. The geometric autograd applies SELECTIVE damping: it attenuates destructive gradient components (radial, collapsing) while passing constructive components (tangential, separating). Uniform damping destroys the geometric harmonic. Selective filtering preserves it.

Weight decay exists to prevent overfitting through unconstrained weight growth. The geometric autograd prevents overfitting through manifold constraint. They solve the same problem through incompatible mechanisms. Using both creates a regulatory conflict where weight decay shrinks the structure the gates are protecting.

---

3. The Patchwork Architecture

3.1 Constellation and Triangulation

The constellation is a set of N anchors on the unit hypersphere, initialized with Xavier normal distribution (near-orthogonal in high dimensions). Each input embedding is triangulated against all anchors, producing a distance vector of cosines to each anchor.

Critically, anchors are NOT classes. They are geometric reference points — abstract coordinates in the embedding space. Multiple classes can share anchor neighborhoods. A class can span multiple anchors. The number of anchors is independent of the number of classes. The classifier learns the mapping from triangulation coordinates to class labels through the patchwork + MLP pipeline.

This distinction matters for rigidity tracking: anchor rigidity is measured by nearest-anchor assignment (which embeddings land near which anchor), not by class label. In early prototypes, rigidity was tracked with labels == i, hardcoding a 1:1 anchor-class mapping that limited architectural flexibility. The corrected version uses tri_dist.argmin(dim=-1) to assign by geometric proximity.

3.2 Compartmentalized Patchwork

The patchwork partitions N anchors into K compartments via interleaved assignment (anchor i belongs to compartment i % K). Each compartment has its own MLP processing the triangulation distances for its assigned anchors:

Compartment 0: anchors [0, K, 2K, ...]  → MLP → (B, d_comp)
Compartment 1: anchors [1, K+1, 2K+1, ...] → MLP → (B, d_comp)
...
Compartment K-1: anchors [K-1, 2K-1, 3K-1, ...] → MLP → (B, d_comp)
→ concatenate → (B, K × d_comp)
→ funnel MLP → (B, n_classes)

The interleaved assignment ensures every compartment sees a cross-section of the constellation — anchors from different geometric regions. Each compartment learns the relationships WITHIN its cross-section. The funnel MLP learns how compartments relate to each other.

3.3 Anchor Count Independence

We tested 30, 240, and 1024 anchors across embedding dimensions of 768 and 256. All configurations converged to the same accuracy ceiling (~0.78 on 15K training samples). The geometric evolution pipeline is robust to anchor count — the consensus alignment extracts the same invariant structure regardless of coordinate system resolution. At 1024 anchors in 256-d embedding space, k-means initialization on consensus embeddings replaced class-centroid initialization, fully decoupling anchors from class labels.

---

4. Dual-Teacher Consensus Distillation

4.1 The Pipeline

Two teachers trained independently on the same data with different configurations:

Both teachers encode the full training set. Generalized Procrustes Analysis (GPA) iteratively aligns their embedding spaces to find the geometric center:

for each iteration:
    mean_shape = average of all aligned embeddings
    for each model:
        Procrustes-rotate model embeddings toward mean_shape
    until convergence (delta < 1e-8)
consensus = L2_normalize(mean_shape)

The consensus is the geometric truth — what both teachers agree on after removing their individual rotational frames. It is more regular than either individual model (consensus CV=0.18 vs teacher CVs of 1.4+).

4.2 Student Training

The student model initializes its constellation anchors from per-class centroids of the consensus embeddings (or k-means for class-decoupled anchors). Training combines:

Plus geometric autograd backward filtering (tang=0.01, sep=1.0).

4.3 Results

The student exceeded both teachers in every category. Stars improved from 0.83 to 0.97. Structure improved from 0.72 to 0.76. CV dropped from 1.4+ to 0.33 — the manifold converged toward the 0.2 target naturally through consensus distillation.

The trajectory is the critical observation:

The teachers plateaued by epoch 15. The student was still accelerating at epoch 30. This is incompatible with standard optimization dynamics where late training decelerates. The geometric autograd creates the conditions for constructive interference — each epoch's geometric improvement makes the next epoch's consensus distillation more precise, producing better gradients, which improve the geometry further. This is resonance.

---

5. Multi-Generational Geometric Evolution

5.1 The Paradigm

We extend consensus distillation to multiple generations:

Generation 0: N founders trained independently with varied configurations. GPA consensus computed. Consensus anchors extracted.

Generation 1: M offspring distilled from Generation 0 consensus. Each offspring uses a different training configuration (variation). A new founder is introduced — fresh random initialization that never saw any consensus (immigration).

Generation 2+: Offspring from previous generation + new founder → GPA → consensus anchors → next generation offspring.

The pipeline has direct analogs to evolutionary mechanisms:

5.2 Data-Diverse Evolution

Each generation trains on differently-perturbed data, so the consensus captures what's INVARIANT across perturbations:

Validation is always on Dataset A — consistent evaluation regardless of training perturbation.

5.3 Generation-by-Generation Results

Gen 0 (2 founders):           mean=0.664  best=0.675
Gen 1 (2 offspring + founder): mean=0.550  best=0.719
Gen 2 (3 offspring + founder): mean=0.750  best=0.754
Gen 3 (5 offspring):           mean=0.742  best=0.773
Gen 4 (3 triplets):            mean=0.765  best=0.775

The generational improvement is monotonic from Generation 0 onward. Generation 1's mean is depressed by a catastrophically trained model (G1_B at 0.255-0.334 across runs) — trained on Dataset B with thick noisy strokes, it failed to generalize to the standard validation set. Yet this model contributed to the consensus, and the lineage recovered completely.

5.4 Robustness to Catastrophic Models

The catastrophically trained G1_B is the robustness proof. Across multiple runs:

Stars at literally 0%. Worse than random guessing. And the lineage still climbed to 0.77+. Procrustes alignment doesn't concentrate errors — it CANCELS them. What survives consensus across independently-initialized models isn't a defect. It's signal. The noise is individual-specific. The signal is what's shared. Even a catastrophic model contributes the small fraction of geometric truth it accidentally discovered.

5.5 Parent Selection Strategy

Three strategies were compared for the final generation:

The diverse strategy consistently matched or beat the best5 strategy. Selecting parents for maximum SPREAD across the ranking provides more independent geometric perspectives than selecting the top performers, who may share similar biases.

---

6. The Data Ceiling

6.1 Fusion Experiment

The final experiment combined all 5 datasets (75K samples) and all 16 models from the evolutionary lineage:

6.2 Three Findings

The ceiling was data, not architecture. 15K samples → 0.78 max across all configurations. 75K samples → 0.81-0.83. The conv backbone (128 channels, 32×32 input) had more capacity than any 15K experiment revealed. The encoder was starving for examples, not parameters.

Distillation adds value even with abundant data. 0.830 vs 0.813 — the 16-parent consensus contributed +0.017 beyond what 5× data alone provided. The consensus captures invariant geometric structure that pure data volume doesn't guarantee.

The epoch-1 head start. The distilled student started at 0.580 validation accuracy on epoch 1. The raw model needed 10 epochs to reach 0.699. Consensus anchors + distillation targets gave the student a 10-epoch convergence advantage. This is the inheritance mechanism working — the coordinate system was pre-solved.

---

7. Resonant Dynamics

7.1 The Observation

In every consensus distillation experiment, the student's late-training acceleration exceeded what standard optimization theory predicts. Typical training curves decelerate as the model approaches the loss landscape minimum — learning rate decays, gradients shrink, the system oscillates around equilibrium and settles. The geometric system accelerated:

Epochs 20→25: +0.033 val_acc
Epochs 25→30: +0.025 val_acc (still accelerating)

The model was gaining accuracy FASTER in late training than in early training, and hadn't plateaued at epoch 30.

7.2 The Mechanism

Standard training:

• Epoch N: gradient pushes embedding somewhere
• Epoch N+1: gradient pushes it somewhere else
• Net effect: oscillation, energy dissipation, plateau

Geometric training with consensus:

• Epoch N: tangential gradient slides embedding along manifold
• Epoch N+1: manifold is slightly better → consensus signal clearer
• Epoch N+2: clearer signal → better tangential gradient → better manifold
• Net effect: constructive interference, energy accumulation, acceleration

The geometric autograd filters out destructive gradient components (radial, collapsing) and passes constructive components (tangential, separating). When destructive interference is removed, what remains is pure constructive signal. Constructive signal compounds across epochs.

The CV trajectory confirms the crystallization: 1.19 → 0.67 → 0.55 → 0.45 → 0.34. The manifold is becoming more regular (approaching the 0.2 target) while accuracy increases. In standard training, geometric regularity and discriminative capacity are inversely related — co-training experiments in Part II proved this. The consensus distillation pipeline inverts this relationship: regularity and accuracy improve simultaneously because the consensus targets encode the geometric truth, and the autograd keeps the system on the resonant path.

7.3 The Resonant Cavity Analogy

The dual-teacher consensus defines a standing wave pattern — the geometric center of two independently-discovered manifolds. The geometric autograd defines the cavity walls — gradient filtering that passes only the harmonic matching the cavity geometry. The student's embeddings are the oscillating field.

Instead of damping, the cavity amplifies. The walls are ACTIVE — they don't just reflect, they filter. Only the harmonic matching the consensus geometry survives. Everything else is absorbed. This is why AdamW kills the resonance: weight decay is uniform damping applied to an active cavity. It removes energy from all modes equally, including the resonant mode. The geometric autograd provides SELECTIVE damping — destructive modes are absorbed, the resonant mode is passed. That's the difference between a dead room and a concert hall.

---

8. Implications for Distributed Training

8.1 The Synchronization Primitive

Standard distributed training (DDP) averages gradients across workers every step. The averaged gradient is applied identically to all workers, keeping weights synchronized. With geometric autograd, the gradient filtering is nonlinear and state-dependent:

Standard DDP:     filter(avg(grads))  ← filter AFTER averaging
Geometric:        avg(filter(grads))  ← filter BEFORE averaging
These are NOT equivalent.

The evolutionary pipeline suggests a different synchronization approach: each worker trains independently with its own geometric autograd for K epochs, then embeddings are extracted on a shared calibration set, GPA-aligned, and consensus anchors are broadcast back to all workers. The backbones continue training independently but the coordinate system re-synchronizes periodically.

The anchors are the synchronization primitive, not the weights. Two workers' weight deltas might point in different directions that are actually the SAME geometric update expressed in different rotational frames. Procrustes resolves the frame. Without alignment, averaging weights across independently-trained models is averaging vectors in misaligned coordinate systems. The magnitudes cancel. That's why naive model averaging barely works but GPA consensus distillation produces resonance.

8.2 Practical Recipe

1. Train 2+ models with different configs         (variation)
2. Extract embeddings on shared data               (phenotype)
3. GPA-align, compute consensus                    (selection)
4. K-means consensus → anchor initialization       (inheritance)
5. Distill with InfoNCE + MSE + micro geometric    (development)
6. Adam, not AdamW. tang=0.01, sep=1.0, cv=0.001  (optimizer)
7. Repeat for additional generations               (evolution)

This recipe is architecture-agnostic. Any encoder that projects onto a hypersphere can use it — BERT, CLIP, vision transformers, convolutional networks. The substrate doesn't matter. The geometric consensus pipeline operates on the manifold, not the architecture.

---

9. Relationship to Prior Work

9.1 Connection to Parts I and II

The geometric autograd is the optimizer that Parts I and II needed but hadn't formalized:

The co-training experiment in Part II proved that consensus fidelity and discrimination are inversely related when optimized simultaneously. Part III resolves this tension: the consensus is computed OFFLINE (Procrustes on frozen teacher embeddings), then distilled. The student never has to trade between matching consensus and discriminating classes — the consensus targets already encode the discriminative structure.

9.2 The CV Constant

The 0.20-0.23 CV band continues to appear across all systems:

Consensus CV is consistently BELOW the universal band — the shared geometric center of multiple models is smoother than any individual model. Student CV starts above and converges toward the band during training. The 0.2 target is an attractor for pretrained models; for randomly-initialized models, it's a destination they reach through consensus distillation rather than a starting point.

---

10. Conclusion

The geometric autograd replaces weight decay with manifold-aware gradient filtering. Adam with tangential projection, separation preservation, and micro CV loss outperforms AdamW by 15% on a 30-class classification benchmark. The mechanism is simple: the geometry IS the regularization. Weight decay damps everything; geometric filtering damps only the destructive modes.

Consensus distillation through Procrustes alignment produces students that exceed both teachers. The mechanism is noise cancellation: what survives alignment across independently-trained models is geometric truth, not individual bias. Even catastrophically-trained models contribute to the consensus — Procrustes extracts agreement, not quality.

Multi-generational evolution with data diversity compounds this effect across generations. Each generation inherits consensus geometry, trains with fresh variation, and passes refined geometry to the next. The convergence is monotonic and robust to individual model failures.

The resonant dynamics — late-training acceleration incompatible with standard optimization theory — suggest that geometric gradient filtering creates conditions for constructive interference. The filtered gradient landscape is smoother in the directions that matter and steeper in the directions that help. Energy accumulates instead of dissipating. The system resonates.

The practical recipe — train diverse models, GPA-align, distill with geometric autograd, repeat — is architecture-agnostic and ready for deployment on any hypersphere embedding system.

---

Reproducibility

All experiments in this work use a synthetic 30-class shape classification benchmark with controlled perturbation profiles. The complete pipeline including shape renderers, geometric autograd, Procrustes alignment, multi-generational evolution, and data-diverse training is available in the following files:

All experiments were conducted on a single NVIDIA GPU. The geometric autograd adds negligible computational overhead — the CV loss (16 pentachoron samples per batch) is the most expensive component at approximately 2ms per batch on consumer hardware.