# Three-layer hierarchical learning: Muon, DMFT, and BBP audit Date: 2026-07-22 ## Executive conclusion The repository already contained a strong deterministic three-layer quadratic ResNet program and a separate, much more mature Muon/Volterra/BBP program for phase retrieval. It did **not** contain a Muon optimizer or a genuine block-Wishart Hessian experiment for the three-layer ResNet. The new runner `experiments/brainstorm/resnetl3/hierarchical3_muon_bbp.py` makes the first controlled bridge. It compares ordinary gradient descent, exact polar Muon, and two transferred Muon powers on identical initializations and datasets. It then lifts trajectory checkpoints to a fresh ambient Gaussian sample, where the orthogonal first-layer Hessian is exactly a matrix-weighted block-Wishart matrix. Its bulk edges are computed with the Montanari--Saeed variational formula, while raw polynomial Hessians and clipped theorem-compatible Hessians are reported separately. The current empirical conclusion is: - Muon is promising but not uniformly dominant: exact polar Muon and the normalized `a=1/3` control each beat GD on two of three paired datasets. - At the endpoint, polar Muon's median exact test risk is `0.0871`, versus `0.7569` for GD. The corresponding means are `0.1463` and `0.5816`. - The variance across only three repetitions is large. These numbers establish a reproducible pilot, not an optimizer ranking theorem. - The higher Hermite sectors are learned sequentially. On repeat zero, GD does not halve the order-4 error by step 1200; polar Muon does so at step 680 and `a=1/3` at step 800. - Informative Hessian eigenvectors exist outside empirical bulk intervals, but their number is not monotone in time and is sensitive to polynomial-tail truncation. This is a dynamic-BBP diagnostic, not a completed BBP theorem. ## 1. Repository state before this bridge The audited Git state was clean at `f6c83833cb00bb36a72ee6843ed7164e19e4ae8e` (`master`, `origin/master`), with 227 commits and roughly 9,475 tracked files. The largest relevant bodies were: - `experiments/brainstorm/resnetl3/`: exact order-4 reduced dynamics, ambient SGD proxies, Hermite-sector Schur complements, and Hessian decompositions; - `experiments/muon/`: phase-retrieval Muon-a theory, Volterra reductions, finite Schur/BBP experiments, and block-Wishart bulk tests; - `refs/report/3/deterministic/`: deterministic multi-index and general-link closures, including the Braun--Loureiro--Quang Minh--Imaizumi extensions. The existing ResNet L3 results are substantial: - exact finite teacher-subspace coefficient map through Hermite order four; - exact loss, coefficient Jacobian, and Hessian decomposition `H = H_GN + H_res`; - exact orderwise weighted Gram matrices and Schur complements `S2,S3,S4`; - many long trajectories and width/cross-term sweeps; - low-loss examples, alongside documented stiffness, local minima, and discretization sensitivity. But searches for `Muon`, `polar`, `SignSVD`, and Newton--Schulz in the L3 subproject returned no optimizer implementation. The old ambient Hessian was explicitly labelled a proxy. Therefore, transferring only a plot or a phase retrieval exponent would have been scientifically incorrect. ## 2. Exact three-layer model For teacher coordinates `z in R^k`, the reduced student is ```text h1 = V z + b, q = He_2(h1) = h1^2 - 1, h2 = U z + A q + c, s = He_2(h2) = h2^2 - 1, f = a0 + alpha^T q + beta^T s. ``` This is a two-hidden-block, scalar-readout network: three trainable layers in the convention used by the cited hierarchy papers. The optional tied model sets `U=V`. The campaign uses the untied model because the repository's tied trajectories are measurably stiffer. Let `c(theta)` be the vector of Hermite coefficients of `f` up to order four, `D=diag(alpha!)`, and `c_*` the teacher coefficients. Gauss--Hermite quadrature is exact for the polynomial integrals used here, so ```text R(theta) = 1/2 (c(theta)-c_*)^T D (c(theta)-c_*), grad R = J_c(theta)^T D (c(theta)-c_*), H = J_c^T D J_c + sum_gamma D_gamma delta_gamma Hess(c_gamma). ``` Consequently, the population trajectory, order-2/3/4 errors, reduced Hessian, and Schur complements are deterministic finite-dimensional objects. This is stronger than a Monte Carlo approximation, but it is not a high-dimensional DMFT. ## 3. What the references actually contribute ### 3.1 Depth as coarse-graining Dandi, Pesce, Zdeborova, and Krzakala, [*The Computational Advantage of Depth*](https://arxiv.org/abs/2502.13961), introduce SIGHT/MIGHT targets with strictly decreasing effective dimensions. The rigorous route trains layers successively: recover the first latent subspace, build nonlinear intermediate features, then fit the final readout. Their joint-backpropagation results are numerical rather than the direct scope of the layerwise theorem. The valid import here is the physical mechanism: ```text ambient input -> first recovered subspace -> nonlinear feature space of smaller effective dimension -> final low-dimensional readout. ``` In this repository, the orderwise Schur objects `S2,S3,S4` are finite-model observables of the same coarse-graining idea. They do not automatically prove the sample thresholds of SIGHT/MIGHT. ### 3.2 Sequential feature recovery and scaling laws Wortsman-Zurich, Tabanelli, Dandi, Krzakala, and Loureiro, [*Scaling Laws from Sequential Feature Recovery*](https://arxiv.org/abs/2605.14567), study a lifted degree-`q` hierarchy with feature strengths `lambda_i = Z_gamma z_i i^{-gamma}`. Their spectral estimator recovers feature `i` at the sharp scale ```text n_i ~ d^q i^(2 gamma) / Z_gamma^2, m(n) ~ (n Z_gamma^2 / d^q)^(1/(2 gamma)). ``` For a summable squared spectrum (`gamma>1/2`), summing the unrecovered tail gives ```text MSE(n) ~ (n/d^q)^(-1 + 1/(2 gamma)). ``` The central import is a cascade of sharp BBP-like feature transitions whose aggregate is a smooth power law. The present fixed-`k=3` experiment tests the ordering mechanism, not this asymptotic exponent: `k` does not grow and the model does not use their lifted dimension `D ~ d^q`. ### 3.3 Three-layer feature-learning guarantees Nichani, Damian, and Jason D. Lee, [*Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks*](https://arxiv.org/abs/2305.06986), and Wang, Nichani, and Lee, [*Learning Hierarchical Polynomials with Three-Layer Neural Networks*](https://arxiv.org/abs/2311.13774), prove that a staged three-layer procedure can first recover a lower-degree feature and then learn the outer function. For a target `g o p` with feature degree `k` and link degree `q`, the hierarchical method can operate near the `d^k` feature-recovery scale instead of the `d^(kq)` shallow-kernel scale. Their guarantees use layerwise schedules, freezing, or sample splitting. They do not prove the joint Muon dynamics used in the new campaign. They justify the staged observable design and motivate a future layerwise control arm. ### 3.4 Joint direction/link learning Pillaud-Vivien and Schertzer, [*Joint Learning in the Gaussian Single Index Model*](https://arxiv.org/abs/2505.21336), prove convergence of an alternating scheme that jointly learns a direction and a univariate link. The rate is controlled by the Gaussian information exponent, and convergence can occur from negative initial correlation. This supports treating readout/link adaptation as part of the dynamics rather than freezing it forever. It remains a single-index alternating theorem, not a theorem for the coupled multi-index matrices `V,U,A`. ### 3.5 Deterministic anisotropic phases Braun, Loureiro, Quang Minh, and Imaizumi, [*Fast Escape, Slow Convergence*](https://arxiv.org/abs/2511.18661), show that power-law anisotropy destroys the usual two-scalar phase-retrieval closure. An infinite hierarchy of weighted overlaps is reduced by Duhamel/Volterra methods. The trajectory has three phases: 1. fast escape from low alignment; 2. slow convergence of macroscopic summaries; 3. spectral-tail learning along low-variance directions. The local repository extension replaces scalar weighted overlaps by matrices `U_m,S_m`. In the low-correlation phase, the escape pole solves ```text lambda_max(8 K_hat_b(rho)) = 1, K_hat_b(rho) = sum_i lambda_i^2 a_i a_i^T / (rho - b lambda_i). ``` This is a deterministic population statement. A random-matrix theorem is still needed to turn the same pole into an empirical Hessian outlier. ### 3.6 Random-feature DMFT benchmark Kramp, Lindner, and Helias, [*Dynamics of neural scaling laws in random feature regression*](https://arxiv.org/abs/2602.23039), derive a dynamical mean-field theory for frozen random features with power-law kernel eigenvalues. The microscopic modes become effective independent modes coupled through a common response and effective noise; their framework includes gradient flow, regularization, and Langevin dynamics. This paper is the correct DMFT benchmark, but not a ready-made closure for the present feature-learning network. Here the features and their spectrum evolve because `V,U,A` move. Importing the frozen-feature equations without extra response kernels would discard the phenomenon being studied. ### 3.7 Block-Wishart bulk Montanari and Saeed, [*Variational Formulas for the Spectrum of Block Wishart Matrices*](https://arxiv.org/abs/2606.27774), analyze ```text W = (1/n) sum_mu T_mu tensor x_mu x_mu^T, n/d -> alpha, ``` for fixed block size. With empirical block law `nu`, the matrix K-transform is ```text K(S) = E_nu[T (I + S T)^(-1)] - alpha^(-1) S^(-1). ``` The matrix Stieltjes transform solves `K(S)=zI`; the paper gives variational formulas for support edges and the exterior logarithmic potential. It does not by itself give the anisotropic local law or finite-rank Schur theorem needed for BBP outliers and their eigenvector residues. ## 4. Muon transfer For a matrix gradient `G=U diag(sigma_i) V^T`, the tested spectral family is ```text Phi_a(G) = U diag((sigma_i / mean(sigma))^a) V^T, ``` followed by unit-RMS normalization. Thus `a=0` is exact polar/SignSVD Muon. Only the matrix parameters `V,U,A` receive this update; biases and readout vectors use ordinary gradient descent. The values `a=1/7` and `a=1/3` have deliberately different labels: - `1/7` is the raw phase-retrieval power-law prediction at spectral exponent `3/2` in the separate Paquette/Volterra program; - `1/3` is its normalized-branch comparator. Neither value is derived for the L3 model. The matrices `V`, `U`, and `A` have different shapes, roles, and gradient laws; a genuine L3 theory should allow block-specific powers `a_V(t),a_U(t),a_A(t)` and clocks. ## 5. Fresh-sample Hessian lift At a fixed reduced state, embed `z` as the first `k` coordinates of an ambient Gaussian vector `x=(z,xi) in R^d`. Let ```text h = (h1,hU) = (Vz+b, Uz), Delta = f(h)-y. ``` For square loss, the per-sample preactivation Hessian is ```text T(z) = grad_h f grad_h f^T + Delta Hess_h f. ``` The extensive Hessian with respect to the direct weights `(V,U)` is exactly ```text H_W = (1/n) sum_mu T(z_mu) tensor x_mu x_mu^T. ``` The orthogonal principal block is ```text H_perp = (1/n) sum_mu T(z_mu) tensor xi_mu xi_mu^T. ``` On a fresh sample, `xi_mu` is independent of `T(z_mu)`. This is precisely the block-Wishart setting. The signal coordinates `z`, the cross blocks, and the finite parameters `(A,b,c,alpha,beta,a0)` form the finite signal sector whose outliers should ultimately be described by a Schur determinant. The polynomial link makes `T(z)` unbounded. The rigorous edge calculation is therefore run on spectrally clipped blocks. The runner separately stores: - clipped variational and clipped empirical bulk edges; - raw empirical bulk edges and raw full-Hessian eigenpairs; - the clipping fraction and operator-norm perturbation. Conflating these objects would produce a false theorem claim. ## 6. Empirical protocol Configuration: ```text k=p=q=3, untied U/V, Hermite quadrature nq=7, teacher sectors 2+3+4 with diagonal and mixed terms, train_n=1024, 1200 full-batch steps, 3 paired repetitions, ambient d=32, d_perp=29, n_fresh=145, alpha_perp=5, checkpoints 0,400,800,1200, block cap 100. ``` Each optimizer receives the same initialization and fixed training sample within a repetition. Fresh BBP samples use common random numbers across optimizers at each checkpoint. Therefore all four spectral states agree exactly at step zero. The test risk is evaluated by exact population Gauss--Hermite quadrature, not on the training sample. The plot shows individual trajectories, their median, and the min--max envelope. With `R=3`, no inferential confidence interval is claimed. ## 7. Results ### 7.1 Paired endpoint risks | optimizer | repeat risks | mean | median | geometric mean | wins vs GD | |---|---|---:|---:|---:|---:| | GD | 0.7569, 0.9416, 0.04620 | 0.5816 | 0.7569 | 0.3205 | baseline | | Muon `a=0` | 0.2712, 0.08062, 0.08712 | 0.1463 | 0.08712 | 0.1240 | 2/3 | | Muon `a=1/7` | 0.4114, 0.06553, 0.6195 | 0.3655 | 0.4114 | 0.2556 | 2/3 | | Muon `a=1/3` | 0.08887, 0.05081, 0.2975 | 0.1457 | 0.08887 | 0.1103 | 2/3 | Paired risk ratios versus GD are: ```text a=0: 0.358, 0.0856, 1.886 a=1/7: 0.544, 0.0696, 13.41 a=1/3: 0.117, 0.0540, 6.438. ``` The third initialization/dataset is easy for GD and exposes large failures of the transferred powers. This is evidence for state-dependent or blockwise Muon control, not for choosing a universal constant from the current panel. ### 7.2 Sequential sector recovery on repeat zero | optimizer | order 2 half-error | order 3 half-error | order 4 half-error | |---|---:|---:|---:| | GD | 120 | 160 | not reached | | Muon `a=0` | 120 | 120 | 680 | | Muon `a=1/7` | 80 | 120 | 1160 | | Muon `a=1/3` | 80 | 80 | 800 | This is the clearest hierarchy signal: early sectors move first and the order-4 sector is the late bottleneck. Muon changes the late-sector access time substantially. One trajectory is not enough to assign a universal scaling exponent. ### 7.3 Spectral diagnostic At step zero, all optimizers share the same clipped variational bulk `[-1.153,1.112]` and the same clipped empirical bulk `[-0.954,0.886]`. Later the bulk becomes broad and optimizer dependent. Raw informative outliers are not monotone: ```text GD: 4 -> 1 -> 3 -> 2 Muon a0: 4 -> 0 -> 1 -> 2 a=1/7: 4 -> 0 -> 1 -> 1 a=1/3: 4 -> 1 -> 2 -> 3. ``` The clipped theorem-compatible counts differ because a few large polynomial blocks have a large operator-norm effect even when the clipping fraction is small. With cap 100, the median clipping fraction is about `2.1%` and the maximum is `6.2%`, but the median full-Hessian operator perturbation is about `32.9`. Rare blocks therefore matter for extremes. The correct claim is limited: > The experiment identifies evolving, teacher-aligned finite modes relative to > a block-Wishart bulk and demonstrates optimizer-dependent spectral motion. > It does not yet establish a sharp asymptotic BBP crossing time. ### 7.4 Nested sample-size panel A second campaign uses five paired repetitions and nested training samples ```text n in {64,128,256,512,1024,2048}, 800 steps per run. ``` At the largest sample size, the median exact population risks are | optimizer | median risk at `n=2048` | descriptive tail slope | |---|---:|---:| | GD | 0.2592 | -0.536 | | Muon `a=0` | 0.1038 | -0.381 | | Muon `a=1/7` | 0.06948 | -0.779 | | Muon `a=1/3` | 0.08954 | -0.521 | The slopes are log--log fits over only the four largest sizes. They are descriptive finite-`k` quantities and must not be identified with the exponent of Wortsman-Zurich et al. The sector errors show a more stable hierarchy than the raw direction alignments. At `n=2048`, the median final/initial error ratios are ```text order 2 order 3 order 4 GD 0.00971 0.327 0.302 Muon a=0 0.00214 0.0985 0.106 Muon a=1/7 0.000775 0.0981 0.0747 Muon a=1/3 0.00266 0.168 0.0887. ``` Direction-wise alignment probabilities are not monotone in `n` with five repetitions. This panel therefore confirms sample-dependent risk improvement and late high-order recovery, but not a sharp ordered cascade of individual directions. Testing that stronger claim needs growing latent rank and many more repetitions. ## 8. DMFT target for the joint L3 model A full joint-feature DMFT must introduce, for every trainable matrix block `ell in {V,U,A}`, two-time correlations and responses, ```text C_ell(t,s) = dimension-normalized , R_ell(t,s) = delta W_ell(t) / delta eta_ell(s), ``` plus cross-layer kernels because `A` transports first-layer features into the second layer. After disorder averaging, the effective process should have the schematic form ```text d W_ell(t) = deterministic drift[Q,M,c] - integral_0^t Sigma_ell(t,s) W_ell(s) ds + effective noise_ell(t), ``` with self-consistency between `(C,R)`, the evolving Hermite coefficient map, and the gradient singular spectra. Muon replaces the raw matrix force by a nonlinear spectral functional, so its response contains the Fréchet derivative of the matrix power/polar map. This is absent from frozen-feature DMFT. A controlled derivation should proceed in three stages: 1. fresh-batch or sample-split dynamics, where current weights are independent of the new Gaussian orthogonal coordinates; 2. finite-time reused-sample dynamics by leave-one-out/cavity control; 3. online same-sample Muon with the corresponding Onsager/response term. Calling the current deterministic coefficient ODE a "DMFT" would skip stages 2 and 3. ## 9. Exact, empirical, conditional, and open statements | statement | status | |---|---| | finite Hermite coefficient map and population risk | exact | | reduced gradient/Hessian and orderwise Schur complements | exact | | analytic preactivation Hessian block | exact; autodiff error about `1e-16` | | orthogonal fresh Hessian is block-Wishart | exact algebraically | | Montanari--Saeed edge for clipped fresh bulk | theorem import under bounded-block assumptions | | GD/Muon paired learning curves | empirical, 3 repetitions | | sequential order-2/3/4 recovery | empirical pilot | | raw polynomial extreme eigenvalues | empirical; unbounded-weight issue open | | finite-rank BBP root and eigenvector residue | open anisotropic local-law/Schur theorem | | reused-sample trajectory transfer | open leave-one-out problem | | joint L3 DMFT | open | | universal L3 Muon exponent | refuted by current variability as a naive claim; blockwise/state-dependent theory open | | Wortsman et al. sample-scaling exponent for this exact ResNet | not tested by fixed-`k` panel | ## 10. Next acceptance campaign The empirical priority should now be: 1. Increase the main paired campaign from `3` to at least `20` repetitions. 2. Extend the completed sample sweep to a joint ambient-dimension/latent-rank sweep, for example `d in {24,32,48,64,96}` with growing `k`. 3. Increase the completed direction-wise recovery panel beyond five repetitions and replace raw row-max alignment by matched subspace overlaps. 4. Add layerwise/frozen controls matching the assumptions of Dandi et al. and Jason Lee's papers. 5. Sweep block-specific powers and learning-rate-matched controls: `(a_V,a_U,a_A)`, not only one global `a`. 6. Repeat BBP spectra over fresh probes and report quantiles of edge/outlier locations. 7. Run a truncation sequence `R in {30,100,300}` and a bounded smooth-link control. The latter is the clean theorem-grade benchmark. 8. Only after fresh-probe convergence, compare held-out, cross-fit, and same-sample Hessians to isolate the leave-one-out correction. ## 11. Reproduction Main run: ```bash python experiments/brainstorm/resnetl3/hierarchical3_muon_bbp.py ``` Focused algebra tests: ```bash python -m unittest \ experiments.brainstorm.resnetl3.test_hierarchical3_muon_bbp -v ``` Artifacts: - `results/hierarchical3_muon_bbp_latest/summary.json`; - `results/hierarchical3_muon_bbp_latest/training_trajectories.csv`; - `results/hierarchical3_muon_bbp_latest/fresh_sample_bbp.csv`; - `results/hierarchical3_muon_bbp_latest/training_and_sequential_recovery.png`; - `results/hierarchical3_muon_bbp_latest/dynamic_bbp_fresh_sample.png`; - `results/hierarchical3_muon_bbp_latest/REPORT.md`; - `results/hierarchical3_sample_scaling_latest/summary.json`; - `results/hierarchical3_sample_scaling_latest/sample_scaling_runs.csv`; - `results/hierarchical3_sample_scaling_latest/sample_scaling_and_feature_recovery.png`.