ICML 2026 Workshop on Weight-Space Symmetries

Low-Rank Networks Recover Weight and Functional Symmetry Better

The paper is an optimization story: low-rank bottlenecks make symmetric internal representations easier for training to find, even when the data stream and the random features are not explicitly symmetrized.

Poster PDF Workshop paper Animated partials

The paper in one breath

Low-rank random-feature networks can learn symmetric targets from asymmetric stochastic training, but the meaningful object is not only the final output. The question is whether symmetry appears inside the model: in active bottleneck partial functions and in mirror-paired first-layer atoms with compatible outgoing weights.

Symmetry is recovered

Batch size one, no paired samples, and no mirror-symmetric initialization. The symmetry that appears is a learned organization of the trainable low-rank channels.

Output symmetry can fool us

A dense network can fit an even function while its hidden features remain asymmetric. The paper separates output, active partials, and weight-space traces.

One dimension is a microscope

After the first FC layer, dense representations are high dimensional. The clean 1D reflection test avoids arbitrary symmetry metrics while still exposing the hidden optimization effect.

Watch partial functions organize

The animation is pedagogical rather than a raw training log. It shows the distinction the experiments measure: RF-LR bottleneck coordinates can move toward even partial functions, while a dense hidden layer can keep asymmetric features even when the output fits.

training time 0%

What is actually measured

The network alternates a frozen random-feature expansion with a trained low-rank contraction. The trainable object is a sequence of channel mixers, not a dense hidden matrix.

$$ u^{(\ell)}(x)=\sigma(A^{(\ell)}z^{(\ell-1)}(x)+b^{(\ell)}), \qquad z^{(\ell)}(x)=B^{(\ell)}u^{(\ell)}(x). $$

The active bottleneck coordinates $p_k^{(\ell)}(x)=z_k^{(\ell)}(x)$ are the partial functions. They are where the representation either keeps or loses symmetry.

Three diagnostics travel together

Output defect: $f(x)$ versus $f(Tx)$.
Active partial defect: $p_k^{(\ell)}(x)$ versus $p_k^{(\ell)}(Tx)$, ignoring dead channels.
Weight-space trace: nearest mirror atoms $(a,b)$ and $(-a,b)$ should have compatible outgoing weights.

$$ D_{\rm out}(T)= \frac{\mathbb E_x[(f(x)-f(Tx))^2]} {\mathbb E_x[f(x)^2]+\varepsilon}. $$

The empirical map

The cleanest result is conditional and sharper than a leaderboard claim: low output loss alone does not force internal symmetry. The low-rank regimes that matter are the lower-left points where loss, active partial defect, and mirror mismatch all become small.

Weight-space regimes across 149 one-dimensional reruns — 149 one-dimensional RF-LR reruns. Lower-left points combine low test MSE, low active partial even defect, and smaller mirror mismatch.

Real figures, same story

RF-LR partial functions at depth — RF-LR depth partials on an even high-frequency target.

MLP asymmetric hidden features — Matched dense MLP hidden features can remain asymmetric.

Aggregate active partial defects — Low-rank partials stay much more symmetric than MLP hidden features at comparable loss.

Positive-bias stress test — Positive-bias sampling makes the stream asymmetric; RF-LR still enters the low-defect regime.

Mirror pairs for partial-symmetric run — A partial-symmetric run shows more compatible outgoing weights on mirror-related atoms.

Mirror pairs for output-only run — An output-only counterexample fits the target but does not organize the hidden representation.

The optimization angle

The poster version of the conjecture is geometric: symmetric low-rank partial minima have a much larger optimizer-accessible volume than their asymmetric dense-feature analogues. That makes them more likely to be found by SGD or Adam.

The landscape work adds the dynamical bridge. On wide plateaus, the learned symmetry is not a fragile endpoint statistic; it can be conserved while the optimizer waits for a learning-rate transition or an adaptive step to enter a lower-loss basin.

Loss evolution under learning-rate schedules — Plateaus and learning-rate transitions help explain why training selects particular symmetry-organized basins.

How this connects to the other low-rank papers

Low rank is enough

RF-LR can preserve the relevant RKHS while reducing trainable parameters from dense matrices to narrow channel readouts.

Edge of convexity

The finite-width NTK paper explains when low rank remains trainable for structural reasons, not just because it is smaller.

Global anchor

The mean-field paper gives a convergence anchor and a rank-channel view of spectral bias and high-frequency recovery.

This paper

Symmetry recovery becomes the new test of whether the optimizer found a meaningful low-rank representation.

Future theorem target

Prove that, for symmetric data distributions, the low-rank bottleneck creates large-volume minima or plateaus where active partial functions are symmetric. The current theorem core is the implication from mirror-paired atoms with matched outgoing coefficients to symmetric partial functions. The missing optimization theorem is why SGD and Adam enter that region so often.

Scale the diagnostic

Use 1D as the clean microscope, then lift the partial-function view to heads, experts, model merging, and transformer internal states.

Make rank a control knob

Intermediate rank should be selected by what it recovers during training: spectrum, high-frequency content, and now symmetry.

Keep counterexamples

The output-only cases are not bad news. They prove that internal symmetry is a real representation phenomenon, not a tautology.