Why Put a Power Inside Muon?
Start with the gradient
The parameters of a neural network layer form a matrix, so its gradient is also a matrix. That gradient contains several directions. Some directions have large singular values and request a large update; others have small singular values and request a small update.
Ordinary gradient descent keeps these differences. Standard Muon removes them: it gives every nonzero singular direction the same update size.
If the gradient is
\[G=U\Sigma V^\top,\]the two updates are
\[\text{gradient descent: }U\Sigma V^\top, \qquad \text{Muon: }UV^\top.\]Add one power
Now insert a power $p$:
\[U\Sigma^pV^\top.\]This gives a continuous family of updates.
- $p=1$ is ordinary gradient descent: keep the original differences.
- $p=0$ is standard Muon: flatten all nonzero singular values to one.
- $0<p<1$ keeps some information about strength while still boosting weaker directions.
The power is a contrast control for the update. It decides how much Muon should flatten the gradient spectrum.
Why change it during training?
The useful update can change over time. Early in training, the largest directions often contain clear signal, so preserving some contrast can accelerate progress. Later, the remaining signal may lie in smaller directions, so stronger flattening can give those directions more weight.
Our proposal is to choose the power from the current state of training rather than treating it as a permanent optimizer setting. The Boltzmann rule in the full article scores the available powers by their predicted present and future effect, then favors the lower-cost choices without switching abruptly whenever two choices are nearly tied.
How this connects to current Muon research
- Dong and Sawin, Muon$^p$: Muon with Fractional Spectral Powers introduces the same fractional-power family between Muon and gradient descent and shows how to compute it using matrix multiplications.
- Wu et al., DynMuon: A Dynamic Spectral Shaping View of Muon schedules the power during training and reports faster progress than fixed Muon across several language-model settings.
- Paquette et al., Phases of Muon: When Muon Eclipses SignSGD shows theoretically that Muon’s advantage depends on the data spectrum, batch size, and target structure. This supports the view that one fixed spectral rule need not be best in every regime.
These works point in the same direction: the amount of spectral flattening is a meaningful optimizer choice, and it can be adapted during training.