New Preprint, Low-Rank Neural Networks and Finite-Width NTK at the Edge of Convexity

Low-rank neural networks are usually presented as smaller, compressed or pruned models. This paper asks a sharper question: when does low rank still preserve the optimization geometry that makes wide neural networks trainable?

We answer this through the finite-width neural tangent kernel. Low-rank NTK training still has an interpretable bottleneck-feature geometry, because the rank controls which feature maps enter the tangent kernel.

Main results.

1. Width-depth-rank compromise: Low-rank training is governed by a genuine three-way tradeoff between width, depth, and rank.

2. Depth-rank certificate: Preserving the NTK spectral margin gives a conservative cubic-depth sufficient rule, $r \gtrsim L^3$, up to dataset-size and logarithmic factors.

3. Finite-network scaling: The finite-network experiments suggest an effective $L^{3/2}$ law for scalar-output contracted cumulants.

4. Full-rank compatibility: The parametrization exactly matches the full-rank case when the rank is maximal, separating true low-rank effects from ordinary finite-width effects.

Experiments. True finite-network NTK experiments confirm the exact full-rank matching, the predicted operator scaling, the rank-depth tradeoff, and the growth of finite-network cumulants. The main takeaway is a criterion for when low-rank networks remain trainable for structural reasons, not merely parameter-efficient ones.

Paper: Low-Rank Neural Networks and Finite-Width NTK at the Edge of Convexity. Joint work with Haizhao Yang and Shijun Zhang.