From Muon to Gradient Clipping: Some Thoughts on QK Stability
Muon optimizerQK weight matricesTransformeroptimizationstabilitygradient descentspectral normpseudoinversebilinear attentionlow-rank approximation.
Author: Eridanus2
Date: 7/13/2026
Article Summary:
A researcher explores the limitations of the Muon optimizer when applied to QK (Query and Key) weight matrices in Transformers, and proposes a modified update rule that follows Muon's philosophy while addressing the stability issues.