Abstract
We analyze a learning-to-optimize (L2O) algorithm that exploits parameterspace symmetry to enhance optimization efficiency. Prior work has shown thatjointly learning symmetry transformations and local updates improvesmeta-optimizer performance. Supporting this, our theoretical analysisdemonstrates that even without identifying the optimal group element, themethod locally resembles Newton's method. We further provide an example wherethe algorithm provably learns the correct symmetry transformation duringtraining. To empirically evaluate L2O with teleportation, we introduce abenchmark, analyze its success and failure cases, and show that enhancementslike momentum further improve performance. Our results highlight the potentialof leveraging neural network parameter space symmetry to advancemeta-optimization.