A Guth Labs publication ↗
Back to News

FOAM trims SOAP’s training overhead, but its public code is still pending

NAVER AI Lab researchers propose FOAM, a lighter version of the SOAP optimizer, in a paper submitted November12. They report competitive validation loss with substantially less optimizer-state storage than SOAP. The central pretraining comparison covers models from 124 million to 760 million parameters.

That is useful research for teams weighing training quality against compute overhead. It is not evidence that frontier training costs have suddenly been halved.

What changes

FOAM keeps input-side preconditioning and factors the rotated second-moment estimate. The authors report half SOAP’s optimizer state and approximately Muon-class runtime in their tested settings. Their full paper explicitly positions the contribution as explaining which side to retain and when factorization works, rather than inventing either ingredient. The paper’s headline memory claim concerns optimizer state, not all GPU memory used during training.

The availability check matters

A paper linking a repository does not establish that runnable code has shipped. At inspection on November15, NAVER’s repository contains a README stating that code remains under internal review and will be released later. No implementation-release date was verified. This article therefore does not describe FOAM as ready-to-install open-source training software.

Guth analysis

The practical next step is reproduction, not switching a production training stack on headline numbers. Small-model experiments can expose a valuable engineering direction while still leaving distribution, hardware and scale questions unanswered. The most useful comparison will hold the training objective, quality target and hardware constant and measure time-to-quality alongside peak memory.

Evidence limits

Performance claims are the authors’ results. No independent replication was verified in this pass. Appendix experiments broaden the paper beyond the headline setup, so 760M should not be described as an absolute limit on every experiment. It is the upper end of the main pretraining comparison cited here. No claim about frontier-scale effectiveness is established.

Sources and citations

Revision history

Published .