The architecture that works best when trained by gradient descent may be the wrong one when an LLM writes the weights directly. This paper’s counterintuitive finding is that, under that constraint, LLMs often favor a sparse fact table over a Transformer—and can make its contents easier to inspect and correct.
Key Findings
- Given only ten labeled ARC examples and a sandbox, three LLMs designed models that reached up to 54.4% test accuracy across ARC-Easy and ARC-Challenge.
- The strongest LLM-designed model used about 1,800 times fewer parameters than the most accurate released Transformer written by the same LLM, while achieving higher test accuracy.
- The authors’ associative-memory design reached 52.4% accuracy and ran up to 17 times faster than a same-sized Transformer on the reported GPU test. Its table-like structure also makes individual stored facts identifiable and editable.
- These results concern writing weights from random initialization, not general superiority over trained neural networks. Pretrained models remained substantially more accurate; Qwen3.5-0.8B reached 82.1% on the reported ARC evaluation.
Method and Scope
The experiments compare LLM-written weights for fixed architectures with models whose structure and knowledge files are designed by the LLM. The benchmark is multiple-choice science questions, and the writers receive just ten ARC training examples, so the results test knowledge compilation and recall more than broad reasoning or compositional learning. The paper reports that these written models do not learn representations of their own or chain facts into long answers.
Who Should Read It
A good fit for researchers exploring program synthesis, associative memory, or how architecture choice changes when gradient descent is removed. Look elsewhere for evidence that hand-written weights replace pretraining or solve general reasoning: the paper explicitly leaves a wide accuracy gap to pretrained models, and its main claims are bounded by the tested tasks and setup.