AIAny
Icon for item

Weight-Space Programming Doesn't Need Transformers

Examines whether Transformers are necessary when LLMs write model weights without gradient training. When allowed to design the architecture, the tested LLMs compile facts into sparse tables, reaching up to 54.4% accuracy on ARC.

Introduction

The architecture that works best when trained by gradient descent may be the wrong one when an LLM writes the weights directly. This paper’s counterintuitive finding is that, under that constraint, LLMs often favor a sparse fact table over a Transformer—and can make its contents easier to inspect and correct.

Key Findings
  • Given only ten labeled ARC examples and a sandbox, three LLMs designed models that reached up to 54.4% test accuracy across ARC-Easy and ARC-Challenge.
  • The strongest LLM-designed model used about 1,800 times fewer parameters than the most accurate released Transformer written by the same LLM, while achieving higher test accuracy.
  • The authors’ associative-memory design reached 52.4% accuracy and ran up to 17 times faster than a same-sized Transformer on the reported GPU test. Its table-like structure also makes individual stored facts identifiable and editable.
  • These results concern writing weights from random initialization, not general superiority over trained neural networks. Pretrained models remained substantially more accurate; Qwen3.5-0.8B reached 82.1% on the reported ARC evaluation.
Method and Scope

The experiments compare LLM-written weights for fixed architectures with models whose structure and knowledge files are designed by the LLM. The benchmark is multiple-choice science questions, and the writers receive just ten ARC training examples, so the results test knowledge compilation and recall more than broad reasoning or compositional learning. The paper reports that these written models do not learn representations of their own or chain facts into long answers.

Who Should Read It

A good fit for researchers exploring program synthesis, associative memory, or how architecture choice changes when gradient descent is removed. Look elsewhere for evidence that hand-written weights replace pretraining or solve general reasoning: the paper explicitly leaves a wide accuracy gap to pretrained models, and its main claims are bounded by the tested tasks and setup.

Information

  • Websitearxiv.org
  • OrganizationsPohang University of Science and Technology (POSTECH), {hyunjong.ok, jaeho.lee}@postech.ac.kr, Code
  • AuthorsHyunjong Ok, Jaeho Lee
  • Published date2026/11/17

More Items

Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.

Details the Xing 4 model architecture, Ascend-scale pretraining systems, and agentic post-training, including a 285B-parameter training run on 8,192 Ascend 910C cards.

Connects single-chain protein folding to a general-purpose language model, using its residue representations to drive a pretrained structure generator. Current results show weaker performance on longer chains and reduced scores on a general-knowledge test after protein training.