Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.
Examines whether Transformers are necessary when LLMs write model weights without gradient training. When allowed to design the architecture, the tested LLMs compile facts into sparse tables, reaching up to 54.4% accuracy on ARC.