AIAny
Large Language Model Papers2026
Icon for item

Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Generates repository-specific LoRA adapters via a hypernetwork to inject repo-level knowledge into code LMs with zero inference-time token overhead. Provides a Static snapshot mode and an Evo mode that updates adapters per commit; evaluated on the 604-repo RepoPeftBench.

Visit Website

Introduction

Most code LMs struggle with repository-local details (imports, APIs, conventions) because those facts either bloat the input or require per-repo fine-tuning. Code2LoRA's core insight is to encode repository context as lightweight LoRA adapters produced by a hypernetwork — so you get repository-specific knowledge without longer prompts or per-repo full fine-tuning.

Key Findings
  • Competitive accuracy with far less deployment cost: Code2LoRA-Static matches the per-repository LoRA upper bound on the static track (63.8% cross-repo / 66.2% in-repo exact match), showing hypernetwork-generated adapters can approach per-repo tuning. This means similar code-comprehension gains with a single adapter-generation step instead of a full fine-tune per repo.
  • Handles active development: Code2LoRA-Evo maintains an adapter state updated by a GRU per code diff and achieves 60.3% cross-repo exact match on the evolution track (+5.2 percentage points over a single shared LoRA). In practice this trades a small runtime state update for staying current with code changes.
  • Scale and benchmark: The authors release RepoPeftBench (604 Python repos; static: ~40K train / 12K test tasks; evolution: ~215K train / 87K test tasks), enabling reproducible comparisons for repo-aware parameter-efficient fine-tuning.
Suitable for and tradeoffs

Great fit if you need repository-aware code completion or synthesis at scale but want to avoid sending large context windows or maintaining full per-repo fine-tuned models. Code2LoRA is especially useful when many repositories share a base model but benefit from lightweight repo-specific adapters. Look elsewhere if you require absolute, immediate token-level grounding of extremely novel runtime data (RAG-style retrieval may still be preferable), or if your infra cannot accept the complexity of adapter generation + small state updates per commit.

How it works (brief)

A hypernetwork consumes a repository snapshot (or a diff-updated hidden state in the Evo variant) and generates LoRA adapter weights for a base code LM. Static mode produces one adapter per snapshot; Evo mode uses a GRU to update a hidden representation per commit and outputs updated adapter weights, avoiding token-length increases at inference time.

Practical notes

Code and an anonymous code release are linked by the paper; model checkpoints and the RepoPeftBench dataset are available on Hugging Face. The method emphasizes parameter-efficient adaptation and continuous updating for evolving codebases rather than retrieval-heavy prompt construction.

Back

Information

  • Websitearxiv.org
  • AuthorsLiliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie
  • Published date2026/06/04

Categories

Tags

More Items

Large Language Model Papers2026
Icon for item

Language Models Can Control Their Own Attention

Namgyu Ho, Huzama Ahmad +4KAIST AI, Google DeepMind

Introduces Declarative Attention (DA), a zero-shot protocol that has LMs declare which parts of long context to attend to during chain-of-thought, letting the runtime build dynamic attention masks and skip most KV-cache reads. Produces large token savings (up to ~52% on Gemma-4-31B) with modest accuracy loss.

llmlong-horizonvllmgemmaqwen+5
Natural Language Processing Papers2026
Icon for item

It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Runpeng Dai, Kaili Huang +2University of North Carolina at Chapel Hill, Apple

Generates compact keyword sets for both queries and items with LLMs and matches them directly via an inverted index. Uses supervised fine-tuning to align keyword spaces, then alternates GRPO-based reinforcement learning on query- and item-side generators to co-evolve representations and maximize retrieval F1 while staying compatible with keyword-based infrastructure.

retrievalLLMRLsftbenchmark+2
Large Language Model Papers2026
Icon for item

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Shaowen Wang, Ge Zhang +7

Studies looping shared transformer layers in Mixture-of-Experts models under matched budgets and proposes SMELT: loop the middle half twice while matching per-token FLOPs, non-embedding parameters, and KV cache. Shows 6.8–18.0% training-FLOPs savings on the compute-optimal frontier, stronger downstream gains on code and long-context tasks.

moetransformersllmresearchbenchmark+2