AIAny
Icon for item

AGE: Adaptive-masking for Graph Embedding in Graph Retrieval-Augmented Generation

Trains a transformer-based graph encoder with RL-guided adaptive masking so retrieved subgraphs embed relationships that better align with frozen LLM text encoders, improving GraphRAG performance with non-parametric retrievers on GraphQA benchmarks.

Introduction

Most frozen LLM workflows treat retrieved graph data as tokenized text or naively embed nodes, which often misaligns graph structure with the LLM's text-based latent space. AGE reframes graph embedding as a mask-based self-supervised task tailored to this gap: instead of random node masking, it learns a node-sampler (via reinforcement learning) that avoids masking irrecoverable "key" nodes and focuses training on auxiliary nodes whose prediction teaches relational structure.

Key Findings
  • Adaptive masking with an RL-guided node sampler yields substantially better graph-to-text embedding alignment than random masking, so downstream frozen-LLM GraphRAG accuracy improves across multiple GraphQA benchmarks.
  • Designed to mimic text encoder SSL (masking + JEPA-style joint embedding predictive objectives), AGE produces embeddings that integrate node relationships rather than just node content, so non-parametric retrievers feed more useful subgraphs to LLMs without costly retriever retraining.
  • Empirical gains are consistent: up to ~26.7 percentage points improvement on ExplaGraphs for a 1B Llama3.2 backbone and notable gains on SceneGraphs/WebQSP/CWQ, so AGE is especially effective when retrieved graphs are relatively small or when retrievers are non-parametric.
Who it's for & tradeoffs

Great fit if you build retrieval-augmented systems that must keep LLMs frozen and rely on non-parametric graph retrieval; AGE boosts graph embedding quality without expensive LLM fine-tuning or LLM-based retrievers. Look elsewhere if your pipeline already uses trainable LLM retrievers or you can fine-tune large LLMs end-to-end; AGE's benefits shrink when retrieval cannot surface critical nodes or graphs contain rich textual context absent from pretraining.

How it works (brief)

AGE uses a transformer encoder trained with mask-based SSL and a JEPA-like decoder objective. A learnable node sampler is trained with RL to avoid masking dominant "key" nodes, focusing prediction loss on auxiliary nodes that better expose relational structure. Trained embeddings are used with existing non-parametric retrieval and a frozen LLM to generate answers, yielding higher GraphQA accuracy with modest compute overhead.

Information

  • Websitearxiv.org
  • OrganizationsOMRON Corporation, OMRON SINIC X Corporation
  • AuthorsBao Long Nguyen Huu, Atsushi Hashimoto
  • Published date2026/06/30

More Items

Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.

Examines whether Transformers are necessary when LLMs write model weights without gradient training. When allowed to design the architecture, the tested LLMs compile facts into sparse tables, reaching up to 54.4% accuracy on ARC.

Details the Xing 4 model architecture, Ascend-scale pretraining systems, and agentic post-training, including a 285B-parameter training run on 8,192 Ascend 910C cards.