Xiaohan Qin, Xiangdong Zhang +8School of AI, Shanghai Jiao Tong University, Shanghai Innovation Institute +3
Enables multi-head, depth-wise residual retrieval in Transformers by expanding a single residual stream into multiple context-aware streams using causal depthwise convolutions. Improves pretraining convergence and downstream benchmark scores with minimal overhead, yielding notable iso-loss compute speedups in large MoE models.
AI Model2026
A System 1 multimodal decision model that picks among up to 256 options and returns calibrated probabilities for choice, yes/no and 0–5 score decisions from text and images. Fine-tuned on Qwen3.5-35B-A3B, available in bf16 and NVFP4 builds (NVFP4 fits ~23.3 GiB GPU), and served via vLLM's POST /v1/decide API.