Serves token-level routed LLM inference by dispatching requests to per-model asynchronous subservers and using delayed-batching scheduling to reduce admission latency and step desynchronization. Exposes a request-centric route-send-receive API and reports 2.01–64.15× decoding throughput gains versus single-LLM servers.
Explores asymmetric compute allocation in looped Mixture-of-Experts models by compressing the first loop visit's sequence and reallocating saved compute to enlarge hidden size, yielding consistent NLL reductions and predicted compute savings across 1.3B–7B configurations.