Soup

Local LLM training is often blocked by GPU memory and brittle infra; Soup's key practical insight is to keep the frozen base out of VRAM and feed it to the GPU one decoder layer at a time, enabling LoRA-style fine-tuning of 8B models on 4 GB cards.

What Sets It Apart

• Layer streaming (opt-in, BETA): the frozen base is streamed from host RAM or NVMe one decoder layer at a time and dequantized to NF4 on the fly, shrinking the stored base ~4× and bounding peak VRAM by a single layer instead of the whole model. Measured example: Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s with a 3.32 GB peak on an RTX 3050 (batch 1, seq 512).
• One-config training workflow: a single soup.yaml drives model selection, quantization, LoRA adapters, batching and evaluation; many recipes and auto heuristics remove manual tuning of low-level infra.
• Evaluation & release gating: built-in eval suites and soup ship regression gate, deterministic reward synthesis from references (soup reward synth), reproducible evidence emission and export paths (GGUF, ONNX, TensorRT, AWQ, GPTQ, etc.).
• Reproducibility focus and measured claims: the layer-streaming approach and its correctness protocol are documented in a citable preprint; the project publishes raw benchmark records and the change history for measurements and bug fixes.

How It Works (concise)

The CLI runs the adapter training on-device while streaming the frozen base from host memory/disk. Preference losses (DPO/ORPO/SimPO/KTO) can use the same streamed base as a reference by switching adapters off rather than instantiating a second full copy (memory-free reference at the cost of extra layer reads). Streaming increases layer reads (e.g., ~1.52× for some preference losses) and has per-layer NF4 dequantization cost; streaming is slower than resident training but enables runs that would otherwise not fit.

Who It's For and Trade-offs

Great fit if you want to iterate LoRA or other PEFT recipes locally without expensive multi-GPU setups, validate tuning with built-in gates, or export models for local deployment. It is explicit about hardware-gated claims and publishes benchmarks to back them.

Look elsewhere if you need the absolute fastest resident training (streaming is slower), production-scale distributed training primitives like ZeRO-3 tuned for many GPUs, or if you require features explicitly excluded (PPO/GRPO/ppo-style runs are intentionally limited). Layer streaming is marked BETA: correctness fixes have landed (v0.72→v0.73 series) and some earlier claims were revised in later preprint versions.

Where It Fits

Use it as an accessible local training and evaluation pipeline for LLM research and small-scale model iteration, as a bridge between single-GPU experimentation and deployable export formats. Expect measurable trade-offs in throughput versus resident training but genuine gains in ability to run larger models on memory-constrained hardware.