Why this matters
Running recent 27–35B-class LLMs on consumer-class GPUs requires careful engineering: the right quant, engine, and compose layout can mean the difference between a stable 200K‑token session and a hard OOM cliff. club-3090 collects community-validated compose files, engine pins, patches and benchmark workflows so you can reproducibly run and compare modern LLMs on one or two RTX 3090-class cards.
What Sets It Apart
- Practical, hardware-aware recipes: curated single- and dual‑card Docker compose manifests and launch scripts covering vLLM (full features), llama.cpp (max context/robustness), and ik_llama (fast GGUF paths). Each variant is hardware-filtered and annotated with VRAM budgets and engine pins so the launcher can choose safe defaults.
- Measured tradeoffs, not marketing claims: published TPS and soak/bench protocols for target models (Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma 4 variants) plus per-config run artifacts. Bench pipeline (bench → verify-stress → quality tests → soak) exposes real-world failure modes like prefill cliffs and accumulation OOMs.
- Escape hatches & diagnostics: interactive wizard, a terminal “cockpit” (c3), direct docker compose lines to bypass preflight, and a triage report tool that captures hardware, drivers, kernel/pinned image digests to aid cross-rig debugging.
Who It's For & Tradeoffs
Great fit if you run a homelab or dev backend with 1–2 RTX 3090 (24 GB) GPUs and want reproducible, benchable LLM serving without redesigning engines. It’s optimized for measured throughput/latency and long-context experiments on Ampere rigs.
Look elsewhere if you need a cloud-native multi-node managed service, sub-12GB GPUs, or a turnkey Windows-native experience (vLLM is Linux+CUDA only; Windows workflows require WSL2 and have limitations). Be aware of Ampere-specific caveats: some engines or quant kernels require specific NVIDIA driver/CUDA versions and certain single-card long-context paths may be blocked or retired; dual‑card variants often escape those cliffs.
Where It Fits
- Local inference & benchmarking: drop-in OpenAI-compatible API on localhost for dev, load testing, and tool-using agents.
- Comparative engine research: easily switch between vLLM/llama.cpp/ik_llama and repro bench runs to evaluate TPS, context ceilings, and quant tradeoffs.
- Not a model zoo UI: weights still must be provided (setup scripts help download/verify), and advanced tuning still needs manual patching when experimenting with head-dim/CUDA kernels.
Key operational notes
- Hardware: designed for 1× or 2× NVIDIA RTX 3090 (24 GB); larger Ampere/Ada cards work but some tight VRAM behaviors differ (4090/5090 notes in FAQ).
- Drivers & CUDA: vLLM/ik-llama pins require specific CUDA runtimes (CUDA ≥13 / 13.2 nuances); check nvidia driver compatibility before launching.
- Bench & validation: canonical bench.sh + quality packs; use soak/stress phases to catch multi-turn agent failures (Cliff 2 variants).
If you maintain a small GPU rig and need reproducible, cross-engine LLM serving recipes with measured performance numbers, club-3090 gives a pragmatic, battle-tested starting point and a path to iterate safely on amps‑class hardware.
