Security teams have decades of tooling to scan networks for weaknesses, but an LLM long stayed a black box you simply hoped behaved. garak flips that by treating a model the way a pentester treats a server: throw thousands of adversarial inputs at it and log every place it breaks. The underlying insight is that LLM safety is empirical, not declarative — you can't read a model card and know it won't leak data or obey a jailbreak. You have to attack it and count the failures.
What Sets It Apart
- Probe-and-detector architecture, not a fixed checklist. Each of its dozen-plus probe families — DAN-style jailbreaks, encoding-based injection, training-data leakage, malware generation, toxicity — pairs with detectors that score responses, so coverage grows as new attack classes land rather than freezing at release.
- Model-agnostic by design. The same suite runs against Hugging Face, OpenAI, AWS Bedrock, Cohere, Groq, Replicate, or any REST endpoint, so you can benchmark a hosted API against a local model on identical attacks.
- Quantified output, not vibes. Runs emit JSONL logs and per-probe hit rates, turning "is this model safer than last quarter's?" into a number you can diff in CI instead of a judgment call.
Who It's For
Great fit if you ship or fine-tune LLMs and need repeatable, attack-based evidence of robustness before release, or if you red-team models and want a standard probe library instead of hand-rolled prompts. Look elsewhere if you need guardrails that block attacks at inference time — garak finds weaknesses, it doesn't patch them — or if you expect a polished GUI; it is a CLI whose output is logs and reports, closer to nmap than to a dashboard.
