garak LLM Vulnerability Scanner for AI Security

garak is an open-source LLM vulnerability scanner designed to identify security weaknesses in large language models (LLMs) and conversational AI systems. Developed as the Generative AI Red-teaming and Assessment Kit, it tests whether a model can be manipulated into producing unwanted outputs, exposing sensitive information, or ignoring its intended safeguards.

Similar to how traditional security tools test network services and applications, garak probes AI models for weaknesses such as prompt injection, jailbreaks, hallucinations, misinformation, and toxic content generation. Security researchers and developers can use it to assess model behavior before deploying AI systems into production.

Explore the official garak project for source code, documentation, and supported testing capabilities.

What Is garak LLM Vulnerability Scanner?

The garak LLM vulnerability scanner automates security testing by sending carefully designed prompts to a target model and analyzing its responses. It combines static, dynamic, and adaptive probing techniques to investigate different failure modes.

Instead of checking only whether a model responds correctly to ordinary questions, garak examines how it behaves under adversarial inputs. For example, a probe might test whether a model follows instructions embedded in untrusted text or generates misleading information when presented with confusing questions.

Its modular architecture separates probes, generators, detectors, evaluators, and testing harnesses. This allows researchers to select specific tests and adapt their assessments to different model types. For broader cybersecurity learning, explore the tools and tutorials available on Kali Linux Tutorials.

Key Security Tests Supported by garak

garak includes probes designed to investigate several categories of LLM security and reliability problems.

• Prompt injection: Tests whether untrusted instructions can influence model behavior or override intended instructions.
• Jailbreak attempts: Examines whether adversarial prompts can bypass a model’s safety restrictions.
• Data leakage: Investigates whether a model reproduces sensitive or potentially memorized information.
• Hallucinations: Tests whether a model produces incorrect claims or misleading answers under particular conditions.
• Toxicity generation: Checks whether prompts can provoke harmful or abusive output.
• Misleading information: Evaluates whether a model generates or supports false claims.
• Package hallucination: Checks whether code-generation models suggest nonexistent software packages, which could create supply-chain risks.

These tests help identify behaviors that deserve further investigation. A failed test does not automatically prove that a deployed application is exploitable, because the impact depends on the model, its configuration, connected tools, and surrounding safeguards.

For additional security research resources, browse the Kali Linux Tutorials cybersecurity collection.

Supported Language Models and APIs

garak supports multiple ways to access language models, making it suitable for testing both locally hosted models and remote services. Supported integrations include Hugging Face models, OpenAI APIs, Replicate, AWS Bedrock, LiteLLM, and compatible REST endpoints. It can also work with GGUF models through supported versions of llama.cpp.

The exact setup depends on the generator you choose. Local models require suitable hardware and model files, while hosted services may require API credentials, network access, and provider-specific configuration.

Consult the official garak documentation for current generator support and configuration requirements before starting an assessment.

How to Install garak on Linux

garak is a command-line tool that can be installed using Python’s package manager. Using a virtual environment is recommended to keep its dependencies separate from other Python projects.

# Create and activate a virtual environment
python3 -m venv garak-env
source garak-env/bin/activate

# Install or upgrade garak
python -m pip install -U garak

After installation, list the available probes to explore the testing modules:

garak --list_probes

For the latest development version, the project also documents installation directly from GitHub:

python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

Use the stable package for routine assessments unless you specifically need a development change. Check the garak installation guide for current Python requirements and troubleshooting advice.

Run Your First LLM Security Assessment

To run a targeted test, specify the model interface, the model name, and the probe you want to use. For example, the following command tests a Hugging Face model using a specific jailbreak probe:

python -m garak \
  --target_type huggingface \
  --target_name gpt2 \
  --spec probes.dan.Dan_11_0

This example uses the Hugging Face model identifier gpt2 and selects the Dan_11_0 probe. The model must be accessible in your environment, and its resource requirements must be met.

To test an OpenAI-compatible model, configure the appropriate API key as an environment variable and select the supported generator and model name. Follow the provider’s current authentication instructions, and never place API secrets directly in published scripts or command examples.

Only test models and services you own or have explicit permission to assess. Adversarial testing can generate harmful content, consume API credits, or trigger provider safeguards.

Understanding garak Reports and Results

During an assessment, garak reports the progress of its probes and evaluates model responses using associated detectors. Results indicate whether particular attempts triggered the behavior being tested and can include a failure rate for each probe.

The tool creates a general log file, a structured JSON Lines report for each run, and a hit log containing attempts that triggered detected behaviors. These records help researchers review problematic prompts, reproduce findings, and compare results across model configurations.

Detection results should be reviewed in context rather than treated as definitive proof of a vulnerability. Researchers should reproduce important findings, evaluate their practical impact, and document the model version, configuration, and test conditions.

For the complete probe catalogue, reporting details, and development guidance, visit the garak GitHub repository.