Dirble : Fast Directory Scanning And Scraping Tool

Dirble is a website directory scanning tool for Windows and Linux. It’s designed to be fast to run and easy to use.

How to Use?

Download one of the precompiled binaries for Linux, Windows, or Mac, or compile the source using Cargo, then run it from a terminal. The default wordlist Dirble uses is dirble_wordlist.txt in the same directory as the executable.

It can be installed in BlackArch using sudo pacman -S dirble

There is also a docker image, which can be run as: docker run --rm -t isona/dirble [dirble arguments]

The help text can be displayed using dirble --help, alternatively it can be found on the github wiki: https://github.com/nccgroup/dirble/wiki/Help-Text

Example Uses

  • Run against a website using the default dirble_wordlist.txt from the current directory: dirble [address]
  • Run with a different wordlist and including .php and .html extensions: dirble [address] -w example_wordlist.txt -x .php,.html
  • With listable directory scraping enabled: dirble [address] --scrape-listable
  • Providing a list of extensions and a list of hosts: dirble [address] -X wordlists/web.lst -U hostlist.txt
  • Providing multiple hosts to scan via command line: dirble [address] -u [address] -u [address]
  • Running with threading in Gobuster’s default style, disabling recursion and having 10 threads scanning the main directory: dirble [address] --max-threads 10 --wordlist-split 10 -r

Also Read – Evil SSDP : Create Fake UPnP Devices To Phish For Credentials

Building from source

To build on your current platform, ensure cargo is installed and then run cargo build --release. Alternatively, running make will build the binary in release mode (internally running cargo build --release).

To cross-compile for 32- and 64-bit Linux and Windows targets, there is a handy makefile. make release will build for all four targets using cross. This depends on having cross and docker installed (cargo install cross).

Features

  • Cookies
  • Custom Headers
  • Extensions and prefixes
  • HTTP basic auth
  • Listable directory detection and scraping
  • Save ouptut to file
  • Save output in XML and JSON formats
  • Proxy support
  • Recursion
  • Status code blacklisting and whitelisting
  • Threading
  • Request throttling
  • Detect not found code of each directory based on response code and length
  • Ability to provide list of URLs to be scanned
  • User agents
  • Scanning with GET, POST or HEAD requests
  • Exclude ranges of response lengths from output

Performance

The following graph was generated by running each tool with Hyperfine against a test server with 5ms latency and 1% packet loss. (Gobuster was omitted due to lack of recursion).

How it works?

Directory Detection

Dirble detects files based on the response code sent by the server. The behaviour can be loosely categorized by response code type.

  • 200: the path exists and is valid
  • 301, 302: redirection; report the code, size, and Location header
  • 404: not found; by default these responses are not reported
  • All other response codes are reported in the Dirble format of + [url] (CODE:[code]|SIZE:[size])

A path is classified as a directory if a request to [url] (with no trailing slash) returns a 301 or 302 redirection to [url]/ (with a trailing slash).

This gets reported with a D prefix and if recursion is enabled will be added to the scan queue.

This method is not dependent on the redirection target existing or being accessible, so a separate request will be made to determine the response code and size of the directory.

Listable directories are detected by inspecting the content of url/: if it returns a 200 response code and the body contains either “parent directory”, “up to ” or “directory listing for” (case insensitive), then it is likely to be a listable directory. If --scrape-listable is enabled, URLs are parsed out of the listing (ignoring sorting links or out of scope links) and added to the scan queue if they have a trailing slash. Listable directories have an L prefix in the output.

Threading

  • The threading behaviour of Dirble is based on the concepts of wordlists and jobs.
  • A job is any task which can be run independently of other tasks, for example requesting a series of URLs.
  • A wordlist is a list of words with a defined transformation, for example the list {admin, config, shop} together with the transformation append ".php" forms a single wordlist instance.
  • To improve performance further, we introduce the concept of wordlist splitting.
  • This is the process by which a single wordlist instance (i.e. words with a transformation) is broken up into multiple jobs, each responsible for a portion of the list.
  • The number of interleaved portions that each wordlist is split into is defined by the --wordlist-split option (default 3).
  • Whenever a directory is detected (and recursion is enabled) new jobs are created for each split wordlist (with transformation) and added to a central job queue.
  • The maximum number of concurrent tasks is defined by the --max-threads parameter, and Dirble will start jobs as they are added to the queue, up to this limit.
  • Whenever a job completes (i.e. a split wordlist is exhausted) Dirble will take the next job from the queue and start it.
R K

Recent Posts

Shadow-rs : Harnessing Rust’s Power For Kernel-Level Security Research

shadow-rs is a Windows kernel rootkit written in Rust, demonstrating advanced techniques for kernel manipulation…

1 week ago

ExecutePeFromPngViaLNK – Advanced Execution Of Embedded PE Files via PNG And LNK

Extract and execute a PE embedded within a PNG file using an LNK file. The…

2 weeks ago

Red Team Certification – A Comprehensive Guide To Advancing In Cybersecurity Operations

Embark on the journey of becoming a certified Red Team professional with our definitive guide.…

2 weeks ago

CVE-2024-5836 / CVE-2024-6778 : Chromium Sandbox Escape via Extension Exploits

This repository contains proof of concept exploits for CVE-2024-5836 and CVE-2024-6778, which are vulnerabilities within…

3 weeks ago

Rust BOFs – Unlocking New Potentials In Cobalt Strike

This took me like 4 days (+2 days for an update), but I got it…

3 weeks ago

MaLDAPtive – Pioneering LDAP SearchFilter Parsing And Security Framework

MaLDAPtive is a framework for LDAP SearchFilter parsing, obfuscation, deobfuscation and detection. Its foundation is…

3 weeks ago