SubScraper for Subdomain Enumeration and Recon

SubScraper is a Python 3 tool for subdomain enumeration that combines DNS brute forcing, search engine scraping, and VirusTotal lookups to discover subdomains associated with a target domain. It helps security researchers and penetration testers expand their reconnaissance coverage without requiring an API key for its core enumeration techniques.

During enumeration, SubScraper validates discovered subdomains through HTTP and HTTPS requests and DNS A record lookups. This helps identify which hosts resolve and respond, giving testers useful information for prioritizing further investigation. After enumeration, the tool performs CNAME lookups to highlight potential subdomain takeover opportunities that require additional verification.

Key Features of SubScraper

• DNS brute forcing: Uses a wordlist to discover possible subdomains by testing DNS records.
• Search engine scraping: Collects subdomain information from Google and Bing.
• VirusTotal integration: Uses VirusTotal as another source of subdomain discovery, subject to the service’s current access and response behavior.
• HTTP and HTTPS validation: Sends requests to discovered hosts to gather additional information about their availability.
• DNS record checks: Performs A record and CNAME lookups to support validation and identify potential takeover candidates.
• Multithreaded scanning: Allows users to adjust the number of threads to control enumeration speed.
• Export options: Saves results in CSV or TXT format for documentation and further analysis.
• Custom wordlists: Supports a user-provided subdomain wordlist for targeted brute-force discovery.

SubScraper is useful during the reconnaissance phase of an authorized security assessment. Its findings can help map an organization’s external attack surface before more detailed testing begins.

Installing SubScraper

SubScraper is written in Python 3. Clone the project from its GitHub repository and install its required dependencies.

git clone https://github.com/m8r0wn/subscraper.git
cd subscraper
python3 -m pip install -r requirements.txt

Use a virtual environment if you want to keep the project’s Python dependencies separate from your system packages. Check the repository for the current installation instructions and supported Python version before running the tool.

How to Use SubScraper

Run a basic enumeration against a domain you own or have permission to assess:

python3 subscraper.py example.com

To increase the number of worker threads and export the results as a CSV file, use:

python3 subscraper.py -t 5 -o csv example.com

Replace example.com with your authorized target domain. The thread count controls the concurrency used by the tool; a higher value may increase speed but can also generate more requests and place additional load on the target or external services.

SubScraper Command-Line Options

• -s: Uses internet-based sources to discover subdomains.
• -b: Uses DNS brute forcing to discover subdomains.
• -o OUTFILE: Selects the output format, such as csv or txt.
• -t MAX_THREADS: Sets the maximum number of threads. The documented default is 10.
• -w SUBLIST: Specifies a custom subdomain wordlist.

Use the tool’s help output or repository documentation to confirm the accepted option syntax for your installed version.

Reviewing Enumeration Results

Discovered subdomains should be treated as leads rather than confirmed vulnerabilities. A responding HTTP service does not automatically indicate a security issue, and a CNAME record alone does not prove that a subdomain is vulnerable to takeover. Verify DNS configuration, the destination service, and ownership status before reporting a finding.

For broader reconnaissance, combine SubScraper results with other authorized discovery techniques and document which hosts resolve, which services respond, and which findings need manual validation. You can also explore our WhatWeb web scanner guide to learn about identifying technologies exposed by discovered web applications.

Project source: SubScraper on GitHub