Dr_Robot : Tool Used To Enumerate The Subdomains Associated With A Company

Dr_Robot is a tool for Domain Reconnaissance and Enumeration. By utilizing containers to reduce the overhead of dealing with dependencies, inconsistency across operating sytems, and different languages, Dr. ROBOT is built to be highly portable and configurable.

Use Case: Gather as many public facing servers that a target organization possesses. Querying DNS resources enables us to quickly develop a large list of possible targets that you can run further analysis on.

Note: Dr. ROBOT is not just a one trick pony. You can easily customize the tools that are used gather information, so that you can enjoy the benefits of using latest and greatest along with your battle tested favorites.

Install and Run

Inspect

Upload Slack

Dump DB

Output

Serve

Also Read – DetExploit : OSS Vulnerability Scanner for Windows Platform

Command Examples

• Run gather using Sublist3r and Aquatone and Shodanpython drrobot.py example.domain gather -sub -aqua -shodan
• Run gather using Sublist3r with Proxypython drrobot.py --proxy http://some.proxy:port example.domain gather -sub
• Run inspect using Eyewitnesspython drrobot.py example.domain inspect -eye
• Run inspect using httpscreenshot and grabbing headerspython drrobot.py example.domain inspect -http -headers
• Run upload using Mattermost/Slackpython drrobot.py example.domain upload -matter

Main

usage: drrobot.py [-h] [–proxy PROXY] [–dns DNS] [–verbose]
[–dbfile DBFILE]
{gather,inspect,upload,rebuild,dumpdb,output,serve}
…

Docker DNS recon tool
positional arguments:
{gather,inspect,upload,rebuild,dumpdb,output,serve}
gather Run scanners against a specified domain and gather the associated
systems. You have the option to run using any
docker_buildfiles/webtools included in your config.
inspect Run further tools against domain information gathered
from the gather step. Note: you must either supply a file
which contains a list of IP/Hostnames, or the targeted
domain must have a db file in the dbs folder
upload Upload recon data to Mattermost. Currently only works
with afolder that contain PNG images.
rebuild Rebuild the database with additional files/all files
from the previous runtime
dumpdb Dump the database of ip, hostname, and banners to a text
file
output Generate output in specified format. Contains all
information from scans (images, headers, hostnames,
ips)
serve Serve database file in docker container using django

optional arguments:
-h, –help show this help message and exit
–proxy PROXY Proxy server URL to set DOCKER http_proxy too
–dns DNS DNS server to add to resolv.conf of DOCKER containers
–verbose Display verbose statements
–dbfile DBFILE Specify what db file to use for saving data too

Gather

usage: drrobot.py domain gather [-h] [-aqua] [-sub] [-brute] [-sfinder]
[-knock] [-amass] [-recong] [-shodan] [-arin]
[-hack] [-dump] [-virus] [–ignore IGNORE]
[–headers]

positional arguments:
domain Domain to run scan against

optional arguments:
-h, –help Show this help message and exit
-aqua, –Aquatone AQUATONE is a set of tools for performing
reconnaissance on domain names
-sub, –Sublist3r Sublist3r is a python tool designed to enumerate
subdomains of websites using OSINT
-brute, –Subbrute SubBrute is a community driven project with the goal
of creating the fastest, and most accurate subdomain
enumeration tool.
-sfinder, –Subfinder
SubFinder is a subdomain discovery tool that discovers
valid subdomains for websites by using passive online
sources
-knock, –Knock Knockpy is a python tool designed to enumerate
subdomains on a target domain through a wordlist
-amass, –Amass The OWASP Amass tool suite obtains subdomain names by
scraping data sources, recursive brute forcing,
crawling web archives, permuting/altering names and
reverse DNS sweeping.
-recon, –Reconng Recon-ng is a full-featured Web Reconnaissance
framework written in Python. DrRobot utilizes several
of the recon/hosts-domain modules in this framework.
-shodan, –Shodan Query SHODAN for publicly facing sites of given domain
-arin, –Arin Query ARIN for public CIDR ranges. This is better as a
brute force option as the ranges
-hack, –HackerTarget
This query will display the forward DNS records
discovered using the data sets outlined above.
-dump, –Dumpster Use the limited response of DNSDumpster. Requires API
access for better results.
-virus, –VirusTotal Utilize VirusTotal’s Observer Subdomain Search
–ignore IGNORE Space seperated list of subnets to ignore
–headers If headers should be scraped from ip addresses
gathered

INSPECT

usage: main.py inspect [-h] [-httpscreen] [-eye] [–proxy PROXY] [–dns DNS]
[–file FILE]
positional arguments:
domain Domain to run scan against
optional arguments:
-h, –help Show this help message and exit
-httpscreen, –HTTPScreenshot
Post enumeration tool for screen grabbing websites.
All images will be downloaded to an output file:
httpscreenshot.tar and unpacked httpscreenshots
-eye, –Eyewitness Post enumeration tool for screen grabbing websites.
All images will be downloaded to outfile:
Eyewitness.tar and unpacked in Eyewitness
–proxy PROXY Proxy server URL to set for DOCKER http_proxy
–dns DNS DNS server for the resolv.conf of DOCKER containers
–file FILE (NOT WORKING) File with hostnames to run further
inspection on

UPLOAD

usage: drrobot.py domain upload [-h] [-matter] [-slack] [–filepath FILEPATH]

positional arguments:
domain Domain to run scan against

optional arguments:
-h, –help Show this help message and exit
-matter, –Mattermost Mattermost server to upload findings to
Mattermost server
-slack, –Slack Slack server
–filepath FILEPATH Filepath to the folder containing images to upload.
This is relative to the domain specified. By default,
this will be the path to the output folder

Rebuild

usage: drrobot.py rebuild [-h] [-f [FILES [FILES …]]]

optional arguments:
-h, –help Show this help message and exit
-f [FILES [FILES …]], –files [FILES [FILES …]]
Additional files to supply in addition to the ones in the
config file

Dumpdb

usage: drrobot.py dumpdb [-h]

positional arguments:
domain Domain to run scan against

optional arguments:
-h, –help Show this help message and exit

OUTPUT

usage: drrobot.py domain output [-h] [–output OUTPUT] {json,xml}

positional arguments:
{json,xml} Generate json file under outputs folder (format)
domain Domain to dump output of

optional arguments:
-h, –help Show this help message and exit
–output OUTPUT Alternative location to create output file

Serve

usage: drrobot.py domain serve [-h]

optional arguments:
-h, –help show this help message and exit

Configurations

This tool is highly dependent on the configuration you provide it. Provided for you is a default_config.json that you can use as a simple template for your user_config.json. Most of the configurations under Scanners are done for you and can be used as is. Note the use of default in this and other sections.

default : specifies a Docker or Ansible instance. Make sure you adjust configurations according to their usage.

• Docker Configuration Requirements
  • Example:

“Sublist3r”: {
“name”: “Sublist3r”,
“default” : true,
“mode” : “DOCKER”,
“docker_name”: “sub”,
“network_mode”: “host”,
“default_conf”: “docker_buildfiles/Dockerfile.Sublist3r.tmp”,
“active_conf”: “docker_buildfiles/Dockerfile.Sublist3r”,
“description”: “Sublist3r is a python tool designed to enumerate subdomains of websites using OSINT”,
“src”: “https://github.com/aboul3la/Sublist3r”,
“output”: “/root/sublist3r”,
“output_folder”: “sublist3r”
},

• • name: Identifiable name for the program/utility you are using
  • default : (Disabled for now)
  • mode : DOCKER (uses docker container with this tool when chosen)
  • docker_name : What the docker image name will be when running docker images
  • network_mode : Network mode to use when creating container. Host uses the host network
  • default_conf : Template Dockerfile to build form
  • active_conf : Target specific configuration that will be used during runtime
  • description : Description of tool (optional)
  • src : Where the tool comes from (optional)
  • output : Location of output on the docker container. Can be hardcoded into Dockerfiles for preference
  • output_folder : Location under the outputs/target folder where output for target will be stored

• Ansible Configuration Requirements
  • Example

“HTTPScreenshot”: {
“name” : “HTTPScreenshot”,
“short_name” : “http”,
“mode” : “ANSIBLE”,
“ansible_arguments” : {
“config” : “$config/httpscreenshot_play.yml”,
“flags”: “-e ‘$extra’ -i ansible_plays/inventory.yml”,
“extra_flags”:{
“1” : “variable_host=localhost”,
“2” : “infile=$infile/aggregated/aggregated_protocol_hostnames.txt”,
“3” : “outfile=$outfile/httpscreenshots.tar”,
“4” : “outfolder=$outfile/httpscreenshots”,
“5” : “variable_user=bitnami”
}
},
“description” : “Post enumeration tool for screen grabbing websites. All images will be downloaded to outfile: httpscreenshot.tar and unpacked httpscreenshots”,
“output” : “/tmp/output”,
“infile” : “/tmp/output/aggregated_protocol_hostnames.txt”,
“enabled” : false

• name: Identifiable name for the program/utility you are using
• default : (Disabled for now)
• mode : ANSIBLE (uses Ansible with this tool when chosen)
• ansible_arguments : Json configuration for specific informaiton
  • config : playbook to use ($config keyword is replaces for full path to file when issuing ansible playbook command)
  • flags : specifies extra flags to be used with the ansible command (specifically useful for any extra flags you would like to use)
  • extra flags : key does not matter so long as it is different from any other key. These extra flags will all be applied to the ansible file in question
• description : Description of tool (optional)
• src : Where the tool comes from (optional)
• output : Where output will be stored on the external file system
• infile : (Unique for certain modules) what files this program will use as input to the program. In this case you will notice that it searches /tmp/output for aggregated_protocol_hostnames.txt. This file is supplied from the above extra flags option.

• Web Modules
  • Example:

“HackerTarget” :
{
“short_name” : “hack”,
“class_name” : “HackerTarget”,
“default” : false,
“description” : “This query will display the forward DNS records discovered using the data sets outlined above.”,
“api_call_unused” : “https://api.hackertarget.com/hostsearch/?q=example.com”,
“output_file” : “hacker.txt”
},

• short_name : quick reference name for use in CLI
• class_name : this must match the name you specify for a given class under the respective module name
  • The reason behind this results from the loading of modules at runtime which requires the use of importlib. This will load the respective class from the classname provided via the CLI options.
• default : false (Disabled for now)
• api_call_unused : (Old, may be used later…)
• description : Description of tool (optional)

• Serve Module:
  • Example

“Serve” : {
“name” : “Django”,
“command” : “python manage.py runserver 0.0.0.0:8888”,
“docker_name”: “django”,
“network_mode”: “host”,
“default_conf”: “serve_api/Dockerfile.Django.tmp”,
“active_conf”: “serve_api/Dockerfile.Django”,
“description” : “Django container for hosting database”,
“ports” : {
“8888” : “8888”
}
}

• command: Command to start server on Docker container (Note: For now only using docker)
• docker_name : What the docker image name will be when running docker images
• network_mode : Network mode to use when creating container. Host uses the host network
• default_conf : Template Dockerfile to build form
• active_conf : Target specific configuration that will be used during runtime
• description : Description of tool (optional)
• ports: Port mapping of localhost to container for docker

Example Configuration For WebTools

Under configs, you will find a default_config that contains a majority of the default scanners you can use. If you wish to extend upon the WebTools list just follow these steps:

• Add the new tool to the user_config.json

{
“WebTools”:
{
“NewTool” :
{
“short_name”: “ntool”,
“class_name”: “NewTool”,
“description” : “NewTool description”,
“output_file” : “newtool.txt”,
“api_key” : null,
“endpoint” : null,
“username” : null,
“password” : null
},

• Open src/web_resources.py and make a class with the class_name specified in the previous step. MAKE SURE IT MATCHES EXACTLY

class NewTool(WebTool):
def init(self, kwargs): super().init(kwargs)
….
def do_query(self):
…. do the query …
store results in
self.results

Example Configurations For Docker Containers

Under configs, you will find a default_config which contains a majority of the default scanners you can utilize. If you wish to extend upon the Scanners list just follow these steps:

• Add the json to the config file (user if generated).

“Scanners” : {
…
“NewTool”: {
“name”: “NewTool”,
“default” : true,
“mode” : DOCKER,
“docker_name”: “ntool”,
“network_mode”: “host”,
“default_conf”: “docker_buildfiles/Dockerfile.NewTool.tmp”,
“active_conf”: “docker_buildfiles/Dockerfile.NewTool”,
“description”: “NewTool is an awesome tool for domain enumeration”,
“src”: “https://github.com/NewTool”,
“output”: “/home/newtool”,
“output_file”: “NewTool.txt”
},
…
}

• Note network_mode is an option specifically for docker containers. It is implementing the --network flag when using docker

Example Ansible Configuration

Under configs you will find a default_config which contains a majority of the default scanners you can have. For this step however, we will be looking at configuring an inspection too Eyewitness for utilization with Ansible.

• Add the json to the config file (user if generated).

“Enumeration” : {
“Eyewitness”: {
“name” : “Eyewitness”,
“short_name” : “eye”,
“docker_name” : “eye”,
“mode” : “ANSIBLE”,
“network_mode”: “host”,
“default_conf” : “docker_buildfiles/Dockerfile.Eyewitness.tmp”,
“active_conf” : “docker_buildfiles/Dockerfile.Eyewitness”,
“ansible_arguments” : {
“config” : “$config/eyewitness_play.yml”,
“flags”: “-e ‘$extra’ -i ansible_plays/inventory”,
“extra_flags”:{
“1” : “variable_host=localhost”,
“2” : “variable_user=root”,
“3” : “infile=$infile/aggregated_protocol_hostnames.txt”,
“4” : “outfile=$outfile/Eyewitness.tar”,
“5” : “outfolder=$outfile/Eyewitness”
}
},
“description” : “Post enumeration tool for screen grabbing websites. All images will be downloaded to outfile: Eyewitness.tar and unpacked in Eyewitness”,
“output” : “/tmp/output”,
“infile” : “/tmp/output/aggregated/aggregated_protocol_hostnames.txt”,
“enabled” : false
},
}

• As you can see, this has a few items that may seem confusing at first, but will be clarified here:

• mode: Allows you to specify how you want to deploy a tool you want to use. Currently DOCKER or ANSIBLE are the only available methods to deploy.
• All options outside of ansible_configuration will be ignored when developing for ANSIBLE.
• Options under ansible_arguments
  • config: specify which playbook to use
  • flags: which flags to pass to the ansible-playbook command. With the exception of the $extra flag, you can add anything you would like to be done uniquely here.
  • extra_flags : this corresponds to the $extra flag as seen above. This will be used to populate variables that you input into your playbook. You can use this to supply command line arguments when utilizing ansible and Dr. Robot in order to add files and other utilities to your script.
    • variable_host : hostname alias found in the inventory file
    • variable_user : user to login as on the variable_host machine
    • infile: file to be used with the tool above. Eyewitness requires hostnames with the format https://some.url, hence aggregated_protocol_hostnames.txt
    • Note the use of the prefix $infile– these names all match as they are placeholders for the default locations that $infile corresponds to in outputs/target_name/aggregated
    • If you have a file in another location you can just specify the entire path without any errors occurring.
    • outfile : The output file location
    • As with the above infile $outfile in the name is just a key to the location outputs/target_name/
    • You may specify a hard coded path for other use. Just remember the location for uploading or other processing with Dr. Robot
    • outfolder : The output folder to unpack/download files too
    • As with the above infile $outfile in the name is just a key to the location outputs/target_name/
    • This is a special case for Eyewitness and HttpScreenshot, which you can see in their playbooks. They generate a lot of files and rather than download each individually having them pack up the files as a step in the playbook and then unpacking allows for some integrity.
    • A quick example below shows how we use the extra_flags to supply the hostname to the playbook for ansible.

Slack

Please check the following for a guide on how to setup your Python bot for messaging.

https://github.com/slackapi/python-slackclient

SQLite DB file schema

Table Data: | domainid | INTEGER | PRIMARY KEY | ——– | ——- | | ip | VARCHAR | | hostname | VARCHAR | | headers | VARCHAR | | http_headers | TEXT | | https_headers| TEXT | | domain | VARCHAR | FOREIGN KEY

Table Domain: | domain | VARCHAR | PRIMARY KEY | ——– | ——- |

Serve

As is often the case, having an API can be nice for automation purposes. Under the serve-api folder, there is a simple Django server implementation that you can stand up locally or serve via Docker. In order to serve the datak, you need to copy your database folder to the root directory of serve-api and rename the file to drrobot.db. If you would like to use an alternative name, simply change the name in the Django serve____-api/drrobot/drrobot/settings.py.