BucketLoot – An Automated S3 Bucket Inspector

Thousands of S3 buckets are left open on the internet, which makes them an easy target for bad people who want to get private information from the files in these buckets that can be tied to a person or an organization.

There isn’t much study or software that can use these S3 buckets to find secret exposures and search text files for specific keywords or regular expression patterns.

BucketLoot is an automated S3 Bucket Inspector that can scan all the text files in an open S3 bucket on systems like AWS, DigitalOcean, etc. at the same time.

It checks the text files that are out in the open for:

• URLs, Domains, and Subdomains
• Exact terms | Regular Expression Patterns (supplied by the user)

The end user can even look for string-based keywords or provide custom regular expression patterns that can be matched with the contents of these text files.

BucketLoot is a great recon tool for both bug hunters and experienced pentesters because of all of these things.

The tool lets users save the output in a JSON format, which makes it easier to use the data as an input in a third-party product or platform.

Setup

1. Tool Setup

The tool is written in Go, so make sure to install it on your system before proceeding. The setup is pretty easy and straight forward.

Just follow the below steps in order to quickly install and get the binary working.

Firstly clone the repo to any directory/path of your liking,

git clone https://github.com/redhuntlabs/BucketLoot.git

Afer this, just run the following command in order to build the binary according to your environment.

go build

2. Credentials Setup (Optional)

In order to setup the tool for the full scan mode (optional), you need to modify the credentials.json file and add the credentials for the target platforms for whom you would like to run a full scan against.

At the moment, BucketLoot only supports AWS for running full mode scans, and we expect the release of another one or two modules for other platforms very soon.

AWS

In order to run the AWS module for the full scan mode, you need to generate the Access Key and Secret Key from the IAM Dashboard by heading to the users section, clicking on any user you would like to use for the tool, going to the Security Credentials tab, clicking on the create access key button, choosing “Third-party service” and finally creating the accees key.

Make sure that the user has the “AmazonS3FullAccess” permission policy attached, since its absence may lead to errors and issues with the scan.

Usage

1. Basic Scan

In order to run a basic scan without any extra flags, you just need to provide a target URL or target(s) file as an argument.

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/

                    OR

umair@redhuntlabs:~/bucketloot$ ./bucketloot targets.txt

2. Additional Flags

BucketLoot also offers several additional flags that can help customise your scan and get the right results that you need.

umair@redhuntlabs:~/bucketloot$ ./bucketloot -h
Usage of ./bucketloot:
  -full
     Go beyond the 1000 file scan limit [Requires additional setup!]
  -log-errors
     Log errors in final output
  -max-size string
     Maximum file size (in bytes)
  -save string
     Save tool output, should either end with .txt or .json [Default output file name is output.json]
  -search string
     Keyword(s) to look for during the scan. [Possible values -> keyword, keyword1:::keyword2, keywords.txt]
  -slow
     Set slow mode for the scan

-full

If you would like to go beyond the maximum 1000 files per bucket limit, you can run BucketLoot’s Full scan mode by setting up the target platform’s access credentials.

Currently we only support full scan mode for Amazon Web Services and expect to release the modules for other platforms very soon.

If for some reason, during the full scan, the tool encounters any authentication or permission issue, it will automatically switch back to scraping mode for that individual bucket.

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -full

-log-errors

BucketLoot allows users to save all the errors it encountered during the scan within the tool output. This can be helpful especially during the debugging process and can even help us to understand the reported issues better.

The flag creates an additional array named Errors within the JSON output.

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -log-errors

-max-size

Often users can encounter buckets that contain huge files. This can add up to the scan completion time and might not be an ideal scenario for systems with less bandwidth.

The -max-size flag allows users to provide the maximum file size which they would like to scan for (in bytes).

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -max-size 13521

-save

Saves the JSON output that your tool generates for users to go through or parse for further processing. The tool saves the output in a file named output.json by default if the flag is provided. User has the ability to provide custom output file names, either in .txt or .json format.

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -save

                                    OR
                                
umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -save myscan.json

-search

Users can use the -search flag in order to query for specific keywords or regular expressions within the file contents from an exposed storage bucket. There are several ways through which the keywords can be passed to the tool.

1. Search an individual keyword/RegEx query

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -search 'admin'

2. Search for multiple keywords/RegEx queries (using ::: as a separator)

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -search 'admin:::login:::password:::API:::.*dev-instance'

3. Search for multiple keywords/RegEx queries (using a .txt file containing the list)

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -search queries.txt

-slow

Designed for systems with low network bandwidth where the consistency of results is important, the -slow flag allows to run all the scans sequentially instead of concurrently [Fast mode] which is the default behavious of the tool.

Although this would definitely increase the overall scan time, the tool will provide consistent results while also making sure that it can run hassle-free locally.

umair@redhuntlabs:~/bucketloot$ ./bucketloot https://myvulninstance.s3.amazonaws.com/ -slow

3. Output

BucketLoot returns a JSON output at the end of every scan. The tool has the following structure:

type bucketLootOpStruct struct {
 Results []struct {
  BucketUrl string `json:"bucketUrl"`
  Assets    []struct {
   URL       string `json:"url"`
   Domain    string `json:"domain"`
   Subdomain string `json:"subdomain"`
  } `json:"Assets"`
  Secrets []struct {
   Name string `json:"name"`
   URL  string `json:"url"`
  } `json:"Secrets"`
  Keywords []struct {
   URL     string `json:"url"`
   Keyword string `json:"keyword"`
   Type    string `json:"type"`
  } `json:"Keywords,omitempty"`
 } `json:"Results"`
 Version string   `json:"version"`
 Scanned []string `json:"Scanned"`
 Skipped []string `json:"Skipped"`
 Errors  []string `json:"Errors,omitempty"`
}

The Keywords and Errors array only show up when their respective flags are provided as an input and thus omitted if empty.

Here, Results is an array storing the scan data for every individual misconfigured bucket that was scanned and from whom results were derivedd. The unique identifier here is the BucketURL field.

Outside of the array we have the version field containing the tool version, Scanned is an array showing the URLs that successfully got scanned, Skipped stores all the URLs which werne’t scanned either because they were not valid S3 endpoints, were private or had some issue while making a request.

The errors array (optional) stores all the errors caught during the scan as raw strings.

Here’s an example output for a basic scan to give you a glimpse of how the tool works,

➜  bucketloot git:(master) ✗ ./bucketloot https://bucketloot-testing.blr1.digitaloceanspaces.com/

,.--'''''''''--.,  ____             _        _   _                 _   
(\'-.,_____,.-'/) |  _ \           | |      | | | |               | |  
 \\-.,_____,.-//  | |_) |_   _  ___| | _____| |_| |     ___   ___ | |_ 
 ;\\         //|  |  _ <| | | |/ __| |/ / _ \ __| |    / _ \ / _ \| __|
 | \\  ___  // |  | |_) | |_| | (__|   <  __/ |_| |___| (_) | (_) | |_ 
 |  '-[___]-'  |  |____/ \__,_|\___|_|\_\___|\__|______\___/ \___/ \__|
 |             |                                                       
 |             |  An Automated S3 Bucket Inspector                                             
 |             |  Developed by Umair Nehri (@umair9747) and Owais Shaikh (@4f77616973)             
 ''-.,_____,.-''                                                        
 

 
Processing arguments...

 
Discovered a total of 3 bucket files...
Total bucket files of interest: 3

 
Starting to scan the files... [FAST]
Discovered SECRET[AWS Access Key ID Value] in https://bucketloot-testing.blr1.digitaloceanspaces.com/credentials.json
Discovered SECRET[AWS Access Key ID] in https://bucketloot-testing.blr1.digitaloceanspaces.com/credentials.json
Discovered URL(s) in https://bucketloot-testing.blr1.digitaloceanspaces.com/credentials.json
Discovered SECRET[Google API key] in https://bucketloot-testing.blr1.digitaloceanspaces.com/config.php
Discovered URL(s) in https://bucketloot-testing.blr1.digitaloceanspaces.com/dashboard.html
Discovered URL(s) in https://bucketloot-testing.blr1.digitaloceanspaces.com/config.php

{
  "Results": [
    {
      "bucketUrl": "https://bucketloot-testing.blr1.digitaloceanspaces.com/",
      "Assets": [
        {
          "url": "https://blackhat.com/",
          "domain": "blackhat.com",
          "subdomain": ""
        },
        {
          "url": "https://certificates.blackhat.com/",
          "domain": "blackhat.com",
          "subdomain": "certificates.blackhat.com"
        },
        {
          "url": "https://google.com/login",
          "domain": "google.com",
          "subdomain": ""
        },
        {
          "url": "https://firecat.toolswatch.org/",
          "domain": "toolswatch.org",
          "subdomain": "firecat.toolswatch.org"
        },
        {
          "url": "https://www.google.com",
          "domain": "google.com",
          "subdomain": "www.google.com"
        },
        {
          "url": "http://example.com/dashboard",
          "domain": "example.com",
          "subdomain": ""
        },
        {
          "url": "https://www.openai.com",
          "domain": "openai.com",
          "subdomain": "www.openai.com"
        },
        {
          "url": "https://www.example.com/admin",
          "domain": "example.com",
          "subdomain": "www.example.com"
        },
        {
          "url": "https://www.example.com/login.php",
          "domain": "example.com",
          "subdomain": "www.example.com"
        },
        {
          "url": "https://www.example.com/reset-password",
          "domain": "example.com",
          "subdomain": "www.example.com"
        },
        {
          "url": "https://example.com/api/endpoint",
          "domain": "example.com",
          "subdomain": ""
        }
      ],
      "Secrets": [
        {
          "name": "AWS Access Key ID Value",
          "url": "https://bucketloot-testing.blr1.digitaloceanspaces.com/credentials.json"
        },
        {
          "name": "AWS Access Key ID",
          "url": "https://bucketloot-testing.blr1.digitaloceanspaces.com/credentials.json"
        },
        {
          "name": "Google API key",
          "url": "https://bucketloot-testing.blr1.digitaloceanspaces.com/config.php"
        }
      ]
    }
  ],
  "version": "1.0",
  "Scanned": [
    "https://bucketloot-testing.blr1.digitaloceanspaces.com/"
  ],
  "Skipped": null
}