Galer : A Fast Tool To Fetch URLs From HTML Attributes By Crawl-In

Galer is a fast tool to fetch URLs from HTML attributes by crawl-in. Inspired by the @omespino Tweet, which is possible to extract src, href, url and action values by evaluating JavaScript through Chrome DevTools Protocol.

Installation

• From Binary

The installation is easy. You can download a prebuilt binary from releases page, unpack and run! or with

(sudo) curl -sSfL https://git.io/galer | sh -s — -b /usr/local/bin

• From Source

If you have go1.15+ compiler installed and configured:

GO111MODULE=on go get github.com/dwisiswant0/galer

• From GitHub

git clone https://github.com/dwisiswant0/galer
cd galer
go build .
(sudo) mv galer /usr/local/bin

Usage

• Basic Usage
  • Simply, galer can be run with:

galer -u “http://domain.tld”

• Flags

galer -h

This will display help for the tool. Here are all the switches it supports.

Flag	Description
-u, –url	Target to fetches (single target URL or list)
-e, –extension	Show only certain extensions (comma-separated, e.g. js,php)
-c, –concurrency	Concurrency level (default: 50)
–in-scope	Show in-scope URLs/same host only
-o, –output	Save fetched URLs output into file
-t, –timeout	Maximum time (seconds) allowed for connection (default: 60)
-s, –silent	Silent mode (suppress an errors)
-v, –verbose	Verbose mode show error details unless you weren’t use silent
-h, –help	Display its helps

Examples

• Single URL

galer -u “http://domain.tld”

• URLs from list

galer -u /path/to/urls.txt

• From Stdin

cat urls.txt | galer

• In case you want to chained with other tools:

subfinder -d domain.tld -silent | httpx -silent | galer

Library

You can use galer as library.

go get github.com/dwisiswant0/galer/pkg/galer

For example:

package main

import (
	"fmt"

	"github.com/dwisiswant0/galer/pkg/galer"
)

func main() {
	cfg := &galer.Config{
		Timeout: 60,
	}
	cfg = galer.New(cfg)

	run, err := cfg.Crawl("https://twitter.com")
	if err != nil {
		panic(err)
	}

	for _, url := range run {
		fmt.Println(url)
	}
}

TODOs

• Enable to set extra HTTP headers
• Provide randomly User-Agent
• Bypass headless browser
• Add exception for specific extensions