Galer is a fast tool to fetch URLs from HTML attributes by crawl-in. Inspired by the @omespino Tweet, which is possible to extract src, href, url and action values by evaluating JavaScript through Chrome DevTools Protocol.
Installation
- From Binary
The installation is easy. You can download a prebuilt binary from releases page, unpack and run! or with
(sudo) curl -sSfL https://git.io/galer | sh -s — -b /usr/local/bin
- From Source
If you have go1.15+ compiler installed and configured:
GO111MODULE=on go get github.com/dwisiswant0/galer
- From GitHub
git clone https://github.com/dwisiswant0/galer
cd galer
go build .
(sudo) mv galer /usr/local/bin
Usage
- Basic Usage
- Simply, galer can be run with:
galer -u “http://domain.tld”
- Flags
galer -h
This will display help for the tool. Here are all the switches it supports.
| Flag | Description |
|---|---|
| -u, –url | Target to fetches (single target URL or list) |
| -e, –extension | Show only certain extensions (comma-separated, e.g. js,php) |
| -c, –concurrency | Concurrency level (default: 50) |
| –in-scope | Show in-scope URLs/same host only |
| -o, –output | Save fetched URLs output into file |
| -t, –timeout | Maximum time (seconds) allowed for connection (default: 60) |
| -s, –silent | Silent mode (suppress an errors) |
| -v, –verbose | Verbose mode show error details unless you weren’t use silent |
| -h, –help | Display its helps |
Examples
- Single URL
galer -u “http://domain.tld”
- URLs from list
galer -u /path/to/urls.txt
- From Stdin
cat urls.txt | galer
- In case you want to chained with other tools:
subfinder -d domain.tld -silent | httpx -silent | galer
Library
You can use galer as library.
go get github.com/dwisiswant0/galer/pkg/galer
For example:
package main
import (
"fmt"
"github.com/dwisiswant0/galer/pkg/galer"
)
func main() {
cfg := &galer.Config{
Timeout: 60,
}
cfg = galer.New(cfg)
run, err := cfg.Crawl("https://twitter.com")
if err != nil {
panic(err)
}
for _, url := range run {
fmt.Println(url)
}
}
TODOs
- Enable to set extra HTTP headers
- Provide randomly User-Agent
- Bypass headless browser
- Add exception for specific extensions
