Gospider Fast Web Spider Written in Go
Gospider is a tool to perform website crawling.
Features
- Fast web crawling
- Brute force and parse sitemap.xml
- Parse robots.txt
- Generate and verify link from JavaScript files
- Link Finder
- Find AWS-S3 from response source
- Find subdomains from response source
- Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault
- Format output easy to Grep
- Support Burp input
- Crawl multiple sites in parallel
- Random mobile/web User-Agent
Installation
GO111MODULE=on go install github.com/jaeles-project/gospider@latest
Demo Video
Example commands
Crawling single site
gospider -s “https://google.com/” -o output -c 10 -d 1
Crawling with site list
gospider -S sites.txt -o output -c 10 -d 1
Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)
gospider -s “https://google.com/” -o output -c 10 -d 1 –other-source
Blacklist url/file extension.
P/s: gospider blacklisted .(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico) as default
gospider -s “https://google.com/” -o output -c 10 -d 1 –blacklist “.(woff|pdf)”
