Gospider – Web Crawling and Content Discovery Tool

Gospider - Web Crawling and Content Discovery Tool

Gospider Fast Web Spider Written in Go

Gospider is a tool to perform website crawling.

Features

  • Fast web crawling
  • Brute force and parse sitemap.xml
  • Parse robots.txt
  • Generate and verify link from JavaScript files
  • Link Finder
  • Find AWS-S3 from response source
  • Find subdomains from response source
  • Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault
  • Format output easy to Grep
  • Support Burp input
  • Crawl multiple sites in parallel
  • Random mobile/web User-Agent

Installation

GO111MODULE=on go install github.com/jaeles-project/gospider@latest

Demo Video

Example commands

Crawling single site

gospider -s “https://google.com/” -o output -c 10 -d 1

Crawling with site list

gospider -S sites.txt -o output -c 10 -d 1

Also get URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)

gospider -s “https://google.com/” -o output -c 10 -d 1 –other-source

Blacklist url/file extension.

P/s: gospider blacklisted .(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico) as default

gospider -s “https://google.com/” -o output -c 10 -d 1 –blacklist “.(woff|pdf)”


You May Also Like

Leave a Reply

Your email address will not be published. Required fields are marked *

seventeen + eight =