List of free crawl stuff for developerππ½
If you're looking for all programming tools around crawl (not just free). Check this page List of crawl Programming Tools

Octoparse
A Free, Simple, and Powerful Web Scraping Tool. Automate Data Extraction from websites within clicks without coding.
Tag: crawl, scraping
Unlimited pages per crawl Unlimited computers 10,000 records per export 2 concurrent local run 10 Crawlers

Puppeteer
Most things that you can do manually in the browser can be done using Puppeteer
Tag: screenshots, ui-testing, crawl