This project on GitHub is close to 20,000 stars. It is a web crawling framework written in Node.js (with a Python version as well).
Its biggest feature is that the crawling behavior mimics a real person, making it harder to be blocked by websites’ anti-crawling mechanisms.
For example, if you want to scrape dynamic content like TikTok or YouTube, it can handle that.
It also comes with a CLI tool that generates the project structure with one click, providing you with ready-made templates.
It supports HTTP requests and can work with browser automation tools like Playwright to scrape complex web pages. It has built-in proxy switching, request queuing, error retrying, data storage, and more…
In a word: it is very suitable for those who want to write crawlers but don’t want to reinvent the wheel.
Source