Crawlee
Developer(s)Apify
Initial release13 July 2022 (2022-07-13)
Written inTypescript, Python
Operating systemWindows, macOS, Linux
TypeWeb crawler
LicenseGPL v3.0

Crawlee is a free and open-source web-crawling and browser automation library developed by Apify. The original TypeScript version was first released in 2022, with a Python version added in 2024.

Crawlee's architecture is built around modular crawlers responsible for extracting data from websites. The library follows a declarative programming approach, where users define crawling logic through a structured set of rules. Crawlee uses queues to manage requests; for each request, a specific function is executed to extract data or perform further processing.

Crawlee supports both headless browser sessions (via Playwright and other browser automation software) and plain HTTP request-based scraping.

It also provides various web-scraping-related utilities, such as a sitemap parser[1] and an automatic HTTP proxy manager.

Notable mentions of Crawlee's use in web-crawling projects include GPT Crawler by Builder.io[2] and various generative AI projects maintained by AWS Labs[3].

History

edit

The first stable TypeScript version was released in 2021 under the name Apify SDK[4]. This version offered both the open-source crawling framework and the proprietary storage implementation for use on the Apify platform.

In 2022, version v3.0.0 was released[5], renaming the library to Crawlee. This update made Crawlee independent of the Apify Platform, moving most of the Apify-specific features into a separate package (also named Apify SDK).

In 2024, a beta version of Crawlee for Python was released[6].

References

edit
  1. ^ "Release v3.7.0 · apify/crawlee". GitHub. Retrieved 22 September 2024.
  2. ^ "BuilderIO/gpt-crawler: Crawl a site to generate knowledge files to create your own custom GPT from a URL". GitHub. Retrieved 21 September 2024.
  3. ^ "awslabs/generative-ai-cdk-constructs: AWS Generative AI CDK Constructs are sample implementations of AWS CDK for common generative AI patterns". GitHub. Amazon Web Services - Labs. 20 September 2024. Retrieved 21 September 2024.
  4. ^ "Release v1.0.0 · apify/crawlee". GitHub.
  5. ^ "Release v3.0.0 · apify/crawlee". GitHub.
  6. ^ "Announcing Crawlee for Python: Now you can use Python to build reliable web crawlers | Crawlee · Build reliable crawlers. Fast". crawlee.dev. 5 July 2024.