🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

Scrapy, a fast high-level web crawling & scraping framework for Python.

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Elegant Scraper and Crawler Framework for Golang

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

🕵️‍♂️ Collect a dossier on a person by username from thousands of sites

Pythonic HTML Parsing for Humans™

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

A scalable web crawler framework for Java.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

List of libraries, tools and APIs for web scraping and data processing.

Tabula is a tool for liberating data tables trapped inside PDF files

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Declarative web scraping

🦊 Anti-detect browser

Distributed crawler powered by Headless Chrome

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Self-hosted webscraper.

Swiss-army tool for scraping and extracting data from online assets, made for hackers

Mechanize is a ruby library that makes automated web interaction easy.

Collection of useful data science topics along with articles, videos, and code

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

Up-to-date simple useragent faker with real world database

Snoop — инструмент разведки на основе открытых данных (OSINT world)

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

Scrape Facebook public pages without an API key

A browser testing and web crawling library for PHP and Symfony