Distributed - Asynchronous - Easy-to-use

An Easy-to-use and Fast Python Spider Framework

Overview

• AirSpider is a high-performance asynchronous crawler framework for developers
• Based on Redis: task distribution, task deduplication, and distributed ☁️

Requirements

• Python 3.6
• Works on Linux, Windows, macOS

Features

• Quick to Start
• Low Coupling
• High Cohesion
• Easy Expansion
• Orderly Workflow

Installation

# For Linux && MacOS
pip3 install airspider


# For Windows
pip3 install airspider

Documents

• Introduction：Introduction to AirSpider
• Tutorials： Quick to program a Spider By AirSpider
• Plugins：Program extensions for AirSpider

Topics

• Item：Define Target in HTML
• Selector：Select Target from HTML
• Request：Request Part
• Response：Response Part
• Middleware：Make Airspider Support to Extensions
• Spider：Entrance for AirSpider

TODO

• Complete Plugins of Redis
• Complete Distributed Architecture

Contributing

AirSpider🕷️ is still under Developing

Feel free to open issues💬 and pull requests

• Report or Fix bugs
• Build Powerful plugins
• Make documentation Better
• Add Examples of Spiders