pip install Protego==0.1.16

Pure-Python robots.txt parser with support for modern conventions

Source
Among top 1% packages on PyPI.
Over 1.4M downloads in the last 90 days.

Commonly used with Protego

Based on how often these packages appear together in public requirements.txt files on GitHub.

hyperlink

A featureful, immutable, and correct URL for Python.

Automat

Self-service finite-state machines for the programmer on the go.

constantly

Symbolic constants in Python

incremental

None

parsel

Parsel is a library to extract data from HTML and XML using XPath and CSS selectors

itemadapter

Common interface for data container classes

PyDispatcher

Multi-producer-multi-consumer signal dispatching mechanism

Twisted

An asynchronous networking framework written in Python

itemloaders

Base library for scrapy's ItemLoader

scraper

Configurable Python Web Scraper

w3lib

Library of web-related functions

queuelib

Collection of persistent (disk-based) queues

Scrapy

A high-level Web Crawling and Web Scraping framework

PyHamcrest

Hamcrest framework for matcher objects

html2markdown

Conservatively convert html to markdown

scrapyd-client

A client for scrapyd

filepath

Object-oriented filesystem path representation.

attrs

Classes Without Boilerplate

autobahntestsuite

AutobahnTestSuite - WebSocket/WAMP protocol implementation test suite.

Version usage of Protego

Proportion of downloaded versions in the last 3 months (only versions over 1%).

0.1.16

99.69%