ProxyBroker is an open-source tool that can asynchronously find public proxies from multiple sources and check them simultaneously.
Features
- Find more than 7,000 proxy servers from about 50 sources.
-
Supported protocols: HTTP(S), SOCKS4 /
5. Connects using the CONNECT method to ports 80 and 23 (SMTP). - Proxies can be filtered by type, anonymity level, response time, country, and status in DNSBL.
- Acts as a proxy server, distributing incoming requests to external proxies. Automatic proxy rotation.
- All proxies are checked for support of Cookies and Referer (and POST requests if needed).
- Automatically removes duplicate proxies.
- Asynchronous.
Installation
Install the latest stable version from PyPI:
$ pip install proxybroker
The latest development version can be installed directly from GitHub:
$ pip install -U git + https：//github.com/constverum/ProxyBroker.git
Usage
CLI Examples
Find
Find and display 10 HTTP(S) proxies from the US with a high level of anonymity:
$ proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10
Grab
Find 10 US proxies and save them to a file (without checking):
$ proxybroker grab --countries US --limit 10 --outfile ./proxies.txt
Serve
Run a local proxy server that distributes incoming requests across a pool of discovered HTTP(S) proxies with a high anonymity level:
$ proxybroker serve --host 127 .0.0.1 --port 8888 --types HTTP HTTPS --lvl high
Note
Run for more information about the available options.proxybroker --help
Run to get more information about the command.proxybroker <command> --help
Basic Code Example
Find and display 10 working HTTP(S) proxies:
import asyncio
from proxybroker import Broker
async def show(proxies):
while True:
proxy = await proxies.get()
if proxy is None: break
print('Proxy found: %s' % proxy)
proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
broker.find(types=['HTTP', 'HTTPS'], limit=10),
show(proxies))
loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)
TODO
- Check ping, response time, and data transfer speed
- Check website access permissions (Google, Twitter, etc.), or even your own custom URLs
- Uptime information
- Checksum of returned data
- Support proxy validation
- Find the outgoing IP for chained proxies
- Ability to specify the address of proxies without a port (try connecting on the default port)
Contributing
- Fork it: https://github.com/constverum/ProxyBroker/fork
- Create your feature branch: git checkout -b my-new-feature
- Commit your changes: git commit -am ‘Add some feature’
- Push to the branch: git push origin my-new-feature
- Submit a pull request!
License
Licensed under the Apache License, Version 2.0
This product includes GeoLite2 data created by MaxMind, available from http://www.maxmind.com.