Requests: gevent + requests blocks when using socks

Created on 10 Feb 2017  ·  3Comments  ·  Source: psf/requests

I use python(2.7.6),gevnet(1.1.2),requests(2.11.1) to make http requests concurrent, and it works good. But when I add socks proxy to requests, it blocks.

This is my code:

import time
import requests
import logging
import click

import gevent
from gevent import monkey
monkey.patch_all()

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('test')

#socks proxy
user = MY_SOCKS_PROXY_USERNAME
password = MY_SOCKS_PROXY_PASSWORD
host = MY_SOCKS_PROXY_HOST
port = MY_SOCKS_PROXY_PORT
proxies = {
        'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
        'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
}
url = 'https://www.youtube.com/user/NBA'


def fetch_url(i,with_proxy):
    while True:
        logger.warning('thread %s fetch url'%i)
        try:
            if with_proxy:
                res = requests.get(url,proxies=proxies, timeout=5)
            else:
                res = requests.get(url, timeout=5)
        except Exception as e:
            logger.error(str(e))
            continue

        logger.warning(res.status_code)

def do_other_thing():
    while True:
        logger.warning('do other thing...')
        time.sleep(1)

@click.command()
@click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
def run(with_proxy):
    if with_proxy:
        logger.warning('with proxy......')
    else:
        logger.warning('without proxy......')
    ts = []
    ts.append(gevent.spawn(do_other_thing))
    for i in xrange(3):
        ts.append(gevent.spawn(fetch_url,i,with_proxy))
    gevent.joinall(ts)

if __name__=='__main__':
    run()

These pictures show the result.

run with proxy
run without proxy

With proxy, do_other_thing will blocks before fetch_url done.
Without proxy, it works good. (timeout error occurs, because of the GFW)

Can anyone help me solving this problem? Thanks very much!

Most helpful comment

My best guess is that you are monkeypatching too late. When using gevent you must monkeypatch first, before importing anything else. Doing it later runs the risk of breaking anything that has done a from import from the things that gevent monkeypatches.

Can you try moving your import and monkeypatch to the very top of your file before doing anything else?

All 3 comments

My best guess is that you are monkeypatching too late. When using gevent you must monkeypatch first, before importing anything else. Doing it later runs the risk of breaking anything that has done a from import from the things that gevent monkeypatches.

Can you try moving your import and monkeypatch to the very top of your file before doing anything else?

Yeah, ' moving your import and monkeypatch to the very top of your file ' solved my problem! Thank you very much!

This suggestion also solves my project's problem.
I hava 3 files called A,B,C, and run sequence is A->B->C. I use gevent in B, so I add monkeypatch in B, but i blocks with requests+socks. Now I add monkeypatch at the very top of A, it works great now! Hope this will help others!

Thank you again!

Happy to help!

Was this page helpful?
0 / 5 - 0 ratings

Related issues

remram44 picture remram44  ·  4Comments

ReimarBauer picture ReimarBauer  ·  4Comments

cnicodeme picture cnicodeme  ·  3Comments

tiran picture tiran  ·  3Comments

JimHokanson picture JimHokanson  ·  3Comments