Requests: ์–‘๋ง์„ ์‚ฌ์šฉํ•  ๋•Œ gevent + ์š”์ฒญ ์ฐจ๋‹จ

์— ๋งŒ๋“  2017๋…„ 02์›” 10์ผ  ยท  3์ฝ”๋ฉ˜ํŠธ  ยท  ์ถœ์ฒ˜: psf/requests

python(2.7.6),gevnet(1.1.2),requests(2.11.1)๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ http ์š”์ฒญ์„ ๋™์‹œ์— ์ˆ˜ํ–‰ํ•˜๊ณ  ์ž˜ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ์š”์ฒญ์— ์–‘๋ง ํ”„๋ก์‹œ๋ฅผ ์ถ”๊ฐ€ํ•˜๋ฉด ์ฐจ๋‹จ๋ฉ๋‹ˆ๋‹ค.

์ด๊ฒƒ์€ ๋‚ด ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.

import time
import requests
import logging
import click

import gevent
from gevent import monkey
monkey.patch_all()

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('test')

#socks proxy
user = MY_SOCKS_PROXY_USERNAME
password = MY_SOCKS_PROXY_PASSWORD
host = MY_SOCKS_PROXY_HOST
port = MY_SOCKS_PROXY_PORT
proxies = {
        'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
        'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
}
url = 'https://www.youtube.com/user/NBA'


def fetch_url(i,with_proxy):
    while True:
        logger.warning('thread %s fetch url'%i)
        try:
            if with_proxy:
                res = requests.get(url,proxies=proxies, timeout=5)
            else:
                res = requests.get(url, timeout=5)
        except Exception as e:
            logger.error(str(e))
            continue

        logger.warning(res.status_code)

def do_other_thing():
    while True:
        logger.warning('do other thing...')
        time.sleep(1)

@click.command()
@click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
def run(with_proxy):
    if with_proxy:
        logger.warning('with proxy......')
    else:
        logger.warning('without proxy......')
    ts = []
    ts.append(gevent.spawn(do_other_thing))
    for i in xrange(3):
        ts.append(gevent.spawn(fetch_url,i,with_proxy))
    gevent.joinall(ts)

if __name__=='__main__':
    run()

์ด ์‚ฌ์ง„์€ ๊ฒฐ๊ณผ๋ฅผ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

ํ”„๋ก์‹œ๋กœ ์‹คํ–‰
ํ”„๋ก์‹œ ์—†์ด ์‹คํ–‰

ํ”„๋ก์‹œ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด fetch_url์ด ์™„๋ฃŒ๋˜๊ธฐ ์ „์— do_other_thing์ด ์ฐจ๋‹จ๋ฉ๋‹ˆ๋‹ค.
ํ”„๋ก์‹œ๊ฐ€ ์—†์œผ๋ฉด ์ž˜ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. (GFW๋กœ ์ธํ•ด ํƒ€์ž„์•„์›ƒ ์—๋Ÿฌ ๋ฐœ์ƒ)

๋ˆ„๊ตฌ๋“ ์ง€์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ๋ฐ ๋„์›€์„ ์ค„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ? ๋งค์šฐ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

๊ฐ€์žฅ ์œ ์šฉํ•œ ๋Œ“๊ธ€

๊ฐ€์žฅ ์ข‹์€ ์ถ”์ธก์€ ๋‹น์‹ ์ด ๋„ˆ๋ฌด ๋Šฆ๊ฒŒ ์›์ˆญ์ด ํŒจ์น˜๋ฅผ ํ•œ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. gevent๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ ๋‹ค๋ฅธ ๊ฒƒ์„ ๊ฐ€์ ธ์˜ค๊ธฐ ์ „์— ๋จผ์ € monkeypatchํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋‚˜์ค‘์— ์ˆ˜ํ–‰ํ•˜๋ฉด gevent monkeypatches์—์„œ from ๊ฐ€์ ธ์˜ค๊ธฐ๋ฅผ ์ˆ˜ํ–‰ํ•œ ๋ชจ๋“  ํ•ญ๋ชฉ์ด ์†์ƒ๋  ์œ„ํ—˜์ด ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค๋ฅธ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๊ธฐ ์ „์— ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ monkeypatch๋ฅผ ํŒŒ์ผ์˜ ๋งจ ์œ„๋กœ ์ด๋™ํ•ด ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

๋ชจ๋“  3 ๋Œ“๊ธ€

๊ฐ€์žฅ ์ข‹์€ ์ถ”์ธก์€ ๋‹น์‹ ์ด ๋„ˆ๋ฌด ๋Šฆ๊ฒŒ ์›์ˆญ์ด ํŒจ์น˜๋ฅผ ํ•œ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. gevent๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ ๋‹ค๋ฅธ ๊ฒƒ์„ ๊ฐ€์ ธ์˜ค๊ธฐ ์ „์— ๋จผ์ € monkeypatchํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋‚˜์ค‘์— ์ˆ˜ํ–‰ํ•˜๋ฉด gevent monkeypatches์—์„œ from ๊ฐ€์ ธ์˜ค๊ธฐ๋ฅผ ์ˆ˜ํ–‰ํ•œ ๋ชจ๋“  ํ•ญ๋ชฉ์ด ์†์ƒ๋  ์œ„ํ—˜์ด ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค๋ฅธ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๊ธฐ ์ „์— ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ monkeypatch๋ฅผ ํŒŒ์ผ์˜ ๋งจ ์œ„๋กœ ์ด๋™ํ•ด ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๊นŒ?

์˜ˆ, ' ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ monkeypatch๋ฅผ ํŒŒ์ผ ๋งจ ์œ„๋กœ ์ด๋™'ํ•˜๋ฉด ๋‚ด ๋ฌธ์ œ๊ฐ€ ํ•ด๊ฒฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค! ๋งค์šฐ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

์ด ์ œ์•ˆ์€ ๋˜ํ•œ ๋‚ด ํ”„๋กœ์ ํŠธ์˜ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•ฉ๋‹ˆ๋‹ค.
A,B,C๋ผ๋Š” 3๊ฐœ์˜ ํŒŒ์ผ์ด ์žˆ๊ณ  ์‹คํ–‰ ์ˆœ์„œ๋Š” A->B->C์ž…๋‹ˆ๋‹ค. ๋‚˜๋Š” B์—์„œ gevent๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ B์—์„œ monkeypatch๋ฅผ ์ถ”๊ฐ€ํ•˜์ง€๋งŒ ์š”์ฒญ+์–‘๋ง๋กœ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค. ์ด์ œ A ๋งจ ์œ„์— monkeypatch๋ฅผ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ์ด์ œ ์ž˜ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค! ์ด๊ฒƒ์ด ๋‹ค๋ฅธ ์‚ฌ๋žŒ๋“ค์—๊ฒŒ ๋„์›€์ด ๋˜๊ธฐ๋ฅผ ๋ฐ”๋ž๋‹ˆ๋‹ค!

๋‹ค์‹œ ๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค!

๋„์™€ ์ค„ ์ˆ˜์žˆ์–ด์„œ ๊ธฐ๋ป!

์ด ํŽ˜์ด์ง€๊ฐ€ ๋„์›€์ด ๋˜์—ˆ๋‚˜์š”?
0 / 5 - 0 ๋“ฑ๊ธ‰