Ccxt: changing aiohttp_proxy after the first request has no effect

Created on 27 Feb 2018 · 3Comments · Source: ccxt/ccxt

OS: Windows
Programming Language version: Python async
CCXT version: 1201

Changing aiohttp_proxy after the first request has no effect. If it's set after the first (for example) await e.load_markets(), it will not be used. If it's set before the first await e.load_markets() , it will keep going to that proxy server, also if you change aiohttp_proxy to another proxy server later on.

What's the easiest way to fix this ? call close() and overwrite the session variable ? And/or is this something that needs to be fixed in ccxt (for example, add another method to restart the session)

import ccxt.async as ccxt
import asyncio
import test2_config as config


async def main():
    exchange = 'gatecoin'
    e = getattr(ccxt, exchange)()
    e.verbose = True
    e.apiKey = config.exchange_keys[exchange]['key']
    e.secret = config.exchange_keys[exchange]['secret']
    e.aiohttp_proxy = 'http://' + config.check_proxylist[0]['host'] + ':' + config.check_proxylist[0]['port']
    await e.load_markets(reload=True)
    e.aiohttp_proxy = 'http://' + config.transaction_proxy['host'] + ':' + config.transaction_proxy['port']
    await e.load_markets(reload=True)
    await e.close()


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Verbose output

Request: GET https://api.gatecoin.com/Public/LiveTickers {'User-Agent': 'python-requests/2.18.4', 'Accept-Encoding': 'gzip, deflate'} None

Response: GET https://api.gatecoin.com/Public/LiveTickers 200 <CIMultiDictProxy('Date': 'Tue, 27 Feb 2018 18:15:41 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': '__cfduid=<snip>; expires=Wed, 27-Feb-19 18:15:40 GMT; path=/; domain=.gatecoin.com; HttpOnly', 'Cache-Control': 'private', 'Content-Encoding': 'gzip', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': '<snip>-DUB')> <snip>

Request: GET https://api.gatecoin.com/Public/LiveTickers {'User-Agent': 'python-requests/2.18.4', 'Accept-Encoding': 'gzip, deflate'} None

Response: GET https://api.gatecoin.com/Public/LiveTickers 200 <CIMultiDictProxy('Date': 'Tue, 27 Feb 2018 18:15:41 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Cache-Control': 'private', 'Content-Encoding': 'gzip', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': '<snip>-DUB')> <snip>

question

Source

gaardiolor

All 3 comments

Apparently proxy works per TCP connection. If it isn't closed, previous proxies are being used. If I add
headers['Connection'] = 'close' to the fetch method in the base exchange class (for async), proxy rotation works as expected because the TCP connection is closed after the first request. When that header is not added, the TCP connection to the previous proxy stays open and aiohttp reuses that one regardless of the new aiohttp_proxy.

So the behaviour could be different depending on proxy and exchange, depending if/how they implement keepalive.

I can't find a 'clean' way to inject headers['Connection'] = 'close' though.

gaardiolor on 28 Feb 2018

I can't find a 'clean' way to inject headers['Connection'] = 'close' though.

Add exchange.headers = {'Connection': 'close'} before issuing the call or right after creating the exchange instance:


# option 1
exchange = ccxt.gatecoin({'headers': {'Connection': 'close'}})

# option 2
exchange = ccxt.gatecoin()
exchange.headers = {'Connection': 'close'}

Let us know if it does not help. Thx!