Celery: Fails with SQS: 'Hub' object has no attribute '_current_http_client'

And if you run python -c 'import pycurl', does that pass? It seems like the only way the code in question would fail is if pycurl really isn't installed: https://github.com/celery/kombu/blob/master/kombu/asynchronous/http/curl.py#L16

I'm having this same issue and have pycurl installed.

~
boto3==1.4.4
pycurl==7.43.0.1
celery==4.1.0
~

Built with:
~
ENV PYCURL_SSL_LIBRARY=openssl
libssl-dev
libcurl4-openssl-dev
~

~
AttributeError: 'NoneType' object has no attribute '_current_http_client'
at get_client (/usr/local/lib/python3.6/site-packages/kombu/async/http/__init__.py:20)
~

Full trace:

~
log: "Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 235, in perform_pending_operations
self._pending_operations.pop()()
File "/usr/local/lib/python3.6/site-packages/vine/promises.py", line 143, in __call__
return self.throw()
File "/usr/local/lib/python3.6/site-packages/vine/promises.py", line 140, in __call__
retval = fun(final_args, *final_kwargs)
File "/usr/local/lib/python3.6/site-packages/kombu/message.py", line 130, in ack_log_error
self.ack(multiple=multiple)
File "/usr/local/lib/python3.6/site-packages/kombu/message.py", line 125, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "/usr/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 380, in basic_ack
self.asynsqs.delete_message(message['sqs_queue'],
File "/usr/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
region=self.region
File "/usr/local/lib/python3.6/site-packages/kombu/async/aws/sqs/connection.py", line 27, in __init__
*kwargs
File "/usr/local/lib/python3.6/site-packages/kombu/async/aws/connection.py", line 178, in __init__
*http_client_params)
File "/usr/local/lib/python3.6/site-packages/kombu/async/aws/connection.py", line 151, in __init__
self._httpclient = http_client or get_client()
File "/usr/local/lib/python3.6/site-packages/kombu/async/http/__init__.py", line 22, in get_client
client = hub._current_http_client = Client(hub, *kwargs)
File "/usr/local/lib/python3.6/site-packages/kombu/async/http/__init__.py", line 13, in Client
return CurlClient(hub, *kwargs)
File "/usr/local/lib/python3.6/site-packages/kombu/async/http/curl.py", line 57, in __init__
self._timeout_check_tref = self.hub.call_repeatedly(
AttributeError: 'NoneType' object has no attribute 'call_repeatedly'
"
~

Any updates on this? It's affecting me as well.

@fmoga can you please provide information on your setup? Output of pip freeze, Python version etc?

I'm getting the same error.
Python 3.6.3

pip freeze:

alabaster==0.7.10
alembic==0.9.6
amqp==2.3.0
arrow==0.12.1
astroid==1.6.0
atomicwrites==1.1.5
attrs==18.1.0
Babel==2.5.3
beautifulsoup4==4.4.1
billiard==3.5.0.3
boto3==1.7.28
botocore==1.10.28
celery==4.1.0
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.7
docutils==0.14
enum-compat==0.0.2
eventlet==0.23.0
fhirclient==3.2.0
Flask==1.0.2
Flask-Cors==3.0.4
flower==0.9.2
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
imagesize==1.0.0
isodate==0.6.0
isort==4.2.15
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
kombu==4.2.0
lazy-object-proxy==1.3.1
Mako==1.0.7
MarkupSafe==1.0
mccabe==0.6.1
more-itertools==4.2.0
mypy==0.600
packaging==17.1
pika==0.11.2
pluggy==0.6.0
psutil==3.3.0
psycopg2==2.7.3.2
psycopg2-binary==2.7.4
py==1.5.3
pyasn1==0.4.2
pycurl==7.43.0.2
Pygments==2.2.0
pylint==1.8.1
pyparsing==2.2.0
pytest==3.6.0
pytest-flask==0.10.0
pytest-html==1.18.0
pytest-metadata==1.7.0
pytest-tornado==0.4.5
pytest-tornado5==1.0.0
python-dateutil==2.7.3
python-editor==1.0.3
python-magic==0.4.6
pytz==2018.4
PyYAML==3.12
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.13
six==1.11.0
snowballstemmer==1.2.1
sphinxcontrib-websupport==1.0.1
SQLAlchemy==1.2.5
tornado==5.0.2
typed-ast==1.1.0
urllib3==1.22
vine==1.1.4
websocket-client==0.47.0
Werkzeug==0.14.1
wrapt==1.10.11
ws4py==0.3.2

Trace:

Traceback (most recent call last):
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/http/__init__.py", line 20, in get_client
return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start
return self.obj.start()
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 320, in start
blueprint.start(self)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(c.loop_args())
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/hub.py", line 291, in create_loop
item()
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/vine/promises.py", line 143, in __call__
return self.throw()
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/vine/promises.py", line 140, in __call__
retval = fun(final_args, *final_kwargs)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
queue, callback=promise(self._loop1, (queue,)),
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
return self._get_async(queue, maxcount, callback=callback)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
qname, count=count, connection=self.asynsqs,
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
region=self.region
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/aws/sqs/connection.py", line 27, in __init__
*kwargs
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/aws/connection.py", line 178, in __init__
*http_client_params)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/aws/connection.py", line 151, in __init__
self._httpclient = http_client or get_client()
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/http/__init__.py", line 22, in get_client
client = hub._current_http_client = Client(hub, *kwargs)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/http/__init__.py", line 13, in Client
return CurlClient(hub, **kwargs)
File "/Users/aseeger/Documents/GitHub/gos/venv/lib/python3.6/site-packages/kombu/async/http/curl.py", line 43, in __init__
raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

Do i need to add anything else?

I am having the same issue, same stack trace.
python 3.6.3
4.15.0-29-generic

Relevant versions:

amqp==2.3.2
boto==2.48.0
boto3==1.7.71
botocore==1.10.71
celery==4.2.1
kombu==4.2.1

I believe the issue caused by the lack of pycurl library.
This is the way I fixed this issue on Amazon Linux 1.

1. Install required yum packages
   yum install -y libcurl-devel python-devel

2. Install pycurl
   PYTHON_INSTALL_LAYOUT='' pip install pycurl

If environment variable PYTHON_INSTALL_LAYOUT=amzn is set, pycurl is installed into unexpected directory and celery/kombu cannot find it without the extra PYTHONPATH settings. So, I cleared it out while installing the pycurl.

I hope this helps.

I got the same error while installing celery on AWS elastic beanstalk with SQS. On my localhost , everything is fine though.

On my Elastic beanstalk, the log shows pycurl==7.43.0.1 has been satisfied already, I still got the error.
" yum install -y libcurl-devel python-devel" did not solve the problem.

Could anyone please give me a help? Thanks a lot

This issue was closed, but it's not clear what the solution is. There seem to be two different issues conflated here, one is the issue of the original poster

ImportError: The curl client requires the pycurl library.

and the second issue is the one in the title, and the one that I'm getting:

AttributeError: 'NoneType' object has no attribute 'call_repeatedly'

I would love to know if there's a solution to this second issue. PyCurl is installed fine for me and I was able to run the sample code on the PyCurl getting started page, without any problems.

Maybe the latter issue is an issue with Kombo, there is a related issue on their tracker that is still open:
https://github.com/celery/kombu/issues/746

None of the conversation on that issue helped me. One fellow said that using --pool=solo in Celery helped his issue, but that did nothing for me.

I had to downgrade Celery version from 4.3.0 to 3.1.25.
Neither 4.2.0 and 4.1.0 works for me but 3.1 did.

could you try celery ==4.4.0rc3?

What worked for me with Celery 4.3.0 was downgrading pycurl to ==7.19.3 instead of ==7.43.0.3.

Hope this helps someone!

could you try celery ==4.4.0rc3?

it's the same

I'm expecting same issue using Elastic Beanstalk and SQS.

AttributeError: 'Hub' object has no attribute '_current_http_client'
ImportError: The curl client requires the pycurl library

none of provided solutions work for me. Can anyone help?

Upgrading pycurl to pycurl==7.43.0.5 worked for me. Where as pycurl==7.19.3 would break for me.

Same here, breaking for me as well

After a couple of try and error and search through some repos this worked for me

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=nss
pip install --compile pycurl

https://stackoverflow.com/questions/21487278/ssl-error-installing-pycurl-after-ssl-is-set

I'm having the same issue:

ImportError: The curl client requires the pycurl library.

I've followed all the above steps, with no joy. I can see that pycurl is installed, it's in my bin folder and when I pip freeze it's in my requirements.txt.

I'm running Mojave 10.14.6, any tips?

On elastic beanstalk, I solved this one by changing the --pool for gevent or eventlet

For me, these steps worked

1. yum install -y gcc libcurl-devel python3-devel libcurl-devel openssl-static.x86_64
2. pip uninstall pycurl
3. pip install pycurl --compile --global-option="--with-openssl" --no-cache-dir

By default I had nss available, but no solution worked fine.
So, installing openssl and other relevant dependencies and installing pycurl with openssl worked fine.

On amazon linux 2:

packages:
  yum:
    gcc: []
    python3-devel: []
    libcurl-devel: []
    openssl-static.x86_64: []

and then executing this in a predeploy hook (.platform/hooks/predeploy/0x-pycurl-reinstall.sh) will do

source ${PYTHONPATH}/activate && ${PYTHONPATH}/pip3 install pycurl==7.43.0.5 --global-option="--with-openssl" --upgrade

Note: Amazon Linux 2 will support systemd by default. Go through this for a detailed understanding on Celery+AWS Elasticbeanstalk: https://medium.com/@rr.16566/celery-on-elasticbeanstalk-with-amazon-linux-2-98b00a42cb65

For anyone running into this on OSX for whom none of the above solutions worked, this worked for me: https://cscheng.info/2018/01/26/installing-pycurl-on-macos-high-sierra.html