Certbot: installation error on amazon linux

From #1458,

I was able to run the client with a modified letsencrypt-auto on Amazon Linux 2015.03 by manually installing the python27-virtualenv package first, then running the RedHat bootstrap path.

I did that and it seems to be working. You need to run sudo yum install python27-virtualenv and sudo ./bootstrap/_rpm_common.sh

@barshinoff I'm curious if this is at all related to #1677 (an out of memory error when compiling cffi dependencies on other redhat-family distros) I've checked that ./letsencrypt-auto works on clean installs of amazon linux 2015.09.1 and 2015.03.1 on t2.micro instances with nothing else running. What instance type and AMI are you trying this on? If you run it verbosely with '-v' ./letsencrypt-auto --help --debug -v it might shed more information on what's going on.

Hi, thanks for the advice.
It is not a memory issue, to make sure I made sure nothing else was running and hooked up a 40gb swap drive.
I am running 2015.09 distro with all packages updated.

Installing python27-virtualenv package first, then running the RedHat bootstrap path as suggested by @kiwimancy got me past the initial error.
The error I now receive is:
Updating letsencrypt and virtual environment dependencies....... Running with virtualenv: /root/.local/share/letsencrypt/bin/letsencrypt Traceback (most recent call last): File "/root/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module> from letsencrypt.cli import main File "/root/.local/share/letsencrypt/local/lib/python2.7/dist-packages/letsencrypt/cli.py", line 17, in <module> import OpenSSL ImportError: No module named OpenSSL

Same error here on the same 2015.09 Amazon Linux.
Below is the verbose log, hope this helps identifying the problem.

$ ./letsencrypt-auto help --debug --verbose
Bootstrapping dependencies for Amazon Linux...
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
Package python26-2.6.9-1.80.amzn1.x86_64 already installed and latest version
Package python26-devel-2.6.9-1.80.amzn1.x86_64 already installed and latest version
Package python26-virtualenv-12.0.7-1.10.amzn1.noarch already installed and latest version
Nothing to do
Loaded plugins: priorities, update-motd, upgrade-helper
Package git-2.4.3-6.40.amzn1.x86_64 already installed and latest version
Package gcc-4.8.3-3.20.amzn1.noarch already installed and latest version
Package dialog-1.1-9.20080819.1.5.amzn1.x86_64 already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1k-10.87.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-11.4.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.27.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.4-65.0.1.14.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Already using interpreter /usr/bin/python2.7
New python executable in /home/ec2-user/.local/share/letsencrypt/bin/python2.7
Also creating executable in /home/ec2-user/.local/share/letsencrypt/bin/python
Installing setuptools, pip...done.
Updating letsencrypt and virtual environment dependencies...
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting setuptools from https://pypi.python.org/packages/3.5/s/setuptools/setuptools-18.7.1-py2.py3-none-any.whl#md5=a793e7b7ad3a91ca796f2e3bbd4ac355
  Using cached setuptools-18.7.1-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 12.0.5
    Uninstalling setuptools-12.0.5:
      Successfully uninstalled setuptools-12.0.5

Successfully installed setuptools-18.7.1
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Successfully uninstalled pip-6.0.8

Successfully installed pip-7.1.2
Collecting letsencrypt
  Using cached letsencrypt-0.1.0-py2-none-any.whl
Collecting letsencrypt-apache
  Using cached letsencrypt_apache-0.1.0-py2-none-any.whl
Collecting ConfigArgParse from git+https://github.com/kuba/[email protected]#egg=ConfigArgParse (from -r ./py26reqs.txt (line 1))
  Cloning https://github.com/kuba/ConfigArgParse.git (to python2.6-0.9.3) to /tmp/pip-build-3Vgs0V/ConfigArgParse
Collecting acme==0.1.0 (from letsencrypt)
  Using cached acme-0.1.0-py2-none-any.whl
Requirement already up-to-date: setuptools in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Collecting python2-pythondialog>=3.2.2rc1 (from letsencrypt)
  Using cached python2-pythondialog-3.3.0.tar.bz2
Collecting PyOpenSSL (from letsencrypt)
  Using cached pyOpenSSL-0.15.1-py2.py3-none-any.whl
Collecting requests (from letsencrypt)
  Using cached requests-2.8.1-py2.py3-none-any.whl
Collecting parsedatetime (from letsencrypt)
  Using cached parsedatetime-1.5-py2-none-any.whl
Collecting configobj (from letsencrypt)
  Using cached configobj-5.0.6.tar.gz
Collecting pytz (from letsencrypt)
  Using cached pytz-2015.7-py2.py3-none-any.whl
Collecting psutil>=2.1.0 (from letsencrypt)
  Using cached psutil-3.3.0.tar.gz
Collecting six (from letsencrypt)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting cryptography>=0.7 (from letsencrypt)
  Using cached cryptography-1.1.1.tar.gz
Collecting zope.interface (from letsencrypt)
  Using cached zope.interface-4.1.3.tar.gz
Collecting zope.component (from letsencrypt)
  Using cached zope.component-4.2.2.tar.gz
Collecting mock (from letsencrypt)
  Using cached mock-1.3.0-py2.py3-none-any.whl
Collecting pyrfc3339 (from letsencrypt)
  Using cached pyRFC3339-1.0-py2.py3-none-any.whl
Collecting python-augeas (from letsencrypt-apache)
  Using cached python-augeas-0.5.0.tar.gz
Collecting pyasn1 (from acme==0.1.0->letsencrypt)
  Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting ndg-httpsclient (from acme==0.1.0->letsencrypt)
  Using cached ndg_httpsclient-0.4.0.tar.gz
Collecting werkzeug (from acme==0.1.0->letsencrypt)
  Using cached Werkzeug-0.11.2-py2.py3-none-any.whl
Collecting idna>=2.0 (from cryptography>=0.7->letsencrypt)
  Using cached idna-2.0-py2.py3-none-any.whl
Collecting enum34 (from cryptography>=0.7->letsencrypt)
  Using cached enum34-1.1.1.tar.gz
Collecting ipaddress (from cryptography>=0.7->letsencrypt)
  Using cached ipaddress-1.0.15-py27-none-any.whl
Collecting cffi>=1.1.0 (from cryptography>=0.7->letsencrypt)
  Using cached cffi-1.3.1.tar.gz
Collecting zope.event (from zope.component->letsencrypt)
  Using cached zope.event-4.1.0.tar.gz
Collecting funcsigs (from mock->letsencrypt)
  Using cached funcsigs-0.4-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock->letsencrypt)
  Using cached pbr-1.8.1-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.1.0->cryptography>=0.7->letsencrypt)
  Using cached pycparser-2.14.tar.gz
Installing collected packages: pytz, requests, pyasn1, idna, six, enum34, ipaddress, pycparser, cffi, cryptography, PyOpenSSL, ndg-httpsclient, werkzeug, funcsigs, pbr, mock, pyrfc3339, acme, python2-pythondialog, parsedatetime, configobj, psutil, zope.interface, zope.event, zope.component, ConfigArgParse, letsencrypt, python-augeas, letsencrypt-apache
  Running setup.py install for enum34
  Running setup.py install for pycparser
  Running setup.py install for cffi
  Running setup.py install for cryptography
  Running setup.py install for ndg-httpsclient
  Running setup.py install for python2-pythondialog
  Running setup.py install for configobj
  Running setup.py install for psutil
  Running setup.py install for zope.interface
  Running setup.py install for zope.event
  Running setup.py install for zope.component
  Running setup.py install for ConfigArgParse
  Running setup.py install for python-augeas
Successfully installed ConfigArgParse-0.9.3 PyOpenSSL acme-0.1.0 cffi configobj-5.0.6 cryptography enum34-1.1.1 funcsigs-0.4 idna-2.0 ipaddress-1.0.15 letsencrypt-0.1.0 letsencrypt-apache-0.1.0 mock-1.3.0 ndg-httpsclient-0.4.0 parsedatetime-1.5 pbr-1.8.1 psutil pyasn1-0.1.9 pycparser-2.14 pyrfc3339 python-augeas-0.5.0 python2-pythondialog-3.3.0 pytz-2015.7 requests-2.8.1 six-1.10.0 werkzeug zope.component-4.2.2 zope.event-4.1.0 zope.interface
Running with virtualenv: sudo /home/ec2-user/.local/share/letsencrypt/bin/letsencrypt help --debug --verbose
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module>
    from letsencrypt.cli import main
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/dist-packages/letsencrypt/cli.py", line 17, in <module>
    import OpenSSL
ImportError: No module named OpenSSL

I tried bluntly to resolve for pyOpenSSL by manually add pip install pyOpenSSL pip install cryptography before https://github.com/letsencrypt/letsencrypt/blob/master/letsencrypt-auto#L181 , but stuck with cffi version incompatibility and gave up.

I have the error as well:

$ ./letsencrypt-auto --apache -d www.3chirurgen.de -d 3chirurgen.de -d incentergy.de -d www.incentergy.de -d shop.incentergy.de
Updating letsencrypt and virtual environment dependencies......Command "/home/ec2-user/.local/share/letsencrypt/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-oV91ZI/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-GU4tNk-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/.local/share/letsencrypt/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-build-oV91ZI/cffi
$ uname -a
Linux ip-172-31-11-205 4.1.7-15.23.amzn1.x86_64 #1 SMP Mon Sep 14 23:20:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Solution as from @kiwimancy worked:

sudo yum install python27-virtualenv
sudo ./bootstrap/_rpm_common.sh
# Output nothing to do

Next Problem:

$ ./letsencrypt-auto --apache -d www.3chirurgen.de -d 3chirurgen.de -d incentergy.de -d www.incentergy.de -d shop.incentergy.de
Running with virtualenv: sudo /home/ec2-user/.local/share/letsencrypt/bin/letsencrypt --apache -d www.3chirurgen.de -d 3chirurgen.de -d incentergy.de -d www.incentergy.de -d shop.incentergy.de
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module>
    from letsencrypt.cli import main
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/dist-packages/letsencrypt/cli.py", line 17, in <module>
    import OpenSSL
ImportError: No module named OpenSSL

Solution:

$ /home/ec2-user/.local/share/letsencrypt/bin/pip install pyopenssl

Next problem:

Complete output from command /home/ec2-user/.local/share/letsencrypt/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-5_HYBt/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-BxpscI-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/.local/share/letsencrypt/include/site/python2.7/cffi:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/gc_weakref.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/c
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DUSE__THREAD -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/home/ec2-user/.local/share/letsencrypt/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-5_HYBt/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-BxpscI-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/.local/share/letsencrypt/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-build-5_HYBt/cffi

Solution:

sudo yum install python27-devel

Next problem:

The apache plugin is not working; there may be problems with your existing configuration.
The error was: NoInstallationError()

2015-12-04 08:18:50,797:DEBUG:letsencrypt.cli:Root logging level set at 20
2015-12-04 08:18:50,798:INFO:letsencrypt.cli:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2015-12-04 08:18:50,807:DEBUG:letsencrypt.cli:letsencrypt version: 0.1.0
2015-12-04 08:18:50,807:DEBUG:letsencrypt.cli:Arguments: ['-v', '--debug', '--apache', '-d', 'www.3chirurgen.de', '-d', '3chirurgen.de', '-d', 'incentergy.de', '-d', 'www.incentergy.de', '-d', 'shop.incentergy.de']
2015-12-04 08:18:50,809:DEBUG:letsencrypt.cli:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2015-12-04 08:18:50,814:DEBUG:letsencrypt.cli:Requested authenticator apache and installer apache
2015-12-04 08:18:50,820:DEBUG:letsencrypt.plugins.disco:No installation (PluginEntryPoint#apache):
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/plugins/disco.py", line 103, in prepare
    self._initialized.prepare()
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt_apache/configurator.py", line 142, in prepare
    raise errors.NoInstallationError
NoInstallationError
2015-12-04 08:18:50,821:DEBUG:letsencrypt.display.ops:No candidate plugin
2015-12-04 08:18:50,821:DEBUG:letsencrypt.cli:Selected authenticator None and installer None

I will dig into this later.

./letsencrypt-auto -v

Updating letsencrypt and virtual environment dependencies...
Requirement already up-to-date: setuptools in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages
Requirement already up-to-date: pip in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages
Requirement already up-to-date: letsencrypt in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages
Requirement already up-to-date: letsencrypt-apache in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages
Requirement already up-to-date: zope.interface in /home/ec2-user/.local/share/letsencrypt/lib64/python2.7/site-packages (from letsencrypt)
Requirement already up-to-date: setuptools in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: python2-pythondialog>=3.2.2rc1 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Collecting PyOpenSSL (from letsencrypt)
  Using cached pyOpenSSL-0.15.1-py2.py3-none-any.whl
Requirement already up-to-date: requests in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Collecting cryptography>=0.7 (from letsencrypt)
  Using cached cryptography-1.1.2.tar.gz
Requirement already up-to-date: parsedatetime in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: configobj in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: pytz in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Collecting psutil>=2.1.0 (from letsencrypt)
  Using cached psutil-3.3.0.tar.gz
Requirement already up-to-date: six in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: acme==0.1.1 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: zope.component in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: mock in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Requirement already up-to-date: ConfigArgParse in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt)
Collecting pyrfc3339 (from letsencrypt)
  Using cached pyRFC3339-1.0-py2.py3-none-any.whl
Requirement already up-to-date: python-augeas in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from letsencrypt-apache)
Requirement already up-to-date: idna>=2.0 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from cryptography>=0.7->letsencrypt)
Requirement already up-to-date: pyasn1>=0.1.8 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from cryptography>=0.7->letsencrypt)
Requirement already up-to-date: enum34 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from cryptography>=0.7->letsencrypt)
Requirement already up-to-date: ipaddress in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from cryptography>=0.7->letsencrypt)
Collecting cffi>=1.1.0 (from cryptography>=0.7->letsencrypt)
  Using cached cffi-1.4.2.tar.gz
Requirement already up-to-date: ndg-httpsclient in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from acme==0.1.1->letsencrypt)
Collecting werkzeug (from acme==0.1.1->letsencrypt)
  Using cached Werkzeug-0.11.3-py2.py3-none-any.whl
Requirement already up-to-date: zope.event in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from zope.component->letsencrypt)
Requirement already up-to-date: funcsigs in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from mock->letsencrypt)
Requirement already up-to-date: pbr>=0.11 in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from mock->letsencrypt)
Requirement already up-to-date: pycparser in /home/ec2-user/.local/share/letsencrypt/lib/python2.7/dist-packages (from cffi>=1.1.0->cryptography>=0.7->letsencrypt)
Installing collected packages: cffi, cryptography, PyOpenSSL, psutil, pyrfc3339, werkzeug
  Running setup.py install for cffi
  Running setup.py install for cryptography
  Running setup.py install for psutil
Successfully installed PyOpenSSL cffi cryptography psutil pyrfc3339 werkzeug
Requesting root privileges to run with virtualenv: sudo /home/ec2-user/.local/share/letsencrypt/bin/letsencrypt -v
Traceback (most recent call last):
  File "/home/ec2-user/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module>
    from letsencrypt.cli import main
  File "/home/ec2-user/.local/share/letsencrypt/local/lib/python2.7/dist-packages/letsencrypt/cli.py", line 17, in <module>
    import OpenSSL
ImportError: No module named OpenSSL

pip install pyopenssl does not resolve this.

OK so I believe I found a solution to this issue.

It all comes down to running as root or running via sudo, the use of multiple package locations for pip installation (lib/lib64, site-packages/dist-packages, /usr vs /usr/local).

I came across this similar issue:
http://stackoverflow.com/questions/33661818/python-virtualenv-importerror-with-celery-and-billiard

It turns out the pip and virtualenv versions that ship with Amazon Linux, even in the latest AMI, are out of date and (possibly) have bugs in how lib64/dist-packages is handled for virtual environments.

Updating both programs via pip and then recreating the virtual environment seems to fix the issue when running the setup as root (not using sudo). The solutions given in other threads and this one to run pip from the virtualenv folder to reinstall dependencies are fixing a symptom, not the cause.

Similar issue reported here:
https://community.letsencrypt.org/t/letsencrypt-auto-openssl-module-not-found-ec2/4356

eric-tucker's solution above worked for me as well. Thank you!

Came here from Google, had similar issues with all compiled python modules in virtualenv on latest Amazon Linux 2015.09.1.

eric-tucker is right, upgrading virtualenv manually helps here.

I found that the culprit was env variable PYTHON_INSTALL_LAYOUT=amzon which by default isn't set on Amazon Linux - and then you don't notice the problem. But once this is set, things do not work with default version of virtualenv, because packages get installed into lib64/python2.7/dist-packages and this is not added by virtualenv's site.py (at least not in the version that comes from rpm package; only lib/python2.7/dist-packages is added).

The tricky part, at least for me, was finding out which package defined this variable for me. It turned out it was caused by installing "@Development tools" group by yum and specifically the system-rpm-config package. Not that obvious...

Yes! I solved!. Thank you eric-tucker !

I ran the following command.

# pip install pip --upgrade
# pip install virtualenv --upgrade
# virtualenv -p /usr/bin/python27 venv27 
# . venv27/bin/activate 
# git clone https://github.com/letsencrypt/letsencrypt
# cd letsencrypt
# ./letsencrypt-auto certonly --debug --standalone -d <domain-name>

Update to @skatsumata for 2017 and certbot-auto:

# rm /root/.local/share/letsencrypt/ -Rf
# pip install pip --upgrade
# pip install virtualenv --upgrade
# virtualenv -p /usr/bin/python27 venv27 
# . venv27/bin/activate 
# ./certbot-auto renew --debug #(or whatever your relevant command is)

I only had to run sudo yum install python27-devel for this to work for me, thanks!

@zarmstrong Thanks a lot, that works. BTW I want to setup a cron to auto renew. How do I do that with your solution?

Not sure if this will work, untested. But you can try it.

#!/bin/bash
date
cd /root
source venv27/bin/activate
./certbot-auto renew --debug
/etc/init.d/httpd reload

Be sure to redirect output to some sort of log and verify it works.

This should be fixed in our next release with #4978.

@bmw I don't believe this has been fixed. I'm experiencing this same lib/lib64 issue still on AWS Elastic Beanstalk environments with latest version of certbot downloaded as follows:

wget https://dl.eff.org/certbot-auto; chmod a+x certbot-auto

# cat /etc/system-release
Amazon Linux AMI release 2017.09

[root@ip-172-31-10-255 venv]# /certbot/certbot-auto --debug renew
Error: couldn't get currently installed version for /opt/eff.org/certbot/venv/bin/letsencrypt:
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 7, in <module>
    from certbot.main import main
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/josepy/__init__.py", line 41, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/OpenSSL/rand.py", line 12, in <module>
    from OpenSSL._util import (
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/dist-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
ImportError: No module named cryptography.hazmat.bindings.openssl.binding

I'm baffled by how much trouble we're having with Python dependencies and what not.

Though, if you install certbot via pip, I think things work. Have you tried this @utdrmac ? I know it solved many of my issues.

Didn't know install via pip was an option. I was following some blog posts about using certbot on beanstalk and their scripts downloaded it as I put above. Perhaps blog post out of date. Testing via pip now:

# rm -rf /opt/eff.org/*
# pip install -U certbot
# certbot renew --debug

Appears to have worked! I will adjust my scripts to install certbot using system pip instead of downloading it directly. Perhaps there's some issue with certbots built-in installer/updater in this regard?

I honesty have no idea why installation via pip isn't the recommended approach on ALL operating systems.

The original solution of downloading of a file which tries to setup a virtualenv and fails miserably half the time needs to DIAF.

The error is with the python path and the use of dist-packages. We can add a path configuration, e.g., dist.pth, to /opt/eff.org/certbot/venv/lib64/python2.7/site-packages/dist.pth. However, pth files are only processed if they are in a site directory (e.g., site-packages) and pth files in the newly appointed path (i.e., dist-packages) will not be read. We create a sitecustomize.py in /opt/eff.org/certbot/venv/lib64/python2.7/site-packages/ with the following:

import site
site.addsitedir('/opt/eff.org/certbot/venv/lib64/python2.7/dist-packages')

The sitecustomize.py sets dist-packages as a site directory and will successfully read the zope.interface-4.4.3-py2.7-nspkg.pth. This avoids the from zope.interface import Interface error ImportError.

Further, I found that /usr/local/bin/certbot-auto re-creates the virtualenv venv. This will result in the same python path error we are trying to avoid. I recommend applying the above and manually sourcing the activate script in venv; e.g., . /opt/eff.org/certbot/venv/bin/activate.

I prefer this approach because it avoids bashing over system-level python packages.

Not that this says anything different than above, but the quick fix for me on certbot 0.21.1 was to effectively "move" the cryptography hazmat bindings to where letsencrypt's custom version of python is looking:

cd /opt/eff.org/certbot/venv/lib64/python2.7
mv site-packages site-packages.sav
ln -s dist-packages/ site-packages

This works for me because the site-packages/ subdir is empty for me. And to be clear, this addresses the following error that appears on recent certbot versions on Amazon Linux, for me certbot 0.21.0 and 0.21.1:

<snip>
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
ImportError: No module named cryptography.hazmat.bindings.openssl.binding

The answer @utdrmac wrote is what I ended up using, Thanks Matthew!

But I got the error: "The requested apache plugin does not appear to be installed"
So I did: sudo -H pip install certbot-apache

i had same error and i somehow solved it

Since i had two versions of python install, python 2 and 3, i had to find out which on certbot/letencrypt was using. that one will report error when trying to import cryptography....

on the command line

python
from cryptography.hazmat.bindings._constant_time import lib

no import error, so i did same with python3 and i got the No module named '_cffi_backend error. So i simple installed the cffi module with pip3
sudo -H pip3 install cffi

sudo -H cuz it was needed as a global package

did certbot help next then got import error for urllib3. Installed that with pip3 and now everything works fine.

Hopes someone finds this helpful

I had this problem too on an AWS EC2 instance, just fixed by running @utdrmac's solution as root. It was upgrading to certbot 0.22.0

Thank you so much @utdrmac for the solution. I have been trying so many things last two days and nothing had worked.

I ran into this issue today and further found that the @utdrmac solution does not work on AWS EC2 running Amazon Linux AMI 2017.09. The error I encounter at the final step is:

$ sudo certbot renew --debug
/usr/lib64/python2.6/dist-packages/cryptography/__init__.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. The next version of cryptography will drop support for Python 2.6
  DeprecationWarning
Traceback (most recent call last):
  File "/usr/bin/certbot", line 7, in <module>
    from certbot.main import main
  File "/usr/lib/python2.6/dist-packages/certbot/main.py", line 11, in <module>
    import zope.component
  File "/usr/lib/python2.6/dist-packages/zope/component/__init__.py", line 28, in <module>
    from zope.component.globalregistry import getGlobalSiteManager
  File "/usr/lib/python2.6/dist-packages/zope/component/globalregistry.py", line 18, in <module>
    from zope.interface.registry import Components
  File "/usr/lib64/python2.6/dist-packages/zope/interface/registry.py", line 167
    filtered_state = {k: v for k, v in reduction[2].items()
                             ^
SyntaxError: invalid syntax

In order to get certbot to run, I was forced to extend this solution by installing a newer Python and utilizing it, thusly:

sudo rm -rf /opt/eff.org/*
sudo yum -y install python36 python36-pip python36-libs python36-tools python36-virtualenv
sudo /usr/bin/pip-3.6 install -U certbot
sudo /usr/bin/pip-3.6 install certbot-apache
sudo /usr/local/bin/certbot renew --debug

Note that I separately sudo each command in this proposal because when I tried to lump them all together (as in sudo su -), this sequence of steps failed because some old Python settings polluted the environment, causing certbot to ultimately fail, anyway.

@wwkimball To be fair, my solution/problem was for Elastic Beanstalk. Seems I'm also running AMI 2017.09 so not sure why yours is different from everybody else's.

I ran into the same problem. Here's what solved it for me:

/opt/eff.org/certbot/venv/local/bin/pip install cryptography interface

@gmegidish 's solution fixed this problem for me.

Nothing but https://github.com/certbot/certbot/issues/1680#issuecomment-374046383 by @wwkimball worked for me.
Thanks for the solution! :100:

FWIW, this is what I threw together to work on my AWS Linux instance:

#!/bin/bash -x

# If run in cron, add /usr/local to cron's sanitized PATH
export PATH=$PATH:/usr/local/sbin:/usr/local/bin

INSTALLED=/usr/local/bin

$INSTALLED/certbot-auto $@
if [ $? -ne 0 ]; then
  # certbot-auto used venv to install modules, but something
  # about the way it put things together put pieces in a subdir
  # that python doesn't look in by default on AWS Linux. Fix it.
  BASEDIR="/opt/eff.org/certbot/venv/lib64/python2.7"
  cat << "  EOM"
  1. Getting rid of empty $BASEDIR/site-packages.
  2. Create symlink "site-packages" to "dist-packages/".

  EOM

  if [ -d $BASEDIR ]; then
    cd $BASEDIR
    rm -rf site-packages
    ln -s dist-packages/ site-packages
  else
    echo "ERROR: $BASEDIR doesn't exist, python version changed?"
  fi

  $INSTALLED/certbot-auto $@
fi

I chose to leave the INSTALLED variable in place in case someone wants to install the certbot-auto script somewhere other than in the path. I call it run_certbot.sh, and I call it with the args I want to pass to certbot-auto:

run_certbot.sh renew --debug

@mrballcb Thanks! work perfectly.

sudo pip3 install cryptography
sudo pip3 install interface
worked for me

./certbot-auto --no-bootstrap

This worked for me

I was able to install and run certbot successfully on Amazon Linux 1 via the pip method (python36 which includes pip). The yum method encountered the below errors which were above my ability to diagnose.

Complete!
Creating virtual environment...
Traceback (most recent call last):
  File "<stdin>", line 27, in <module>
  File "<stdin>", line 19, in create_venv
  File "/usr/lib64/python2.7/subprocess.py", line 185, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 172, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

@micharu123 Thanks for your post your solution fixed the same issue for me. Cheers

I had some of the same problems that are mentioned here, and the solution was simply move the "certbot-auto" executable to /usr/local/bin/ . This fixed every single error inmediatly on my AWS server.