Aws-cli: PyYAML requires python-dev dependency

We found that this did not work for us (as our CI also broke) and we had to downgrade to 1.11.18
I think the issue is the version bump to 1.4.76 of botocore

Same here. My CI on Travis is broken now.

Installing collected packages: awscli, PyYAML, docutils, botocore, rsa, colorama, s3transfer, jmespath, python-dateutil, pyasn1, futures, six
  Running setup.py install for PyYAML
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/check_libyaml.c -o build/temp.linux-x86_64-2.7/check_libyaml.o
    checking if libyaml is linkable
    x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-2.7/check_libyaml.o -lyaml -o build/temp.linux-x86_64-2.7/check_libyaml
    building '_yaml' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c ext/_yaml.c -o build/temp.linux-x86_64-2.7/ext/_yaml.o
    ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

We have had some success here by running

apt-get update && apt-get install -y python-pip libpython-dev
pip install awscli

@DeploymentKing Did you get a lot warnings from PyYAML installation?

I used: apt-get update && apt-get install -y python-dev

Part of my output:

Installing collected packages: awscli, PyYAML, docutils, botocore, rsa, colorama, s3transfer, jmespath, python-dateutil, pyasn1, futures, six
  Running setup.py install for PyYAML
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/check_libyaml.c -o build/temp.linux-x86_64-2.7/check_libyaml.o
    checking if libyaml is linkable
    x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-2.7/check_libyaml.o -lyaml -o build/temp.linux-x86_64-2.7/check_libyaml
    building '_yaml' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c ext/_yaml.c -o build/temp.linux-x86_64-2.7/ext/_yaml.o
    In file included from ext/_yaml.c:271:0:
    ext/_yaml.h:6:0: warning: "PyUnicode_FromString" redefined
     #define PyUnicode_FromString(s) PyUnicode_DecodeUTF8((s), strlen(s), "strict")
     ^
    In file included from /usr/include/python2.7/Python.h:85:0,
                     from ext/_yaml.c:4:
    /usr/include/python2.7/unicodeobject.h:281:0: note: this is the location of the previous definition
     # define PyUnicode_FromString PyUnicodeUCS4_FromString
     ^
    ext/_yaml.c: In function '__pyx_pf_5_yaml_get_version_string':
    ext/_yaml.c:1410:17: warning: assignment discards 'const' qualifier from pointer target type
       __pyx_v_value = yaml_get_version_string();
                     ^
    ext/_yaml.c: In function '__pyx_pf_5_yaml_7CParser___init__':
    ext/_yaml.c:2577:52: warning: passing argument 2 of 'yaml_parser_set_input' from incompatible pointer type
         yaml_parser_set_input((&__pyx_v_self->parser), __pyx_f_5_yaml_input_handler, ((void *)__pyx_v_self));
                                                        ^
    In file included from ext/_yaml.h:2:0,
                     from ext/_yaml.c:271:
    /usr/include/yaml.h:1367:1: note: expected 'int (*)(void *, unsigned char *, size_t,  size_t *)' but argument is of type 'int (*)(void *, char *, size_t,  size_t *)'
     yaml_parser_set_input(yaml_parser_t *parser,
     ^

...

Successfully installed awscli PyYAML docutils botocore rsa colorama s3transfer jmespath python-dateutil pyasn1 futures six

But it worked.

@prodis I did yes, but as it did for you, it worked for us too

Just adding that this also broke our CI and local development. We are pulling the aws cli into a docker container for some setup scripts and the update to 1.11.19 breaks any fresh (no-cache) builds.

As a workaround for now, we are pinning our awscli-bundle download to version 1.11.18 by downloading the bundle from https://s3.amazonaws.com/aws-cli/awscli-bundle-1.11.18.zip

This broke our docker build environments as well. We pinned to 1.11.18 and that has corrected the issue for us:

RUN pip install awscli==1.11.18

Can anyone confirm if this is still an issue for the latest version of the CLI? We loosened the dependency range on PyYaml so that if you have PyYaml previous installed (>=3.10) we will not try to update it if you had it already installed which was causing issues for users who had PyYaml installed on some of the different versions of Linux.

@kyleknap That sounds like a different issue than the one reported above (which is the requirement to install python-dev in order to install aws-cli after 1.11.18).

Just tried with the latest available bundle installer (looks like 1.11.28) and the install process still says that it requires Python.h for PyYaml (meaning there is still a dependency on python-dev).

For my own use case (downloading a file from S3 during the entrypoint script of a docker container), we changed our docker container to replace the aws-cli command we were using with a node script since those containers already have node and the aws-sdk installed. It would be nice to use the aws-cli as it would work the same in our non-nodejs containers, but we at least have a workaround.

It looks like the aws documentation has since been updated to state the python-dev dependency.

@kyleknap Hi, thanks for your response. For now, it works with my CI test since it has PyYAML == 3.10 by default so it does not need to rebuild PyYAML.

For other use case, as @bodnarbm faced, I think it will also break in the future if there's no appropriate version of PyYAML installed. If there is no tough work which is needed with using PyYAML, I personally suggest to use more lightweight YAML library (no *-dev dependency to build).

Anyway, thank you.

@kyleknap here is what I just tried:

docker run -i -t node:6.9 bash
apt-get update
apt-get -y install unzip python2.7 curl
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

and it still breaks, for it to work I had to install libpython-dev also.

Still broken for me, need to add libpython-dev

Needed both libpython-dev and python-dev

So the issue here is that PyYAML doesn't correctly detect all failures when determining whether or not to fallback to pure python. We'll need to get this fixed upstream.

CentOS/RHEL 7 needs python-devel

msys2 (ArchLinux) :
_x86_64-pc-msys-gcc_: Error: ext/_yaml.c: No such file or directory

broken for me still even when including both libpython-dev and python-dev. Ran docker build as of Dec 13, 2017 using debian jessie.

FROM debian:jessie

RUN     apt-get update -q
RUN     apt-get install -qy python-pip libpython-dev python-dev
RUN     pip install awscli

snippet of failure:

Installing collected packages: awscli, PyYAML, botocore, docutils, rsa, s3transfer, jmespath, python-dateutil, futures
  Running setup.py install for PyYAML
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/check_libyaml.c -o build/temp.linux-x86_64-2.7/check_libyaml.o
    build/temp.linux-x86_64-2.7/check_libyaml.c:2:18: fatal error: yaml.h: No such file or directory
     #include <yaml.h>
                      ^
    compilation terminated.

    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)

Any updates on this issue? I'm having the same problem.

Broken for me on ubuntu for circleci

This issue occurred yesterday for me on an EC2 instance while installing awscli to a virtual environment.

Specifically,
pip install pyyaml - fails
easy-install pyyaml - works

Same here, broken on Heroku deploy through CircleCI:

Collecting PyYAML<=3.12,>=3.10 (from awscli)
  Using cached https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz
    Complete output from command python setup.py egg_info:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: invalid command 'egg_info'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-cwzxyqvc/PyYAML/

Funny enough, I have to go back as far as 1.11.18 to reach an installable version. 1.11.19 already fails.

I'm having the same issue with Debian(Stretch):

Dockerfile:

FROM debian:stretch-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    python \
    python-dev \
    libpython-dev \
    python-pip \
    python-setuptools

COPY requirements.txt /

RUN pip install -r requirements.txt

cat requirements.txt

awscli==1.15.46
boto3==1.7.46

Output:

Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.10.46->awscli==1.15.46->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: PyYAML
  Running setup.py bdist_wheel for PyYAML: started
  Running setup.py bdist_wheel for PyYAML: finished with status 'error'
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-LiBDht/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpSAayIYpip-wheel- --python-tag cp27:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for PyYAML
  Running setup.py clean for PyYAML
Failed to build PyYAML

I am having the same problem with pip install awscli on CircleCI, image python:3. I get the following errors:

Failed building wheel for PyYAML

Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-g6h03py7/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-nniaucuh/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-g6h03py7/PyYAML/

I am installing all necessary dependencies with:

sudo apt-get update && sudo apt-get install -y python-pip libpython-dev && sudo apt-get install -y libxml2-dev libxslt1-dev build-essential python-dev libssl-dev

@antonioned Are you using docker? If so, the newest version of Python (3.7.0) was just released and there is apparently some compatibility issues with PyYAML. We had to explicitly state our box as box: python:3.6.6-stretch. Seems to be unrelated to aws-cli. Checkout https://github.com/yaml/pyyaml/issues/126

Hope this helps!

@jrmeier Yes, thanks Jed! This docker image fixed my problem, hopefully this compatibility issue will be resolved soon.

On Linux Mint 18:
pip install awscli --upgrade --user

Collecting awscli
  Using cached https://files.pythonhosted.org/packages/41/f4/60232985cfdbb4ecd0e74d6318c439bc4ffcd49ea2c576539c5a2674c19a/awscli-1.16.55-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Using cached https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl
Collecting botocore==1.12.45 (from awscli)
  Using cached https://files.pythonhosted.org/packages/ab/fa/5853b620519ee0f4fbc9c08c0598c6b51b09b09774f130e98df7d7ce4fce/botocore-1.12.45-py2.py3-none-any.whl
Collecting PyYAML<=3.13,>=3.10 (from awscli)
  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli)
  Using cached https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Using cached https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl
Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
  Using cached https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.20; python_version == "2.7" (from botocore==1.12.45->awscli)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.45->awscli)
  Using cached https://files.pythonhosted.org/packages/b7/31/05c8d001f7f87f0f07289a5fc0fc3832e9a57f2dbd4d3b0fee70e0d51365/jmespath-0.9.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore==1.12.45->awscli)
  Using cached https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.12->awscli)
  Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Using cached https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.45->awscli)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: PyYAML
  Running setup.py bdist_wheel for PyYAML ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-kSUd1J/PyYAML/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpguSnswpip-wheel- --python-tag cp27:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for PyYAML
  Running setup.py clean for PyYAML
Failed to build PyYAML
Installing collected packages: docutils, urllib3, jmespath, six, python-dateutil, botocore, PyYAML, futures, s3transfer, pyasn1, rsa, colorama, awscli
  Running setup.py install for PyYAML ... done
Successfully installed PyYAML-3.13 awscli-1.16.55 botocore-1.12.45 colorama-0.3.9 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.4 python-dateutil-2.7.5 rsa-3.4.2 s3transfer-0.1.13 six-1.11.0 urllib3-1.24.1
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@ConstantinElse run:
pip install wheel first

After several hours of troubleshooting this on WSL Ubuntu to no avail, I finally gave up and used this Docker image instead.

RUN apt-get update \
    && apt-get install -y --no-install-recommends python-setuptools=33.1.1-1 python-pip=9.0.1-2 \
    python-wheel=0.29.0-2 libpython-dev=2.7.13-2 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN pip install awscli --upgrade --user

This works for me (based on ruby:2.5.3-slim-stretch)

Hi,
trying to clean up older issues, and this seems to be no longer a problem, do correct me if I'm wrong, otherwise I'll be closing this.

Hi,

I tried using the version python:3.9.0a5-alpine3.11 and I start having this issue, but when I downgrade it to to python:3.8.2-alpine3.11 works

@amaiaeskisabela 3.9 is in alpha and we don't explicitly support it so I wouldn't really expect that one to succeed.

From a lot of these posted stack traces it says it succeeds the install of pyyaml at the bottom. The error is during pyyaml's installation it tries to compile optional C extensions for speed ups. In the case where it can't it falls back to a pure python implementation. So there isn't really a hard requirement on the dev tools, it is an optional dependency if you want the speedups.

Is anyone having an issue where this is actually failing the install altogether? I don't see any posted stack traces where it actually fails. The only failures posted are from the middle of the logs. In my repro cases it then says it succeeded at the bottom and I can use the CLI just fine. I've also tried every docker image posted in this thread to repro it, and the cli works fine in all of them after building and the rc is 0.