Pandas: importing pandas breaks after successfully installing pandas with pip

Looks pretty similar to https://github.com/pandas-dev/pandas/issues/11383#issuecomment-294242777

Can you try with verbose mode to see what version of numpy it pip grabs when you do them at the same time?

It looks like it is picking up numpy 1.13.0:

gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include -I/usr/local/include/python2.7 -c pandas/_libs/testing.c -o build/temp.linux-x86_64-2.7/pandas/_libs/testing.o -Wno-unused-function
  gcc -shared build/temp.linux-x86_64-2.7/pandas/_libs/testing.o -L/usr/local/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/pandas/_libs/testing.so
  building 'pandas._libs.lib' extension
  gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include -I/usr/local/include/python2.7 -c pandas/_libs/lib.c -o build/temp.linux-x86_64-2.7/pandas/_libs/lib.o -Wno-unused-function
  In file included from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                   from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:18,
                   from /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/arrayobject.h:4,
                   from pandas/_libs/lib.c:435:
  /tmp/pip-build-WbJa3a/pandas/.eggs/numpy-1.13.0-py2.7-linux-x86_64.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

Maybe try reporting on the pip issue tracker then. It's possible there's something wrong with our setup.py but a bug pip or setuptools seems more likely.

@rvernica is there a specific reason that you are building pandas from source, instead of downloading and installing the wheel? Because with the wheel, it should install without problems I think.

Ah I didn't see that the pip version was that old. It may already be fixed then.

@jorisvandenbossche I am just doing pip install, pip chooses to build it from source probably because there is no wheel.

I tried upgrading pip first:

/ # pip install --upgrade pip
Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 480kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-9.0.1

But the result is the same, pip succeeds but import pandas fails.

There should certainly be a wheel for pandas-0.20.2, so not sure why it is not downloading it ..

However, what system is it? It is maybe not compatible with manylinux requirements.

But it is indeed the same issue as in https://github.com/pandas-dev/pandas/issues/11383. So when doing install from source, installing with pip breaks if you try to install numpy and pandas in one go (and if you are not installing the latest version of numpy). The reason is that pip seems to build pandas in a separate env for which it uses the latest numpy, but in the actual env an older version of numpy is installed, and hence the errors. Solution is just do it in two steps (then numpy is already installed, and when building pandas it will use that and not fetch another one).
So either pip is just broken for this use case, or we do something wrong in our setup.py (but no expert in that)

Makes sense. The system is Alpine Linux 3.4.

@rvernica did you report this on the pip issue tracker? I assumed I would be able to reproduce this with the --no-binary option, but that pulled down the correct numpy version.

No, I didn't get the chance to report it.

I assumed I would be able to reproduce this with the --no-binary option, but that pulled down the correct numpy version.

@TomAugspurger did you specify an older version of numpy? (as that is needed to trigger it, eg pip install numpy==1.11 pandas==0.20)

Yeah I did pip install numpy==1.11.3 pandas==0.20.2 --no-binary=':all:' --verbose into a clean environment.

OK I'm not going to report this upstream as https://github.com/pypa/pip/issues/3691 (implemented in https://github.com/pypa/pip/pull/4144) is the correct solution. Let's give that a shot.

It's possible the version information will still be ignored for the build step, in which case we should report it to pip. I'll make a PR adding the pyproject.toml

I am not sure PEP518 is going to change anything about this, because the problem still remains that at the moment you build the pandas wheel (with specification of numpy>0.17 from the pyproject.toml, which will download the latest 1.13) it does not seem to know about the exact numpy version that is installed at the same time (and can thus be older than the one used in the wheel building step)

Should we actually pin the numpy version to the oldest supported one in the build-system requires? (what we in practice do when building wheels to distribute?) Since this is done in an isolated environment, it shouldn't influence your actual environment

Still having issues, any solution ?

Did you read the comments above? Try installing numpy and pandas in two separate steps. And otherwise you will need to give much more details about the specific problem you have.

Hi,
I have installed manually also ( note: already same version only), by using "pip install numpy==1.12.0 pandas==0.21.0"

Though after, while doing python -c " import pandas" getting error as

"the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

As I said above and please read the comments above, if it is the same problem you need to install them separately, so pip install numpy=1.12.0 and then pip install pandas=0.21.0

@jorisvandenbossche I have installed seperately, still same error. I did other way around. I have cloned pandas in seperate directory and installed pandas, where I can do import pandas, but when i do same from outside directory its not, it has always looking for source directory path to run

sudo pip install numpy==1.12.0
Requirement already satisfied: numpy==1.12.0 in /usr/local/lib/python2.7/dist-packages
ubuntu@ocata:~/devstack$ sudo pip install pandas==0.21.0
Requirement already satisfied: pandas==0.21.0 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas==0.21.0)
Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from pandas==0.21.0)
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from pandas==0.21.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/dist-packages (from python-dateutil->pandas==0.21.0)
ubuntu@ocata:~/devstack$ python -c "import pandas"
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/pandas/__init__.py", line 35, in
"the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

You need to give much more detailed information. Eg the full console output of the different steps you did, ideally with some verbose option, some info about the environment (which OS, python version, ...), why you are installing from source and not wheels, ...

Python 2.7.12
OS: ubuntu 16.04 xenial

I need to install gnocchi, as per http://gnocchi.xyz/install.html#installation-using-devstack, after adding plugin, ran ./stack.sh, where gnocchi installation https://github.com/gnocchixyz/gnocchi has pandas dependencies, when it try to install/import pandas getting error

Here the gnocchi file dependencies of pandas https://github.com/gnocchixyz/gnocchi/blob/ffbb446f3f12d637a6ff45b3eb522191638e8be0/tools/duration_perf_analyse.py#L37

In another way of around, just assume that i need to install pandas, I have to clone in seperate directory and use their modules, if i installed in one directory and try to import from outside directory, its not getting identified, where I can able import modules which are all under /python/dist-packages/ anywhere irrelevant to source directory

sudo pip install pandas
Requirement already satisfied: pandas in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/dist-packages (from python-dateutil->pandas)
ubuntu@ocata:/usr/local/lib/python2.7/dist-packages$ python -c "import pandas"
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
File "", line 1, in
File "pandas/__init__.py", line 35, in
"the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

@jorisvandenbossche let me know details required if any

The last log output says "Requirement already satisfied: pandas", so you are not actually installing it.
Further, make sure that pip references the same python (eg do python -m pip), and why are you importing pandas from dist-packages?

@jorisvandenbossche s "Requirement already satisfied: pandas" - Because I have installed separately as you said, when i trying to do again, definitely it will show like that.

why are you importing pandas from dist-packages? -->> I tried from everywhere, its not running.

Let me know how to install pandas in ubuntu 16.04 where python is 2.7.

as of now in fresh machine I have installed

pip install numpy==1.12.0 and then pip install pandas==0.21.0,

after installation, getting same error

pip install numpy==1.12.0 and then pip install pandas==0.21.0, after installation, getting same error

then show the full log of that (and not a log that is not actually installing anything, you need to do this from a fresh environment where numpy and pandas is not yet installed)

@jorisvandenbossche here in fresh machine installation

sudo pip install pandas
Collecting pandas
Using cached pandas-0.21.0-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/dist-packages (from python-dateutil->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.21.0

ubuntu@ocata:~$ sudo pip install numpy
Requirement already satisfied: numpy in /usr/lib/python2.7/dist-packages

ubuntu@ocata:~$ python -c "import pandas"
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/pandas/__init__.py", line 35, in
"the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

You should install numpy before pandas (although in this case it wouldn't have mattered, as numpy didn't install itself as it was already installed). Can you show the same but with freshly installing both numpy and pandas (and numpy first)?
BTW, does importing numpy work?

@jorisvandenbossche installed numpy first its working fine to import, installed pandas and importing is fine, but while doing ./stack.sh as per gnocchi, pandas importing is error where import numpy is fine

Your sudo python may be different than your python. Use python -m pip install ..., and only use sudo if nescessary.

@TomAugspurger i haven't combined both, i have installed pandas using pip install no issues in installation,

Here my question; whether numpy=1.12.0 is compatible with pandas=0.21.0

or what is the compatible one ?

I have tried numpy latest and pandas latest one. still not working, is there any compatibility issue ? or best compatible numpy & pandas version ?

@maestropandy look at your paths:

sudo pip install pandas
Collecting pandas
Using cached pandas-0.21.0-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas)

ubuntu@ocata:~$ sudo pip install numpy
Requirement already satisfied: numpy in /usr/lib/python2.7/dist-packages

The first is /usr/local/lib, the second is /usr/lib, so something is off there. Your exception comes from the python in usr/local/lib.

@TomAugspurger Does "sudo python -m pip install numpy" will install in /usr/local/lib/ ? Will try out.

Also let me know is there any compatiblity issue with numpy & pandas, because i have installed numpy>=1.9.0 & pandas>=0.18.0 which mean both numpy and pandas latest versions, but seems in compatible ?

Also let me know is there any compatiblity issue with numpy & pandas,

There is no incompatibility between those versions you mention in general. The issue is how you install both.

There is problem with new pandas version.. so till the issue will be fixed use pip install pandas==0.20.2

@abhishekg2389 can you provide more info? If there's an issue with our build files we'd like to know.

I'm seeing a similar (same?) issue under Fedora 25.

$ python --version
Python 2.7.13

$ which python
/usr/bin/python



md5-c7b975f7c1ac4eb89f5fe68ce1d09cd7



$ sudo pip install pandas
[sudo] password for .....:
Requirement already satisfied: pandas in /usr/lib64/python2.7/site-packages
Requirement already satisfied: pytz>=2011k in /usr/lib/python2.7/site-packages (from pandas)
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/site-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib64/python2.7/site-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil->pandas)

$ sudo pip install numpy
Requirement already satisfied: numpy in /usr/lib64/python2.7/site-packages



md5-79f115fb1bb7ca0fecb38162c5f341da



$ python testPandas.py 
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "runProphet.py", line 1, in <module>
    import pandas as pd
  File "/usr/lib64/python2.7/site-packages/pandas/__init__.py", line 35, in <module>
    "the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.



md5-fca2a04366ebb14ff185882c94092f99



$ sudo pip install pandas==0.20.2
[sudo] password for jbooker: 
Collecting pandas==0.20.2
  Downloading pandas-0.20.2-cp27-cp27mu-manylinux1_x86_64.whl (22.4MB)
    100% |████████████████████████████████| 22.4MB 79kB/s 
Requirement already satisfied: pytz>=2011k in /usr/lib/python2.7/site-packages (from pandas==0.20.2)
Requirement already satisfied: numpy>=1.7.0 in /usr/lib64/python2.7/site-packages (from pandas==0.20.2)
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/site-packages (from pandas==0.20.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil->pandas==0.20.2)
Installing collected packages: pandas
  Found existing installation: pandas 0.21.0
    Uninstalling pandas-0.21.0:
      Successfully uninstalled pandas-0.21.0
Successfully installed pandas-0.20.2

What version of numpy is installed? And can you show the output of installing pandas 0.21.0? (the one you show doesn't actually install, as it is already installed at that point)

$ pip show numpy
Name: numpy
Version: 1.11.2
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: NumPy Developers
Author-email: [email protected]
License: BSD
Location: /usr/lib64/python2.7/site-packages

I don't seem to have the output from installing pandas 0.21.0. It seems it was already installed on my system (??). I don't see a RPM package, so it wasn't installed via yum/dnf system packages.

I was attempting to install Facebook's Prophet package (has a dependency on Pandas and Numpy), and it noted that pandas 0.18.1 was already installed. Interesting that it's 0.18.1 and not 0.21.0.

$ sudo pip install fbprophet
Collecting fbprophet
  Using cached fbprophet-0.2.1.tar.gz
Requirement already satisfied: matplotlib in /usr/lib64/python2.7/site-packages (from fbprophet)
Requirement already satisfied: pandas>=0.18.1 in /usr/lib64/python2.7/site-packages (from fbprophet)
Requirement already satisfied: pystan>=2.14 in /usr/lib64/python2.7/site-packages (from fbprophet)
Requirement already satisfied: cycler>=0.10 in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: subprocess32 in /usr/lib64/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: pytz in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: six>=1.10 in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: backports.functools-lru-cache in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: numpy>=1.7.1 in /usr/lib64/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: python-dateutil>=2.0 in /usr/lib/python2.7/site-packages (from matplotlib->fbprophet)
Requirement already satisfied: Cython!=0.25.1,>=0.22 in /usr/lib64/python2.7/site-packages (from pystan>=2.14->fbprophet)
Installing collected packages: fbprophet
  Running setup.py install for fbprophet ... done
Successfully installed fbprophet-0.2.1

Without knowing how pandas was installed is difficult to help. Would you want to upgrade pandas again to 0.21.0 to see if that works and log the output?

Interesting that it's 0.18.1 and not 0.21.0.

It only says it is >=0.18.1, so it can be 0.21.0

Ahh right. I missed the '>='.. just that it satisfied the requirement, not the specific version.

Tried upgrading to 0.21.0 again, which recreated error on import (install works fine, though):

$ sudo pip install pandas==0.21.0
[sudo] password for jbooker: 
Collecting pandas==0.21.0
  Using cached pandas-0.21.0-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: pytz>=2011k in /usr/lib/python2.7/site-packages (from pandas==0.21.0)
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/site-packages (from pandas==0.21.0)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib64/python2.7/site-packages (from pandas==0.21.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil->pandas==0.21.0)
Installing collected packages: pandas
  Found existing installation: pandas 0.20.2
    Uninstalling pandas-0.20.2:
      Successfully uninstalled pandas-0.20.2
Successfully installed pandas-0.21.0

Test script:

import pandas as pd
import numpy as np

print "Hello, world."

Outputs the error again:

$ python testPandas.py 
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "runProphet.py", line 1, in <module>
    import pandas as pd
  File "/usr/lib64/python2.7/site-packages/pandas/__init__.py", line 35, in <module>
    "the C extensions first.".format(module))
ImportError: C extension: numpy.core.multiarray failed to import not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

Downgrading to 0.20.2 worked fine, and the test script runs successfully after doing so.

Anything else I can do to help narrow this down?

OK, I can actually reproduce this. Thanks for the report!
Reopening this issue, but as a new one (as the original issue here was something else, building from source)

@jlbooker opened an issue to track this here: https://github.com/pandas-dev/pandas/issues/18530
Hopefully we can quickly fix this

for me this issue was resolved this by uninstalling and reinstalling pandas by itself