Numpy: Buggy Accelerate Backend when using numpy 1.19

This is by design. Did you read the error message? Is it unclear?

so how can I fix this error, by updating the Accelerate backend? I had a search and don't think I can update that for my MacOS.

@billlyzhaoyh Requesting to use OpenBLAS while building on MacOS instead. Use site.cfg in the root repo directory to build using the same.

Closing. We strongly recommend not using Accelerate.

Closing. We strongly recommend not using Accelerate.

Sure, how do we not use it? I got the same error when just run simple import numpy in my macOS

If you are building your own NumPy, get OpenBlas. You can use python3 tools/openblas_support.py to download the one used to build the official wheel. If you are using a provider's NumPy, ask them why they are using Accelerate.

If you are building your own NumPy, get OpenBlas. You can use python3 tools/openblas_support.py to download the one used to build the official wheel. If you are using a provider's NumPy, ask them why they are using Accelerate.

I am definitely not building on my own. I am using macOS, and run pip3 install -U numpy , so I am not sure what "provider" do you mean by that .

Pip can build from source behind your back. If it takes more than a few seconds to install, it may be compiling. To force pip to only use binary wheels and never compile, use

pip3 install --upgrade --only-binary :all: <package>

Can you try to uninstall your numpy and install with the --only-binary :all: argument?

This fixed the problem for me. (I replaced pip with pip3 ... and the numpy version as of today will be 1.19.1)
https://gist.github.com/yatsu/47bdde35e8abbe7d14bbe730342aa9e0#file-numpy-openblas-macos-pip-sh

EDIT: Pasting complete solution in case gist gets deleted :)

# Setup HomeBrew if not already installed: https://brew.sh/
brew install openblas
mkdir /tmp/numpy_local
cd /tmp/numpy_local
pip3 download --no-binary :all: --no-deps numpy
unzip numpy-*.zip  # (assuming there's only one version in this folder)
cd numpy-1.19.1 # the version may be a later version than this

cat > site.cfg <<EOF
[openblas]
libraries = openblas
library_dirs = $(brew --prefix openblas)/lib
include_dirs = $(brew --prefix openblas)/include
runtime_library_dirs = $(brew --prefix openblas)/lib
EOF

pip3 install .
# cleanup
cd /tmp
rm -rf numpy_local

A slightly simpler solution:

$ rm -v ~/Library/Caches/pip/wheels/////numpy # clear the pip wheel cache of any built numpy wheels

Per https://twitter.com/pradyunsg/status/1317081239526936576?s=20 this can now be done with a built-in pip command:

$ pip cache remove numpy
$ brew install openblas # make sure OpenBLAS is installed
$ # activate your pypy3 virtualenv
$ OPENBLAS="$(brew --prefix openblas)" pip install numpy # let numpy's setup.py know where OpenBLAS is installed

This has the added benefit that the wheel will not need to be rebuilt, and any future pip install numpy's on pypy3 will work.

Workarounds aside though, why is the default build behavior to produce a broken wheel that can't even import? It seems like it being "by design" to build such a broken artifact is a bug either way; a self-test at the end of the build that aborts if it finds itself in this state would be a welcome addition. (Especially if it could provide a helpful error message like "try brew install openblas" or somesuch.)

Trying to import after the build sounds like a good idea.

Why not simply refuse to build on macOS if openblas is not found?

There are various valid blas backends, besides OpenBLAS. The only invalid one is Accelerate. Unfortunately, macOS does not provide a way to tell us that they are providing Accelerate, we can only detect it at runtime by triggering a known bad calculation. If you can figure out how to detect Accelerate earlier that would be great. The best we had so far is to detect a symlink, but that is very fragile.

If any python-poetry users come across this thread, I would just like to add that @glyph's solution works as well if you just replace the last line with OPENBLAS="$(brew --prefix openblas)" poetry add numpy

I've tried a previous version of numpy and it worked for me
pip3 install numpy==1.18.0

Hope this help

uninstall and install 1.18.0 -- worked for me! kuddos to the above ^^

I'm here to report that the suggestion to use openblas doesn't work. This crashes:

OPENBLAS="$(brew --prefix openblas)" pip install numpy
python -c 'import numpy'

# =>  RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.

while this works without any issues:

pip install numpy==1.18.0
python -c 'import numpy'

@fny check the output to see if pip is using a cached wheel. You might have to clear the cache in that case.

@fny see https://github.com/numpy/numpy/issues/15947#issuecomment-686159427 for updated instructions

Can anyone provide a link where one can read about why Accelerate is a bad BLAS implementation which should not be used? I tried googling "why apple accelerate is a bad blas" and found nothing.

I don't know where a comprehensive breakdown is, but Accelerate has threading problems, some wrong results, and exposes an outdated version of LAPACK.

For reference this solution fails. Why isn't this working? And why is any of this even necessary?

@orome as for "why" ask Apple. They are the ones you bought the system from and are shipping buggy software that the mostly volunteer team of NumPy has to work around. Any anger should should be directed at them, not at us.

@orome as for "why" ask Apple ... Any anger should should be directed at them, not at us.

That's definitely where it's directed (again)!

This fixed the problem for me. (I replaced pip with pip3 ... and the numpy version as of today will be 1.19.1)
https://gist.github.com/yatsu/47bdde35e8abbe7d14bbe730342aa9e0#file-numpy-openblas-macos-pip-sh

EDIT: Pasting complete solution in case gist gets deleted :)

# Setup HomeBrew if not already installed: https://brew.sh/
brew install openblas
mkdir /tmp/numpy_local
cd /tmp/numpy_local
pip3 download --no-binary :all: --no-deps numpy
unzip numpy-*.zip  # (assuming there's only one version in this folder)
cd numpy-1.19.1 # the version may be a later version than this

cat > site.cfg <<EOF
[openblas]
libraries = openblas
library_dirs = $(brew --prefix openblas)/lib
include_dirs = $(brew --prefix openblas)/include
runtime_library_dirs = $(brew --prefix openblas)/lib
EOF

pip3 install .
# cleanup
cd /tmp
rm -rf numpy_local

This somehow didn't work for me. I compiled OpenBLAS from source and installed it in /opt/OpenBLAS, and replaced every occurrence of $(brew --prefix openblas) with /opt/OpenBLAS. My Python 3 installation is from the Xcode command line tools, so I used pip3 install --user . instead of pip3 install .. Apart from that I followed the script exactly. However, after I run python3 -c "import numpy", the error message is exactly as before:

RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

Any ideas why this might have happened?

pip install pandas
pip3 install pandas
pip3 install pandas --no-build-isolation --no-cache-dir
sudo pip3 install pandas --no-cache-dir

Problem description

I've done a clean installation of macOS Big Sur and a clean install of Python3.9 via Homebrew (brew install python3). After that I installed pip3 with brew install pip command and I wanted to install Pandas but I'm always getting the following error. I've tried with all the commands listed above to install Pandas on macOS Big Sur and Python3...

Anyone can help? These are the only packages actually installed:

Package Version

pip 20.2.4
setuptools 50.3.2
wheel 0.35.1

Expected Output

Output of pip install pandas

In this issue Pandas#37880 they say that it's a numpy problem with the latest version of macOS. Any help??

Thanks in advance!

same issue with same error

This worked for me on MacOS 11.0.1 Big Sur, Python 3.8.2:

pip3 install --force-reinstall numpy\<1.19
pip3 install --upgrade wheel

brew install libjpeg # optional
pip3 install pillow # optional
pip3 install matplotlib

....

Successfully installed matplotlib-3.3.3 

This issue is closed. If you are using MacOS 11 with a python you installed from HomeBrew, as of Nov 2020 you should use brew install NumPy not pip install numpy. Please do not comment here, open a new issue if a careful reading of gh-17784 does not help you figure out what is going on.

These steps helped me to resolve the issue:
https://gist.github.com/mohammadnassiri/31ca8f4f2298d8694a8327814631455c