Scikit-learn: I don't understand how to built with anaconda / MKL

how did you iinstall it?

it's pretty automatic as for me I just need to add a license file and run
conda upgrade (or something like that)

I did that too, and that gives me numpy with mkl. But if I then try to compile sklearn I get linking errors on import.

hum I don't know. Maybe a conflict with your local python and anaconda
during setup :(

hm, maybe i need to set LD_LIBRARY_PATH or something?

I have mine set to something like

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/u/kkastne/miniconda/lib/

though ordering may be important if it finds other libraries from other
python installs first.

On Wed, Jul 29, 2015 at 5:01 PM, Andreas Mueller [email protected]
wrote:

hm, maybe i need to set LD_LIBRARY_PATH or something?

—
Reply to this email directly or view it on GitHub
https://github.com/scikit-learn/scikit-learn/issues/5046#issuecomment-126094651
.

it was a combination of mixing up system numpy and the LD_LIBRARY_PATH... always good fun.... sorry for the noise.

I am getting the same problem. @amueller Noob question: where do you search for the mixup? (ld_lib_path)

I am having the same problem, using pyinstaller in anaconda to compile my program. When I execute it on certain machines, it raises the above mentioned error. In my case, if I echo $LD_LIBRARY_PATH I see a couple of paths that our system manager set for some programs to work. I tried to set it to the anaconda lib path, and to leave it empty, but it keeps producing the error.

I think there is a new error, I just got it on a travis instance.

When does your problem occur? Because in my case, I could install with conda install numpy, and did not get error messages. Afterwards, I successfully generated a one-file binary with pyinstaller. It is when actually using the executable that, at some point, a call to some Biopython code is raising the error. So I guess you are also having the problem when testing the code, at runtime, right?

If I check my conda installation, both libmkl_avx2.so and libmkl_def.so are there, at the $home/anaconda2/lib folder. I have also tried to add the files as binaries in the pyintaller spec file, but none of it works...

I think there is a new error, I just got it on a travis instance.

@amueller I have seen this error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

on Travis as well. From what I could gather it was due to numpy 1.10.2 missing some mkl stuff. numpy 1.10.4 was fine. This was noted in https://github.com/scikit-learn/scikit-learn/pull/6508#issuecomment-194908920.

For the record I never had to set LD_LIBRARY_PATH to build scikit-learn with anaconda (with or without MKL).

@clacri is your problem related to scikit-learn at all ? If not it looks like you should ask your question to either the pyinstaller or the Biopython folks ...

From what I could gather it was due to numpy 1.10.2 missing some mkl stuff.

Here is a snippet that reproduces the problem on my machine:

conda create -n mkl --yes python=3.5 pip nose numpy=1.10.2 scipy=0.16.1 cython=0.23.4 mkl
. activate mkl
cd ~/dev/scikit-learn
make in
python -c 'import sklearn.linear_model.tests.test_randomized_l1'

Intel MKL FATAL ERROR: Cannot load libmkl_mc3.so or libmkl_def.so.

No, you are right. I should post on their issues list. It is just that I saw the error pointed here was the same and also while using anaconda, so I thought it might be related. I already asked on the pyinstaller project, because it seems to have a similar issue with other mkl libraries. In any case, I checked my numpy version in anaconda, and it is 1.10.4. Thanks anyway :)

Just for the record, browsing through your referenced comment, I found the post at Anaconda blog where they describe how to inactivate the mkl versions of the libraries (https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations) and now at least I can compile a working version. Thanks!

Because of the same problem to me, I turn off mkl by conda install nomkl. I am now happy. If this problem is fixed, please share with us so that I can turn on again. Thank you for elaborating this problem.

Same problem

Followed the link above and solved my problem. Thanks.

It was solved now at least to me. You can try with new numpy 11.
On Wed, Apr 27, 2016 at 10:42 PM gjzhu [email protected] wrote:

Same problem. Don't know how to deal with that.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/scikit-learn/scikit-learn/issues/5046#issuecomment-215292024

Sungjin (James) Kim, PhD

Postdoc, CCB in Harvard

[email protected]

[Web] http://aspuru.chem.harvard.edu/james-sungjin-kim/

[Linkedin] https://www.linkedin.com/in/jamessungjinkim

[Facebook] https://www.facebook.com/jamessungjin.kim

[alternative email] jamessungjin.[email protected]

conda install numpy
Fixed the error for me

Good. It also works for me as I noted before.

On Tue, May 3, 2016 at 11:26 AM alinabee [email protected] wrote:

conda install numpy
Fixed the error for me

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/scikit-learn/scikit-learn/issues/5046#issuecomment-216563766

Sungjin (James) Kim, PhD

Postdoc, CCB in Harvard

[email protected]

[Web] http://aspuru.chem.harvard.edu/james-sungjin-kim/

[Linkedin] https://www.linkedin.com/in/jamessungjinkim

[Facebook] https://www.facebook.com/jamessungjin.kim

[alternative email] jamessungjin.[email protected]

Hi,

Just wanted to note that Anaconda 4.0.0, shipped with mkl enabled by default, has this issue.
The problem is indeed with Anaconda, as it can be reproduced with the python sklearn test suggested above by @pcgreat.

The actual issue is that Anaconda linked with mkl, but not with libmkl_core.so, thus it has a missing symbol, and can be seen by running:

$ LD_DEBUG=symbols python -c 'import sklearn.linear_model.tests.test_randomized_l1' 2>&1 | grep -i error
      2200:     /opt/anaconda/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

I didn't want to uninstall mkl, as I'd like to have the performance boost, so I found a workaround which worked for me - preload libmkl_core.so before execution.

$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
$

$ LD_PRELOAD=/opt/anaconda/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'
$

Regards,
Yanir.

It is so wonderful solution thought.

On Wed, May 11, 2016 at 7:00 AM yanirj [email protected] wrote:

Hi,

Just wanted to note that Anaconda 4.0.0, shipped with mkl enabled by
default, has this issue.
The problem is indeed with Anaconda, as it can be reproduced with the
python sklearn test suggested above by @pcgreat
https://github.com/pcgreat.

The actual issue is that Anaconda linked with mkl, but not with
libmkl_core.so, thus it has a missing symbol, and can be seen by running:

$ LD_DEBUG=symbols python -c 'import sklearn.linear_model.tests.test_randomized_l1' 2>&1 | grep -i error
2200: /opt/anaconda/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

I didn't want to uninstall mkl, as I'd like to have the performance boost,
so I found a workaround which worked for me - preload libmkl_core.so before
execution.

$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
$

$ LD_PRELOAD=/opt/anaconda/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'
$

Regards,
Yanir.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/scikit-learn/scikit-learn/issues/5046#issuecomment-218427199

Sungjin (James) Kim, PhD

Postdoc, CCB in Harvard

[email protected]

[Web] http://aspuru.chem.harvard.edu/james-sungjin-kim/

[Linkedin] https://www.linkedin.com/in/jamessungjinkim

[Facebook] https://www.facebook.com/jamessungjin.kim

[alternative email] jamessungjin.[email protected]

I had the same problem, despite using the latest available packages. Turns out the solution was easier than I thought: for whatever reason Anaconda installed the MKL-enabled versions of the numpy/scipy stack, but did not actually install mkl itself. I have seen this when building Docker images based on the Jupyter minimal notebook stack.

A simple conda install --yes mkl mkl-service solved it for me.

I was opening an ipython terminal with the library preloaded and run into the following error when i run one of the pca routines.
:
/home/rp2801/anaconda2/bin/python: symbol lookup error: /home/rp2801/anaconda2/lib/libmkl_core.so: undefined symbol: mkl_serv_get_max_threads
I opened my ipython terminal as
$ LD_PRELOAD=/path/to/libmkl_core.so ipython

https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations
this link solved the problem

I was getting the same error. Alternative to removing MKL switching to earlier cvxopt built also solved the issue for me :

conda install cvxopt=1.1.8=py35_0

I solve the problem successfully, this is the tutorial https://docs.continuum.io/mkl-optimizations/
the command:
conda update conda
conda update anaconda
conda update mkl

I found that I had the same problem under numpy: 1.11.2, which could be reproduced by
$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

After downgrading numpy to 1.11.1 this error disappeared.
conda install numpy=1.11.1
The following packages will be DOWNGRADED due to dependency conflicts:
numpy: 1.11.2-py35_0 --> 1.11.1-py35_0
Proceed ([y]/n)? y

[SOLVED] http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error

Adding
import numpy as np
at the begin of the script solved the problem :-)

Didn't want to disable MKL with nomkl.
Adding

import mkl
mkl.get_max_threads()

Solved this problem for me.

conda install numpy
solved the problem for me.

conda install -c intel mkl

Worked for me

conda install numpy
solved the problem for me too.