Numpy: Numpy 1.10.2 + MKL = crash (Windows)

Hmm, don't know what is going on here. Intel made some changes to the build, but I don't think those are involved. Python 3.5 is a possible culprit as it is compiled with a newer VS on windows. @cgohlke thoughts?

I think your best bet is to provide a reproducer (what are the "some functions" that produce this?), and either hope that @cgohlke pops up with the answer or else file a bug with Anaconda (since they have the resources to support their MKL builds and we don't).

Sorry for the "some functions" :) - it crashed indirectly though matplotlib.

This crashes just the same:

A = np.arange(2*3).reshape(2,3).astype(np.float32)
B = np.arange(2*3*2731).reshape(2,3,2731).astype(np.int16)

np.dot(A, B)

I did a quick check with the latest @cgohlke numpy (1.10.2) python-3.4 (64bit) as well as python-3.5 (64bit) on my laptop:

>>> np.show_config()
openblas_lapack_info:
  NOT AVAILABLE
lapack_mkl_info:
    include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
    libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
    library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
lapack_opt_info:
    include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
    libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt', 'mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
    library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
mkl_info:
    include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
    libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
    library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
blas_mkl_info:
    include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
    libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
    library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
blas_opt_info:
    include_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include']
    libraries = ['mkl_lapack95_lp64', 'mkl_blas95_lp64', 'mkl_rt']
    library_dirs = ['C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/lib/intel64_win']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]

>>> import numpy as np
>>> A = np.arange(2*3).reshape(2,3).astype(np.float32)
>>> B = np.arange(2*3*2731).reshape(2,3,2731).astype(np.int16)
>>> np.dot(A, B)
array([[[  13655.,   13658.,   13661., ...,   21839.,   21842.,   21845.],
        [  38234.,   38237.,   38240., ...,   46418.,   46421.,   46424.]],

       [[  38234.,   38246.,   38258., ...,   70970.,   70982.,   70994.],
        [ 136550.,  136562.,  136574., ...,  169286.,  169298.,  169310.]]], dtype=float32)

Both works as expected for me.

I guess the error Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll could be explained due to a wrong versioned mkl_intel_thread.dll found somwhere on PATH. Locating MKL DLLs by extending the PATH as done in numpy+MKL is fragile if another MKL runtime is installed on the system.

Are there any libiomp5md.dll or mkl_*.dll files in the C:\Windows* directories?

@cgohlke, in numpy+MKL you introduced _init_numpy_mkl() to add numpy.core to the PATH.

Did you consider the following approach from https://gist.github.com/matham/f6a07f8fb5e403feb440 to change the DLL search order?

See also:
https://pytools.codeplex.com/workitem/1627 (where it comes from)
https://github.com/numpy/numpy/wiki/windows-dll-notes#python-dlls
http://stackoverflow.com/questions/23804438/find-library-in-ctypes

@cgohlke These are the DLLs found in my Windows directory:

['System32\\libiomp5md.dll',
 'System32\\mkl_core.dll',
 'System32\\mkl_def.dll',
 'System32\\mkl_intel_thread.dll',
 'SysWOW64\\libiomp5md.dll',
 'SysWOW64\\mkl_core.dll',
 'SysWOW64\\mkl_intel_thread.dll',
 'SysWOW64\\mkl_p4.dll',
 'SysWOW64\\mkl_p4m.dll',
 'SysWOW64\\mkl_p4m3.dll',
 'SysWOW64\\mkl_p4p.dll']

Trying to reset my PATH with SET PATH=; before invoking python still crashes but removing above DLLs entirely from the Windows directory stops the crash. Seems like a path precedency issue @carlkl.

@cgohlke, I propose following change to test with numpy+MKL's __init__.py

def _init_numpy_mkl():
    # Numpy+MKL on Windows only
    import os
    import ctypes
    if os.name != 'nt':
        return
    # disable Intel Fortran default console event handler
    env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
    if env not in os.environ:
        os.environ[env] = '1'
    # extend DLL search order with numpy.core
    # See https://github.com/numpy/numpy/wiki/windows-dll-notes#python-dlls
    # and https://pytools.codeplex.com/workitem/1627
    try:
        _AddDllDirectory = ctypes.windll.kernel32.AddDllDirectory
        _AddDllDirectory.argtypes = [ctypes.c_wchar_p]
        # Needed to initialize AddDllDirectory modifications
        ctypes.windll.kernel32.SetDefaultDllDirectories(0x1000)
    except AttributeError:
        _AddDllDirectory = ctypes.windll.kernel32.SetDllDirectoryW
        _AddDllDirectory.argtypes = [ctypes.c_wchar_p]
    try:
        _core = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
        _AddDllDirectory(_core)
    except Exception:
        pass

_init_numpy_mkl()
del _init_numpy_mkl

removing above DLLs entirely from the Windows directory stops the crash.

Good riddance.

I propose following change to test with numpy+MKL's __init__.py

That won't really work on older systems and more importantly it breaks packages that depend on the system's DLL search order. It might be acceptable for distributions like Anaconda or WinPython.

I find it amusing that I initially found the AddDllDirectory method in the numpy wiki which was credited partially to @carlkl, but then my usage of it was proposed here by @carlkl as an example, and I ended up here because of an issue with that example code when searching numpy for AddDllDirectory. It's full circle...

Anyway, I did want to note for others who consider using this, that the line ctypes.windll.kernel32.SetDefaultDllDirectories(0x1000) (and any other of the none-zero flags) causes ImportError: DLL load failed: The parameter is incorrect exceptions when loading some dlls. For me it only happened specifically with a mingw-w64 (mingwpy) built extension that linked to ffmpeg dlls that were cross-compiled on debian with mingw-w64. But I don't know if that was responsible.

However, because python calls LoadLibraryEx only with LOAD_WITH_ALTERED_SEARCH_PATH and not with LOAD_LIBRARY_SEARCH_USER_DIRS, unless SetDefaultDllDirectories is also called with LOAD_LIBRARY_SEARCH_USER_DIRS, LoadLibraryEx will not search the directories added with AddDllDirectory. To me this seems a little bit to fragile to use, so I think I'll switch back to adding the dll directory to path with os.environ. I do wonder whether I should update the wiki with this info, though?

Anyway, hopefully this wasn't too far off topic.

@matham, this code snippet should be credited to _zooba_ - see https://pytools.codeplex.com/workitem/1627. The line ctypes.windll.kernel32.SetDefaultDllDirectories(0x1000) doesn't change the behaviour of LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH. It does _hopefully_ change the behaviour of LoadLibraryA (_without_ LOAD_WITH_ALTERED_SEARCH_PATH) that it used to load MKL_INTEL_THREAD.DLL right after random\mtrand.pyd is loaded and MKL_RT.DLL then calls MKL_INTEL_THREAD.DLL:

log from depency walker during python -c "import numpy"

...
LoadLibraryExW("d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\random\mtrand.cp35-win_amd64.pyd", 0x0000000000000000, LOAD_WITH_ALTERED_SEARCH_PATH) called from "d:\devel\py\python-3.5.0.amd64\PYTHON35.DLL" at address 0x00000000587E7747.
Loaded "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\random\MTRAND.CP35-WIN_AMD64.PYD" at address 0x000007FEF3310000.  Successfully hooked module.
DllMain(0x000007FEF3310000, DLL_PROCESS_ATTACH, 0x0000000000000000) in "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\random\MTRAND.CP35-WIN_AMD64.PYD" called.
DllMain(0x000007FEF3310000, DLL_PROCESS_ATTACH, 0x0000000000000000) in "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\random\MTRAND.CP35-WIN_AMD64.PYD" returned 1 (0x1).
LoadLibraryExW("d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\random\mtrand.cp35-win_amd64.pyd", 0x0000000000000000, LOAD_WITH_ALTERED_SEARCH_PATH) returned 0x000007FEF3310000.
LoadLibraryA("d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\core\mkl_intel_thread.dll") called from "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\core\MKL_RT.DLL" at address 0x000007FEE17C4772.
Loaded "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\core\MKL_INTEL_THREAD.DLL" at address 0x000007FEE0520000.  Successfully hooked module.
Loaded "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\core\MKL_CORE.DLL" at address 0x000007FEDEEC0000.  Successfully hooked module.
Loaded "d:\devel\py\python-3.5.0.amd64\lib\site-packages\numpy\core\LIBIOMP5MD.DLL" at address 0x000007FEE43E0000.  Successfully hooked module.
...

@carlkl true, but I was referring to the wiki: (thanks to Steve Dower for the fragment, Carl Kleffner for finding it), which I assumed was you :)

The fragility I was referring to is that in general there may be some other code executed between the SetDefaultDllDirectories call which sets that flag and when the library is actually loaded. E.g. if you import something that itself calls SetDefaultDllDirectories perhaps with a value (e.g. 0) which may result in in the dll not found later when imported. Although I guess this may not be an issue here if it's immediately imported.

@cgohlke, is there something wrong with preloading MKL_RT.DLL, MKL_INTEL_THREAD.DLL and MKL_CORE.DLL with explicit pathname to numpy.core within _init_numpy_mkl()? I'm not sure though if gradually loading of other necessary MKL DLLs works then.

@cgohlke,

the following snippet ensures(?) that the MKL DLLs are loaded from numpy.core and not from any windows system or Intel redist folder. The order of loading the DLLs matters. At least it works for Windows 7 python-3.4 bit.

def _init_numpy_mkl():
    # Numpy+MKL on Windows only
    import os
    import ctypes
    if os.name != 'nt':
        return
    # disable Intel Fortran default console event handler
    env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
    if env not in os.environ:
        os.environ[env] = '1'
    # preload MKL DLLs from numpy.core
    try:
        _core = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
        for _dll in ('mkl_rt', 'libiomp5md', 'mkl_core', 'mkl_intel_thread', 
                     'libmmd', 'libifcoremd', 'libimalloc'):
            ctypes.cdll.LoadLibrary(os.path.join(_core,_dll))
    except Exception:
        pass

_init_numpy_mkl()
del _init_numpy_mkl

I had the same Problem and traced it back to installed mkl_intel_thread.dll in C:\Windows\System32 as indicated by danielgeier.
The older version didn't have the mkl_aa_fw_get_max_memory among other things.
The creation time coincided with the installation of Asio4all audio drivers. After I uninstalled those, which removed several dlls , it worked again (better than just deleting dlls in system32 on my own).
I played around with rolling back and updating the anaconda distribution but that did not work. I also couldn't disable mkl with anaconda on windows.

@sebastian-schmidt, can you try out, if explicit loading MKL DLLs from the numpy/core folder works for you, even with installed outdated MKL DLLs in the Windows system folder. See above: https://github.com/numpy/numpy/issues/6923#issuecomment-169073613

@carlkl , where do I have to add the snippet?
I tried adding it to the file I used as a test: https://gist.github.com/sebastian-schmidt/9bb97354b481750209fd3dac1e748d31 and it did not work.
The joined path is just a subdirectory from where I call the file.
Do I have to add it to the init of Numpy module?

The function _init_numpy_mkl() is part of @cgohlke numpy+mkl wheels. You can find it in the __init__.py file. It is not part of the official numpy sources.

Here is the version found in numpy-1.10.4+mkl-cp34-none-win32.whl

def _init_numpy_mkl():
    # Numpy+MKL on Windows only
    import os
    if os.name != 'nt':
        return
    # disable Intel Fortran default console event handler
    env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
    if env not in os.environ:
        os.environ[env] = '1'
    # prepend the path of the Intel runtime DLLs to os.environ['PATH']
    try:
        path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
        if path not in os.environ.get('PATH', ''):
            os.environ['PATH'] = os.pathsep.join((path, os.environ.get('PATH', '')))
    except Exception:
        pass

_init_numpy_mkl()
del _init_numpy_mkl

This function adds the numpy/core folder to the PATH environment. However, DLLs found in the the windows system folder are favoured to DLLs found in the PATH. Welcome to the DLL hell.

The alternative version of this function https://github.com/numpy/numpy/issues/6923#issuecomment-169073613 loads the MKL DLLs from numpy/core in correct order _without_ changing the PATH environment. All DLLs names are given as explicit filenames with fully qualified pathname, so I don't expect problems with shadowing DLLs in the windows system folder, but I haven't tested this.

However, I didn't care about frozen numpy in this code snippet.

I changed the _init_numpy_mkl() in the Anaconda distribution to your version and then it worked (with incompatible dlls in windows/system32.

Great to hear!

The code in _init_numpy_mkl() is part of @cgohlke's numpy distribution and under his responibility.
@cgohlke: any comments on https://github.com/numpy/numpy/issues/6923#issuecomment-169073613?

@sebastian-schmidt, can you test the version using AddDllDirectory (or SetDllDirectoryW as a fallback): see above: https://github.com/numpy/numpy/issues/6923#issuecomment-168490201

@carlkl , the AddDllDirectory / SetDllDirectoryW version also works.
Test was always only the np.dot() example by danielgeier.
On a side note, the software that installs the mkl libraries was Amplitube and not the Asio4all drivers.

I, too, traced back the issue to Amplitube.

when i tried : import matplotlib.pyplot as plt with both init versions I got an error:

---> 60 import matplotlib._png as _png
     61 ####################
     62 

ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

So I switched back to the Anaconda original version and and leave Amplitube desinstalled.

@sebastian-schmidt, at least it works for me. Importing numpy, scipy, matplotlib._png works without problems on my box.

However, deinstalling MKL DLLs from the windows system folders seems the best solution for this kind of problems.

Time to close this issue?

Hi guys. Just in case someone is still with this problem. I have numpy 1.12 + mkl and had the same issue. I uninstalled Amplitube 3 and now everything its working!

Hi guys,

I just installed the latest numpy+mkl 1.13 package from here by @cgohlke .

I'm running Anaconda w/ Python 3.6.1 64-bit on Windows 7.

After that I'm getting this DLL issue as well, it was working fine for v1.12 from the same site before.

To add, actually all code works in a Jupyter notebook.... This only happens in a console.

@carlkl I searched in my Anaconda folder for _init_numpy_mkl but couldn't find file that contained this string. Manually appending numpy/core to my PATH env variable did not work either.

Any other suggestions?

Thanks.

when i imported skimage , i met the problem "cannot load the mkl_intel_thread.dll".
I can't find any mkl_intel_thread.dll in windows/system32, so i deleted the libiomp5md.dll in that folder just as @cgohlke mentioned , it works！ thanks you a lot! the problem has troubled me weeks. Anyway, i hope others met this problem can get help from this. Try to find libiomp5md.dll , it may help.

I wonder if https://github.com/numpy/numpy/issues/6923#issuecomment-169073613 works in that case without the need to remove MKL DLLs from windows/system32. This snippet shows, that it is possible to load seven (but not all) MKL DLLs in the numpy/core folder without the need to pass extra libs to the PATH environment variable or use other tricks. It is important to load these 7 DLLs in the correct sequence. All other required MKL DLLs are then automatically loaded from numpy/core instead of the windows/system32 folder.

It is yet unclear if this works in all conceivable szenarios. For example frozen python applications. Also Windows OS versions may differ in the loading behaviour of the DLL.

It seems to me advantageous to further investigate this variant.

We went back and forth with SetDefaultDllDirectories, #10229 added it then #11493 removed it. I think we have gone as far as we can go with this. It seems the best option is to remove other copies of mkl from PATH.

Please reopen if you disagree.