Numpy: RuntimeError: implement_array_function method already has a docstring

After quite some testing, I'm pretty sure that is related to being run within uwsgi.

This rings a vague bell. IIRC uwsgi does some kind of docstring manipulation, like running with -OO.

This also happens in Spyder, which uses IPython for its Python console. They have a technique called "user module reloader", which "forces Python to reload modules which were imported when executing a file in a Python or IPython console". Basically this does del sys.modules[modname].

When trying to run a script which imports numpy, I get a stack trace ending in

[...]
import numpy as np

File "C:\Program Files\Python35\lib\site-packagesnumpy\__init__.py", line 142, in
from . import core

File "C:\Program Files\Python35\lib\site-packagesnumpy\core\__init__.py", line 17, in
from . import multiarray

File "C:\Program Files\Python35\lib\site-packagesnumpy\core\multiarray.py", line 14, in
from . import overrides

File "C:\Program Files\Python35\lib\site-packagesnumpy\core\overrides.py", line 47, in
""")

RuntimeError: implement_array_function method already has a docstring

Happens to me too (Django server):

  File "/opt/project/consensx/graph/values_graph.py", line 2, in <module>
    import matplotlib.pyplot as plt
  File "/pyroot/lib/python3.7/site-packages/matplotlib/__init__.py", line 138, in <module>
    from . import cbook, rcsetup
  File "/pyroot/lib/python3.7/site-packages/matplotlib/cbook/__init__.py", line 31, in <module>
    import numpy as np
  File "/pyroot/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/pyroot/lib/python3.7/site-packages/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/pyroot/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/pyroot/lib/python3.7/site-packages/numpy/core/overrides.py", line 47, in <module>
    """)
RuntimeError: implement_array_function method already has a docstring

Python 7.3.7
numpy: 1.17.2

The real issue here is NumPy does not support being re-initialized in the process. This goes back to gh-665 from 2012. Solving this would require carefully capturing all the global state we create in a struct, and accessing that via a getter on the module.

Oh, globals. Those are always fun.

Downgrading numpy to numpy==1.15.4 did the trick for me.

Downgrading numpy to numpy==1.15.4 did the trick for me.

This version of numpy (1.15.4) works with witch version of pandas ?
Because with pandas 1.0.3 =
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

Other question, do you think this bug will be fixed soon?

Thx a lot.

Other question, do you think this bug will be fixed soon?

If by "this bug" you mean allowing numpy to be deleted and reimported, then the answer is "It is not on our roadmap". This is open source so anyone can contribute toward fixing it, but it will be a rather large undertaking.

If all you have is an import warning, then you can ignore it. But I guess you are using uwsgi...

If this is due to uwsgi, then I only have the answer that we will not support it and you would have to put a very significant effort into this. At this time not even all of Python proper supports this, they are working on it with a vague hope that one day it will work. But asking us to make this work will simply not go anywhere. Sure, you can try to invest time and money into improving the situation, that is welcome, but don't get your hope up.

After Python gets their thing together on their end (this may take a while, they are working on this for many years) and there is a reasonably large userbase there might be some interest in it.
At this time, we have public API which to my knowledge is fundamentally incompatible with what wsgi does (use sub-interpreters), and we probably have some optimized caches which we probably want and Python to my knowledge does not yet provide any mechanism for making them fast in this setting.

The fact is that sub-interpreters which wsgi uses by default are simply not supported by a reasonable size of the C-extension system. And this will not happen for the foreseeable future. And if anyone thinks or expects this as an important thing of the NumPy roadmap, then in my opinion Python developers got their signals wrong (and I have said so, and it seemed to be acknowledged there). Because they should tell everyone that sub-interpreters must be assumed to fail horribly with the vast majority of C-extensions, that nobody should expect the situation to change in the mid term (next few years), and basically that the feature is experimental when it comes to C-extension support.

Yes many Python core devs are happily moving in the direction of making things work, but from the perspective of a project such as NumPy, supporting sub-interpreters (wsgi defaults) is no easier then the Python 2 to Python 3 transition with a fraction of potential users and currently not even the technology to properly support everything!

So, after this rant... I will close this, if anyone wants to spend the effort into making NumPy work (better with sub-interpreters) that is a noble effort. And I am happy to be proven wrong... But I stand with the point that sub-interpreters (wsgi), must be considered an experimental and feature. I.e. if you use sub-interpreters, its more like using some fringe Python implementation (like an early PyPy). And it took years before anyone using PyPy expected to run NumPy within PyPy (and PyPy did all the work in making it happen)...

Eh, I totally agree that we don't care about subinterpreters (yet anyway), but this is a simpler issue - it worked in 1.15.4 and the Django, Spyder, etc. reports do not involve uwsgi. I'll have a quick go at changing the docstring attachment in __array_function__.

True missed those. Although I wonder also there... it should be fairly safe in that context. In the sense that it hopefully only leaks memory. (I still wonder a bit if Spyder shouldn't just skip C-extension modules, at least until CPython gets more machinery to say that a C-extension module implements it safely... And even then, for most C-extension modules reloading probably does almost nothing).

And even then, for most C-extension modules reloading probably does almost nothing).

I don't think that's the point. It's much easier to say reload(any_module) than to figure out before trying that if there's some compiled extension hanging around as part of a Python package.

Not sure what Django does exactly, but it's probably similar.

I didn't want to annoy you too much.

I have a web service that uses Flask and Numpy, I guess I'm not the only one.
I have no errors with the development server (certainly Werkzeug) included in Flask.

And indeed, in production, I use the apache2 WSGI module for Python3, which allows me to hold the load (several threads)

I'm open, tell me what to use :

• Gunicorn or other server? but that's WSGI too
• mod_wsgi-express ?
• or what option should I change in mod_wsgi?, I read that in paragraph sub-interpreters in modwsgi's documentaion: WSGIApplicationGroup %{GLOBAL}

Thank you very much.

Anything in the sub-interpreter paragraph should be sound advice with regards to NumPy. I cannot give any advice on what may work for web-services, but maybe someone else in this thread can.

I just wanted to be clear that making the situation slightly better may be good (and help some), but will not solve the deep down problems in the foreseeable future. In any case, I guess we may try to remove this failure but instead give a huge warning that this can lead to hard to debug issues.

Maybe it will help someone as it wasn't mentioned before:
I'm using uwsgi (with Nginx) and the "single-interpreter = true" ini config (or --single-interpreter switch) solved this issue for me.

Here I found the explanation of this flag:

By default uWSGI will execute Python code within a sub interpreter of the process rather than the main Python interpreter created when Python is first initialized. This is done to allow multiple separate Python web applications to be run within the one process but to be sufficiently separated so as to not interfere with each other. Older versions of uWSGI can fail however when using sub interpreters with threading enabled. It is therefore safest to use this option and restrict yourself to a single web application per process. Running a single web application in each process with uWSGI is the normal use case so it would be unlikely that this restriction would be an issue.

So as a workaround you can switch to uwsgi with this config or try to find something similar for your environment.

Will reopen until we actually merge a fix/workaround - should be soon.

I had this same error occur to me yesterday, and I just couldn't find out what had happened. Just for completion's sake, I'll had my cause here: I had a logger module, named logger.py, that was overwriting something inside Numpy. Changed the module name, no errors. Maybe someone else will come across this comment and figure out something similar is happening to them.

We have pushed a fix for 1.19. Or maybe rather a workaround, since the error still indicates that something is going on that might create issues.

@seberg with the workaround in 1.19 should we close this?