Numpy: [BUG] f2py unable to compile extension module if C sources are generated first

Created on 22 Nov 2019  ·  7Comments  ·  Source: numpy/numpy

Following the docs/User Guide from f2py, if one wants to generate a .c file with the extension module one must do

$ f2py -m fib1 fib1.f

to generate a fib1module.c file. This works. However, the next step to build the extension module to be importable in Python, according to the docs, is

$ f2py -c fib1module.c

However, this fails for me with the following messages:

$ f2py -c fib1module.c --verbose
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "untitled" sources
build_src: building npy-pkg config files
running build_ext
new_compiler returns <class 'distutils.unixccompiler.UnixCCompiler'>
customize UnixCCompiler
customize UnixCCompiler using build_ext
********************************************************************************
<class 'distutils.unixccompiler.UnixCCompiler'>
preprocessor  = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-E']
compiler      = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-std=c99']
compiler_so   = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/', '-Wsign-compare', '-DNDEBUG', '-g', '-fwrapv', '-O3', '-Wall', '-Wstrict-prototypes', '-fPIC', '-std=c99']
compiler_cxx  = ['g++', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/']
linker_so     = ['gcc', '-pthread', '-shared', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-L/opt/miniconda/envs/numpy-dev/lib', '-Wl,-rpath=/opt/miniconda/envs/numpy-dev/lib', '-Wl,--no-as-needed', '-Wl,--sysroot=/']
linker_exe    = ['gcc', '-pthread', '-B', '/opt/miniconda/envs/numpy-dev/compiler_compat', '-Wl,--sysroot=/']
archiver      = ['ar', 'rc']
ranlib        = None
libraries     = []
library_dirs  = []
include_dirs  = ['/opt/miniconda/envs/numpy-dev/include/python3.6m']
********************************************************************************
building 'untitled' extension
compiling C sources
C compiler: gcc -pthread -B /opt/miniconda/envs/numpy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c99

compile options: '-I/home/melissa/projects/numpy/numpy/core/include -I/opt/miniconda/envs/numpy-dev/include/python3.6m -c'
gcc: fib1module.c
fib1module.c:16:10: fatal error: fortranobject.h: Arquivo ou diretório inexistente
   16 | #include "fortranobject.h"
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
error: Command "gcc -pthread -B /opt/miniconda/envs/numpy-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c99 -I/home/melissa/projects/numpy/numpy/core/include -I/opt/miniconda/envs/numpy-dev/include/python3.6m -c fib1module.c -o /tmp/tmpyjgu4iq5/fib1module.o -MMD -MF /tmp/tmpyjgu4iq5/fib1module.o.d" failed with exit status 1

Doing

$ f2py -c fib1module.c -I/home/melissa/projects/numpy/numpy/f2py/src -m fib1module

seems to work (this is the location of my fortranobject.h file), but then the module is not importable:

>>> import fib1module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /home/melissa/projects/numpy/doc/source/f2py/fib1module.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFortran_Type

TL; DR: Is this a bug, or something we are not supposed to do (generate .c sources and then build the extension module in a separate step)?

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.18.0.dev0+8d217b0 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42) 
[GCC 7.3.0]
00 - Bug numpy.f2py

Most helpful comment

I am not sure which docs say this but the following is wrong:

$ f2py -c fib1module.c

f2py -c ... must be applied either to existing .pyf file (plus the source/object/library files) or one must specify -m <modulename> option (plus the sources/object/library files). f2py -c is not meant to compile the generated C/API extension modules directly (although it is likely able doing that with proper include directory flags and the source/object file inputs).

Use one of the followinig options:

f2py -c -m fib1 fib1.f

or

f2py -m fib1 fib1.f -h fib1.pyf
f2py -c fib1.pyf fib1.f

These examples are minimal in the sense that if anything is skipped in the above f2py command lines then a failure is expected.

All 7 comments

Thanks for the report @melissawm. We're talking about this page right: https://numpy.org/devdocs/f2py/getting-started.html?

I can confirm what you're seeing:

$ f2py -m fib1 fib1.f
$ f2py -c fib1module.c
...
fib1module.c:16:10: fatal error: 'fortranobject.h' file not found
#include "fortranobject.h"

However, the doc page does $ f2py -c -m fib1 fib1.f instead (generate code and compile it in one go), and that works as expected for me. I'm not sure if separating -m and -c was ever supported. It should be possible to make -c-only work, by automatically adding the right include path - probably not hard, but not sure if it's needed.

@pearu any thoughts?

I am not sure which docs say this but the following is wrong:

$ f2py -c fib1module.c

f2py -c ... must be applied either to existing .pyf file (plus the source/object/library files) or one must specify -m <modulename> option (plus the sources/object/library files). f2py -c is not meant to compile the generated C/API extension modules directly (although it is likely able doing that with proper include directory flags and the source/object file inputs).

Use one of the followinig options:

f2py -c -m fib1 fib1.f

or

f2py -m fib1 fib1.f -h fib1.pyf
f2py -c fib1.pyf fib1.f

These examples are minimal in the sense that if anything is skipped in the above f2py command lines then a failure is expected.

Actually, I'm talking about this page: https://numpy.org/devdocs/f2py/usage.html

There, in items 2 and 3, we have the following: Item 2 is to run f2py without the -m or -c options, which apparently doesn't work. Item 3 says

To build an extension module, use

f2py -c <options> <fortran files>       \
  [[ only: <fortran functions>  : ]     \
   [ skip: <fortran functions>  : ]]... \
  [ <fortran/c source files> ] [ <.o, .a, .so files> ]

If <fortran files> contains a signature file, then a source for an extension module is constructed,
all Fortran and C sources are compiled, and finally all object and library files are linked to the
extension module <modulename>.so which is saved into the current directory.

If <fortran files> does not contain a signature file, then an extension module is constructed by
scanning all Fortran source codes for routine signatures.

From what you said, I understand we're not supposed to use the .c sources directly, right? If we generate a .c file, is it unusable later?

Thanks for the input!

Actually, I'm talking about this page: https://numpy.org/devdocs/f2py/usage.html

This page describes the f2py command-line options and the different modes. It presumes that one understand how Python extension modules can be built, see for instance https://docs.python.org/3.8/extending/index.html

While f2py can be used for creating extension modules that wrap also C functions (contained in .c files) and hence the statement "we're not supposed to use the .c sources directly" is inaccurate.

One should distinguish the .c files that f2py generates and the .c files that contain user implementations of C functions. The .c file that f2py generates is useful only when one needs to have full control of how the extension module is built or for debugging f2py. In all other cases, one does not need to generate the extension module as it will be generated automatically in the build process.

I'm adding the explanation from @pearu above to the docs, if you have any other ideas let me know. Thank you all for your coments!

Feel free to close the issue (or should I do it only after that PR gets accepted?)

I suggest closing when the PR that tackles this issue is accepted.

I suggest closing when the PR that tackles this issue is accepted.

+1

@melissawm you can also add closes gh-14960 to one of your commit messages, then the issue gets auto-closed when the PR gets merged.

Was this page helpful?
0 / 5 - 0 ratings