Numpy: Problem compiling numpy with GCC 10.1

Created on 28 May 2020  ·  4Comments  ·  Source: numpy/numpy

When attempting to build numpy locally I had received the following error:

gcc: numpy/core/src/common/mem_overlap.c
gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fno-semantic-interposition -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o -L/usr/lib -Lbuild/temp.linux-x86_64-3.8 -lnpymath -o numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so
/usr/bin/ld: build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o:/home/ross/repos/numpy/numpy/core/include/numpy/ndarraytypes.h:1824: multiple definition of `PyArrayDTypeMeta_Type'; build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o:/home/ross/repos/numpy/numpy/core/include/numpy/ndarraytypes.h:1824: first defined here
collect2: error: ld returned 1 exit status
error: Command "gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fno-semantic-interposition -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o -L/usr/lib -Lbuild/temp.linux-x86_64-3.8 -lnpymath -o numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so" failed with exit status 1
`bisect`ing identified 1809a0fde as the source of the issue. Note that this error only occurred for me with gcc 10.1. No problems were encountered with gcc 9.3. ### Reproducing code example: 
$ git clone https://github.com/numpy/numpy.git
$ git checkout 1809a0fde
$ pip install -r test_requirements.txt
$ python setup.py build_ext -i
### Error message:
Full build log 
Running from numpy source directory.                                                                                         
Cythonizing sources                                                                                                          
numpy/random/_bounded_integers.pxd.in has not changed                                                                        
Processing numpy/random/mtrand.pyx                                                                                           
numpy/random/_common.pyx has not changed                                                                                     
Processing numpy/random/_mt19937.pyx                                                                                         
Processing numpy/random/_pcg64.pyx                                                                                           
Processing numpy/random/_generator.pyx                                                                                       
numpy/random/_bounded_integers.pyx has not changed                                                                           
numpy/random/_sfc64.pyx has not changed                                                                                      
numpy/random/_philox.pyx has not changed                                                                                     
numpy/random/bit_generator.pyx has not changed                                                                               
numpy/random/_bounded_integers.pyx.in has not changed                                                                        
numpy/core/setup.py:406: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 13, with checksum 17a0f366e55ec05e5c5c149123478452, but recorded checksum for C API version 13 in core/codegen_dir/cversions.txt is 5b0e8bbded00b166125974fc71e80a33. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.py.                                                                                             
  check_api_version(C_API_VERSION, codegen_dir)                                                                              
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_info:
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC

creating /tmp/tmpp_pb97eg/tmp
creating /tmp/tmpp_pb97eg/tmp/tmpp_pb97eg
compile options: '-c'
gcc: /tmp/tmpp_pb97eg/source.c
gcc -pthread /tmp/tmpp_pb97eg/tmp/tmpp_pb97eg/source.o -L/usr/lib64 -lopenblas -o /tmp/tmpp_pb97eg/a.out
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
  libraries mkl_rt not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_lapack_info:
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC

creating /tmp/tmpeicqr4ho/tmp
creating /tmp/tmpeicqr4ho/tmp/tmpeicqr4ho
compile options: '-c'
gcc: /tmp/tmpeicqr4ho/source.c
gcc -pthread /tmp/tmpeicqr4ho/tmp/tmpeicqr4ho/source.o -L/usr/lib64 -lopenblas -o /tmp/tmpeicqr4ho/a.out
/usr/bin/ld: /tmp/tmpeicqr4ho/tmp/tmpeicqr4ho/source.o: in function `main':
/tmp/tmpeicqr4ho/source.c:4: undefined reference to `zungqr_'
collect2: error: ld returned 1 exit status
  NOT AVAILABLE

openblas_clapack_info:
  libraries openblas,lapack not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries tatlas,tatlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries tatlas,tatlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib64
  libraries tatlas,tatlas not found in /usr/lib64
  libraries lapack_atlas not found in /usr/lib
  libraries tatlas,tatlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib/
  libraries tatlas,tatlas not found in /usr/lib/

  NOT AVAILABLE

atlas_3_10_info:
  libraries lapack_atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries satlas,satlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries satlas,satlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib64
  libraries satlas,satlas not found in /usr/lib64
  libraries lapack_atlas not found in /usr/lib
  libraries satlas,satlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib/
  libraries satlas,satlas not found in /usr/lib/

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries ptf77blas,ptcblas,atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib64
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
  libraries lapack_atlas not found in /usr/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib/
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/

  NOT AVAILABLE

atlas_info:
  libraries lapack_atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries f77blas,cblas,atlas not found in /home/ross/.virtualenvs/tmp/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib64
  libraries f77blas,cblas,atlas not found in /usr/lib64
  libraries lapack_atlas not found in /usr/lib
  libraries f77blas,cblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib/
  libraries f77blas,cblas,atlas not found in /usr/lib/

  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

lapack_info:
  libraries lapack not found in ['/home/ross/.virtualenvs/tmp/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

/home/ross/repos/numpy/numpy/distutils/system_info.py:1748: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  return getattr(self, '_calc_info_{}'.format(name))()
lapack_src_info:
  NOT AVAILABLE

/home/ross/repos/numpy/numpy/distutils/system_info.py:1748: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  return getattr(self, '_calc_info_{}'.format(name))()
  NOT AVAILABLE

numpy_linalg_lapack_lite:
  FOUND:
    language = c
    define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]

non-existing path in 'numpy/random': '{gen}.pxd'
non-existing path in 'numpy/random': '{gen}.pxd'
/usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running build_ext
running build_src
build_src
building py_modules sources
building library "npymath" sources
  adding 'numpy/core/src/npymath' to include_dirs.
None - nothing done with h_files = ['numpy/core/src/npymath/npy_math_internal.h']
building library "npysort" sources
  adding 'numpy/core/src/common' to include_dirs.
None - nothing done with h_files = ['numpy/core/src/common/npy_sort.h', 'numpy/core/src/common/npy_partition.h', 'numpy/core/src/common/npy_binsearch.h']
building library "npyrandom" sources
building extension "numpy.core._multiarray_tests" sources
conv_template:> numpy/core/src/multiarray/_multiarray_tests.c
building extension "numpy.core._multiarray_umath" sources
conv_template:> numpy/core/src/multiarray/arraytypes.c
conv_template:> numpy/core/src/multiarray/einsum.c
conv_template:> numpy/core/src/multiarray/lowlevel_strided_loops.c
conv_template:> numpy/core/src/multiarray/nditer_templ.c
conv_template:> numpy/core/src/multiarray/scalartypes.c
conv_template:> numpy/core/src/umath/funcs.inc
  adding 'numpy/core/src/umath' to include_dirs.
conv_template:> numpy/core/src/umath/simd.inc
conv_template:> numpy/core/src/umath/loops.h
conv_template:> numpy/core/src/umath/loops.c
conv_template:> numpy/core/src/umath/matmul.h
conv_template:> numpy/core/src/umath/matmul.c
conv_template:> numpy/core/src/umath/clip.h
conv_template:> numpy/core/src/umath/clip.c
conv_template:> numpy/core/src/umath/scalarmath.c
conv_template:> numpy/core/src/npymath/npy_math_internal.h
  adding 'numpy/core/src/npymath' to include_dirs.
conv_template:> numpy/core/src/npymath/ieee754.c
conv_template:> numpy/core/src/npymath/npy_math_complex.c
conv_template:> numpy/core/src/common/templ_common.h
  adding 'numpy/core/src/common' to include_dirs.
conv_template:> numpy/core/src/common/npy_cpu_features.c
numpy.core - nothing done with h_files = ['numpy/core/src/umath/funcs.inc', 'numpy/core/src/umath/simd.inc', 'numpy/core/src/umath/loops.h', 'numpy/core/src/umath/matmul.h', 'numpy/core/src/umath/clip.h', 'numpy/core/src/npymath/npy_math_internal.h', 'numpy/core/src/common/templ_common.h', 'numpy/core/include/numpy/config.h', 'numpy/core/include/numpy/_numpyconfig.h', 'numpy/core/include/numpy/__multiarray_api.h', 'numpy/core/include/numpy/__ufunc_api.h']
building extension "numpy.core._umath_tests" sources
building extension "numpy.core._rational_tests" sources
building extension "numpy.core._struct_ufunc_tests" sources
building extension "numpy.core._operand_flag_tests" sources
building extension "numpy.fft._pocketfft_internal" sources
building extension "numpy.linalg.lapack_lite" sources
### Warning:  Using unoptimized lapack ###
building extension "numpy.linalg._umath_linalg" sources
### Warning:  Using unoptimized lapack ###
building extension "numpy.random._mt19937" sources
building extension "numpy.random._philox" sources
building extension "numpy.random._pcg64" sources
building extension "numpy.random._sfc64" sources
building extension "numpy.random._common" sources
building extension "numpy.random.bit_generator" sources
building extension "numpy.random._generator" sources
building extension "numpy.random._bounded_integers" sources
building extension "numpy.random.mtrand" sources
building data_files sources
build_src: building npy-pkg config files
customize UnixCCompiler
customize UnixCCompiler using new_build_clib
building 'npymath' library
compiling C sources
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC

compile options: '-Inumpy/core/src/npymath -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/ross/.virtualenvs/tmp/include -I/usr/include/python3.8 -Inumpy/core/src/common -Inumpy/core/src/npymath -c'
extra options: '-std=c99'
gcc: numpy/core/src/npymath/halffloat.c
gcc: numpy/core/src/npymath/npy_math.c
gcc: numpy/core/src/npymath/npy_math_complex.c
gcc: numpy/core/src/npymath/ieee754.c
ar: adding 4 object files to build/temp.linux-x86_64-3.8/libnpymath.a
building 'npyrandom' library
compiling C sources
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC

compile options: '-Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/ross/.virtualenvs/tmp/include -I/usr/include/python3.8 -Inumpy/core/src/common -Inumpy/core/src/npymath -c'
extra options: '-std=c99'
gcc: numpy/random/src/distributions/distributions.c
gcc: numpy/random/src/distributions/random_mvhg_count.c
gcc: numpy/random/src/distributions/random_mvhg_marginals.c
gcc: numpy/random/src/distributions/random_hypergeometric.c
ar: adding 5 object files to build/temp.linux-x86_64-3.8/libnpyrandom.a
customize UnixCCompiler
customize UnixCCompiler using new_build_ext
building 'numpy.core._multiarray_tests' extension
compiling C sources
C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC

compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/ross/.virtualenvs/tmp/include -I/usr/include/python3.8 -Inumpy/core/src/common -Inumpy/core/src/npymath -c'
extra options: '-std=c99'
gcc: numpy/core/src/common/mem_overlap.c
gcc: numpy/core/src/multiarray/_multiarray_tests.c
gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fno-semantic-interposition -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o -L/usr/lib -Lbuild/temp.linux-x86_64-3.8 -lnpymath -o numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so
/usr/bin/ld: build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o:/home/ross/repos/numpy/numpy/core/include/numpy/ndarraytypes.h:1825: multiple definition of `PyArrayDTypeMeta_Type'; build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o:/home/ross/repos/numpy/numpy/core/include/numpy/ndarraytypes.h:1825: first defined here
collect2: error: ld returned 1 exit status
error: Command "gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fno-semantic-interposition -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.8/numpy/core/src/multiarray/_multiarray_tests.o build/temp.linux-x86_64-3.8/numpy/core/src/common/mem_overlap.o -L/usr/lib -Lbuild/temp.linux-x86_64-3.8 -lnpymath -o numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so" failed with exit status 1

Numpy/Python version information:

Python version: 3.8.3
NumPy version: 1809a0fde
GCC version: 10.1

00 - Bug build
Source
picture rossbar

Most helpful comment

The error is a missing extern on the declaration, gh-16435 should fix it, Ross tested it for me.

picture seberg on 29 May 2020
👍2

All 4 comments

I guess those definitions are probably not strictly OK to be there (now that you note it, clang static analysis is also giving me some warnings about it). They probably need to be moved to the specific header files until we make them public?

seberg picture seberg on 28 May 2020

I see that too, gcc 10.1.1. Upgraded my OS last night, there are always problems after...

charris picture charris on 29 May 2020

I am seeing this with

jupiter@23:06  ➤  gcc --version
gcc (GCC) 10.1.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
tacaswell picture tacaswell on 29 May 2020

The error is a missing extern on the declaration, gh-16435 should fix it, Ross tested it for me.

seberg picture seberg on 29 May 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

MareinK picture MareinK  ·  3Comments
Foadsf picture Foadsf  ·  3Comments
astrofrog picture astrofrog  ·  4Comments
Kreol64 picture Kreol64  ·  3Comments
thouis picture thouis  ·  4Comments