Psutil: [macOS] psutil compilation fails with the final version of Xcode 12

Created on 21 Sep 2020  ·  5Comments  ·  Source: giampaolo/psutil

Platform

  • macOS 10.15.6 (19G2021)
  • psutil 5.7.2
  • Python 3.8.2
  • Xcode 12.0 (12A7209)
  • clang --version output:
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Bug description

psutil setup in venv fails with

Collecting psutil
  Using cached https://files.pythonhosted.org/packages/aa/3e/d18f2c04cf2b528e18515999b0c8e698c136db78f62df34eee89cee205f1/psutil-5.7.2.tar.gz
Installing collected packages: psutil
  Running setup.py install for psutil ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/lsoltic/Development/testxc12/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"'; __file__='"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-record-qtwmxusr/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lsoltic/Development/testxc12/include/site/python3.8/psutil
         cwd: /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/
    Complete output (141 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14.6-x86_64-3.8
    creating build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pswindows.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_common.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psosx.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psbsd.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psaix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pslinux.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_compat.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psposix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pssunos.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    creating build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/runner.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_memleaks.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_testutils.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    running build_ext
    building 'psutil._psutil_osx' extension
    creating build/temp.macosx-10.14.6-x86_64-3.8
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil/arch
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil/arch/osx
    xcrun -sdk macosx clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_OSX=1 -I/Users/lsoltic/Development/testxc12/include -I/Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c psutil/_psutil_common.c -o build/temp.macosx-10.14.6-x86_64-3.8/psutil/_psutil_common.o
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
    In file included from /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/limits.h:63:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
    #error Unsupported architecture
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
    In file included from /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/limits.h:64:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
    #error architecture not supported
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:33:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
    #error architecture not supported
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
    typedef __int64_t       __darwin_blkcnt_t;      /* total blocks */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_blksize_t;     /* preferred block size */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_dev_t;         /* dev_t */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_gid_t;         /* [???] process and group IDs */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_id_t;          /* [XSI] pid_t, uid_t, or gid_t*/
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t'
    typedef __uint64_t      __darwin_ino64_t;       /* [???] Used for 64 bit inodes */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t'
    typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'?
    typedef __uint16_t      __darwin_mode_t;        /* [???] Some file attributes */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t'
    typedef __int64_t       __darwin_off_t;         /* [???] Used for file sizes */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_pid_t;         /* [???] process and group IDs */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_sigset_t;      /* [???] signal set */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_suseconds_t;   /* [???] microseconds */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_uid_t;         /* [???] user IDs */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_useconds_t;    /* [???] microseconds */
            ^
    note: '__uint128_t' declared here
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:43:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_wctype_t;
            ^
    note: '__uint128_t' declared here
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:75:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types/_va_list.h:31:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/types.h:37:2: error: architecture not supported
    #error architecture not supported
     ^
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
    error: command 'xcrun' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/lsoltic/Development/testxc12/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"'; __file__='"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-record-qtwmxusr/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lsoltic/Development/testxc12/include/site/python3.8/psutil Check the logs for full command output.

Steps

python3 -m venv testxc12
cd testxc12
. bin/activate
pip install psutil

Comments

In clang call I see the options -arch arm64 -arch x86_64 but macOS 10.15 SDK is still used (which doesn't support ARM). So either -arch arm64 should be removed or macOS 11 SDK should be used.

After a quick look inside Xcode to view the available SDK, it looks like macOS 11 SDK is missing, so the only option looks to remove -arch arm64 for now.

bug macos
Source
picture Ceylo
👍9 👀1 😕1

Most helpful comment

While I was hitting this when installing a different package (cffi) I was able to work around this by exporting an environment variable
export ARCHFLAGS="-arch x86_64"

picture japaniel on 7 Oct 2020
👍20 🎉6 ❤5 🚀2

All 5 comments

It is not a problem of psutils (it doesn't add -arch options to setup.py) but an Apple's fault. Apple builds python3 with -arch arm64 -arch x86_64 flags that are used as default flags in setup.py that can't be skipped . You can check your python3 flags with:

import sysconfig
print(sysconfig.get_config_vars())

and -arch arm64 should be there.

darthunix picture darthunix on 25 Sep 2020

Oh I wasn't aware that changing the active Xcode version was also changing the Python 3 being used (unless using Python 3 from Homebrew I guess), I thought only clang & xcodebuild related tools were affected.

And I'm not familiar with Python module dev so wasn't aware that build flags were taken from Python's config.

Then I suppose final Xcode 12 is just not usable if we need psutil, and we need Xcode 12 beta or Xcode 12.2 beta or any Xcode 11 for now.

Ceylo picture Ceylo on 25 Sep 2020

Reverting back to 11.7 is fixing the issue, although it is still annoying.

Exitare picture Exitare on 25 Sep 2020

BTW, python3 from brew is not affected by -arch arm64. Just install it and add to your $PATH

darthunix picture darthunix on 25 Sep 2020

While I was hitting this when installing a different package (cffi) I was able to work around this by exporting an environment variable
export ARCHFLAGS="-arch x86_64"

japaniel picture japaniel on 7 Oct 2020
👍20 🎉6 ❤5 🚀2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

SimiCode picture SimiCode  ·  3Comments
RedBearAK picture RedBearAK  ·  4Comments
cyril23 picture cyril23  ·  12Comments
wemakeweb picture wemakeweb  ·  3Comments
AdnanHodzic picture AdnanHodzic  ·  4Comments