Mayavi: Python 3 support

@hadim The plans for Mayavi on Python 3 are linked to VTK being available on Python 3. The roadmap to get there is unclear. See http://www.vtk.org/Wiki/VTK/Python_Wrapping_FAQ.

:+1:

:+1:

:+1:

I don't think a :thumbsup: here is gone do much good. You should rather try to convince Kitware to support python3 in VTK. Without that there is not any chance for a python3 version of Mayavi.

I don't think a :thumbsup: here is gone do much good. You should rather try to
convince Kitware to support python3 in VTK. Without that there is not any
chance for a python3 version of Mayavi.

Thank you for stressing that!

Maybe you can provide link to proper bugreport in VTK about porting it to Python 3, to redirect users saying +1 there?

For the record I opened a ticket on VTK bug tracker : http://www.vtk.org/Bug/view.php?id=15554

Great news! David Gobbi (the maintainer of the VTK Python bindings) has responded on that bug ticket and stated that he's already landed merge requests to prepare the bindings for Py3 support, and he's concretely working on Py3 support starting this weekend:
http://www.vtk.org/Bug/view.php?id=15554#c34787

I completely understand that this is just the start of the process and it may be some time before Python 3 support lands in a release of VTK, but I wonder if there is anything Mayavi could actively be doing to prepare itself for Python 3? I take it there will be changes needed to the Mayavi codebase itself to support Python 3, is it worth starting a discussion around what this work would be?

Thanks @jabooth,

As far as I can tell the full picture for Mayavi moving to Python 3 depends on the main dependencies supporting python 3. The current status is:

• [ ] python vtk
• [x] traits
• [x] traitsui
• [x] pyface
• [ ] envisage

Python 3 VTK bindings is the main part and the fact that there is work taking place is good news, but without the bindings available on a stable release of VTK it would be very difficult to work and test on a python 3 friendly Mayavi.

Furthermore, Mayavi itself depends heavily on traits, traitsui and pyface which need to also support python 3. While there is work-in-progress in these libraries support is not there yet.

_Nevertheless_, what we can do at this moment, (and @dmsurti started already) is to fix pending bugs improve testing coverage and simplify the Mayavi codebase so that when the dependencies are available in python 3 the effort for moving Mayavi to Python 3 will be less.

You can also do things like switch to "from future import
print_function, division", get rid of any relative imports, and switching
everything over to six.

I think doing anything right now is a waste of time. Mayavi does not do too much which cannot be fixed in a day of work, the bigger issue is VTK supporting Python 3 and as soon as that is available we will make the necessary change. Without VTK, we will not be able to test. I am aware of David Gobbi's recent work but I am going to wait until it is officially supported and merged into VTK.

Looks like the python-py3k branch was just merged: https://gitlab.kitware.com/vtk/vtk/merge_requests/478

Does anyone know the status of tvtk? If i could finally transition to py3k this year that would so great!

$ python
Python 3.4.3 (default, Mar 25 2015, 17:13:50) 
[GCC 4.9.2 20150304 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
>>>

Yeehaa!

Is there any progress on this? It looks like Python 3 support will be in the next minor release vtk 6.3.1.

See #250

pull request #250 is merged. Closing.

Just some quick postive feedback
I am using WinPython-64bit-3.4.4.1 with Mayavi 4.4.4 and it works fine. Thanks to all devs!

1.) VTK-7.0.0-cp34-cp34m-win_amd64.whl (http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk)
2.) pip install mayavi

@IVIUPPET,
does the workaround mentioned here
https://github.com/enthought/mayavi/pull/250#issuecomment-207719214
fix the problem?

@solarjoe

HAven't tried it, but I came back here to post my working results for others to follow, coming from very limited Python background, but experience with Matlab and C#.

I was getting an unsupported platform error that I commented about then promptly deleted, because I was trying to use your link (for Python 3.4) with Python 3.5
But I appreciate your reply. Comment with my setup to follow.

I installed/am using Anaconda for Python 3.5. Version 3.19.1. I used pip 8.1.1. (not that either of these matter much). 64 bit whenver possible.

I then downloaded:
1) VTK-7.0.0-cp35-cp35m-win_amd64.whl
From: http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk
python 3.5 64 bit version of VTK, a critical dependency for Mayavi (Thanks solarjoe!)

cd to downloadDir:
pip install VTK-7.0.0-cp35-cp35m-win_amd64.whl

Took me a bit to solve absence of vcvarsall.bat, which was the next issue I ran into once I found this thread to get VTK working with python 3.5. Tried adding stuff to different environment variables, installing old versions of Visual Studio. Didn't work. Ended up just getting Visual Studio Community 2015, which contains C++ compiler for Python 3.5. I know, it's overkill, but I spent hours doing the other stuff, so for me this was the path of least resistance (I'm an EE). During install, be sure to do "custom" and then under "programming languages", check anything that has to do with Python or C++.

2) pip install mayavi

It worked. After a day of googling I finally have 3D graphing that seem to match or exceed Matlab! my Matlab to Python transition is complete :D

As I was reading @IVIUPPET 's comments above, I remembered a very recent (April 11th) blog post from Microsoft named How to deal with the pain of "unable to find vcvarsall.bat". Although I am not certain, I thought that post might help.

After installing Mayavi in Anaconda python 3.5 following IVIUPPET's helpful instructions, I have hit a road block in actually running it.

When I import mayavi I get the following runtime error:
RuntimeError: Invalid Qt API 'pyqt5', valid values are: 'pyqt' or 'pyside'

Does mayavi not run under pyqt5 (which is the anaconda default)? If so, should I install pyqt4 and how do I nominate pyqt4 as the Qt API?

@dreme Unfortunately I have not yet implemented any kind of GUI or have a need for QT API in my programs, but based on skimming the documentation it seems there is still some work to be done to transition over to QT5. I would not be surprised if there is some incompatibility with mayavi, but your error doesn't seem to suggest that to me. Try without mayavi to isolate problem or contacting help related to QT, I would like to have a GUI eventually when I leave my code for my lab once I leave. If you found an incompatibility with mayavi and better yet a solution, please keep us informed!

Thanks for the response IVIUPPET.

I guess I'll first try installing Qt4. Hopefully it doesn't introduce nasty software clashes with Anaconda's Qt5

Also, when compiling VTK you can select if you want Qt4 or Qt5 widgets at compile time, I think.

Well, I tried compiling VTK from source (http://www.vtk.org/download/) using the instructions at http://www.vtk.org/Wiki/VTK/Building/Windows

However, I couldn't see any option to specify Qt5 widgets, and the compile process (using Microsoft Visual Studio Community 2015) failed anyway.

115>  Generating Code...
115>LINK : fatal error LNK1104: cannot open file '..\..\lib\Debug\vtkViewsCore-7.1.lib'
117>------ Build started: Project: ALL_BUILD, Configuration: Debug Win32 ------
117>  Building Custom Rule C:/MyProjects/VTK-src/CMakeLists.txt
117>  CMake does not need to re-run because C:\MyProjects\CMakeFiles\generate.stamp is up-to-date.
========== Build: 18 succeeded, 99 failed, 0 up-to-date, 0 skipped ==========

You would want to turn VTK_Group_Qt on, and then set VTK_QT_VERSION to 5 (it defaults to 4). I have been developing and packaging Tomviz, which uses Qt 5 with VTK/ParaView for quite some time now. It works pretty well, there are some bugs we need to get fixed. We are also building with Visual Studio 2015 on Windows and have been for a few months now.

Thanks @cryos!

I'm not a C++ programmer and have nearly zero experience in compiling programs, so if you don't mind I have a follow-up question.

When you say "turn VTK_Group_Qt on, and then set VTK_QT_VERSION to 5" what exactly do you mean? I see that the VTK_Group_Qt and VTK_QT_VERSION terms appear in quite a number of the CMakeLists.txt files, as well as in the vtkiOS.cmake, vtkAndroid.cmake, VTKConfig.cmake.in and vtkQt.cmake files.

Easiest way is to use cmake-gui, and point it at the build directory. You would want to click on VTK_Group_Qt to put a check in the box, click on Configure, and then the VTK_QT_VERSION would appear as a new variable. Alternatively a CMake command line such as the following could be used

cmake -DVTK_Group_Qt:BOOL=ON -DVTK_QT_VERSION:STRING=5 C:/path/to/vtk/src

These are all stored in the CMakeCache.txt in the build directory, but cmake-gui, cmake command line etc can be used to manipulate them. You would also need to turn on the Python wrapping in the same way. Hope that makes it clearer.

@dreme Are you sure you want to compile VTK to use Qt5? Mayavi/ETS only supports Qt4 (or WX): https://github.com/enthought/mayavi/issues/450 . Sorry if I am missing something...

Thanks for your patience @cryos. I didn't realise that a cmake-gui program is also provided when cmake is installed.

OK, so I have started cmake-gui, and indicated the source (C:/MyProjects/VTK-src) and build (C:/MyProjects/VTK-bin) folders.

However, when I click the Configure button I get a bunch of error messages (see attached).

I also don't see the VTK_Group_Qt term amongst the variables that are listed (see attached pic).

Would I need to add it with the Add Entry button?

It looks like it can't find the C or C++ compilers. They aren't there because it is failing really early on, you said you were using Visual Studio 2015, but the generator in the screenshot is saying Visual Studio 2017 (which is really new, and I would avoid for now). You probably need to delete the build tree, and start again but select Visual Studio 2015.

Hi @jonathanrocher, my ultimate aim is to run mayavi under anaconda python 3.5, much as @IVIUPPET has seemed to be able to do.

Following his instructions, I managed to install mayavi OK, but when I try to import it I get the following runtime error:
RuntimeError: Invalid Qt API 'pyqt5', valid values are: 'pyqt' or 'pyside'

So, I thought maybe the solution is to compile VTK so that it automatically uses Qt5 which is the default Qt version shipped with Anaconda.

But as you can see from the preceding comments, I am floundering in this task ...

Is there perhaps a way to get mayavi to use WX instead?

@dreme -- I tested the following instructions with conda for the pysph package (see here) :

$ conda install -c conda-forge matplotlib jupyter pyside
$ conda install -c menpo mayavi

If you have any issues please let us know.

Hi @prabhuramachandran, thanks for that advice, but it didn't seem to work for me.

When I entered: conda install -c conda-forge matplotlib jupyter pyside

I got the following error message:

Fetching package metadata ...........
Solving package specifications: ....

UnsatisfiableError: The following specifications were found to be in conflict:
  - matplotlib
  - python 3.5*
Use "conda info <package>" to see the dependencies for each package.

It doesn't seem to like python 3.5 so I created a new env for python 3.4, for which both of the conda install commands did actually work, i.e.:

$ conda install -c conda-forge matplotlib jupyter pyside
$ conda install -c menpo mayavi

However, when I try to import mayavi in a jupyter notebok, I still get the same old runtime error:
RuntimeError: Invalid Qt API 'pyqt5', valid values are: 'pyqt' or 'pyside'

Hey @dreme, I made the mayavi build on the menpo channel.

We've hit this same issue. Solution for us was just to force a downgrade in pyqt after the installation, so

> conda install -c menpo mayavi
> conda install pyqt=4.11.4

I found the final downgrade played fine with everything else, at least drawn from the standard anaconda channel and our own.

Actually, @scienceopen has a more elegant solution if you are happy making a fresh env here https://github.com/enthought/mayavi/issues/448#issuecomment-258065329 (turns out you can request a QT4 env from the get-go)

Yes using a QT4 env is the way to go. Then you can use QT5 for HiDPI non-Mayavi work.

Grateful for Python 3 support; hopefully it will take less than three years for QT5 support

Also, as I mentioned above (https://github.com/enthought/mayavi/issues/84#issuecomment-205720564) if you are on Windows and not bound to Anaconda, it is really easy to get Mayavi to work with WinPython.

Hey all, thx for your continuing help with this problem. Really appreciate it.

I tried both of the solutions offered by @jabooth (downgrading pyqt, and creating a new env with the python=3 pyqt=4 settings), but neither worked. I still get that same darn runtime error when I try to import mayavi:

C:\Users\dreme\Anaconda3\envs\py34\lib\site-packages\traitsui\qt4\__init__.py in <module>()
     16 # import pyface.qt before anything else is done so the sipapi
     17 # can be set correctly if needed
---> 18 import pyface.qt
     19 
     20 #----------------------------------------------------------------------------

C:\Users\dreme\Anaconda3\envs\py34\lib\site-packages\pyface\qt\__init__.py in <module>()
     42 elif qt_api != 'pyside':
     43     raise RuntimeError("Invalid Qt API %r, valid values are: 'pyqt' or 'pyside'"
---> 44                        % qt_api)

RuntimeError: Invalid Qt API 'pyqt5', valid values are: 'pyqt' or 'pyside'

[I don't know if this is significant, but I'm running my code in a Jupyter notebook]

I really like the Anaconda python distribution, but I'm getting to the point where I'm ready in desperation to switch to WinPython (not that I think it's necessarily bad) as @solarjoe suggested.

In that case @solarjoe, which installer did you use? WinPython 3.5.2.3Qt5 or WinPython 3.5.2.3 ?

@dreme I'm assuming after you did

conda create -n oldqt python=3 pyqt=4

that you activated that environment with (for Windows)

activate oldqt

or for Mac/Linux

source activate oldqt

Have you set the QT_API environmental variable to something? See https://github.com/enthought/pyface/blob/master/pyface/qt/__init__.py for the logic used.

@dreme I'm assuming after you did

conda create -n oldqt python=3 pyqt=4
that you activated that environment with (for Windows)

activate oldqt

Yep, that's what I did @scienceopen

Have you set the QT_API environmental variable to something? See https://github.com/enthought/pyface/blob/master/pyface/qt/__init__.py for the logic used.

Nope, this is all being done on a pretty fresh install of anaconda 3.5

But I think you may be right about it having something to do with the QT_API variable. Does the Jupyter notebook have a default setting for this? Can I perhaps set this to something that will be OK for mayavi?

You can always try changing the env variable. I doubt that Jupyter changes it but perhaps something else. You can inspect the current value with:

echo %QT_API%

and change it with

set QT_API= pyqt

on windows http://superuser.com/questions/79612/setting-and-getting-windows-environment-variables-from-the-command-prompt

@dreme, I am using an older version, WinPython-64bit-3.4.4.1 with Qt4 and Mayavi 4.4.4

1.) VTK-7.0.0-cp34-cp34m-win_amd64.whl (http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk)
2.) pip install mayavi

You can also try to install mayavi from http://www.lfd.uci.edu/~gohlke/pythonlibs/, search the page for Mayavi. Just pick the correct versions.
I am not sure if they work with the Qt5 version of WinPython. Maybe you have to install http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyqt4 then.

You can always try changing the env variable. I doubt that Jupyter changes it but perhaps something else. You can inspect the current value with:

echo %QT_API%
and change it with

set QT_API= pyqt

Well, this is just driving me nuts. I set the windows environmental variable for QT_API to pyqt as @jenshnielsen advised, and double checked that it had been set by running the SET command.

But fmd, I still get the same old runtime error complaining that the Qt API is pyqt5. I checked again in a cmd terminal, and sure enough the QT_API envirnoment variable was pyqt.

Then when I shut down Anaconda Navigator and checked again, the QT_API variable had just disappeared altogether!

BUT, when I restarted Anaconda navigator and opened the a Jupyter notebook under the the "oldqt" kernel, I get the same old runtime error complaining about pyqt5

When you ude set it only takes effect in the current cmd shell. No effect on programs outside the shell or in other shells. You can google how to set the env varible permanently from the windows. To test I would suggest set QT_API. In the same shell open python and run a simple mlab example i.e. From here http://docs.enthought.com/mayavi/mayavi/mlab.html#a-demo

You should also be able to set the env variable from python before importing mayavi. http://stackoverflow.com/questions/5971312/how-to-set-environment-variables-in-python

@dreme: we had a user with a similar issue see if their resolution helps: https://github.com/menpo/menpo3d/issues/16

The resulting instructions in our README might be easier to follow:
https://github.com/menpo/menpo3d#visualizing-3d-objects

Key part:

If you are running Windows and recieve this error, try:

```
set QT_API=pyqt
set ETS_TOOLKIT=qt4
````

@jenshnielsen I tried to automate this so that users (of Menpo at least) wouldn't have to worry about it, but I couldn't get it to work: https://github.com/menpo/menpo3d/issues/22

I'm going to raise a separate issue to try and discuss improving the situation here.

You should also be able to set the env variable from python before importing mayavi. http://stackoverflow.com/questions/5971312/how-to-set-environment-variables-in-python

Bingo! That did the trick. Thx @jenshnielsen =:-)

I much prefer this in-script approach compared to doing a system-wide OS setting of the QT_API environmental variable, as I may still want to do pyqt5 development in my root Anaconda environment.

So, in summary for any poor shmuck like me who is struggling to get mayavi running on anaconda python 3.5, here is my suggested approach distilled from the helpful advice in preceding comments:

1. Create a new env (oldqt) which is downgraded to pyqt4 by running the following command in the anaconda root env terminal:
   conda create -n oldqt python=3 pyqt=4

2. Switch to the newly created env:
   activate oldqt for Windows OS
   source activate oldqt for Mac/linux OS

3. install mayavi using the menpo package installer:
   conda install -c menpo mayavi

4. Use the oldqt env to run your python mayavi scripts, and include the snippet below at the top of these scripts to ensure that a valid QT API for mayavi is specified:

import os
os.environ["QT_API"] = "pyqt"

VTK-7.0.0-cp34-cp34m-win_amd64.whl is not avalable at http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk, from where I can get VTK-7.0.0-cp34-cp34m-win_amd64.whl.

It was available a week ago, I downloaded it. It is still on my office computer I could send you a link on
Monday (also for the matching Mayavi version).

Why don't you use the newer Mayavi and VTK?

Please send link for VTK-7.0.0-cp34-cp34m-win_amd64.whl. Thank you.

Link below should be valid about 30 days.

VTK-7.0.0-cp34-cp34m-win_amd64.whl

http://ge.tt/8QvtGkl2

so... approximately where is this at? I been keeping my eye on this since inception and it's another one of those projects that just doesn't seem to happen with.

@nevion How do you mean? This is a closed issue, and I have mayavi working on Python 3.6 on Windows as well as Linux.

@mabl It wan't really clear from the issue trail that this was the case, and things are often left in an unclear state in scientific software with something like qt support laggin. Between this and the anaconda distribution only supporting 2.7, I thought this was still the case. Any idea why anaconda still only supports 2.7?

conda install mayavi
Fetching package metadata .........
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:
  - mayavi -> apptools 4.2.0 -> python 2.7* -> openssl 1.0.1*
  - python 3.6*
Use "conda info <package>" to see the dependencies for each package.

Just installed Anaconda3 and tried to install mayavi. Got:

(C:\ProgramData\Anaconda3) C:\Users\nort.ARP>conda install mayavi
Fetching package metadata ...........
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

• mayavi -> apptools 4.2.0 -> python 2.7*
• python 3.6*
  Use "conda info " to see the dependencies for each package.

Did you look at/try the 4-point instructions above from @dreme ?

Specifically from @dreme

1. Create a new env (oldqt) which is downgraded to pyqt4 by running the following command in the anaconda root env terminal:

       conda create -n oldqt python=3.5 pyqt=4
   

2. Switch to the newly created env:

       activate oldqt #for Windows OS
       source activate #oldqt for Mac/linux OS
   

3. install mayavi using the menpo package installer:

       conda install -c menpo mayavi
   

4. Use the oldqt env to run your python mayavi scripts, and include the snippet below at the top of these scripts to ensure that a valid QT API for mayavi is specified:

       import os
       os.environ["QT_API"] = "pyqt"
   

@jonathanrocher No, fair enough. I was perhaps misreading the assertion that this was solved and just worked. Yes, it installed following the recipe.

I just tried @dreme 4-point instructions and it failed with python 3.6. I had to force 3.5:
conda create -n oldqt python=3.5 pyqt=4 and that worked.

yes that's correct, Mayavi 4.5 requires Python 3.5 due to QT4

So I can get mayavi working on Qt5 and Python 3.6 + VTK7.1 using e.g. conda install python=3.6.1 vtk>=7 -c clinicalgraphics, and using some recent commit numbers from pyface, traits, and traitsui. This seems to work on OSX, Windows, and Linux according to our CIs.

However, on OSX machines with HiDPI support, only 1/4 of the screen area is used:

Is this something that can be fixed at the Mayavi level?

It seems like it was maybe meant to be part of VTK7.1:

https://public.kitware.com/pipermail/vtk-developers/2016-November/034456.html

But before looking deeper I figured I'd ask in case others had ideas.

One option (sad but would fill the window) is to wait for VTK8 to come out, then use QVTKOpenGLWidget, which you can use to turn HiDPI off:

https://www.vtk.org/doc/nightly/html/classQVTKOpenGLWidget.html#details

Or maybe HiDPI will "just work" when using this widget.

I've tried the environment variables listed here but they do not fix it.

Hi @larsoner, are there some news now that vtk 8.1 is available?

Installing vtk 8.1.0 conda install -c conda-forge vtk, python is downgraded to 3.6.2 (and matplotlib to 2.1.0 between others).
I installed pyface 5.1.0, traits 4.6.0, traitsui 5.1.0, conda install -c conda-forge pyface traits traitsui
I have qt 5.6.2.

Installing mayavi 4.5.0 from conda-forge conda install -c conda-forge mayavi it complains about dependencies pyqt 4.*, python 3.6* and vtk 7.1.*.

Any suggestion to have it working with the current packages or, alternatively, exactly how did you made it work with Python3.6, Qt5 and VTK7.1? (Still there is the solution by @dreme for Python3.5+Qt4) Thank you.

We have been using this environment file for CI for about 6 months on Linux, OSX, and Windows with good success (other than the HiDPI stuff):

https://github.com/mne-tools/mne-python/blob/master/environment.yml

It has a bunch of stuff that you don't need. The critical components for Mayavi were getting VTK7 from clinicalgraphics because they had a 3.6 version, and pinning to some commits of pyface, traits, and traitsui from GitHub in lieu of using releases as the Qt5-supporting versions have not been released yet.

If VTK8 is available on conda-forge, it could probably safely be swapped in, but I haven't tried it yet. I'll give it a shot at some point and see if it fixes the HiDPI stuff. If you beat me to it, let me know how it goes!

@larsoner, @mpanighel -- this is an issue with VTK 7.x (the window being only quarter the size) and works fine with VTK 8.x. Stay tuned for an easier way to install VTK in a few days.

Just FYI, I have pushed wheels for VTK 8.1.0 on PyPI, see here: http://prabhuramachandran.blogspot.in/2018/01/vtk-810-wheels-for-all-platforms-on-pypi.html

Awesome @prabhuramachandran !

That is great @prabhuramachandran ! Thanks!

In these days I followed your latest fixes and tried to reproduce your setup and test mayavi on python3.6/pyqt5 on Linux, but with no luck (surely for my very basic python knowledge), most probably I just missing some obvious package. I thought to post more info in https://github.com/enthought/mayavi/issues/595 to be tidier.