Mopidy: Port Mopidy to Python 3

Hey, I'd like to help with Python 3 migration, anything in particular that's under way? don't want to step on anyone's toes

I've started a branch at jodal/feature/py3-compat where I've:

• updated the Python version check that stops you from starting Mopidy under Python 3
• updated the tox.ini setup
• starting fixing Py3 issues module-by-module, running the tests under both Python 2 and Python 3

I haven't touched this since November, except that I rebased the branch on top of Mopidy 2.0.0 a couple of days ago. The current status is that I've fixed around 1000 tests, with about 240 remaining.

The plan ahead is to:

1. Fix the rest of the issues module-by-module.
2. Take the experience from fixing all the issues and try to land as much of it as possible in Mopidy 2.x. My current module-by-module approach changes things (e.g. it starts to threat file paths as Unicode instead of bytes) that I don't want to land in Mopidy until the 3.0 release, where we're allowed to break things. The can't-land-until-3.0 diff should be as small as possible.
3. Use Mopidy with Python 3, finding all the issues not covered by tests.

ok, I will fork off of that branch and try to fix whatever tests I can

Hello guys, any news on this issue?

3½ years left for this 😉

Are there any updates on this?

Porting to Python 3 is my personal highest priority task when working on Mopidy, problem is I have been quite busy with work for the last year.

My current plan, in approximate order, is to:

• [x] Fix the Mopidy-MPRIS test suite, so that it can survive the upcoming changes. This is a bit boring and my doorstep mile.
• [x] Remove all the deprecated stuff from Mopidy and make a major release. See this milestone for the detailed issues. This I look forward to.
• [x] Ensure that all extensions in the Mopidy org still works after the removal, and make necessary changes and releases.
• [x] Port Mopidy to Python 2 + 3 and make another major release.
• [ ] Port all extensions in the Mopidy org to Python 3 only and release.
• [ ] Port/help port all other extensions to Python 3 only and urge for release.
• [ ] Remove the Python 2 support from Mopidy itself.

As you can see, there's a couple of tasks that should be done first before starting the porting itself. That said, the largest body of work is already complete: Rewriting pyspotify to workon Python 3 (two years of my spare time was spent here) and porting Mopidy to GStreamer 1.x (one paternity leave of spare time spent here).

I'll get there, but it takes time. If people want to help out, I think the most helpful would be helping port extensions to Python 3 as soon as Mopidy supports 2+3.

Thanks for the update, @jodal.

I Think that is a good idea move to Python3.

I am new in mopidy and have never contributed to it, do you think that I
can help you with something?

El mié., 22 de mar. de 2017 a la(s) 07:08, Frederick Gnodtke <
[email protected]> escribió:

Thanks for the update, @jodal https://github.com/jodal.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mopidy/mopidy/issues/779#issuecomment-288352544, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHQMO8UqLNKzfVQZAZJ3Pnjx2N7MsHg4ks5roPMEgaJpZM4CMq8p
.

Any help with the project that frees some of the maintenance burden helps with the big hairy goals like Python 3, as it frees time from the maintainers to either work on the larger issues themselves or to do proper code review on larger contributes, which can be quite exhausting and time consuming. Two prime examples are helping other users with debugging their problems, and to reproduce possible bugs, minimize the steps needed to reproduce, and document the results in the issues.

@jodal Where we can see your work with python3?

regards

@jodal the py3-compat branch you mentioned looks a bit... abandoned. Is there a new one somewhere?
I'm considering writing an extension and I'd prefer digging into some py3 bugs in unstable mopidy branch rather than remembering nasty py2 workarounds.

What's the status on this? is there a version of mopidy that can be used with python3?

Dependency pylast will be dropping support for legacy Python 2 soon: https://github.com/pylast/pylast/issues/265

Hi,
Currently I maintain pylast in Debian. I intend to upload the latest version ([1]) which no longer supports Python 2.7
I plan to publish this version at the end of February, if mopidy still uses Python 2 will not be included
in the next Debian release which could be a shame since mopidy is an amazing piece of software :smiley:

Is there any official branch where this is going on? I am wiling to help this to happen

Regards

@jodal Are the extensions the blocker here? If so, I would suggest opening an issue against each extension and referencing them here to keep track. If you can give some instructions in the issue on what changes are likely to be required, then people how aren't that familiar with Python (like me) could hopefully still help.

PS I'm happy to help with the extentions that I use (GMusic, Iris, Podcast, Scrobbler, Spotify), I just don't really know what to do at the moment.

Hey folks, may someone please clarify the status of adaptation to python3? Do we have any estimates when the first python3 version will be released? Thanks.

I'm still committed to porting Mopidy to Python 3. I won't let Mopidy die with Python 2.

History

Most of the work of moving us to Python 3 was done years ago in assuring that all our dependencies works on Python 3:

• Pykka has been Python 3 compatible since it was extracted from Mopidy in 2011.
• pyspotify 2.x was a full rewrite to get Python 3 support and complete libspotify bindings. This alone took almost two years from start to finish, work that is now mostly void thanks to Spotify breaking large parts of libspotify.
• The port of Mopidy from GStreamer 0.10/PyGObject to GStreamer 1.x/PyGI, released in Mopidy 2.0 three years ago, was the primary project of my second paternity leave.

As you can probably read between the lines from the long history of this issue and the state of the issue tracker, this project and open source in general, hasn't been my top priority since I moved from working in a large enterprise to a startup three years ago.

However, I'm slowly starting to pick up my open source projects again. I'm trying to focus on one project at the time and ship it before taking on the next. So far Mopidy.js and Mopidy-MPRIS has received some tender love and care. Currently, I'm working on a new release of Pykka, which among other small improvements will reduce the startup time off many Mopidy setups by a huge factor.

Next up

Once Pykka 2 has shipped, my primary goal is moving Mopidy to Python 3.

I won't be giving any time estimate for when that work will be complete, as that would simply be setting myself up for failure. It all depends a lot on external factors and motivation.

As for helping out, I don't see that there's much new contributors can do to significantly help the effort directly. I'm still not back to working on open source in a predictable and regular cadence, so PR reviews would quickly become demotivating for both parties. As previously mentioned in this issue, any help to reduce the general maintenance burden always helps indirectly.

As for other core devs, I'm sure they would be able to help out, but we all have our priorities and projects. I won't wait for our time and motivation to all align up. If they join in once I get properly started on Mopidy 3, that would be lovely, but I have no expectations.

Putting all my rambling about life and everything aside, this is neither a gigantic task nor a task I'm unfamiliar with. It's mostly a matter of completing the other ongoing stuff so that this can become the primary focus for a while.

Mopidy 3

Since I wrote the last plan here in March 2017, Python 3 adoption has gotten so far that I no longer see the point in taking gradual steps from Python 2 only via 2+3 to Python 3 only. Mopidy 2.x has served us quite well for three years, and will be a good and stable place to leave those stuck on Python 2 for whatever reason.

Thus, revising the plan from March 2017, I imagine a process like this:

• [x] Remove all the deprecated stuff from Mopidy. See the v3.0 milestone for the detailed issues.
• [x] Port Mopidy to Python 3.
• [x] Make a pre-release to PyPI, e.g. 3.0.0-rc1. This won't affect those running pip install mopidy, but makes it possible to run pip install --pre mopidy to install the pre-release for testing of ported extensions.
• [x] Port all extensions in the Mopidy org to Python 3, and optionally make pre-releases to PyPI.
• [ ] Port/help port other extensions to Python 3. Help wanted once we get here.
• [ ] Make a coordinated major release of Mopidy core and all extensions in the Mopidy org.

Since there has been five months, I guess it is time to update this issue. The Pykka release I spoke of last time was shipped as Pykka 2.0 six weeks ago. Since then, I've started working on Mopidy again:

• Bugfix release 2.2.3 is out, so we have currently nothing unreleased in the release-2.2 branch.
• In the develop branch which will become Mopidy 3.0 most of the deprecated stuff has been removed.

Next up for me is probably:

• [x] Simplifying logging (#1452)
• [x] Decide and implement how to handle file system paths when moving from byte-oriented Python 2 APIs to text-oriented Python 3 file system APIs. (No issue tracking this yet.)
• [x] Start making the test suite pass on Python 3.

Is there anything I could do to help with making either mopidy core, or the mopidy-soundcloud plugin work with python3?

Thanks for your work so far @jodal!

I'm wondering, how are we supposed to install the docs when using the Python 3 version of Sphinx to build them (asking for packaging purposes)?

I used to do make -C docs SPHINXBUILD=sphinx-build-2 man but changing that for sphinx-build-3 just errors out:

make: Entering directory '/home/builder/aports/community/mopidy/src/Mopidy-3.0.0a1/docs'
sphinx-build-3 -b man -d _build/doctrees   . _build/man
Running Sphinx v1.8.4

Configuration error:
The configuration file (or one of the modules it imports) called sys.exit()

This happens on both 2.2.3 and 3.0.0a1. Or is this part just not ported yet?

Given that v3 will break backwards compatibility anyway, and python2 will be end-of-lifed in january, is there a reason to keep python2 compatibility in v3?

@tmccombs said:

Given that v3 will break backwards compatibility anyway, and python2 will be end-of-lifed in january, is there a reason to keep python2 compatibility in v3?

No, we're not planning on keeping Python 2 compatability in Mopidy 3.

Quoting myself from February:

Since I wrote the last plan here in March 2017, Python 3 adoption has gotten so far that I no longer see the point in taking gradual steps from Python 2 only via 2+3 to Python 3 only. Mopidy 2.x has served us quite well for three years, and will be a good and stable place to leave those stuck on Python 2 for whatever reason.

@PureTryOut said:

This happens on both 2.2.3 and 3.0.0a1. Or is this part just not ported yet?

None of Mopidy has been ported to Python 3 yet. We've just been doing other simplifications and removed deprecated stuff to make the transition easier. Once we've ported, we'll build the docs with Sphinx running on Python 3.

Could I advise making sure the next release (2.4.0?) is compatible with Python 3? Most distributions are working on getting rid of their Python 2 packages. For my case, Alpine Linux is dropping Python 2 by the next release (3.11, end of January next year) which means Mopidy will be dropped from the repositories if it's not Python 3 compatible by then.

Mopidy 3.0 will be compatible with Python 3. The plan is to have it out before the end of the year.

Hi @jodal, I might have some spare cycles in the coming weeks to help with some Python 3 functionality. Either in mopidy directly or in the extensions. Do you have any particular issues to tackle?

Hi @zubieta!

We've recently merged a test setup which successfully runs ~10% of the test suite on Python 3. There are a couple of PRs soon to be merged that increases that to ~20%. Please check what PRs are already open, so you don't duplicate any work, and see the description of #1809 for a step-by-step guide on how to port a few more modules and their tests.

Extensions will follow as soon as Mopidy core is running on Python 3.

I think it's time for a status update on Mopidy and Python 3...

Mopidy 3.0.0a2 runs on Python 3 :tada:

Every single one of the 2016 tests in Mopidy's test suite now runs on both Python 2.7 and Python 3.7. All of this work is merged in the develop branch. A huge thanks to @kingosticks for helping out with the porting effort!

I've just released the develop branch as-is to PyPI as Mopidy 3.0.0a2. It can be installed with:

python3.7 -m pip install --pre mopidy

This release has not been subject to any extensive testing, other than having a passing test suite, answering a few requests across MPD and HTTP, and playing a handful of MP3 files.

Road ahead :arrow_right:

Mopidy 3.0.0a2 will probably be the only Mopidy release ever to support both Python 2 and 3. We'll promptly start removing Python 2.7 support and make Mopidy a cleaner and more modern Python code base.

The plan towards a final release is approximately as follows:

• [x] Remove Python 2.7 support from Mopidy and clean up after the porting effort.
• [x] Make sure Python 3.8 is supported.
• [x] Format the source code with Black.
• [x] Make another alpha release.
• [ ] Fix the issues in the v3.0 milestone.
• [ ] Make a beta release.
• [ ] Fix any bugs discovered through use and porting of extensions.
• [ ] Make a release candidate.
• [ ] Once a large enough set of extensions have been ported...
• [ ] Release Mopidy 3.0 final.

Help needed :heart_eyes:

In parallel with the march towards 3.0 final, we need help with the rest of the ecosystem of 122 packages on PyPI found by searching for "mopidy".

For each extension you care about:

• [ ] Port to Python >= 3.7. Drop Python 2.7 support.
• [ ] Consider including project setup modernizations from the extension cookiecutter. I'll update the cookiecutter shortly, once I get around to porting some extensions myself.
• [ ] Make pre-releases of the ports to PyPI.
• [ ] Once Mopidy 3.0 final is released, make final releases to PyPI.

76 commits, 204 files changed, 9832 insertions(+), 9612 deletions(-) later, we have another pre-release: Mopidy 3.0.0a3 is now on PyPI. It can be installed with:

python3 -m pip install --pre mopidy

New since 3.0.0a2:

• Python 2.7 support is gone, including lots of legacy/compatibility code:
  
  
  
  • The mopidy.compat module is gone.
  
  
  • # encoding: utf-8 comments are gone.
  
  
  • from __future__ ... imports are gone.
  
  
  • All subclassing of object are gone.
  
  
  • .encode() and .decode() no longer include an explicit "utf-8" argument.
  
  
• Source code is formatted with black. :black_heart:
• isort are now configured, and can be used to clean up imports when needed.
• mock is replaced with unittest.mock.
• unittest assertion methods are replaced with pytest assert statements, which means better readability and better error messages.
• Most instances of % and .format() are replaced with f-strings.
• All linters run on Python 3.
• Docs are built on Python 3.
• Tests in CI runs on Python 3.8 in addition to Python 3.7.
• setup.py has been minimized and replaced by a declarative setup.cfg.
• Dependencies in tox.ini, dev-requirements.txt, and docs/requirements.txt are all replaced by "extras" in setup.cfg. This means that dev dependencies are now installed with python3 -m pip install -e ".[dev]".

The road ahead still looks like I drafted in the previous comment.

With regards to Mopidy core, I think we're done here.

The rest of the way to Mopidy 3.0 final is tracked in the v3.0 milestone:
https://github.com/mopidy/mopidy/milestone/55

The porting of key extensions to Python 3 is tracked on this project board:
https://github.com/orgs/mopidy/projects/2

If you test out Mopidy on Python 3 in the near future, please open issues for any problems you run into!