Mycroft-core: Python 3.9+ issues
@j1nx reported that Exceptions were raised during Padatious training on python 3.9 similar to
Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.9/site-packages/padatious/training_manager.py", line 98, in train
pool = mp.Pool()
File "/usr/lib/python3.9/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/usr/lib/python3.9/multiprocessing/pool.py", line 191, in __init__
self._setup_queues()
File "/usr/lib/python3.9/multiprocessing/pool.py", line 343, in _setup_queues
self._inqueue = self._ctx.SimpleQueue()
File "/usr/lib/python3.9/multiprocessing/context.py", line 113, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/usr/lib/python3.9/multiprocessing/queues.py", line 342, in __init__
self._rlock = ctx.Lock()
File "/usr/lib/python3.9/multiprocessing/context.py", line 67, in Lock
from .synchronize import Lock
ImportError: cannot import name 'Lock' from partially initialized module 'multiprocessing.synchronize' (most likely due to a circular import) (/usr/lib/python3.9/multip
rocessing/synchronize.py)
The issue seems to be related to a change in how modules/packages are imported from python 3.9 and onwards See bug report.
A quick and dirty workaround for testing is available in here. Basically it preloads the modules that may be incomplete when trying to create the Pool object.
It's a bit hard to tell exactly what is happening but one possible culprit is msm, which also use a multiprocessing Pool (threadpool but it's the same base class)...unless training is somehow launched in separate threads as well as processes internally in padatious but that sounds farfetched.
Not sure what the most correct action is a couple of things that can be tested:
- Lock around msm.apply calls and padatious training
- Move the execution of training to the same thread as the skill download (SkillManager thread)
To Reproduce
Steps to reproduce the behavior:
- Install mycroft with a python3.9 environment
- Startup mycroft
- watch skills log
- Check for ImportError in multiprocessing
Environment (please complete the following information):
- Ubuntu 18.04
- Ubuntu 20.04
- Open Voice OS
forslund
All 10 comments
I hope I soon have a separate OVOS image with Python 3.9 again for testing this. Will see if I can make a clean OVOS-Python3.9 build on another (temperarily) build machine. Report back as soon as possible.
j1nx
on 15 Jan 2021
Also appears on Arch Linux
adocampo
on 19 Jan 2021
Tried to install mycroft with Python 3.9:
When I start mycroft cli I got some errors:
Starting cli
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/pi/mycroft-core/mycroft/client/text/__main__.py", line 19, in <module>
import curses
File "/usr/local/lib/python3.9/curses/__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
With your fixed process_utils.py I got those errors:
Starting cli
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 188, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/local/lib/python3.9/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/home/pi/mycroft-core/mycroft/__init__.py", line 17, in <module>
from mycroft.api import Api
File "/home/pi/mycroft-core/mycroft/api/__init__.py", line 22, in <module>
from mycroft.configuration import Configuration
File "/home/pi/mycroft-core/mycroft/configuration/__init__.py", line 15, in <module>
from .config import Configuration, LocalConf, RemoteConf
File "/home/pi/mycroft-core/mycroft/configuration/config.py", line 23, in <module>
from mycroft.util.json_helper import load_commented_json, merge_dict
File "/home/pi/mycroft-core/mycroft/util/__init__.py", line 32, in <module>
from .process_utils import (reset_sigint_handler, create_daemon,
File "/home/pi/mycroft-core/mycroft/util/process_utils.py", line 7
<!DOCTYPE html>
Is there a workaround or should I relink Python to 3.7 and reinstall mycroft?
simical
on 27 Jan 2021
It should work with python 3.9. I have it working, in fact, with python 3.9. There is the multiprocessing error, and I need to kill skills manually after stopping services, because they didn't stop properly, but is definitely working.
Do you have installed curses-menu? it seems it complains about it
adocampo
on 27 Jan 2021
It should work with python 3.9. I have it working, in fact, with python 3.9. There is the multiprocessing error, and I need to kill skills manually after stopping services, because they didn't stop properly, but is definitely working.
Would like to +1 this behaviour. Indeed, with Python 3.9 stopping the mycroft-skills service takes a long time (till time out).
j1nx
on 27 Jan 2021
Has anyone tried the workaround branch I mentioned at the top? would be interesting to see if it makes things work better, or it's just for me(TM) :)
forslund
on 27 Jan 2021
Not by myself as of yet. When the OVOS release is out of the door, I will get back to it.
j1nx
on 27 Jan 2021
Not me either, can I apply just this or need to do some git magic (if you can guide me to easily test it without the need to reinstall everything, would be great, I usually just clone, push and pull, so other git stuff are still a mystery for me)
adocampo
on 27 Jan 2021
Yeah that's what's the total change.
you could just copy-paste it. Or the following should work (written from memory without testing so it may be wrong)
git checkout -b py39-test
git pull https://github.com/forslund/mycroft-core.git bugfix/mp-workaround
Then to go back to dev you can just do
git checkout dev
git remote add forslund https://github.com/forslund/mycroft-core.git
git pull forslund
git checkout forslund/bugfix/mp-workaround
Found that easier for in case you test, give feedback, forslund changes some stuff again. Then all you need to do is
git pull
j1nx
on 28 Jan 2021
Related issues
tiagochiavericosta
·
6Comments
Zacki84
·
10Comments
krisgesling
·
3Comments
el-tocino
·
4Comments
krisgesling
·
5Comments
Most helpful comment
I hope I soon have a separate OVOS image with Python 3.9 again for testing this. Will see if I can make a clean OVOS-Python3.9 build on another (temperarily) build machine. Report back as soon as possible.