Gunicorn: reload does not work for uvicorn.workers.UvicornWorker

Created on 26 May 2020  ·  28Comments  ·  Source: benoitc/gunicorn

this is my commandline gunicorn -k uvicorn.workers.UvicornWorker --reload-engine=inotify --workers=1 --log-level debug --access-logfile - --error-logfile - --disable-redirect-access-to-syslog --reload main:app

note that i have tested with both master branch of gunicorn as well as last stable release.

This is the code im using

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def read_root():
    # raise 1
    return {"Hello": "World"}


@app.get("/items/{item_id}")
async def read_item(item_id: int, q: str = None):

    return {"item_id 2": item_id, "q": q}

reload does not pickup when i make changes to the app. im using vscode as an editor

(
Source
picture sandys
👍9

Most helpful comment

The issue seems to be in the gunicorn -> workers/base.py -> init_process() -> def changed(fname) method. This function is called by the change watcher thread and it ends with sys.exit(0) which, unfortunately, results in only killing that thread (that's why changes are detected only once).

To workaround the issue, you can define a custom UvicornWorker with additional thread that sends a KILL signal to the current process if the worker's active flag is set to False (which is done by the changed(fname) function above).

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker


class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)


class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

To use that worker you need to specify it with the gunicorn's worker_class parameter, e.g.:

gunicorn -k some_package.RestartableUvicornWorker ...

picture jvasi on 24 Jun 2021
👍5

All 28 comments

You should just use uvicorn itself rather than adding gunicorn on top of it for local development. Something like this:

uvicorn main:app --reload

And then if you have watchgod installed, uvicorn should automatically pick that up to make the reloading process better. If you're on Docker, add --host 0.0.0.0

Andrew-Chen-Wang picture Andrew-Chen-Wang on 29 Jun 2020

This is my start command gunicorn my_pro.asgi:application -b unix:/run/day01/gunicorn.socket --reload -w 2 -t 1 -k uvicorn.workers.UvicornWorker, It can be reloaded normally, but there seems to be a 1-2 second delay， @sandys

systemime picture systemime on 14 Aug 2020

I've observed similar behavior as @sandys .

The workaround proposed by @Andrew-Chen-Wang is not always useful (not close to production setup; problems with Windows + docker).

@sandys , @systemime : Which python versions, and which OS were you using?

tobias-hd picture tobias-hd on 13 Oct 2020

@tobias-hd This is not my workaround; it is the recommended method by uvicorn's documentation: https://www.uvicorn.org/deployment/

Run gunicorn's CLI during deployment; run uvicorn's during development.

If you're using --reload and DEBUG log level, I'm assuming you're developing locally. If you're acting for production as I'm assuming tobias is saying, then you should use gunicorn and do not use the reload flag (use supervisord).

Additionally, in your IDE, you should do CTRL + S or save the file with some hotkeys. That's usually how flask, Django, or uvicorn works/checks for new file changes.

Andrew-Chen-Wang picture Andrew-Chen-Wang on 13 Oct 2020

This is my start command gunicorn my_pro.asgi:application -b unix:/run/day01/gunicorn.socket --reload -w 2 -t 1 -k uvicorn.workers.UvicornWorker, It can be reloaded normally, but there seems to be a 1-2 second delay， @sandys

Thats because the -t argument causes the idle workers to reload every second.

cochiseruhulessin picture cochiseruhulessin on 12 Nov 2020

Is there any chance this could be revisited? I'm having some problems with Uvicorn in which Migrations are loaded on app startup, triggering django.core.exceptions.SynchronousOnlyOperation. Running Gunicorn of course solves this as we are now using workers rather than threads, but I would really like to have an option to reload.

By any means, what's the problem with having your development server try to be as close as possible to your production server?

For instance, if you try to use Django Prometheus with Uvicorn without Gunicorn, you'll get a trace like this (also using the Django Cookie Cutter Library @Andrew-Chen-Wang :

django            | Process SpawnProcess-1:
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
django            |     self.run()
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
django            |     self._target(*self._args, **self._kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/subprocess.py", line 61, in subprocess_started
django            |     target(sockets=sockets)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 47, in run
django            |     loop.run_until_complete(self.serve(sockets=sockets))
django            |   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 54, in serve
django            |     config.load()
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 306, in load
django            |     self.loaded_app = import_from_string(self.app)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
django            |     module = importlib.import_module(module_str)
django            |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django            |     return _bootstrap._gcd_import(name[level:], package, level)
django            |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
django            |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
django            |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
django            |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
django            |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
django            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django            |   File "./config/asgi.py", line 23, in <module>
django            |     django.setup(set_prefix=False)
django            |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
django            |     apps.populate(settings.INSTALLED_APPS)
django            |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
django            |     app_config.ready()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/apps.py", line 23, in ready
django            |     ExportMigrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/migrations.py", line 52, in ExportMigrations
django            |     executor = MigrationExecutor(connections[alias])
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
django            |     self.loader = MigrationLoader(self.connection)
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 49, in __init__
django            |     self.build_graph()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 212, in build_graph
django            |     self.applied_migrations = recorder.applied_migrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 76, in applied_migrations
django            |     if self.has_table():
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 56, in has_table
django            |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
django            |   File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
django            |     raise SynchronousOnlyOperation(message)
django            | django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

I have no easy access to modify the library.

omarsumadi picture omarsumadi on 2 Feb 2021

I guess this has to do with asyncio event loop interfering with the rest. I will have a look on it.

benoitc picture benoitc on 2 Feb 2021

@benoitc Someone on Stackoverflow actually had the fix to my problem exactly, but it means I needed to add my Project Root to my path which... is something I didn't do previously and I'm looking into the repercussions of doing that.

Here's the link:
https://stackoverflow.com/questions/64306187/how-to-make-django-3-channels-and-uvicorn-work-together

omarsumadi picture omarsumadi on 2 Feb 2021

@omarsumadi To run migrations, use the manage.py command python manage.py makemigrations && python manage.py migrate. To run the server: uvicorn config.asgi:application --host 0.0.0.0 --reload. The --host 0.0.0.0 allows your mobile device (e.g. like a phone) to access your web server too. Additionally, in your IDE, you should do CTRL + S or save the file with some hotkeys. That's usually how flask, Django, or uvicorn works/checks for new file changes.

Really, just follow uvicorn's documentation. The comment Tobias said about Windows + Docker being an issue is not an issue if you just use uvicorn without docker in the first place...

By any means, what's the problem with having your development server try to be as close as possible to your production server?

It's just the settings. The differences? Default log levels, setting up HTTPS settings, django debug toolbar, DEBUG=False, etc.. Otherwise, yea your dev server shouldn't deviate too much from your production server (and that's also down to personal opinions).

Andrew-Chen-Wang picture Andrew-Chen-Wang on 3 Feb 2021

Hi Andrew
Thanks for the reply. I know you're trying to be helpful, but most of
working on production apps would like 1) docker and 2) gunicorn so that dev
is close to production.

All of us here working in large teams have faced the common issues of mac
(intel vs m1) vs windows vs linux.

Trust me when I say this - we would pay to get production consistency on
dev... versus choosing the convenient option. 🙏

On Wed, 3 Feb, 2021, 21:24 Andrew Chen Wang, notifications@github.com
wrote:

@omarsumadi https://github.com/omarsumadi To run migrations, use the
manage.py command python manage.py makemigrations && python manage.py
migrate. To run the server: uvicorn config.asgi:application --host
0.0.0.0 --reload. The --host 0.0.0.0 allows your mobile device (e.g. like
a phone) to access your web server too. Additionally, in your IDE, you
should do CTRL + S or save the file with some hotkeys. That's usually how
flask, Django, or uvicorn works/checks for new file changes.

Really, just follow uvicorn's documentation. The comment Tobias said about
Windows + Docker being an issue is not an issue if you just use uvicorn
without docker in the first place...

By any means, what's the problem with having your development server try
to be as close as possible to your production server?

It's just the settings. The differences? Default log levels, setting up
HTTPS settings, django debug toolbar, DEBUG=False, etc.. Otherwise, yea
your dev server shouldn't deviate too much from your production server (and
that's also down to personal opinions).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/benoitc/gunicorn/issues/2339#issuecomment-772613108,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYUY4YKMKAGBQTLCR6OTS5FWSJANCNFSM4NKBSMLQ
.

sandys picture sandys on 3 Feb 2021
👍3

Hey Sandy! I love Docker, but some people might not (addressing Tobias' early comment). The commands I just posted work the same way in Docker and with a virtualenv, regardless.

we would pay to get production consistency on dev.

ik I feel the pain as well... But yea Docker: super useful and needed, regardless if you're in a team or not.

...gunicorn so that dev is close to production.

I would still recommend that you use uvicorn itself when working in the Dev environment. It's close enough to gunicorn, and it'll be a little easier on your computer (at least for me) in terms of resource consumption (and noise reduction due to a phenomenal fan). But yea I agree: dev environment shouldn't deviate too much from prod, if at all.

Andrew-Chen-Wang picture Andrew-Chen-Wang on 3 Feb 2021
❤1

@Andrew-Chen-Wang Hey Andrew - As you said it might be a Docker Problem, whether or not it is, I did put in an Enhancement on the Django Cookie Cutter repo to fix Uvicorn spawning this error for the Docker version - run Uvicorn programmatically. Also using VSCode, so yes everything migrations and watching for file changes works perfectly. The problem is that Uvicorn is not working with this - not a problem with following documentation.

Still, it's sort of an interesting perspective because I did have to add the Project Root to path, run django.setup(), then run asgi.py from the Docker start scripts which is not what the Cookie Cutter setup does currently. Running Gunicorn with Uvicorn workers does solve this without any modifications, which makes me agree if Gunicorn had reloading work with Uvicorn workers, this might be a nice positive addition to making dev and prod as similar as possible. Perhaps this serves as an example of what happens when they aren't - things have to change.

Thanks,
Omar

omarsumadi picture omarsumadi on 3 Feb 2021

@omarsumadi Taking a look at that traceback again, I don't have django.setup() in my asgi, wsgi, or manage.py. Take a look at cookiecutter Django's files again. I'll also take a look at that issue you rose.

Andrew-Chen-Wang picture Andrew-Chen-Wang on 3 Feb 2021

@omarsumadi Taking a look at that traceback again, I don't have django.setup() in my asgi, wsgi, or manage.py. Take a look at cookiecutter Django's files again. I'll also take a look at that issue you rose.

I don't either - just undid all the changes I suggested and ran it again (I have some comments in the file, so it doesn't match up exactly with Cookie Cutter's ASGI.py file). I should have probably told you I'm running Django Channels as well (but I know you don't like it haha)

django            | Process SpawnProcess-1:
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
django            |     self.run()
django            |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
django            |     self._target(*self._args, **self._kwargs)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/subprocess.py", line 61, in subprocess_started
django            |     target(sockets=sockets)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 47, in run
django            |     loop.run_until_complete(self.serve(sockets=sockets))
django            |   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_impl/asyncio.py", line 54, in serve
django            |     config.load()
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 306, in load
django            |     self.loaded_app = import_from_string(self.app)
django            |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
django            |     module = importlib.import_module(module_str)
django            |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django            |     return _bootstrap._gcd_import(name[level:], package, level)
django            |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
django            |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
django            |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
django            |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
django            |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
django            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django            |   File "./config/asgi.py", line 29, in <module>
django            |     django_application = get_asgi_application()
django            |   File "/usr/local/lib/python3.8/site-packages/django/core/asgi.py", line 12, in get_asgi_application
django            |     django.setup(set_prefix=False)
django            |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
django            |     apps.populate(settings.INSTALLED_APPS)
django            |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
django            |     app_config.ready()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/apps.py", line 23, in ready
django            |     ExportMigrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django_prometheus/migrations.py", line 52, in ExportMigrations
django            |     executor = MigrationExecutor(connections[alias])
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
django            |     self.loader = MigrationLoader(self.connection)
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 49, in __init__
django            |     self.build_graph()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 212, in build_graph
django            |     self.applied_migrations = recorder.applied_migrations()
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 76, in applied_migrations
django            |     if self.has_table():
django            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 56, in has_table
django            |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
django            |   File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 24, in inner
django            |     raise SynchronousOnlyOperation(message)
django            | django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

My suggestion in the Issues you saw is a way to work around this issue I have here. Something is calling Django Setup from this package.

omarsumadi picture omarsumadi on 3 Feb 2021

@omarsumadi I see. I think it'll be wise to move this discussion to that issue then, since, in my belief, this issue should be closed as correct answers have already been given several times by multiple people, and your current problem is more Django related.

Andrew-Chen-Wang picture Andrew-Chen-Wang on 3 Feb 2021

@Andrew-Chen-Wang And, what is the correct answer?

gunicorn main:app --reload -w 2 -k uvicorn.workers.UvicornWorker

still does not work for me.
Simply using uvicorn is not the answer I'm looking for.

alanwilter picture alanwilter on 11 Mar 2021

@alanwilter Hm, I'm no guru on gunicorn, but try installing the inotify package. If that doesn't work, try using watchgod. It could just be gunicorn not picking up on sys notifs.

Edit: I think you need to add --reload-engine inotify

Andrew-Chen-Wang picture Andrew-Chen-Wang on 11 Mar 2021

I face the same issue here on gunicorn

On Thu, 11 Mar, 2021, 20:53 Andrew Chen Wang, @.*>
wrote:

@alanwilter https://github.com/alanwilter Hm, I'm no guru on gunicorn,
but try installing the inotify package. If that doesn't work, try using
watchgod. It could just be gunicorn not picking up on sys notifs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/benoitc/gunicorn/issues/2339#issuecomment-796815523,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAASYU3UM7OXNTOIFZGLBGTTDDG5XANCNFSM4NKBSMLQ
.

sandys picture sandys on 11 Mar 2021

something very strange is happening with the latest version of gunicorn and uvicornworker.

This works ONE TIME
gunicorn p:app -p 8080 --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker

gunicorn correctly picks up any file changes and does a hot-reload 

unicorn p:app -p 8080 --reload --reload-engine inotify   -k uvicorn.workers.UvicornWorker                                                (git)-[master]-
[2021-03-17 14:35:42 +0530] [298736] [INFO] Starting gunicorn 20.0.4
[2021-03-17 14:35:42 +0530] [298736] [INFO] Listening at: http://127.0.0.1:8000 (298736)
[2021-03-17 14:35:42 +0530] [298736] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 14:35:42 +0530] [298739] [INFO] Booting worker with pid: 298739
[2021-03-17 14:35:43 +0530] [298739] [INFO] Started server process [298739]
[2021-03-17 14:35:43 +0530] [298739] [INFO] Waiting for application startup.
[2021-03-17 14:35:43 +0530] [298739] [INFO] Application startup complete.
[2021-03-17 14:35:48 +0530] [298739] [INFO] Worker reloading: p.py.a42bb679bd860346e8c7d564fdc6cc56.tmp modified

But after this one-reload, it stops working. It doesnt pick up any subsequent file changes. Not sure why its happening

sandys picture sandys on 17 Mar 2021
👍2

I've seen it, but it actually is doing nothing.
gunicorn main:app --reload -k uvicorn.workers.UvicornWorker

However, if I add inotify:

gunicorn main:app --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker
[2021-03-17 10:43:49 +0100] [31332] [INFO] Starting gunicorn 20.0.4
[2021-03-17 10:43:49 +0100] [31332] [INFO] Listening at: http://127.0.0.1:8000 (31332)
[2021-03-17 10:43:49 +0100] [31332] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 10:43:49 +0100] [31333] [INFO] Booting worker with pid: 31333
[2021-03-17 10:43:49 +0100] [31333] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/uvicorn/workers.py", line 63, in init_process
    super(UvicornWorker, self).init_process()
  File "/Users/alan/Downloads/fastapi/venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 132, in init_process
    self.reloader = reloader_cls(extra_files=self.cfg.reload_extra_files,
TypeError: __init__() got an unexpected keyword argument 'extra_files'
[2021-03-17 10:43:49 +0100] [31333] [INFO] Worker exiting (pid: 31333)
[2021-03-17 10:43:49 +0100] [31332] [INFO] Shutting down: Master
[2021-03-17 10:43:49 +0100] [31332] [INFO] Reason: Worker failed to boot.

It doesn't work at all. Is there anything special to do about inotify? It seems a feature for Linux kernels, I'm running macOS Big Sur. Anyway I did pip install -U uvloop httptools inotify.

alanwilter picture alanwilter on 17 Mar 2021

So I tested on Linux (Ubuntu 18.04) as well, failed. My whole setup (you need Python 3.7 or higher):

mkdir fastapi
cd fastapi

create file main.py

from typing import Optional
from fastapi import FastAPI
from pydantic import BaseModel

class Item(BaseModel):
    name: str
    description: Optional[str] = None
    price: float
    tax: Optional[float] = None

app = FastAPI()

@app.post("/items/")
async def create_item(item: Item):
    return item

Now, set python venv:

python3 -m venv venv
source venv/bin/activate
pip install -U fastapi gunicorn pip wheel uvicorn uvloop httptools

Standard way work as expected when changing main.py:

uvicorn main:app --reload
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [53638] using statreload
INFO:     Started server process [53640]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
WARNING:  StatReload detected file change in 'main.py'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [53640]
INFO:     Started server process [53758]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Now, try with basic gunicorn, inducing a change that would cause main.py to failed (like a typo returns item):

gunicorn main:app --reload -k uvicorn.workers.UvicornWorker
[2021-03-17 11:43:37 +0000] [95542] [INFO] Starting gunicorn 20.0.4
[2021-03-17 11:43:37 +0000] [95542] [INFO] Listening at: http://127.0.0.1:8000 (95542)
[2021-03-17 11:43:37 +0000] [95542] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 11:43:37 +0000] [95544] [INFO] Booting worker with pid: 95544
[2021-03-17 11:43:37 +0000] [95544] [INFO] Started server process [95544]
[2021-03-17 11:43:37 +0000] [95544] [INFO] Waiting for application startup.
[2021-03-17 11:43:37 +0000] [95544] [INFO] Application startup complete.
[2021-03-17 11:44:02 +0000] [95544] [INFO] Worker reloading: /home/awilter/fastapi/main.py modified

So, it may catch a "change", but it's not reloaded.

Using, inotify:

gunicorn main:app --reload --reload-engine=inotify -k uvicorn.workers.UvicornWorker
[2021-03-17 11:49:58 +0000] [99660] [INFO] Starting gunicorn 20.0.4
[2021-03-17 11:49:58 +0000] [99660] [INFO] Listening at: http://127.0.0.1:8000 (99660)
[2021-03-17 11:49:58 +0000] [99660] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 11:49:58 +0000] [99662] [INFO] Booting worker with pid: 99662
[2021-03-17 11:49:58 +0000] [99662] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line
 583, in spawn_worker
    worker.init_process()
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/uvicorn/workers.py", line
63, in init_process
    super(UvicornWorker, self).init_process()
  File "/home/awilter/fastapi/venv/lib/python3.8/site-packages/gunicorn/workers/base.py",
 line 132, in init_process
    self.reloader = reloader_cls(extra_files=self.cfg.reload_extra_files,
TypeError: __init__() got an unexpected keyword argument 'extra_files'
[2021-03-17 11:49:58 +0000] [99662] [INFO] Worker exiting (pid: 99662)
[2021-03-17 11:49:58 +0000] [99660] [INFO] Shutting down: Master
[2021-03-17 11:49:58 +0000] [99660] [INFO] Reason: Worker failed to boot.

As for watchgod, I don't know how to use it, doing simply --reload-engine=watchgod does not work.

alanwilter picture alanwilter on 17 Mar 2021

I think you get got an unexpected keyword argument 'extra_files' if gunicorn cannot detect you have inotify installed, either because you are not running on linux or there were import errors from inotify package. https://github.com/benoitc/gunicorn/blob/master/gunicorn/reloader.py#L121

In my case I'm running on Mac so it looks like I can't use the inotify engine and my only option is to use the poll engine. I don't know enough about gunicorn + uvicorn to know if the reloading has ever worked - but to me it looks like we need to get the uvicorn server to exit via SIGINT or SIGTERM.

In order to get the poll engine to reload the server I have added a small worker_int hook in my gunicorn.conf.py file which signals to exit the uvicorn server via SIGINT.

import os
import signal


def worker_int(worker):
    os.kill(worker.pid, signal.SIGINT)

This has allowed code reloading to work but it feels pretty barbaric. I'm sure there's a better way to achieve this. I run gunicorn like this:
gunicorn main:app -w 4 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker --log-file=- --log-level DEBUG --reload

c-py picture c-py on 13 Apr 2021
🚀1

@c-py u seem to be on the right track. I'm roughly getting it to work on linux...however I'm having to save TWICE, not once to get this to work

sandys picture sandys on 17 Apr 2021
👍2

If you're already using supervisor inside a container then you can add a process like this to the end of your development config. You will need procps and inotify-tools installed for pkill and inotify-wait.

[program:inotifywait]
command=bash -c "inotifywait --excludei \"[^(\.py)]+$\" -e modify -r path-to-watch/ && pkill -SIGHUP gunicorn"
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

I like this method because it can be adapted to work with practically anything and you don't need to change anything else in production vs development.

kylemacfarlane picture kylemacfarlane on 8 May 2021

something very strange is happening with the latest version of gunicorn and uvicornworker.

This works ONE TIME
gunicorn p:app -p 8080 --reload --reload-engine inotify -k uvicorn.workers.UvicornWorker

gunicorn correctly picks up any file changes and does a hot-reload

unicorn p:app -p 8080 --reload --reload-engine inotify   -k uvicorn.workers.UvicornWorker                                                (git)-[master]-
[2021-03-17 14:35:42 +0530] [298736] [INFO] Starting gunicorn 20.0.4
[2021-03-17 14:35:42 +0530] [298736] [INFO] Listening at: http://127.0.0.1:8000 (298736)
[2021-03-17 14:35:42 +0530] [298736] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-03-17 14:35:42 +0530] [298739] [INFO] Booting worker with pid: 298739
[2021-03-17 14:35:43 +0530] [298739] [INFO] Started server process [298739]
[2021-03-17 14:35:43 +0530] [298739] [INFO] Waiting for application startup.
[2021-03-17 14:35:43 +0530] [298739] [INFO] Application startup complete.
[2021-03-17 14:35:48 +0530] [298739] [INFO] Worker reloading: p.py.a42bb679bd860346e8c7d564fdc6cc56.tmp modified

But after this one-reload, it stops working. It doesnt pick up any subsequent file changes. Not sure why its happening

Experiencing the same exact behaviour here on Mac, the reload works only once. I didn't specify --reload-engine though.

After adding the gunicorn.conf.py by @c-py it works well enough for now. Thanks.

liquiddandruff picture liquiddandruff on 1 Jun 2021
👍1

The issue seems to be in the gunicorn -> workers/base.py -> init_process() -> def changed(fname) method. This function is called by the change watcher thread and it ends with sys.exit(0) which, unfortunately, results in only killing that thread (that's why changes are detected only once).

To workaround the issue, you can define a custom UvicornWorker with additional thread that sends a KILL signal to the current process if the worker's active flag is set to False (which is done by the changed(fname) function above).

import os
import signal
import threading
import time
from typing import Any, List, Dict

from uvicorn.workers import UvicornWorker


class ReloaderThread(threading.Thread):

    def __init__(self, worker: UvicornWorker, sleep_interval: float = 1.0):
        super().__init__()
        self.setDaemon(True)
        self._worker = worker
        self._interval = sleep_interval

    def run(self) -> None:
        while True:
            if not self._worker.alive:
                os.kill(os.getpid(), signal.SIGINT)
            time.sleep(self._interval)


class RestartableUvicornWorker(UvicornWorker):

    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools"}

    def __init__(self, *args: List[Any], **kwargs: Dict[str, Any]):
        super().__init__(*args, **kwargs)
        self._reloader_thread = ReloaderThread(self)

    def run(self) -> None:
        if self.cfg.reload:
            self._reloader_thread.start()
        super().run()

To use that worker you need to specify it with the gunicorn's worker_class parameter, e.g.:

gunicorn -k some_package.RestartableUvicornWorker ...

jvasi picture jvasi on 24 Jun 2021
👍5

Thanks @jvasi that worked great.

I get one INFO notification whenever gunicorn goes down this way:

[INFO] Error while closing socket [Errno 9] Bad file descriptor

I believe this is coming from this line in gunicorn: https://github.com/benoitc/gunicorn/blob/1299ea9e967a61ae2edebe191082fd169b864c64/gunicorn/sock.py#L69

rotten picture rotten on 9 Jul 2021

This seems related to https://github.com/benoitc/gunicorn/issues/2604

fvalverd picture fvalverd on 10 Jul 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

haolujun picture haolujun  ·  3Comments
leonardbinet picture leonardbinet  ·  4Comments
davidfstr picture davidfstr  ·  3Comments
joekohlsdorf picture joekohlsdorf  ·  4Comments
twosigmajab picture twosigmajab  ·  4Comments