gunicorn 20.0.0: --paste not detecting arguments under [server:main]

My apologies. The changelog entry only says "Simplify Paste Deployment documentation", and I should have helped to prepare a better news entry here.

The PR for this is here: https://github.com/benoitc/gunicorn/pull/1957

Previously, we had deprecated use = egg:gunicorn#main but that is no longer deprecated. The change is intended to clarify the role of Gunicorn in a Paste Deploy-compatible environment.

There are two options for using Gunicorn with this style of .ini file.

The first option is to use the gunicorn CLI. When you do this, you must use Gunicorn's own CLI flags or Gunicorn's own configuration module (default gunicorn.conf.py) to configure server arguments. Gunicorn bind's the socket, manages reloading, writes PID files and so on. Gunicorn can use an app section in your .ini to configure the application callable.

The second option is to use a Paste Script runner such as pserve. In this case, this script runner manages reloading, writes PID files and so on. Most other options should still work, but to use the server block of your .ini file you need to invoke a script runner that is Paste compatible. Gunicorn is no longer such a script runner.

Let me know if I can add more clarity.

In your case, you should be able to continue as before, but use pserve raither than gunicorn to start your application. All the server configuration for Gunicorn can then be in your server block, as you seem to have done already.

The previous behavior was potentially confusing because it allowed specifying options on the command line that would conflict with the file. We also had requests for adding the ability to specify config vars for interpolation in the .ini file and specifying different server blocks (in addition to different app blocks). As a result, the decision was made to deprecate using Gunicorn as a Paste _server_ runner rather than try to add support for all these features. The Gunicorn CLI now supports reading a Paste Deploy .ini file to construct an application, but using server blocks is left to dedicated tooling in that ecosystem.

In your case, you should be able to continue as before, but use pserve raither than gunicorn to start your application.

Thanks for the fast response @tilgovi!

Following your recommendation:

$ pserve development.ini
# ...
Starting server in PID 40148.
[2019-11-12 14:26:30 -0800] [40148] [INFO] Starting gunicorn 20.0.0
[2019-11-12 14:26:30 -0800] [40148] [INFO] Listening at: https://0.0.0.0:9090 (40148)
[2019-11-12 14:26:30 -0800] [40148] [INFO] Using worker: gevent
[2019-11-12 14:26:30 -0800] [40263] [INFO] Booting worker with pid: 40263

Which is great, since part of the internal PR I opened up involved having to change the command pserve to gunicorn, which I was a bit wary of as I'm not the original developer of our internal API server.

That resolves my problems, feel free to close this issue =)

Thanks,
Correy

One final note, and then I think I've added all the detail I can recall. It was potentially confusing that invoking gunicorn --paste production.ini would use Gunicorn as the server _even if the server block specified something other than egg:gunicorn#main_!

Since Gunicorn is primarily a server and process manager, it doesn't make sense for Gunicorn to be a general CLI for invoking arbitrary Paste-compatible servers. Instead, Gunicorn is a server that supports Paste Deploy applications and it _is a_ Paste-compatible server. It is _not_ a Paste Script runner CLI, though!

I opened an issue on the Pyramid cookbook for this: https://github.com/Pylons/pyramid_cookbook/issues/222

I thought I had documented this thoroughly in Gunicorn itself, but I couldn't find the reference at first. It's here: http://docs.gunicorn.org/en/stable/run.html#paste-deployment

@tilgovi just a heads up this was a breaking change for my team as well. Perhaps worth moving to the breaking change portion of the changelog?

I'll re-open the issue and self-assign. I'll close it when I update the changelog to be more legible here.

Again, my apologies for not calling it out more clearly in the changelog initially.

@tilgovi bump

Please let me know if this should be opened as a separate issue.

This may be an isolated issue with our codebase, but after a bit more testing, my team has noticed that for our API server, gunicorn 20.0.0 breaks the function pyramid_ldap3.get_ldap_connector.

gunicorn 20.0.0:

On Startup:

$ pip list | grep gunicorn
gunicorn             20.0.0
$ pserve bioapps/development.ini
[2019-11-20 15:55:30 -0800] [9902] [INFO] Starting gunicorn 20.0.0
[2019-11-20 15:55:30 -0800] [9902] [INFO] Listening at: https://0.0.0.0:10999 (9902)
[2019-11-20 15:55:30 -0800] [9902] [INFO] Using worker: gevent
[2019-11-20 15:55:30 -0800] [10034] [INFO] Booting worker with pid: 10034
/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/gunicorn/workers/ggevent.py:53: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/urllib3/util/__init__.py)'].
  monkey.patch_all()

After Attempting to Authenticate:

[2019-11-20 15:57:54,189] INFO  [access:342][DummyThread-1] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:10999/session HTTP/1.1" {'username': 'colim', 'password': ''}
[2019-11-20 15:57:57,276] ERROR [exc_logger:114][DummyThread-1] 'https://bioappsdev02.bcgsc.ca:10999/session'
Traceback (most recent call last):
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/tweens.py", line 39, in excview_tween
    response = handler(request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/router.py", line 156, in handle_request
    view_name
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/view.py", line 642, in _call_view
    response = view_callable(context, request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/config/views.py", line 181, in __call__
    return view(context, request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/viewderivers.py", line 390, in attr_view
    return view(context, request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/viewderivers.py", line 368, in predicate_wrapper
    return view(context, request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/viewderivers.py", line 439, in rendered_view
    result = view(context, request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid/viewderivers.py", line 148, in _requestonly_view
    response = view(request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/cornice/service.py", line 493, in wrapper
    response = view_(request)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/bioapps/api/endpoints/session.py", line 139, in session_post
    username, request.validated['password'], request,
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/bioapps/api/endpoints/session.py", line 27, in get_ldap_groups
    auth = connector.authenticate(username, password)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid_ldap3/__init__.py", line 208, in authenticate
    password=escape_for_search(password))
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid_ldap3/__init__.py", line 82, in execute
    with manager.connection() as conn:
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/pyramid_ldap3/__init__.py", line 165, in connection
    auto_bind=True, lazy=False, read_only=True)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/ldap3/core/connection.py", line 326, in __init__
    self.do_auto_bind()
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/ldap3/core/connection.py", line 343, in do_auto_bind
    self.bind(read_server_info=True)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/ldap3/core/connection.py", line 585, in bind
    _, result = self.get_response(response)
  File "/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/ldap3/strategy/base.py", line 370, in get_response
    raise LDAPResponseTimeoutError('no response from server')
ldap3.core.exceptions.LDAPResponseTimeoutError: no response from server
[2019-11-20 15:57:57,298] INFO  [access:362][DummyThread-1] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:10999/session HTTP/1.1" 500 206

gunicorn 19.9.0

On Startup:

$ pip install gunicorn==19.9.0
Collecting gunicorn==19.9.0
  Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Installing collected packages: gunicorn
  Found existing installation: gunicorn 20.0.0
    Uninstalling gunicorn-20.0.0:
      Successfully uninstalled gunicorn-20.0.0
Successfully installed gunicorn-19.9.0
$ pip list | grep unicorn
gunicorn             19.9.0
$ gunicorn --paste bioapps/development.ini
[2019-11-20 16:03:45 -0800] [12015] [INFO] Starting gunicorn 19.9.0
[2019-11-20 16:03:45 -0800] [12015] [INFO] Listening at: https://0.0.0.0:10999 (12015)
[2019-11-20 16:03:45 -0800] [12015] [INFO] Using worker: gevent
[2019-11-20 16:03:45 -0800] [12018] [INFO] Booting worker with pid: 12018

After Attempting to Authenticate:

[2019-11-20 16:04:39,292] INFO  [access:342][DummyThread-1] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:10999/session HTTP/1.1" {'username': 'colim', 'password': ''}
[2019-11-20 16:04:39,527] INFO  [access:362][DummyThread-1] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:10999/session HTTP/1.1" 200 639

No changes were made to development.ini between gunicorn 20.0.0 and gunicorn 19.9.0.

Interestingly, we can stop the error with gunicorn 20.0.0 if we start the server with the following command:

$ pip list | grep unicorn
gunicorn             20.0.0
$ gunicorn --paste bioapps/development.ini -b 0.0.0.0:8999 --workers 1 --certfile /etc/ssl/certs/current/webserver.cer  --keyfile /etc/ssl/certs/current/private.key.u
[2019-11-20 16:14:27 -0800] [14783] [INFO] Starting gunicorn 20.0.0
[2019-11-20 16:14:27 -0800] [14783] [INFO] Listening at: https://0.0.0.0:8999 (14783)
[2019-11-20 16:14:27 -0800] [14783] [INFO] Using worker: sync
[2019-11-20 16:14:27 -0800] [14798] [INFO] Booting worker with pid: 14798

[2019-11-20 16:16:39,550] INFO  [access:342][MainThread] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:8999/session HTTP/1.1" {'username': 'colim', 'password': ''}
[2019-11-20 16:16:39,768] INFO  [access:362][MainThread] 10.9.202.54 - - "POST https://bioappsdev02.bcgsc.ca:8999/session HTTP/1.1" 200 639

I'm not sure if it's related at all, but starting the server using gunicorn 20.0.0 with pserve is the only time we see this warning:

/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/gunicorn/workers/ggevent.py:53: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/home/colim/Projects/bioapps/bioapps.api.ssl/centos7venv/lib/python3.6/site-packages/urllib3/util/__init__.py)'].
  monkey.patch_all()

@tilgovi what would you change in the changelog?

@benoitc I would like to call out the removal of support for Paste Deploy server definitions in the Gunicorn CLI. I can do this today.

Is it alright if I modify the changelog retro-actively to make it more clear (make breaking changes section in the 20.0 release)?

@CorreyL interesting! You can definitely run gunicorn with the options specified on the command line like that. I think that is the preferred and safest way to run gunicorn. The integration with pserve is a convenience, but it's nice to know it causes issues here. I hope I didn't make a mistake to un-deprecate it.

Is it alright if I modify the changelog retro-actively to make it more clear (make breaking changes section in the 20.0 release)?

@tilgovi yes definitely

@tilgovi can you add this change today? Would be cool to have it for 20.0.1 :)

Added a one line note to c25563f. The documentation has already been updated since the change happened. Hopefully, anyone who sees that note can find the documentation and these issues. 😅

@tilgovi thanks

Just wanted to add another confirmation of getting surprisingly impacted by this. It wasn't very significant in my case, but I was confused about why gunicorn stopped auto-reloading in dev since upgrading as well as a few other minor behavior changes. I spent some time trying to figure it out today and realized that the settings in my --paste INI files were no longer working, which helped me find my way to this issue.

I have no idea if this is feasible, but would it be possible to have gunicorn output a warning if it detects that you're still trying to set server arguments through the Paster file?

My apologies for the disruption, @Deimos. I would review PRs, but I have no specific plans to add more here.

How about a case when you actually want to use --paste together with --config? Which in our case (RhodeCode) is a big requirement for the special logic of memory monitoring we've got in the gunicorn config.

@marcinkuzminski that's the ideal use case. Just specify both --paste and --config. However, Gunicorn will not read the "server" section of the paste ini file because the expectation is that you will configure the server in the gunicorn configuration file.

That's unfortunate.

We're shipping gunicorn to customers in installer, and all logic and configuration has been delegated to the .ini files. This is how also most of examples over internet specifies for Pyramid projects.

This change breaks that, and it's probably easier for us to fork gunicorn to bring that back, then change the logic and delegate configuration to gunicorn_conf.py :(

What about if the --paste options would be read, with a special prefix. e.g you could configure gunicorn with --paste but it would read only config options that would be prefixed with gunicorn.

e.g

gunicorn.workers = 2
gunicor.XXX = YYY

You don't need to use --config. You can use the paste INI entirely. For that use pserve instead of gunicorn. See the documentation: https://docs.gunicorn.org/en/stable/run.html#paste-deployment

The change that was made was only to remove support for using Gunicorn as a general Paste Deployment CLI that can run servers. Gunicorn still _is a_ Paste compatible server itself.

This change was made to remove potential confusion where a .ini file would specify waitress, or any other server, in its server block, but running it with gunicorn --paste production.ini would not actually use waitress at all. People also often requested the ability to specify an alternate server block other than server:main. Maintaining support for these features when perfectly good CLIs like pserve exist for this did not seem to make sense.

The gunicorn CLI can read an application definition from an INI file, but it uses its own configuration file for configuring a server. Use another tool (like pserve) as the script / process runner if you want to use the INI file to configure a Gunicorn server.

but we have to use --config together with --paste, as per my 1 st comment.
In our project everything is managed by single configuration file (.ini) There's lot of upgrade/autoscale logic that just adjusts the .ini file. Then we use also --config to specify a custom python config that sets

• custom logger format (this technically isn't possible to be specified using .ini file)
• worker memory management (Python code)

Gunicorn is Paste compatible, but functionality is limited in this way, and it created a problem for us we're unable to recover from because we cannot have 2 configuration files, and also moving to the configuration on another file is more work then actually forking Gunicorn and maintaining that fork just to bring that behaviour back.

I know the rationale for this ticket, but we used to use gunicorn and waitress together, and i thinking running gunicorn binary is explicit enough, IMHO. Additionally, i think you can even detect if users use different egg and make it a hard error.

We didn't considered such usage if I remember. We can probably bring back
the support of it as the usecase sounds good. Would it be OK to have a
log warning for it ?

On Fri 16 Oct 2020 at 08:28 Marcin Kuźmiński notifications@github.com
wrote:

but we have to use --config together with --paste, as per my 1 st
comment.
In our project everything is managed by single configuration file (.ini)
There's lot of upgrade/autoscale logic that just adjusts the .ini file.
Then we use also --config to specify a custom python config that sets

• custom logger format (this technically isn't possible to be
  specified using .ini file)
• worker memory management

Gunicorn is paste compatible, but functionality is limited in this way,
and it created a problem for us we're unable to recover from.

I know the rationale, but we used to use gunicorn and waitress together,
and i thinking running gunicorn binary is explicit enough, IMHO.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/benoitc/gunicorn/issues/2169#issuecomment-709838842,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAADRIQR2CLVUOYK6FDY2ZDSK7RZFANCNFSM4JMI65YA
.

>

Sent from my Mobile

I actually thought on another solution if possible. When using pserve with unicorn egg, it would be also nice that the configuration file would be set inside the .ini file.

e.g

use = egg:gunicorn#main
workers = 2
config = /path/to/gunicorn_conf.py

So it would load the gunicorn_conf.py exactly like --config=/path/to/gunicorn_conf.py does

So the above would work for us, and it's solving also the problem of this ticket. Not sure how easy and feasible that is to implement.

Otherwise, If you could bring the functionality of loading config from .ini file when running gunicorn binary it would be awesome, that would save us lot of hassle. Having a warning about it is no problem

I actually thought on another solution if possible. When using pserve with unicorn egg, it would be also nice that the configuration file would be set inside the .ini file.

That should work and is documented. If it does not, please file a bug!

Ok, we'll check this. But AFAIR there were slight changes in how the gunicorn vs pserve binaries work. If I recall, gunicorn --paste had access to the .ini file path, while pserve using gunicorn egg didn't. We'll check this and open a relevant ticket if needed.