Zammad: Scheduler not running because of Bad file descriptor in PGConsumeInput()

This usually happens when there is no space left on temp or the data store. I don't think this is Zammad related. The traceback is directly from
ActiveRecord

I suggest to check the file system( space and inodes) and perform a check on the PG database.

Edit:
What happens when you restart both, Scheduler and Postgresql?

Note:
Please don't use the develop branch in production! 2.0 is stable

Filesystem does not look anywhere close to full.

hexa@tickets:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G  6.2G   13G  34% /
udev             10M     0   10M   0% /dev
tmpfs           502M   45M  457M   9% /run
tmpfs           1.3G   16K  1.3G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.3G     0  1.3G   0% /sys/fs/cgroup
tmpfs           251M     0  251M   0% /run/user/2000
hexa@tickets:~$ df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/xvda1     1310720 97329 1213391    8% /
udev            314738   268  314470    1% /dev
tmpfs           320664   313  320351    1% /run
tmpfs           320664     5  320659    1% /dev/shm
tmpfs           320664     3  320661    1% /run/lock
tmpfs           320664    13  320651    1% /sys/fs/cgroup
tmpfs           320664     4  320660    1% /run/user/2000

PostgreSQL does not seem to have any internal checking mechanic. Restarting both doesn't change the situation.

I understand not to use the develop branch, but we needed to start with 1.6 since we required LDAP and then forget to pin the version.

Also PostgreSQL is clearly up and accessible. This also works with the credentials set in /opt/zammad/config/database.yml.

root@tickets:/home/hexa# sudo -u postgres psql
psql (9.4.13)
Type "help" for help.

postgres=# \connect zammad
You are now connected to database "zammad" as user "postgres".

zammad=# \dt
                      List of relations
 Schema |                Name                | Type  | Owner  
--------+------------------------------------+-------+--------
 public | activity_streams                   | table | zammad
 public | ar_internal_metadata               | table | zammad
 public | authorizations                     | table | zammad
 public | avatars                            | table | zammad
 public | calendars                          | table | zammad
 public | channels                           | table | zammad
 public | chat_agents                        | table | zammad
 public | chat_messages                      | table | zammad
 public | chat_sessions                      | table | zammad
 public | chat_topics                        | table | zammad
 public | chats                              | table | zammad
 public | cti_caller_ids                     | table | zammad
 public | cti_logs                           | table | zammad
 public | delayed_jobs                       | table | zammad
 public | email_addresses                    | table | zammad
 public | external_credentials               | table | zammad
 public | external_syncs                     | table | zammad
 public | groups                             | table | zammad
 public | groups_users                       | table | zammad
 public | histories                          | table | zammad
 public | history_attributes                 | table | zammad
 public | history_objects                    | table | zammad
 public | history_types                      | table | zammad
 public | http_logs                          | table | zammad
 public | import_jobs                        | table | zammad
 public | jobs                               | table | zammad
 public | karma_activities                   | table | zammad
 public | karma_activity_logs                | table | zammad
 public | karma_users                        | table | zammad
 public | link_objects                       | table | zammad
 public | link_types                         | table | zammad
 public | links                              | table | zammad
 public | locales                            | table | zammad
 public | macros                             | table | zammad
 public | network_categories                 | table | zammad
 public | network_categories_moderator_users | table | zammad
 public | network_category_subscriptions     | table | zammad
 public | network_category_types             | table | zammad
 public | network_item_comments              | table | zammad
 public | network_item_plus                  | table | zammad
 public | network_item_subscriptions         | table | zammad
 public | network_items                      | table | zammad
 public | network_privacies                  | table | zammad
 public | networks                           | table | zammad
 public | notifications                      | table | zammad
 public | oauth_access_grants                | table | zammad
 public | oauth_access_tokens                | table | zammad
 public | oauth_applications                 | table | zammad
 public | object_lookups                     | table | zammad
 public | object_manager_attributes          | table | zammad
 public | online_notifications               | table | zammad
 public | organizations                      | table | zammad
 public | organizations_users                | table | zammad
 public | overviews                          | table | zammad
 public | overviews_groups                   | table | zammad
 public | overviews_roles                    | table | zammad
 public | overviews_users                    | table | zammad
 public | package_migrations                 | table | zammad
 public | packages                           | table | zammad
 public | permissions                        | table | zammad
 public | permissions_roles                  | table | zammad
 public | postmaster_filters                 | table | zammad
 public | recent_views                       | table | zammad
 public | report_profiles                    | table | zammad
 public | roles                              | table | zammad
 public | roles_groups                       | table | zammad
 public | roles_users                        | table | zammad
 public | schedulers                         | table | zammad
 public | schema_migrations                  | table | zammad
 public | sessions                           | table | zammad
 public | settings                           | table | zammad
 public | signatures                         | table | zammad
 public | slas                               | table | zammad
 public | stats_stores                       | table | zammad
 public | store_files                        | table | zammad
 public | store_objects                      | table | zammad
 public | store_provider_dbs                 | table | zammad
 public | stores                             | table | zammad
 public | tag_items                          | table | zammad
 public | tag_objects                        | table | zammad
 public | tags                               | table | zammad
 public | taskbars                           | table | zammad
 public | templates                          | table | zammad
 public | templates_groups                   | table | zammad
 public | text_modules                       | table | zammad
 public | text_modules_groups                | table | zammad
 public | ticket_article_flags               | table | zammad
 public | ticket_article_senders             | table | zammad
 public | ticket_article_types               | table | zammad
 public | ticket_articles                    | table | zammad
 public | ticket_counters                    | table | zammad
 public | ticket_flags                       | table | zammad
 public | ticket_priorities                  | table | zammad
 public | ticket_state_types                 | table | zammad
 public | ticket_states                      | table | zammad
 public | ticket_time_accountings            | table | zammad
 public | tickets                            | table | zammad
 public | tokens                             | table | zammad
 public | translations                       | table | zammad
 public | triggers                           | table | zammad
 public | type_lookups                       | table | zammad
 public | user_devices                       | table | zammad
 public | users                              | table | zammad
(103 rows)

Today I have the same on mac os 10.12.6 (but it was working the days before 🤥) with develop (Zammad 2.1). All is working (rails c, rails s, ...) expecting the scheduler.

I have the same Upgrade from 1.5 to 2.0 100% CPU usage and this error

tail: scheduler_err.log: Datei abgeschnitten
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() could not receive data from server: Bad file descriptor
: SELECT "delayed_jobs".* FROM "delayed_jobs"
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in async_exec' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/postgresql_adapter.rb:598:inblock in exec_no_cache'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/abstract_adapter.rb:590:in block in log' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.5/lib/active_support/notifications/instrumenter.rb:21:ininstrument'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/abstract_adapter.rb:583:in log' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/postgresql_adapter.rb:598:inexec_no_cache'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in execute_and_clear' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:inexec_query'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/abstract/database_statements.rb:377:in select_prepared' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/abstract/database_statements.rb:39:inselect_all'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in select_all' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/querying.rb:39:infind_by_sql'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/relation.rb:702:in exec_queries' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/relation.rb:583:inload'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/relation.rb:260:in records' /opt/zammad/vendor/bundle/ruby/2.4.0/gems/activerecord-5.0.5/lib/active_record/relation/delegation.rb:38:ineach'
/opt/zammad/app/models/scheduler.rb:78:in cleanup' /opt/zammad/app/models/scheduler.rb:24:inthreads'
script/scheduler.rb:66:in `block in

@Hagelsturm which os? And what process takes 100%?

In our case it is script/scheduler.rb start -t.

In our case it is script/scheduler.rb start -t

That's also on my side. Normally the scheduler is running in daemon mode (script/scheduler.rb start or script/scheduler.rb stop).

script/scheduler.rb start -t is usually only used for developers because the scheduler is not running in background (as daemon).

@mweinelt A question for me is, why you are using -t?

JFI: The scheduler is working fine for me in background (as daemon).

I'm running 2.1.1-1505985142.807a1d88.jessie because I forgot to pin to a stable version. The -t option must come from the service files, I haven't changed anything in that regard.

root@tickets:/opt/zammad# grep -ri "scheduler.rb start" *
contrib/systemd/zammad-scheduler.service:ExecStart=/bin/bash -l -c "${BUNDLE_BINARY} exec script/scheduler.rb start -t"
Procfile:worker: bundle exec script/scheduler.rb start -t
Procfile.frontend:worker: bundle exec ruby script/scheduler.rb start -t
script/init-script-normal-user-rvm-fedora:    script/scheduler.rb start &> /dev/null && echo_success || echo_failure
script/init.d/zammad:  execute "RAILS_ENV=production script/scheduler.rb start $SCHEDULER_OPTS"
script/build/test_startup.sh:bundle exec script/scheduler.rb start
script/local_browser_tests.sh:script/scheduler.rb start
vendor/pkgr/processes/worker:exec bundle exec script/scheduler.rb start -t $@

I use Debian 8 yesterday i Try Version 9 also
But The error its The same
I dont modify anything the -t came from the package not from me

The Process with 100% is Ruby
script/scheduler.rb start -t
Everytime is starting and crashing

from scheduler.rb --help:

• "-t, --ontop Stay on top (does not daemonize)"

Maybe but the Scheduler crash all the Time
Die You Think the reason is the ontop Parameter?

@Hagelsturm
No, just wanted to clarify what it does. Did not see martinis answer.
Daemonizing is done by systemd so normally its no problem.

@martini
If i remember right "-t" is used because systemd thinks scheduler is failed & stopped when it goes into backround from the unit file.

@mweinelt
If you're using 2.1.1 you're in the develop repo.
There is no way to pin a stable package in there.
Use stable repo to get stable packages: https://packager.io/gh/zammad/zammad/builds/2414/install/debian-8

@monotek
Setting Type=forking should allow for starting without -t if I'm not wrong.

Regarding the downgrade to a stable release (2.0.x):
I would have to revert some migrations if I don't have a recent backup, right?

Thanks for the hint. Will try change it later. Not sure if possible in packager.io.

okay i use version 2.0 now
but now i have a problem with the email Channel

Email::Notification out Can't use Channel::Driver::Smtp: #

I cant check the Settings because the Account is empty :-(

channel is active but not fetched for 1 hour
Channel: Email::Notification out Can't use Channel::Driver::Smtp: #

how its possible to set is new on the cli?

@Hagelsturm try checking your smtp settings, probably the host parameter for SMTP in your email channel settings is not there. Also, since your problem is a new problem, you should create a new issue?

Hi there! The very same problem here. I get here from #1473 . I installed version 2.0 but could not add a new email account on email channel configuration, maybe related to the last comment from @Hagelsturm.

I would have to revert some migrations if I don't have a recent backup, right?

@mweinelt sorry, I did not see this feedback. At the moment there are no database migrations between stable and develop. So right now you can switch between stable/master and develop without problems.

Anyway, it seems that I got the issue "Bad file descriptor in PGConsumeInput" fixed. So if you update develop first (before switching back to stable) would be fine!

Thanks for feedback!

-Martin

@Hagelsturm JFI your email issue has noting todo with this scheduler issue. Another issue would be good for that.

@martini I can confirm that update did the trick. Thank you very much!

Will try a downgrade to 2.0 in a bit.