Ini adalah lingkungan kita:
pengirim log (500+ server dengan konfigurasi rsyslog) ---> penerus log (2 server) --> server target
Masalah ini terjadi di server penerusan log.
Setelah berjalan 2 hari, ada banyak koneksi terbuka di 2 server ini.
Ini memiliki lebih dari 1 koneksi dari 1 IP.
Saya telah memeriksa pengirim log tertentu, hanya 1 koneksi aktif di sana.
Jadi, masalah 1 => mengapa ada begitu banyak koneksi mati terbuka di sisi server?
Saya mencoba menyelesaikan masalah koneksi mati dengan menambahkan konfigurasi keepalive:
$InputTCPServerKeepAlive on
Koneksi berkurang sangat cepat, tetapi kesalahan yang berbeda muncul di log:
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad4bc9820 from 169.61.224.213 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad4b89910 from 52.116.56.204 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad4b885d0 from 169.61.246.243 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad4ba1330 from 149.81.89.147 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:12:33 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:14:28 qrada-log-forwarder-lbaas-2 rsyslogd: message repeated 2 times: [unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]]
Sep 20 02:14:28 qrada-log-forwarder-lbaas-2 rsyslogd: rsyslogd[internal_messages]: 139 messages lost due to rate-limiting
Sep 20 02:14:28 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad41ca310 from 141.125.112.94 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:26:29 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:26:29 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad5741740 from 168.1.224.168 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:32:23 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:32:23 qrada-log-forwarder-lbaas-2 rsyslogd: netstream session 0x7f7ad411ded0 from 130.198.104.90 will be closed due to error [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Sep 20 02:39:35 qrada-log-forwarder-lbaas-2 rsyslogd: unexpected GnuTLS error -54 in nsd_gtls.c:546: Error in the pull function. [v8.32.0 try http://www.rsyslog.com/e/2078 ]
server adalah VM di cloud
versi rsyslog:
rsyslogd 8.32.0, compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Number of Bits in RainerScript integers: 64
See http://www.rsyslog.com for more information.
Ubuntu 18.04-64
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
# use gtls netstream driver
$DefaultNetstreamDriver gtls
# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/cacert.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/keys/servercert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/keys/serverkey.pem
global(debug.gnutls="10" debug.logFile="/var/log/rsyslogdebug")
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="10514")
$ModLoad imtcp
$InputTCPServerRun 10514
$InputTCPServerKeepAlive on
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
$InputTCPMaxSessions 10000
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
authpriv.* @@10.94.170.164:514
Catatan: dokumentasi rsyslog tersedia di sini: - rilis stabil saat ini: http://www.rsyslog.com/doc/v8-stable/ - pra-rilis: http://www.rsyslog.com/doc/master/
Mencoba google, tidak berhasil, buka debug dengan menambahkan baris berikut ke rsyslog.conf
global(debug.gnutls="10" debug.logFile="/var/log/rsyslogdebug")
Dan mulai rsyslog dengan mode debug: /usr/sbin/rsyslogd -dn
Berikut adalah file lognya.
versi rsyslog yang ditingkatkan, masih error:
Sep 21 02:14:37 qrada-log-forwarder-lbaas-2 rsyslogd[4895]: unexpected GnuTLS error -54 in nsd_gtls.c:594: Error in the pull function. [v8.1910.0.9814b01e74e0 try https://www.rsyslog.com/e/2078 ]
Sep 21 02:14:37 qrada-log-forwarder-lbaas-2 rsyslogd[4895]: netstream session 0x7fe2cc071890 from 135.90.112.13 will be closed due to error [v8.1910.0.9814b01e74e0 try https://www.rsyslog.com/e/2078 ]
rsyslogd 8.1910.0.9814b01e74e0 (aka 2019.10) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /var/run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
See https://www.rsyslog.com for more information.
Sejak Versi Versi 8.32.0 (2018-01-09), telah terjadi banyak perubahan pada kode gnutls / openssl.
Saya sarankan Anda mencoba rsyslog terbaru dari repositori kami, untuk memverifikasi apakah masalahnya belum diperbaiki:
https://www.rsyslog.com/ubuntu-repository/
Jika masalah berlanjut, kami dapat melihat lebih dalam masalah Anda.
Saya akan berasumsi bahwa ini hanya koneksi terputus. Gnutls hanya melaporkannya dengan pesan kesalahan umum.
@alorbach Saya telah mencoba versi 8.1910.0.9814b01e74e0, masalah yang sama.
@rgerhards
Setelah beberapa hari berjalan, kami telah mengamati log kesalahan yang sama di sisi klien.
Sep 23 21:28:06 lb-bd1247c1-65065 rsyslogd[1495]: unexpected GnuTLS error -53 - this could be caused by a broken connection. GnuTLS reports: Error in the push function. [v8.32.0 try http://www.rsyslog.com/e
Sep 23 21:28:06 lb-bd1247c1-65065 rsyslogd[1495]: omfwd: TCPSendBuf error -2078, destruct TCP Connection to logforwarder.lb.appdomain.cloud:10514 [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Ketika koneksi terputus, apakah rsyslog akan menyambungkan kembali dan mengirim ulang log yang gagal terkirim?
Saya pikir kami telah mengamati log yang terlewatkan, dan ini adalah masalah nyata.
Baca
https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
agar dapat diandalkan, Anda perlu menggunakan relp
David Lang
Sepertinya koneksi rusak.
@lichen2013 Anda dapat mencoba driver openssl ("ossl") yang memberikan pelaporan dan penanganan kesalahan yang jauh lebih baik:
# use ossl netstream driver
$DefaultNetstreamDriver ossl
Untuk informasi lebih lanjut:
https://www.rsyslog.com/doc/v8-stable/concepts/ns_ossl.html
Terima kasih atas bantuannya, @alorbach @rgerhards.
Akan memeriksa relp dan openssl.
Karena pesan kesalahan ini bukan masalah sebenarnya, tutup masalah ini.
Utas ini telah dikunci secara otomatis karena tidak ada aktivitas terbaru setelah ditutup. Silakan buka edisi baru untuk bug terkait.