Aws-iot-device-sdk-python-v2: AWS_ERROR_MQTT_UNEXPECTED_HANGUP Endlose Wiederverbindungsschleife
Bestätigen Sie, indem Sie unten [ ] in [x] ändern, um sicherzustellen, dass es sich um einen Fehler handelt:
- [x] Ich habe nach ähnlichen Problemen gesucht und keine Lösung gefunden
Bekanntes Problem
- [x] Ich verwende einen Endpunkt des ATS-Datentyps: Der Endpunkt sollte wie folgt aussehen:
<prefix>-ats.iot.<region>.amazonaws.com
Hallo AWS-Team,
wir stecken seit 2 Tagen mit einem kritischen Problem mit dem SDK fest (Details unten).
Beschreibe den Fehler
Das Herunterfahren des TLS-Kanals durch den Task-Scheduler unmittelbar nach der Host-Auflösung, was zu einer unendlichen RECONNECTING <> AWS_ERROR_MQTT_UNEXPECTED_HANGUP-Schleife führt (Essen viele Daten !!).
Die SDK-Verbindung zum Endpunkt wird nie hergestellt. TLS-Öffnen und -Schließen von Sitzungen in einem sehr schnellen Tempo, was einen enormen Datenverbrauch verursacht (wir haben aufgrund des plötzlichen Problems mehrere Hundert US-Dollar an Daten-Roaming-Rechnungen auf einem Feldgerät erhalten).
SDK-Versionsnummer
Getestet mit 1.5.12 & 1.5.14 und jeweils passendem aws-crt
Plattform/Betriebssystem/Gerät
Armhf-Gerät (Cortex-A7)
Docker-Container basierend auf Ubuntu 20.04
Reproduzieren (beobachtetes Verhalten)
Die Standardverwendung des SDK wurde beobachtet und das Gerät lief 10 Tage lang, bevor die Wiederverbindungsschleife auftrat. Danach konnte sich das Gerät nie wieder richtig mit unserem Endpunkt verbinden und die Wiederverbindungsschleife war immer vorhanden. Verhalten geändert am 30.04.2021 nach 10 Tagen Normalbetrieb (Verbindungen & Veröffentlichung alle 3 Minuten). Die Zertifikate & CA sind nicht abgelaufen und eine erneute Geräteregistrierung unter einem neuen Zertifikat hat nicht geholfen. Update von SDK 1.5.12 auf 1.15.14 hat auch nicht geholfen.
AWS_KEEP_ALIVE_SECS = 30
# setup AWS connection
io.init_logging(io.LogLevel.Debug, 'stderr')
event_loop_group = io.EventLoopGroup(1)
host_resolver = io.DefaultHostResolver(event_loop_group)
client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)
client_id = "auto-%s" % str(uuid.uuid4()).upper()
aws_mqtt = mqtt_connection_builder.mtls_from_path(
endpoint=args.endpoint,
cert_filepath="%s/device.crt" % (args.provision_location),
pri_key_filepath="%s/device.key" % (args.provision_location),
client_bootstrap=client_bootstrap,
ca_filepath="%s/AmazonRootCA1.pem" % (args.cert_location),
on_connection_interrupted=aws_on_connection_interrupted,
on_connection_resumed=aws_on_connection_resumed,z
client_id=client_id,
clean_session=False,
keep_alive_secs=AWS_KEEP_ALIVE_SECS)
Erwartetes Verhalten
Kein unerwartetes Auflegen. Und keine unendliche Wiederverbindungsschleife.
Protokolle/Ausgabe
Die Netzwerknutzung führte zu einer SIM-Datennutzung von mehreren GB auf einem bereitgestellten Roaming-Gerät (16 GB). Hier ist die Netzwerkdatennutzung bei einem eth0-basierten Verbindungsversuch:
Trace auf Debug-Ebene
[?2004l[?1l>[?25h[>4;m[?1049l[23;0;0t]0;root@aeler-imx7ulpea-ucom: /root@aeler-imx7ulpea-ucom:/# pyu thon3 - service.py
aws-iotsdk> [2021-05-04 13:53:48.925804] AWS Topic: /d/22f66ba6f37b3ede6115c86389e530eb1d408103
aws-iotsdk> [2021-05-04 13:53:48.952523] [mc] connecting to localhost on port 1883
aws-iotsdk> [2021-05-04 13:53:49.104120] [mc] connect rc: 0
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Initializing edge-triggered epoll
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Using eventfd for cross-thread notifications.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Starting event-loop thread.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [dns] - id=0x162d7a0: Initializing default host resolver with 16 max host entries.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [channel-bootstrap] - id=0x164f998: Initializing client bootstrap with event-loop group 0x16eddb0
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: main loop started
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: default timeout 100000, and max events to process per tick 100
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Certificate and key have been set, setting them up now.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: cannot enable ocsp stapling: OCSP stapling was requested, but is not supported
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Setting ALPN list x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - client=0x17347e8: Initalizing MQTT client
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Creating new connection
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-topic-tree] - tree=0x163db24: Creating new topic tree
aws-iotsdk> [2021-05-04 13:53:53.308811] checking internet connection.
aws-iotsdk> [2021-05-04 13:53:53.734491] [aws] Connecting to <my_endpoint>-ats.iot.us-east-1.amazonaws.com as auto-2C288BE4-A555-4653-92D8-A632727D5B1C ...
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Begin connecting process, switch state to CONNECTING.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: using ping timeout of 3000000000 ns
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: No cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com starting new resolver thread.
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 52.21.198.210 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.147.70.168 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 34.206.245.43 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.158.175.156 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 3.225.200.224 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.145.247.28 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.84.18.235 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 18.207.61.253 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::23ad:4123 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3db:a390 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::34c8:5c29 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::36a3:c4c1 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:b950 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, invoking resolution callback for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com with 2 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3e1:f981:443.
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:53Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3e1:f981 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 52.21.198.210:443.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Scheduling (null) task for future execution at time 2076022788913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:39232
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: no message pool is currently stored in the event-loop local storage, adding 0xb4805c40 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket-handler] - id=0xb4817418: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2081199692246
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Scheduling mqtt_connack_timeout task for future execution at time 2074947672913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 0 to CONNECTED.
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2102147419913
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:56.689408] checking internet connection.
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:57.606042] [aws] Publishing message to topic '/d/22f66ba6f37b3ede6115c86389e530eb1d408103
'
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Scheduling schedule_cross_thread_tasks task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Running schedule_cross_thread_tasks task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b6314460] [mqtt-client] - id=0x163d9b8: Starting publish 1 to topic /d/22f66ba6f37b3ede6115c86389e530eb1d408103
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Running mqtt_connack_timeout task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2075467825247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:53:58.359515] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::22cd:842b:443.
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::22cd:842b for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.158.175.156:443.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4804268: Scheduling (null) task for future execution at time 2080568943914
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:47820
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket-handler] - id=0xb482ec00: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2085748031247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Scheduling mqtt_connack_timeout task for future execution at time 2079454727914
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
aws-iotsdk> [2021-05-04 13:53:59.568730] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2106715588247
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2077377824580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48353c8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:00.265671] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3417:2507:443.
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3417:2507 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.147.70.168:443.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4803ea8: Scheduling (null) task for future execution at time 2082467830580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:50080
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket-handler] - id=0xb482b660: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2087638712914
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb482d770: Scheduling mqtt_connack_timeout task for future execution at time 2081369652914
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:54:01.464140] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2108610288247
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2079278874581
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb482d770
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482d770: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:02.177324] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
Zusätzlicher Kontext
Lösungsversuche: mqtt builder-Parameter ändern, Gerät mit einem anderen Zertifikat registriert
Das Gerät wird automatisch über den JITP-Prozess registriert und die folgende JTIP_PubSub-Richtlinie angehängt:
bug
qcabrol
👍3
Alle 8 Kommentare
+1
Hadatko
am 4. Mai 2021
Für etwas mehr Kontext:
Ein anderes Gerät mit genau dem gleichen Build und der gleichen Konfiguration verbindet sich gut. Der andere ist es immer noch nicht. Ich frage mich, ob wir vielleicht ein Kontingent für die Veröffentlichungsgröße erreichen oder so. Wird noch untersucht... In diesem Fall würde ich jedoch einen Fehler erwarten.
qcabrol
am 4. Mai 2021
+1 – Wir sehen dies bei ungefähr 4 % der Geräte im Feld
Wir verwenden Ubuntu 18.04 auf arm64-Geräten mit awsiotsdk=1.5.11 und awscrt=0.11.8
JoelHensley
am 4. Mai 2021
👍2
Ein anderes Gerät mit genau dem gleichen Build und der gleichen Konfiguration verbindet sich gut. Der andere ist es immer noch nicht.
Jedes Gerät verbindet sich mit einer eindeutigen Client-ID, oder?
Wenn sich ein zweites Gerät mit derselben Client-ID verbindet, akzeptiert der Server diese neue Verbindung und beendet die alte Verbindung. Das führt dazu, dass das erste Gerät wieder verbunden wird, wodurch das zweite Gerät gestartet wird, sodass das zweite Gerät erneut verbunden wird, wodurch das erste Gerät gestartet wird usw
Die "Client-ID", von der ich spreche, ist das Argument client_id , das an mqtt_connection_builder- Funktionen oder an die awscrt.mqtt.Connection- Klasse übergeben wird
graebm
am 5. Mai 2021
Auch WOW, das über die schnelle Wiederverbindungsschleife nervt, die GB an Daten aufsaugt.
Unsere Implementierung versucht, während der Wiederverbindung einen exponentiellen Backoff-Timer zu verwenden, aber sobald die Verbindung als "erfolgreich" angesehen wird, wird der Timer auf Null zurückgesetzt. Wenn die Verbindung sofort nach erfolgreichem Abschluss unterbrochen wird, wird das Gerät sofort wieder eine Verbindung herstellen, da dieser Timer auf Null gesetzt wurde.
Klingt so, als ob wir warten möchten, bis die Verbindung für einige Zeit "stabil" bleibt, bevor wir diesen Timer zurücksetzen...
graebm
am 5. Mai 2021
👍1
Wenn der Server Ihr Gerät trennt, können Sie den Grund dafür finden. Siehe Überwachen von AWS IoT mit CloudWatch Logs
graebm
am 5. Mai 2021
Hallo @graebm
Danke für die Hinweise
Leider ist die INFO-Ebene von AWS Logs hier nicht sehr hilfreich:
Bezüglich der Eindeutigkeit der Client-ID wird diese bei jedem Anwendungsstart mit Python uuid4() aus dem uuid-Modul generiert:
client_id = "auto-%s" % str(uuid.uuid4()).upper()
Dies bedeutet, dass das Problem, selbst wenn wir extrem viel Pech hatten, doppelte UUIDs hatte, niemals einen Anwendungsneustart überleben sollte, der eine neue Client-Uuid verwendet, richtig? Ich werde ein wenig auf diese Möglichkeit eingehen.
Eine andere Sache , die ich überlege , ist , ob wir eine Art Kontingent für die Veröffentlichungspaketgröße erreichen . Wenn ein Gerät aufgrund eines Netzwerkproblems getrennt wird, bündeln wir viele Pakete, die wir in einem einzigen Aufruf veröffentlicht haben, um die Gesamtdatennutzung zu begrenzen. Wir versuchen jetzt, diese maximale Bundle-Größe zu begrenzen, um zu sehen, ob dies das Verhalten beeinflusst. 128 KB ist die maximale Größe.
Die Frage ist, gibt es auf der SDK-Clientseite kontingentbezogene Filter, bevor eine Anfrage gesendet wird? (Das würde viele nicht konforme Pakete verhindern und explizite Fehler erhalten, die das Debuggen im Falle einer schlechten Anfrage erleichtern, bevor eine Verbindung hergestellt und Netzwerkdaten verbraucht werden).
qcabrol
am 5. Mai 2021
Unterm Strich also:
- wir serialisieren rohe Bytes so, dass die Paketgröße etwas aufgebläht wird (wird später Binärpakete verwenden)
- Wenn die Verbindung unterbrochen wird, senden wir Pakete in einer gruppierten Liste von Byte-Arrays (bis zu 1000 Nachrichten gruppiert)
Wir haben die Berechnungen für verschiedene Szenarien durchgeführt und waren in dieser Situation über der Quote.
Wir haben die maximale Anzahl von Nachrichten, die in einem Paket gebündelt sind, begrenzt und es hat funktioniert.
Dies geschah, weil das Gerät für eine Weile die Verbindung verloren hatte und einige Warnungen protokolliert wurden, wodurch ein Datenburst über dem von der Quote unterstützten Limit verursacht wurde.
IMHO sollte das SDK Pakete auf Verstöße gegen feste Quoten überprüfen und einen Fehler ausgeben, bevor es versucht, sie zu senden, um das Debuggen solcher Situationen zu erleichtern. Hier war die Quelle unklar.
Sie sollten die Situation reproduzieren können, indem Sie ein Paket mit einer Größe von >128kB in einer Try-Schleife veröffentlichen.
qcabrol
am 5. Mai 2021
War diese Seite hilfreich?
0 / 5 - 0 Bewertungen
Verwandte Themen
mkozjak
·
17Kommentare
banuprathap
·
10Kommentare
supertick
·
7Kommentare
satay99
·
6Kommentare
victorct-pronto
·
3Kommentare
Hilfreichster Kommentar
+1 – Wir sehen dies bei ungefähr 4 % der Geräte im Feld
Wir verwenden Ubuntu 18.04 auf arm64-Geräten mit awsiotsdk=1.5.11 und awscrt=0.11.8