Aws-iot-device-sdk-python-v2: AWS_ERROR_MQTT_UNEXPECTED_HANGUP Infinite reconnection loop

Created on 4 May 2021 · 8Comments · Source: aws/aws-iot-device-sdk-python-v2

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[x] I've searched for previous similar issues and didn't find any solution

Known Issue

[x] I'm using ATS data type endpoint: the endpoint should look like <prefix>-ats.iot.<region>.amazonaws.com

Hi AWS team,

we are a bit stuck with a critical issue we are facing with the SDK since 2 days (details below).

Describe the bug
TLS channel shutdown by the task-scheduler immediately after host resolution resulting in infinite RECONNECTING <> AWS_ERROR_MQTT_UNEXPECTED_HANGUP loop (Eating a lot of data !!).
SDK connection to the endpoint is never established. TLS opening and closing of sessions at a very fast pace causing tremendous data usage burst (we got several hundred $ of data roaming bill on a field device due to the sudden issue).

SDK version number
Tested with 1.5.12 & 1.5.14 and matching aws-crt in each case

Platform/OS/Device
Armhf device (Cortex-A7)
Docker container based on ubuntu 20.04

To Reproduce (observed behavior)

Standard use of the SDK was observed and the device was running for 10 days before the reconnection loop occurred. After that, the device was never able to reconnect properly to our endpoint and the reconnection loop would always be present. Behavior changed on 30.04.2021 after 10 days of normal operation (connections & publish every 3 minutes). The certificates & CA are not expired and renewed device registration under a new certificate did not help. Update from SDK 1.5.12 to 1.15.14 did not help either.

AWS_KEEP_ALIVE_SECS = 30

    # setup AWS connection
    io.init_logging(io.LogLevel.Debug, 'stderr')
    event_loop_group = io.EventLoopGroup(1)
    host_resolver = io.DefaultHostResolver(event_loop_group)
    client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)

    client_id = "auto-%s" % str(uuid.uuid4()).upper()
    aws_mqtt = mqtt_connection_builder.mtls_from_path(
        endpoint=args.endpoint,
        cert_filepath="%s/device.crt" % (args.provision_location),
        pri_key_filepath="%s/device.key" % (args.provision_location),
        client_bootstrap=client_bootstrap,
        ca_filepath="%s/AmazonRootCA1.pem" % (args.cert_location),
        on_connection_interrupted=aws_on_connection_interrupted,
        on_connection_resumed=aws_on_connection_resumed,z
        client_id=client_id,
        clean_session=False,
        keep_alive_secs=AWS_KEEP_ALIVE_SECS)

Expected behavior

No unexpected hangup. And no infinite reconnection loop.

Logs/output

Network usage caused a SIM data usage of several GB on a deployed roaming device (16GB). Here is the network data usage on a eth0 based connection attempt:

Debug level trace

[?2004l[?1l>[?25h[>4;m[?1049l[23;0;0t]0;root@aeler-imx7ulpea-ucom: /root@aeler-imx7ulpea-ucom:/# pyu thon3 - service.py 
aws-iotsdk> [2021-05-04 13:53:48.925804] AWS Topic: /d/22f66ba6f37b3ede6115c86389e530eb1d408103

aws-iotsdk> [2021-05-04 13:53:48.952523] [mc] connecting to localhost on port 1883
aws-iotsdk> [2021-05-04 13:53:49.104120] [mc] connect rc: 0
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Initializing edge-triggered epoll
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Using eventfd for cross-thread notifications.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Starting event-loop thread.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [dns] - id=0x162d7a0: Initializing default host resolver with 16 max host entries.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [channel-bootstrap] - id=0x164f998: Initializing client bootstrap with event-loop group 0x16eddb0
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: main loop started
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: default timeout 100000, and max events to process per tick 100
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Certificate and key have been set, setting them up now.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: cannot enable ocsp stapling: OCSP stapling was requested, but is not supported
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Setting ALPN list x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - client=0x17347e8: Initalizing MQTT client
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Creating new connection
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-topic-tree] - tree=0x163db24: Creating new topic tree
aws-iotsdk> [2021-05-04 13:53:53.308811] checking internet connection.
aws-iotsdk> [2021-05-04 13:53:53.734491] [aws] Connecting to <my_endpoint>-ats.iot.us-east-1.amazonaws.com as auto-2C288BE4-A555-4653-92D8-A632727D5B1C ...
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Begin connecting process, switch state to CONNECTING.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: using ping timeout of 3000000000 ns
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: No cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com starting new resolver thread.
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 52.21.198.210 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.147.70.168 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 34.206.245.43 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.158.175.156 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 3.225.200.224 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.145.247.28 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.84.18.235 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 18.207.61.253 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::23ad:4123 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3db:a390 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::34c8:5c29 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::36a3:c4c1 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:b950 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, invoking resolution callback for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com with 2 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3e1:f981:443.
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:53Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3e1:f981 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 52.21.198.210:443.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Scheduling (null) task for future execution at time 2076022788913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:39232
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: no message pool is currently stored in the event-loop local storage, adding 0xb4805c40 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket-handler] - id=0xb4817418: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2081199692246
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Scheduling mqtt_connack_timeout task for future execution at time 2074947672913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 0 to CONNECTED.
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2102147419913
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:56.689408] checking internet connection.
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:57.606042] [aws] Publishing message to topic '/d/22f66ba6f37b3ede6115c86389e530eb1d408103
'
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Scheduling schedule_cross_thread_tasks task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Running schedule_cross_thread_tasks task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b6314460] [mqtt-client] - id=0x163d9b8: Starting publish 1 to topic /d/22f66ba6f37b3ede6115c86389e530eb1d408103

[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Running mqtt_connack_timeout task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2075467825247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:53:58.359515] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::22cd:842b:443.
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::22cd:842b for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.158.175.156:443.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4804268: Scheduling (null) task for future execution at time 2080568943914
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:47820
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket-handler] - id=0xb482ec00: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2085748031247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Scheduling mqtt_connack_timeout task for future execution at time 2079454727914
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
aws-iotsdk> [2021-05-04 13:53:59.568730] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2106715588247
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2077377824580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48353c8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:00.265671] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3417:2507:443.
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3417:2507 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.147.70.168:443.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4803ea8: Scheduling (null) task for future execution at time 2082467830580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:50080
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket-handler] - id=0xb482b660: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2087638712914
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb482d770: Scheduling mqtt_connack_timeout task for future execution at time 2081369652914
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:54:01.464140] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2108610288247
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2079278874581
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb482d770
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482d770: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:02.177324] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.

Additional context

Resolution attempts: modify mqtt builder parameters, registered the device with a different certificate

Device is automatically registered via JITP process and attaching the JTIP_PubSub policy that follows:

bug

Source

qcabrol

👍3

Most helpful comment

+1 - we're seeing this on roughly 4% of devices in the field

We're using Ubuntu 18.04 on arm64 devices with awsiotsdk=1.5.11 and awscrt=0.11.8

JoelHensley on 4 May 2021

👍2

All 8 comments

Hadatko on 4 May 2021

For a bit more of context:
another device with exactly the same build and config is connecting fine. The other still isn't. I wonder if maybe we are hitting a publish size quota or something. Still investigating... In which case I would expect an error though.

qcabrol on 4 May 2021

+1 - we're seeing this on roughly 4% of devices in the field

We're using Ubuntu 18.04 on arm64 devices with awsiotsdk=1.5.11 and awscrt=0.11.8

JoelHensley on 4 May 2021

👍2

another device with exactly the same build and config is connecting fine. The other still isn't.

Every device is connecting with a unique client-id, right?

If a second device connects with the same client-id, the server will accept that new connection and kill the old connection. That will result in the first device reconnecting, which kicks off the second device, so the second device reconnects which kicks off the first device, etc etc

The "client-id" I'm talking about is the client_id argument passed to mqtt_connection_builder functions, or to the awscrt.mqtt.Connection class

graebm on 5 May 2021

Also, WOW that sucks about the rapid reconnect loop sucking down GB of data.

Our implementation does try to use an exponential backoff timer during reconnect, but once the connection is considered "successful" the timer resets to zero. If something happens to kill the connection immediately after it had succeeded, the device will immediately reconnect because that timer has been zeroed out.

Sounds like we might want to wait for the connection to remain "stable" for some period of time before resetting that timer...

graebm on 5 May 2021

👍1

If the server is disconnecting your device, you can find out why. See Monitor AWS IoT using CloudWatch Logs

graebm on 5 May 2021

Hi @graebm
Thanks for the pointers

Unfortunately the INFO level of AWS Logs is not very helpful here:

Regarding the uniqueness of the client id, it is generated at each application start using the python uuid4() from uuid module:
client_id = "auto-%s" % str(uuid.uuid4()).upper()

This means that even if because we were extremely unlucky we had duplicate uuids, the issue should never survive an application restart that will use a new client uuid right ? I will dive a bit into this possibility.

Another thing that I am considering is whether we are hitting some kind of publish packet size quota. As, when a device is disconnected because of a network issue, we bundle many packets together that we published in a single call to limit the overall data usage. We are trying to limit that max bundle size now, see if it affects the behavior. 128KB is the max size.

Question is, are there any quota-related filters on SDK client side before sending a request ? (that would prevent a lot of non-compliant packets and get explicit errors that ease debugging in case of bad request, before connecting and eating any network data).

qcabrol on 5 May 2021

So bottom line is:

we serialize raw bytes in a way that is bloating the packet size a bit (will use binary packets later on)
when connection is lost, we sent packets in a grouped list of byte arrays (up to 1000 messages grouped)

We did the calculations for various scenarios and we were over the quota in this situation.
We limited the max number of messages bundled in one packet and it worked.
This happened because the device lost connection for a while and some alerts logging occurred, creating a data burst above the limit of what the quota is supporting.

IMHO, the SDK should check packets for hard set quota violations and throw an error before trying to send them to ease the debugging of such situations. Here the source was unclear.

You should be able to reproduce the situation by publishing a packet of size >128kB in a try loop.

qcabrol on 5 May 2021

Was this page helpful?

0 / 5 - 0 ratings