Aws-iot-device-sdk-python-v2: AWS_ERROR_MQTT_UNEXPECTED_HANGUP 无限重连循环

创建于 2021-05-04  ·  8评论  ·  资料来源: aws/aws-iot-device-sdk-python-v2

通过将下面的 [ ] 更改为 [x] 来确认它是一个错误：

已知问题

  • [x] 我正在使用 ATS 数据类型端点：端点应该看起来像<prefix>-ats.iot.<region>.amazonaws.com

嗨，AWS 团队，

自 2 天以来，我们对 SDK 面临的一个关键问题感到有些困惑（详情如下）。

描述错误
任务调度程序在主机解析后立即关闭 TLS 通道，导致无限 RECONNECTING <> AWS_ERROR_MQTT_UNEXPECTED_HANGUP 循环（消耗大量数据！！）。
从未建立到端点的 SDK 连接。 TLS 以非常快的速度打开和关闭会话，导致巨大的数据使用量爆发（由于突然出现的问题，我们在现场设备上收到了数百美元的数据漫游账单）。

SDK版本号
用 1.5.12 和 1.5.14 测试并在每种情况下匹配 aws-crt

平台/操作系统/设备
Armhf 设备 (Cortex-A7)
基于ubuntu 20.04的Docker容器

再现（观察到的行为）

观察到 SDK 的标准使用，并且设备在重新连接循环发生之前运行了 10 天。 在那之后，设备永远无法正确重新连接到我们的端点，并且重新连接循环将始终存在。 在正常运行 10 天后（每 3 分钟连接和发布一次），行为在 2021 年 4 月 30 日发生了变化。 证书和 CA 未过期，在新证书下续订设备注册也无济于事。 从 SDK 1.5.12 更新到 1.15.14 也没有帮助。

AWS_KEEP_ALIVE_SECS = 30

    # setup AWS connection
    io.init_logging(io.LogLevel.Debug, 'stderr')
    event_loop_group = io.EventLoopGroup(1)
    host_resolver = io.DefaultHostResolver(event_loop_group)
    client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)

    client_id = "auto-%s" % str(uuid.uuid4()).upper()
    aws_mqtt = mqtt_connection_builder.mtls_from_path(
        endpoint=args.endpoint,
        cert_filepath="%s/device.crt" % (args.provision_location),
        pri_key_filepath="%s/device.key" % (args.provision_location),
        client_bootstrap=client_bootstrap,
        ca_filepath="%s/AmazonRootCA1.pem" % (args.cert_location),
        on_connection_interrupted=aws_on_connection_interrupted,
        on_connection_resumed=aws_on_connection_resumed,z
        client_id=client_id,
        clean_session=False,
        keep_alive_secs=AWS_KEEP_ALIVE_SECS)

预期行为

没有意外挂断。 并且没有无限重连循环。

日志/输出

网络使用导致在部署的漫游设备 (16GB) 上使用了几 GB 的 SIM 数据。 以下是基于 eth0 的连接尝试的网络数据使用情况：
image

调试级别跟踪

[?2004l[?1l>[?25h[>4;m[?1049l[23;0;0t]0;root@aeler-imx7ulpea-ucom: /root@aeler-imx7ulpea-ucom:/# pyu thon3 - service.py 
aws-iotsdk> [2021-05-04 13:53:48.925804] AWS Topic: /d/22f66ba6f37b3ede6115c86389e530eb1d408103

aws-iotsdk> [2021-05-04 13:53:48.952523] [mc] connecting to localhost on port 1883
aws-iotsdk> [2021-05-04 13:53:49.104120] [mc] connect rc: 0
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Initializing edge-triggered epoll
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Using eventfd for cross-thread notifications.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Starting event-loop thread.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [dns] - id=0x162d7a0: Initializing default host resolver with 16 max host entries.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [channel-bootstrap] - id=0x164f998: Initializing client bootstrap with event-loop group 0x16eddb0
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: main loop started
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: default timeout 100000, and max events to process per tick 100
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Certificate and key have been set, setting them up now.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: cannot enable ocsp stapling: OCSP stapling was requested, but is not supported
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Setting ALPN list x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - client=0x17347e8: Initalizing MQTT client
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Creating new connection
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-topic-tree] - tree=0x163db24: Creating new topic tree
aws-iotsdk> [2021-05-04 13:53:53.308811] checking internet connection.
aws-iotsdk> [2021-05-04 13:53:53.734491] [aws] Connecting to <my_endpoint>-ats.iot.us-east-1.amazonaws.com as auto-2C288BE4-A555-4653-92D8-A632727D5B1C ...
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Begin connecting process, switch state to CONNECTING.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: using ping timeout of 3000000000 ns
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: No cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com starting new resolver thread.
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 52.21.198.210 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.147.70.168 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 34.206.245.43 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.158.175.156 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 3.225.200.224 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.145.247.28 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.84.18.235 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 18.207.61.253 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::23ad:4123 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3db:a390 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::34c8:5c29 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::36a3:c4c1 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:b950 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, invoking resolution callback for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com with 2 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3e1:f981:443.
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:53Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3e1:f981 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 52.21.198.210:443.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Scheduling (null) task for future execution at time 2076022788913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:39232
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: no message pool is currently stored in the event-loop local storage, adding 0xb4805c40 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket-handler] - id=0xb4817418: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2081199692246
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Scheduling mqtt_connack_timeout task for future execution at time 2074947672913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 0 to CONNECTED.
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2102147419913
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:56.689408] checking internet connection.
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:57.606042] [aws] Publishing message to topic '/d/22f66ba6f37b3ede6115c86389e530eb1d408103
'
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Scheduling schedule_cross_thread_tasks task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Running schedule_cross_thread_tasks task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b6314460] [mqtt-client] - id=0x163d9b8: Starting publish 1 to topic /d/22f66ba6f37b3ede6115c86389e530eb1d408103

[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Running mqtt_connack_timeout task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2075467825247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:53:58.359515] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::22cd:842b:443.
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::22cd:842b for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.158.175.156:443.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4804268: Scheduling (null) task for future execution at time 2080568943914
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:47820
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket-handler] - id=0xb482ec00: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2085748031247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Scheduling mqtt_connack_timeout task for future execution at time 2079454727914
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
aws-iotsdk> [2021-05-04 13:53:59.568730] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2106715588247
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2077377824580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48353c8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:00.265671] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3417:2507:443.
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3417:2507 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.147.70.168:443.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4803ea8: Scheduling (null) task for future execution at time 2082467830580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:50080
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket-handler] - id=0xb482b660: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2087638712914
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb482d770: Scheduling mqtt_connack_timeout task for future execution at time 2081369652914
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:54:01.464140] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2108610288247
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2079278874581
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb482d770
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482d770: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:02.177324] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.

附加上下文

解决尝试：修改 mqtt builder 参数，用不同的证书注册设备

设备通过 JITP 过程自动注册并附加以下 JTIP_PubSub 策略：
image

picture qcabrol
👍3

最有用的评论

+1 - 我们在现场大约 4% 的设备上看到了这一点

我们在 awsiotsdk=1.5.11 和 awscrt=0.11.8 的 arm64 设备上使用 Ubuntu 18.04

picture JoelHensley 于 2021-05-04
👍2

所有8条评论

+1

Hadatko picture Hadatko 于 2021-05-04

有关更多上下文：
另一个具有完全相同构建和配置的设备连接正常。 另一个仍然不是。 我想知道我们是否达到了发布大小配额或其他什么。 仍在调查......在这种情况下，我预计会出现错误。

qcabrol picture qcabrol 于 2021-05-04

+1 - 我们在现场大约 4% 的设备上看到了这一点

我们在 awsiotsdk=1.5.11 和 awscrt=0.11.8 的 arm64 设备上使用 Ubuntu 18.04

JoelHensley picture JoelHensley 于 2021-05-04
👍2

另一个具有完全相同构建和配置的设备连接正常。 另一个仍然不是。

每个设备都使用唯一的客户端 ID 连接，对吗？

如果第二个设备使用相同的客户端 ID 连接，服务器将接受该新连接并终止旧连接。 这将导致第一个设备重新连接，启动第二个设备，因此第二个设备重新连接，启动第一个设备，依此类推

我所说的“客户端 ID”是传递给mqtt_connection_builder函数或awscrt.mqtt.Connection类的client_id参数

graebm picture graebm 于 2021-05-05

此外，WOW 令人讨厌的是快速重新连接循环吸收了 GB 的数据。

我们的实现确实尝试在重新连接期间使用指数退避计时器，但是一旦连接被认为“成功”，计时器就会重置为零。 如果连接成功后立即终止连接，设备将立即重新连接，因为该计时器已清零。

听起来我们可能希望在重置该计时器之前等待连接保持“稳定”一段时间......

graebm picture graebm 于 2021-05-05
👍1

如果服务器与您的设备断开连接，您可以找出原因。 请参阅使用 CloudWatch Logs 监控 AWS IoT

graebm picture graebm 于 2021-05-05

嗨@graebm
谢谢指点

不幸的是，AWS 日志的 INFO 级别在这里不是很有帮助：
image

关于客户端 id 的唯一性，它是在每个应用程序启动时使用 uuid 模块中的 python uuid4() 生成的：
client_id = "auto-%s" % str(uuid.uuid4()).upper()

这意味着即使因为我们非常不走运我们有重复的 uuid，问题也不应该在使用新客户端 uuid 的应用程序重新启动后继续存在，对吗？ 我将深入探讨这种可能性。

我正在考虑的另一件事是我们是否达到了某种发布数据包大小配额。 因为，当设备由于网络问题而断开连接时，我们将在一次调用中发布的许多数据包捆绑在一起，以限制整体数据使用。 我们现在正在尝试限制最大包大小，看看它是否会影响行为。 128KB 是最大大小。

问题是，在发送请求之前，SDK 客户端是否有任何与配额相关的过滤器？ （这将防止大量不合规的数据包，并在连接和食用任何网络数据之前，获得显式错误，从而在出现错误请求时简化调试）。

qcabrol picture qcabrol 于 2021-05-05

所以底线是：

  • 我们以稍微膨胀数据包大小的方式序列化原始字节（稍后将使用二进制数据包）
  • 当连接丢失时，我们在字节数组的分组列表中发送数据包（最多 1000 条消息分组）

我们对各种场景进行了计算，在这种情况下我们超过了配额。
我们限制了捆绑在一个数据包中的最大消息数，它起作用了。
发生这种情况是因为设备失去了一段时间的连接，并且发生了一些警报日志记录，创建了超出配额支持限制的数据突发。

恕我直言，SDK 应该检查数据包是否违反硬设置配额并在尝试发送它们之前抛出错误，以简化此类情况的调试。 这里的来源不清楚。

您应该能够通过在 try 循环中发布大小大于 128kB 的数据包来重现这种情况。

qcabrol picture qcabrol 于 2021-05-05
此页面是否有帮助？
0 / 5 - 0 等级

相关问题

supertick picture supertick  ·  7评论
GauravPatni picture GauravPatni  ·  4评论
Sanrro10 picture Sanrro10  ·  16评论
mkozjak picture mkozjak  ·  17评论
satay99 picture satay99  ·  6评论