Aws-iot-device-sdk-python-v2: AWS_ERROR_MQTT_UNEXPECTED_HANGUP Bucle de reconexión infinito
Confirme cambiando [] a [x] a continuación para asegurarse de que se trata de un error:
- [x] Busqué problemas similares anteriores y no encontré ninguna solución.
Problema conocido
- [x] Estoy usando un punto final de tipo de datos ATS: el punto final debería verse como
<prefix>-ats.iot.<region>.amazonaws.com
Hola, equipo de AWS:
estamos un poco atascados con un problema crítico al que nos enfrentamos con el SDK desde hace 2 días (detalles a continuación).
Describe el error
Cierre del canal TLS por parte del programador de tareas inmediatamente después de la resolución del host, lo que da como resultado un bucle infinito RECONNECTING <> AWS_ERROR_MQTT_UNEXPECTED_HANGUP (¡¡Consumiendo muchos datos !!).
La conexión del SDK al punto final nunca se establece. La apertura y el cierre de sesiones de TLS a un ritmo muy rápido, provocando una tremenda explosión en el uso de datos (obtuvimos varios cientos de $ de factura de roaming de datos en un dispositivo de campo debido al problema repentino).
Número de versión del SDK
Probado con 1.5.12 y 1.5.14 y aws-crt coincidente en cada caso
Plataforma / SO / Dispositivo
Dispositivo Armhf (Cortex-A7)
Contenedor de Docker basado en ubuntu 20.04
Reproducir (comportamiento observado)
Se observó el uso estándar del SDK y el dispositivo estuvo funcionando durante 10 días antes de que ocurriera el ciclo de reconexión. Después de eso, el dispositivo nunca pudo volver a conectarse correctamente a nuestro punto final y el bucle de reconexión siempre estaría presente. El comportamiento cambió el 30.04.2021 después de 10 días de funcionamiento normal (conexiones y publicación cada 3 minutos). Los certificados y la CA no han expirado y el registro renovado del dispositivo con un nuevo certificado no ayudó. La actualización de SDK 1.5.12 a 1.15.14 tampoco ayudó.
AWS_KEEP_ALIVE_SECS = 30
# setup AWS connection
io.init_logging(io.LogLevel.Debug, 'stderr')
event_loop_group = io.EventLoopGroup(1)
host_resolver = io.DefaultHostResolver(event_loop_group)
client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)
client_id = "auto-%s" % str(uuid.uuid4()).upper()
aws_mqtt = mqtt_connection_builder.mtls_from_path(
endpoint=args.endpoint,
cert_filepath="%s/device.crt" % (args.provision_location),
pri_key_filepath="%s/device.key" % (args.provision_location),
client_bootstrap=client_bootstrap,
ca_filepath="%s/AmazonRootCA1.pem" % (args.cert_location),
on_connection_interrupted=aws_on_connection_interrupted,
on_connection_resumed=aws_on_connection_resumed,z
client_id=client_id,
clean_session=False,
keep_alive_secs=AWS_KEEP_ALIVE_SECS)
Comportamiento esperado
Sin colgar inesperado. Y sin bucle de reconexión infinito.
Registros / salida
El uso de la red provocó un uso de datos SIM de varios GB en un dispositivo de itinerancia implementado (16 GB). Aquí está el uso de datos de red en un intento de conexión basado en eth0:
Seguimiento del nivel de depuración
[?2004l[?1l>[?25h[>4;m[?1049l[23;0;0t]0;root@aeler-imx7ulpea-ucom: /root@aeler-imx7ulpea-ucom:/# pyu thon3 - service.py
aws-iotsdk> [2021-05-04 13:53:48.925804] AWS Topic: /d/22f66ba6f37b3ede6115c86389e530eb1d408103
aws-iotsdk> [2021-05-04 13:53:48.952523] [mc] connecting to localhost on port 1883
aws-iotsdk> [2021-05-04 13:53:49.104120] [mc] connect rc: 0
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Initializing edge-triggered epoll
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Using eventfd for cross-thread notifications.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [event-loop] - id=0x1684fc8: Starting event-loop thread.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [dns] - id=0x162d7a0: Initializing default host resolver with 16 max host entries.
[INFO] [2021-05-04T13:53:50Z] [b6f3f460] [channel-bootstrap] - id=0x164f998: Initializing client bootstrap with event-loop group 0x16eddb0
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: main loop started
[INFO] [2021-05-04T13:53:50Z] [b51fe460] [event-loop] - id=0x1684fc8: default timeout 100000, and max events to process per tick 100
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Certificate and key have been set, setting them up now.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: cannot enable ocsp stapling: OCSP stapling was requested, but is not supported
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [tls-handler] - ctx: Setting ALPN list x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - client=0x17347e8: Initalizing MQTT client
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Creating new connection
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-topic-tree] - tree=0x163db24: Creating new topic tree
aws-iotsdk> [2021-05-04 13:53:53.308811] checking internet connection.
aws-iotsdk> [2021-05-04 13:53:53.734491] [aws] Connecting to <my_endpoint>-ats.iot.us-east-1.amazonaws.com as auto-2C288BE4-A555-4653-92D8-A632727D5B1C ...
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: Begin connecting process, switch state to CONNECTING.
[INFO] [2021-05-04T13:53:53Z] [b6f3f460] [mqtt-client] - id=0x163d9b8: using ping timeout of 3000000000 ns
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b6f3f460] [dns] - id=0x162d7a0: No cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com starting new resolver thread.
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 52.21.198.210 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.147.70.168 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 34.206.245.43 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.158.175.156 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 3.225.200.224 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.145.247.28 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 54.84.18.235 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 18.207.61.253 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::23ad:4123 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3db:a390 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::34c8:5c29 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::36a3:c4c1 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static: new address resolved 2406:da00:ff00::3e1:b950 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com caching
[DEBUG] [2021-05-04T13:53:53Z] [b467f460] [dns] - static, invoking resolution callback for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com with 2 addresses
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804c88: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3e1:f981:443.
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:53Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3e1:f981 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3e1:f981 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:53Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4804de0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 52.21.198.210:443.
[DEBUG] [2021-05-04T13:53:53Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Scheduling (null) task for future execution at time 2076022788913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:39232
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805628: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4804ef8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel] - id=0xb4806838: no message pool is currently stored in the event-loop local storage, adding 0xb4805c40 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [socket-handler] - id=0xb4817418: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2081199692246
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Scheduling mqtt_connack_timeout task for future execution at time 2074947672913
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:54Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:54Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 0 to CONNECTED.
[DEBUG] [2021-05-04T13:53:55Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2102147419913
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:55Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:56.689408] checking internet connection.
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:56Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:57Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:53:57.606042] [aws] Publishing message to topic '/d/22f66ba6f37b3ede6115c86389e530eb1d408103
'
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Scheduling schedule_cross_thread_tasks task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4806950: Running schedule_cross_thread_tasks task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b6314460] [mqtt-client] - id=0x163d9b8: Starting publish 1 to topic /d/22f66ba6f37b3ede6115c86389e530eb1d408103
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb48349a0: Running mqtt_connack_timeout task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:57Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:58Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2075467825247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4817478: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb4817418 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:53:58.359515] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4834048: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::22cd:842b:443.
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::22cd:842b for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::22cd:842b for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48338c0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.158.175.156:443.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4804268: Scheduling (null) task for future execution at time 2080568943914
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:47820
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb482bd18: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4801cc8: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [socket-handler] - id=0xb482ec00: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2085748031247
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:58Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4803a48: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:53:59Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Scheduling mqtt_connack_timeout task for future execution at time 2079454727914
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
aws-iotsdk> [2021-05-04 13:53:59.568730] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2106715588247
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:53:59Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:00Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2077377824580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482b970: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482ec60: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482ec00 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48353c8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48353c8: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:00.265671] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: Host resolution requested for <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: cached entries found for <my_endpoint>-ats.iot.us-east-1.amazonaws.com returning to caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: destroying channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: releasing bootstrap reference
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482fc00: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 1 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 2406:da00:ff00::3417:2507:443.
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connect failed with error code 101.
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [dns] - id=0x162d7a0: recording failure for record 2406:da00:ff00::3417:2507 for <my_endpoint>-ats.iot.us-east-1.amazonaws.com, moving to bad list
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [dns] - static: purging address 2406:da00:ff00::3417:2507 for host <my_endpoint>-ats.iot.us-east-1.amazonaws.com from the cache due to cache eviction or shutdown
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: is still open, closing...
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[ERROR] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: failed to create socket with error 1049
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb482f5e0: Running attempt_connection task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: initializing with domain 0 and type 0
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[WARN] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: setsockopt() for NO_SIGNAL failed with errno 92. If you are having SIGPIPE signals thrown, you may want to install a signal trap in your application layer.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: beginning connect.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connecting to endpoint 54.147.70.168:443.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4803ea8: Scheduling (null) task for future execution at time 2082467830580
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: connection success
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: local endpoint 10.42.0.177:50080
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket] - id=0xb4805118 fd=10: assigning to event loop 0x1684fc8
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: client connection on socket 0xb4805118 completed with error 0.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: Beginning creation and setup of new channel.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805680: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4835560: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: setup complete, notifying caller.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel] - id=0xb4806838: message pool 0xb4805c40 found in event-loop local storage: using it.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 setup succeeded: bootstrapping.
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [socket-handler] - id=0xb482b660: Socket handler created with max_read_size of 16384
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Scheduling tls_timeout task for future execution at time 2087638712914
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:00Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 52.21.198.210
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.147.70.168
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 34.206.245.43
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.158.175.156
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 3.225.200.224
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alpn protocol negotiated as x-amzn-mqtt-ca
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [tls-handler] - id=0xb48175d0: Remote server name is <my_endpoint>-ats.iot.us-east-1.amazonaws.com
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [channel-bootstrap] - id=0x164f998: tls negotiation result 0 on channel 0xb4806838
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection successfully opened, sending CONNECT packet
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb482d770: Scheduling mqtt_connack_timeout task for future execution at time 2081369652914
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Adding username ?SDK=PythonV2&Version=dev to connection
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.145.247.28
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 54.84.18.235
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 18.207.61.253
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:f981
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3417:2507
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::23ad:4123
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::22cd:842b
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3db:a390
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::34c8:5c29
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::36a3:c4c1
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static: resolved record: 2406:da00:ff00::3e1:b950
[DEBUG] [2021-05-04T13:54:01Z] [b467f460] [dns] - static, resolving host <my_endpoint>-ats.iot.us-east-1.amazonaws.com successful, returned 16 addresses
aws-iotsdk> [2021-05-04 13:54:01.464140] [aws] Connection resumed. return_code: ConnectReturnCode.ACCEPTED session_present: True
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was accepted, switch state from 2 to CONNECTED.
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Scheduling mqtt_outgoing_request_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x163dac8: Scheduling mqtt_ping task for future execution at time 2108610288247
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0x1715368: Running mqtt_outgoing_request_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:01Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Alert code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806978: Running channel_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: beginning shutdown process
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Shutting down read direction with error code 0
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in read dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[INFO] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: sending disconnect message as part of graceful shutdown.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: failed to send courteous disconnect io message
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0x163d9cc shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Scheduling delayed write direction shutdown
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Scheduling s2n_delayed_shutdown task for future execution at time 2079278874581
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48176e0: Running s2n_delayed_shutdown task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [tls-handler] - id=0xb48175d0: Delayed shut down in write direction
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Scheduling socket_written_task task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb48175d0 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [socket] - id=0xb4805118 fd=10: closing
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4805260: Running socket_written_task task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Scheduling socket_handler_close task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48349b0: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482b6c0: Running socket_handler_close task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: handler 0xb482b660 shutdown in write dir completed.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Scheduling (null) task for immediate execution
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb4806850: Running (null) task with <Running> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb48175e8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb48175e8: Running tls_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0xb482d770
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0xb482d770: Running mqtt_connack_timeout task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel] - id=0xb4806838: during shutdown, canceling task 0x163dac8
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [task-scheduler] - id=0x163dac8: Running mqtt_ping task with <Canceled> status
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [channel-bootstrap] - id=0x164f998: channel 0xb4806838 shutdown with error 0.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: connection was unexpected interrupted, switch state to RECONNECTING.
[DEBUG] [2021-05-04T13:54:02Z] [b51fe460] [mqtt-client] - id=0x163d9b8: Connection interrupted, calling callback and attempting reconnect
aws-iotsdk> [2021-05-04 13:54:02.177324] [aws] Connection interrupted. error: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
Contexto adicional
Intentos de resolución: modificar los parámetros del constructor de mqtt, registrar el dispositivo con un certificado diferente
El dispositivo se registra automáticamente a través del proceso JITP y adjunta la política JTIP_PubSub que sigue:
bug
qcabrol
👍3
Todos 8 comentarios
+1
Hadatko
en 4 may. 2021
Para un poco más de contexto:
otro dispositivo con exactamente la misma compilación y configuración se está conectando bien. El otro todavía no lo está. Me pregunto si tal vez estamos alcanzando una cuota de tamaño de publicación o algo así. Aún estoy investigando ... En cuyo caso esperaría un error.
qcabrol
en 4 may. 2021
+1: estamos viendo esto en aproximadamente el 4% de los dispositivos en el campo
Estamos usando Ubuntu 18.04 en dispositivos arm64 con awsiotsdk = 1.5.11 y awscrt = 0.11.8
JoelHensley
en 4 may. 2021
👍2
otro dispositivo con exactamente la misma compilación y configuración se está conectando bien. El otro todavía no lo está.
Cada dispositivo se conecta con una identificación de cliente única, ¿verdad?
Si un segundo dispositivo se conecta con el mismo ID de cliente, el servidor aceptará esa nueva conexión y eliminará la conexión anterior. Eso dará como resultado que el primer dispositivo se vuelva a conectar, lo que inicia el segundo dispositivo, por lo que el segundo dispositivo se vuelve a conectar, lo que inicia el primer dispositivo, etc., etc.
El "client-id" del que estoy hablando es el argumento client_id pasado a las funciones awscrt.mqtt.Connection
graebm
en 5 may. 2021
Además, WOW que apesta sobre el bucle de reconexión rápida que absorbe GB de datos.
Nuestra implementación intenta usar un temporizador de retroceso exponencial durante la reconexión, pero una vez que la conexión se considera "exitosa", el temporizador se restablece a cero. Si sucede algo que interrumpe la conexión inmediatamente después de que se haya realizado correctamente, el dispositivo se volverá a conectar inmediatamente porque ese temporizador se ha puesto a cero.
Parece que es posible que deseemos esperar a que la conexión permanezca "estable" durante un período de tiempo antes de restablecer ese temporizador ...
graebm
en 5 may. 2021
👍1
Si el servidor está desconectando su dispositivo, puede averiguar por qué. Consulte Supervisión de AWS IoT mediante CloudWatch Logs
graebm
en 5 may. 2021
Hola @graebm
Gracias por los consejos
Lamentablemente, el nivel INFO de los registros de AWS no es muy útil aquí:
Con respecto a la unicidad de la identificación del cliente, se genera en cada inicio de la aplicación usando el python uuid4 () del módulo uuid:
client_id = "auto-%s" % str(uuid.uuid4()).upper()
Esto significa que incluso si tuvimos mucha mala suerte de tener uuids duplicados, el problema nunca debería sobrevivir al reinicio de una aplicación que usará un nuevo uuid de cliente, ¿verdad? Me sumergiré un poco en esta posibilidad.
Otra cosa que estoy considerando es si estamos alcanzando algún tipo de cuota de tamaño de paquete de publicación. Como, cuando un dispositivo se desconecta debido a un problema de red, agrupamos muchos paquetes que publicamos en una sola llamada para limitar el uso general de datos. Estamos tratando de limitar ese tamaño máximo de paquete ahora, vea si afecta el comportamiento. 128 KB es el tamaño máximo.
La pregunta es, ¿existen filtros relacionados con las cuotas en el lado del cliente del SDK antes de enviar una solicitud? (eso evitaría una gran cantidad de paquetes no compatibles y obtendría errores explícitos que facilitan la depuración en caso de una solicitud incorrecta, antes de conectarse y consumir cualquier dato de red).
qcabrol
en 5 may. 2021
Entonces, la conclusión es:
- serializamos bytes sin procesar de una manera que aumenta un poco el tamaño del paquete (usaremos paquetes binarios más adelante)
- cuando se pierde la conexión, enviamos paquetes en una lista agrupada de matrices de bytes (hasta 1000 mensajes agrupados)
Hicimos los cálculos para varios escenarios y estábamos por encima de la cuota en esta situación.
Limitamos el número máximo de mensajes agrupados en un paquete y funcionó.
Esto sucedió porque el dispositivo perdió la conexión por un tiempo y se produjeron algunos registros de alertas, creando una ráfaga de datos por encima del límite de lo que admite la cuota.
En mi humilde opinión, el SDK debería comprobar los paquetes en busca de violaciones de la cuota fijadas y arrojar un error antes de intentar enviarlos para facilitar la depuración de tales situaciones. Aquí la fuente no estaba clara.
Debería poder reproducir la situación publicando un paquete de tamaño> 128kB en un ciclo de prueba.
qcabrol
en 5 may. 2021
¿Fue útil esta página
0 / 5 - 0 calificaciones
Temas relacionados
victorct-pronto
·
3Comentarios
GauravPatni
·
4Comentarios
shravan097
·
6Comentarios
satay99
·
6Comentarios
mkozjak
·
17Comentarios
Comentario más útil
+1: estamos viendo esto en aproximadamente el 4% de los dispositivos en el campo
Estamos usando Ubuntu 18.04 en dispositivos arm64 con awsiotsdk = 1.5.11 y awscrt = 0.11.8