Lorawan-stack: C 类消息间歇性地无法调度，绝对时间无效

创建于 2020-11-17 · 15评论 · 资料来源: TheThingsNetwork/lorawan-stack

概括

当间隔很近时，C 类消息间歇性地无法调度“在应用程序下行链路中设置的绝对时间无效”。

发送到同一设备的消息之间是否有最短时间？理想情况下，我们会将它们安排为相距 130 毫秒，但已尝试将间距增加到 1000 毫秒以缓解该问题

重现步骤

在当前时间安排下行链路 + 7 秒到网关 X
在当前时间安排下行链路 + 8 秒到网关 X
等待10秒，观察mqtt主题DowlinkFailed
如果需要，循环运行以增加重现错误的机会

我们大约每 15 个下行链接就会看到这一点

你现在看到了什么？

17/11/2020 14:09:30.783 +1300   Sending TTN V3 multicast. topic: v3/halter/devices/2704aee7-7c77-4c99-8d26-cf110b1a90a7/down/push. Payload: {"downlinks":[{"f_port":1,"frm_payload":"ChQKEgoQEg4IAxIFDbQiy10YASDjAQ==","priority":"ABOVE_NORMAL","class_b_c":{"absolute_time":"2020-11-17T01:09:37.000Z","gateways":[{"gateway_ids":{"gateway_id":"eui-00800000a0003582"}}]}}]}
17/11/2020 14:09:30.883 +1300   Sending TTN V3 multicast. topic: v3/halter/devices/2704aee7-7c77-4c99-8d26-cf110b1a90a7/down/push. Payload: {"downlinks":[{"f_port":1,"frm_payload":"ChQKEgoQEg4IAxIFDbQiy10YASDjAQ==","priority":"ABOVE_NORMAL","class_b_c":{"absolute_time":"2020-11-17T01:09:38.000Z","gateways":[{"gateway_ids":{"gateway_id":"eui-00800000a0005314"}}]}}]}
17/11/2020 14:09:37.068 +1300   Serial 2704aee7-7c77-4c99-8d26-cf110b1a90a7: Receiving DownlinkFailed. Payload: {"end_device_ids":{"device_id":"2704aee7-7c77-4c99-8d26-cf110b1a90a7","application_ids":{"application_id":"halter"},"dev_addr":"01415A3E"},"correlation_ids":["as:downlink:01EQ9VZT44AARKQ3V190WBSXZA","as:up:01EQ9W005A64G2W1Y3GQA9S3SR"],"received_at":"2020-11-17T01:09:37.066774351Z","downlink_failed":{"downlink":{"f_port":1,"f_cnt":56,"frm_payload":"ChQKEgoQEg4IAxIFDbQiy10YASDjAQ==","class_b_c":{"gateways":[{"gateway_ids":{"gateway_id":"eui-00800000a0005314"}}],"absolute_time":"2020-11-17T01:09:38Z"},"priority":"ABOVE_NORMAL","correlation_ids":["as:downlink:01EQ9VZT44AARKQ3V190WBSXZA"]},"error":{"namespace":"pkg/networkserver","name":"absolute_time","message_format":"invalid absolute time set in application downlink","code":3}}}
17/11/2020 14:09:37.068 +1300   Serial 2704aee7-7c77-4c99-8d26-cf110b1a90a7: Receiving DownlinkFailed. Payload: {"end_device_ids":{"device_id":"2704aee7-7c77-4c99-8d26-cf110b1a90a7","application_ids":{"application_id":"halter"},"dev_addr":"01415A3E"},"correlation_ids":["as:downlink:01EQ9VZT44AARKQ3V190WBSXZA","as:up:01EQ9W005A64G2W1Y3GQA9S3SR"],"received_at":"2020-11-17T01:09:37.066774351Z","downlink_failed":{"downlink":{"f_port":1,"f_cnt":56,"frm_payload":"ChQKEgoQEg4IAxIFDbQiy10YASDjAQ==","class_b_c":{"gateways":[{"gateway_ids":{"gateway_id":"eui-00800000a0005314"}}],"absolute_time":"2020-11-17T01:09:38Z"},"priority":"ABOVE_NORMAL","correlation_ids":["as:downlink:01EQ9VZT44AARKQ3V190WBSXZA"]},"error":{"namespace":"pkg/networkserver","name":"absolute_time","message_format":"invalid absolute time set in application downlink","code":3}}}

你想看什么？

没有从网关发出的下行链路故障和消息

环境

LoRaWAN 的事物堆栈：ttn-lw-stack
版本：3.9.4
构建日期：2020-09-23T09:56:19Z
Git 提交：c4be55c
转版本：go1.15.2
操作系统/架构：linux/amd64

您建议如何实施？

确定绝对时间无效的原因并解决是否有错误

你打算如何测试这个？

很高兴在我们的开发环境中测试 PR

你能自己做这个并提交一个拉取请求吗？

@rvolosatovs ？

bug gateway server

资料来源

virtualguy

👍3

所有15条评论

@virtualguy这个问题持续存在吗？

数据速率是多少？您在哪个地区？

当您说要以 130 毫秒的间隔进行传输时，您是否考虑了播出时间？

您能否通过$ ttn-lw-cli events ...订阅网关和终端设备事件，并粘贴网关服务器报告的有关调度失败原因的确切错误消息？

johanstokking 于 2020-12-23

@virtualguy我添加了一些额外的调试行。选择以下任何一项：

应用补丁调查-3487.txt
摘樱桃https://github.com/TheThingsNetwork/lorawan-stack/commit/f9bbda5db5090dd7eb7bc2d798c92bed82efc2dd
退房https://github.com/TheThingsNetwork/lorawan-stack/tree/investigate/3487-abs-downlink-timing （基于v3.10.7 ）

请按#3487 grep 输出并复制到这里。它也是转到stdout的唯一输出（因为日志转到stderr ）。

如果在ScheduleAt说 RTT 太少，您可能需要增加TTN_LW_EXP_RTT_TTL （持续时间，例如6h 6 小时，默认为30m 30 分钟）和/或将TTN_LW_EXP_SCHEDULE_MIN_RTT_COUNT到 3 或其他（默认为 5）。这些是临时功能标志。

抄送@ymgupta

johanstokking 于 2021-01-22

👍1

仅供参考，在我们从您的补丁中收集日志之前，我们在 #3736 上构建了一些令人不安的运行二进制文件@johanstokking

virtualguy 于 2021-02-03

👍1

@virtualguy @johanstokking以下是运行修补版本的一些日志： https : //gist.github.com/kurtmc/75f4ecf93c2f7a1ee8373d3a9c7f181a

kurtmc 于 2021-02-04

非常感谢。为延迟响应道歉。

这将是非常有帮助的。为了找到根本原因，我添加了更多日志条目。

您可以使用https://github.com/TheThingsNetwork/lorawan-stack/tree/investigate/3487-abs-downlink-timing再次运行这个新版本

如果你想在 v3.10.x 上挑选，挑选 50f56055ba2c0172c002784d0ef22f140b60903c 和 d1e5305d2363aa8ce8dc485b36c9977857da6b6

同样对于输出，我需要从头开始的完整跟踪。

请注意，我们现在正在打印网关 ID（或 EUI）。如果这是敏感信息，请将其编辑为另一个有意义的值或通过电子邮件发送日志。

johanstokking 于 2021-02-10

@kurtmc @virtualguy让我知道我们如何帮助这个设置。如果我需要向您发送 Docker 映像的二进制文件，请告诉我。

johanstokking 于 2021-02-12

@johanstokking更新日志： https : //gist.github.com/kurtmc/041dd593d24fd9f01784e56ec1deb325

kurtmc 于 2021-02-14

谢谢@kurtmc。我没有看到此处出现“无绝对时间”错误，仅在开始时出现，但这是正常的。所以在这里，一切都按预期进行，对吗？

@adriansmares使用https://github.com/TheThingsNetwork/lorawan-stack/pull/3794修复同步的竞赛实际上发生在这里。所以这是真的，请参阅：

"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: median is 30.900256ms"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: median is 30.900256ms"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: relative time downlink"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: relative time downlink"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: scheduled"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 1 ScheduleAt: scheduled"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 Record: 30.629582ms at 2021-02-14 21:58:36.014795727 +0000 UTC m=+86.734546258"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 Stats: sorted items: [{30446978 {13835767378842655859 75767631763 0x3a7a1c0}} {30504816 {13835767372765423622 70132850448 0x3a7a1c0}} {30639590 {13835767387106897922 83515681039 0x3a7a1c0}} {30830194 {13835767336362635015 36237283864 0x3a7a1c0}} {30834436 {13835767341445058316 40950998050 0x3a7a1c0}} {30835582 {13835767336764530097 36639178949 0x3a7a1c0}} {30842181 {13835767341958726238 41464666012 0x3a7a1c0}} {30845267 {13835767320071738555 21052514770 0x3a7a1c0}} {30847416 {13835767323811969378 24571520108 0x3a7a1c0}} {30889585 {13835767369324628633 66913280952 0x3a7a1c0}} {30910927 {13835767324288677194 24974486184 0x3a7a1c0}} {30943729 {13835767344594801745 43879516001 0x3a7a1c0}} {30952793 {13835767349113727476 48103474433 0x3a7a1c0}} {30963569 {13835767326445257112 26983582371 0x3a7a1c0}} {30993783 {13835767329486798315 29803898110 0x3a7a1c0}} {31018584 {13835767328128331563 28592914998 0x3a7a1c0}} {31045641 {13835767318515021603 19643281462 0x3a7a1c0}} {31073394 {13835767334606151403 34628283902 0x3a7a1c0}} {31090774 {13835767382891657128 79595407630 0x3a7a1c0}} {31249183 {13835767338921164328 38648329532 0x3a7a1c0}}]"
"1613339916024","15/02/2021 10:58:36.024 +1300","stack_1  | #3487 Stats: sorted items: [{30446978 {13835767378842655859 75767631763 0x3a7a1c0}} {30504816 {13835767372765423622 70132850448 0x3a7a1c0}} {30639590 {13835767387106897922 83515681039 0x3a7a1c0}} {30830194 {13835767336362635015 36237283864 0x3a7a1c0}} {30834436 {13835767341445058316 40950998050 0x3a7a1c0}} {30835582 {13835767336764530097 36639178949 0x3a7a1c0}} {30842181 {13835767341958726238 41464666012 0x3a7a1c0}} {30845267 {13835767320071738555 21052514770 0x3a7a1c0}} {30847416 {13835767323811969378 24571520108 0x3a7a1c0}} {30889585 {13835767369324628633 66913280952 0x3a7a1c0}} {30910927 {13835767324288677194 24974486184 0x3a7a1c0}} {30943729 {13835767344594801745 43879516001 0x3a7a1c0}} {30952793 {13835767349113727476 48103474433 0x3a7a1c0}} {30963569 {13835767326445257112 26983582371 0x3a7a1c0}} {30993783 {13835767329486798315 29803898110 0x3a7a1c0}} {31018584 {13835767328128331563 28592914998 0x3a7a1c0}} {31045641 {13835767318515021603 19643281462 0x3a7a1c0}} {31073394 {13835767334606151403 34628283902 0x3a7a1c0}} {31090774 {13835767382891657128 79595407630 0x3a7a1c0}} {31249183 {13835767338921164328 38648329532 0x3a7a1c0}}]"

请注意，由于底层刷新，这些语句是乱序的；看起来简短的语句会立即刷新，而较长的语句会延迟。

我认为问题在于，当(*rtts).Record()释放写锁时，两个并发(*rtts).Stats()调用都会获得一个读锁，因此两个并发(*Scheduler).ScheduleAt()调用变得完全同步。由于那些正在获取（另一个）读锁，它们同时发生，导致Scheduler状态损坏。

这已通过https://github.com/TheThingsNetwork/lorawan-stack/pull/3794修复