Lorawan-stack: Downlinks rejected by gateways for certain TX powers

Error seems to occur with Lora-net/packet_forwarder. Relevant code: https://github.com/Lora-net/packet_forwarder/blob/master/lora_pkt_fwd/src/lora_pkt_fwd.c#L2517-L2527

With the gateway configs that have been distributed through TTN (https://github.com/TheThingsNetwork/gateway-conf/), the following TX powers are accepted by gateways:

-6, -3, 0, 3, 6, 10, 11, 12, 13, 14, 16, 20, 23, 25, 26, 27

In The Things Stack we use the same list when we build up the dynamic config for gateways: https://github.com/TheThingsNetwork/lorawan-stack/blob/master/pkg/pfconfig/shared/shared.go#L259-L276

This means that the _ERROR: Packet REJECTED, unsupported RF power for TX - 11_ is just a misconfiguration of the gateway, but the other reports are valid.

I suspect that these are caused by a v3 Gateway Server sending an unsupported TX power value. We need to make sure that the GS only sends TX power values to UDP gateways if they're in the list of supported TX powers.

Any recent updates or insights on this?

Can you please add a report to https://status.thethings.network/ ? Thanks!

https://status.thethings.network/ is meant for network incidents. That's not the place for this. We'll release a new update where the server will check for supported TX Power before sending.

Hi Kirshna,

Can you be a little more precise when this update will be released? There are many people waiting for this fix.

By the way, what to you mean with "will check for supported TX Power before sending"? Will the server ensure the TX power is "standard" or will it drop the packet?

Thanks for your help

https://status.thethings.network/ is meant for network incidents.

Just to express that I feel that's very rigid. This _is_ an operational problem for some, right? At least for some users devices drop to SF12 due to not receiving ADR downlinks, and others suddenly see OTAA fail. Without access to a gateway log they're in the dark about the cause, even more so as things were working fine until recently something was changed for the routing of the V2 traffic. (Or something like that; there is no clear communication about those changes either.)

@avbentem: This would not be classified as an operational issue since it's related to software behaviour and not connectivity/throughput/availability etc. However I do understand that this is important to be fixed. You can expect a fix in the coming days.
PS: You're completely right. This change wasn't communicated properly.

This change wasn't communicated properly.

Aside: it's never too late to at least fix that? It's completely unclear to me what changed, and when, and whether people should look at V2 or V3 when debugging problems, @KrishnaIyer and @htdvisser.

I very much welcome, say, a forum post or something on https://www.thethingsnetwork.org/updates about this change, and future changes. Thanks!

We've just deployed a patch that should revert the behaviour. I'll wait for some feedback before closing this.

Thanks Krishna!
Can you give us some details what was fixed? Not to blame anyone, just because that would help us better understand the issues we had with our gateways and packet forwarders and to make sure the behaviour we observed was caused by this change.

The change basically was this:

make sure that the GS only sends TX power values to UDP gateways if they're in the list of supported TX powers

https://github.com/TheThingsNetwork/lorawan-stack/issues/2106#issuecomment-596502171

In addition to this change, we discovered that the conversion of the TX power in downlinks received from v2 systems wasn't done correctly, resulting in a 3 dBm difference (11 should have been 14; 17 should have been 20 and 24 should have been 27). This was also fixed.

Thanks for the details, "_24 should have been 27_" is exactly what caused our problems. With knowing this detail we don't need to debug the GW on site any further.

I was not able to reproduce the problem anymore on my side. Thanks for the update!

Thanks, @htdvisser and @KrishnaIyer. Since this is somehow related to connecting V2 and V3: can you please document somewhere how things are connected nowadays, for the community network?

Apparently, as was briefly announced during the late January 2020 Conference, traffic of gateways that were set up in V2 is being routed through V3? Maybe I missed something, but it's really, really unclear to me what components of V2 are still used, and what is now being delegated to V3. Also, it's unclear when the changes were made.

(Like: the community network still uses V2 TTN Console, and that works fine. I also assume that ADR is still being handled by V2, as https://github.com/TheThingsNetwork/ttn/issues/767 is still being reported for OTAA devices that joined prior to October 2019. But given that the issue above was fixed in V3, something is done by V3 too? Knowing when things changed might help future debugging.)

@dermatthias

we don't need to debug the GW on site any further

Note that it might still be a good idea to fix the gateway's side too, to implement a fallback mechanism for future changes or for future usage of V3? I am not sure; see some comments from Slack in https://www.thethingsnetwork.org/forum/t/error-packet-rejected-unsupported-rf-power-for-tx-24/34833/14

I can't see how this has fixed the problem when someone has (for example) a 5dBi antenna on their gateway. If they leave the Antenna Gain in the Config file as 0dBi then they will exceed the maximum legal transmit power and if they change the config file to 5dBi then there will be cases where the packet will be rejected by the gateway.

reply to @avbentem:

Since our community network clusters all look different, and things are changing all the time, it would cost us too much time to maintain an up-to-date overview of how things are connected in each region.

In the image below you'll see how gateways can connect to a v2 network. The old situation (yellow) will slowly be migrated to the new situation (green). We do this by first routing a small percentage of traffic for a given gateway protocol through the new components, and slowly increasing that percentage. The same for the other gateway protocols.

As @johanstokking mentioned during his presentation at the conference, many community network gateways already connect through a v3 gateway server, and we are slowly connecting more gateways through v3 Gateway Servers over time.

Apart from these Gateway Servers the entire Public Community Network still runs v2 components.

reply to @tonysmith55:

This hasn't. It just changes the behavior back to what it was before. Yes, it is (and always will be) possible that gateways exceed the maximum legal transmit power if these gatways haven't been correctly configured by their owners.

I hope this answers your questions. Since we're going a bit off-topic here, and since this issue is now confirmed to be resolved, I'll now close the issue.