Summary:
The add device form (added in #573) is currently not composing fields based on constraints (except for ABP / OTAA selection). For example, it could hide the fields that are only relevant for specific LoRaWAN Versions, or rename fields accordingly (eg. NwkSKey
vs. FNwkSIntKey
).
Why do we need this?
For better UX, avoid faulty inputs
What is already there? What do you see now?
The add device form, with conditional fields based on OTAA/ABP
What is missing? What do you want to see?
More sophisticated checks and constraints applied into the form, preventing faulty inputs
Environment:
Console in browser
How do you propose to implement this?
Likely hook into field change events and compose fields accordingly
Can you do this yourself and submit a Pull Request?
Yes.
The main problem with the current implementation of the device form is that everything is coupled together. This causes
I propose to implement the device form as the multi-step form. This can look something like this:
Such approach addresses all issues mentioned above:
Join EUI
label to App EUI
for lorawan version 1.0.x
.Editing devices can be implemented as a stack of accordions:
Where each accordion expands with a standalone form.
Besides the device form we can use wizard for the application form as well to:
cc @kschiffer @johanstokking @htdvisser
Sounds good to me, especially if we can group fields for components together in a step. That way, we can simply skip/disable steps when a component is not available.
Referencing https://github.com/TheThingsNetwork/lorawan-stack/issues/1234 also; even if JS is available, the user can skip the JS fields.
I came up with such diagram:
Every node is a step
It looks complicated, but it the current implementation has even bigger state space regarding validation/submission/field mask generation/etc.
Consider the diagram as an initial proposition to start the discussion.
I think this is a good start.
And some small things:
DevEUI
is not forbidden for ABP devices (it can optionally be set)NwkKey
, only an AppKey
FNwkSIntKey
is called NwkSKey
(towards the user)NwkKey
or AppKey
Yes, great start.
- Maybe the flow would be better if the Join Server came first.
I agree with this. If the activation mode is OTAA and the cluster JS is enabled, on the JS page users get the choice to enter root keys and store them on the cluster JS. (if they disable that, NS uses JS lookup, just like when there is no JS in the cluster enabled)
multicast bool
and supports_join bool
. There are 3 possible options, since multicast && supports_join
is not valid.frequency_plan
-> frequency_plan_id
resets_f_cnt
is optional, false
by default.
FNwkSIntKey
should be presented as NwkKey
only for <1.1
FNwkSIntKey
is missing in ABP NS settings for >= 1.1multicast
devices only need AppSKey
- NS does not need any keys for multicast
devices to work, only AS@htdvisser
Maybe the flow would be better if the Join Server came first.
Why?
An important thing we haven't really made any progress on yet is pre-filling fields from device templates in the device repository #263. I think that could really simplify the process for production deployments.
For me it seems outside of the scope of this issue
A DevEUI is not forbidden for ABP devices (it can optionally be set)
Do we want to set it for ABP devices as well? I would say the fewer fields we have, the better. However, it is needed we can add it.
FNwkSIntKey is called NwkSKey (towards the user)
So the label should be NwkSKey
for both 1.0.x and 1.1.x ? Here is what we have atm in the console:
- LoRaWAN 1.0.x devices don't have a NwkKey, only an AppKey
- The Application Server does not get the NwkKey or AppKey
Fixed 👌
@johanstokking
If the activation mode is OTAA and the cluster JS is enabled, on the JS page users get the choice to enter root keys and store them on the cluster JS. (if they disable that, NS uses JS lookup, just like when there is no JS in the cluster enabled).
So this is just the matter of allowing users to skip the join server submission step? No requests to js at all?
Regarding https://github.com/TheThingsNetwork/lorawan-stack/issues/1134, could you also specify where do these fields belong in the diagram?
@rvolosatovs
Activation mode" is in payload - in your diagram it corresponds to actually 2 fields - multicast bool and supports_join bool. There are 3 possible options, since multicast && supports_join is not valid.
Just to clarify:
supports_join=true
- OTAAsupports_join=false
- ABPmulticast=true && **no** supports_join
- multicastresets_f_cnt is optional, false by default.
I think it is still fine to show it to the users. Should allow the users to set it for multicast devices?
FNwkSIntKey should be presented as NwkKey only for <1.1
You mean NwkSKey
?
frequency_plan -> frequency_plan_id
FNwkSIntKey is missing in ABP NS settings for >= 1.1
Fixed 👌
Updated diagram:
@htdvisser
Maybe the flow would be better if the Join Server came first.
Why?
Yeah I'm coming back on this; I think it makes more sense to enter MAC versions before entering keys. So I'm supporting the current flow. If MAC version is entered (when NS is enabled), we don't have to ask for NwkKey
since that's 1.1.x.
An important thing we haven't really made any progress on yet is pre-filling fields from device templates in the device repository #263. I think that could really simplify the process for production deployments.
For me it seems outside of the scope of this issue
It is out of scope indeed.
A DevEUI is not forbidden for ABP devices (it can optionally be set)
Do we want to set it for ABP devices as well? I would say the fewer fields we have, the better. However, it is needed we can add it.
Yes, we want to optionally ask for it. The more identification information we have about end devices, the better. It is also required in stateful passive roaming. The reason we don't force users to enter the DevEUI, is because if there is no DevEUI, we don't want users to enter bogus values.
FNwkSIntKey is called NwkSKey (towards the user)
So the label should be
NwkSKey
for both 1.0.x and 1.1.x ? Here is what we have atm in the console:
It's NwkSKey
for 1.0.x and FNwkSIntKey
for 1.1.x.
If the activation mode is OTAA and the cluster JS is enabled, on the JS page users get the choice to enter root keys and store them on the cluster JS. (if they disable that, NS uses JS lookup, just like when there is no JS in the cluster enabled).
So this is just the matter of allowing users to skip the join server submission step? No requests to js at all?
Indeed.
An example is a device featuring Semtech's modem which uses the Semtech Join Server. We only need to know EUIs in that case.
Regarding #1134, could you also specify where do these fields belong in the diagram?
They belong in the JS step; these fields to go JS, if the JS is enabled in the cluster and if the user wants to provision the devices on the JS.
These fields are optional.
Just to clarify:
supports_join=true
- OTAAsupports_join=false
- ABPmulticast=true && **no** supports_join
- multicast
Is this correct?
Strictly:
supports_join
!supports_join && !multicast
multicast
Invalid is supports_join && multicast
, but this is (or should be) validated at NS.
resets_f_cnt is optional, false by default.
I think it is still fine to show it to the users. Should allow the users to set it for multicast devices?
No, this is not for multicast. resets_f_cnt
refers to uplink, but there ain't uplink in multicast.
FNwkSIntKey should be presented as NwkKey only for <1.1
You mean
NwkSKey
?
Should be NwkSKey
indeed.
@bafonins please see @johanstokking 's answer.
In regards to multicast
- device address is required as well
DevEUI
to abp/multicast devicesDevice Address
to multicast devicesDevice Address still missing from multicast
device Network Server settings
Device Address still missing from multicast device Network Server settings
Updated 👌
Multicast can be 1.0.x and 1.1.x. Entering the session information (DevAddr
and keys) is the same for multicast as for ABP.
Also resets_join_nonces
can be set in JS with 1.1.x
Splitting up the device creation is definitely needed and a nice way to get the flow and implementation closer to the requirements of the backend, while reducing complexity for the user.
Some thoughts and challenges that I see:
Unnecessary complexity in the js sdk for batched creation/update/deletion as well as creation rollback logic
I think the functionality there in the SDK for splitting and merging device requests is still pretty valuable, even if we don't use it in this case
@johanstokking for multicast devices NS only needs the SNwkSIntKey
in 1.1 for MIC calculation. FNwkSIntKey
and NwkSEncKey
, in fact, should not be allowed to set IMO.
- We should add functionality to abort the whole flow, resulting in a deletion of whatever registry entries have already been created.
- Likewise, going to the previous step needs to put the form in "update mode" (should be relatively easy as the endpoints are the same, except for IS)
I don't think we should be creating anything before hitting Finish or something. Going back and forth in the wizard and closing the browser window should not result in half-created devices.
- One problem I see is the Application Server step being very shallow (will this likely change later?) and adding the payload format options there is also not really in line with the user's needs while creating a device.
We will add configuration for application packages here (i.e. remote multicast setup, fragmented data block transfer, Semtech modem options, etc). So yes it shallow now but it will be extended.
- A solution might be to merge the AS and JS step given that both are pretty straightforward and not interdependent. I realize this goes against the separation by component but I think this would simplify the whole flow with a reasonable abstraction. Especially given that we don't communicate any connection to the respective stack components within the steps.
For our future selves I would recommend keeping them separate. Also combining them makes rendering stuff on those pages conditional on component availability which adds complexity to an already complex flow.
Depending on the stack config we might end up with a wizard that only has one or two steps, which is an anti-pattern for wizards.
- For the case of just one step, we can simply hide/remove the wizard aspect
- For two steps, I think we have to live with the issue 🤷♂
It should be considered that the wizard solution will increase the _time-to-complete_ for the user story quite a bit
- This could become an issue in situations where a lot of devices need to be created by hand, though we will introduce batch creation features for this use case and the CLI/scripting can help with such situations as well
- Still, consider the difference with the V2 console device creation and how users might perceive this change
The separation of components and the flexible deployment scenarios is one of the key changes compared to V2, so it makes total sense that this is effective in V3 Console.
Indeed, for creating large amounts of devices, people should use APIs and CLI anyway.
There is no scenario with one step, it's always at least two steps.
How about rendering tabs instead of pages? That way it's still one page but things are easily accessible without going back and forth in steps.
@johanstokking
I don't think we should be creating anything before hitting Finish or something.
If we make the actual request only at the last step, then it means that we would make 3-4 request. How do we handle errors returned by different components? Handling this, navigating the user to the errored step/resetting store/etc. is complex. That is why I suggest submitting values at every step, because every successive step can rely on submitted values as valid.
Going back and forth in the wizard and closing the browser window should not result in half-created devices.
I am against allowing users to edit data in the wizard (for the steps that result in submission). Mb, only optional fields that are stored in a single compoent (and no other components need those), say name
, description
. Otherwise handling this becomes quite complicated.
@kschiffer
Depending on the stack config we might end up with a wizard that only has one or two steps, which is an anti-pattern for wizards.
Well, we are open to alternative solutions. Any propositions?
If we make the actual request only at the last step, then it means that we would make 3-4 request. How do we handle errors returned by different components? Handling this, navigating the user to the errored step/resetting store/etc. is complex. That is why I suggest submitting values at every step, because every successive step can rely on submitted values as valid.
OK. That's fine, as long as the Console can handle devices that are in ER but cannot be found in JS/NS/AS because that step failed or the user closed the tab or something.
I am against allowing users to edit data in the wizard (for the steps that result in submission). Mb, only optional fields that are stored in a single compoent (and no other components need those), say
name
,description
. Otherwise handling this becomes quite complicated.
What do you mean?
Note that that AS, for example, doesn't need any fields, it just needs the device to exist. So if the AS is there, even if the user doesn't enter any values for AS, it should still create an (empty) device in AS.
What do you mean?
Sorry. This was towards @kschiffer suggestion:
Likewise, going to the previous step needs to put the form in "update mode" (should be relatively easy as the endpoints are the same, except for IS)
Going back and forth in the wizard and closing the browser window should not result in half-created devices.
I would consider this as a future improvement. For now we can just show a notification to the user on uncompleted flow. Later, the user can edit/delete the device in the general settings page.
Well, we are open to alternative solutions. Any propositions?
Well, I think that given the _edge-casiness_ of this particular situation, it is ok to live with this issue. Your wording suggests that I should not word concerns without proposed solutions, but I like to give others the opportunity to come up with some if I can't immediately find any.
I am against allowing users to edit data in the wizard (for the steps that result in submission). Mb, only optional fields that are stored in a single compoent (and no other components need those), say
name
,description
. Otherwise handling this becomes quite complicated.
This would again break with the best practices for the wizard pattern. Users will likely want to revise or simply inspect earlier fields, especially given that these are interdependent. I'm okay with not including this functionality initially, but it should be added eventually (as in: kept track of in an issue).
Otherwise handling this becomes quite complicated.
Generally speaking, the necessity to implement complex frontend logic should not affect our commitment to UX.
I would consider this as a future improvement. For now, we can just show a notification to the user on uncompleted flow. Later, the user can edit/delete the device in the general settings page.
Not optimal but acceptable for me, as long as we will enhance this eventually.
As discussed offline, here's the initial proposal;
ids
name
description
attributes
lorawan_version
lorawan_phy_version
LoRaWAN settings
lorawan_version
(required)lorawan_phy_version
(required)supports_join
set if Activation Mode is OTAA (required)multicast
set if Activation Mode is multicastsupports_class_b
supports_class_c
frequency_plan_id
(required)mac_settings.supports_32_bit_f_cnt
mac_settings.factory_preset_frequencies
mac_settings.ping_slot_data_rate_index.value
mac_settings.ping_slot_frequency
mac_settings.ping_slot_periodicity.value
mac_settings.rx2_data_rate_index.value
min_frequency
max_frequency
session.dev_addr
session.keys.session_key_id
session.keys.f_nwk_s_int_key.key
- requiredsession.keys.s_nwk_s_int_key.key
(if LW >= 1.1.0) - requiredsession.keys.nwk_s_enc_key.key
(if LW >= 1.1.0) - requiredsession.keys.app_s_key.key
- requiredsession.last_conf_f_cnt_down
session.last_n_f_cnt_down
mac_settings.resets_f_cnt
session.last_f_cnt_up
- this is required for securityIf NS in cluster: Fields if activation mode is ABP or OTAA
mac_settings.adr_margin
mac_settings.class_b_timeout
mac_settings.class_c_timeout
mac_settings.desired_adr_ack_delay_exponent.value
mac_settings.desired_adr_ack_limit_exponent.value
mac_settings.desired_max_duty_cycle.value
mac_settings.desired_rx1_data_rate_offset
mac_settings.desired_rx1_delay.value
mac_settings.desired_rx2_data_rate_index.value
mac_settings.desired_rx2_frequency
mac_settings.max_duty_cycle.value
mac_settings.rx1_data_rate_offset
mac_settings.rx1_delay.value
mac_settings.status_count_periodicity
mac_settings.status_time_periodicity
mac_settings.supports_32_bit_f_cnt
mac_settings.use_adr
This sets the device in NS and AS (if in cluster). If we have an OTAA device, we don't have anything to set in AS at this point, but I would still make the call for consistency
payload_formatters
root_keys.root_key_id
root_keys.app_key.key
root_keys.nwk_key.key
(if LW >= 1.1.0)resets_join_nonces
home_net_id
network_server_kek_label
application_server_kek_label
application_server_id
Here, I would go with a tabbed approach, basically with wizard steps as tabs.
Key here is that;
ids
, supports_join
, multicast
are read-onlysupports_join
!supports_join && !multicast
!supports_join && multicast
(although multicast
should be enough)Some use cases to support in updating devices:
lorawan_version
and lorawan_phy_version
(this changes constraints)root_keys != nil
) but they're not exposed (root_keys.app_key == null
etc). The user should be able to leave the root keys intactI think https://github.com/TheThingsNetwork/lorawan-stack/issues/579#issuecomment-525719408 already contains everything necessary for NS.
I think all mac_settings
should be available to be set for all non-multicast devices.
For multicast devices only the following make sense:
Maybe some things to clarify
Create:
supports_class_b
?Update:
true
, or to reset device state in the NSGeneral stuff:
If we're talking about top-level fields:
https://github.com/TheThingsNetwork/lorawan-stack/blob/375c82cc068bbadb72b887e25631f8f2dc03a366/api/end_device.proto#L395-L418 this whole chunk belongs in NS(but not all of these are required)
m{in,ax}_frequency
is not applicable to multicast
@rvolosatovs
I think #579 (comment) already contains everything necessary for NS.
I think all
mac_settings
should be available to be set for all non-multicast devices.
For multicast devices only the following make sense:
One of the problems with this issue is the amount of information buried in comments.
Can you please add _exactly_ the fields in the right places? I'm fine if you copy my whole list of bullet points so we incrementally work on that until we have a final version.
@htdvisser
- General Settings
a. When we create the device in the ER, we don't set the NS/AS/JS addresses. Do we update the ER registration when the device is set in those? What if we're using an external JS?
I think we should update the addresses when we set the devices in AS/NS/JS.
b. Do we store the activation mode in the ER?
No, we don't have a field for that at the moment, we don't really need it and it creates room for inconsistencies.
c. Do we store the phy/mac versions in the ER?
No, see API documentation. Again, I don't see the need and it creates room for inconsistencies.
a. How about
supports_class_b
?
Yeah I guess we should add that, pending #19. @rvolosatovs please include this in an updated version if relevant.
- Application settings
a. Yes, this could mean that we set the device on the AS twice
Yes, that doesn't hurt
Update:
- Activation mode: would be a lot easier if we store this in ER
Yes but I'm not sure if we should go for easiness and if that fully applies. We need to have it available in the hot path.
Also, it would only be easy if we can start from a clean slate. However, we need to be backward compatible, adding heuristics when lorawan_version
is not in ER, introducing conditional gets from NS etc.
- Do we only look at 404's or also at the NS/AS/JS addresses in the ER?
We can only get a 404 if there's an address set, otherwise there's nothing to get. We should interpret both as "not in that registry". We cannot error here (like we did before), exactly for the reason that creation in IS and AS/NS/JS does not happen at the same time, and users should be able to recover inconsistencies created by the Console.
- I think it would be nice to be able to delete an individual NS/AS/JS registration for example when changing "external JS" to
true
, or to reset device state in the NS
Yes, let's do that later
- _"External JS is where the JS is not in the same cluster as the NS"_: I think this should be something like "join_server_address is not same as JS address in console config"
Yes, that's indeed the way to check it. join_server_address
can be empty also, using interop.
@bafonins do you still have the source for the graph?
Since a lot of work went into constructing that already, I think we should just update that to make it complete to have a clear representation?
We can also work on updating the NS part offline to not clutter the discussion here.
I updated the graph with all possible NS fields
The fields in red are mandatory
@rvolosatovs we're aiming to define the flow and fields that we're presenting in the Console when creating and updating devices.
As such,
mac_state.lorawan_version
mac_state.ping_slot_periodicity
through CLI for nowsupports_class_b
, supports_class_c
and mac_state.device_class
; which is part of create, which is part of update, how does one relate to each other?0
)mac_settings
and mac_state.desired_parameters
; @htdvisser can you think along here?I changed the order in https://github.com/TheThingsNetwork/lorawan-stack/issues/579#issuecomment-553347858. Please work incrementally on that list.
Can you please add exactly the fields in the right places?
I answered this question - i.e. I presented all fields, that can be set and should be set on NS. Or at least that's how I understood your question.
Regarding what should be in the console, everything mac_state
should probably only be settable through CLI, however we may require that for ABP/multicast devices and/or OTAA devices, which are being registered with an existing session, and, hence MAC state.
I will update your comment.
We should not be changing individual counters; a "reset frame counters" button would do (which afaik can be a separate action, like we have in V2 Console, that's setting the counters to 0)
We do need to be able to set downlink frame counters for ABP and multicast - otherwise NS cannot send downlinks
For uplink frame counters are also very useful from security standpoint for ABP
Update device
Here, I would go with a tabbed approach, basically with wizard steps as tabs.
Let's use the accordion approach here, as presented in the first comment from @bafonins:
It will give us fewer issues with horizontal space, and allows to distinguish modes with Edit
/ Create
buttons.
@rvolosatovs
We do need to be able to set downlink frame counters for ABP and multicast - otherwise NS cannot send downlinks
For uplink frame counters are also very useful from security standpoint for ABP
If this is simple we can do it. In fact, two/three input boxes may be simpler than a reset button which triggers a specific action.
Regarding what should be in the console, everything
mac_state
should probably only be settable through CLI, however we may require that for ABP/multicast devices and/or OTAA devices, which are being registered with an existing session, and, hence MAC state.
I understand. I think we should separate "new and reset ABP devices" from "existing ABP devices that have already been on a network".
The former should be complete, so it should include some mac_state
(i.e. factory reset frequencies, RX1 delay, RX2 settings, etc) but not mac_state.desired_parameters
.
The latter should be done via migration and we can add the fine tuning settings later in the Console; for now CLI should be used.
Thanks for updating
LoRaWAN settings
- If NS in cluster: Fields
mac_settings.factory_preset_frequencies
mac_settings.ping_slot_data_rate_index.value
mac_settings.ping_slot_frequency
mac_settings.ping_slot_periodicity.value
mac_settings.rx2_data_rate_index.value
Are these necessary for OTAA? Isn't this only for ABP and Multicast?
@johanstokking
LoRaWAN settings
- If NS in cluster: Fields
mac_settings.factory_preset_frequencies
mac_settings.ping_slot_data_rate_index.value
mac_settings.ping_slot_frequency
mac_settings.ping_slot_periodicity.value
mac_settings.rx2_data_rate_index.value
Are these necessary for OTAA? Isn't this only for ABP and Multicast?
All MAC settings are optional by design.
The only case where these are "required" are class B multicast devices, that is due to the class B operation specifics.
Apart from that, factory_preset_frequencies
is likely to be required for ABP for best results, but nothing prevents OTAA device to have that set.
All MAC settings are optional by design.
The only case where these are "required" are class B multicast devices, that is due to the class B operation specifics.
OK
Apart from that,
factory_preset_frequencies
is likely to be required for ABP for best results, but nothing prevents OTAA device to have that set.
It's ineffective for OTAA; the default join frequencies are required by spec, and the channels come in via join-accept and MAC commands. There should not be out-of-band configuration of preset frequencies for OTAA devices.
@bafonins can you give a status update and timeline on this issue?
For app_s_key
and skip_payload_crypto
, here's the thing;
skip_payload_crypto
of the end device. If true
, AS does not perform payload encryption and decryptionskip_payload_crypto
on the application-level too (via Link, like payload formatters), which takes precedence over the end device's settingapp_s_key
in AS, but it may be wrapped, i.e. session.keys.app_s_key.key
is not set (but encrypted_key
and kek_label
is set)app_s_key
as is to clientsskip_payload_crypto
is set true
(on end device or application link), don't bother with app_s_key
: disable it, don't care