Lorawan-stack: One webhook overwrites another settings?

Created on 6 Jul 2019  ·  5Comments  ·  Source: TheThingsNetwork/lorawan-stack

Summary

I',m trying to create on the same instance of lorawan-stack:

  • 1'st application + webhook for production environment (standard data exchange)
  • 2'd application + webhook for firmware update of microcontroller device
  • the same end-device belongs to both applications (the same name/eui - other app eui/keys)

After performing full initialization of one application it looks it overwrite access rights for previous webhook (it reports no access rights - status 403 on server console)

Steps to Reproduce

Create ap2 for firmware upgrade and webhook for it:

ttn-lw-cli end-devices create ap2 dv1 \
  --dev-eui 00217E00000003FF \
  --app-eui 800000000000008C \
  --frequency-plan-id EU_863_870 \
  --root-keys.app-key.key 852BAEC23EAE7964AF27C325F4C23C9A \
  --lorawan-version 1.0.2 \
  --lorawan-phy-version 1.0.2-b
{
  "ids": {
    "device_id": "dv1",
    "application_ids": {
      "application_id": "ap2"
    },
    "dev_eui": "00217E00000003FF",
    "join_eui": "800000000000008C"
  },
  "created_at": "2019-06-26T07:49:08.966Z",
  "updated_at": "2019-06-26T07:49:09.197889667Z",
  "attributes": {
  },
  "network_server_address": "localhost",
  "application_server_address": "localhost",
  "join_server_address": "localhost",
  "lorawan_version": "1.0.2",
  "lorawan_phy_version": "1.0.2-b",
  "frequency_plan_id": "EU_863_870",
  "supports_join": true,
  "root_keys": {
    "app_key": {
      "key": "852BAEC23EAE7964AF27C325F4C23C9A"
    }
  }
}

ttn-lw-cli applications api-keys create \
  --name link \
  --application-id ap2 \
  --right-application-link

  INFO API key ID: LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY
  INFO API key value: NNSXS.LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY.NGYQQRDUJW36KJC3UP6PZYP2DNUYZTGISDJEXACL2TB5BB5EWQCA
  WARN The API key value will never be shown again
  WARN Make sure to copy it to a safe place
{
  "id": "LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY",
  "key": "NNSXS.LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY.NGYQQRDUJW36KJC3UP6PZYP2DNUYZTGISDJEXACL2TB5BB5EWQCA",
  "name": "link",
  "rights": [
    "RIGHT_APPLICATION_LINK"
  ]
}

ttn-lw-cli applications link set ap2 --api-key NNSXS.LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY.NGYQQRDUJW36KJC3UP6PZYP2DNUYZTGISDJEXACL2TB5BB5EWQCA
{
  "api_key": "NNSXS.LHUIKUKWZ62A2LXUYP4ZFC54423D3A3J36G33JY.NGYQQRDUJW36KJC3UP6PZYP2DNUYZTGISDJEXACL2TB5BB5EWQCA"
}

ttn-lw-cli applications api-keys create --name wh-client --application-id ap2 --right-application-link --right-application-all --right-application-delete --right-application-devices-read --right-application-devices-read-keys --right-application-devices-write --right-application-devices-write-keys --right-application-info --right-application-link --right-application-settings-api-keys --right-application-settings-basic --right-application-settings-collaborators --right-application-traffic-down-write --right-application-traffic-read --right-application-traffic-up-write

  INFO API key ID: CLCIYOYYEDPLJSSWRNMYS5KCDI45HOE6M3WZIDY
  INFO API key value: NNSXS.CLCIYOYYEDPLJSSWRNMYS5KCDI45HOE6M3WZIDY.E6DXAAZ4HSX2V6VL7C3244HGNKBO24SEROTXOZURJHWWOMWZQSPA
  WARN The API key value will never be shown again
  WARN Make sure to copy it to a safe place
{
  "id": "CLCIYOYYEDPLJSSWRNMYS5KCDI45HOE6M3WZIDY",
  "key": "NNSXS.CLCIYOYYEDPLJSSWRNMYS5KCDI45HOE6M3WZIDY.E6DXAAZ4HSX2V6VL7C3244HGNKBO24SEROTXOZURJHWWOMWZQSPA",
  "name": "wh-client",
  "rights": [
    "RIGHT_APPLICATION_DEVICES_WRITE",
    "RIGHT_APPLICATION_DEVICES_READ",
    "RIGHT_APPLICATION_TRAFFIC_DOWN_WRITE",
    "RIGHT_APPLICATION_SETTINGS_BASIC",
    "RIGHT_APPLICATION_DEVICES_WRITE_KEYS",
    "RIGHT_APPLICATION_INFO",
    "RIGHT_APPLICATION_SETTINGS_API_KEYS",
    "RIGHT_APPLICATION_SETTINGS_COLLABORATORS",
    "RIGHT_APPLICATION_TRAFFIC_READ",
    "RIGHT_APPLICATION_TRAFFIC_UP_WRITE",
    "RIGHT_APPLICATION_DELETE",
    "RIGHT_APPLICATION_LINK",
    "RIGHT_APPLICATION_ALL",
    "RIGHT_APPLICATION_DEVICES_READ_KEYS"
  ]
}

ttn-lw-cli applications webhooks set \
  --application-id ap2 \
  --webhook-id fwup \
  --format json \
  --base-url http://192.168.0.8/IoT/ \
  --join-accept.path lorafw.php \
  --uplink-message.path lorafw.php

{
  "ids": {
    "application_ids": {
      "application_id": "ap2"
    },
    "webhook_id": "fwup"
  },
  "created_at": "2019-06-26T07:54:51.099460917Z",
  "updated_at": "2019-06-26T07:54:51.099460917Z",
  "base_url": "http://192.168.0.8/IoT/",
  "format": "json",
  "uplink_message": {
    "path": "lorafw.php"
  },
  "join_accept": {
    "path": "lorafw.php"
  }
}

TEST:

curl http://localhost:1885/api/v3/as/applications/ap2/webhooks/fwup/devices/dv1/down/push -X POST   -H 'Authorization: Bearer NNSXS.CLCIYOYYEDPLJSSWRNMYS5KCDI45HOE6M3WZIDY.E6DXAAZ4HSX2V6VL7C3244HGNKBO24SEROTXOZURJHWWOMWZQSPA'  --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'

Webhooks works OK Now

Create ap3 for production environment:
*Normal work * 

**ttn-lw-cli applications create ap3 --user-id admin**

{

  "ids": {

    "application_id": "ap3"

  },

  "created_at": "2019-07-06T09:45:28.540Z",

  "updated_at": "2019-07-06T09:45:28.540Z"

}

**ttn-lw-cli end-devices create ap3 dv1 \
  --dev-eui 00217E00000003FF \
  --app-eui 100000000000001C \
  --frequency-plan-id EU_863_870 \
  --root-keys.app-key.key 152BAEC23EAE7964AF27C325F4C23C9A \
  --lorawan-version 1.0.2 \
  --lorawan-phy-version 1.0.2-b**

{

  "ids": {

    "device_id": "dv1",

    "application_ids": {

      "application_id": "ap3"

    },

    "dev_eui": "00217E00000003FF",

    "join_eui": "100000000000001C"

  },

  "created_at": "2019-07-06T09:46:16.897Z",

  "updated_at": "2019-07-06T09:46:17.144655816Z",

  "attributes": {

  },

  "network_server_address": "localhost",

  "application_server_address": "localhost",

  "join_server_address": "localhost",

  "lorawan_version": "1.0.2",

  "lorawan_phy_version": "1.0.2-b",

  "frequency_plan_id": "EU_863_870",

  "supports_join": true,

  "root_keys": {

    "app_key": {

      "key": "852BAEC23EAE7964AF27C325F4C23C9A"

    }

  }

}

**ttn-lw-cli applications api-keys create \
  --name link \
  --application-id ap3 \
  --right-application-link**
  INFO API key ID: 77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI

  INFO API key value: NNSXS.77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI.UFJ2Y5ITTD6DKTMIPSH3UOUMVGPAIOGBQGF547KWPCAZ6WZE6VXQ

  WARN The API key value will never be shown again

  WARN Make sure to copy it to a safe place

{

  "id": "77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI",

  "key": "NNSXS.77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI.UFJ2Y5ITTD6DKTMIPSH3UOUMVGPAIOGBQGF547KWPCAZ6WZE6VXQ",

  "name": "link",

  "rights": [

    "RIGHT_APPLICATION_LINK"

  ]

}

**ttn-lw-cli applications link set ap3 --api-key NNSXS.77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI.UFJ2Y5ITTD6DKTMIPSH3UOUMVGPAIOGBQGF547KWPCAZ6WZE6VXQ**

{

  "api_key": "NNSXS.77EPVZXEKOCAD7G6G4UGAKERJGNL3MHNYIWA7YI.UFJ2Y5ITTD6DKTMIPSH3UOUMVGPAIOGBQGF547KWPCAZ6WZE6VXQ"

}

**ttn-lw-cli applications api-keys create --name wh-client --application-id ap3 --right-application-link --right-application-all --right-application-delete --right-application-devices-read --right-application-devices-read-keys --right-application-devices-write --right-application-devices-write-keys --right-application-info --right-application-link --right-application-settings-api-keys --right-application-settings-basic --right-application-settings-collaborators --right-application-traffic-down-write --right-application-traffic-read --right-application-traffic-up-write**


INFO API key ID: JIXW5XX3UITDZASY4L4A3FZ4Z53YF2S63ZASO3Y
  INFO API key value: NNSXS.JIXW5XX3UITDZASY4L4A3FZ4Z53YF2S63ZASO3Y.7TDNWLRAKDETDFJXHVWHYQ3J47CDD7SRQR66FGT7F72BHTZMSYGQ
  WARN The API key value will never be shown again
  WARN Make sure to copy it to a safe place
{

  "id": "JIXW5XX3UITDZASY4L4A3FZ4Z53YF2S63ZASO3Y",

  "key": "NNSXS.JIXW5XX3UITDZASY4L4A3FZ4Z53YF2S63ZASO3Y.7TDNWLRAKDETDFJXHVWHYQ3J47CDD7SRQR66FGT7F72BHTZMSYGQ",

  "name": "wh-client",

  "rights": [

    "RIGHT_APPLICATION_TRAFFIC_READ",

    "RIGHT_APPLICATION_SETTINGS_API_KEYS",

    "RIGHT_APPLICATION_DEVICES_READ",

    "RIGHT_APPLICATION_SETTINGS_BASIC",

    "RIGHT_APPLICATION_DEVICES_READ_KEYS",

    "RIGHT_APPLICATION_TRAFFIC_DOWN_WRITE",

    "RIGHT_APPLICATION_ALL",

    "RIGHT_APPLICATION_INFO",

    "RIGHT_APPLICATION_LINK",

    "RIGHT_APPLICATION_SETTINGS_COLLABORATORS",

    "RIGHT_APPLICATION_TRAFFIC_UP_WRITE",

    "RIGHT_APPLICATION_DEVICES_WRITE_KEYS",

    "RIGHT_APPLICATION_DELETE",

    "RIGHT_APPLICATION_DEVICES_WRITE"

  ]

}

**ttn-lw-cli applications webhooks set \
  --application-id ap3 \
  --webhook-id production \
  --format json \
  --base-url http://192.168.0.8/IoT/ \
  --join-accept.path join.php \
  --uplink-message.path up.php**



{

  "ids": {

    "application_ids": {

      "application_id": "ap3"

    },

    "webhook_id": "production"

  },

  "created_at": "2019-07-06T09:57:36.729241177Z",

  "updated_at": "2019-07-06T09:57:36.729241177Z",

  "base_url": "http://192.168.0.8/IoT/",

  "format": "json",

  "uplink_message": {

    "path": "up.php"

  },

  "join_accept": {

    "path": "join.php"

  }

}

**ttn-lw-cli applications api-keys create --name wh-prod --application-id ap3 --right-application-link --right-application-all --right-application-delete --right-application-devices-read --right-application-devices-read-keys --right-application-devices-write --right-application-devices-write-keys --right-application-info --right-application-link --right-application-settings-api-keys --right-application-settings-basic --right-application-settings-collaborators --right-application-traffic-down-write --right-application-traffic-read --right-application-traffic-up-write**

  INFO API key ID: SC626ESUGY3E5RO2NB6NOMCPKT6SDR3MJWTAQZA

  INFO API key value: NNSXS.SC626ESUGY3E5RO2NB6NOMCPKT6SDR3MJWTAQZA.VCUMNMSQQFDNBD7BSKKXEXCCMJH77254DEVTDCUICO45RDAT2TVA

  WARN The API key value will never be shown again

  WARN Make sure to copy it to a safe place

{

  "id": "SC626ESUGY3E5RO2NB6NOMCPKT6SDR3MJWTAQZA",

  "key": "NNSXS.SC626ESUGY3E5RO2NB6NOMCPKT6SDR3MJWTAQZA.VCUMNMSQQFDNBD7BSKKXEXCCMJH77254DEVTDCUICO45RDAT2TVA",

  "name": "wh-prod",

  "rights": [

    "RIGHT_APPLICATION_DELETE",

    "RIGHT_APPLICATION_LINK",

    "RIGHT_APPLICATION_DEVICES_WRITE",

    "RIGHT_APPLICATION_TRAFFIC_READ",

    "RIGHT_APPLICATION_TRAFFIC_UP_WRITE",

    "RIGHT_APPLICATION_SETTINGS_COLLABORATORS",

    "RIGHT_APPLICATION_DEVICES_WRITE_KEYS",

    "RIGHT_APPLICATION_TRAFFIC_DOWN_WRITE",

    "RIGHT_APPLICATION_DEVICES_READ_KEYS",

    "RIGHT_APPLICATION_SETTINGS_API_KEYS",

    "RIGHT_APPLICATION_SETTINGS_BASIC",

    "RIGHT_APPLICATION_DEVICES_READ",

    "RIGHT_APPLICATION_ALL",

    "RIGHT_APPLICATION_INFO"

  ]

}

a) AP2 application webhook downlink stop working (downlink - with access rights issue)
b) AP3 application webhook start working
c) When modify AP1 api-key and exchange it on webhook php file and reboot network server AP2 get back to work and AP3 stops downlink working (downlink - with access rights issue)

Manipulation API

What do you see now?

...

What do you want to see instead?

...

Environment


Not relevant Ubuntu 16/kerlink/Telit/1.0.2 Class A
...

How do you propose to implement this?

...

Can you do this yourself and submit a Pull Request?

...

blocking release bug application server
Source
picture ecities

Most helpful comment

I can confirm that I've been able to reproduce this bug on the v3.0.3. My run can be found here. I'll investigate and come back with the results.

picture adriansmares on 9 Jul 2019
👍2

All 5 comments

I can confirm that I've been able to reproduce this bug on the v3.0.3. My run can be found here. I'll investigate and come back with the results.

adriansmares picture adriansmares on 9 Jul 2019
👍2

@adriansmares Thanks.

ecities picture ecities on 9 Jul 2019

I've migrated my v3.0.3 setup to the latest master and apparently the bug has been fixed in the meantime. 

adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ git checkout master
Previous HEAD position was e8450dac8 all: Bump to version 3.0.3
Switched to branch 'master'
Your branch is up to date with 'ttn/master'.
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app1/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.PRF4BDQMASLMRWXTV5HRM4TQUROQAHNTDR7EZTY.JK6XNLQYXQSNCAWZHFOXG6LCBINIEMVSDPR3OIORDVQF5FJCI6YA'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app2/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.V3ZM2DMCSFK4YEBB55WA6MMY34YXQ6SARLJS4TQ.OILYCKNO5R25HYYZB6BPOJN5XLKWQZFXOIB77TJ464IDQBQ2YI7A'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ git checkout v3.0.3
Note: checking out 'v3.0.3'.
...
HEAD is now at e8450dac8 all: Bump to version 3.0.3
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app1/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.PRF4BDQMASLMRWXTV5HRM4TQUROQAHNTDR7EZTY.JK6XNLQYXQSNCAWZHFOXG6LCBINIEMVSDPR3OIORDVQF5FJCI6YA'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app2/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.V3ZM2DMCSFK4YEBB55WA6MMY34YXQ6SARLJS4TQ.OILYCKNO5R25HYYZB6BPOJN5XLKWQZFXOIB77TJ464IDQBQ2YI7A'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'
error:pkg/auth/rights:no_application_rights (no rights for application `app2`)

I did look however which commits fixed this issue, and the they are a2e7e4c6b8929dfd52ba62046cc70d4529c4f25e and fe95aafb9959e841e920eaad39eb2e39b6ed24e6

adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ git checkout v3.0.3
...
HEAD is now at e8450dac8 all: Bump to version 3.0.3
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ git cherry-pick a2e7e4c6b8929dfd52ba62046cc70d4529c4f25e
[detached HEAD d8f1ee6fc] as: Use request context in webhook downlink
 Author: Johan Stokking <[email protected]>
 Date: Mon May 27 14:39:05 2019 +0200
 4 files changed, 27 insertions(+), 4 deletions(-)
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ git cherry-pick fe95aafb9959e841e920eaad39eb2e39b6ed24e6
[detached HEAD dce35a353] as: Fill Webhook downlink queue request context
 Date: Tue Jul 2 11:44:10 2019 +0200
 1 file changed, 1 insertion(+), 1 deletion(-)
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app1/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.PRF4BDQMASLMRWXTV5HRM4TQUROQAHNTDR7EZTY.JK6XNLQYXQSNCAWZHFOXG6LCBINIEMVSDPR3OIORDVQF5FJCI6YA'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'
adriansmares@A98BCD2222F1:~/go/src/go.thethings.network/lorawan-stack$ curl http://localhost:1885/api/v3/as/applications/app2/webhooks/wh1/devices/dev1/down/push   -X POST   -H 'Authorization: Bearer NNSXS.V3ZM2DMCSFK4YEBB55WA6MMY34YXQ6SARLJS4TQ.OILYCKNO5R25HYYZB6BPOJN5XLKWQZFXOIB77TJ464IDQBQ2YI7A'   --data '{"downlinks":[{"frm_payload":"vu8=","f_port":15,"priority":"NORMAL"}]}'

Since we have no master docker image, should we have a v3.0.3b release which includes these fixes ? @johanstokking @htdvisser

adriansmares picture adriansmares on 9 Jul 2019

I've investigated _why_ the two commits actually fix this issue, and found that the real cause is actually not related to them directly, but rather to the fact that in v3.0.3 the context used for rights checking is shared between requests.

In v3.0.3, the context used by the Webhooks component to check the rights of the request is w.ctx, which if we follow up to its creation, is the context of the Component, then filled using FillContext.
https://github.com/TheThingsNetwork/lorawan-stack/blob/e8450dac84c1d7ec685121957d7e8cd4ef67c013/pkg/applicationserver/applicationserver.go#L163
This looks inoffensive (or at least looked until today), until you realize that one of the context fillers is a rights cache, _which stays inside the context after the first use_.
https://github.com/TheThingsNetwork/lorawan-stack/blob/e8450dac84c1d7ec685121957d7e8cd4ef67c013/pkg/identityserver/entity_access.go#L46-L69
https://github.com/TheThingsNetwork/lorawan-stack/blob/e8450dac84c1d7ec685121957d7e8cd4ef67c013/pkg/identityserver/identityserver.go#L146-L150

What happens is that the first request (by app1, or app2 - doesn't really matter) sets the cache of the Webhooks context (w.ctx). w.ctx is FillContexted on creation, and contains the auth cache - so after the first request is done, the rights are saved inside the context.
https://github.com/TheThingsNetwork/lorawan-stack/blob/e8450dac84c1d7ec685121957d7e8cd4ef67c013/pkg/identityserver/entity_access.go#L64-L68
Follow up requests are no longer served by the IS database, and instead get filled from the cache. But since the API key that made the first request does not have rights on the second application, the 2nd request fails. The corollary is that any further requests that are not made by the first API key will fail by default.

TLDR: Probably https://github.com/TheThingsNetwork/lorawan-stack/pull/902/commits/e6ab950fa76bc8cdbe6dc519ec71b9deed2e7231 should get merged as well in the quickfix release. FillContext should never be used on standalone contexts, only request contexts.

adriansmares picture adriansmares on 9 Jul 2019
👍1

Released in https://github.com/TheThingsNetwork/lorawan-stack/releases/tag/v3.0.4

johanstokking picture johanstokking on 10 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

adamsondelacruz picture adamsondelacruz  ·  7Comments
htdvisser picture htdvisser  ·  4Comments
johanstokking picture johanstokking  ·  8Comments
kschiffer picture kschiffer  ·  7Comments
adriansmares picture adriansmares  ·  8Comments