Edge-home-orchestration-go: [DataStorage] Runtime error

Created on 19 Jan 2021  ·  7Comments  ·  Source: lf-edge/edge-home-orchestration-go

Describe the bug
A runtime error when edgex foundry servers are not running, 

level=ERROR ts=2021-01-19T01:51:50.815737266Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
level=INFO ts=2021-01-19T01:51:51.817181193Z app=datastorage source=init.go:144 msg="Check Metadata service's status by ping..."
level=INFO ts=2021-01-19T01:51:51.818287982Z app=datastorage source=init.go:144 msg="Check Data service's status by ping..."
level=ERROR ts=2021-01-19T01:51:51.822577012Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48081/api/v1/ping\": dial tcp 127.0.0.1:48081: connect: connection refused"
level=ERROR ts=2021-01-19T01:51:51.824049381Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
level=INFO ts=2021-01-19T01:51:52.825768174Z app=datastorage source=init.go:144 msg="Check Metadata service's status by ping..."
level=INFO ts=2021-01-19T01:51:52.826907105Z app=datastorage source=init.go:144 msg="Check Data service's status by ping..."
level=ERROR ts=2021-01-19T01:51:52.830784824Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48081/api/v1/ping\": dial tcp 127.0.0.1:48081: connect: connection refused"
level=ERROR ts=2021-01-19T01:51:52.83209855Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
INFO[2021-01-19T01:51:53Z]discovery.go:833 activeDiscovery [discoverymgr] activeDiscovery!!!
INFO[2021-01-19T01:51:53Z]discovery.go:571 func1 [deviceDetectionRoutine] edge-orchestration-3125da9e-1e9a-41aa-ac83-004725eb2d1e
level=ERROR ts=2021-01-19T01:51:53.83359109Z app=datastorage source=init.go:139 msg="dependency Metadata service checking time out"
level=ERROR ts=2021-01-19T01:51:53.834663766Z app=datastorage source=init.go:139 msg="dependency Data service checking time out"
level=INFO ts=2021-01-19T01:51:53.840074015Z app=datastorage source=httpserver.go:116 msg="Web server shutting down"
level=INFO ts=2021-01-19T01:51:53.841736032Z app=datastorage source=httpserver.go:107 msg="Web server stopped"
level=INFO ts=2021-01-19T01:51:54.341966491Z app=datastorage source=httpserver.go:118 msg="Web server shut down"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8d1a0e]

goroutine 44 [running]:
github.com/edgexfoundry/device-sdk-go/internal/autoevent.(*manager).StopAutoEvents(0x0)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/internal/autoevent/manager.go:69 +0x4e
github.com/edgexfoundry/device-sdk-go/pkg/service.(*DeviceService).Stop(0xc0004c2780, 0xc0005ae600)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/service/service.go:134 +0x45
github.com/edgexfoundry/device-sdk-go/pkg/service.Main(0xe2dbda, 0xb, 0xe3a91d, 0x1a, 0xda46e0, 0xc0005a5de0, 0xf46e20, 0xc0005ae600, 0xc0000330a0, 0xc0005ac300, ...)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/service/main.go:69 +0x6fa
github.com/edgexfoundry/device-sdk-go/pkg/startup.Bootstrap(0xe2dbda, 0xb, 0xe3a91d, 0x1a, 0xda46e0, 0xc0005a5de0)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/startup/bootstrap.go:19 +0x117
created by github.com/lf-edge/edge-home-orchestration-go/src/controller/storagemgr.StorageImpl.StartStorage
        /home/t25kim/edge-home-orchestration-go/src/controller/storagemgr/storage.go:51 +0xef

To Reproduce

  1. Put necessary configuration files in /var/edge-orchestration/datastorage/
  2. Run edge-home-orchestration-go

Expected behavior
Check the edgex foundry server in advance before starting Data Storage.

Test environment configuration (please complete the following information):

  • Firmware version: Ubuntu 18.04
  • Hardware: x86-64
  • Edge Orchestration Release: Coconut
bug
Source
picture t25kim
👍1

Most helpful comment

Post communication with EdgeX members, they have provided us with a fork : github.com/hahattan/device-sdk-go v1.4.1 with the fix on the Hanoi branch, causing this crash.

I had been testing the fix by using the replace directive in our go.mod. That seems to work well.

As you have mentioned, we can wait till the Ireland release or we can also clone the fix to our repo till the time EdgeX releases Ireland.

Please share your thoughts on the same ?
@t25kim

Thanks !

picture sun-sharma on 24 Mar 2021
👍2

All 7 comments

@suresh-lc PTAL. Thank you in advance.

MoonkiHong picture MoonkiHong on 19 Jan 2021

Hi Sunchit,

I've figured out why it crashed. It's a bug, and it's not so hard to fix.

The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).

In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134

We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.

We have resolved this issue in V2 (Ireland).

Hi Jim and Iain,

May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

suresh-lc picture suresh-lc on 21 Mar 2021
👍1

Hi Sunchit,

I've figured out why it crashed. It's a bug, and it's not so hard to fix.

The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).

In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134

We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.

We have resolved this issue in V2 (Ireland).

Hi Jim and Iain,

May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

@suresh-lc Who wrote this email?

t25kim picture t25kim on 23 Mar 2021

Hi Sunchit,
I've figured out why it crashed. It's a bug, and it's not so hard to fix.
The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).
In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134
We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.
We have resolved this issue in V2 (Ireland).
Hi Jim and Iain,
May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

@suresh-lc Who wrote this email?

Cloud Tsai from EdgeX- Device Service team

suresh-lc picture suresh-lc on 23 Mar 2021

This issue will be resolved by using the edgexfoundry 'Ireland' version, but it has not been officially released.
I'd like to leave this as a known issue and upgrade edgexfoundry when Ireland is released.

t25kim picture t25kim on 23 Mar 2021
👍1

Post communication with EdgeX members, they have provided us with a fork : github.com/hahattan/device-sdk-go v1.4.1 with the fix on the Hanoi branch, causing this crash.

I had been testing the fix by using the replace directive in our go.mod. That seems to work well.

As you have mentioned, we can wait till the Ireland release or we can also clone the fix to our repo till the time EdgeX releases Ireland.

Please share your thoughts on the same ?
@t25kim

Thanks !

sun-sharma picture sun-sharma on 24 Mar 2021
👍2

@sun-sharma Thank you for letting us know.
Could you modify the go.mod file with the replace flag?
It would be nice to add a note that it is temporary and will be removed.

t25kim picture t25kim on 24 Mar 2021
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

MoonkiHong picture MoonkiHong  ·  6Comments
DoomsdayT picture DoomsdayT  ·  3Comments
MoonkiHong picture MoonkiHong  ·  5Comments
t25kim picture t25kim  ·  4Comments
t25kim picture t25kim  ·  5Comments