Похоже на:

  WARN No cookie hash key configured, generated a random one hash_key=2FECCF81EFFEF0C8110CCBE2BE8B23C26E337279E62756ACFE16375BB6897E3EBA5759D84E150BB7E17A99EBEDE387B096EE41BB4DD44BC19C9609EFC8328F06 namespace=web test_name=TestGatewayServer
  WARN No cookie block key configured, generated a random one block_key=822E142EF552CFEEC7E546D22F7E36ED26CBCE6D79A8C4415898EC242919D649 namespace=web test_name=TestGatewayServer
  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer
  WARN No rights TTL configured                 test_name=TestGatewayServer
  WARN No cluster key configured, generated a random one key=eaadd899806bb4c13450d4257d7c6110c592da1b724a8978068a8f9187eb53d9 test_name=TestGatewayServer
  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer
  INFO Finished unary call                      duration=5.012417ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H70FPFWJRBCCZAJTAS4W test_name=TestGatewayServer
  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:41030 request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  INFO Finished streaming call                  duration=245.644341ms error=error:unknown:unknown (context canceled) error_correlation_id=1257f760d52f4150a71f3ee60703974a grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  WARN Finished unary call                      duration=604.26µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) error_correlation_id=46b6a81d7eeb4c089f7e3c4eedc841f5 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78GTQJD37YRDRTY6Y9V test_name=TestGatewayServer
  WARN Finished unary call                      duration=503.568µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) error_correlation_id=eb72c1e0978a4702b6bc36439c272e00 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78JP2MV2CWX8NVV1848 test_name=TestGatewayServer
  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:41030 request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  INFO Finished unary call                      duration=113.698µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=efc5641d14db4e939380d3daefa580bb error_name=identifiers error_namespace=pkg/ttnpb grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78M598DAMETTSJ2FZBP test_name=TestGatewayServer
  INFO Finished unary call                      duration=98.372µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=55e3f96c3e1e4744a74efa5eb9fd426a error_name=identifiers error_namespace=pkg/ttnpb grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78NJW6NKZKT4H6YX47J test_name=TestGatewayServer
  INFO Finished unary call                      duration=765.197µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=b07dd87e51824fc29bba3c808e38a183 error_name=not_connected error_namespace=pkg/gatewayserver grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7GQBS44PQD1EWMJ90MV test_name=TestGatewayServer
  INFO Finished unary call                      duration=863.468µs grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H7GS4ZEYAXTN31QZ4WJ9 test_name=TestGatewayServer
  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:41042 request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
  INFO Finished unary call                      duration=580.106µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7RW3SJ569D1GQZ7HF3B test_name=TestGatewayServer
  INFO Finished unary call                      duration=655.422µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7RY0P6F19NTX7W9P5ZE test_name=TestGatewayServer
  INFO Finished unary call                      duration=444.661µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7S0143EYQES1ABD6GM6 test_name=TestGatewayServer
  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:41042 request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
  INFO Finished streaming call                  duration=263.036878ms error=error:unknown:unknown (context canceled) error_correlation_id=06aac79e327f4b86b2dd0e8520fb2a5e grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
--- FAIL: TestGatewayServer (1.86s)
    --- FAIL: TestGatewayServer/Traffic/grpc (0.52s)
        --- FAIL: TestGatewayServer/Traffic/grpc/Upstream (0.26s)
            --- FAIL: TestGatewayServer/Traffic/grpc/Upstream/OneValidLoRa (0.00s)
                gatewayserver_test.go:812: gs.up.forward event published
                doc.go:66: 
                    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver_test.go:878
                    Expected: '1'
                    Actual:   '0'
                    (Should be equal)
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    1.915s

Похоже, я нашел еще один:

  WARN No cookie hash key configured, generated a random one hash_key=AF8CAC67C022D6738C91ACEC222EB0FEEB1EED3075DCC91EB38660FB9C8C05B72CD2CCBA4A6CE81DDDD937D6C1C86AE10ACE17BB57D6F44BEC7114FC36055BD7 namespace=web test_name=TestGatewayServer

  WARN No cookie block key configured, generated a random one block_key=D0DA3B8FF79A7F4A35AF7199A95465A5DA4E99D5EB7B8FEEF7A2A4CD3B4D2BC8 namespace=web test_name=TestGatewayServer

  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer

  WARN No rights TTL configured                 test_name=TestGatewayServer

  WARN No cluster key configured, generated a random one key=f6d812ec6397f4cd4b12e7725ca88d99334ada3fe94e5b1181b5b8101f48d66d test_name=TestGatewayServer

  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer

  INFO Finished unary call                      duration=2.932065ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1H2JKQ3YDTN36EFNCV0CS56 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Finished streaming call                  duration=247.357924ms error=context deadline exceeded grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Link failed                              error=rpc error: code = DeadlineExceeded desc = context deadline exceeded gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Task failed                              error=mux: listener closed invocation=1 namespace=gatewayserver task_id=serve_mqtt/:1882 test_name=TestGatewayServer

--- FAIL: TestGatewayServer (4.66s)

    --- FAIL: TestGatewayServer/Authenticate/grpc (0.26s)

        --- FAIL: TestGatewayServer/Authenticate/grpc/ValidIDAndKey (0.26s)

            gatewayserver_test.go:559: Expected deadline exceeded with valid auth, but have error:unknown:unknown (context deadline exceeded)

  WARN Task failed                              error=read udp [::]:1700: use of closed network connection invocation=1 namespace=gatewayserver task_id=serve_udp/:1700 test_name=TestGatewayServer

  WARN Task failed                              error=http: Server closed invocation=1 namespace=gatewayserver task_id=serve_basicstation/:1887 test_name=TestGatewayServer

FAIL

  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestGatewayServer

coverage: 41.4% of statements

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    4.714s

И еще один с basicstation этот раз:

  WARN No cookie hash key configured, generated a random one hash_key=686D148310C51B627BF34558528A47B84D1001620F48956E6B042A7CCBC82CC5B3F7AA873BA65BD7C7B43AEC7DA75CEE8F82761886A14AAF7182B394E1513879 namespace=web test_name=TestGatewayServer

  WARN No cookie block key configured, generated a random one block_key=A52935E61E7592777715B8CB46FAF8E3221A0DAC214BDD6A23607E668F3C7BAA namespace=web test_name=TestGatewayServer

  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer

  WARN No rights TTL configured                 test_name=TestGatewayServer

  WARN No cluster key configured, generated a random one key=965535d0e2ca0d5883c0e8c0fa0eef8fe4d9f97f25b7458e77dfa6b3cd15d5e0 test_name=TestGatewayServer

  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer

  INFO Finished unary call                      duration=14.054791ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9X43KPMHN775400EDVSY test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60682 request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  INFO Finished streaming call                  duration=239.767717ms error=context canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60682 request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  WARN Finished unary call                      duration=566.11µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) error_correlation_id=35b22abf384e4bf69c03e09931b4e689 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XC31MX21ZTFNCYARFMD test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Finished unary call                      duration=444.016µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) error_correlation_id=248c07bcb42b45cd951f72d683cf3804 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XC5152VJWPBT2F7TB8E test_name=TestGatewayServer uid=invalid-gateway

  INFO Finished unary call                      duration=173.105µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=eed7ec8dbca0442d8ab04037d4f1b217 error_name=identifiers error_namespace=pkg/ttnpb field=gateway_id grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs name=GatewayIdentifiersValidationError namespace=grpc reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$" request_id=01E1KP9XC71EYJ5KJSGZWQZYJZ test_name=TestGatewayServer

  INFO Finished unary call                      duration=81.663µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=85ad95d2b6d046fc9fd1ab3c5aa3d17b error_name=identifiers error_namespace=pkg/ttnpb field=gateway_id grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs name=GatewayIdentifiersValidationError namespace=grpc reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$" request_id=01E1KP9XC8JVRETETN2B9HYEAE test_name=TestGatewayServer

  INFO Finished unary call                      duration=923.455µs grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XMAVY68WF94PTMENQXW test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60692 request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.109666ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XWAD7DW8MBEZGR97BXR test_name=TestGatewayServer

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  WARN Finished streaming call                  duration=257.213855ms error=error:pkg/gatewayserver:new_connection (new connection from same gateway) error_correlation_id=b652b79bda0f404c875e3da58601241c error_name=new_connection error_namespace=pkg/gatewayserver grpc_code=Aborted grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60692 request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Finished streaming call                  duration=254.052548ms error=error:unknown:unknown (context canceled) error_correlation_id=8ce5108c56c34e27bce9f005609d7348 grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Finished unary call                      duration=600.622µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=8106732c22904b4b90f56163a78c7b7e error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YCCSKMR0Q62G1N6A7X8 test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.026905ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9YCEVSHC8H81J9H3NER2 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=602.597µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMHMZGJ83XM2MHZVJND test_name=TestGatewayServer

  INFO Finished unary call                      duration=427.272µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMK5Q9HJQTM6GWAK7YA test_name=TestGatewayServer

  INFO Finished unary call                      duration=357.239µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMN0Y39J7P2GVYW9SD3 test_name=TestGatewayServer

  INFO Finished unary call                      duration=599.939µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMPJWGMNBS7JJ0M0HFK test_name=TestGatewayServer

  INFO Finished unary call                      duration=558.232µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWTVE97R1D6SQVWVR54 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=411.31µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWWECN00MWPSZB272RE test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=551.542µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWXR9BVHT56K1DN4GB0 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.068656ms grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YX8TEAG9S680RQ6DHB6 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=639.377µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YXA0F3EV7EMQ27NVXW7 test_name=TestGatewayServer

  INFO Finished streaming call                  duration=541.847903ms error=error:unknown:unknown (context canceled) error_correlation_id=0612e7a48e594d33abce482385099ca8 grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=584.252µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=b4f5f60b8b0d40e7b3e9d3a46a57688d error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9Z5EJ4EW0588SY6DWM04 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40724 test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40726 test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40728 test_name=TestGatewayServer uid=invalid-gateway

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40730 test_name=TestGatewayServer uid=invalid-gateway

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  WARN Error receiving packet                   client_id=127.0.0.1:40732-3419925607 error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection namespace=gatewayserver/io/mqtt test_name=TestGatewayServer username=eui-aaee000000000000

  WARN Error when reading packet                error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Finished unary call                      duration=604.953µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=f2068f9ebfba4cb28eb76ce2238964a0 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA0XH5P41MNQC1Y5ZGSEY test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=966.471µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15N9RXZWA8SK7QACS4J test_name=TestGatewayServer

  INFO Finished unary call                      duration=635.035µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15QT8H8XENDG6TP87W0 test_name=TestGatewayServer

  INFO Finished unary call                      duration=347.509µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15VA5BHCW8YXNNWJ03G test_name=TestGatewayServer

  INFO Finished unary call                      duration=395.257µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15W4T4NZEA7Q4G3F1ZT test_name=TestGatewayServer

  INFO Finished unary call                      duration=626.299µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1DZB7G1ENE86N3PHAEK test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=437.273µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E14D8M1QE5NGCM6RPV test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=385.207µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E2HE36ZE2MT0FQ1P4F test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=691.985µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E4ZRHDHTPVVHNMNJ5P test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=505.736µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E67BX01H78KFQR0Q2S test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=714.237µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=545034895c8f403498fad3634aed83e5 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1P8T99QZMEABNQJ96CG test_name=TestGatewayServer

  INFO Connected                                gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  WARN Connect unregistered gateway             gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Connected                                gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=521.446µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=6d78bc53308141b3959bfcf87328bf2f error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3DSMQTCPYG6DGVMEBJV test_name=TestGatewayServer

  INFO Connected                                gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=496.765µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NW2S7PRDJB4ECGKDRR test_name=TestGatewayServer

  INFO Finished unary call                      duration=343.624µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NYY200GY8YFS97VYXB test_name=TestGatewayServer

  INFO Finished unary call                      duration=612.383µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NZCP9ZPVN1GKB55QVR test_name=TestGatewayServer

  INFO Finished unary call                      duration=511.471µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3P1GB5GGMN4D3HF7DE2 test_name=TestGatewayServer

  INFO Finished unary call                      duration=557.428µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y4FYVCE6CMXR0FXF1R test_name=TestGatewayServer

  INFO Finished unary call                      duration=384.58µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y5WXMSFQEM11XH81A2 test_name=TestGatewayServer

  INFO Finished unary call                      duration=334.846µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y6HVCGQKC8K7GZZSSR test_name=TestGatewayServer

  INFO Finished unary call                      duration=373.463µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y8Y57REGBHVYB4RG0C test_name=TestGatewayServer

  INFO Finished unary call                      duration=323.76µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y9NGDB4ZWEJVBJXF8V test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=586.868µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=df3fd6a0d98040a8822291fccd8f7767 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA4J16P0TPSZEXATTBV5Z test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.667636ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.990103ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=256.949369ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.937262ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Finished unary call                      duration=511.423µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=48680752385f4f7bab0314f8eb2e3225 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA625W72DV35C84B7CFTF test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Finished unary call                      duration=458.982µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA627JZH5S5WZBTZXX95N test_name=TestGatewayServer

  INFO Finished unary call                      duration=288.972µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA6286VHRWHBM6ZEN8HW0 test_name=TestGatewayServer

  INFO Finished unary call                      duration=293.552µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA62AKSZ3S2M3XD57QKZ1 test_name=TestGatewayServer

  INFO Finished unary call                      duration=288.987µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA62CCZRXH7JN8XX0YY3X test_name=TestGatewayServer

  INFO Request handled                          duration=6.202961ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  WARN Task failed                              error=read udp [::]:1700: use of closed network connection invocation=1 namespace=gatewayserver task_id=serve_udp/:1700 test_name=TestGatewayServer

  WARN Task failed                              error=mux: listener closed invocation=1 namespace=gatewayserver task_id=serve_mqtt/:1882 test_name=TestGatewayServer

  WARN Task failed                              error=http: Server closed invocation=1 namespace=gatewayserver task_id=serve_basicstation/:1887 test_name=TestGatewayServer

--- FAIL: TestGatewayServer (11.77s)

    --- FAIL: TestGatewayServer/Traffic/basicstation (0.26s)

        --- FAIL: TestGatewayServer/Traffic/basicstation/Upstream (0.01s)

            --- FAIL: TestGatewayServer/Traffic/basicstation/Upstream/OneValidFSK (0.00s)

                gatewayserver_test.go:857: gs.up.forward event published

                doc.go:66: 

                    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver_test.go:923

                    Expected: '2'

                    Actual:   '1'

                    (Should be equal)

  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestGatewayServer

FAIL

coverage: 78.1% of statements

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    11.830s

И еще один, исх. № 2003.

2020-02-26T12:09:37.2950440Z   WARN No cookie hash key configured, generated a random one hash_key=2F2F7C66987C8D056542955EF87C2E847403300EB8902617A2F10CAB64A0BF43E65A02381BB8E70DD428A5BBBF6A5CA9D97F57D086553B7AC354335BF2CC2557 namespace=web test_name=TestClientTokenAuth
2020-02-26T12:09:37.2951747Z   WARN No cookie block key configured, generated a random one block_key=7A6A1716EA81C20770D8AE6606279EE50AA66BCEDA4BF45960D21F7457DD1908 namespace=web test_name=TestClientTokenAuth
2020-02-26T12:09:37.2952326Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestClientTokenAuth
2020-02-26T12:09:37.2952700Z   WARN No rights TTL configured                 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2953277Z   WARN No cluster key configured, generated a random one key=4828de4e338f805fd9872013c35c32ab574805eeaabeead7b302be4142ad58fa test_name=TestClientTokenAuth
2020-02-26T12:09:37.2953659Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestClientTokenAuth
2020-02-26T12:09:37.2954826Z   INFO Finished unary call                      duration=25.801µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBNVNTJ15PBK9BHE31Y test_name=TestClientTokenAuth
2020-02-26T12:09:37.2955957Z   INFO Request handled                          duration=1.543711ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58330 request_id=01E20PDKBMZ3TNAGG60ZA3C1FR response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2957459Z   INFO Finished unary call                      duration=11.5µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBPXGSGGN7JH07XH7J6 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2958673Z   INFO Request handled                          duration=729.705µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58332 request_id=01E20PDKBPM7HAKKRQHP2P2NEG response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2959790Z   INFO Finished unary call                      duration=12.9µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBQVGBE3GQGEAHEYBSE test_name=TestClientTokenAuth
2020-02-26T12:09:37.2960871Z   WARN Failed to connect                        endpoint=traffic error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-0101010101010101`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58334 request_id=01E20PDKBQXW3VQ1QATA9B6BMS test_name=TestClientTokenAuth uid=eui-0101010101010101 url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2963193Z   INFO Request handled                          duration=630.705µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-0101010101010101`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58334 request_id=01E20PDKBQXW3VQ1QATA9B6BMS status=403 test_name=TestClientTokenAuth uid=eui-0101010101010101 url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2964592Z   INFO Request handled                          duration=273.002µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58336 request_id=01E20PDKBRCEFWGBN88QV1VFDT response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2965683Z   INFO Finished unary call                      duration=12.1µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBST5716W4V4RBKHQG8 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2967665Z   WARN Failed to connect                        endpoint=traffic error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-1122334455667788`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58338 request_id=01E20PDKBRBFF9XF21D1WERF9R test_name=TestClientTokenAuth uid=eui-1122334455667788 url=/traffic/eui-1122334455667788
2020-02-26T12:09:37.2971339Z   INFO Request handled                          duration=525.004µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-1122334455667788`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58338 request_id=01E20PDKBRBFF9XF21D1WERF9R status=403 test_name=TestClientTokenAuth uid=eui-1122334455667788 url=/traffic/eui-1122334455667788
2020-02-26T12:09:37.2972518Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestClientTokenAuth
2020-02-26T12:09:37.2973051Z   WARN No cookie hash key configured, generated a random one hash_key=D834925B52BC94223C4E7CA1FE9B6C83BE9EA2EBB03F5724EEBEAE1B05454FE5D5299CBE7984154F63003120EB536D2789904F1598614566E2F016617E672B0C namespace=web test_name=TestDiscover
2020-02-26T12:09:37.2973480Z   WARN No cookie block key configured, generated a random one block_key=D93A7217E0F0D4278F38BB522C167599F5130449C112D8FFECB3F3B0D18CC318 namespace=web test_name=TestDiscover
2020-02-26T12:09:37.2973841Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestDiscover
2020-02-26T12:09:37.2974085Z   WARN No rights TTL configured                 test_name=TestDiscover
2020-02-26T12:09:37.2974437Z   WARN No cluster key configured, generated a random one key=4185bee11ced6f7d9d2b5fc0db6b47b7c6a45b26fe42bdfb0df843e0916e659b test_name=TestDiscover
2020-02-26T12:09:37.2974689Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestDiscover
2020-02-26T12:09:37.2974992Z   INFO Request handled                          duration=300ns error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60578 request_id=01E20PDKCFWJTSJHTP136QEKSJ status=404 test_name=TestDiscover url=/api/v3/gs/io/basicstation/discover
2020-02-26T12:09:37.2976679Z   INFO Request handled                          duration=300ns error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60580 request_id=01E20PDKCFR039DECP6DKE2F9X status=404 test_name=TestDiscover url=/router-info/eui-0101010101010101
2020-02-26T12:09:37.2977633Z  ERROR Request error                            duration=205.102µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60582 request_id=01E20PDKCG80X7HQ9Z2SZGE6F5 status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2977896Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2978106Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2979469Z  ERROR Request error                            duration=127.101µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60584 request_id=01E20PDKCGY7M3M2AG09S5NBGJ status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2979843Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2980047Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2980896Z  ERROR Request error                            duration=134.501µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60586 request_id=01E20PDKCH7G8V10W6T3PQJXED status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2981164Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2981367Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2982281Z   INFO Request handled                          duration=388.303µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60588 request_id=01E20PDKCH891Y49M37TWCA7ZC status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2982520Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2982718Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2984207Z   INFO Request handled                          duration=468.504µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60590 request_id=01E20PDKCJ9QBPXRQ58WB83ZVY status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2984478Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2984680Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2985505Z   INFO Request handled                          duration=476.904µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60592 request_id=01E20PDKCJ9C6FCBRB0J2KA0A2 status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2985738Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2985961Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2986810Z   INFO Request handled                          duration=249.801µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60594 request_id=01E20PDKCJT4XT2TSCKERMZTMR response_size=0 status=200 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2987372Z   WARN No cookie hash key configured, generated a random one hash_key=945021EB77E419898A7155BA2A452690636593668F47C7978EC16EF42E92B4F8996CDA0C156F1E9B597FBCDC1A55059FFD4B80AC69ABF2F8376C13505E92CC80 namespace=web test_name=TestVersion
2020-02-26T12:09:37.2987798Z   WARN No cookie block key configured, generated a random one block_key=A26709BEB30BAE31B780BA0BA9AB61D490FF9B79A3055C38A0EAB27878373CF1 namespace=web test_name=TestVersion
2020-02-26T12:09:37.2988101Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestVersion
2020-02-26T12:09:37.2988495Z   WARN No rights TTL configured                 test_name=TestVersion
2020-02-26T12:09:37.2989583Z   WARN No cluster key configured, generated a random one key=094e698636bc15731746b7dc6d719928250b0248e61f0cdebcf68f9ec58ecc14 test_name=TestVersion
2020-02-26T12:09:37.2989896Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestVersion
2020-02-26T12:09:37.2990391Z   WARN No cookie hash key configured, generated a random one hash_key=1885A3A4F1CAF4364173D33E8129A2FC53AE80F3823E5A48207D13FECA63A9F3358320589A74FA343CDEF6BD08A64662664844967168AC6A04C9982D682F9230 namespace=web test_name=TestTraffic
2020-02-26T12:09:37.2990867Z   WARN No cookie block key configured, generated a random one block_key=0878491E0147A2FC43AEA720E91EEF8326EC92C3985B315691524F4B1483B0D4 namespace=web test_name=TestTraffic
2020-02-26T12:09:37.2991149Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestTraffic
2020-02-26T12:09:37.2992220Z   INFO Request handled                          duration=2.911722ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:53514 request_id=01E20PDKD7X8JBYBCA6QTNT2RP response_size=0 status=200 test_name=TestVersion url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2992502Z   WARN No rights TTL configured                 test_name=TestTraffic
2020-02-26T12:09:37.2992863Z   WARN No cluster key configured, generated a random one key=6807b19d6e375f3161d8cd6ae07e1e58bca0f6625d530c819fd818a8b6aebb07 test_name=TestTraffic
2020-02-26T12:09:37.2993108Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestTraffic
2020-02-26T12:09:37.2993993Z   INFO Send downlink message                    endpoint=traffic method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:47540 request_id=01E20PDKDZQCP0CHFM1W3D1VG6 test_name=TestTraffic url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2994505Z   WARN No cookie hash key configured, generated a random one hash_key=5674A555CE98B8027001DFA7E37C6181DDD3AC5FFAC7C8646979E885E36BA42E0F50EB23EB5358406D2E3A6BC8B3FFF95214AED0D11059E4ED2C8E14F329B884 namespace=web test_name=TestRTT
2020-02-26T12:09:37.2994924Z   WARN No cookie block key configured, generated a random one block_key=5F7B37EBFA095E33328BE9E6B986640B1F952AA4AB9BFFD13F54C5AB418F361B namespace=web test_name=TestRTT
2020-02-26T12:09:37.2995217Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestRTT
2020-02-26T12:09:37.2995448Z   WARN No rights TTL configured                 test_name=TestRTT
2020-02-26T12:09:37.2995797Z   WARN No cluster key configured, generated a random one key=20348a8a71643c45f04f30e03f77824d6c2d25f2f65785744bb9cd32e9e7fcb1 test_name=TestRTT
2020-02-26T12:09:37.2996037Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestRTT
2020-02-26T12:09:37.2996264Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestRTT
2020-02-26T12:09:37.2997144Z   INFO Request handled                          duration=259.906187ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:47540 request_id=01E20PDKDZQCP0CHFM1W3D1VG6 response_size=0 status=200 test_name=TestTraffic url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2997967Z   INFO Send downlink message                    endpoint=traffic method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58718 request_id=01E20PDKPQP8WZBT8RRPCE18GB test_name=TestRTT url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2998824Z   INFO Request handled                          duration=194.390386ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58718 request_id=01E20PDKPQP8WZBT8RRPCE18GB response_size=0 status=200 test_name=TestRTT url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2999500Z   WARN No cookie hash key configured, generated a random one hash_key=5C8598449079E978D61D613DE18D8930D4DDCD381A0762157EF1B7DE824C75D15439AF4479FA644DAD49065D562258A0AA6F207B206ACA22D3386114407EF08F namespace=web test_name=TestPingPong
2020-02-26T12:09:37.3000003Z   WARN No cookie block key configured, generated a random one block_key=C7DA538C00B720F7ED23C1CBD1CDEDAE5AC46DEA1D7ACDA3654D7BDC1BAB577E namespace=web test_name=TestPingPong
2020-02-26T12:09:37.3000252Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestRTT
2020-02-26T12:09:37.3000538Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestPingPong
2020-02-26T12:09:37.3000776Z   WARN No rights TTL configured                 test_name=TestPingPong
2020-02-26T12:09:37.3001139Z   WARN No cluster key configured, generated a random one key=1df4193fc199e5fee3f94439a29f0ded8eef94222aa5749c18ca323ab66efa8d test_name=TestPingPong
2020-02-26T12:09:37.3001390Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestPingPong
2020-02-26T12:09:37.3001586Z concurrent write to websocket connection
2020-02-26T12:09:37.3001724Z goroutine 306 [running]:
2020-02-26T12:09:37.3001926Z runtime/debug.Stack(0x1c78220, 0xc0000b6010, 0xc000362120)
2020-02-26T12:09:37.3002153Z    /opt/hostedtoolcache/go/1.13.8/x64/src/runtime/debug/stack.go:24 +0x9d
2020-02-26T12:09:37.3002715Z go.thethings.network/lorawan-stack/pkg/web/middleware.Recover.func1.1.1(0xc000363808)
2020-02-26T12:09:37.3003206Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/recover.go:36 +0xcd
2020-02-26T12:09:37.3003376Z panic(0x16077c0, 0x1c4e120)
2020-02-26T12:09:37.3003590Z    /opt/hostedtoolcache/go/1.13.8/x64/src/runtime/panic.go:679 +0x1b2
2020-02-26T12:09:37.3003894Z github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0003623c8, 0x1, 0x0, 0x0, 0x0, 0xc0001c96c0, 0x2937820)
2020-02-26T12:09:37.3004126Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:610 +0x612
2020-02-26T12:09:37.3004406Z github.com/gorilla/websocket.(*Conn).WriteMessage(0xc0000e8840, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-02-26T12:09:37.3004655Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:763 +0x23d
2020-02-26T12:09:37.3005261Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns.(*srv).handleTraffic.func2(0x0, 0x0, 0x0, 0x0)
2020-02-26T12:09:37.3005840Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns.go:263 +0x83
2020-02-26T12:09:37.3006098Z github.com/gorilla/websocket.(*Conn).advanceFrame(0xc0000e8840, 0xc0003625d8, 0x18, 0x18)
2020-02-26T12:09:37.3006343Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:929 +0x6de
2020-02-26T12:09:37.3006666Z github.com/gorilla/websocket.(*Conn).NextReader(0xc0000e8840, 0x43a90e, 0xc000362600, 0x45e5e0, 0xc000362648, 0xc000362650)
2020-02-26T12:09:37.3006919Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:980 +0xa0
2020-02-26T12:09:37.3007273Z github.com/gorilla/websocket.(*Conn).ReadMessage(0xc0000e8840, 0x195b220, 0xc0002a8a80, 0xc0000e8840, 0xc00034a500, 0x1cde100, 0xc0001c86c0)
2020-02-26T12:09:37.3007521Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:1064 +0x2f
2020-02-26T12:09:37.3008195Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns.(*srv).handleTraffic(0xc0002c0000, 0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3008779Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns.go:335 +0xcc5
2020-02-26T12:09:37.3009062Z github.com/labstack/echo/v4.(*Echo).add.func1(0x1cf3220, 0xc000114240, 0x450468, 0x511b615ab5)
2020-02-26T12:09:37.3009306Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:500 +0x8a
2020-02-26T12:09:37.3009857Z go.thethings.network/lorawan-stack/pkg/web/middleware.Recover.func1.1(0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3010362Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/recover.go:44 +0x85
2020-02-26T12:09:37.3011061Z go.thethings.network/lorawan-stack/pkg/web/middleware.Log.func1.1(0x1cf3220, 0xc000114240, 0xc00034a0f0, 0xc0008aaa48)
2020-02-26T12:09:37.3011663Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/log.go:44 +0x513
2020-02-26T12:09:37.3011945Z github.com/labstack/echo/v4/middleware.BodyLimitWithConfig.func1.1(0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3012273Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/body_limit.go:87 +0x1c4
2020-02-26T12:09:37.3012897Z go.thethings.network/lorawan-stack/pkg/web/middleware.ID.func1.1(0x1cf3220, 0xc000114240, 0x4, 0x4)
2020-02-26T12:09:37.3013385Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/id.go:47 +0x3bf
2020-02-26T12:09:37.3013672Z github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc00000c1e0, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3013904Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:611 +0x22a
2020-02-26T12:09:37.3014557Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns_test.TestPingPong.func1.1(0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3015165Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1312 +0x48
2020-02-26T12:09:37.3015436Z net/http.HandlerFunc.ServeHTTP(0xc000168030, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3015675Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2007 +0x44
2020-02-26T12:09:37.3015927Z net/http.serverHandler.ServeHTTP(0xc00016a000, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3016164Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2802 +0xa4
2020-02-26T12:09:37.3016385Z net/http.(*conn).serve(0xc0001783c0, 0x1cbc060, 0xc0001c8380)
2020-02-26T12:09:37.3016613Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:1890 +0x875
2020-02-26T12:09:37.3016776Z created by net/http.(*Server).Serve
2020-02-26T12:09:37.3017004Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2928 +0x384
2020-02-26T12:09:37.3017874Z  ERROR Request error                            duration=65.378599ms error=error:pkg/web/middleware:http_recovered (Internal Server Error) method=GET namespace=gatewayserver/io/basicstation panic=concurrent write to websocket connection remote_addr=127.0.0.1:40070 request_id=01E20PDKXJA20PE0D34DV61RFH status=500 test_name=TestPingPong url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.3018139Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.3018709Z --- FAIL: TestPingPong (0.35s)
2020-02-26T12:09:37.3062032Z ##[error]    basicstationlns_test.go:1352: Received server ping
2020-02-26T12:09:37.3066025Z ##[error]    basicstationlns_test.go:1367: Server pong timeout
2020-02-26T12:09:37.3066374Z FAIL
2020-02-26T12:09:37.3067122Z FAIL   go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.937s

Еще одно: https://github.com/TheThingsNetwork/lorawan-stack/pull/2036/checks?check_run_id=475381911

ERROR Request error                            duration=65.476945ms error=error:pkg/web/middleware:http_recovered (Internal Server Error) method=GET namespace=gatewayserver/io/basicstation panic=concurrent write to websocket connection remote_addr=127.0.0.1:41304 request_id=01E26E87M66F6VC188T4N2VFRF status=500 test_name=TestPingPong url=/traffic/eui-0101010101010101
2020/02/28 17:42:05 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe--- FAIL: TestPingPong (0.34s)
##[error]    basicstationlns_test.go:1352: Received server ping
##[error]    basicstationlns_test.go:1367: Server pong timeout
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.934s

Это может быть серьезно, так как это может вызвать панику и в производстве:

goroutine 1682 [running]:
go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xabffad0, 0xcb, 0xcb, 0x0, 0xc, 0x0, 0xcb)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0x1b6
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xa94f380, 0xabffad0, 0xcb, 0xcb, 0x301e5412, 0x0, 0xd5fb1121)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x314
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xa94f380, 0xabffad0, 0x0, 0xcb, 0xa94f300, 0xe7d33501, 0xaf1b110, 0x812373b, 0xaf1b15e, 0xade2ba0)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x46
github.com/gogo/protobuf/proto.Marshal(0x95fa230, 0xa94f380, 0x62, 0x62, 0x62, 0x1, 0x2)
    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x203
go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x95fa230, 0xa94f380, 0xae254a8, 0xaf1b110, 0xae1de34, 0xaf1b110)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x2d
go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x9629a20, 0xad276c0, 0xaf1b110, 0x62, 0x95fa230, 0xa94f380, 0x0, 0x0, 0x0, 0x2, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:164 +0x2d
go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xa96c000, 0x9601d20, 0xaed1c60, 0xade2ba0, 0x14, 0xa9da7a0, 0x0, 0xa94f380, 0x8e1dacd, 0xaed1c40)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:41 +0x1f8
go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xab1c190, 0x9601d20, 0xaed1c60, 0xa8c0c00, 0xae1df01, 0xae1df54)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:659 +0x71
go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xab1c190, 0xa8c0c00, 0xafd5580)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:678 +0x16e
created by go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).Connect
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:451 +0xf74

Они просто продолжают приходить

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:59384: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:59384 test_name=TestGatewayServer

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:59478: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:59478 test_name=TestGatewayServer

panic: runtime error: index out of range [-1]

goroutine 1683 [running]:

go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xc00018e000, 0xcb, 0xcb, 0x0, 0xc, 0xcb)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0xa9

go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xc0007228a0, 0xc00018e000, 0xcb, 0xcb, 0x21, 0xc0000c41e0, 0xc0008a69b0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x2f7

go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xc0007228a0, 0xc00018e000, 0x0, 0xcb, 0xc000722800, 0xffffff01, 0x0, 0x1, 0x1a01e01, 0x7)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x5b

github.com/gogo/protobuf/proto.Marshal(0x1e083c0, 0xc0007228a0, 0xc0008a6ad8, 0xc000937460, 0xc000937460, 0x1921d60, 0xc0008a6af8)

    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x2d6

go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x1e083c0, 0xc0007228a0, 0xc000136930, 0xc0008a6c98, 0xc000136930, 0x62)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x50

go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x1e5be40, 0xc0005e1aa0, 0xc000136930, 0x62, 0x1e083c0, 0xc0007228a0, 0x0, 0x2, 0xc000615630, 0x0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:167 +0x39

go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xc0003fca68, 0x1e18ac0, 0xc0008d5020, 0xc0004f3160, 0x14, 0xc0008be498, 0x0, 0xc0007228a0, 0x146cfac, 0xc0008d1dc0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:41 +0x24f

go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xc000427b90, 0x1e18ac0, 0xc0008d5020, 0xc0003e8270, 0x44f501, 0xc000a00148)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:659 +0xa6

go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xc000427b90, 0xc0003e8270, 0xc00094efc0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:678 +0x239

created by go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).Connect

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:451 +0x12be

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    22.446s

@neoaggelos @KrishnaIyer не могли бы вы скоординировать, кто это рассмотрит? Нам нужно будет исправить разные вещи.

В # 2108 были внесены изменения, которые делают https://github.com/TheThingsNetwork/lorawan-stack/issues/1332#issuecomment -600045502 и https://github.com/TheThingsNetwork/lorawan-stack/issues/1332#issuecomment -598169195 не имеет значения.

Кроме того, даже если это не настоящее решение проблемы, добавление большего тайм-аута здесь https://github.com/TheThingsNetwork/lorawan-stack/pull/2108/files?file-filters%5B%5D=.go# diff -b91e68ddfed2ddb93b15d4b9ab05bde0R1045, похоже, устранил большую часть нестыковки.

В общем, все, что связано со статистикой, рассматривается как часть # 2108 (и она неизменно не дает сбоев).

После множества тестов все вышеперечисленные тесты (и те, которые продолжают нарушать CI несколько раз в день) могут быть сгруппированы (пока) в 4 категории:

1. concurrent write to websocket connection : AFAIK это было исправлено уже в # 2153

2. Expected deadline exceeded with valid auth, but have error:unknown:unknown (context deadline exceeded) Это происходит в тестах Authentication (например, https://github.com/TheThingsNetwork/lorawan-stack/blob/3e7ae2ca9f1180cf3d937acf11bf1637bc29ca80/pkg/gatewayserver/gatewayserver_test.go562#L559 . ошибка неверна не потому, что тест не прошел, а потому, что контекст истекает до того, как аутентификация может быть даже проверена. Это на 100% воспроизводится, если мы вызываем cancel() перед ptc.Link . Предлагаемое исправление: добавьте небольшую задержку перед отменой контекста в строке 562.

3. Expected: 'N+1' 'Actual': 'N' Это происходит из-за того, что тайм-аут недостаточен для того, чтобы интерфейс действительно обработал и учесть сообщение. Механизм обновления статистики недоступен извне, поэтому лучшее, что мы можем сделать, - это подождать достаточно долго. Наивное увеличение тайм-аута теста - неправильное решение. Предлагаемое исправление - повторять попытку проверки до тех пор, пока она не будет успешной, с ограничением максимально допустимого количества повторных попыток.

4. panic: runtime error: index out of range [-1] Это появляется нерегулярно при сохранении статистики соединений шлюза в Redis, и это происходит при маршалинге прото. Код изменился в # 2108, поэтому тесты смотрят там. Похоже на состояние гонки, но мне не удалось определить источник.

К вашему сведению:

1. У нас есть test.Delay для этого
2. См. Https://github.com/TheThingsNetwork/lorawan-stack/blob/da4c2ac4044188c6c28fc6d799d20bbb1918b9fb/pkg/networkserver/networkserver_flow_test.go#L621 -L634 для примера этого подхода (используя также test.Delay

Новенький:

--- FAIL: TestTraffic (0.03s)
    mqtt_test.go:169: Connection timeout
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/mqtt    0.104s

2. Предлагаемое исправление - добавить небольшую задержку перед отменой контекста в строке 562.

Звучит отлично

3. Предлагаемое исправление - повторять попытку проверки до тех пор, пока она не будет успешной, ограничив максимально допустимое количество повторных попыток.

Тоже хорошо звучит

4. panic: runtime error: index out of range [-1] Это появляется нерегулярно при сохранении статистики соединения шлюза в Redis, и это происходит при маршалинге протокола. Код изменился в # 2108, поэтому тесты смотрят там. Похоже на состояние гонки, но мне не удалось определить источник.

Это может быть серьезно. Это может произойти, когда вы пытаетесь маршалировать структуру nil или когда структура изменяется во время маршалинга. Это обе ошибки реализации.

 ERROR Request error                            duration=259.278µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36438 request_id=01E4CWTRYGCFG9R79E322774MJ status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
 ERROR Request error                            duration=136.484µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36440 request_id=01E4CWTRYGVGM6XQ4KMR66T3B7 status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
 ERROR Request error                            duration=1.681437ms error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36442 request_id=01E4CWTRYHK6KM1CCM7G5TYVZM status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
--- FAIL: TestRTT (0.04s)
    --- FAIL: TestRTT/RepeatedTxAck (0.00s)
        doc.go:66: 
            pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1233
            Expected: '2'
            Actual:   '1'
            (Should be equal)
        basicstationlns_test.go:1234: Incorrect Stats entries recorded: 1
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.247s

@neoaggelos какой здесь статус?

После объединения # 2260 и # 2190 мы должны быть в гораздо лучшем положении. Я бы не стал его закрывать, давай дадим Трэвису несколько дней, чтобы посмотреть, как он справится с изменениями.

Новый:

--- FAIL: TestFlow (0.00s)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:111
        Expected: true
        Actual:   false
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:112
        Expected 'nil' to NOT be nil (but it was)!
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:113
        Expected '2562047h47m16.854775807s' to be less than '64ms' (but it wasn't)!
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io 0.020s

Кроме того, снова получил это:

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:45706: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:45706 test_name=TestGatewayServer
 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:45912: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:45912 test_name=TestGatewayServer
panic: runtime error: index out of range [-1]

goroutine 1721 [running]:
go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xc0000aeb60, 0xcb, 0xcb, 0x0, 0xc, 0xcb)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0xa9
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xc000a9e4e0, 0xc0000aeb60, 0xcb, 0xcb, 0x21, 0xc000a9e360, 0xc0004ed6f0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x2f4
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xc000a9e4e0, 0xc0000aeb60, 0x0, 0xcb, 0xc000a9e400, 0x1, 0x1d6dbc0, 0xc000897710, 0x1d8b120, 0x2acf700)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x5b
github.com/gogo/protobuf/proto.Marshal(0x1d611c0, 0xc000a9e4e0, 0x18fa180, 0xc000a9e360, 0xc000881220, 0xc000881340, 0xc00089e1e8)
    /home/runner/go/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x2d6
go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x1d611c0, 0xc000a9e4e0, 0x40c245, 0x1934f20, 0x19396e0, 0x0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/redis/redis.go:49 +0x50
go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x7f731299a868, 0xc0008854c0, 0xc0007ba4d0, 0x69, 0x1d611c0, 0xc000a9e4e0, 0x0, 0x3, 0xc0004edaa8, 0x12bdd43)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/redis/redis.go:167 +0x39
go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set.func1(0x1da3f80, 0xc0008854c0, 0x7f733c4c07d0, 0x0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/redis/registry.go:64 +0x693
github.com/go-redis/redis.(*Pipeline).pipelined(0xc0008854c0, 0xc000897c20, 0x13b77f7, 0x17668e0, 0xc000a932f0, 0x0, 0xc0008854c0)
    /home/runner/go/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:111 +0x40
github.com/go-redis/redis.(*Pipeline).Pipelined(0xc0008854c0, 0xc000897c20, 0xc0008854c0, 0x30, 0x185a180, 0x1, 0xc000897c20)
    /home/runner/go/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:120 +0x35

Последнее кажется, что глубокая копия не сработала?

Разве это не конкретный случай тестирования?

И еще один:

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:39250: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:39250 test_name=TestGatewayServer
--- FAIL: TestGatewayServer (36.35s)
    --- FAIL: TestGatewayServer/DetectDisconnect/grpc#01 (0.77s)
        gatewayserver_test.go:650: Expected deadline exceeded on second connection but have error:unknown:unknown (context deadline exceeded)
FAIL
coverage: 75.7% of statements
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    36.387s

Эту проблему можно решить, снова используя события GS, как мы это делали для тестов Authentication , но это немного сложнее.

Последнее кажется, что глубокая копия не сработала?

Ага, наверное, статус шлюза - не единственное. Но я действительно не могу найти, что еще может вызывать здесь проблему. Может ли быть так, что тесты проводятся в незаконном состоянии (например, одна и та же статистика обновляется двумя источниками одновременно, gc берет на себя и выпускает вещи, которые не должны)?

Эту проблему можно решить, снова используя события GS, как мы это делали для тестов Authentication , но это немного сложнее.

Я думаю, что этот похож на предыдущий; это также выполняет полный Link() в тесте, в то время как контекст уже отменяется до правильной настройки. Таким образом, вам может потребоваться снова удвоить timeout для Link() .

Отметим, что тесты GS занимают уже 36 секунд, что довольно много. Вы можете начать экспериментировать, снижая общее значение GS timeout и увеличивая его в умных местах, таких как связывание и, возможно, обработка трафика.

Ага, наверное, статус шлюза - не единственное. Но я действительно не могу найти, что еще может вызывать здесь проблему. Может ли быть так, что тесты проводятся в незаконном состоянии (например, одна и та же статистика обновляется двумя источниками одновременно, gc берет на себя и выпускает вещи, которые не должны)?

Это вполне могло быть. Вы можете проверить это с помощью -run (для запуска конкретного теста) и -race для обнаружения гонок.

Отметим, что тесты GS занимают уже 36 секунд, что довольно много. Вы можете начать экспериментировать, понижая общее значение GS timeout и увеличивая его в умных местах, таких как связывание и, возможно, обработка трафика.

Вот почему я почти не хочу просто увеличивать таймауты, когда Трэвис терпит неудачу. Я думаю, что тесты были бы очень полезны, если бы мы использовали события вместо тайм-аутов.

Кроме того, я думаю, что уменьшение тайм-аута приведет только к большему количеству нестабильных тестов (я действительно не знаю, как стали использоваться используемые значения, поэтому, возможно, @KrishnaIyer больше подойдет для этого.)

Я лучше использую события, но у нас не может быть событий для неудачных ссылок, если есть предварительные условия, которые не выполняются. Неизбежно, что что-то происходит асинхронно и нет события для отказа, в то время как мы хотим, чтобы он потерпел неудачу, т.е. нам нужно подождать, чтобы убедиться, что это не удалось.

@neoaggelos, у нас их слишком много:

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:56960: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:56960 test_name=TestGatewayServer
panic: runtime error: index out of range [-1]
goroutine 780 [running]:
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.encodeVarintGateway(0xc1e6700, 0xdd, 0xdd, 0x0, 0xc, 0x0, 0xdd)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4410 +0x1b6
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xbe2d580, 0xc1e6700, 0xdd, 0xdd, 0x1d4e291c, 0x0, 0xd64ead82)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4298 +0x4b6
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xbe2d580, 0xc1e6700, 0x0, 0xdd, 0xbe2d500, 0x909ab01, 0x42, 0xbfa92d8, 0xbc536d0, 0x906ef60)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1503 +0x46
github.com/gogo/protobuf/proto.Marshal(0x95e0130, 0xbe2d580, 0x0, 0x4, 0x55555555, 0x7, 0x10)
    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x203
go.thethings.network/lorawan-stack/v3/pkg/redis.MarshalProto(0x95e0130, 0xbe2d580, 0xc1ceaa8, 0x8050b39, 0x926a420, 0x926eb60)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x3d
go.thethings.network/lorawan-stack/v3/pkg/redis.SetProto(0xe7ce4a88, 0xc2d88d0, 0xbdefb20, 0x69, 0x95e0130, 0xbe2d580, 0x0, 0x0, 0xffffffe1, 0x0, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:167 +0x2d
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set.func1(0x9607320, 0xc2d88d0, 0xf7f773ec, 0x20)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:64 +0x597
github.com/go-redis/redis.(*Pipeline).pipelined(0xc2d88d0, 0xc1cea40, 0x8dbb048, 0x90bc700, 0xbfa9510, 0xc0b4f60, 0xc2d88d0)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:111 +0x32
github.com/go-redis/redis.(*Pipeline).Pipelined(0xc2d88d0, 0xc1cea40, 0xc2d88d0, 0x20, 0x91763a0, 0x1, 0xc1cea40)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:120 +0x29
github.com/go-redis/redis.(*Client).Pipelined(0xbdbb260, 0xc1cea40, 0x92a3af7, 0x1c, 0xa1e9ca0, 0x14, 0x1d4e291c)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/redis.go:457 +0x3c
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xbddf928, 0x95e8160, 0xc1c4520, 0xbc3ff40, 0x14, 0xbdda390, 0x0, 0xbe2d580, 0x9010000, 0x0, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:51 +0x156
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xbd4d270, 0xbd31000, 0x9010000, 0xc24ade0, 0x0)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:883 +0x9e
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xbd4d270, 0xbd31000, 0xbfcba80)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:673 +0x91
created by go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).Connect
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:458 +0xff1

Пожалуйста, разрешите

Новый;

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:43472: use of closed network connection gateway_uid=eui-aaee000000000000@foo-tenant namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:43472 tenant_id=foo-tenant test_name=TestGatewayServer
--- FAIL: TestGatewayServer (36.36s)
    --- FAIL: TestGatewayServer/DetectDisconnect/grpc#01 (0.77s)
        gatewayserver_test.go:653: Expected deadline exceeded on second connection but have error:unknown:unknown (context deadline exceeded)
FAIL
FAIL    go.thethings.network/lorawan-stack/v3/pkg/gatewayserver 36.408s

Это требует точной настройки тайм-аута, используемого в этом конкретном месте.

ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:41304: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:41304 test_name=TestGatewayServer
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x144bd95]

goroutine 90 [running]:
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io.(*Connection).Context(...)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/io.go:162
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.(*srv).gc.func1(0x18ff0a0, 0xc00097dcb8, 0x1679ba0, 0xc0005ac000, 0x1)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:425 +0xf5
sync.(*Map).Range(0xc0001aaf98, 0xc0004a3f58)
    /opt/hostedtoolcache/go/1.14.3/x64/src/sync/map.go:333 +0x1c5
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.(*srv).gc(0xc0001aaf20)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:421 +0xdf
created by go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.Serve
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:72 +0x1e2

2624 исправил последний комментарий.

--- FAIL: TestSubBandEIRPOverride (0.00s)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:476
        Expected: true
        Actual:   false
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:477
        Expected: '1'
        Actual:   '0'
        (Should be equal)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:478
        Expected '2562047h47m16.854775807s' to be less than '64ms' (but it wasn't)!
FAIL
FAIL    go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io  0.087s

Это похоже на переполнение int, что довольно страшно. Убедитесь, что это вызвано неясностью теста, а не тем, что может произойти в производственной среде.

Другой:

2020-07-13T11:28:13.8444055Z  ERROR Request error                            duration=218.2µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35116 request_id=01ED3YVKE5E4MZCJ6TJDV5KMH6 status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8444793Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8445194Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8446392Z  ERROR Request error                            duration=138.4µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35118 request_id=01ED3YVKE5Z7A5CRAJWN5ARF1P status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8447014Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8447418Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8449101Z  ERROR Request error                            duration=137.8µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35120 request_id=01ED3YVKE6SKB9TPP9BN74YD74 status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8449873Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8453477Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8454049Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8454795Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8455937Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8456759Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8457157Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8459526Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8460320Z --- FAIL: TestRTT (0.15s)
2020-07-13T11:28:13.8461043Z     --- FAIL: TestRTT/FollowUpTxAck (0.13s)
2020-07-13T11:28:13.8461263Z         doc.go:66: 
2020-07-13T11:28:13.8461807Z             /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1235
2020-07-13T11:28:13.8462139Z             Expected: '1'
2020-07-13T11:28:13.8462441Z             Actual:   '0'
2020-07-13T11:28:13.8462601Z             (Should be equal)
2020-07-13T11:28:13.8462766Z         basicstationlns_test.go:1236: Incorrect Stats entries recorded: 0
2020-07-13T11:28:13.8462920Z FAIL
2020-07-13T11:28:13.8463066Z coverage: 70.2% of statements
2020-07-13T11:28:13.8463424Z FAIL   go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/basicstationlns  0.545s

Последний комментарий очень старый, закрывающий. При необходимости откройте заново.