外观相关：

  WARN No cookie hash key configured, generated a random one hash_key=2FECCF81EFFEF0C8110CCBE2BE8B23C26E337279E62756ACFE16375BB6897E3EBA5759D84E150BB7E17A99EBEDE387B096EE41BB4DD44BC19C9609EFC8328F06 namespace=web test_name=TestGatewayServer
  WARN No cookie block key configured, generated a random one block_key=822E142EF552CFEEC7E546D22F7E36ED26CBCE6D79A8C4415898EC242919D649 namespace=web test_name=TestGatewayServer
  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer
  WARN No rights TTL configured                 test_name=TestGatewayServer
  WARN No cluster key configured, generated a random one key=eaadd899806bb4c13450d4257d7c6110c592da1b724a8978068a8f9187eb53d9 test_name=TestGatewayServer
  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer
  INFO Finished unary call                      duration=5.012417ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H70FPFWJRBCCZAJTAS4W test_name=TestGatewayServer
  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:41030 request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  INFO Finished streaming call                  duration=245.644341ms error=error:unknown:unknown (context canceled) error_correlation_id=1257f760d52f4150a71f3ee60703974a grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  WARN Finished unary call                      duration=604.26µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) error_correlation_id=46b6a81d7eeb4c089f7e3c4eedc841f5 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78GTQJD37YRDRTY6Y9V test_name=TestGatewayServer
  WARN Finished unary call                      duration=503.568µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) error_correlation_id=eb72c1e0978a4702b6bc36439c272e00 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78JP2MV2CWX8NVV1848 test_name=TestGatewayServer
  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:41030 request_id=01DPN4H70SHSF2C64N1HDPVRJJ test_name=TestGatewayServer
  INFO Finished unary call                      duration=113.698µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=efc5641d14db4e939380d3daefa580bb error_name=identifiers error_namespace=pkg/ttnpb grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78M598DAMETTSJ2FZBP test_name=TestGatewayServer
  INFO Finished unary call                      duration=98.372µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=55e3f96c3e1e4744a74efa5eb9fd426a error_name=identifiers error_namespace=pkg/ttnpb grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H78NJW6NKZKT4H6YX47J test_name=TestGatewayServer
  INFO Finished unary call                      duration=765.197µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=b07dd87e51824fc29bba3c808e38a183 error_name=not_connected error_namespace=pkg/gatewayserver grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7GQBS44PQD1EWMJ90MV test_name=TestGatewayServer
  INFO Finished unary call                      duration=863.468µs grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H7GS4ZEYAXTN31QZ4WJ9 test_name=TestGatewayServer
  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:41042 request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
  INFO Finished unary call                      duration=580.106µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7RW3SJ569D1GQZ7HF3B test_name=TestGatewayServer
  INFO Finished unary call                      duration=655.422µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7RY0P6F19NTX7W9P5ZE test_name=TestGatewayServer
  INFO Finished unary call                      duration=444.661µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01DPN4H7S0143EYQES1ABD6GM6 test_name=TestGatewayServer
  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:41042 request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
  INFO Finished streaming call                  duration=263.036878ms error=error:unknown:unknown (context canceled) error_correlation_id=06aac79e327f4b86b2dd0e8520fb2a5e grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01DPN4H7GT1NBTJKJCGEMN4QKT test_name=TestGatewayServer
--- FAIL: TestGatewayServer (1.86s)
    --- FAIL: TestGatewayServer/Traffic/grpc (0.52s)
        --- FAIL: TestGatewayServer/Traffic/grpc/Upstream (0.26s)
            --- FAIL: TestGatewayServer/Traffic/grpc/Upstream/OneValidLoRa (0.00s)
                gatewayserver_test.go:812: gs.up.forward event published
                doc.go:66: 
                    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver_test.go:878
                    Expected: '1'
                    Actual:   '0'
                    (Should be equal)
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    1.915s

看来我又找到了一个：

  WARN No cookie hash key configured, generated a random one hash_key=AF8CAC67C022D6738C91ACEC222EB0FEEB1EED3075DCC91EB38660FB9C8C05B72CD2CCBA4A6CE81DDDD937D6C1C86AE10ACE17BB57D6F44BEC7114FC36055BD7 namespace=web test_name=TestGatewayServer

  WARN No cookie block key configured, generated a random one block_key=D0DA3B8FF79A7F4A35AF7199A95465A5DA4E99D5EB7B8FEEF7A2A4CD3B4D2BC8 namespace=web test_name=TestGatewayServer

  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer

  WARN No rights TTL configured                 test_name=TestGatewayServer

  WARN No cluster key configured, generated a random one key=f6d812ec6397f4cd4b12e7725ca88d99334ada3fe94e5b1181b5b8101f48d66d test_name=TestGatewayServer

  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer

  INFO Finished unary call                      duration=2.932065ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1H2JKQ3YDTN36EFNCV0CS56 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Finished streaming call                  duration=247.357924ms error=context deadline exceeded grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Link failed                              error=rpc error: code = DeadlineExceeded desc = context deadline exceeded gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:36532 request_id=01E1H2JKQ7T0A088T8FZY72T0W test_name=TestGatewayServer

  WARN Task failed                              error=mux: listener closed invocation=1 namespace=gatewayserver task_id=serve_mqtt/:1882 test_name=TestGatewayServer

--- FAIL: TestGatewayServer (4.66s)

    --- FAIL: TestGatewayServer/Authenticate/grpc (0.26s)

        --- FAIL: TestGatewayServer/Authenticate/grpc/ValidIDAndKey (0.26s)

            gatewayserver_test.go:559: Expected deadline exceeded with valid auth, but have error:unknown:unknown (context deadline exceeded)

  WARN Task failed                              error=read udp [::]:1700: use of closed network connection invocation=1 namespace=gatewayserver task_id=serve_udp/:1700 test_name=TestGatewayServer

  WARN Task failed                              error=http: Server closed invocation=1 namespace=gatewayserver task_id=serve_basicstation/:1887 test_name=TestGatewayServer

FAIL

  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestGatewayServer

coverage: 41.4% of statements

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    4.714s

这次还有一个basicstation ：

  WARN No cookie hash key configured, generated a random one hash_key=686D148310C51B627BF34558528A47B84D1001620F48956E6B042A7CCBC82CC5B3F7AA873BA65BD7C7B43AEC7DA75CEE8F82761886A14AAF7182B394E1513879 namespace=web test_name=TestGatewayServer

  WARN No cookie block key configured, generated a random one block_key=A52935E61E7592777715B8CB46FAF8E3221A0DAC214BDD6A23607E668F3C7BAA namespace=web test_name=TestGatewayServer

  WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestGatewayServer

  WARN No rights TTL configured                 test_name=TestGatewayServer

  WARN No cluster key configured, generated a random one key=965535d0e2ca0d5883c0e8c0fa0eef8fe4d9f97f25b7458e77dfa6b3cd15d5e0 test_name=TestGatewayServer

  INFO Listening for connections                address=:9187 namespace=grpc protocol=gRPC test_name=TestGatewayServer

  INFO Finished unary call                      duration=14.054791ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9X43KPMHN775400EDVSY test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60682 request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  INFO Finished streaming call                  duration=239.767717ms error=context canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60682 request_id=01E1KP9X4J3M4X91F397PTWGHG test_name=TestGatewayServer

  WARN Finished unary call                      duration=566.11µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) error_correlation_id=35b22abf384e4bf69c03e09931b4e689 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XC31MX21ZTFNCYARFMD test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Finished unary call                      duration=444.016µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) error_correlation_id=248c07bcb42b45cd951f72d683cf3804 error_name=no_gateway_rights error_namespace=pkg/auth/rights grpc_code=PermissionDenied grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XC5152VJWPBT2F7TB8E test_name=TestGatewayServer uid=invalid-gateway

  INFO Finished unary call                      duration=173.105µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=eed7ec8dbca0442d8ab04037d4f1b217 error_name=identifiers error_namespace=pkg/ttnpb field=gateway_id grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs name=GatewayIdentifiersValidationError namespace=grpc reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$" request_id=01E1KP9XC71EYJ5KJSGZWQZYJZ test_name=TestGatewayServer

  INFO Finished unary call                      duration=81.663µs error=error:pkg/ttnpb:identifiers (invalid identifiers) error_cause=error:pkg/errors:validation (invalid `gateway_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$") error_correlation_id=85ad95d2b6d046fc9fd1ab3c5aa3d17b error_name=identifiers error_namespace=pkg/ttnpb field=gateway_id grpc_code=InvalidArgument grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs name=GatewayIdentifiersValidationError namespace=grpc reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$" request_id=01E1KP9XC8JVRETETN2B9HYEAE test_name=TestGatewayServer

  INFO Finished unary call                      duration=923.455µs grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XMAVY68WF94PTMENQXW test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60692 request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.109666ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XWAD7DW8MBEZGR97BXR test_name=TestGatewayServer

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  WARN Finished streaming call                  duration=257.213855ms error=error:pkg/gatewayserver:new_connection (new connection from same gateway) error_correlation_id=b652b79bda0f404c875e3da58601241c error_name=new_connection error_namespace=pkg/gatewayserver grpc_code=Aborted grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60692 request_id=01E1KP9XMCPBPQ2Z859GA9PGJ8 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60694 request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Finished streaming call                  duration=254.052548ms error=error:unknown:unknown (context canceled) error_correlation_id=8ce5108c56c34e27bce9f005609d7348 grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9XWCSB4Q9YMKQSE7KT4D test_name=TestGatewayServer

  INFO Finished unary call                      duration=600.622µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=8106732c22904b4b90f56163a78c7b7e error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YCCSKMR0Q62G1N6A7X8 test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.026905ms grpc_method=GetConcentratorConfig grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9YCEVSHC8H81J9H3NER2 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=602.597µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMHMZGJ83XM2MHZVJND test_name=TestGatewayServer

  INFO Finished unary call                      duration=427.272µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMK5Q9HJQTM6GWAK7YA test_name=TestGatewayServer

  INFO Finished unary call                      duration=357.239µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMN0Y39J7P2GVYW9SD3 test_name=TestGatewayServer

  INFO Finished unary call                      duration=599.939µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YMPJWGMNBS7JJ0M0HFK test_name=TestGatewayServer

  INFO Finished unary call                      duration=558.232µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWTVE97R1D6SQVWVR54 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=411.31µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWWECN00MWPSZB272RE test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=551.542µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YWXR9BVHT56K1DN4GB0 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=1.068656ms grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YX8TEAG9S680RQ6DHB6 test_name=TestGatewayServer

  INFO Send downlink message                    gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=639.377µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9YXA0F3EV7EMQ27NVXW7 test_name=TestGatewayServer

  INFO Finished streaming call                  duration=541.847903ms error=error:unknown:unknown (context canceled) error_correlation_id=0612e7a48e594d33abce482385099ca8 grpc_code=Canceled grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=grpc request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 grpc_method=LinkGateway grpc_service=ttn.lorawan.v3.GtwGs namespace=gatewayserver/io/grpc protocol=grpc remote_addr=127.0.0.1:60698 request_id=01E1KP9YCF6FZJBKZHH460NERV test_name=TestGatewayServer

  INFO Finished unary call                      duration=584.252µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=b4f5f60b8b0d40e7b3e9d3a46a57688d error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KP9Z5EJ4EW0588SY6DWM04 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40722 test_name=TestGatewayServer

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40724 test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-aaee000000000000`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40726 test_name=TestGatewayServer uid=eui-aaee000000000000

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40728 test_name=TestGatewayServer uid=invalid-gateway

  WARN Failed to setup connection               error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `invalid-gateway`) namespace=gatewayserver/io/mqtt remote_addr=127.0.0.1:40730 test_name=TestGatewayServer uid=invalid-gateway

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  WARN Error receiving packet                   client_id=127.0.0.1:40732-3419925607 error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection namespace=gatewayserver/io/mqtt test_name=TestGatewayServer username=eui-aaee000000000000

  WARN Error when reading packet                error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:40732: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40732 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40734 test_name=TestGatewayServer

  INFO Finished unary call                      duration=604.953µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=f2068f9ebfba4cb28eb76ce2238964a0 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA0XH5P41MNQC1Y5ZGSEY test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=966.471µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15N9RXZWA8SK7QACS4J test_name=TestGatewayServer

  INFO Finished unary call                      duration=635.035µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15QT8H8XENDG6TP87W0 test_name=TestGatewayServer

  INFO Finished unary call                      duration=347.509µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15VA5BHCW8YXNNWJ03G test_name=TestGatewayServer

  INFO Finished unary call                      duration=395.257µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA15W4T4NZEA7Q4G3F1ZT test_name=TestGatewayServer

  INFO Finished unary call                      duration=626.299µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1DZB7G1ENE86N3PHAEK test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=437.273µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E14D8M1QE5NGCM6RPV test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=385.207µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E2HE36ZE2MT0FQ1P4F test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=691.985µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E4ZRHDHTPVVHNMNJ5P test_name=TestGatewayServer

  INFO Publish downlink message                 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=505.736µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1E67BX01H78KFQR0Q2S test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:40738 test_name=TestGatewayServer

  INFO Finished unary call                      duration=714.237µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=545034895c8f403498fad3634aed83e5 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA1P8T99QZMEABNQJ96CG test_name=TestGatewayServer

  INFO Connected                                gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  WARN Connect unregistered gateway             gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Connected                                gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=BBFF000000000000 gateway_uid=eui-bbff000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=521.446µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=6d78bc53308141b3959bfcf87328bf2f error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3DSMQTCPYG6DGVMEBJV test_name=TestGatewayServer

  INFO Connected                                gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=496.765µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NW2S7PRDJB4ECGKDRR test_name=TestGatewayServer

  INFO Finished unary call                      duration=343.624µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NYY200GY8YFS97VYXB test_name=TestGatewayServer

  INFO Finished unary call                      duration=612.383µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3NZCP9ZPVN1GKB55QVR test_name=TestGatewayServer

  INFO Finished unary call                      duration=511.471µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3P1GB5GGMN4D3HF7DE2 test_name=TestGatewayServer

  INFO Finished unary call                      duration=557.428µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y4FYVCE6CMXR0FXF1R test_name=TestGatewayServer

  INFO Finished unary call                      duration=384.58µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y5WXMSFQEM11XH81A2 test_name=TestGatewayServer

  INFO Finished unary call                      duration=334.846µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y6HVCGQKC8K7GZZSSR test_name=TestGatewayServer

  INFO Finished unary call                      duration=373.463µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y8Y57REGBHVYB4RG0C test_name=TestGatewayServer

  INFO Finished unary call                      duration=323.76µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA3Y9NGDB4ZWEJVBJXF8V test_name=TestGatewayServer

  INFO Disconnected                             gateway_eui=AAEE000000000000 gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/udp protocol=udp test_name=TestGatewayServer

  INFO Finished unary call                      duration=586.868µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=df3fd6a0d98040a8822291fccd8f7767 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA4J16P0TPSZEXATTBV5Z test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.667636ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48180 request_id=01E1KPA4J3DTQVAKNSS4M4N039 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.990103ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48182 request_id=01E1KPA4T3CKACW9SXT4FTEKK5 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  WARN Disconnect existing connection           gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=256.949369ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48184 request_id=01E1KPA5A4VAJVYJK09146F8QQ test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Request handled                          duration=255.937262ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48186 request_id=01E1KPA5J4CJCZ0A3DJ05ZVPSY test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Finished unary call                      duration=511.423µs error=error:pkg/gatewayserver:not_connected (gateway `eui-aaee000000000000` not connected) error_correlation_id=48680752385f4f7bab0314f8eb2e3225 error_name=not_connected error_namespace=pkg/gatewayserver gateway_uid=eui-aaee000000000000 grpc_code=NotFound grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA625W72DV35C84B7CFTF test_name=TestGatewayServer

  INFO Connected                                gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Finished unary call                      duration=458.982µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA627JZH5S5WZBTZXX95N test_name=TestGatewayServer

  INFO Finished unary call                      duration=288.972µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA6286VHRWHBM6ZEN8HW0 test_name=TestGatewayServer

  INFO Finished unary call                      duration=293.552µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA62AKSZ3S2M3XD57QKZ1 test_name=TestGatewayServer

  INFO Finished unary call                      duration=288.987µs grpc_method=GetGatewayConnectionStats grpc_service=ttn.lorawan.v3.Gs namespace=grpc request_id=01E1KPA62CCZRXH7JN8XX0YY3X test_name=TestGatewayServer

  INFO Request handled                          duration=6.202961ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G response_size=0 status=200 test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  INFO Disconnected                             gateway_uid=eui-aaee000000000000 method=GET namespace=gatewayserver/io/basicstation protocol=basicstation remote_addr=127.0.0.1:48190 request_id=01E1KPA6261VZ0WG7JTTK8MT7G test_name=TestGatewayServer url=/traffic/eui-aaee000000000000

  WARN Task failed                              error=read udp [::]:1700: use of closed network connection invocation=1 namespace=gatewayserver task_id=serve_udp/:1700 test_name=TestGatewayServer

  WARN Task failed                              error=mux: listener closed invocation=1 namespace=gatewayserver task_id=serve_mqtt/:1882 test_name=TestGatewayServer

  WARN Task failed                              error=http: Server closed invocation=1 namespace=gatewayserver task_id=serve_basicstation/:1887 test_name=TestGatewayServer

--- FAIL: TestGatewayServer (11.77s)

    --- FAIL: TestGatewayServer/Traffic/basicstation (0.26s)

        --- FAIL: TestGatewayServer/Traffic/basicstation/Upstream (0.01s)

            --- FAIL: TestGatewayServer/Traffic/basicstation/Upstream/OneValidFSK (0.00s)

                gatewayserver_test.go:857: gs.up.forward event published

                doc.go:66: 

                    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver_test.go:923

                    Expected: '2'

                    Actual:   '1'

                    (Should be equal)

  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestGatewayServer

FAIL

coverage: 78.1% of statements

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    11.830s

还有一个，参考文献#2003

2020-02-26T12:09:37.2950440Z   WARN No cookie hash key configured, generated a random one hash_key=2F2F7C66987C8D056542955EF87C2E847403300EB8902617A2F10CAB64A0BF43E65A02381BB8E70DD428A5BBBF6A5CA9D97F57D086553B7AC354335BF2CC2557 namespace=web test_name=TestClientTokenAuth
2020-02-26T12:09:37.2951747Z   WARN No cookie block key configured, generated a random one block_key=7A6A1716EA81C20770D8AE6606279EE50AA66BCEDA4BF45960D21F7457DD1908 namespace=web test_name=TestClientTokenAuth
2020-02-26T12:09:37.2952326Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestClientTokenAuth
2020-02-26T12:09:37.2952700Z   WARN No rights TTL configured                 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2953277Z   WARN No cluster key configured, generated a random one key=4828de4e338f805fd9872013c35c32ab574805eeaabeead7b302be4142ad58fa test_name=TestClientTokenAuth
2020-02-26T12:09:37.2953659Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestClientTokenAuth
2020-02-26T12:09:37.2954826Z   INFO Finished unary call                      duration=25.801µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBNVNTJ15PBK9BHE31Y test_name=TestClientTokenAuth
2020-02-26T12:09:37.2955957Z   INFO Request handled                          duration=1.543711ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58330 request_id=01E20PDKBMZ3TNAGG60ZA3C1FR response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2957459Z   INFO Finished unary call                      duration=11.5µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBPXGSGGN7JH07XH7J6 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2958673Z   INFO Request handled                          duration=729.705µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58332 request_id=01E20PDKBPM7HAKKRQHP2P2NEG response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2959790Z   INFO Finished unary call                      duration=12.9µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBQVGBE3GQGEAHEYBSE test_name=TestClientTokenAuth
2020-02-26T12:09:37.2960871Z   WARN Failed to connect                        endpoint=traffic error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-0101010101010101`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58334 request_id=01E20PDKBQXW3VQ1QATA9B6BMS test_name=TestClientTokenAuth uid=eui-0101010101010101 url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2963193Z   INFO Request handled                          duration=630.705µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-0101010101010101`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58334 request_id=01E20PDKBQXW3VQ1QATA9B6BMS status=403 test_name=TestClientTokenAuth uid=eui-0101010101010101 url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2964592Z   INFO Request handled                          duration=273.002µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58336 request_id=01E20PDKBRCEFWGBN88QV1VFDT response_size=0 status=200 test_name=TestClientTokenAuth url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2965683Z   INFO Finished unary call                      duration=12.1µs grpc_method=ListRights grpc_service=ttn.lorawan.v3.GatewayAccess namespace=grpc request_id=01E20PDKBST5716W4V4RBKHQG8 test_name=TestClientTokenAuth
2020-02-26T12:09:37.2967665Z   WARN Failed to connect                        endpoint=traffic error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-1122334455667788`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58338 request_id=01E20PDKBRBFF9XF21D1WERF9R test_name=TestClientTokenAuth uid=eui-1122334455667788 url=/traffic/eui-1122334455667788
2020-02-26T12:09:37.2971339Z   INFO Request handled                          duration=525.004µs error=error:pkg/auth/rights:no_gateway_rights (no rights for gateway `eui-1122334455667788`) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58338 request_id=01E20PDKBRBFF9XF21D1WERF9R status=403 test_name=TestClientTokenAuth uid=eui-1122334455667788 url=/traffic/eui-1122334455667788
2020-02-26T12:09:37.2972518Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestClientTokenAuth
2020-02-26T12:09:37.2973051Z   WARN No cookie hash key configured, generated a random one hash_key=D834925B52BC94223C4E7CA1FE9B6C83BE9EA2EBB03F5724EEBEAE1B05454FE5D5299CBE7984154F63003120EB536D2789904F1598614566E2F016617E672B0C namespace=web test_name=TestDiscover
2020-02-26T12:09:37.2973480Z   WARN No cookie block key configured, generated a random one block_key=D93A7217E0F0D4278F38BB522C167599F5130449C112D8FFECB3F3B0D18CC318 namespace=web test_name=TestDiscover
2020-02-26T12:09:37.2973841Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestDiscover
2020-02-26T12:09:37.2974085Z   WARN No rights TTL configured                 test_name=TestDiscover
2020-02-26T12:09:37.2974437Z   WARN No cluster key configured, generated a random one key=4185bee11ced6f7d9d2b5fc0db6b47b7c6a45b26fe42bdfb0df843e0916e659b test_name=TestDiscover
2020-02-26T12:09:37.2974689Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestDiscover
2020-02-26T12:09:37.2974992Z   INFO Request handled                          duration=300ns error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60578 request_id=01E20PDKCFWJTSJHTP136QEKSJ status=404 test_name=TestDiscover url=/api/v3/gs/io/basicstation/discover
2020-02-26T12:09:37.2976679Z   INFO Request handled                          duration=300ns error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60580 request_id=01E20PDKCFR039DECP6DKE2F9X status=404 test_name=TestDiscover url=/router-info/eui-0101010101010101
2020-02-26T12:09:37.2977633Z  ERROR Request error                            duration=205.102µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60582 request_id=01E20PDKCG80X7HQ9Z2SZGE6F5 status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2977896Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2978106Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2979469Z  ERROR Request error                            duration=127.101µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60584 request_id=01E20PDKCGY7M3M2AG09S5NBGJ status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2979843Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2980047Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2980896Z  ERROR Request error                            duration=134.501µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60586 request_id=01E20PDKCH7G8V10W6T3PQJXED status=500 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2981164Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2981367Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2982281Z   INFO Request handled                          duration=388.303µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60588 request_id=01E20PDKCH891Y49M37TWCA7ZC status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2982520Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2982718Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2984207Z   INFO Request handled                          duration=468.504µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60590 request_id=01E20PDKCJ9QBPXRQ58WB83ZVY status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2984478Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2984680Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2985505Z   INFO Request handled                          duration=476.904µs error=error:pkg/basicstation:format (invalid format) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60592 request_id=01E20PDKCJ9C6FCBRB0J2KA0A2 status=400 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2985738Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.2985961Z 2020/02/26 12:09:23 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-02-26T12:09:37.2986810Z   INFO Request handled                          duration=249.801µs method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:60594 request_id=01E20PDKCJT4XT2TSCKERMZTMR response_size=0 status=200 test_name=TestDiscover url=/router-info
2020-02-26T12:09:37.2987372Z   WARN No cookie hash key configured, generated a random one hash_key=945021EB77E419898A7155BA2A452690636593668F47C7978EC16EF42E92B4F8996CDA0C156F1E9B597FBCDC1A55059FFD4B80AC69ABF2F8376C13505E92CC80 namespace=web test_name=TestVersion
2020-02-26T12:09:37.2987798Z   WARN No cookie block key configured, generated a random one block_key=A26709BEB30BAE31B780BA0BA9AB61D490FF9B79A3055C38A0EAB27878373CF1 namespace=web test_name=TestVersion
2020-02-26T12:09:37.2988101Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestVersion
2020-02-26T12:09:37.2988495Z   WARN No rights TTL configured                 test_name=TestVersion
2020-02-26T12:09:37.2989583Z   WARN No cluster key configured, generated a random one key=094e698636bc15731746b7dc6d719928250b0248e61f0cdebcf68f9ec58ecc14 test_name=TestVersion
2020-02-26T12:09:37.2989896Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestVersion
2020-02-26T12:09:37.2990391Z   WARN No cookie hash key configured, generated a random one hash_key=1885A3A4F1CAF4364173D33E8129A2FC53AE80F3823E5A48207D13FECA63A9F3358320589A74FA343CDEF6BD08A64662664844967168AC6A04C9982D682F9230 namespace=web test_name=TestTraffic
2020-02-26T12:09:37.2990867Z   WARN No cookie block key configured, generated a random one block_key=0878491E0147A2FC43AEA720E91EEF8326EC92C3985B315691524F4B1483B0D4 namespace=web test_name=TestTraffic
2020-02-26T12:09:37.2991149Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestTraffic
2020-02-26T12:09:37.2992220Z   INFO Request handled                          duration=2.911722ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:53514 request_id=01E20PDKD7X8JBYBCA6QTNT2RP response_size=0 status=200 test_name=TestVersion url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2992502Z   WARN No rights TTL configured                 test_name=TestTraffic
2020-02-26T12:09:37.2992863Z   WARN No cluster key configured, generated a random one key=6807b19d6e375f3161d8cd6ae07e1e58bca0f6625d530c819fd818a8b6aebb07 test_name=TestTraffic
2020-02-26T12:09:37.2993108Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestTraffic
2020-02-26T12:09:37.2993993Z   INFO Send downlink message                    endpoint=traffic method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:47540 request_id=01E20PDKDZQCP0CHFM1W3D1VG6 test_name=TestTraffic url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2994505Z   WARN No cookie hash key configured, generated a random one hash_key=5674A555CE98B8027001DFA7E37C6181DDD3AC5FFAC7C8646979E885E36BA42E0F50EB23EB5358406D2E3A6BC8B3FFF95214AED0D11059E4ED2C8E14F329B884 namespace=web test_name=TestRTT
2020-02-26T12:09:37.2994924Z   WARN No cookie block key configured, generated a random one block_key=5F7B37EBFA095E33328BE9E6B986640B1F952AA4AB9BFFD13F54C5AB418F361B namespace=web test_name=TestRTT
2020-02-26T12:09:37.2995217Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestRTT
2020-02-26T12:09:37.2995448Z   WARN No rights TTL configured                 test_name=TestRTT
2020-02-26T12:09:37.2995797Z   WARN No cluster key configured, generated a random one key=20348a8a71643c45f04f30e03f77824d6c2d25f2f65785744bb9cd32e9e7fcb1 test_name=TestRTT
2020-02-26T12:09:37.2996037Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestRTT
2020-02-26T12:09:37.2996264Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestRTT
2020-02-26T12:09:37.2997144Z   INFO Request handled                          duration=259.906187ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:47540 request_id=01E20PDKDZQCP0CHFM1W3D1VG6 response_size=0 status=200 test_name=TestTraffic url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2997967Z   INFO Send downlink message                    endpoint=traffic method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58718 request_id=01E20PDKPQP8WZBT8RRPCE18GB test_name=TestRTT url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2998824Z   INFO Request handled                          duration=194.390386ms method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:58718 request_id=01E20PDKPQP8WZBT8RRPCE18GB response_size=0 status=200 test_name=TestRTT url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.2999500Z   WARN No cookie hash key configured, generated a random one hash_key=5C8598449079E978D61D613DE18D8930D4DDCD381A0762157EF1B7DE824C75D15439AF4479FA644DAD49065D562258A0AA6F207B206ACA22D3386114407EF08F namespace=web test_name=TestPingPong
2020-02-26T12:09:37.3000003Z   WARN No cookie block key configured, generated a random one block_key=C7DA538C00B720F7ED23C1CBD1CDEDAE5AC46DEA1D7ACDA3654D7BDC1BAB577E namespace=web test_name=TestPingPong
2020-02-26T12:09:37.3000252Z   WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe namespace=grpc test_name=TestRTT
2020-02-26T12:09:37.3000538Z   WARN No static assets found in any search path namespace=web search_paths=[] test_name=TestPingPong
2020-02-26T12:09:37.3000776Z   WARN No rights TTL configured                 test_name=TestPingPong
2020-02-26T12:09:37.3001139Z   WARN No cluster key configured, generated a random one key=1df4193fc199e5fee3f94439a29f0ded8eef94222aa5749c18ca323ab66efa8d test_name=TestPingPong
2020-02-26T12:09:37.3001390Z   INFO Listening for connections                address=:0 namespace=grpc protocol=gRPC test_name=TestPingPong
2020-02-26T12:09:37.3001586Z concurrent write to websocket connection
2020-02-26T12:09:37.3001724Z goroutine 306 [running]:
2020-02-26T12:09:37.3001926Z runtime/debug.Stack(0x1c78220, 0xc0000b6010, 0xc000362120)
2020-02-26T12:09:37.3002153Z    /opt/hostedtoolcache/go/1.13.8/x64/src/runtime/debug/stack.go:24 +0x9d
2020-02-26T12:09:37.3002715Z go.thethings.network/lorawan-stack/pkg/web/middleware.Recover.func1.1.1(0xc000363808)
2020-02-26T12:09:37.3003206Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/recover.go:36 +0xcd
2020-02-26T12:09:37.3003376Z panic(0x16077c0, 0x1c4e120)
2020-02-26T12:09:37.3003590Z    /opt/hostedtoolcache/go/1.13.8/x64/src/runtime/panic.go:679 +0x1b2
2020-02-26T12:09:37.3003894Z github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0003623c8, 0x1, 0x0, 0x0, 0x0, 0xc0001c96c0, 0x2937820)
2020-02-26T12:09:37.3004126Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:610 +0x612
2020-02-26T12:09:37.3004406Z github.com/gorilla/websocket.(*Conn).WriteMessage(0xc0000e8840, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-02-26T12:09:37.3004655Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:763 +0x23d
2020-02-26T12:09:37.3005261Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns.(*srv).handleTraffic.func2(0x0, 0x0, 0x0, 0x0)
2020-02-26T12:09:37.3005840Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns.go:263 +0x83
2020-02-26T12:09:37.3006098Z github.com/gorilla/websocket.(*Conn).advanceFrame(0xc0000e8840, 0xc0003625d8, 0x18, 0x18)
2020-02-26T12:09:37.3006343Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:929 +0x6de
2020-02-26T12:09:37.3006666Z github.com/gorilla/websocket.(*Conn).NextReader(0xc0000e8840, 0x43a90e, 0xc000362600, 0x45e5e0, 0xc000362648, 0xc000362650)
2020-02-26T12:09:37.3006919Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:980 +0xa0
2020-02-26T12:09:37.3007273Z github.com/gorilla/websocket.(*Conn).ReadMessage(0xc0000e8840, 0x195b220, 0xc0002a8a80, 0xc0000e8840, 0xc00034a500, 0x1cde100, 0xc0001c86c0)
2020-02-26T12:09:37.3007521Z    /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:1064 +0x2f
2020-02-26T12:09:37.3008195Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns.(*srv).handleTraffic(0xc0002c0000, 0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3008779Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns.go:335 +0xcc5
2020-02-26T12:09:37.3009062Z github.com/labstack/echo/v4.(*Echo).add.func1(0x1cf3220, 0xc000114240, 0x450468, 0x511b615ab5)
2020-02-26T12:09:37.3009306Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:500 +0x8a
2020-02-26T12:09:37.3009857Z go.thethings.network/lorawan-stack/pkg/web/middleware.Recover.func1.1(0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3010362Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/recover.go:44 +0x85
2020-02-26T12:09:37.3011061Z go.thethings.network/lorawan-stack/pkg/web/middleware.Log.func1.1(0x1cf3220, 0xc000114240, 0xc00034a0f0, 0xc0008aaa48)
2020-02-26T12:09:37.3011663Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/log.go:44 +0x513
2020-02-26T12:09:37.3011945Z github.com/labstack/echo/v4/middleware.BodyLimitWithConfig.func1.1(0x1cf3220, 0xc000114240, 0x0, 0x0)
2020-02-26T12:09:37.3012273Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/body_limit.go:87 +0x1c4
2020-02-26T12:09:37.3012897Z go.thethings.network/lorawan-stack/pkg/web/middleware.ID.func1.1(0x1cf3220, 0xc000114240, 0x4, 0x4)
2020-02-26T12:09:37.3013385Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/web/middleware/id.go:47 +0x3bf
2020-02-26T12:09:37.3013672Z github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc00000c1e0, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3013904Z    /home/runner/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:611 +0x22a
2020-02-26T12:09:37.3014557Z go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns_test.TestPingPong.func1.1(0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3015165Z    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1312 +0x48
2020-02-26T12:09:37.3015436Z net/http.HandlerFunc.ServeHTTP(0xc000168030, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3015675Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2007 +0x44
2020-02-26T12:09:37.3015927Z net/http.serverHandler.ServeHTTP(0xc00016a000, 0x1cb3aa0, 0xc00016a0e0, 0xc000100600)
2020-02-26T12:09:37.3016164Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2802 +0xa4
2020-02-26T12:09:37.3016385Z net/http.(*conn).serve(0xc0001783c0, 0x1cbc060, 0xc0001c8380)
2020-02-26T12:09:37.3016613Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:1890 +0x875
2020-02-26T12:09:37.3016776Z created by net/http.(*Server).Serve
2020-02-26T12:09:37.3017004Z    /opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2928 +0x384
2020-02-26T12:09:37.3017874Z  ERROR Request error                            duration=65.378599ms error=error:pkg/web/middleware:http_recovered (Internal Server Error) method=GET namespace=gatewayserver/io/basicstation panic=concurrent write to websocket connection remote_addr=127.0.0.1:40070 request_id=01E20PDKXJA20PE0D34DV61RFH status=500 test_name=TestPingPong url=/traffic/eui-0101010101010101
2020-02-26T12:09:37.3018139Z 2020/02/26 12:09:23 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-02-26T12:09:37.3018709Z --- FAIL: TestPingPong (0.35s)
2020-02-26T12:09:37.3062032Z ##[error]    basicstationlns_test.go:1352: Received server ping
2020-02-26T12:09:37.3066025Z ##[error]    basicstationlns_test.go:1367: Server pong timeout
2020-02-26T12:09:37.3066374Z FAIL
2020-02-26T12:09:37.3067122Z FAIL   go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.937s

再来一个： https :

ERROR Request error                            duration=65.476945ms error=error:pkg/web/middleware:http_recovered (Internal Server Error) method=GET namespace=gatewayserver/io/basicstation panic=concurrent write to websocket connection remote_addr=127.0.0.1:41304 request_id=01E26E87M66F6VC188T4N2VFRF status=500 test_name=TestPingPong url=/traffic/eui-0101010101010101
2020/02/28 17:42:05 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
  WARN transport: http2Server.HandleStreams failed to read frame: io: read/write on closed pipe--- FAIL: TestPingPong (0.34s)
##[error]    basicstationlns_test.go:1352: Received server ping
##[error]    basicstationlns_test.go:1367: Server pong timeout
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.934s

这可能很严重，因为它也可能在生产中出现恐慌：

goroutine 1682 [running]:
go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xabffad0, 0xcb, 0xcb, 0x0, 0xc, 0x0, 0xcb)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0x1b6
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xa94f380, 0xabffad0, 0xcb, 0xcb, 0x301e5412, 0x0, 0xd5fb1121)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x314
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xa94f380, 0xabffad0, 0x0, 0xcb, 0xa94f300, 0xe7d33501, 0xaf1b110, 0x812373b, 0xaf1b15e, 0xade2ba0)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x46
github.com/gogo/protobuf/proto.Marshal(0x95fa230, 0xa94f380, 0x62, 0x62, 0x62, 0x1, 0x2)
    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x203
go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x95fa230, 0xa94f380, 0xae254a8, 0xaf1b110, 0xae1de34, 0xaf1b110)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x2d
go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x9629a20, 0xad276c0, 0xaf1b110, 0x62, 0x95fa230, 0xa94f380, 0x0, 0x0, 0x0, 0x2, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:164 +0x2d
go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xa96c000, 0x9601d20, 0xaed1c60, 0xade2ba0, 0x14, 0xa9da7a0, 0x0, 0xa94f380, 0x8e1dacd, 0xaed1c40)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:41 +0x1f8
go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xab1c190, 0x9601d20, 0xaed1c60, 0xa8c0c00, 0xae1df01, 0xae1df54)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:659 +0x71
go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xab1c190, 0xa8c0c00, 0xafd5580)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:678 +0x16e
created by go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).Connect
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:451 +0xf74

他们只是不断地来

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:59384: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:59384 test_name=TestGatewayServer

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:59478: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:59478 test_name=TestGatewayServer

panic: runtime error: index out of range [-1]

goroutine 1683 [running]:

go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xc00018e000, 0xcb, 0xcb, 0x0, 0xc, 0xcb)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0xa9

go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xc0007228a0, 0xc00018e000, 0xcb, 0xcb, 0x21, 0xc0000c41e0, 0xc0008a69b0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x2f7

go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xc0007228a0, 0xc00018e000, 0x0, 0xcb, 0xc000722800, 0xffffff01, 0x0, 0x1, 0x1a01e01, 0x7)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x5b

github.com/gogo/protobuf/proto.Marshal(0x1e083c0, 0xc0007228a0, 0xc0008a6ad8, 0xc000937460, 0xc000937460, 0x1921d60, 0xc0008a6af8)

    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x2d6

go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x1e083c0, 0xc0007228a0, 0xc000136930, 0xc0008a6c98, 0xc000136930, 0x62)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x50

go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x1e5be40, 0xc0005e1aa0, 0xc000136930, 0x62, 0x1e083c0, 0xc0007228a0, 0x0, 0x2, 0xc000615630, 0x0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:167 +0x39

go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xc0003fca68, 0x1e18ac0, 0xc0008d5020, 0xc0004f3160, 0x14, 0xc0008be498, 0x0, 0xc0007228a0, 0x146cfac, 0xc0008d1dc0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:41 +0x24f

go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xc000427b90, 0x1e18ac0, 0xc0008d5020, 0xc0003e8270, 0x44f501, 0xc000a00148)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:659 +0xa6

go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xc000427b90, 0xc0003e8270, 0xc00094efc0)

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:678 +0x239

created by go.thethings.network/lorawan-stack/pkg/gatewayserver.(*GatewayServer).Connect

    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:451 +0x12be

FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    22.446s

@neoaggelos @KrishnaIyer你能协调谁调查这个吗？ 我们可能有不同的事情要解决。

#2108 中的更改使https://github.com/TheThingsNetwork/lorawan-stack/issues/1332#issuecomment -600045502 和https://github.com/TheThingsNetwork/lorawan-stack/issues/1332#issuecomment -598169195 无关。

此外，即使它不是问题的真正解决方案，在此处添加更大的超时https://github.com/TheThingsNetwork/lorawan-stack/pull/2108/files?file-filters%5B%5D=.go# diff -b91e68ddfed2ddb93b15d4b9ab05bde0R1045 似乎已经清除了很多片状。

一般来说，与统计相关的所有内容都被视为#2108 的一部分（并且它始终不会失败）。

在进行大量测试后，上述所有测试（以及每天多次破坏 CI 的测试）可以分为（到目前为止）4 类：

1. concurrent write to websocket connection ：AFAIK 这已经在 #2153 中修复了

2. Expected deadline exceeded with valid auth, but have error:unknown:unknown (context deadline exceeded)这发生在Authentication测试中（例如https://github.com/TheThingsNetwork/lorawan-stack/blob/3e7ae2ca9f1180cf3d937acf11bf1637bc29ca80/pkg/gatewayserver/gatewayserver_test.go#L659错误不是因为测试失败，而是因为上下文在验证甚至可以验证之前就过期了。如果我们在ptc.Link之前调用cancel() ，这是 100% 重现的。建议的修复方法是在第 562 行取消上下文之前添加一个小的延迟。

3. Expected: 'N+1' 'Actual': 'N'发生这些是因为超时不足以让前端实际处理和解释消息。 更新统计数据的机制在外部不可用，所以我们能做的最好的事情就是等待足够长的时间。 天真地增加测试超时不是一个合适的解决方案。 建议的解决方法是重试检查直到成功，并限制允许的最大重试次数。

4. panic: runtime error: index out of range [-1]在 Redis 上存储网关连接统计信息时会不规则地出现，并且在编组 proto 时会发生这种情况。 #2108 中的代码已更改，因此正在查看测试。 看起来像是竞争条件，但我无法确定来源。

供参考：

1. 我们有test.Delay
2. 有关此方法的示例，请参见https://github.com/TheThingsNetwork/lorawan-stack/blob/da4c2ac4044188c6c28fc6d799d20bbb1918b9fb/pkg/networkserver/networkserver_flow_test.go#L621 -L634（也使用test.Delay

一个新的：

--- FAIL: TestTraffic (0.03s)
    mqtt_test.go:169: Connection timeout
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/mqtt    0.104s

2. 建议的修复是在取消第 562 行的上下文之前添加一个小的延迟。

听起来不错

3. 建议的解决方法是重试检查直到成功，并限制允许的最大重试次数。

也很好听

4. panic: runtime error: index out of range [-1]这在Redis上存储网关连接统计信息时不规则出现，并且在编组proto时发生。 #2108 中的代码已更改，因此正在查看测试。 看起来像是竞争条件，但我无法确定来源。

这个可能很严重。 当您尝试编组nil结构或在编组时结构发生更改时，可能会发生这种情况。 这些都是实现错误。

 ERROR Request error                            duration=259.278µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36438 request_id=01E4CWTRYGCFG9R79E322774MJ status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
 ERROR Request error                            duration=136.484µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36440 request_id=01E4CWTRYGVGM6XQ4KMR66T3B7 status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
 ERROR Request error                            duration=1.681437ms error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:36442 request_id=01E4CWTRYHK6KM1CCM7G5TYVZM status=500 test_name=TestDiscover url=/router-info
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020/03/27 04:23:43 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020/03/27 04:23:43 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
--- FAIL: TestRTT (0.04s)
    --- FAIL: TestRTT/RepeatedTxAck (0.00s)
        doc.go:66: 
            pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1233
            Expected: '2'
            Actual:   '1'
            (Should be equal)
        basicstationlns_test.go:1234: Incorrect Stats entries recorded: 1
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io/basicstationlns 0.247s

@neoaggelos这里的状态是什么？

#2260 和 #2190 合并后，我们应该处于一个更好的位置。 我还不会关闭它，让我们给 Travis 几天时间看看它如何随着变化而变化。

新的一个：

--- FAIL: TestFlow (0.00s)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:111
        Expected: true
        Actual:   false
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:112
        Expected 'nil' to NOT be nil (but it was)!
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:113
        Expected '2562047h47m16.854775807s' to be less than '64ms' (but it wasn't)!
FAIL
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver/io 0.020s

另外，再次得到这个：

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:45706: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:45706 test_name=TestGatewayServer
 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:45912: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:45912 test_name=TestGatewayServer
panic: runtime error: index out of range [-1]

goroutine 1721 [running]:
go.thethings.network/lorawan-stack/pkg/ttnpb.encodeVarintGateway(0xc0000aeb60, 0xcb, 0xcb, 0x0, 0xc, 0xcb)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:4225 +0xa9
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xc000a9e4e0, 0xc0000aeb60, 0xcb, 0xcb, 0x21, 0xc000a9e360, 0xc0004ed6f0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:4158 +0x2f4
go.thethings.network/lorawan-stack/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xc000a9e4e0, 0xc0000aeb60, 0x0, 0xcb, 0xc000a9e400, 0x1, 0x1d6dbc0, 0xc000897710, 0x1d8b120, 0x2acf700)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/ttnpb/gateway.pb.go:1501 +0x5b
github.com/gogo/protobuf/proto.Marshal(0x1d611c0, 0xc000a9e4e0, 0x18fa180, 0xc000a9e360, 0xc000881220, 0xc000881340, 0xc00089e1e8)
    /home/runner/go/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x2d6
go.thethings.network/lorawan-stack/pkg/redis.MarshalProto(0x1d611c0, 0xc000a9e4e0, 0x40c245, 0x1934f20, 0x19396e0, 0x0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/redis/redis.go:49 +0x50
go.thethings.network/lorawan-stack/pkg/redis.SetProto(0x7f731299a868, 0xc0008854c0, 0xc0007ba4d0, 0x69, 0x1d611c0, 0xc000a9e4e0, 0x0, 0x3, 0xc0004edaa8, 0x12bdd43)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/redis/redis.go:167 +0x39
go.thethings.network/lorawan-stack/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set.func1(0x1da3f80, 0xc0008854c0, 0x7f733c4c07d0, 0x0)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/redis/registry.go:64 +0x693
github.com/go-redis/redis.(*Pipeline).pipelined(0xc0008854c0, 0xc000897c20, 0x13b77f7, 0x17668e0, 0xc000a932f0, 0x0, 0xc0008854c0)
    /home/runner/go/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:111 +0x40
github.com/go-redis/redis.(*Pipeline).Pipelined(0xc0008854c0, 0xc000897c20, 0xc0008854c0, 0x30, 0x185a180, 0x1, 0xc000897c20)
    /home/runner/go/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:120 +0x35

最后一个似乎深层复制没有成功？

这不是测试中的特定情况吗？

还有一个：

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:39250: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:39250 test_name=TestGatewayServer
--- FAIL: TestGatewayServer (36.35s)
    --- FAIL: TestGatewayServer/DetectDisconnect/grpc#01 (0.77s)
        gatewayserver_test.go:650: Expected deadline exceeded on second connection but have error:unknown:unknown (context deadline exceeded)
FAIL
coverage: 75.7% of statements
FAIL    go.thethings.network/lorawan-stack/pkg/gatewayserver    36.387s

这可以通过再次使用 GS 事件来解决，就像我们在Authentication测试中所做的那样，但是这个测试有点复杂。

最后一个似乎深层复制没有成功？

是的，可能网关状态不是唯一的事情。 但我真的找不到其他可能导致问题的原因。 是否会以某种方式导致测试处于非法状态（例如，两个来源同时更新相同的统计数据，gc 接管并发布不应该的东西）？

这可以通过再次使用 GS 事件来解决，就像我们在Authentication测试中所做的那样，但是这个测试有点复杂。

我认为这个和上一个相似； 这也在测试中执行了完整的Link() ，而上下文在正确设置之前已经被取消。 因此，您可能需要再次为Link()加倍timeout 。

请注意，GS 测试已经需要 36 秒，这是相当多的。 您可能会开始尝试降低一般 GS timeout值，并在智能位置增加它，例如链接和潜在的流量处理。

是的，可能网关状态不是唯一的事情。 但我真的找不到其他可能导致问题的原因。 是否会以某种方式导致测试处于非法状态（例如，两个来源同时更新相同的统计数据，gc 接管并发布不应该的东西）？

那很可能是。 您可以使用-run （运行特定测试）和-race来检测种族。

请注意，GS 测试已经需要 36 秒，这是相当多的。 您可能会开始尝试降低一般 GS timeout值，并在智能位置增加它，例如链接和潜在的流量处理。

这就是为什么当 Travis 失败时，我大多不愿意简单地增加超时时间。 我认为如果我们使用事件而不是超时，测试将受益匪浅。

另外，我认为减少超时只会导致更不稳定的测试（我真的不知道所使用的值是如何产生的，所以也许@KrishnaIyer更适合这个测试。）

我可以更好地使用事件，但是如果不满足先决条件，我们就不能为失败的链接设置事件。 某些事情异步发生是不可避免的，并且没有失败事件，而我们希望它失败，即我们需要等待以确保它没有成功。

@neoaggelos我们有太多这样的：

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:56960: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:56960 test_name=TestGatewayServer
panic: runtime error: index out of range [-1]
goroutine 780 [running]:
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.encodeVarintGateway(0xc1e6700, 0xdd, 0xdd, 0x0, 0xc, 0x0, 0xdd)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4410 +0x1b6
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.(*GatewayConnectionStats).MarshalToSizedBuffer(0xbe2d580, 0xc1e6700, 0xdd, 0xdd, 0x1d4e291c, 0x0, 0xd64ead82)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:4298 +0x4b6
go.thethings.network/lorawan-stack/v3/pkg/ttnpb.(*GatewayConnectionStats).XXX_Marshal(0xbe2d580, 0xc1e6700, 0x0, 0xdd, 0xbe2d500, 0x909ab01, 0x42, 0xbfa92d8, 0xbc536d0, 0x906ef60)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/ttnpb/gateway.pb.go:1503 +0x46
github.com/gogo/protobuf/proto.Marshal(0x95e0130, 0xbe2d580, 0x0, 0x4, 0x55555555, 0x7, 0x10)
    /home/travis/gopath/pkg/mod/github.com/gogo/[email protected]/proto/table_marshal.go:2940 +0x203
go.thethings.network/lorawan-stack/v3/pkg/redis.MarshalProto(0x95e0130, 0xbe2d580, 0xc1ceaa8, 0x8050b39, 0x926a420, 0x926eb60)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:49 +0x3d
go.thethings.network/lorawan-stack/v3/pkg/redis.SetProto(0xe7ce4a88, 0xc2d88d0, 0xbdefb20, 0x69, 0x95e0130, 0xbe2d580, 0x0, 0x0, 0xffffffe1, 0x0, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/redis/redis.go:167 +0x2d
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set.func1(0x9607320, 0xc2d88d0, 0xf7f773ec, 0x20)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:64 +0x597
github.com/go-redis/redis.(*Pipeline).pipelined(0xc2d88d0, 0xc1cea40, 0x8dbb048, 0x90bc700, 0xbfa9510, 0xc0b4f60, 0xc2d88d0)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:111 +0x32
github.com/go-redis/redis.(*Pipeline).Pipelined(0xc2d88d0, 0xc1cea40, 0xc2d88d0, 0x20, 0x91763a0, 0x1, 0xc1cea40)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/pipeline.go:120 +0x29
github.com/go-redis/redis.(*Client).Pipelined(0xbdbb260, 0xc1cea40, 0x92a3af7, 0x1c, 0xa1e9ca0, 0x14, 0x1d4e291c)
    /home/travis/gopath/pkg/mod/github.com/go-redis/[email protected]+incompatible/redis.go:457 +0x3c
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/redis.(*GatewayConnectionStatsRegistry).Set(0xbddf928, 0x95e8160, 0xc1c4520, 0xbc3ff40, 0x14, 0xbdda390, 0x0, 0xbe2d580, 0x9010000, 0x0, ...)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/redis/registry.go:51 +0x156
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).UpdateConnectionStats(0xbd4d270, 0xbd31000, 0x9010000, 0xc24ade0, 0x0)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:883 +0x9e
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).updateConnStats(0xbd4d270, 0xbd31000, 0xbfcba80)
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:673 +0x91
created by go.thethings.network/lorawan-stack/v3/pkg/gatewayserver.(*GatewayServer).Connect
    /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/gatewayserver.go:458 +0xff1

请解决

新的一个;

 ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:43472: use of closed network connection gateway_uid=eui-aaee000000000000@foo-tenant namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:43472 tenant_id=foo-tenant test_name=TestGatewayServer
--- FAIL: TestGatewayServer (36.36s)
    --- FAIL: TestGatewayServer/DetectDisconnect/grpc#01 (0.77s)
        gatewayserver_test.go:653: Expected deadline exceeded on second connection but have error:unknown:unknown (context deadline exceeded)
FAIL
FAIL    go.thethings.network/lorawan-stack/v3/pkg/gatewayserver 36.408s

这需要微调在这个特定地方使用的超时

ERROR Send failed, close session               error=read tcp 127.0.0.1:1882->127.0.0.1:41304: use of closed network connection gateway_uid=eui-aaee000000000000 namespace=gatewayserver/io/mqtt protocol=mqtt remote_addr=127.0.0.1:41304 test_name=TestGatewayServer
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x144bd95]

goroutine 90 [running]:
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io.(*Connection).Context(...)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/io.go:162
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.(*srv).gc.func1(0x18ff0a0, 0xc00097dcb8, 0x1679ba0, 0xc0005ac000, 0x1)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:425 +0xf5
sync.(*Map).Range(0xc0001aaf98, 0xc0004a3f58)
    /opt/hostedtoolcache/go/1.14.3/x64/src/sync/map.go:333 +0x1c5
go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.(*srv).gc(0xc0001aaf20)
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:421 +0xdf
created by go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/udp.Serve
    /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/udp/udp.go:72 +0x1e2

2624 修复了最后一条评论。

--- FAIL: TestSubBandEIRPOverride (0.00s)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:476
        Expected: true
        Actual:   false
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:477
        Expected: '1'
        Actual:   '0'
        (Should be equal)
    doc.go:66: 
        /home/travis/gopath/src/go.thethings.network/lorawan-stack/pkg/gatewayserver/io/io_test.go:478
        Expected '2562047h47m16.854775807s' to be less than '64ms' (but it wasn't)!
FAIL
FAIL    go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io  0.087s

这看起来像一个int溢出，相当可怕。 请确保这是由测试的模糊性引起的，而不是生产中可能发生的事情。

另一个：

2020-07-13T11:28:13.8444055Z  ERROR Request error                            duration=218.2µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35116 request_id=01ED3YVKE5E4MZCJ6TJDV5KMH6 status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8444793Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8445194Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8446392Z  ERROR Request error                            duration=138.4µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35118 request_id=01ED3YVKE5Z7A5CRAJWN5ARF1P status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8447014Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8447418Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8449101Z  ERROR Request error                            duration=137.8µs error=error:pkg/gatewayserver/io/basicstationlns:empty_gateway_eui (empty gateway EUI) method=GET namespace=gatewayserver/io/basicstation remote_addr=127.0.0.1:35120 request_id=01ED3YVKE6SKB9TPP9BN74YD74 status=500 test_name=TestDiscover url=/router-info
2020-07-13T11:28:13.8449873Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8453477Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8454049Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8454795Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8455937Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8456759Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8457157Z 2020/07/13 11:27:13 http: response.WriteHeader on hijacked connection from github.com/labstack/echo/v4.(*Response).WriteHeader (response.go:63)
2020-07-13T11:28:13.8459526Z 2020/07/13 11:27:13 http: response.Write on hijacked connection from github.com/labstack/echo/v4.(*Response).Write (response.go:72)
2020-07-13T11:28:13.8460320Z --- FAIL: TestRTT (0.15s)
2020-07-13T11:28:13.8461043Z     --- FAIL: TestRTT/FollowUpTxAck (0.13s)
2020-07-13T11:28:13.8461263Z         doc.go:66: 
2020-07-13T11:28:13.8461807Z             /home/runner/work/lorawan-stack/lorawan-stack/pkg/gatewayserver/io/basicstationlns/basicstationlns_test.go:1235
2020-07-13T11:28:13.8462139Z             Expected: '1'
2020-07-13T11:28:13.8462441Z             Actual:   '0'
2020-07-13T11:28:13.8462601Z             (Should be equal)
2020-07-13T11:28:13.8462766Z         basicstationlns_test.go:1236: Incorrect Stats entries recorded: 0
2020-07-13T11:28:13.8462920Z FAIL
2020-07-13T11:28:13.8463066Z coverage: 70.2% of statements
2020-07-13T11:28:13.8463424Z FAIL   go.thethings.network/lorawan-stack/v3/pkg/gatewayserver/io/basicstationlns  0.545s

最后一条评论很旧，关闭。 根据需要重新打开。