Bug description
I am running a docker swarm with 7 nodes 3 managers and 4 workers. I have deployed portainer via the stack file, with agents and portainer it's self. After some time an agent instance will report high cpu usage 60-70% and the portainer instance will not load any data about services, stacks etc.
When i look at the log files for the agent with the high CPU i get:
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
2018/06/22 13:48:35 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Expected behavior
I would expect that the portainer instance loads all the data it can, so i can continue to use it as much as possible. An error message explaining that it was unable to contact an agent would also be helpful.
Steps to reproduce the issue:
I do not have any specific steps to reproduce it. Just seems to happen randomly.
Technical details:
- Portainer version: 1.18.0
- Docker version (managed by Portainer): 18.03.1-ce
- Platform (windows/linux): Amazon Linux AMI release 2018.03
- Command used to start Portainer (
docker run -p 9000:9000 portainer/portainer): sudo curl -L https://portainer.io/download/portainer-agent-stack.yml -o portainer-agent-stack.yml
sudo docker stack deploy --compose-file=portainer-agent-stack.yml portainer - Browser: Firefox 60.0.2 (64-bit)
Umaaz
All 47 comments
We have experienced this issue again with nearly 1000000 log messages in 3 mins. Any update on this?
Umaaz
on 13 Jul 2018
Sorry @Umaaz we were not able to investigate this yet, I've referenced a network instability issue with serf inside AWS, might be related: https://github.com/hashicorp/serf/issues/512
I know that @mrmachine is another AWS user, have you seen the same issues?
deviantony
on 13 Jul 2018
I am running portainer as a service on a dedicated server. The same error occurs for me. A restart helps, but after some time (within 24 hours) the error re-occurs. So this is not only occurring on AWS.
portainer[25005]: 2018/08/06 12:33:03 http: Accept error: accept tcp 127.0.0.1:8080: accept4: too many open files; retrying in 1s
daneZz
on 6 Aug 2018
@daneZz are you using the agent setup?
deviantony
on 6 Aug 2018
@deviantony I used the "Deploy Portainer without Docker" part from the documentation on https://portainer.readthedocs.io/en/latest/deployment.html.
daneZz
on 6 Aug 2018
I figured why this happens. It seems that the connection amount of portainer continuously increases. I suspect that connections are properly closed.
Environment
Portainer version: 1.19.1 (runs as a service in Linux, not in a docker container).
I am running a docker swarm with 10 docker servers. (DESTINATION_*). The portainer service is running on a dedicated server.
Current open connections:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
portainer 24681 portainer 5u IPv4 1661338056 0t0 TCP SOURCE:49156->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 6u IPv4 1666996540 0t0 TCP SOURCE:60668->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 7u IPv4 1667123453 0t0 TCP SOURCE:37898->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 8u IPv4 1667118482 0t0 TCP SOURCE:40450->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 9u IPv4 1661338057 0t0 TCP SOURCE:34826->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 10u IPv4 1667116664 0t0 TCP SOURCE:48878->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 11u IPv4 1667117731 0t0 TCP SOURCE:42462->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 12u IPv4 1667123468 0t0 TCP SOURCE:49850->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 13u IPv4 1667125453 0t0 TCP localhost:webcache (LISTEN)
portainer 24681 portainer 14u IPv4 1667544168 0t0 TCP SOURCE:59596->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 15u IPv4 1665297050 0t0 TCP SOURCE:49680->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 16u IPv4 1667544169 0t0 TCP SOURCE:42882->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 17u IPv4 1666995473 0t0 TCP SOURCE:49686->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 18u IPv4 1667541323 0t0 TCP SOURCE:48350->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 19u IPv4 1661598653 0t0 TCP SOURCE:50902->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 20u IPv4 1667541324 0t0 TCP SOURCE:45284->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 21u IPv4 1667544170 0t0 TCP SOURCE:59342->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 22u IPv4 1667544172 0t0 TCP SOURCE:52908->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 23u IPv4 1667536230 0t0 TCP SOURCE:60308->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 24u IPv4 1667957567 0t0 TCP SOURCE:40926->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 25u IPv4 1667930880 0t0 TCP SOURCE:52450->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 26u IPv4 1667953275 0t0 TCP SOURCE:57918->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 27u IPv4 1667957573 0t0 TCP SOURCE:60470->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 28u IPv4 1667957574 0t0 TCP SOURCE:54846->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 29u IPv4 1667964094 0t0 TCP SOURCE:40666->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 30u IPv4 1667957575 0t0 TCP SOURCE:34232->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 31u IPv4 1667957576 0t0 TCP SOURCE:41620->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 32u IPv4 1668392513 0t0 TCP SOURCE:49886->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 33u IPv4 1668400465 0t0 TCP SOURCE:33166->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 34u IPv4 1668403233 0t0 TCP SOURCE:38628->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 35u IPv4 1668400467 0t0 TCP SOURCE:41186->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 36u IPv4 1668400468 0t0 TCP SOURCE:35562->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 37u IPv4 1668400470 0t0 TCP SOURCE:49614->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 38u IPv4 1668403236 0t0 TCP SOURCE:43180->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 39u IPv4 1668392514 0t0 TCP SOURCE:50574->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 40u IPv4 1668848037 0t0 TCP SOURCE:59476->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 41u IPv4 1668851750 0t0 TCP SOURCE:42756->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 42u IPv4 1668848759 0t0 TCP SOURCE:48218->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 43u IPv4 1668851751 0t0 TCP SOURCE:50770->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 44u IPv4 1668843039 0t0 TCP SOURCE:45152->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 45u IPv4 1668848038 0t0 TCP SOURCE:59210->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 46u IPv4 1668851755 0t0 TCP SOURCE:52776->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 47u IPv4 1668851756 0t0 TCP SOURCE:60164->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 48u IPv4 1668891221 0t0 TCP SOURCE:40768->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 49u IPv4 1669302607 0t0 TCP SOURCE:52280->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 50u IPv4 1669302610 0t0 TCP SOURCE:57750->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 51u IPv4 1669302612 0t0 TCP SOURCE:60306->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 52u IPv4 1669304343 0t0 TCP SOURCE:54682->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 53u IPv4 1669304344 0t0 TCP SOURCE:40502->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 54u IPv4 1661338249 0t0 TCP SOURCE:34068->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 55u IPv4 1669304345 0t0 TCP SOURCE:41456->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 56u IPv4 1669746836 0t0 TCP SOURCE:50410->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 57u IPv4 1669745985 0t0 TCP SOURCE:33694->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 58u IPv4 1669745986 0t0 TCP SOURCE:39176->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 59u IPv4 1669748911 0t0 TCP SOURCE:41728->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 60u IPv4 1669745589 0t0 TCP SOURCE:36104->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 61u IPv4 1669745590 0t0 TCP SOURCE:50156->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 62u IPv4 1669461959 0t0 TCP SOURCE:43722->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 63u IPv4 1669745591 0t0 TCP SOURCE:51110->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 64u IPv4 1670199401 0t0 TCP SOURCE:44070->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 65u IPv4 1670197674 0t0 TCP SOURCE:60790->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 66u IPv4 1670199402 0t0 TCP SOURCE:49538->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 67u IPv4 1670196790 0t0 TCP SOURCE:52090->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 68u IPv4 1670196791 0t0 TCP SOURCE:46466->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 69u IPv4 1670196792 0t0 TCP SOURCE:60524->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 70u IPv4 1670196794 0t0 TCP SOURCE:54096->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 71u IPv4 1670200687 0t0 TCP SOURCE:33252->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 72u IPv4 1670626194 0t0 TCP SOURCE:54820->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 73u IPv4 1670623075 0t0 TCP SOURCE:43308->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 74u IPv4 1670623076 0t0 TCP SOURCE:60288->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 75u IPv4 1670619762 0t0 TCP SOURCE:34614->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 76u IPv4 1670619763 0t0 TCP SOURCE:57222->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 77u IPv4 1670623692 0t0 TCP SOURCE:43042->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 78u IPv4 1670619764 0t0 TCP SOURCE:36608->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 79u IPv4 1670623693 0t0 TCP SOURCE:44002->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 80u IPv4 1671067751 0t0 TCP SOURCE:55022->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 81u IPv4 1671074820 0t0 TCP SOURCE:38302->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 82u IPv4 1671070856 0t0 TCP SOURCE:43770->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 83u IPv4 1671067752 0t0 TCP SOURCE:46326->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 84u IPv4 1671055820 0t0 TCP SOURCE:40734->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 85u IPv4 1671070858 0t0 TCP SOURCE:54792->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 86u IPv4 1671067757 0t0 TCP SOURCE:48358->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 87u IPv4 1671062402 0t0 TCP SOURCE:55746->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 88u IPv4 1671510542 0t0 TCP SOURCE:50478->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 89u IPv4 1671503635 0t0 TCP SOURCE:38966->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 90u IPv4 1671510543 0t0 TCP SOURCE:55940->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 91u IPv4 1671503636 0t0 TCP SOURCE:58496->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 92u IPv4 1671510545 0t0 TCP SOURCE:52874->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 93u IPv4 1671510546 0t0 TCP SOURCE:38694->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 94u IPv4 1671510547 0t0 TCP SOURCE:60498->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 95u IPv4 1671510549 0t0 TCP SOURCE:39666->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 96u IPv4 1671972691 0t0 TCP SOURCE:34750->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 97u IPv4 1671978118 0t0 TCP SOURCE:51470->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 98u IPv4 1671979088 0t0 TCP SOURCE:40218->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 99u IPv4 1671979089 0t0 TCP SOURCE:42770->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 100u IPv4 1671979092 0t0 TCP SOURCE:37158->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 101u IPv4 1671979093 0t0 TCP SOURCE:51210->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 102u IPv4 1671979094 0t0 TCP SOURCE:44776->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 103u IPv4 1671979097 0t0 TCP SOURCE:52170->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 104u IPv4 1672468200 0t0 TCP SOURCE:46970->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 105u IPv4 1672472884 0t0 TCP SOURCE:35458->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 106u IPv4 1672468203 0t0 TCP SOURCE:52438->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 107u IPv4 1672468204 0t0 TCP SOURCE:54990->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 108u IPv4 1672468206 0t0 TCP SOURCE:49372->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 109u IPv4 1672474765 0t0 TCP SOURCE:35198->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 110u IPv4 1672468209 0t0 TCP SOURCE:57002->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 111u IPv4 1672468211 0t0 TCP SOURCE:36158->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 112u IPv4 1672963762 0t0 TCP SOURCE:59388->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 113u IPv4 1672951247 0t0 TCP SOURCE:47876->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 114u IPv4 1672964595 0t0 TCP SOURCE:36630->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 115u IPv4 1672963766 0t0 TCP SOURCE:39188->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 116u IPv4 1672963768 0t0 TCP SOURCE:33576->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 117u IPv4 1672964600 0t0 TCP SOURCE:47640->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 118u IPv4 1672963772 0t0 TCP SOURCE:41206->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 119u IPv4 1672963773 0t0 TCP SOURCE:48594->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 120u IPv4 1673436783 0t0 TCP SOURCE:41866->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 121u IPv4 1673437301 0t0 TCP SOURCE:58568->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 122u IPv4 1673436785 0t0 TCP SOURCE:47328->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 123u IPv4 1673436786 0t0 TCP SOURCE:49880->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 124u IPv4 1673436787 0t0 TCP SOURCE:44256->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 125u IPv4 1673436788 0t0 TCP SOURCE:58308->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 126u IPv4 1673436789 0t0 TCP SOURCE:51874->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 127u IPv4 1673436790 0t0 TCP SOURCE:59262->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 128u IPv4 1673907078 0t0 TCP SOURCE:53058->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 129u IPv4 1673907077 0t0 TCP SOURCE:41546->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 130u IPv4 1673907081 0t0 TCP SOURCE:58526->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 131u IPv4 1673907082 0t0 TCP SOURCE:32846->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 132u IPv4 1673907083 0t0 TCP SOURCE:55454->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 133u IPv4 1673907085 0t0 TCP SOURCE:41280->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 134u IPv4 1673907086 0t0 TCP SOURCE:34846->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 135u IPv4 1673907087 0t0 TCP SOURCE:42238->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 136u IPv4 1673241716 0t0 TCP SOURCE:35400->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 137u IPv4 1674384529 0t0 TCP SOURCE:52114->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 138u IPv4 1674345383 0t0 TCP SOURCE:40868->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 139u IPv4 1674374334 0t0 TCP SOURCE:43420->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 140u IPv4 1674345384 0t0 TCP SOURCE:37802->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 141u IPv4 1674345385 0t0 TCP SOURCE:51854->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 142u IPv4 1674345386 0t0 TCP SOURCE:45420->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 143u IPv4 1674380533 0t0 TCP SOURCE:52814->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 144u IPv4 1674864995 0t0 TCP SOURCE:45974->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 145u IPv4 1674850690 0t0 TCP SOURCE:34462->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 146u IPv4 1674863810 0t0 TCP SOURCE:51448->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 147u IPv4 1674863811 0t0 TCP SOURCE:54000->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 148u IPv4 1674864998 0t0 TCP SOURCE:48382->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 149u IPv4 1674854887 0t0 TCP SOURCE:34202->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 150u IPv4 1674856413 0t0 TCP SOURCE:56000->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 151u IPv4 1674856414 0t0 TCP SOURCE:35156->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 152u IPv4 1675339606 0t0 TCP SOURCE:56046->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 153u IPv4 1675341366 0t0 TCP SOURCE:44532->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 154u IPv4 1675345013 0t0 TCP SOURCE:33276->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 155u IPv4 1675341367 0t0 TCP SOURCE:35828->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 156u IPv4 1675348475 0t0 TCP SOURCE:58436->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 157u IPv4 1675348476 0t0 TCP SOURCE:44256->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 158u IPv4 1675348477 0t0 TCP SOURCE:37822->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 159u IPv4 1675348478 0t0 TCP SOURCE:45210->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 160u IPv4 1673241861 0t0 TCP SOURCE:37426->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 161u IPv4 1675848789 0t0 TCP SOURCE:54146->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 162u IPv4 1675845727 0t0 TCP SOURCE:42894->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 163u IPv4 1675835177 0t0 TCP SOURCE:45446->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 164u IPv4 1675845474 0t0 TCP SOURCE:39822->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 165u IPv4 1675835178 0t0 TCP SOURCE:53874->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 166u IPv4 1675835179 0t0 TCP SOURCE:47440->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 167u IPv4 1675845476 0t0 TCP SOURCE:54834->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 168u IPv4 1676320733 0t0 TCP SOURCE:46832->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 169u IPv4 1676316265 0t0 TCP SOURCE:35320->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 170u IPv4 1676326915 0t0 TCP SOURCE:52300->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 171u IPv4 1676321266 0t0 TCP SOURCE:54858->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 172u IPv4 1676321267 0t0 TCP SOURCE:49240->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 173u IPv4 1676317085 0t0 TCP SOURCE:35078->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 174u IPv4 1676328237 0t0 TCP SOURCE:56878->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 175u IPv4 1676324805 0t0 TCP SOURCE:36034->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 176u IPv4 1676833576 0t0 TCP SOURCE:56750->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 177u IPv4 1676831375 0t0 TCP SOURCE:45238->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 178u IPv4 1676827178 0t0 TCP SOURCE:33986->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 179u IPv4 1676833577 0t0 TCP SOURCE:36538->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 180u IPv4 1676838133 0t0 TCP SOURCE:59152->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 181u IPv4 1676838149 0t0 TCP SOURCE:44972->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 182u IPv4 1676833707 0t0 TCP SOURCE:38538->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 183u IPv4 1676827218 0t0 TCP SOURCE:45932->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 184u IPv4 1677362406 0t0 TCP SOURCE:39226->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 185u IPv4 1677362401 0t0 TCP SOURCE:55940->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 186u IPv4 1677357296 0t0 TCP SOURCE:44694->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 187u IPv4 1677353970 0t0 TCP SOURCE:47246->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 188u IPv4 1677363636 0t0 TCP SOURCE:41628->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 189u IPv4 1677363637 0t0 TCP SOURCE:55680->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 190u IPv4 1677360214 0t0 TCP SOURCE:49246->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 191u IPv4 1677363831 0t0 TCP SOURCE:56634->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 192u IPv4 1677870447 0t0 TCP SOURCE:48904->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 193u IPv4 1677870356 0t0 TCP SOURCE:37392->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 194u IPv4 1677870473 0t0 TCP SOURCE:54366->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 195u IPv4 1677861760 0t0 TCP SOURCE:56918->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 196u IPv4 1677864786 0t0 TCP SOURCE:51312->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 197u IPv4 1677864787 0t0 TCP SOURCE:37134->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 198u IPv4 1677873442 0t0 TCP SOURCE:58936->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 199u IPv4 1677873575 0t0 TCP SOURCE:38092->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 200u IPv4 1678364090 0t0 TCP SOURCE:59606->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 201u IPv4 1678360049 0t0 TCP SOURCE:48094->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 202u IPv4 1678364091 0t0 TCP SOURCE:36836->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 203u IPv4 1678364752 0t0 TCP SOURCE:39388->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 204u IPv4 1678367691 0t0 TCP SOURCE:33770->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 205u IPv4 1678366489 0t0 TCP SOURCE:47822->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 206u IPv4 1678370847 0t0 TCP SOURCE:41388->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 207u IPv4 1678370873 0t0 TCP SOURCE:48776->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 208u IPv4 1678876750 0t0 TCP SOURCE:42684->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 209u IPv4 1678733309 0t0 TCP SOURCE:59392->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 210u IPv4 1678876761 0t0 TCP SOURCE:48146->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 211u IPv4 1678876793 0t0 TCP SOURCE:50698->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 212u IPv4 1678876817 0t0 TCP SOURCE:45086->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 213u IPv4 1678878852 0t0 TCP SOURCE:59138->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 214u IPv4 1678878853 0t0 TCP SOURCE:52704->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 215u IPv4 1678878854 0t0 TCP SOURCE:60098->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 216u IPv4 1679367010 0t0 TCP SOURCE:53886->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 217u IPv4 1679387107 0t0 TCP SOURCE:42366->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 218u IPv4 1679378815 0t0 TCP SOURCE:59354->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 219u IPv4 1679387136 0t0 TCP SOURCE:33674->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 220u IPv4 1679387137 0t0 TCP SOURCE:56282->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 221u IPv4 1679387138 0t0 TCP SOURCE:42102->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 222u IPv4 1679387139 0t0 TCP SOURCE:35668->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 223u IPv4 1679387140 0t0 TCP SOURCE:43056->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 224u IPv4 1679894060 0t0 TCP SOURCE:35852->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 225u IPv4 1679894059 0t0 TCP SOURCE:52572->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 226u IPv4 1679894062 0t0 TCP SOURCE:41320->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 227u IPv4 1679888169 0t0 TCP SOURCE:43872->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 228u IPv4 1679888170 0t0 TCP SOURCE:38260->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 229u IPv4 1679894063 0t0 TCP SOURCE:52312->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 230u IPv4 1679895198 0t0 TCP SOURCE:45884->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 231u IPv4 1679892798 0t0 TCP SOURCE:53272->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 232u IPv4 1680403556 0t0 TCP SOURCE:45320->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 233u IPv4 1680391865 0t0 TCP SOURCE:33808->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 234u IPv4 1680399953 0t0 TCP SOURCE:50794->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 235u IPv4 1680403635 0t0 TCP SOURCE:53346->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 236u IPv4 1680399954 0t0 TCP SOURCE:47722->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 237u IPv4 1680399067 0t0 TCP SOURCE:33542->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 238u IPv4 1680403857 0t0 TCP SOURCE:55340->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 239u IPv4 1680403858 0t0 TCP SOURCE:34496->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 240u IPv4 1680879067 0t0 TCP SOURCE:54836->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 241u IPv4 1680860943 0t0 TCP SOURCE:43324->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 242u IPv4 1680874277 0t0 TCP SOURCE:60304->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 243u IPv4 1680879072 0t0 TCP SOURCE:34630->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 244u IPv4 1680874281 0t0 TCP SOURCE:57244->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 245u IPv4 1680879107 0t0 TCP SOURCE:43064->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 246u IPv4 1680879108 0t0 TCP SOURCE:36630->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 247u IPv4 1680869294 0t0 TCP SOURCE:44018->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 248u IPv4 1681360677 0t0 TCP SOURCE:36948->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 249u IPv4 1681360592 0t0 TCP SOURCE:53662->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 250u IPv4 1681363127 0t0 TCP SOURCE:42414->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 251u IPv4 1681361136 0t0 TCP SOURCE:44968->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 252u IPv4 1681361137 0t0 TCP SOURCE:39356->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 253u IPv4 1681360848 0t0 TCP SOURCE:53414->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 254u IPv4 1681354292 0t0 TCP SOURCE:46980->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 255u IPv4 1681363333 0t0 TCP SOURCE:54368->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 256u IPv4 1681811381 0t0 TCP SOURCE:47850->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 257u IPv4 1681811347 0t0 TCP SOURCE:36332->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 258u IPv4 1681815341 0t0 TCP SOURCE:53324->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 259u IPv4 1681815342 0t0 TCP SOURCE:55876->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 260u IPv4 1681815345 0t0 TCP SOURCE:50258->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 261u IPv4 1681820730 0t0 TCP SOURCE:36078->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 262u IPv4 1681818434 0t0 TCP SOURCE:57876->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 263u IPv4 1681815346 0t0 TCP SOURCE:37032->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 264u IPv4 1682259433 0t0 TCP SOURCE:57416->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 265u IPv4 1682245111 0t0 TCP SOURCE:45904->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 266u IPv4 1682262017 0t0 TCP SOURCE:34646->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 267u IPv4 1682249663 0t0 TCP SOURCE:37198->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 268u IPv4 1682249664 0t0 TCP SOURCE:59812->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 269u IPv4 1682249665 0t0 TCP SOURCE:45638->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 270u IPv4 1681164304 0t0 TCP SOURCE:39204->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 271u IPv4 1682245220 0t0 TCP SOURCE:46592->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 272u IPv4 1682716691 0t0 TCP SOURCE:39346->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 273u IPv4 1682703129 0t0 TCP SOURCE:56054->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 274u IPv4 1682713355 0t0 TCP SOURCE:44808->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 275u IPv4 1682717794 0t0 TCP SOURCE:47370->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 276u IPv4 1682717808 0t0 TCP SOURCE:41758->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 277u IPv4 1681164314 0t0 TCP SOURCE:55818->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 278u IPv4 1681164315 0t0 TCP SOURCE:49384->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 279u IPv4 1681164316 0t0 TCP SOURCE:56772->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 280u IPv4 1683216233 0t0 TCP SOURCE:52400->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 281u IPv4 1683221421 0t0 TCP SOURCE:40882->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 282u IPv4 1683216235 0t0 TCP SOURCE:57862->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 283u IPv4 1683222846 0t0 TCP SOURCE:60414->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 284u IPv4 1683231049 0t0 TCP SOURCE:54796->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 285u IPv4 1683231052 0t0 TCP SOURCE:40616->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 286u IPv4 1683232785 0t0 TCP SOURCE:34182->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 287u IPv4 1683232849 0t0 TCP SOURCE:41576->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 288u IPv4 1683739616 0t0 TCP SOURCE:37268->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 289u IPv4 1683704759 0t0 TCP SOURCE:53988->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 290u IPv4 1683733255 0t0 TCP SOURCE:42736->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 291u IPv4 1681164350 0t0 TCP SOURCE:45294->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 292u IPv4 1683733256 0t0 TCP SOURCE:39670->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 293u IPv4 1683742501 0t0 TCP SOURCE:53740->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 294u IPv4 1683739620 0t0 TCP SOURCE:47310->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 295u IPv4 1683745178 0t0 TCP SOURCE:54706->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 296u IPv4 1684009384 0t0 TCP SOURCE:37564->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 297u IPv4 1683946019 0t0 TCP SOURCE:49088->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 298u IPv4 1684245417 0t0 TCP SOURCE:54556->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 299u IPv4 1684245431 0t0 TCP SOURCE:57114->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 300u IPv4 1684256396 0t0 TCP SOURCE:51490->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 301u IPv4 1684263246 0t0 TCP SOURCE:37310->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 302u IPv4 1684263282 0t0 TCP SOURCE:59114->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 303u IPv4 1684009385 0t0 TCP SOURCE:38270->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 304u IPv4 1677255941 0t0 TCP SOURCE:48118->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 305u IPv4 1684798763 0t0 TCP SOURCE:59632->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 306u IPv4 1684795620 0t0 TCP SOURCE:36862->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 307u IPv4 1684785817 0t0 TCP SOURCE:39414->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 308u IPv4 1681164366 0t0 TCP SOURCE:33796->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 309u IPv4 1684798785 0t0 TCP SOURCE:47860->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 310u IPv4 1684774009 0t0 TCP SOURCE:41426->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 311u IPv4 1684799853 0t0 TCP SOURCE:48814->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 312u IPv4 1676725813 0t0 TCP SOURCE:59290->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 313u IPv4 1685337516 0t0 TCP SOURCE:42582->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 314u IPv4 1685340346 0t0 TCP SOURCE:48068->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 315u IPv4 1685113271 0t0 TCP SOURCE:50620->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 316u IPv4 1685340349 0t0 TCP SOURCE:44996->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 317u IPv4 1685107329 0t0 TCP SOURCE:59048->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 318u IPv4 1685340350 0t0 TCP SOURCE:52614->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 319u IPv4 1685333743 0t0 TCP SOURCE:60002->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 320u IPv4 1678500644 0t0 TCP SOURCE:41418->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 321u IPv4 1681164401 0t0 TCP SOURCE:52936->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 322u IPv4 1685866465 0t0 TCP SOURCE:58404->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 323u IPv4 1685846044 0t0 TCP SOURCE:60956->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 324u IPv4 1685640303 0t0 TCP SOURCE:55332->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 325u IPv4 1681164402 0t0 TCP SOURCE:41158->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 326u IPv4 1681164403 0t0 TCP SOURCE:34730->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 327u IPv4 1685865450 0t0 TCP SOURCE:42118->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 328u IPv4 1686211156 0t0 TCP SOURCE:50746->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 329u IPv4 1686351219 0t0 TCP SOURCE:34026->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 330u IPv4 1686346896 0t0 TCP SOURCE:39488->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 331u IPv4 1686350114 0t0 TCP SOURCE:42040->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 332u IPv4 1686350115 0t0 TCP SOURCE:36422->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 333u IPv4 1686351225 0t0 TCP SOURCE:50480->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 334u IPv4 1686351226 0t0 TCP SOURCE:44046->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 335u IPv4 1686361452 0t0 TCP SOURCE:51434->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 336u IPv4 1686843600 0t0 TCP SOURCE:34048->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 337u IPv4 1686769136 0t0 TCP SOURCE:45566->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 338u IPv4 1686848589 0t0 TCP SOURCE:51034->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 339u IPv4 1686843616 0t0 TCP SOURCE:53592->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 340u IPv4 1686804845 0t0 TCP SOURCE:nimhub->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 341u IPv4 1686819546 0t0 TCP SOURCE:33822->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 342u IPv4 1686819547 0t0 TCP SOURCE:55620->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 343u IPv4 1686819548 0t0 TCP SOURCE:34776->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 344u IPv4 1687348195 0t0 TCP SOURCE:46022->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 345u IPv4 1687337202 0t0 TCP SOURCE:57544->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 346u IPv4 1687325159 0t0 TCP SOURCE:34774->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 347u IPv4 1687337203 0t0 TCP SOURCE:37332->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 348u IPv4 1687348208 0t0 TCP SOURCE:59946->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 349u IPv4 1687322261 0t0 TCP SOURCE:45778->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 350u IPv4 1687349920 0t0 TCP SOURCE:39360->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 351u IPv4 1687340030 0t0 TCP SOURCE:46752->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 352u IPv4 1687882837 0t0 TCP SOURCE:56662->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 353u IPv4 1687869690 0t0 TCP SOURCE:39948->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 354u IPv4 1687869699 0t0 TCP SOURCE:45416->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 355u IPv4 1687880096 0t0 TCP SOURCE:47968->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 356u IPv4 1687867204 0t0 TCP SOURCE:42350->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 357u IPv4 1687864733 0t0 TCP SOURCE:56402->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 358u IPv4 1687867425 0t0 TCP SOURCE:49974->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 359u IPv4 1687867426 0t0 TCP SOURCE:57362->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 360u IPv4 1688382162 0t0 TCP SOURCE:39430->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 361u IPv4 1688391352 0t0 TCP SOURCE:50942->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 362u IPv4 1688391365 0t0 TCP SOURCE:56404->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 363u IPv4 1688391366 0t0 TCP SOURCE:58956->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 364u IPv4 1688382251 0t0 TCP SOURCE:53332->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 365u IPv4 1688391399 0t0 TCP SOURCE:39152->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 366u IPv4 1688385769 0t0 TCP SOURCE:60950->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 367u IPv4 1688393906 0t0 TCP SOURCE:40106->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 368u IPv4 1683047857 0t0 TCP SOURCE:49626->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 369u IPv4 1688905952 0t0 TCP SOURCE:32906->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 370u IPv4 1688897574 0t0 TCP SOURCE:38368->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 371u IPv4 1688912908 0t0 TCP SOURCE:40932->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 372u IPv4 1688904726 0t0 TCP SOURCE:35308->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 373u IPv4 1688912909 0t0 TCP SOURCE:49360->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 374u IPv4 1688904727 0t0 TCP SOURCE:42932->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 375u IPv4 1688912089 0t0 TCP SOURCE:50320->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 376u IPv4 1689423946 0t0 TCP SOURCE:59786->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 377u IPv4 1689414453 0t0 TCP SOURCE:43078->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 378u IPv4 1689423223 0t0 TCP SOURCE:48540->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 379u IPv4 1689414454 0t0 TCP SOURCE:51102->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 380u IPv4 1689423228 0t0 TCP SOURCE:45484->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 381u IPv4 1689414455 0t0 TCP SOURCE:59536->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 382u IPv4 1689414456 0t0 TCP SOURCE:53102->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 383u IPv4 1689426115 0t0 TCP SOURCE:60490->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 384u IPv4 1689850991 0t0 TCP SOURCE:43746->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 385u IPv4 1689963258 0t0 TCP SOURCE:55258->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 386u IPv4 1689950614 0t0 TCP SOURCE:60720->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 387u IPv4 1689953958 0t0 TCP SOURCE:35052->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 388u IPv4 1689963282 0t0 TCP SOURCE:57672->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 389u IPv4 1689950712 0t0 TCP SOURCE:43504->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 390u IPv4 1689953959 0t0 TCP SOURCE:37070->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 391u IPv4 1689953960 0t0 TCP SOURCE:44458->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 392u IPv4 1690461659 0t0 TCP SOURCE:56780->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 393u IPv4 1690464102 0t0 TCP SOURCE:40074->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 394u IPv4 1690478983 0t0 TCP SOURCE:45536->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 395u IPv4 1690464103 0t0 TCP SOURCE:48088->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 396u IPv4 1690464104 0t0 TCP SOURCE:42464->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 397u IPv4 1690465271 0t0 TCP SOURCE:56522->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 398u IPv4 1690478999 0t0 TCP SOURCE:50094->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 399u IPv4 1690479018 0t0 TCP SOURCE:57484->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 400u IPv4 1690978976 0t0 TCP SOURCE:41966->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 401u IPv4 1690991152 0t0 TCP SOURCE:53488->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 402u IPv4 1690989921 0t0 TCP SOURCE:58950->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 403u IPv4 1690989925 0t0 TCP SOURCE:33282->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 404u IPv4 1690989927 0t0 TCP SOURCE:55890->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 405u IPv4 1690983864 0t0 TCP SOURCE:41716->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 406u IPv4 1690991168 0t0 TCP SOURCE:35282->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 407u IPv4 1690991169 0t0 TCP SOURCE:42674->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 408u IPv4 1691503342 0t0 TCP SOURCE:56680->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 409u IPv4 1691515455 0t0 TCP SOURCE:45430->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 410u IPv4 1691498425 0t0 TCP SOURCE:39968->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 411u IPv4 1691429518 0t0 TCP SOURCE:47988->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 412u IPv4 1691429519 0t0 TCP SOURCE:42364->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 413u IPv4 1691510324 0t0 TCP SOURCE:56422->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 414u IPv4 1691503492 0t0 TCP SOURCE:49988->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 415u IPv4 1691518072 0t0 TCP SOURCE:57382->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 416u IPv4 1691995512 0t0 TCP SOURCE:41600->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 417u IPv4 1692051540 0t0 TCP SOURCE:53118->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 418u IPv4 1692048532 0t0 TCP SOURCE:58580->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 419u IPv4 1692048537 0t0 TCP SOURCE:32900->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 420u IPv4 1692048541 0t0 TCP SOURCE:55514->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 421u IPv4 1692042143 0t0 TCP SOURCE:41346->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 422u IPv4 1692045727 0t0 TCP SOURCE:34918->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 423u IPv4 1692051651 0t0 TCP SOURCE:42312->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 424u IPv4 1692345247 0t0 TCP SOURCE:54016->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 425u IPv4 1692563165 0t0 TCP SOURCE:37296->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 426u IPv4 1692557550 0t0 TCP SOURCE:42758->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 427u IPv4 1692563224 0t0 TCP SOURCE:45316->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 428u IPv4 1692563225 0t0 TCP SOURCE:39706->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 429u IPv4 1692563228 0t0 TCP SOURCE:53762->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 430u IPv4 1692563272 0t0 TCP SOURCE:47328->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 431u IPv4 1692560072 0t0 TCP SOURCE:54716->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 432u IPv4 1693044490 0t0 TCP SOURCE:38848->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 433u IPv4 1693066298 0t0 TCP SOURCE:50372->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 434u IPv4 1693061475 0t0 TCP SOURCE:55846->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 435u IPv4 1693061476 0t0 TCP SOURCE:58398->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 436u IPv4 1693063694 0t0 TCP SOURCE:52774->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 437u IPv4 1693066421 0t0 TCP SOURCE:38612->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 438u IPv4 1693055583 0t0 TCP SOURCE:60410->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 439u IPv4 1693061701 0t0 TCP SOURCE:39578->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 440u IPv4 1693566843 0t0 TCP SOURCE:51756->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 441u IPv4 1693558730 0t0 TCP SOURCE:35042->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 442u IPv4 1693544321 0t0 TCP SOURCE:40504->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 443u IPv4 1681165051 0t0 TCP SOURCE:43062->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 444u IPv4 1693560906 0t0 TCP SOURCE:37438->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 445u IPv4 1693573162 0t0 TCP SOURCE:51490->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 446u IPv4 1693572342 0t0 TCP SOURCE:45056->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 447u IPv4 1693551007 0t0 TCP SOURCE:52444->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 448u IPv4 1694046294 0t0 TCP SOURCE:35358->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 449u IPv4 1694049740 0t0 TCP SOURCE:46876->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 450u IPv4 1694050327 0t0 TCP SOURCE:52344->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 451u IPv4 1694032841 0t0 TCP SOURCE:54896->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 452u IPv4 1694032849 0t0 TCP SOURCE:49272->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 453u IPv4 1694050412 0t0 TCP SOURCE:35094->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 454u IPv4 1694048837 0t0 TCP SOURCE:56892->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 455u IPv4 1694050492 0t0 TCP SOURCE:36060->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 456u IPv4 1694485102 0t0 TCP SOURCE:46950->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 457u IPv4 1694486508 0t0 TCP SOURCE:58462->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 458u IPv4 1694486782 0t0 TCP SOURCE:35692->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 459u IPv4 1694496140 0t0 TCP SOURCE:38250->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 460u IPv4 1694486785 0t0 TCP SOURCE:60864->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 461u IPv4 1694493293 0t0 TCP SOURCE:46684->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 462u IPv4 1694496236 0t0 TCP SOURCE:40262->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 463u IPv4 1694498278 0t0 TCP SOURCE:47652->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 464u IPv4 1694569164 0t0 TCP SOURCE:58466->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 465u IPv4 1694892894 0t0 TCP SOURCE:41752->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 466u IPv4 1694884037 0t0 TCP SOURCE:47214->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 467u IPv4 1694896482 0t0 TCP SOURCE:49766->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 468u IPv4 1694896483 0t0 TCP SOURCE:44142->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 469u IPv4 1694892896 0t0 TCP SOURCE:58200->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 470u IPv4 1694895164 0t0 TCP SOURCE:51772->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 471u IPv4 1694896488 0t0 TCP SOURCE:59160->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 472u IPv4 1695271966 0t0 TCP SOURCE:42580->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 473u IPv4 1695277485 0t0 TCP SOURCE:54098->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 474u IPv4 1695259451 0t0 TCP SOURCE:59560->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 475u IPv4 1695271069 0t0 TCP SOURCE:33880->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 476u IPv4 1695259452 0t0 TCP SOURCE:56500->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 477u IPv4 1695277488 0t0 TCP SOURCE:42320->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 478u IPv4 1695271070 0t0 TCP SOURCE:35892->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 479u IPv4 1695277493 0t0 TCP SOURCE:43280->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 480u IPv4 1695499211 0t0 TCP SOURCE:52950->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 481u IPv4 1695499212 0t0 TCP SOURCE:36230->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 482u IPv4 1695609813 0t0 TCP SOURCE:41692->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 483u IPv4 1695624996 0t0 TCP SOURCE:44244->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 484u IPv4 1695624997 0t0 TCP SOURCE:38626->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 485u IPv4 1695619653 0t0 TCP SOURCE:52678->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 486u IPv4 1695619654 0t0 TCP SOURCE:46244->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 487u IPv4 1695613091 0t0 TCP SOURCE:53632->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 488u IPv4 1695932453 0t0 TCP SOURCE:34362->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 489u IPv4 1695934501 0t0 TCP SOURCE:45874->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 490u IPv4 1695905572 0t0 TCP SOURCE:51336->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 491u IPv4 1695922685 0t0 TCP SOURCE:53888->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 492u IPv4 1695922686 0t0 TCP SOURCE:48264->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 493u IPv4 1695932454 0t0 TCP SOURCE:34094->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 494u IPv4 1695932319 0t0 TCP SOURCE:55894->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 495u IPv4 1695932320 0t0 TCP SOURCE:35050->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 496u IPv4 1696179022 0t0 TCP SOURCE:44602->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 497u IPv4 1696240031 0t0 TCP SOURCE:56120->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 498u IPv4 1696240032 0t0 TCP SOURCE:33350->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 499u IPv4 1696217972 0t0 TCP SOURCE:35908->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 500u IPv4 1696245526 0t0 TCP SOURCE:58516->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 501u IPv4 1696240033 0t0 TCP SOURCE:44336->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 502u IPv4 1696245527 0t0 TCP SOURCE:37908->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 503u IPv4 1696245528 0t0 TCP SOURCE:45296->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 504u IPv4 1696558188 0t0 TCP SOURCE:53380->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 505u IPv4 1696559150 0t0 TCP SOURCE:36660->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 506u IPv4 1696553544 0t0 TCP SOURCE:42134->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 507u IPv4 1696553545 0t0 TCP SOURCE:44686->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 508u IPv4 1696553546 0t0 TCP SOURCE:39062->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 509u IPv4 1696558189 0t0 TCP SOURCE:53114->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 510u IPv4 1696553548 0t0 TCP SOURCE:46704->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 511u IPv4 1696556533 0t0 TCP SOURCE:54092->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 512u IPv4 1696873256 0t0 TCP SOURCE:33752->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 513u IPv4 1696877765 0t0 TCP SOURCE:45264->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 514u IPv4 1696877766 0t0 TCP SOURCE:50726->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 515u IPv4 1696877767 0t0 TCP SOURCE:53278->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 516u IPv4 1696877768 0t0 TCP SOURCE:47654->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 517u IPv4 1696877769 0t0 TCP SOURCE:33474->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 518u IPv4 1696877770 0t0 TCP SOURCE:55272->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 519u IPv4 1696877771 0t0 TCP SOURCE:34428->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 520u IPv4 1697178341 0t0 TCP SOURCE:43298->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 521u IPv4 1697177088 0t0 TCP SOURCE:54816->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 522u IPv4 1697184077 0t0 TCP SOURCE:60278->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 523u IPv4 1697184081 0t0 TCP SOURCE:34604->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 524u IPv4 1697184082 0t0 TCP SOURCE:57212->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 525u IPv4 1697184083 0t0 TCP SOURCE:43032->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 526u IPv4 1697184084 0t0 TCP SOURCE:36598->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 527u IPv4 1697184085 0t0 TCP SOURCE:43986->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 528u IPv4 1697475659 0t0 TCP SOURCE:52372->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 529u IPv4 1697479770 0t0 TCP SOURCE:35652->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 530u IPv4 1697479771 0t0 TCP SOURCE:41114->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 531u IPv4 1697479772 0t0 TCP SOURCE:43666->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 532u IPv4 1697479773 0t0 TCP SOURCE:38042->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 533u IPv4 1697478726 0t0 TCP SOURCE:52106->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 534u IPv4 1697479777 0t0 TCP SOURCE:45672->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 535u IPv4 1697479779 0t0 TCP SOURCE:53066->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 536u IPv4 1697729899 0t0 TCP SOURCE:60484->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 537u IPv4 1697729900 0t0 TCP SOURCE:43764->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 538u IPv4 1697778308 0t0 TCP SOURCE:49232->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 539u IPv4 1697778309 0t0 TCP SOURCE:51784->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 540u IPv4 1697775151 0t0 TCP SOURCE:46160->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 541u IPv4 1697775152 0t0 TCP SOURCE:60212->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 542u IPv4 1697775153 0t0 TCP SOURCE:53784->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 543u IPv4 1697778313 0t0 TCP SOURCE:32940->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 544u IPv4 1698100189 0t0 TCP SOURCE:40462->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 545u IPv4 1698113938 0t0 TCP SOURCE:51974->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 546u IPv4 1698112734 0t0 TCP SOURCE:57442->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 547u IPv4 1698113941 0t0 TCP SOURCE:59994->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 548u IPv4 1698113942 0t0 TCP SOURCE:54370->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 549u IPv4 1698113943 0t0 TCP SOURCE:40190->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 550u IPv4 1698113944 0t0 TCP SOURCE:33756->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 551u IPv4 1698113945 0t0 TCP SOURCE:41144->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 552u IPv4 1698349191 0t0 TCP SOURCE:46432->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 553u IPv4 1698360038 0t0 TCP SOURCE:57944->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 554u IPv4 1698360039 0t0 TCP SOURCE:35174->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 555u IPv4 1698360041 0t0 TCP SOURCE:37732->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 556u IPv4 1698360042 0t0 TCP SOURCE:60340->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 557u IPv4 1698360043 0t0 TCP SOURCE:46160->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 558u IPv4 1698360044 0t0 TCP SOURCE:39726->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 559u IPv4 1698360045 0t0 TCP SOURCE:47114->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 560u IPv4 1698557751 0t0 TCP SOURCE:51564->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 561u IPv4 1698557780 0t0 TCP SOURCE:34844->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 562u IPv4 1698557781 0t0 TCP SOURCE:40306->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 563u IPv4 1698557782 0t0 TCP SOURCE:42858->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 564u IPv4 1698552079 0t0 TCP SOURCE:37234->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 565u IPv4 1698552080 0t0 TCP SOURCE:51286->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 566u IPv4 1698504676 0t0 TCP SOURCE:44858->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 567u IPv4 1698552084 0t0 TCP SOURCE:52252->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 568u IPv4 1698744895 0t0 TCP SOURCE:56082->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 569u IPv4 1698735038 0t0 TCP SOURCE:39362->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 570u IPv4 1698751827 0t0 TCP SOURCE:44824->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 571u IPv4 1698744904 0t0 TCP SOURCE:47382->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 572u IPv4 1698735039 0t0 TCP SOURCE:41759->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 573u IPv4 1698744905 0t0 TCP SOURCE:55810->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 574u IPv4 1698735040 0t0 TCP SOURCE:49376->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 575u IPv4 1698720095 0t0 TCP SOURCE:56764->DESTINATION_H:2376 (ESTABLISHED)
portainer 24681 portainer 576u IPv4 1698924141 0t0 TCP SOURCE:60916->DESTINATION_A:2376 (ESTABLISHED)
portainer 24681 portainer 577u IPv4 1698930714 0t0 TCP SOURCE:44208->DESTINATION_B:2376 (ESTABLISHED)
portainer 24681 portainer 578u IPv4 1698926888 0t0 TCP SOURCE:49670->DESTINATION_C:2376 (ESTABLISHED)
portainer 24681 portainer 579u IPv4 1698926889 0t0 TCP SOURCE:52222->DESTINATION_D:2376 (ESTABLISHED)
portainer 24681 portainer 580u IPv4 1698926890 0t0 TCP SOURCE:46600->DESTINATION_E:2376 (ESTABLISHED)
portainer 24681 portainer 581u IPv4 1698926891 0t0 TCP SOURCE:60652->DESTINATION_F:2376 (ESTABLISHED)
portainer 24681 portainer 582u IPv4 1698926892 0t0 TCP SOURCE:54224->DESTINATION_G:2376 (ESTABLISHED)
portainer 24681 portainer 583u IPv4 1698926893 0t0 TCP SOURCE:33382->DESTINATION_H:2376 (ESTABLISHED)
_Edit by xAt0mZ for better lisibility_
daneZz
on 6 Aug 2018
Any Updates on this? I ran into the same issue again running portainer simply with:
docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
On an AWSLinux2 Host connected to 4 Docker Endpoints. It wasn't even running for 24 Hours before i had to restart portainer.
Portainer Version: 1.19.1
Docker Version: Docker version 18.03.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
OS: AWS Linux 2
mrWinston
on 8 Aug 2018
I got the same issue , that I have to reboot the hosts everyday
yexingqi
on 21 Aug 2018
I am experiencing the same issue as well. I am running v1.19.1 on an AWS EC2 container
guppie70
on 21 Aug 2018
I'm curious if this could be caused by the recently introduced snapshot feature. Can anyone try to reproduce this error with the snapshot feature disabled ? Use the --no-snapshot flag to disable it.
E.g. docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer --no-snaspshot
deviantony
on 22 Aug 2018
Have never heard of a snapshot feature - and if I search in the documentation (https://portainer.readthedocs.io/en/stable/search.html?q=snapshot&check_keywords=yes&area=default) for "snapshot" nothing pops up. So I have no clue if adding the flag will help.
guppie70
on 22 Aug 2018
@guppie70 snapshots were introduced in 1.19.1, it will periodically poll the endpoints you have defined in Portainer to refresh the information available in the home view.
Flags are missing from the configuration section of the docs, will update it.
I'm thinking that this might be related to the snapshot feature but I'm not sure so it would be keen of you to try with this flag.
deviantony
on 22 Aug 2018
@deviantony I have re-initiated my portainer container with the --no-snapshot flag. Since I do not know how to enforce the error, we will need to wait and see if portainer keeps running in the coming period. I am happy to share any other information that might help you to locate the problem though. Just let me know!
guppie70
on 22 Aug 2018
Awesome, I do not have any lead on this problem yet. Please report it there if you still encounter the issue.
deviantony
on 22 Aug 2018
Got the same issue on our setup:
- Docker Swarm
- Portainer 1.19.1
- Running Portainer Agent Stack Deployment
It takes 2 - 3 days till these errors come up. I only need to restart portainer UI. Agents can keep on running.
We see three error types:
First, Portainer UI gets 4 - 8 of the following errors, 5 minutes in between these errors:
cron error: endpoint snapshot error (endpoint=someHost, URL=tcp://tasks.agent:9001) (err=Cannot connect to the Docker daemon at tcp://tasks.agent:9001. Is the docker daemon running?)
Afterwards it becomes, 5 minutes in between these errors:
cron error: endpoint snapshot error (endpoint=someHost, URL=tcp://tasks.agent:9001) (err=error during connect: Get https://tasks.agent:9001/_ping: dial tcp: lookup tasks.agent on 127.0.0.11:53: dial udp 127.0.0.11:53: socket: too many open files)
After 4 - 5 hours logs get flooded by this error, one line each second:
http: Accept error: accept tcp [::]:9000: accept4: too many open files; retrying in 1s
Error type 2 still comes up every 5 minutes.
Will try --no-snapshot as well.
CWiesbaum
on 28 Aug 2018
Having same issue almost every day. Swarm with 5 nodes, ~300 containers on AWS.
I'll be glad to establish chat contact with you (gtalk, slack etc) to be able to provide more information about problem (straces etc) while it happening
random-ua
on 30 Aug 2018
Anybody keen to try the portainer/portainer:fix1991-open-files image (linux amd64 build) and report if the problem is solved?
ping @random-ua @CWiesbaum @guppie70 @daneZz @mrWinston @Umaaz
deviantony
on 1 Sep 2018
@deviantony
Portainer seems to be running much more stable with the --no-snaspshot added. It's now up for 4 days without any issues!
guppie70
on 2 Sep 2018
The --no-snapshot flag is not a solution here, it was just used to isolate the cause of the problem.
I believe that #2235 solves this issue, waiting for some feedback from other users before we decide to merge it.
deviantony
on 3 Sep 2018
@deviantony we are now running portainer/portainer:1.19.2 and we are still facing the issue when using the portainer/agent:latest
2018/09/26 21:20:37 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Please let us know if we need to make any other changes.
peteryhwong
on 27 Sep 2018
Will re-open as we only fix this for the Portainer part actually, not for the agent.
deviantony
on 27 Sep 2018
@deviantony thanks for this. In the mean time would --no-snapshot help?
peteryhwong
on 27 Sep 2018
Not sure but you can try it and report if it brings any change, that would help our investigation.
deviantony
on 27 Sep 2018
@deviantony FYI so far adding --no-snapshot seems to have avoid the issue.
peteryhwong
on 1 Oct 2018
I had no luck with the --no-snapshot for the agent. Still the error occurs after a day. I have Amazon Linux 2 image running in Vbox. Compose file:
agent:
image: portainer/agent
environment:
AGENT_CLUSTER_ADDR: tasks.agent
#LOG_LEVEL: debug
command: --no-snapshot # https://github.com/portainer/portainer/issues/1991
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- portainer_agent
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
restart_policy:
delay: 30s
window: 30s
condition: on-failure
logging:
driver: journald
Premik
on 12 Nov 2018
@Premik can you see this error in portainer logs or agent logs?
deviantony
on 12 Nov 2018
This was only in the agent log. Not in the Portainer log.
One thing I've noticed is the error usually affected agent only on the node where the Swarm manager is running. I have two other worker nodes and I haven't seen the error there.
Premik
on 12 Nov 2018
@Premik thanks for the report, yes this issue is related to the agent. The --no-snapshot was supposed to fix the same bug Portainer side in 1.19.1, so no need to keep this flag.
We're investigating this problem.
deviantony
on 13 Nov 2018
@Premik Are you seeing other issues with your swarm itself? Or other containers? I've experienced several times if I have a service that fails to start, and I leave it for several hours, it will continually try to start and fail. This consumes filehands and causes the ec2 instance to end up failing.
kendrickm
on 13 Nov 2018
@kendrickm It could be. I had some failing containers here and there since this is may playground environment. But I restarted the Docker daemon pretty soon after I noticed the high cpu usage so can't tell for sure. I'll see if I can isolate the problem more the next time.
Although I think the file handles were drained only in the agent's container. I run both portainer and agent in a container and docker sets the default ulimits pretty low (1024 for the file handlers I think).
Following is the system log around that time:
75e975d6612f = agent
dacabed9f5bb = portainer
Nov 12 21:30:01 node1 systemd[1]: Created slice User Slice of root.
Nov 12 21:30:01 node1 systemd[1]: Starting User Slice of root.
Nov 12 21:30:01 node1 systemd[1]: Started Session 703 of user root.
Nov 12 21:30:01 node1 systemd[1]: Starting Session 703 of user root.
Nov 12 21:30:01 node1 CROND[30587]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Nov 12 21:30:01 node1 systemd[1]: Removed slice User Slice of root.
Nov 12 21:30:01 node1 systemd[1]: Stopping User Slice of root.
Nov 12 21:33:30 node1 dockerd[30308]: time="2018-11-12T21:33:30.237017361Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:9kjzpblmb4imxanaqvob58y05 leaving:false netPeers:3 entries:9 Queue qLen:0 netMsg/s:0"
Nov 12 21:33:30 node1 dockerd[30308]: time="2018-11-12T21:33:30.237190457Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:vqxianltzhxqfxez24nv56eq7 leaving:false netPeers:3 entries:14 Queue qLen:0 netMsg/s:0"
Nov 12 21:33:30 node1 dockerd[30308]: time="2018-11-12T21:33:30.237346370Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:utqf7lwusdr6be6ggee2z3i33 leaving:false netPeers:3 entries:11 Queue qLen:0 netMsg/s:0"
Nov 12 21:38:30 node1 dockerd[30308]: time="2018-11-12T21:38:30.238596045Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:vqxianltzhxqfxez24nv56eq7 leaving:false netPeers:3 entries:14 Queue qLen:0 netMsg/s:0"
Nov 12 21:38:30 node1 dockerd[30308]: time="2018-11-12T21:38:30.238726712Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:utqf7lwusdr6be6ggee2z3i33 leaving:false netPeers:3 entries:11 Queue qLen:0 netMsg/s:0"
Nov 12 21:38:30 node1 dockerd[30308]: time="2018-11-12T21:38:30.238779981Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:9kjzpblmb4imxanaqvob58y05 leaving:false netPeers:3 entries:9 Queue qLen:0 netMsg/s:0"
Nov 12 21:40:01 node1 systemd[1]: Created slice User Slice of root.
Nov 12 21:40:01 node1 systemd[1]: Starting User Slice of root.
Nov 12 21:40:01 node1 systemd[1]: Started Session 704 of user root.
Nov 12 21:40:01 node1 systemd[1]: Starting Session 704 of user root.
Nov 12 21:40:01 node1 CROND[30800]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Nov 12 21:40:01 node1 systemd[1]: Removed slice User Slice of root.
Nov 12 21:40:01 node1 systemd[1]: Stopping User Slice of root.
Nov 12 21:43:18 node1 dhclient[4746]: DHCPREQUEST on eth1 to 192.168.1.1 port 67 (xid=0x42f1443d)
Nov 12 21:43:18 node1 dhclient[4746]: DHCPACK from 192.168.1.1 (xid=0x42f1443d)
Nov 12 21:43:20 node1 dhclient[4746]: bound to 192.168.1.202 -- renewal in 15082 seconds.
Nov 12 21:43:30 node1 dockerd[30308]: time="2018-11-12T21:43:30.435330566Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:9kjzpblmb4imxanaqvob58y05 leaving:false netPeers:3 entries:9 Queue qLen:0 netMsg/s:0"
Nov 12 21:43:30 node1 dockerd[30308]: time="2018-11-12T21:43:30.435681069Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:vqxianltzhxqfxez24nv56eq7 leaving:false netPeers:3 entries:14 Queue qLen:0 netMsg/s:0"
Nov 12 21:43:30 node1 dockerd[30308]: time="2018-11-12T21:43:30.435722077Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:utqf7lwusdr6be6ggee2z3i33 leaving:false netPeers:3 entries:11 Queue qLen:0 netMsg/s:0"
Nov 12 21:48:30 node1 dockerd[30308]: time="2018-11-12T21:48:30.438736620Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:vqxianltzhxqfxez24nv56eq7 leaving:false netPeers:3 entries:14 Queue qLen:0 netMsg/s:0"
Nov 12 21:48:30 node1 dockerd[30308]: time="2018-11-12T21:48:30.438922906Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:utqf7lwusdr6be6ggee2z3i33 leaving:false netPeers:3 entries:11 Queue qLen:0 netMsg/s:0"
Nov 12 21:48:30 node1 dockerd[30308]: time="2018-11-12T21:48:30.439008482Z" level=info msg="NetworkDB stats node1(d6efd5ef8ef9) - netID:9kjzpblmb4imxanaqvob58y05 leaving:false netPeers:3 entries:9 Queue qLen:0 netMsg/s:0"
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 5ms
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: proxy error: dial tcp 10.0.2.10:9001: socket: too many open files
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 10ms
Nov 12 21:49:13 node1 dacabed9f5bb[30308]: 2018/11/12 21:49:13 http: proxy error: unexpected end of JSON input
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 20ms
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 40ms
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 80ms
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 160ms
Nov 12 21:49:13 node1 75e975d6612f[30308]: 2018/11/12 21:49:13 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 320ms
Nov 12 21:49:14 node1 75e975d6612f[30308]: 2018/11/12 21:49:14 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 640ms
Nov 12 21:49:14 node1 75e975d6612f[30308]: 2018/11/12 21:49:14 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 1s
Nov 12 21:49:15 node1 75e975d6612f[30308]: 2018/11/12 21:49:15 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 1s
Nov 12 21:49:16 node1 75e975d6612f[30308]: 2018/11/12 21:49:16 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 1s
Nov 12 21:49:17 node1 75e975d6612f[30308]: 2018/11/12 21:49:17 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 1s
Nov 12 21:49:18 node1 75e975d6612f[30308]: 2018/11/12 21:49:18 http: Accept error: accept tcp [::]:9001: accept4: too many open files; retrying in 1s
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
Nov 12 21:49:19 node1 75e975d6612f[30308]: 2018/11/12 21:49:19 [ERR] memberlist: Error accepting TCP connection: accept tcp [::]:7946: accept4: too many open files
I also encounter this bug. My setup is a swarm cluster having 2 nodes (1 master and 1 worker). I observe that each time I load / refresh the dashboard page (ie https://abc.yyy/portainer/#/dashboard), the portainer agent containers have increasing number of open files (/proc/agent proc id/fd). This leak also happens on previous verions: 1.18.1, 1.19.1, 1.19.2 and 1.20.0.
tle211212
on 12 Dec 2018
can you try the following image portainer/agent:fix1991 to see if it fixes the problem?
chiptus
on 13 Dec 2018
portainer/agent:fix1991
I tried agent:fix1991 but the leak is still not resolved.
tle211212
on 13 Dec 2018
Thanks for trying, we will continue trying with other solutions.
On Thu, Dec 13, 2018, 16:51 tle211212 <[email protected] wrote:
portainer/agent:fix1991
I tried agent:fix1991 but the leak is still not resolved.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/portainer/portainer/issues/1991#issuecomment-446995224,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUVF81XNA6Wt2H9mb_QYuVRtsIJAqdrks5u4mltgaJpZM4Uz2-d
.
chiptus
on 13 Dec 2018
@tle211212
I also encounter this bug. My setup is a swarm cluster having 2 nodes (1 master and 1 worker). I observe that each time I load / refresh the dashboard page (ie https://abc.yyy/portainer/#/dashboard), the portainer agent containers have increasing number of open files (/proc/agent proc id/fd). This leak also happens on previous verions: 1.18.1, 1.19.1, 1.19.2 and 1.20.0.
I can't reproduce this one, I have an agent deployed locally and on a swarm and both file descriptor count stay the same (I tried refreshing the dashboard, changing pages...)
chiptus
on 16 Dec 2018
Hi chiptus, here is the detail of my setup.
OS: Centos7 + Oracle UEK R5 kernel
Docker: docker-ce-18.06.1.ce-3.el7.x86_64
The yml file I used to deploy portainer is attached for your reference.
tle211212
on 16 Dec 2018
I can confirm the dashboard reloading and fd going up behavior. I created a Vagrant+Ansible+AWS Linux2 image setup to reproduce this.
v.zip
1) Unzip the three files in the attachment to a single folder and run vagrant up there.
1) This would provision two VBox nodes.
Note: There should be one Bridged Net adapter attached to each node (portnode1=081027D43801, portnode2=081027D43802). For some reason (Vagrant bug?) sometimes I got node2 with two Bridged adapters instead. If that happens stop the node2 and remove the extra network adapter and start it back up again.
1) On first run Ansible should print a debug message with Portainer port and details. Eventually re-run later with vagrant provision
1) vagrant ssh portnode1 to get to the node 1 shell and run:
sudo su
export p=$(docker inspect --format '{{.State.Pid}}' $(docker ps | grep agent | cut -c-12))
yes 'ls /proc/$p/fd | wc -l; sleep 2' | /usr/bin/sh
This will list the number of Agent's process fds every 2 seconds.
1) Go to portainer web portnode2:9000/#/dashboard and click between Dashboard, Stack and back etc.The number of fd would go up aprox 9 new on each refresh..
Note:
- It seems at least two nodes are required. I couldn't reproduce this while using a single node only.
- Ansible 2.8.0+ (dev version) is required because of the Docker modules
Premik
on 16 Dec 2018
any update for this bug? same on AWS AMI2
iwater
on 20 Feb 2019
Guys, we have a potential fix for this issue - would you be able to test this image for us?
portainerci/agent:feat49-update-library
ssbkang
on 14 Mar 2019
Hi,
I've tried the portainerci/agent:feat49-update-library:
[root@portnode1 vagrant]# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
wuh6dfrc0ztc portainerStack_agent global 2/2 portainerci/agent:feat49-update-library
cqn3heevayfi portainerStack_portainer replicated 1/1 portainer/portainer:latest *:9000->9000/tcp
but unfortunately no luck. The number of FDs is still growing up on each page refresh:
[root@portnode1 vagrant]# yes 'ls /proc/$p/fd | wc -l; sleep 10' | /usr/bin/sh
78
83
85
85
92
118
125
125
129
137
148
160
160
edit: Adding logs. I can see some TLS errors. Not sure if this is related:
Portainer:
2019/03/17 16:05:52 Creating admin user with password hash $2y$05$cTEbk/D17AFDT4bOmEgD.eOxx8/StO5HkICdKHmClxL9IJ2a6yeWi,
2019/03/17 16:05:52 Starting Portainer 1.20.2 on :9000,
2019/03/17 16:05:55 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2019/03/17 16:05:55 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2019/03/17 16:05:55 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
2019/03/17 16:05:55 http error: Invalid JWT token (err=Invalid JWT token) (code=401),
Agent1:
2019/03/17 16:05:47 [DEBUG] - Agent details: map[NodeName:portnode1 Mode:swarm NodeRole:manager AgentPort:9001],
2019/03/17 16:05:47 [DEBUG] - Using cluster address: tasks.agent,
2019/03/17 16:05:47 [DEBUG] - Using advertiseAddr: 10.0.1.5,
2019/03/17 16:05:47 [DEBUG] - Generating TLS files...,
2019/03/17 16:05:47 [DEBUG] - Using agent port: 9001,
2019/03/17 16:05:50 [DEBUG] - Serf configured with AdvertiseAddr: 10.0.1.5,
2019/03/17 16:05:50 [INFO] serf: EventMemberJoin: 52f06fab0640 10.0.1.5,
2019/03/17 16:05:50 [DEBUG] - Will join cluster via: tasks.agent,
2019/03/17 16:05:50 [INFO] serf: EventMemberJoin: 0cfc8aea9ae6 10.0.1.6,
2019/03/17 16:05:50 [INFO] - Starting Portainer agent version 1.2.1 on 0.0.0.0:9001 (cluster mode: true),
2019/03/17 16:09:05 http: TLS handshake error from 10.0.1.8:50102: EOF,
2019/03/17 16:10:50 http: TLS handshake error from 10.0.1.8:50158: EOF,
2019/03/17 16:10:50 http: TLS handshake error from 10.0.1.8:50160: EOF,
Agent2:
2019/03/17 16:05:47 [DEBUG] - Agent details: map[NodeRole:worker AgentPort:9001 NodeName:portnode2 Mode:swarm],
2019/03/17 16:05:47 [DEBUG] - Using cluster address: tasks.agent,
2019/03/17 16:05:47 [DEBUG] - Using advertiseAddr: 10.0.1.6,
2019/03/17 16:05:47 [DEBUG] - Generating TLS files...,
2019/03/17 16:05:47 [DEBUG] - Using agent port: 9001,
2019/03/17 16:05:50 [DEBUG] - Serf configured with AdvertiseAddr: 10.0.1.6,
2019/03/17 16:05:50 [INFO] serf: EventMemberJoin: 0cfc8aea9ae6 10.0.1.6,
2019/03/17 16:05:50 [DEBUG] - Will join cluster via: tasks.agent,
2019/03/17 16:05:50 [INFO] serf: EventMemberJoin: 52f06fab0640 10.0.1.5,
2019/03/17 16:05:50 [INFO] - Starting Portainer agent version 1.2.1 on 0.0.0.0:9001 (cluster mode: true),
2019/03/17 16:05:52 http error: Missing request signature headers (err=Unauthorized) (code=403),
Hi there,
We have been able to reproduce this bug based on Premik's advice and instructions. Hopefully this will help us get to the bottom of this bug! Our reproduction steps are below.
Reproducing using a Stack file
Step 1: Spin up 2 Vagrant VMs to form a Docker cluster
Step 2: Connect to the manager node via ssh:
vagrant ssh swarmnode1
Step 3: Using our portainer-agent-stack file add the Agent and Portainer UI to the swarm
curl -L https://downloads.portainer.io/portainer-agent-stack.yml \
-o portainer-agent-stack.yml
docker stack deploy --compose-file=portainer-agent-stack.yml portainer
Step 4: Enter below commands suggested by Premik (we used /bin/sh instead of /usr/bin/sh)
sudo su
export p=$(docker inspect --format '{{.State.Pid}}' $(docker ps | grep agent | cut -c-12))
yes 'ls /proc/$p/fd | wc -l; sleep 2' | /bin/sh
Every 2 seconds the number of pages loaded in the Agent will be displayed.
Step 5: Open the Portainer UI in the browser, create an account and navigate to the dashboard. _Triggering the behaviour:_ Reloading the browser on the dashboard as well as switching between the dashboard and stacks should increase number of pages loaded by between 1&7 and will occasionally decrease by 1 before going up again.
Reproducing as Separate Services
When deploying the Agent and the Portainer UI separately as services, this behaviour is repeatable.
Deployment commands:
What doesn't work
- When the Agent and Portainer UI are deployed on a singular manager node using the stack file deployment used earlier this behaviour was not repeatable. (User Premik who suggested this method had the same behaviour on a single node.)
- When the Agent is deployed from a stack file saved on the vagrant vm using the CLI and the Portainer UI is deployed using the docker run command below, and Portainer is connected to the Agent endpoint, this behaviour is NOT repeatable.
Deployment commands used:
itsconquest
on 19 Mar 2019
We have done some further investigation into this issue.
The ever increasing fd behaviour is reproducible when executing GET requests to the following endpoints of the API, only after loading the Portainer dashboard for the first time. If you don’t physically load the dashboard in the browser first, get requests to the following endpoints won’t trigger the behaviour.
Step 1: Follow the instructions in my previous comment for reproducing using a stack file. This will output the number of fd every 2s in your CLI
Step 2: Navigate to endpoint-ip:9000, create your user and navigate to the dashboard
Step 3: Refresh the browser
Step 4: Use the CLI or a tool like postman to complete get requests against one or more of the above endpoints
_You should see the number of fd increase_
Stopping the behaviour:
- Step 1: Load the dashboard in the browser and refresh a few times to trigger the behaviour
Step 2: Go to another view e.g services and refresh the page a few times and don’t go back to dashboard.
Step 3: Execute one or more get requests against the above endpoints
_You should now see this doesn’t increase the total fd. However loading the dashboard again and refreshing a few times will trigger it again_
Other things we tried/discovered
If we took all the requests that the dashboard makes and executed them together in postman in the same order, this behaviour can be reproduced, only after first loading the dashboard. If you execute these requests, without first loading the dashboard, this behaviour will not be reproduced.
Swarm visualiser triggers this behaviour every 5s.
Step 1: Log into Portainer in the browser and load the dashboard and make sure that the behaviour is triggered (if this hasn’t triggered the behaviour refresh it a few times and it should).
Step 2: Enter the swarm visualiser, and now you will notice fd is incremented every 5 seconds
If swarm visualiser is left triggering this behaviour every 5s, then memory used by the agent slowly increases over time.
- Following on from the above steps, enter the below command to output physical memory usage of the agent process every 2 seconds:
while [ 1 ] ; do
grep VmRSS /proc/$p/status ; sleep 2
done
- This behaviour can also be observed via the top command, the below value highlighted in red should keep increasing:
itsconquest
on 20 Mar 2019
I've reproduced @itsconquest's result above, and discovered that the issue only occurs when I have non-master nodes in my cluster. I.e., I deployed 2 nodes using https://github.com/deviantony/vagrant-swarm-cluster (per the above instructions), and sure enough, the issue is present.
Then I promoted the worker node to a master, and redeployed portainer+agent. I was unable to reproduce the increasing-amount-of-FDs problem.
Once I demoted the worker node again, and redeployed, the increasing-amount-of-FDs problem recurred.
This might explain why the problem doesn't exist on a single-node swarm.
D
funkypenguin
on 20 Jun 2019
Has any progress been made on this?
respectTheCode
on 17 Sep 2019
@respectTheCode no update on this yet, although we recently open-sourced the agent, any help appreciated ! https://github.com/portainer/agent
deviantony
on 17 Sep 2019
Closing via https://github.com/portainer/agent/issues/43
Will be part of the agent version 1.5.0.
deviantony
on 20 Sep 2019
Hi,
The problem still remains even with the last version, i haven't notice that in the past but now i understand why sometimes my disks was full sometimes
alfonsodg
on 18 Oct 2020
Related issues
cbrherms
·
3Comments
adi90x
·
4Comments
himred
·
3Comments
chrisvanderpennen
·
3Comments
aprcloud
·
3Comments
Most helpful comment
@deviantony
Portainer seems to be running much more stable with the
--no-snaspshotadded. It's now up for 4 days without any issues!