Portainer: "no such exec" failure in Container console on remote host

Thanks for the detailed report ! I'll try to reproduce.

Unfortunately I was not able to reproduce using the following steps:

• Deploy Portainer locally
• Add a new endpoint being a remote Swarm manager node
• Create a new service using the image nginx:latest and a replica count of 1
• Wait for the service to be deployed and check where the container is located
• Add a new endpoint being the remote Swarm worker where the container is located
• Access the container console view for that container
• Start the console, works without issue

docker exec works using the CLI against these containers.

How do you use the CLI ? Do you use the -H host:port flag or do you use the CLI locally on each node? Also is there anything displayed in the console of the browser developer tools?

Also are you sure that the bash binary is available in that container?

I used the CLI direclty on the nodes for the time being but I will check with the "-H" options if it can helps.
Concerning bash, yes I am sure it is in the image (the docker exec works with the bash command)
For the developer tools, I will send you the answer as soon as I can
Thanks for the help !

Had the same issue with 1.16.3. Push from yesterday fixed it of me.

@nzwei an issue was shipped with the images/binary of the release, fixed a few hours later.

Just tried the latest version this morning (1.16.3) : still having the same issue.
However, some new info after some additional tests :

With Portainer running on my local host and managing remote Swarm nodes endpoints :

• Container console on a container on remote manager endpoint : Failure no such exec
• Container console on a container on any remote worker endpoint : Failure rpc error: code = 2 desc = containerd: process not found for container

With Portainer running on remote hosts managing local node endpoint(either a manager or a worker) :

• Container console on a local container : No error message but no console displayed

@yogeek could you open the developer tools in your browser and check for any errors in the console?

I'm especially curious about the last case.

Hi, I had the same issue with Apache proxy_wstunnel.

ProxyPass /api/websocket ws://:9000/api/websocket/
RequestHeader set Connection "upgrade"
Header set Connection "Upgrade"
RequestHeader setifempty Connection "Upgrade"
Header set Upgrade "websocket"
RequestHeader setifempty Upgrade "websocket"

WebSocket connection to 'wss://mypage/api/websocket/exec?id=561747b024dcc2627c62881f83108610ef2f132cae7c88355db8c7b94c46de43&endpointId=1' failed: Error during WebSocket handshake: Unexpected response code: 400
WrappedWebSocket @ VM2450:164
j @ app.0c4b0cd0.js:44
(anonymous) @ app.0c4b0cd0.js:44
(anonymous) @ app.0c4b0cd0.js:3
$eval @ app.0c4b0cd0.js:3
$digest @ app.0c4b0cd0.js:3
$apply @ app.0c4b0cd0.js:3
i @ app.0c4b0cd0.js:3
u @ app.0c4b0cd0.js:3
y.onload @ app.0c4b0cd0.js:3
app.0c4b0cd0.js:3 POST https://mypage/api/endpoints/1/docker/exec/561747b024dcc2627c62881f83108610ef2f132cae7c88355db8c7b94c46de43/resize?h=30&w=182 404 (Not Found)

@kukam is it possible that your proxy configuration removes any query parameter?

Hi, it is not possible. The apache configuration looks like this:

ServerName my.domain.ltd
DocumentRoot "/var/www/html"
ProxyPass / http://172.17.0.1:9000/
ProxyPassReverse / http://172.17.0.1:9000/
RequestHeader set Connection ""

RequestHeader set Connection "upgrade"
Header set Connection "Upgrade"
RequestHeader setifempty Connection "Upgrade"
Header set Upgrade "websocket"
RequestHeader setifempty Upgrade "websocket"
ProxyPass ws://172.17.0.1:9000/api/websocket/



ServerName my.domain.ltd
DocumentRoot "/var/www/html"
SSLEngine on
SSLCertificateFile /etc/acme/my.domain.ltd/public.cer
SSLCertificateKeyFile /etc/acme/my.domain.ltd/private.key
SSLCertificateChainFile /etc/acme/my.domain.ltd/ca.cer
ProxyPass / http://172.17.0.1:9000/
ProxyPassReverse / http://172.17.0.1:9000/
RequestHeader set Connection ""

RequestHeader set Connection "upgrade"
Header set Connection "Upgrade"
RequestHeader setifempty Connection "Upgrade"
Header set Upgrade "websocket"
RequestHeader setifempty Upgrade "websocket"
ProxyPass ws://172.17.0.1:9000/api/websocket/

@deviantony Here is the console message in chrome dev tools when I hit "Connect" button on a container running on the manager node :

• when Portainer is running on my local host (and the gui displaying the error info box) :

app.0c4b0cd0.js:3 POST http://localhost/portainer/api/endpoints/2/docker/exec/4c0d9bb3ee811f24d076f89c8707da1b6540bb8eaa4a5f7d7a17f26f77bb4410/resize?h=30&w=190 404 (Not Found)

• when Portainer is running (exposed on 8015 port) on the manager node (and the gui not displaying any error) :

app.eb337e10.js:46 WebSocket connection to 'ws://XXX.XXX.XXX.XXX:8015/api/websocket/exec?id=4df5b8c0923b3067a4a70d827fb6f667cf178503196ca2444fc7292f5ea2ee49&endpointId=1' failed: Error during WebSocket handshake: net::ERR_CONNECTION_RESET

In the last message, the IP (replaced here by XXX.XXX.XXX.XXX) is the external IP of the manager (the same IP I use to connect to Portainer web page)

I have same issue, after reading this, I tested accessing directly using 127.0.0.1:9000 and it worked.
In my case I'm using nginx.
What should be the correct configuration, or how can I help debug this?

@cmsalmeida I do not understand your answer : the issue is fixed if you access portainer on 127.0.0.1 to manage a remote docker daemon ? If so, this does not fix the issue I have : whatever IP I use to access Portainer (localhost, 127.0.0.1...), I still cannot launch a console on a container of a remote environment

app.0c4b0cd0.js:3 POST http://127.0.0.1/portainer/api/endpoints/2/docker/exec/25d893ac3e3db6284200771b02cf5e4b611dd25ccb35ab4d3cfcad50b3d3b620/resize?h=30&w=190 404 (Not Found)

@yogeek Exactly. Not fixed if not accessing on 127.0.0.1.

@cmsalmeida ok so it does not seem to be the same issue as mine... same result, but different cause. But maybe it will help portainer guys to investigate ? ^^

@cmsalmeida When using Nginx in front of Portainer, a specific configuration must be added, see http://portainer.readthedocs.io/en/stable/faq.html#how-can-i-configure-my-reverse-proxy-to-serve-portainer

We're investigating this issue.

Thank you for the link, updating the Nginx configuration accordingly worked.
:+1:

@deviantony I have been experiencing a very similar problem: I am running portainer behind a caddy-proxy and got the same error. I fixed it by changing my config to:

portainer.example.com {
  log stdout
  errors stdout
  proxy / example.com:9000 {
    websocket
    transparent
  }
}

Maybe you would like to include a caddy example as well in your documentation...

I had the same issue here with caddy + portainer...

Thanks @NonsenseInc, the websocket keyword on the proxy directive did all the work, now it is working properly.

I didn't experience this issue with the traefik in front of portainer and it didn't require any additional configuration, maybe, that is why I didn't notice the websocket connection.

@deviantony Just had the same error on "play-with-docker". Steps to reproduce :

• go to http://play-with-docker.com
• click on the "wrench" icon to create a 5 nodes swarm cluster
• on the manager1, deploy portainer :

docker run -d -p 9000:9000 --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /opt/portainer:/data portainer/portainer

• on the manager1, run a container :

docker run -p 8080:80 nginx

• got to portainer by clicking the "9000" link in PWD
• Go to "Containers" and try to connect into the previously started container
  The following error is displayed :

The chrome debugger shows the websocket errors.

@yogeek this is funny, I just tried to reproduce the issue using the exact same steps and was able to exec into the container without any issue.

@deviantony indeed, I just tested at home and no error. Is it possible that websocket are blocked when I am at work ? In any case, it does not seem to be the same problem as the one I raised originally in this issue.

Is it possible that websocket are blocked when I am at work ?

Well, you tell me :-)

Version 1.17.0 introduced a new version of the websocket implementation, might solve your problems.

Hello, I still have this issue when trying to access trough Nginx with version 1.17.0. If I access directly (bypassing nginx), then it works.
Chrome console logs:
WebSocket connection to 'wss://website.net:9999/portainer/api/websocket/exec?id=8d1318e442defe1d54c3d7b5560620084b3d9e505bed3925110731d3efdbae39&endpointId=1' failed: Error during WebSocket handshake: Unexpected response code: 403
POST https://website.net:9999/portainer/api/endpoints/1/docker/exec/8d1318e442defe1d54c3d7b5560620084b3d9e505bed3925110731d3efdbae39/resize?h=30&w=79 404 (Not Found)

Nginx config for Portainer part:

server {
   access_log /dev/stdout;
   listen 9999 ssl;
   server_name website.net;
   ssl                  on;
   ssl_certificate      /etc/nginx/ssl/website.net.crt;
   ssl_certificate_key  /etc/nginx/ssl/website.net.key;
   ssl_session_timeout 1d;
   ssl_session_cache shared:SSL:50m;
   ssl_session_tickets off;
   proxy_ssl_session_reuse off;

/.../

   location /portainer/ {
       proxy_pass  http://portainer:9000/;
       proxy_set_header Host $host;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_http_version 1.1;
   }
   location /portainer/api/websocket/ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_http_version 1.1;
       proxy_pass http://portainer:9000/api/websocket/;
   }

@naegleria is it working with Portainer 1.16.5?

Fix for me was to add the following in both locations:
proxy_set_header Origin '';

@deviantony It is working with 1.16.5 indeed, but it's also working now with 1.17.1. Sadly I don't have any environments that have an in-between version. I did not need to add what was suggested by christian-ek for it to work, neither was the location /portainer/api/websocket/ part necessary. I assume something was changed in 1.17.1 or this issue happens randomly.

@naegleria we added some changes regarding the Origin header in 1.17.1.

Hi All...

I am also experiencing this issue in 18.03.1-ce. In my case i am not using any proxies and portainer is installed in the docker host itself. Restarting the portainer container does not seem to solve the issue but rebooting the host itself temporarily fixes it. But it comes to bite us again later.

Google Chrome is showing me the following error.

WebSocket connection to 'wss://XXXXXXX:9000/api/websocket/exec?id=27ed1cd0f73c…GUiOjEsImV4cCI6MTUzMDUyMTk5M30.InwNAPNz7PpkEzILimQvOBgzS5kWTdOOHw2RLMP_7wU' failed: Error in connection establishment: net::ERR_CERT_AUTHORITY_INVALID
app.c628b0f2.js:3

POST https://XXXXXXX:9000/api/endpoints/1/docker/exec/27ed…0f73c4e…/resize?h=30&w=55 404 ()

Let me know if any further information is needed. I am hitting this quite often so it shouldn't be hard to get another error

Hi @GoncaloBorges, can you tell us how did you deploy Portainer?

• Deploying container on a RHEL7 box with selinux enabled.
• Running container as user portainer which is part of the docker group. The command to deploy it is the following...

docker run -d \
--label owner=PortainerUsyd \
--privileged \
-p 9000:9000 --name portainer \
--restart always \
-v $CERTS:/certs -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer \
-H unix:///var/run/docker.sock \
--logo "" \
-l owner=PortainerUsyd \
--ssl --sslcert /certs/selfsigned_portainer.crt --sslkey /certs/selfsigned_portainer.key

• Things work fine on deployment but after some time, we just loose this functionality.
• Let me know what else you need.

@GoncaloBorges this SSL issue looks strange. Can you try with a HTTP Portainer instance and tell me if you can reproduce ? (e.g. deploy a Portainer instance without the --ssl* flags)

@deviantony will do so and let you know if the problem persists. Just as a remark, in the current install i am using self signed certificates (which i was not expecting to be a problem).

@deviantony The issue is not occurring on the HTTP instance since i deployed it yesterday. Moreover, the HTTPS instances kind of recovered and start working again. Today the error is not there... Strange...

@GoncaloBorges this might be an issue related to the SSL flags of Portainer, could you open an new issue for this?

@deviantony Done (see issue #2022)

For me, the fix was to add proxy_set_header directives to my nginx server block.

server {
    listen      80;
    server_name portainer.example.com;

    if ($http_x_forwarded_proto = 'http') {
         return 301 https://$server_name$request_uri;
    }

    location / {
        proxy_pass http://localhost:9000;
        include /etc/nginx/proxy_params;
        proxy_set_header Upgrade websocket;
        proxy_set_header Connection Upgrade;
    }
}

This issue has been marked as stale as it has not had recent activity, it will be closed if no further activity occurs in the next 7 days. If you believe that it has been incorrectly labelled as stale, leave a comment and the label will be removed.

Since no further activity has appeared on this issue it will be closed. If you believe that it has been incorrectly closed, leave a comment and mention @itsconquest. One of our staff will then review the issue.
Note - If it is an old bug report, make sure that it is reproduceable in the latest version of Portainer as it may have already been fixed.

I have the same issue:

main.bd742767a7e5e3fe6f78.js:1 WebSocket connection to 'wss://portainer. ... .com/api/websocket/exec?token={token}&endpointId=1&id={id}&nodeName={nodeName}' failed: Error during WebSocket handshake: Unexpected response code: 400