I need something like this.
https://github.com/rancher/rancher-ecr-credentials

Portainer only support auth by login/password on registry
The rancher ecr credential project is created to enable this feature on rancher, by periodically update credentials
If you want to use ecr images in portainer, we need to have similar behavior

Yes, I believe that we will need to integrate a feature similar to the rancher-ecr-credentials project to automatically update credentials.

Access to an ECR registry is controlled by AWS IAM, an IAM user must request temporary credentials to the registry using the AWS API, see http://rancher.com/using-amazon-container-registry-service/ for more details.

I would like to use portainer to manage my swarm cluster running on AWS. However, I need portainer to be able to access my Docker images stored in ECR. Without such a feature, I cannot use portainer. :(

I have a similar use case where I launch a swarm in AWS and use ECR for the images (and CodeCommit as repository, which is relevant to a different issue). I have applied an IAM role to my instances which grants Pull permissions to the repositories (i.e., I do not have to run aws ecr get-login or docker login). My workflow for launching stacks was too manual: ssh to the manager node, execute a shell script. Yet, registry auth worked.

I have previously used https://github.com/dockersamples/docker-swarm-visualizer as a "GUI" but want to use portainer as a management layer for my stacks. Since I run portainer as a service on my swarm, it necessarily runs as a container itself, which puts the burden of the registry authentication inside the container, and what works on the host doesn't work in the container.

My workaround is this: wrap the portainer docker image in my own Dockerfile and use my own image instead:

Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:latest
ENV HOME=/
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]

This gives me a portainer image that's a little bigger but contains the AWS ecr-login helper binary (and a busybox for good measure so I can get a shell and poke around). Given the HOME environment and the config.json (straight from the ecr helper README, see below), I am able to launch stacks from portainer with images from the ECR.

config.json

{
    "credsStore": "ecr-login"
}

Your mileage may vary, and I do +1 a feature that would bring support for revolving ecr credentials into portainer, but with IAM roles my workaround might be useful to others.

@tcjennings @deviantony unfortunately after the update to 1.17.0 aws login doesn't work anymore with the ecr login helper. If I'll have time I'll check what is changed why the binary is not called when docker pull initiated. I rebuilt the same with 1.16.5 which works perfectly but with the latest version ecr cache never get filled and as I see the helper binary is not called when a stack deployment ran.

@jgrasl starting with 1.17 Portainer is using a specifc config.json file located in /config.json.

If you're using the Dockerfile example above, then changing this line might solve your issue:

COPY config.json /config.json

Note that the location of the config.json might be updated after solving https://github.com/portainer/portainer/issues/1888

@deviantony As I see from the changes in the release the config json file is always overwritten by main() - > initStackManager -> NewStackManager - > updateDockerCLIConfiguration. It works fine if I overwrite the HTTPheaders in the json file after portainer process is started but I cannot overwrite it at build time.
I'll take a look if I can start portainer with a script inside the container instead of with the /portainer entry point so then I can modify the content of the json file after the process was initiated. But I fear the HTTP headers information are stored for a reason so I should not really remove it from that file. I'm not using the agent currently but it can be a problem for someone else.

Right, we'll try to preserve the content of the config.json file when solving #1888

@jgrasl keen to give a try with portainer/portainer:pr1898 ?

@deviantony With the pr1898 the content of the /config.json is preserved and a new config.json is created under /data. Altough that new config.json file only contains the HTTP header json nothing from the file copied build time and so the ecr login is not working with the cred helper.

@jgrasl well with pr1898 you'd need to update your Dockerfile:

COPY config.json /data/config.json

@deviantony sorry I forgot that. Now I did change it unfortunately now the data from my config json is not preserved. Under /data/config.json is only the HTTP headers. I am mapping the data volume to an external volume trough docker volumes to persist portainer config, can that be a problem?

@jgrasl you're using FROM portainer/portainer:pr1898 in your Dockerfile right? Any existing config.json in that volume?

@deviantony yes using the pr docker image

FROM golang:1.9 as ecr-helper                                                                                            
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login                  
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper                                                          
RUN make                                                                                                                 

FROM busybox as busybox                                                                                                  
RUN which busybox                                                                                                        

FROM portainer/portainer:pr1898                                                                                          
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /data/config.json                                                                                       
COPY --from=busybox /bin/busybox /bin/busybox                                                                            
RUN ["/bin/busybox","--install","-s","/bin"]

Before starting up the container I deleted the config.json file from the mounted data volume so should be no json file there at startup.

Actually, mounting a volume to persist Portainer data will map a folder on disk to /data, making that COPY config.json /data/config.json statement useless.

I'm trying to think of a way to persist user changes to config.json before deployment.

@jgrasl How are you persisting Portainer data? Using a named volume? If you're using a named volume AND the volume is empty (e.g. just created) it will have the config.json copied from the Dockerfile. If you're reusing an existing data volume, I believe the config.json file will not be copied.

Might be problematic for people upgrading their Portainer instance, still searching for a better way to keep config.json changes.

@deviantony no, just using a mapped directory with docker run, but it can be named of course if it's needed.
-v /portainer/data/path/on/server:/data

@deviantony for now I just put the config.json file under the mapped directory with the credsStore setup and startup portainer. This way my config got merged with the HTTP headers and ecr login works fine with the pr1898.

@jgrasl this is what I'm thinking as well. If anybody using this workaround wants to continue using it and have existing Portainer data, they'll have to put their config.json in the root of the existing data directory. If people are starting to use this workaround and do not have any existing data, then the config.json will be created in their data directory.

Nonetheless we'll need to find a real solution for this later on.

@deviantony I don't know if you plan to do any other support for ecr login but I think the credential helper could be a final solution. It only needs the credentials file which is already used by everyone who uses the aws cli so if the credsStore in the config json could be included by default in the portainer image we wouldn't need to put he file manually under the persisted /data directory only provide the credentials file.
But I'm completely happy with the current solution too since it works fine for me.
Thanks for the fix!

We have not thought about it yet, quite busy with other topics atm :-)

@deviantony Of course I understand, busy release time:) I can prepare a PR with what I thought and you guys can decide later if you'd like to merge it for a later release or not.

I've tried all workarounds with the new 1.17.0, may be because I'm using agents, it's not working! I tried adding config.json on /, on data on .docker...
Any idea on how have it working?

@reineruhry see the posts above.

Basically, use portainer/portainer:develop and copy your config.json in the data volume (you need to find the folder on disk first) of Portainer.

@deviantony thanks! that worked, I was trying with the pr1898, do you know if it's now part of the 1.17.1?

@reineruhry yup, it's now part of 1.17.1

Have you any idea to test if it working without waiting 12 hours for ECR token expired?

@dawidpawlowski You can log in to the container and remove /.ecr/cache.json file which stores login credentials. After that you run a new deployment it should recreate the file with valid login credentials to ecr.

@jgrasl I tried your solution and it seems to not working for me. Below I described my example stack setup and all steps what I done.

1. I builded portainer workaround docker image as you wrote in posts above:

FROM golang:1.9 as ecr-helper                                                                                            
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login                  
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper                                                          
RUN make                                                                                                                 

FROM busybox as busybox                                                                                                  
RUN which busybox                                                                                                        

FROM portainer/portainer:1.17.1                                                                                         
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /data/config.json                                                                                       
COPY --from=busybox /bin/busybox /bin/busybox                                                                            
RUN ["/bin/busybox","--install","-s","/bin"]

1. Created empty directory /path/on/server/portainer_data. Logged manually to ECR using aws-cli (to be able to download my test_service image from ECR) and deploy stack using docker-compose.yml file (with 2 nodes in swarm):

version: '3.2'
services:
  test_service:
    image: 111111111.dkr.ecr.eu-west-1.amazonaws.com/test_service:1.0.10
    ports:
      - target: 80
        mode: host
    deploy:
      replicas: 1
      mode: replicated
      endpoint_mode: dnsrr

  agent:
    image: portainer/agent
    environment:
      AGENT_CLUSTER_ADDR: tasks.agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - agent_network
    deploy:
      mode: global

  portainer:
    image: workaround-portainer:test5
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    environment:
      AWS_ACCESS_KEY_ID: MY_ACCESS_KEY
      AWS_SECRET_ACCESS_KEY: MY_SECRET_ACCESS_KEY
    ports:
      - "9000:9000"
    volumes:
      - /path/on/server/portainer_data:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay

volumes:
  portainer_data:

1. All runs properly. /path/on/server/portainer_data/config.json looks:

{"HttpHeaders":{"X-PortainerAgent-ManagerOperation":"1","X-PortainerAgent-PublicKey":"somePublicKey","X-PortainerAgent-Signature":"someSignature"}}

There isn't config credsStore property so I replace evertything in /path/on/server/portainer_data/config.json with this:

{
    "credsStore": "ecr-login"
}

and restart portainer container. After restart /path/on/server/portainer_data/config.json looks:

{"HttpHeaders":{"X-PortainerAgent-ManagerOperation":"1","X-PortainerAgent-PublicKey":"somePublicKey","X-PortainerAgent-Signature":"someSignature"},"credsStore":"ecr-login"}

so, everythings looks fine.

1. Next, I logged to portainer http://my-swam-manager-address:9000

and update test_service version to some previous version 111111111.dkr.ecr.eu-west-1.amazonaws.com/test_service:1.0.9. It works, but when try to pull some image from my ECR using http://my-swam-manager-address:9000/#/images it doesn't work. It works when I downloading some images from DockerHub.

I logged to the portainer container and there isn't /.ecr/cache.json file. It creates only when I run manually command docker-credential-ecr-login list. So I'm sure that after 12 hours I won't be able to update service image version.

What I'm doing wrong? I can add that It works exactly the same without workaround - I can update services only for 12 hours after deploy stack.

@dawidpawlowski I tried now to pull an image via portainer images it doesn't work me either but to be honest I never used that feature since stack will automatically pull the image it needs. I think pulling an individual image works differently. Is the stack cache.json file gets created if you try to deploy a stack file trough http://my-swam-manager-address:9000/#/stacks ? I think your configuration is fine.

@jgrasl @dawidpawlowski pulling an image through Portainer implies to pass credentials to the pull operation Docker API request: https://docs.docker.com/engine/api/v1.37/#section/Authentication

Portainer does this automatically by looking at the image you're trying to pull, it will automatically find the associated registry (provided that it is created inside the app) and pass the credentials.

We're aware that this is not working with ECR registries.

So, @jgrasl when I deploy stack using http://my-swam-manager-address:9000/#/stacks then it works, /.ecr/cache.json file is recreating. Thanks for help.

Unfortunately this workaround works only for deploying new stack. When I tried to update service version or scale service to the node where image is not pulled yet, operation fails with message 'No such image...'. .ecr/cache.json file is recreated only if I deploy new stack. With scaling and updating image version cache.json is not recreated.

A workaround I found if your servers are not in aws but you are using ECR, is updating the registry every 12 hours through the api using a cron job

I'm using httpie to execute HTTP queries from the CLI and jq to parse the response

!/bin/sh

echo "get portainer token"
PORTAINER_TOKEN=$(http --ignore-stdin POST :9000/api/auth Username=username Password=password | jq -r '.jwt')

echo "get aws token"
AWS_TOKEN=$(/root/.local/bin/aws ecr get-authorization-token --region sa-east-1 --output text --query authorizationData[].authorizationToken)

AWS_DECODED_PASSWORD=$(echo $AWS_TOKEN | base64 -d | cut -d: -f2)

echo "update the registry"
http --ignore-stdin PUT :9000/api/registries/1 "Authorization: Bearer $PORTAINER_TOKEN" username=AWS password=$AWS_DECODED_PASSWORD Authentication:=true

__Proposed Workaround__:

Create an image with the following entrypoint (and the needed dependencies like awscli installed):

eval $(aws ecr get-login --no-include-email --region eu-west-2)
docker pull <image_needed_from_ecr_1>
docker pull <image_needed_from_ecr_2>
...

You should of course expose AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID vars to the above image (and make available via template.json)

Add:

  deploy:
     restart_policy: none

To the service of the above image;

Add:

depends_on:
   service_above

to services in docker-compose.yml using the above ecr images that need to become locally available first.

Happy ecr-pulling

@pkaramol how is that supposed to refresh the aws auth token when it expires after 12 hours? You'd have to restart Portainer every time your tokens expire.

I have put @dawidpawlowski solution into an auto build here

I use IAM roles on the machines but AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY will work the same

It only works for me if i am deploying a stack using protainer, running a service directly with a private ECR image doesn't seem to work.

I've been able to get the registry created on AWS using https://github.com/johnvmt/portainer-ecr-credentials but even if the credentials are there I get No such image trying to use it in a service and/or a stack

Update: my bad, I've typed the wrong AWS region :) now it seems to be working

I've been able to get the registry created on AWS using https://github.com/johnvmt/portainer-ecr-credentials but even if the credentials are there I get No such image trying to use it in a service and/or a stack

Update: my bad, I've typed the wrong AWS region :) now it seems to be working

I tweaked this container's index.js node file for my enterprise proxy settings and got it working. Pretty slick, it runs every 11 hours to pull down new ECR credentials and updates the portainer registry with the new credential.

https://github.com/awslabs/amazon-ecr-credential-helper

This might be of help to some people

Hey, @deviantony,
Is this being taken into consideration for one of the future versions of portainer? I know that you mentioned that this was not being progressed because of focus on other topics back in May 2018.

We basically use ECR to store all our images and want to be able to pull the images locally (and eventually build containers). The present work around is not compatible for normal docker image pulls using app templates or docker-compose up. The work around only works with docker stack deploy (docker compose in swarm) as you maybe aware.

We mainly only use docker-compose up and docker runs (docker pull in the background when the container does not exist locally) for our application development and the native support for ECR from portainer would be great as we would need to pull images ourselves externally before doing a docker-compose up or docker run (App templates on portainer).

Just trying to get a status of this to make a decision of using portainer or not..

Thanks!

+1 on comment above, thanks!

Please fix that soon!

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• Big note You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• _Big note_ You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

Have you tried this with Portainer 2.0?

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• _Big note_ You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

Have you tried this with Portainer 2.0?

I've tried it with Portainer's latest version and works just fine. I managed to adapt it and made a little repo using @xgenvn 's answer

https://github.com/filadd/ecr_portainer

We can use httpd server to expose an api to execute ./docker pull <ECR Repo/image>. (Helped me with CI/CD)
Implementation - https://github.com/mek97/portainer-ecr

Am new to portainer firstly tnx its awesome.

Another way that just worked for me is to add the registry in the usual manner with username of AWS and password as the output of aws ecr get-login-password. I know it will need updating but it pulls successfully in the first instance.

Using latest version of portainer-ce.

We're planning to add support for AWS ECR registry later this year, this is currently earmarked around Q2 2021.