Moby: Peer-to-peer push/pull between docker hosts

Created on 29 Mar 2013 · 22Comments · Source: moby/moby

Following discussion with @shykes on irc.

When the docker daemon is running, it should accept push/pull requests. This will allow two daemons to send images between each other in a p2p or centralized fashion.

In addition, there should be a -norun option to disable execution. This will be useful when running a dedicated image repo. In this mode, docker should work even without lxc or the aufs module.

Once this exists, the current public repo implementation could be replaced with a docker daemon.

aredistribution exexpert kinfeature

Source

cespare

All 22 comments

Correct on all counts :)

What we're shooting for is a system comparable to the go language. eg. you can host a package anywhere, and as a convenience there's a central curated namespace with guidelines for quality, auditing, safety etc.

shykes on 29 Mar 2013

This is a subcomponent of #21.

titanous on 8 Apr 2013

I changed the title to clarify the difference with #350.

This issue is about adding the peer-to-peer push/pull feature to Docker. With this feature, 2 docker hosts can exchange images directly as if they were exchanging images with the registry.

shykes on 8 Apr 2013

Thinking a little more of specifics here.

Do we want to support pushing to _and_ pulling from another daemon?
Should the docker always accept image pushes (and pulls), or does it need to be run in promiscuous mode?
What about daemon - daemon authentication and authorization?
What's the command for pushing to another daemon? docker push -d other.docker.com myimage?
Is the transfer going to use the exact same mechanism as the regular registry push/pull? (HTTP, etc.)
#21 is tied into this issue. For example, it would include a route for POSTing an image to another daemon. Or maybe that's superseded by whatever API the registry uses.

cespare on 9 Apr 2013

On Mon, Apr 8, 2013 at 8:21 PM, Caleb Spare [email protected]:

Do we want to support pushing to _and_ pulling from another daemon?

Yes, I think so. If I had to choose one, I would choose push to start.

Should the docker always accept image pushes (and pulls), or does it
need to be run in promiscuous mode?

I think it's fine to start that way. We can make it a conditional switch,
eg. 'docker -d --no-push-pull'

What about daemon - daemon authentication and authorization?

I think we can worry about that later.

What's the command for pushing to another daemon? docker push -d
other.docker.com myimage?

This seems reasonable. @samalba @kencochrane and @shin- who are
implementing the registry might have an opinion here.

Is the transfer going to use the exact same mechanism as the regular
registry push/pull? (HTTP, etc.)

Yes, that is the goal.

#21 https://github.com/dotcloud/docker/issues/21 is tied into this
issue. For example, it would include a route for POSTing an image to
another daemon. Or maybe that's superseded by whatever API the registry
uses.

Absolutely. I started working on #21, how about I share the base with you
and we work on parallel on the 2 different parts of the API?

shykes on 10 Apr 2013

@shykes That sounds great. Just push the #21 code to a branch?

cespare on 10 Apr 2013

This will be significantly easier using the 1.0 api.

This would be a great plugin, if somebody is interested in working on this, please say so here. I will hook you up with early API docs and tips for getting started.

shykes on 24 Jul 2013

This sounds interesting. Is there any hard deadline for this feature? If not, I can take it and work on it by the end of this month (or may be September. I have some other work to do these recent weeks.)

@shykes It would be great if I could access more detailed plan of 1.0 APIs.

monnand on 9 Aug 2013

@tobstarr, if you feel like using that go registry implementation of yours to allow docker to receive a push... I think that would be a killer feature! I will be happy to help you get it merged.

shykes on 24 Aug 2013

I am still interested in this. If somebody wants to give it a try, let me know :)

shykes on 11 Nov 2013

+1 for me

GaretJax on 26 Dec 2013

I would really like to get this built. Is the basic idea to overlay the registry API on top of the remote API? It looks like the /images/:id/json endpoint is more-or-less compatible, and everything else doesn't collide.

It would be really neat if the registry API was a subset of the remote API. Alternatively, I guess we could have a separate port / URL namespace that was the registry. Or even a different API entirely?

bfirsh on 28 Jan 2014

+1 - I would love to see it work the same way the registry works so you didn't have to care if you were talking to a registry or to a remote docker daemon.

I'm not sure how much work this would be, but if they're the same then the push / pull functionality of the registry could essentially become a lightweight authentication that sits in front of a docker daemon.

pguelpa on 21 Nov 2014

A working implementation over SSH here: https://github.com/docker/docker/pull/9304

bfirsh on 28 Nov 2014

👍1

Can anyone please tell me if we are going to get this feature in near future in any release?
I would love to see this feature

sahus5 on 2 May 2015

👍1

+1 Would love to see that too!

fgutmann on 20 May 2015

+1.

hanzheng on 26 May 2015

+1 on this. Currently we use docker save | ssh -C docker load to transfer images, but that transfers _everything_, including the pieces we already have. Would be much easier if I could only transfer the bits that mattered.