Machine: Create operation with vmwarefusion driver does not complete

/cc @frapposelli

this is very weird, apparently your machine was created correctly (it executed runScriptInGuest correctly which requires the machine to be up in fusion with vmware tools correctly running) but somehow failed to enter the provisioning process.

@ehazlett are those binaries created from master ?

Let me know if you need me to run some sort of debugging aid/script. Happy
to help.

On 13 August 2015 at 07:26, Fabio Rapposelli [email protected]
wrote:

this is _very_ weird, apparently your machine was created correctly (it
executed runScriptInGuest correctly which requires the machine to be up
in fusion with vmware tools correctly running) but somehow failed to enter
the provisioning process.

@ehazlett https://github.com/ehazlett are those binaries created from
master ?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/machine/issues/1671#issuecomment-130605035.

Robin Bate Boerop

I'm seeing the same behaviour. runScriptInGuest and co. work fine, but vmrun list doesn't list the docker-machine VM.

Trying to run the VM manually via /Applications/VMware\ Fusion.app/Contents/Library/vmrun start ~/.docker/machine/machines/dev/dev.vmx results in:

Error: Unknown error

Not so helpful, I know.

I can confirm this issue on Docker 1.8.1, Machine 0.4.1, and Mac OS X 10.10.4.

@mikew can you please post the vmware.log file that's in ~/.docker/machine/machines/dev ? that would help troubleshooting the issue.

Trying ... cannot even get to where it was failing previously
Here's the log for that run

Ignore previous comment, I was forgetting the -D flag. Here's another attempt with logs:

https://gist.github.com/mikew/9a20b864156f610923de#docker-output

https://gist.github.com/mikew/9a20b864156f610923de#vmware-fusion-logs

Same problem here. My details incase it helps...

System details:

❯ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10.5
BuildVersion:   14F27

❯ docker -v
Docker version 1.8.1, build d12ea79

❯ docker-machine -v
docker-machine version 0.4.1 (e2c88d6)

❯ /Applications/VMware\ Fusion.app/Contents/Library/vmrun
vmrun version 1.14.2 build-2779224

Log files:

docker-machine -D create --driver vmwarefusion fusion

~/.docker/machine/machines/fusion/vmware.log

Ok, this is weird, I tried hard but cannot reproduce this at all, this is my system configuration:

~ ⟩ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10.5
BuildVersion:   14F27
~ ⟩ docker -v
Docker version 1.8.1, build d12ea79
~ ⟩ docker-machine -v
docker-machine version 0.5.0-dev (49cbc6b)
~ ⟩ "/Applications/VMware Fusion.app/Contents/Library/vmware-vmx" -v

VMware Fusion Information:
VMware Fusion 8.0.0 build-2985594 Release

and docker-machine just works:

~ ⟩ docker-machine create -d vmwarefusion test-GH1671
Creating SSH key...
Creating VM...
Starting test-GH1671...
Waiting for VM to come online...
To see how to connect Docker to this machine, run: docker-machine env test-GH1671
~ ⟩ eval (docker-machine env test-GH1671)
~ ⟩ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 19:47:52 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64
~ ⟩ docker run busybox date
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
cf2616975b4a: Pull complete
6ce2e90b0bc7: Pull complete
8c2e06607696: Already exists
library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d
Status: Downloaded newer image for busybox:latest
Thu Aug 27 15:53:17 UTC 2015

I strongly suggest you guys to open a tech support issue on the [msg.vnet.padrConflict] problem you're encountering, that is most likely the culprit of your problem with Fusion.

[msg.vnet.padrConflict] MAC address 00:0C:29:3E:BF:B2 of adapter Ethernet0 is within the reserved address range or is in use by another virtual adapter on your system. Adapter Ethernet0 may not have network connectivity.

Cool, I'll look into it. The VM created by docker-machine fails to start, and attempting to start it makes Fusion start acting really weird. It complains that another virtual machine is already running when trying to start another.

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10.5
BuildVersion:   14F27

$ docker -v
Docker version 1.8.1, build d12ea79

$ docker-machine -v
docker-machine version 0.4.1 (e2c88d6)

$ "/Applications/VMware Fusion.app/Contents/Library/vmware-vmx" -v

VMware Fusion Information:
VMware Fusion 8.0.0 build-2985594 Release

No luck:

$ docker-machine create -d vmwarefusion test-GH1671
Creating SSH key...
Creating VM...
Starting test-GH1671...
Waiting for VM to come online...
Error creating machine: Maximum number of retries (60) exceeded
You will want to check the provider to make sure the machine and associated resources were properly removed.

Thanks @frapposelli, I tried upgrading to Fusion 8 yesterday and received the MAC address error with that as well. Will look into opening a ticket with VMware.

@frapposelli I have upgraded the software on my machine, and it now matches with the versions that you posted above. The problem persists. In my vmware.log file, I see the same line that you mention ("msg.vnet.padrConflict"). I would open an issue with VMware, but if I were to do that, I would be unable to explain where the chosen MAC address came from, how it was chosen, and why I expected it to work. I would simply have to refer VMware to the docker-machine code. Can you shed some light on how docker-machine chooses the MAC address?

@robinbb @johnallen3d the docker-machine driver uses a vmx template that is embedded in the driver (see here), this template contains the ethernet0.addressType = "generated" directive that forces the hypervisor to autogenerate a new MAC on first boot.

For anyone opening tickets to VMware support, feel free to mention my name directly (so they can loop me in internally) and please post your SR number once you get one so I can track them.

Thanks again @frapposelli, I submitted a ticket and received SR #15745066808. Hoping we can get this resolved. I've been itching to run docker on Fusion for a while now!

I did some debugging and I can confirm that the "vmrun start" that is issued in Create() (https://github.com/docker/machine/blob/93366f22be4200bffb8b547a8a1f1052f3fb63e5/drivers/vmwarefusion/fusion_darwin.go#L207) does not succeed. The vmware.log (~/.docker/machine/machines/spinzo-vm/vmware.log) contains the following interesting lines:

2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: Msg_Post: Warning
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: [msg.vnet.padrConflict] MAC address 00:0C:29:87:83:87 of adapter 'Ethernet0' is within the reserved address range or is in use by another virtual adapter on your system. Adapter 'Ethernet0' may not have network connectivity.
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: ----------------------------------------
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: MsgIsAnswered: Using builtin default 'OK' as the answer for 'msg.vnet.padrConflict'
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VMXNET3 user: Ethernet0 Driver Info: version = 16974848 gosBits = 2 gosType = 1, gosVer = 0, gosMisc = 0
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:49.120-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:50.369-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)
2015-08-28T18:57:50.369-04:00| vcpu-0| I125: VNET: MACVNetPort_SetPADR: Ethernet0: can't set PADR (Resource busy)

I can confirm that, for sure, there are no other VMs with exactly that MAC address on my system.

Might this be related to #1434? I had this exact issue with Fusion installed viahomebrew-cask, and once I installed manually with the desktop installer, things appeared to work properly. (If so, I hope it can be resolved, as I would strongly prefer not to have to install Fusion outside of brewcask.)

I'm on VMWare Fusion 8.0, with MacOSX 10.10.5.

@mroth Could be, but I did indeed install Fusion the traditional manual way.

I didn't rely on brew for fusion or docker-machine. I installed machine using docker toolbox.

I used the VMware-supplied installation method, also.

I reinstalled vmware and have 0 machines installed. I then did a machine-create and had the same error on run.

I did, however, succeed in opening the vmx file with fusion and starting it, it gave me the option to upgrade it and I did. It starts and shows as running in docker-machine ls.

$ docker-machine ls
NAME    ACTIVE   DRIVER         STATE     URL                         SWARM
test2            vmwarefusion   Running   tcp://192.168.40.251:2376

However, if I try to ssh it gives me

$ docker-machine ssh test2
exit status 255

$ docker-machine ip test2
192.168.40.251

$ docker-machine url test2
tcp://192.168.40.251:2376

$ docker-machine env test2
open /Users/***/.docker/machine/machines/test2/ca.pem: no such file or directory

$ docker-machine regenerate-certs test2
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Error getting SSH command: exit status 255

I do have a padrConflict error in the vmware.log. However, if I regenerate the mac address in fusion I get an error running docker-machine ls:

$ docker-machine ls
error getting URL for host test2: couldn't find MAC address in VMX file /Users/**/.docker/machine/machines/test2/test2.vmx
error getting URL for host test2: couldn't find MAC address in VMX file /Users/**/.docker/machine/machines/test2/test2.vmx
error determining if host is active for host test2: couldn't find MAC address in VMX file /Users/**/.docker/machine/machines/test2/test2.vmx

Ok, lets remove that broken attempt: docker-machine rm test2

Now lets try the create with sudo:

$ sudo docker-machine create -d vmwarefusion test3 -D

That succeeds :/

Next, I chown -R that machine directory with my local user.

$ sudo chown -R *** test3

where *** is my local user

I now remove the lck and vmem files/directories. Open the vmx file in Fusion and start it, again doing the upgrade when it gives me the option. I can run docker-machine ls and see it running:

 docker-machine ls
NAME    ACTIVE   DRIVER         STATE     URL                         SWARM
test3            vmwarefusion   Running   tcp://192.168.40.128:2376

Now I can set my local env variables:

$ docker-machine env test3
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.40.128:2376"
export DOCKER_CERT_PATH="/Users/***/.docker/machine/machines/test3"
export DOCKER_MACHINE_NAME="test3"
# Run this command to configure your shell:
# eval "$(docker-machine env test3)"

$ eval "$(docker-machine env test3)"

After this, docker-machine ssh works!

$ docker-machine ssh test3
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79

Hope this helps serve as a temporary workaround and shed light into the problem.

So based on that, I'm using this:

https://gist.github.com/mikew/66e15a8be8eaf7d6043c

Which gets docker-machine ls to display it as running and active, however ssh still fails with exit status 255 here.

Viewing the running VM in Fusion shows [guestinfo] Failed to get vmstats. which can't be good.

@mikew looks good, but no need for the pkill. For some reason that breaks things. Here is the updated script:

#!/bin/bash

name="${1:-test}"
dir="${HOME}/.docker/machine/machines/${name}"
vmx="${dir}/${name}.vmx"

echo "Running docker-machine create, will need sudo for vmwarefusion"
sudo docker-machine -D create -d vmwarefusion "${name}"

echo "Changing owner of ${dir} to ${USER}"
sudo chown -R "${USER}" "${dir}"

echo "Cleaning vmem/lck files"
rm -r \
  "${dir}"/*.vmem \
  "${dir}"/*.lck

echo "Opening in Fusion to upgrade"
open "${vmx}"

echo "You should be able to run 'eval \"\$(docker-machine env ${name})\"'"

FYI - this worked for me after upgrading VMware Fusion to v8. Thanks for the troubleshooting @geek and script @mikew!

Well, it would appear that the machine that is created with the above script is no longer accessible if I suspend the Fusion gui or perform a docker-machine stop/start. That's been my recent experience anyway.

same here, unfortunately

• Mike Wyatt

On Mon, Aug 31, 2015 at 11:44 PM, John Allen [email protected]
wrote:

Well, it would appear that the machine that is created with the above
script is no longer accessible if I suspend the Fusion gui or perform a docker-machine
stop/start. That's been my recent experience anyway.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/machine/issues/1671#issuecomment-136557908.

@frapposelli have you had any luck tracking down the issue? I noticed in the repro steps that you are trying with master instead of the release version of docker-machine, have you tried again with the latest release version?

I've updated the script, while the pkill made things worse, doing vmrun stop means you don't have to remove the lock files or anything.

After the VM shuts down cleanly, simply change the permissions and things work as expected. You only need sudo for the first few commands.

https://gist.github.com/mikew/66e15a8be8eaf7d6043c

Thanks for the update @mikew , I plan to give it a try soon. One question though, is the upgradevm command necessary? I notice you have it commented out in the gist.

Not really, no. Just removes the prompt if you ever want to run the VM in Fusion itself.

@mikew thanks for the updated script :)

After you run the script, are you then able to run docker-machine start and it works, or are there other manual steps you do after running the script?

I was able to do everything, repeatedly, without needing sudo after the initial setup.

The only issue that came up was when doing "docker-machine start" when the VM was suspended, ssh had errors after that. But after stopping and starting the VM via docker-machine everything worked a charm.

The script from @mikew's gist worked for me, though I had to issue "sudo docker-machine start xyz" after it finished, and 'eval "$(sudo docker-machine env xyz)"' to make things work.

@robinbb If you are comfortable with using sudo docker-machine ... throughout, you don't need the script at all.

It's strange that you still need sudo. Is you copy of Fusion installed to /Applications?

@mikew The 'sudo' is required because, without it, docker-machine has the behaviour for which I originally opened this issue - and, to be clear, the VM will not start with a "docker-machine start xyz" after the create fails.

The script is not required, but it is a convenient way to remember to change the permissions of ~/.docker/machine/machines/xyz. Otherwise, even issuing "docker" results in a permissions failure.

@frapposelli Would the fact that my umask is set to 0077 cause permissions problems which are overcome by issuing docker-machine as root?

@frapposelli any update?

@vmware ^^^?

@geek nothing yet, I'll let you know as soon as I get word from support.

Tried to reproduce again the other day but failed to.

My problem disappeared after upgrade to El Capitan GM Candidate (15A282b)

I had this problem on El Capitan GM Candidate, after upgrading from VMware Fusion 6 to 8, and then installing docker-machine.

Rebooting solved the problem for me.

I'm pretty sure it's some form of install corruption/permission problem/configuration issue with Fusion.

To the folks who are still facing this issue, can you please try and remove Fusion entirely (follow this KB http://kb.vmware.com/kb/1017838) and reinstall? that should clear any existing condition.

@frapposelli I gave the reinstall a try, no luck :(

@frapposelli I have uninstalled VMware Fusion entirely according to the cited directions, and re-installed Fusion 8. The problem remains, with the same symptoms.

The associated vmware.log is here: https://dl.dropboxusercontent.com/u/31368575/vmware.log, with no more "padrConflict" message.

I can work around the problem by issuing 'docker-machine' commands with 'sudo'.

Tried this a few weeks ago, tried again today. I can't seem to get a docker-machine with fusion to create and start, even after updating various things / reinstalling.

ProductName: Mac OS X
ProductVersion: 10.10.5
BuildVersion: 14F27
Docker version 1.8.2, build 0a8c2e3
docker-machine version 0.4.1 (HEAD)
vmrun version 1.15.0 build-3094680

I've tried the homebrew install and the old style VMWare Fusion .dmg download.

Running into this as well, @mikew's and @geek's script does not work at all for me.

Getting this error even though there is a ethernet0.address in dev.vmx.

Not there yet 1/60, error: couldn't find MAC address in VMX file /Users/msch/.docker/machine/machines/dev/dev.vmx

Still failing for me with docker 1.9 and vmware fusion 8.0.2

I am getting the same error as @MSch

-(~)$-> sw_vers && docker -v && docker-machine -v && "/Applications/VMware Fusion.app/Contents/Library/vmware-vmx" -v
ProductName: Mac OS X
ProductVersion: 10.11.1
BuildVersion: 15B42
Docker version 1.9.0, build 76d6bc9
docker-machine version 0.5.0 (HEAD)

VMware Fusion Information:
VMware Fusion 8.0.2 build-3164312 Release

To all the folks still having the problem, can you please file a service request with VMware support and post the SR number so I can track? I want to nail this down but so far I was not able to reproduce it at all.

@frapposelli have a link and anyone we need to tag in the issue?

@geek use this link as a starting point: https://www.vmware.com/support/file-sr/ and feel free to mention me directly in the SR.

@frapposelli can you help direct me to a support number to call, I can't seem to get anywhere on the site.

At this point I just want a refund. I am not a fan of Oracle, so I went with VMWare to avoid using virtualbox. VMWare advertised that it works with docker. The docker site even makes this claim and links to your driver @frapposelli. Definitely is not working as advertised and at this points its been broken for months.

I appreciate you trying to help, @frapposelli.

@geek numbers are listed here https://www.vmware.com/support/file-sr/file-sr-phone.html:

U.S. and Canada: 1-877-4-VMWARE (1-877-486-9273) or 1-650-475-5345.

for international numbers: https://www.vmware.com/support/contacts/us_support.html

I created an SR as requested: 15802564411.

Okay, now that I've opened that I figured it out (at least in my case). I am a boxen user, and I had originally installed vmware fusion 7 using boxen (which means using homebrew/cask). When that was setup, it symlinked the files from the fusion7 app into my homebrew bin directory. When I upgraded to 8, that wasn't deleted, so docker-machine was still using that to call vmrun. And, I found that cask wasn't able to remove it through the normal means either. Somehow it thought it was no longer installed, but yet it was still there.

To fix this I followed these steps:

• sudo rm -rf /opt/homebrew-cask/Caskroom/vmware-fusion
• sudo rm -f /opt/boxen/homebrew/bin/vm* (Check this one, it matched all the ones I wanted to delete)
• sudo rm -rf /Users/myusername/.docker /var/root/.docker (just to be sure)

I then trashed the Fusion app from the Application folder, and installed fresh from the dmg I had downloaded to do the upgrade. After this, when I ran docker-machine everything ran clean from start to finish with no issues.

Here's what I had to do to get this working:

• Completely remove VMWare Fusion
• Completely remove docker and docker-machine
• Install VMWare Fusion using the .dmg from VMWare. _Don't_ use brew cask for this!
• brew cask install docker docker-machine

There's some issue with the permissions on vmrun. If I install Fusion using brew-cask, vmrun doesn't have the right permissions, and making it setuid root still doesn't work. Installing from the .dmg is successful though.

It also seems that the driver is searching for vmrun when it gets installed and never updates the path even if you remove Fusion via brew cask and install it via the .dmg.

I had been having problems like this as well, and as @robinbb suggested, in my case it was being caused by a 077 umask. Changing my umask to 022 resolved it. I've created a PR to have the driver do this itself, and it's been working well for me for a few weeks now.

Curiously, I noticed this morning that the stock 0.5.1 version of docker-machine seems to be working fine on El Capitan (which I upgraded to yesterday), so this may be less of an issue in El Capitan. I'm not completely convinced yet though.

I just hit the same problem with docker-toolbox 1.12-rc3, vmware fusion 8.1.
The above scripts didn't work but rebooting my laptop did.

With VMWare Fusion version 8.1.1 (3771013), macOS Sierra 10.12, and Docker version 8.1.1 (3771013), this problem no longer occurs for me. Closing.