packer produces 0 byte vmdk

Oh, I can repro the problem on macOS 10.13.3 and APFS, Packer 1.2.1, VirtualBox 5.2.8

$ packer build --only=virtualbox-iso windows_2016_docker.json
$ ls -l windows_2016_docker_virtualbox.box
-rw-r--r--   1 stefan  staff  10146394489 Mar  8 12:16 windows_2016_docker_virtualbox.box
$ tar tvf windows_2016_docker_virtualbox.box
-rw-r--r--  0 stefan staff    2286 Mar  8 12:16 Vagrantfile
-rw-r--r--  0 stefan staff       0 Mar  8 12:16 WindowsServer2016Docker-disk001.vmdk
-rw-r--r--  0 stefan staff   10411 Mar  8 12:16 box.ovf
-rw-r--r--  0 stefan staff      26 Mar  8 12:16 metadata.json

I have retried with an older Packer version 1.1.3, still macOS 10.13.3 and APFS, VirtualBox 5.2.8 and the box file looks fine afterwards:

$ ls -l *box
-rw-r--r--  1 stefan  staff  10817278351 Mar 11 14:28 windows_2016_docker_virtualbox.box
$ tar tvf windows_2016_docker_virtualbox.box 
-rw-r--r--  0 501    20        2286 Mar 11 14:27 Vagrantfile
-rw-r--r--  0 501    20 10952051712 Mar 11 14:27 WindowsServer2016Docker-disk001.vmdk
-rw-r--r--  0 501    20       10412 Mar 11 14:27 box.ovf
-rw-r--r--  0 501    20          26 Mar 11 14:27 metadata.json
~/code/packer-windows on my*

What might be the difference between Packer 1.1.3 and Packer 1.2.1?

Is the func DirToBox significantly different in 1.2.1?

Sorry, not very good with git, so I don't know how to pull up that code like you did with

https://github.com/hashicorp/packer/blob/4d3a762e85e7e8050134a4e33e1f87a54029dcb8/post-processor/vagrant/util.go#L71-L155

A git blame shows no changes since packer 1.1.1 in packer/post-processor/vagrant/util.go.
Packer 1.1.3 is compiled with Go 1.9, Packer 1.2.1 is compiled with Go 1.10.

It really looks like a problem with Golang 1.10.

I've extracted the packer/post-processor/vagrant/util.go file into a small main.go calling the DirToBox function without the packer.Ui.

Compiling that code with Golang 1.9 the created tar file looks good.
Compiling that code with Golang 1.10 the tar file shows a zero byte file.

The problem also occurs with compression level = 0, so it's independent from the pgzip package.

The sample source dirtobox.go, the Dockerfile and build.sh script to build the Darwin binary can be found in this gist: https://gist.github.com/StefanScherer/c55fd0ae752687c806f8d5fb40cadc78

I have created a sample in directory with a big file, compiled the dirtobox binary for Darwin and run the test

$ ./build.sh 

$ ls -l in
total 21509960
-rw-r--r--  1 stefan  staff  11012556800 Mar 12 15:11 abigfile.tar
-rw-r--r--  1 stefan  staff            6 Mar 12 15:08 asmallfile.txt

$ ./dirtobox 
2018/03/12 15:35:02 Turning dir into box: in => out.tar
2018/03/12 15:35:02 Skipping directory 'in' for box 'out.tar'
2018/03/12 15:35:02 Box add: 'in/abigfile.tar' to 'out.tar'
Compressing: %s abigfile.tar
2018/03/12 15:35:32 Box add: 'in/asmallfile.txt' to 'out.tar'
Compressing: %s asmallfile.txt

$ ls -l out.tar
-rw-r--r--  1 stefan  staff  11012560384 Mar 12 15:27 out.tar

$ tar tvf out.tar
-rw-r--r--  0 stefan staff       0 Mar 12 15:11 abigfile.tar
-rw-r--r--  0 stefan staff       6 Mar 12 15:08 asmallfile.txt

Changing the Dockerfile back to golang:1.9

FROM golang:1.9

WORKDIR /go/src/app
COPY . .

RUN go get -d -v ./...
RUN GOOS=darwin go build dirtobox.go

and running ./build.sh and ./dirtobox again produces a correct tar file:

$ ls -l out.tar 
-rw-r--r--  1 stefan  staff  11012559360 Mar 12 15:37 out.tar

$ tar tvf out.tar 
-rw-r--r--  0 501    20 11012556800 Mar 12 15:11 abigfile.tar
-rw-r--r--  0 501    20           6 Mar 12 15:08 asmallfile.txt

It seems to happen with files > 8GByte.

Time to open an issue over at golang ?

Excellent triaging @StefanScherer

I ran into a very similar issue with Vagrant today - see HERE.

Long and short the tar utility embedded with Vagrant 2.0.3 is having problems unpacking archives with large files - when using bsdtar to extract the files from the box tar archive I end up with a very slimline vmdk of zero bytes! Smaller boxes don't cause any issues.

Note that the system tar has no issues extracting the files from the same box/tar archive...

$ /opt/vagrant/embedded/bin/bsdtar --version
bsdtar 3.2.2 - libarchive 3.2.2 zlib/1.2.11 liblzma/5.2.3

$ tar --version
tar (GNU tar) 1.29

@StefanScherer - Are you sure it's not the tar version you are using that's causing the issues?

@StefanScherer Just for reference:

$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G19009

$ go version
go version go1.10 darwin/amd64

$ packer version
Packer v1.2.2-dev...

@basictheprogram @StefanScherer Sorry - just looked at the original issue HERE.

That is the same issue as has been reported HERE - not a Packer issue!

Thanks @DanHam for the heads-up.
There may be a problem with bsdtar on macOS.
The problem in Packer seems to be with Golang 1.10. It obviously creates different tar files between Golang 1.9 and 1.10.

I used another Go binary from https://github.com/mholt/archiver/releases to extract the tar files created with the dirtobox sample code above compiled with Golang 1.9 and 1.10.

The archiver can extract the file from the tar file created with the dirtobox Golang 1.9 binary.
And the archiver just wrote an empty file extracting the tar file created with the dirtobox Golang 1.10 binary.

No tar / bsdtar binary was included in that test.

A test with the embedded bsdtar of my Vagrant 2.0.2 also can extract big files

$ /opt/vagrant/embedded/bin/bsdtar tvf out-go1.9.tar 
-rw-------  0 501    20 10737418240 Mar 23 23:02 10Gigfile
$ /opt/vagrant/embedded/bin/bsdtar xvf out-go1.9.tar 
x 10Gigfile
$ ls -l 10Gigfile 
-rw-------  1 stefan  staff  10737418240 Mar 23 23:02 10Gigfile


$ /opt/vagrant/embedded/bin/bsdtar tvf out-go1.10.tar 
-rw-------  0 stefan staff       0 Mar 23 23:02 10Gigfile
$ /opt/vagrant/embedded/bin/bsdtar xvf out-go1.10.tar 
x 10Gigfile
$ ls -l 10Gigfile 
-rw-------  1 stefan  staff  0 Mar 23 23:02 10Gigfile

@StefanScherer Ah... OK. Thought we had that one explained away! Good luck tracking down the bug...

I've been having the issue recently, but am unable to make any headway with reverting to Packer 1.1.3. I don't suppose you have any other ideas?

(Issue is documented in https://github.com/Parallels/vagrant-parallels/issues/319, as I thought it was a plugin issue!)

So investigating some more;
I reverted to Packer 1.1.2 on OSX 10.12, and that managed to produce a stable Box I could use.
Testing with Packer 1.1.2 on OSX 10.13 and that fails for some reason? The harddrive says it's 0:

$ tar -tvf sdk-mac.box 
-rw-r--r--  0 user group     180 27 Mar 12:57 Vagrantfile
-rw-r--r--  0 user group      25 27 Mar 12:57 metadata.json
-rw-r--r--  0 user group   65536 27 Mar 12:56 sdk-mac.pvm/NVRAM.dat
-rw-r--r--  0 user group    7224 27 Mar 12:56 sdk-mac.pvm/VmInfo.pvi
-rw-r--r--  0 user group   26381 27 Mar 12:56 sdk-mac.pvm/config.pvs
-rw-r--r--  0 user group    1710 27 Mar 12:56 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r--  0 user group       0 27 Mar 12:56 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r--  0 user group       0 27 Mar 12:57 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r--  0 user group    8816 27 Mar 12:57 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh

So definitely looks like it's a packer issue somewhere... how do you go about debugging this, as I really need to get vagrant working for these machines!

FYI. There is a new packer and golang from homebrew today for Sierra. And new packer for High Sierra.

@basictheprogram Thanks for the update!

I've updated them both to the latest Packer available on Homebrew (1.2.2)

• Sierra works fine
• High Sierra still cannot connect the hard drive

I'm not sure what else could be causing this issue apart from the OS difference?

What's weird is that running $ tar .box shows that for both OSs think that the hard disk has a size of 0... but doing vagrant up works fine for the Sierra one. (They are both running Vagrant 2.0.3).

Sierra:

$ tar tvf sdk-mac.box 
-rw-r--r--  0 user group     180 27 Mar 15:25 Vagrantfile
-rw-r--r--  0 user group      25 27 Mar 15:25 metadata.json
-rw-r--r--  0 user group   65536 27 Mar 15:23 sdk-mac.pvm/NVRAM.dat
-rw-r--r--  0 user group    7224 27 Mar 15:23 sdk-mac.pvm/VmInfo.pvi
-rw-r--r--  0 user group   26385 27 Mar 15:23 sdk-mac.pvm/config.pvs
-rw-r--r--  0 user group    1710 27 Mar 15:23 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r--  0 user group       0 27 Mar 15:23 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r--  0 user group       0 27 Mar 15:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r--  0 user group    6784 27 Mar 15:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh

High Sierra:

$ tar tvf sdk-mac.box 
-rw-r--r--  0 user group     180 27 Mar 15:29 Vagrantfile
-rw-r--r--  0 user group      25 27 Mar 15:29 metadata.json
-rw-r--r--  0 user group   65536 27 Mar 15:29 sdk-mac.pvm/NVRAM.dat
-rw-r--r--  0 user group    7224 27 Mar 15:29 sdk-mac.pvm/VmInfo.pvi
-rw-r--r--  0 user group   26381 27 Mar 15:29 sdk-mac.pvm/config.pvs
-rw-r--r--  0 user group    1710 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r--  0 user group       0 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r--  0 user group       0 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r--  0 user group    7728 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh

On Sierra upgrading packer installed go.

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G1212

$ go version
go version go1.10 darwin/amd64

On High Sierra upgrading packer did not install go

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.3
BuildVersion: 17D102

$ which go

$ ls -l /usr/local/bin/go*
ls: /usr/local/bin/go*: No such file or directory

I manually installed go and things are working for me now.

I need to confirm all of this on other installations.

Packer 1.2.2 binary for macOS (https://releases.hashicorp.com/packer/1.2.2/packer_1.2.2_darwin_amd64.zip) is compiled with Golang 1.8.1. I assume it should work again.

Homebrew Packer 1.2.2 is compiled with Golang 1.10
Man, why do they have to compile binaries when binaries are already deployed by HashiCorp?

Awesome investigating, folks. It sounds to me like this is a bug in Homebrew rather than a bug in Packer -- would you all agree? Should I close this ticket?

I think the bug is in Golang and the packer is compiled with a buggy Golang. Hashicorp should compile their packer with Golang 1.10.

Sounds like the buggy binary is Homebrew's and the HashiCorp binary works?

Someone should test the HashiCorp 1.2.2 binary. Can test it tomorrow.

I'm gonna recompile and re-release the HashiCorp 1.2.2. binary on golang 1.10 now. We should have that on latest golang.

Hm, but then it probably will be broken as 1.2.1.

Here's the golang-1.8-compiled binary, for those folks who need a workaround:
packer_1.2.2_darwin_amd64.zip . Releasing it compiled with 1.8 was an accident, and just because it fixes this bug doesn't mean it doesn't introduce others.

@basictheprogram My systems already had go installed when I did the check:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.6
BuildVersion:   16G1212
$ go version
go version go1.10 darwin/amd64

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.3
BuildVersion:   17D102
$ go version
go version go1.10 darwin/amd64

I just tested the 1.8 compiled binary of Packer 1.2.2 @SwampDragons posted above:
Sierra: vagrant box fails (to find hard drive)
High Sierra: the vagrant box failed again (same issue)

Testing the default Packer 1.2.2 compiled with 1.10:
Sierra: vagrant box fails (to find hard drive)
High Sierra: the vagrant box failed again (same issue)

The .pvm file is 16GB it tries to compress, and the output box is 9GB...
I'm beginning to think it has something to do with my system now.

Can anyone confirm 1.2.2 is working, and if so, what setup?

(Weird note; I said earlier that I had it working on Sierra with Brew's Packer, it's now not working...)

@SwampDragons as expected:

• Packer 1.2.2 compiled with Golang 1.8 creates box files that work fine with the macOS tar (bsdtar 2.8.3 - libarchive 2.8.3).
• Packer 1.2.2 compiled with Golang 1.10 creates box files that macOS tar cannot extract (0 byte files)
• Both generated box files work fine with GNU tar (I only tried a Linux version in a container with docker run -it -v $(pwd):/test -w /test ubuntu tar tvf windows_2016_docker_virtualbox-brew-packer-1.2.2-golang-1.10.box.

@SwampDragons I've been using go 1.10 (installed via MacPorts) to compile my Packer binaries. As stated above, I can create large boxes with out issue...

@SurferL My set up:

$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G19009

$ go version
go version go1.10 darwin/amd64

$ packer version
Packer v1.2.2-dev...

Unpacking the box (using GNU tar) from my Windows 2016 build gives me:

.
└── [        340]  vmware_desktop
    ├── [        983]  Vagrantfile
    ├── [         30]  metadata.json
    ├── [       8684]  windows2016-vmware-iso.nvram
    ├── [13725794304]  windows2016-vmware-iso.vmdk
    ├── [          0]  windows2016-vmware-iso.vmsd
    ├── [       3515]  windows2016-vmware-iso.vmx
    └── [        277]  windows2016-vmware-iso.vmxf

1 directory, 7 files

This was why I was so convinced (at first) that we were seeing this issue rather than an issue with Packer. However, the work @StefanScherer has done seems to suggest the problem lies elsewhere...

EDIT: Just seen @StefanScherer 's comment above... so this could be the same issue albeit with some additional info provided by Stefan?

This might have been fixed in https://golang.org/doc/devel/release.html#go1.10.minor

I can see a fix for archive/zip in 1.10.1. I've tried with a stripped down sample code, but there is still the problem. It also occurs with a Linux Golang binary. The macOS tar (bsdtar) cannot read the file size correctly.

I've created https://github.com/golang/go/issues/24599

Here is packer built with go 1.10.1 for darwin. Would appreciate any help in testing if this solves the problem.

packer-1.2.2-go-1.10.1.zip

Thanks @StefanScherer for filing the upstream bug. There's a ton of useful information in it

What can you do?
Go1.10 gives the user finer control over the output format. You can see tar.Header.Format to tar.FormatGNU to work around the libarchive bug.

That seems like it should be it. If anyone would like to submit a PR, please do, otherwise we will add the workaround for the next point release. Compiling packer with an older version of go should be a viable workaround for now.

Will do after some tests.

Afraid I'm still getting the same error with Packer built with go 1.10.1 for Darwin which @mwhooker provided... the .box generated at the end contains an empty hard disk supposedly for box size of 12GB

$ tar -tvf sdk-mac.box 
-rw-r--r--  0 user group     180  3 Apr 11:26 Vagrantfile
-rw-r--r--  0 user group      25  3 Apr 11:26 metadata.json
-rw-r--r--  0 user group   65536  3 Apr 11:25 sdk-mac.pvm/NVRAM.dat
-rw-r--r--  0 user group    7224  3 Apr 11:25 sdk-mac.pvm/VmInfo.pvi
-rw-r--r--  0 user group   26381  3 Apr 11:25 sdk-mac.pvm/config.pvs
-rw-r--r--  0 user group    1710  3 Apr 11:25 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r--  0 user group       0  3 Apr 11:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r--  0 user group       0  3 Apr 11:26 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r--  0 user group    8336  3 Apr 11:26 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh

(and running vagrant up fails with the hard disk error)

I'm also still getting the (vagrant up hard disk) error using Packer 1.1.3, so I'm not sure if it's something with my system?
Even though the hard disk seems to be initialised here?

$ tar -tvf sdk-mac.box 
-rw-r--r--  0 501    20        180  3 Apr 12:24 Vagrantfile
-rw-r--r--  0 501    20         25  3 Apr 12:24 metadata.json
-rw-r--r--  0 501    20      65536  3 Apr 12:23 sdk-mac.pvm/NVRAM.dat
-rw-r--r--  0 501    20       7224  3 Apr 12:23 sdk-mac.pvm/VmInfo.pvi
-rw-r--r--  0 501    20      26381  3 Apr 12:23 sdk-mac.pvm/config.pvs
-rw-r--r--  0 501    20       1710  3 Apr 12:23 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r--  0 501    20          0  3 Apr 12:23 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r--  0 501    20 19503513600  3 Apr 12:24 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r--  0 501    20        7472  3 Apr 12:24 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh

Have you tried packer from brew?

@basictheprogram Yeah, I've tried the latest 1.2.2, as well as going back to 1.1.3 with Brew but they both didn't work with the same issues above

@SurferL Are you using vagrant box add...? I'm pretty sure that Vagrants embedded version of tar will cause these issues irrespective of the fix that went in to Packer. See my comment here.

Long and short you may need to manually unpack the box into the right location in the .vagrant.d... directory using GNU tar rather than bsdtar until the next version of Vagrant is released.

@DanHam Thanks for pointing that out - I was not using vagrant box add, and this made my build work with Packer 1.1.3!

Just some more info:
Using Packer 1.2.2 from brew (and using vagrant box add did not work), nor did using Packer built with go 1.10.1 for Darwin (and using vagrant box add)...

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.