Packer: ansible 2.8 breaks packer ansible provisioner

Ran into exactly the same issue - downgraded to ansible 2.7.10 and it worked perfectly

was anyone else able to repro this?

Same here. I had to downgrade.

On Tue 28 May 2019 at 20:58, AndrewCi notifications@github.com wrote:

was anyone else able to repro this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/packer/issues/7667?email_source=notifications&email_token=AAAAFFDXJF7SZ3RCR7ZC4XLPXV6GTA5CNFSM4HOEP2H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWNDJDY#issuecomment-496645263,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAAFFFYN7SKLPFDER6IOYDPXV6GTANCNFSM4HOEP2HQ
.

I am also able to reproduce this using the Azure ARM provisioner

yup, also downgraded ansible

I think we are hitting this on ansible 2.6.2

@intinig What version you downgrade to?

Rolling back to 2.7.10 fixed it for me.

Alright for me, I got around this by removing the -vv option. WILD!

2.7.10

On Thu 30 May 2019 at 19:50, adamday2 notifications@github.com wrote:

Rolling back to 2.7.10 fixed it for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/packer/issues/7667?email_source=notifications&email_token=AAAAFFDBJO4EIUY3O43JGOTPYAHWNA5CNFSM4HOEP2H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWTASNA#issuecomment-497420596,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAAFFESVO36RH5QHKQV2F3PYAHWNANCNFSM4HOEP2HQ
.

This is (in at least some cases) related to the automated Python interpreter discovery added to Ansible 2.8; hard-coding /usr/bin/python fixed the hangs that I observed.

      "extra_arguments": [
        "--extra-vars",
        "ansible_python_interpreter=/usr/bin/python"
      ],

Hit this, but really need 2.8 for some of the vSphere enhanced modules that have been released.

Just making a note here since this is a popular issue -- this provisioner is one of our community-supported provisioners, which means that the HashiCorp maintainers don't spend much engineering time on it; this means that the best way to see your suggestion make it into Packer is to open a PR.

@flowerysong thanks for the tips, it works! The virtualenv does the help.

A Windows machine with the packer connenction has the same problem, does anyone have a workaround?

Does anyone at least have a root cause of what's causing this hang and a potential plan to fix this? Happy to help where I can, but not familiar with Go (might be more use with a point in the right direction).

Also, could running packer from a different OS make a difference?

Got the same. Attempted with packer 1.3.3, 1.3.4, 1.4.1, 1.4.2, ansible versions 2.7.10, 2.8.0 and 2.8.1 all exhibiting the same behavior. Setting ANSIBLE_PYTHON_INTERPRETER env var succeeded.

I took a quick look at this today; my gut instinct based on your ansible_python_interpreter discoveries is that the warning being generated when using the fallback list is causing the hang because we're not processing stderr properly somewhere.

In the hour or so I tinkered I had trouble reproducing a situation where I could generate this fallback warning to test that theory, though. If any of you who are coming across this issue can share your guest OS arch and what version of python is installed (and where it's installed), that would be very helpful so I don't spent a bunch of time spinning my wheels trying to get a repro case.

For now it sounds like setting the ansible_python_interpreter directly works for you all. I'd be curious to know if setting it to the value ansible_python_interpreter=auto_silent is enough to solve this; that would add some credence my theory that Packer's mishandling the pipe that the warning is coming through somewhere.

@SwampDragons It's the same reason pipelining causes the provisioner to hang, which I don't know the precise root cause of. Interpreter discovery always uses pipelining, regardless of whether it's enabled for module execution.

This is potentially going to be addressed on the Ansible side by adding a timeout to the interpreter discovery, but there's not currently any timeline for that fix.

@flowerysong thanks for the info. Do you have a link to a GH issue or something that talks about the timeout discussion that we can track here?

https://github.com/ansible/ansible/issues/57542

SO @SwampDragons - per your question, I can confirm that using:
"extra_arguments": [
"--extra-vars",
"ansible_python_interpreter=auto_silent"
]
Allows a linux execution using ansible-local to work without issue using the following:
Ansible 2.8.1
Packer 1.4.2
KVM Build of RHEL7

However - those same arguments result in an error still when attempting to provision a WINDOWS Server 2019 host with the following Error:

2019-07-15T14:05:46-04:00: ==> qemu: Executing Ansible: ansible-playbook --extra-vars packer_build_name=qemu packer_builder_type=qemu -o IdentitiesOnly=yes -i /tmp/packer-provisioner-ansible556061269 /opt/jenkins/workspace/-templates_2019_imagebuild_PR-10/windows/ansible/initial_config.yaml -e ansible_ssh_private_key_file=/tmp/ansible-key458833230 --extra-vars packer_http_addr=10.0.2.2:8084 --connection packer --extra-vars ansible_shell_type=powershell ansible_shell_executable=None ansible_python_interpreter=auto_silent

2019-07-15T14:05:55-04:00: qemu:

2019-07-15T14:05:55-04:00: qemu: PLAY [all] ***********************

2019-07-15T14:05:55-04:00: qemu:

2019-07-15T14:05:55-04:00: qemu: TASK [Gathering Facts] *******************

2019-07-15T14:05:56-04:00: qemu: fatal: [default]: FAILED! => {"ansible_facts": {}, "changed": false, "msg": "The following modules failed to execute: setup\n setup: MODULE FAILURE\nSee stdout/stderr for the exact error\n"}

2019-07-15T14:05:56-04:00: qemu:

2019-07-15T14:05:56-04:00: qemu: PLAY RECAP ***********************

2019-07-15T14:05:56-04:00: qemu: default : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

2019-07-15T14:05:56-04:00: qemu:

2019-07-15T14:05:56-04:00: ==> qemu: Deleting output directory...

2019-07-15T14:05:56-04:00: Build 'qemu' errored: Error executing Ansible: Non-zero exit status: exit status 2

So I think we have a valid workaround for "Linux" while we await the pipeline "fix" in ansible core, but "Windows" still needs something more to allow it to work for the time being?

Anyone currently working on this or have any ideas on how to approach a fix?

Not that I'm aware of.

My fix was to add this: "extra_arguments": ["-e", "ansible_python_interpreter=/usr/bin/python", "-vv"]

I still can't reproduce this issue to save my life, but I've decided to move forward on trying to create a workaround based on the theory that the underlying issue is Packer's ssh proxy that it creates to forward Ansible calls.

I've created a PR #8625 from a branch that removes this localhost proxy altogether and modifies the provisioner to instead use the host IP in the inventory file. I'd love if some of y'all affected by the bug could pull down a build (available here: https://circleci.com/gh/hashicorp/packer/29969#artifacts/containers/0) and let me know if it solves the issue for you. Please note that as it stands right now, that branch breaks Docker builds. I'll figure out how to un-break them after we figure out if this move will actually solve the issue.

Please also let me know if the proxy removal causes issues for you; that PR is very much in a Work In Progress state.

Any takers on testing out the above PR and letting me know if it resolves the ansible 2.8+ incompatability? I have new builds, available here: https://circleci.com/gh/hashicorp/packer/32248#artifacts/containers/0, which allow you to use or not-use the proxy adapter based on the boolean option "use_proxy"(currently defaults to true, but I'll change the default in the future if it seems worthwhile)

@SwampDragons , thanks for providing these new packer builds (v 1.5.2).

I have tried this new 1.5.2 build both on macos (Python 3.7.3) and from a docker
container (Python 3.6.9) but the packer build now stalls before running the ansible provisioner:

==> azure-arm: Waiting for WinRM to become available...
==> azure-arm: Timeout waiting for WinRM.

... on both architectures.

If I revert to packer 1.5.1, the connection to WinRM succeeds, powershell
provisioners run successfully, but the ansible provisioner fails no matter what
option or extra arguments are given. The 'ansible_python_interpreter' workaround
mentioned above does not work for me unfortunately.

The 2 environments I have tried to build:
1. macos [Darwin Kernel Version 19.3.0: root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64]
- packer 1.5.1
- builder: azure-arm
- os_type: Windows
- communicator: winrm
- ansible 2.9.2
- Python 3.7.3

1. Docker / mcr.microsoft.com/azure-cli:latest [ Linux 1cba84bd80dd
   4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 Linux ]
2. packer 1.5.1
   
   
   
   • builder: azure-arm
   
   
   • os_type: Windows
   
   
   • communicator: winrm
   
   
3. ansible 2.9.4
4. Python 3.6.9

debug logs:

---------
    azure-arm: [azure-arm]
    azure-arm: XX.XXX.142.52
==> azure-arm: Provisioning with Ansible...
==> azure-arm: Executing Ansible: ansible-playbook --extra-vars packer_build_name=azure-arm packer_builder_type=azure-arm -o IdentitiesOnly=yes -i /var/folders/08/_km87dpn38zf4c0yr8lnq8880000gp/T/packer-provisioner-ansible557376101 /Users/Laurent/work/ansible/win-playboom.yml -e ansible_ssh_private_key_file=/var/folders/08/_km87dpn38zf4c0yr8lnq8880000gp/T/ansible-key717334430 -vvvv --connection packer --inventory-file=../ansible/inventory/inventory_azure_rm.yml --extra-vars ansible_python_interpreter=/Users/Laurent/.pyenv/shims/python ansible_shell_type=powershell ansible_shell_executable=None
    azure-arm: ansible-playbook 2.9.2
    azure-arm: <XX.XXX.142.52> ESTABLISH WINRM CONNECTION FOR USER: packer on PORT 5986 TO XX.XXX.142.52
    azure-arm: fatal: [pkrvmnzc8laeuz0_3a38]: UNREACHABLE! => {
    azure-arm:     "changed": false,
    azure-arm:     "msg": "ssl: the specified credentials were rejected by the server",
    azure-arm:     "unreachable": true
    azure-arm: }
...
    azure-arm: fatal: [default]: FAILED! => {
    azure-arm:     "ansible_facts": {},
    azure-arm:     "changed": false,
    azure-arm:     "failed_modules": {
    azure-arm:         "setup": {
    azure-arm:             "failed": true,
    azure-arm:             "module_stderr": "OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1:
    ...
    ...
        azure-arm:             "module_stdout": "",
    azure-arm:             "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    azure-arm:             "rc": 1
    azure-arm:         }
    azure-arm:     },
    azure-arm:     "msg": "The following modules failed to execute: setup\n"
    azure-arm: }
    azure-arm:
    azure-arm: PLAY RECAP *********************************************************************
    azure-arm: default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
    azure-arm: pkrvmnzc8laeuz0_3a38       : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

---------

Thanks for the update -- I pushed a few fixes and new binaries can be found on the PR. I haven't had a chance to set up the no-proxy option for winrm, though.

I was finally able to reproduce the hang and can confirm that disabling the proxy, as possible in the linked Pr, fixes the issue for SSH builds. I haven't had a chance to research and implement the fix for WinRM yet.

Artifacts for those of you on SSH who need this workaround can be found here: https://circleci.com/gh/hashicorp/packer/33086#artifacts/containers/0 .

Because I haven't gotten this going for Windows, this will probably not make it into the 1.5.2 release, but I'll pick this back up and continue to work on it in a couple of days.

Thanks @SwampDragons , that's great news! Looking forward to get the fixes for Windows builds when you're able to continue work on these.

I can confirm that using the build above fixes the issue with Ansible connecting to the Packer instance via SSH. 🚀

I'm experiencing the same issue in ansible 2.9 with winrm. Then I downgrade the ansible to 2.7 after that it was work fine once. But now I'm experience the same issue in ansible 2.7 as well.

ansible = 2.7.0
python version = 3.7.6
packer = 1.5.4

<127.0.0.1> (0, b'', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 127.0.0.1 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 14869\r\ndebug3: mux_client_request_session: session request sent\r\n#< CLIXML\r\nSystem.Management.Automation.PSCustomObjectSystem.Object1Preparing modules for first use.0-1-1Completed-1 debug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')

@SwampDragons any luck with windows update

Not yet -- I've been traveling and haven't had hands on keyboard this week. I'll try to pick the task back up next week though.

@SwampDragons is there a status on Windows? Thank you!

Yes! On Friday I got a POC for proxy-less Windows builds working using WinRM with basic auth, but I still need to do testing to make sure it works with ssl.

It lives! Binaries that work for ansible with winrm can be downloaded here: https://circleci.com/gh/hashicorp/packer/42423#artifacts/containers/0

See the docs added in the PR for detailed usage instructions: https://github.com/hashicorp/packer/pull/8625

Hi @SwampDragons Thank you for all of your work (and to others on reporting)! :)
I have tried the nightly build listed above, and it still fails for me. I still have to roll back to Ansible 2.7.10 for it to work for me.
[dev-user@centos-7-dev Downloads]$ ansible --version ansible 2.9.6 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/dev-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] [dev-user@centos-7-dev Downloads]$ packer --version 1.5.6 [dev-user@centos-7-dev Downloads]$
This is from a Centos 7.7 host using the vsphere-iso (great to see it built in now!) building a Centos 7.7 minimal image.
Anyone else found any other issues?

@ChrisGWarp I'll need a full repro case in order to look more at your failure, since it Works On My Machine ;)

packer_test.zip
Done!
Including log of failure, rectification (downgrade ansible) and success.
Hope this helps. :)

So, looking at your config:

        {
            "type":            "ansible",
            "playbook_file":   "./ansible/packer-test.yml",
            "galaxy_file":     "./ansible/requirements.yml",
            "user":            "root",
            "extra_arguments": [ "-v" ]
        }

Now I didn't test this with Galaxy, but also in your config it doesn't seem that you've actually turned off the proxy.

        {
            "type":            "ansible",
            "playbook_file":   "./ansible/packer-test.yml",
            "galaxy_file":     "./ansible/requirements.yml",
            "user":            "root",
            "use_proxy": false,
            "extra_arguments": [ "-v" ]
        }

Can you try the above with PACKER_DEBUG=1 in your environment to get verbose logs?, and link those in a gist?

Ok, I managed to get further, but then ran into other problems.
This is what I found/observed:

I was not sure about the use_proxy, if it was an existing parameter who's value needed to be changed, or if a new one.

Packer 1.5.5 chokes on it, so I'm assuming a new variable and therefore not backwards compatible.

Packer 1.5.6-dev did work, as it it did not hang at the gathering facts stage (yey!), but it then choked on the host key issue. Where does it load ansible.cfg from? Or, the same question another way, where (as in what directory) does ansible-playbook get spawned from?

This is the .json file fragment, the env vars did not appear to change the host key behaviour.

"provisioners": [ { "type": "ansible", "playbook_file": "./ansible/packer-test.yml", "galaxy_file": "./ansible/requirements.yml", "user": "root", "use_proxy": false, "ansible_env_vars": [ "ANSIBLE_HOST_KEY_CHECKING=False", "ANSIBLE_NOCOLOR=True" ], "extra_arguments": [ "-v" ] } ]

Here is the log output:

`[dev-user@centos-7-dev packer-test]$ PACKER_LOG=1 packer build -force vsphere-packer-test-x86_64.json
2020/04/26 20:46:14 [INFO] Packer version: 1.5.6-dev (d824b7e969d0d54ce23b42aa2a577a73a4780765) [go1.13.9 linux amd64]
2020/04/26 20:46:14 Checking 'PACKER_CONFIG' for a config file path
2020/04/26 20:46:14 'PACKER_CONFIG' not set; checking the default config file path
2020/04/26 20:46:14 Attempting to open config file: /home/dev-user/.packerconfig
2020/04/26 20:46:14 [WARN] Config file doesn't exist: /home/dev-user/.packerconfig
2020/04/26 20:46:14 Setting cache directory: /home/dev-user/eclipse-workspace/packer-test/packer_cache
2020/04/26 20:46:14 Creating plugin client for path: /usr/bin/packer
2020/04/26 20:46:14 Starting plugin: /usr/bin/packer []string{"/usr/bin/packer", "plugin", "packer-builder-vsphere-iso"}
2020/04/26 20:46:14 Waiting for RPC address for: /usr/bin/packer
2020/04/26 20:46:14 Received unix RPC address for /usr/bin/packer: addr is /tmp/packer-plugin421608791
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: [INFO] Packer version: 1.5.6-dev (d824b7e969d0d54ce23b42aa2a577a73a4780765) [go1.13.9 linux amd64]
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Checking 'PACKER_CONFIG' for a config file path
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: 'PACKER_CONFIG' not set; checking the default config file path
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Attempting to open config file: /home/dev-user/.packerconfig
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: [WARN] Config file doesn't exist: /home/dev-user/.packerconfig
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Setting cache directory: /home/dev-user/eclipse-workspace/packer-test/packer_cache
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: args: []string{"packer-builder-vsphere-iso"}
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Plugin address: unix /tmp/packer-plugin421608791
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Waiting for connection...
2020/04/26 20:46:14 packer-builder-vsphere-iso plugin: Serving a plugin connection...
2020/04/26 20:46:14 Creating plugin client for path: /usr/bin/packer
2020/04/26 20:46:14 Starting plugin: /usr/bin/packer []string{"/usr/bin/packer", "plugin", "packer-provisioner-ansible"}
2020/04/26 20:46:14 Waiting for RPC address for: /usr/bin/packer
2020/04/26 20:46:14 Received unix RPC address for /usr/bin/packer: addr is /tmp/packer-plugin434205582
2020/04/26 20:46:14 packer-provisioner-ansible plugin: [INFO] Packer version: 1.5.6-dev (d824b7e969d0d54ce23b42aa2a577a73a4780765) [go1.13.9 linux amd64]
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Checking 'PACKER_CONFIG' for a config file path
2020/04/26 20:46:14 packer-provisioner-ansible plugin: 'PACKER_CONFIG' not set; checking the default config file path
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Attempting to open config file: /home/dev-user/.packerconfig
2020/04/26 20:46:14 packer-provisioner-ansible plugin: [WARN] Config file doesn't exist: /home/dev-user/.packerconfig
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Setting cache directory: /home/dev-user/eclipse-workspace/packer-test/packer_cache
2020/04/26 20:46:14 packer-provisioner-ansible plugin: args: []string{"packer-provisioner-ansible"}
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Plugin address: unix /tmp/packer-plugin434205582
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Waiting for connection...
2020/04/26 20:46:14 packer-provisioner-ansible plugin: Serving a plugin connection...
vsphere-iso: output will be in this color.

2020/04/26 20:46:14 Build debug mode: false
2020/04/26 20:46:14 Force build: true
2020/04/26 20:46:14 On error:
2020/04/26 20:46:14 Preparing build: vsphere-iso
2020/04/26 20:46:15 Waiting on builds to complete...
2020/04/26 20:46:15 Starting build run: vsphere-iso
2020/04/26 20:46:15 Running builder: vsphere-iso
2020/04/26 20:46:15 [INFO] (telemetry) Starting builder vsphere-iso
2020/04/26 20:46:15 packer-provisioner-ansible plugin: ansible-playbook version: 2.9.6
==> vsphere-iso: Creating VM...
==> vsphere-iso: Customizing hardware...
==> vsphere-iso: Mounting ISO images...
2020/04/26 20:46:17 packer-builder-vsphere-iso plugin: Creating CD-ROM on controller '&{{{} 200 0xc00055e2a0 0 } 0 []}' with iso '[ISOs] CentOS/CentOS-7-x86_64-Minimal-1908.iso'
==> vsphere-iso: Creating floppy disk...
vsphere-iso: Copying files flatly from floppy_files
2020/04/26 20:46:18 packer-builder-vsphere-iso plugin: Floppy path: /tmp/packer579447498
2020/04/26 20:46:18 packer-builder-vsphere-iso plugin: Initializing block device backed by temporary file
2020/04/26 20:46:18 packer-builder-vsphere-iso plugin: Formatting the block device with a FAT filesystem...
2020/04/26 20:46:18 packer-builder-vsphere-iso plugin: Initializing FAT filesystem on block device
2020/04/26 20:46:18 packer-builder-vsphere-iso plugin: Reading the root directory from the filesystem
vsphere-iso: Copying file: http/ks-7.7-minimal-static.cfg
vsphere-iso: Done copying files from floppy_files
vsphere-iso: Collecting paths from floppy_dirs
vsphere-iso: Resulting paths from floppy_dirs : []
vsphere-iso: Done copying paths from floppy_dirs
==> vsphere-iso: Uploading created floppy image
==> vsphere-iso: Adding generated Floppy...
==> vsphere-iso: Starting HTTP server on port 8081
2020/04/26 20:46:19 packer-builder-vsphere-iso plugin: Found available port: 8081 on IP: 0.0.0.0
==> vsphere-iso: Set boot order temporary...
==> vsphere-iso: Power on VM...
==> vsphere-iso: Waiting 10s for boot...
==> vsphere-iso: HTTP server is working at http://A.B.C.E:8081/
==> vsphere-iso: Typing boot command...
2020/04/26 20:46:32 packer-builder-vsphere-iso plugin: Special code '' found, replacing with: CodeTab
2020/04/26 20:46:40 packer-builder-vsphere-iso plugin: Special code '' found, replacing with: CodeReturnEnter
2020/04/26 20:46:40 packer-builder-vsphere-iso plugin: Waiting 1 second
==> vsphere-iso: Waiting for IP...
2020/04/26 20:46:41 packer-builder-vsphere-iso plugin: [INFO] Waiting for IP, up to total timeout: 30m0s, settle timeout: 5s
2020/04/26 20:55:12 packer-builder-vsphere-iso plugin: VM IP aquired: A.B.C.D
2020/04/26 20:55:12 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
2020/04/26 20:55:13 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
2020/04/26 20:55:14 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
2020/04/26 20:55:15 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
2020/04/26 20:55:16 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
==> vsphere-iso: IP address: A.B.C.D
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: VM IP is still the same: A.B.C.D
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: VM IP seems stable enough: A.B.C.D
==> vsphere-iso: Using ssh communicator to connect: A.B.C.D
==> vsphere-iso: Waiting for SSH to become available...
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [INFO] Waiting for SSH, up to timeout: 5m0s
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [INFO] Attempting SSH connection to A.B.C.D:22...
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [DEBUG] reconnecting to TCP connection for SSH
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [DEBUG] handshaking with SSH
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [DEBUG] handshake complete!
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [DEBUG] Opening new ssh session
==> vsphere-iso: Connected to SSH!
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: [INFO] agent forwarding enabled
2020/04/26 20:55:17 packer-builder-vsphere-iso plugin: Running the provision hook
2020/04/26 20:55:17 [INFO] (telemetry) Starting provisioner ansible
==> vsphere-iso: Provisioning with Ansible...
vsphere-iso: Not using Proxy adapter for Ansible run:
vsphere-iso: Using ssh keys from Packer communicator...
vsphere-iso: Using ssh keys from Packer communicator...
2020/04/26 20:55:17 packer-provisioner-ansible plugin: Creating inventory file for Ansible run...
vsphere-iso: Executing Ansible Galaxy
vsphere-iso: [WARNING]: - dovry.ansible_role_sample (master) is already installed - use
vsphere-iso: --force to change version to unspecified
2020/04/26 20:55:18 packer-provisioner-ansible plugin: Megan cmd is &exec.Cmd{Path:"/usr/bin/ansible-playbook", Args:[]string{"ansible-playbook", "-e", "packer_build_name=vsphere-iso", "-e", "packer_builder_type=vsphere-iso", "-e", "ansible_ssh_private_key_file=/tmp/ansible-key848613781", "-e", " packer_http_addr=A.B.C.E:8081", "--ssh-extra-args", "-o IdentitiesOnly=yes", "-i", "/tmp/packer-provisioner-ansible807514096", "/home/dev-user/eclipse-workspace/packer-test/ansible/packer-test.yml", "-v"}, Env:[]string(nil), Dir:"", Stdin:io.Reader(nil), Stdout:io.Writer(nil), Stderr:io.Writer(nil), ExtraFiles:[]os.File(nil), SysProcAttr:(syscall.SysProcAttr)(nil), Process:(os.Process)(nil), ProcessState:(os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)}
==> vsphere-iso: Executing Ansible: ansible-playbook -e packer_build_name=vsphere-iso -e packer_builder_type=vsphere-iso -e ansible_ssh_private_key_file=/tmp/ansible-key848613781 -e packer_http_addr=A.B.C.E:8081 --ssh-extra-args -o IdentitiesOnly=yes -i /tmp/packer-provisioner-ansible807514096 /home/dev-user/eclipse-workspace/packer-test/ansible/packer-test.yml -v
vsphere-iso: Using /etc/ansible/ansible.cfg as config file
vsphere-iso:
vsphere-iso: PLAY [Configure the base VM] ***************
vsphere-iso:
vsphere-iso: TASK [Gathering Facts] *****************
Enter passphrase for key '/tmp/ansible-key848613781':
vsphere-iso: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added 'A.B.C.D' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}
vsphere-iso:
vsphere-iso: PLAY RECAP **********************
vsphere-iso: default : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
vsphere-iso:
2020/04/26 20:55:50 [INFO] (telemetry) ending ansible
==> vsphere-iso: Provisioning step had errors: Running the cleanup provisioner, if present...
==> vsphere-iso: Clear boot order...
==> vsphere-iso: Power off VM...
==> vsphere-iso: Deleting Floppy image ...
==> vsphere-iso: Destroying VM...
2020/04/26 20:55:51 packer-builder-vsphere-iso plugin: Deleting floppy disk: /tmp/packer579447498
Build 'vsphere-iso' errored: Error executing Ansible: Non-zero exit status: exit status 4

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.
2020/04/26 20:55:52 [INFO] (telemetry) ending vsphere-iso
2020/04/26 20:55:52 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2020/04/26 20:55:52 machine readable: vsphere-iso,error []string{"Error executing Ansible: Non-zero exit status: exit status 4"}
==> Builds finished but no artifacts were created.
2020/04/26 20:55:52 [INFO] (telemetry) Finalizing.
2020/04/26 20:55:53 waiting for all plugin processes to complete...
2020/04/26 20:55:53 /usr/bin/packer: plugin process exited
2020/04/26 20:55:53 /usr/bin/packer: plugin process exited
[dev-user@centos-7-dev packer-test]$
`

Packer 1.5.5 chokes on it, so I'm assuming a new variable and therefore not backwards compatible.

Correct. Docs for the new feature can be found in the linked PR.

Packer 1.5.6-dev did work, as it it did not hang at the gathering facts stage (yey!), but it then choked on the host key issue. Where does it load ansible.cfg from? Or, the same question another way, where (as in what directory) does ansible-playbook get spawned from?

ansible-playbook gets spawned from the same directory you're running Packer from.

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.