Toolbox: Containers don't start when /etc/localtime is an absolute symbolic link on the host (eg., Arch and Ubuntu)

I'm getting the exact same problem (on Arch Linux as well). Downgrading to 0.0.96 did the trick for me.

These are the Podman logs for a Fedora 32 toolbox:

$ podman logs fedora-toolbox-32
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1000"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Resolved /etc/localtime to /usr/share/zoneinfo/Europe/Amsterdam"
Error: /etc/localtime points to unknown location

I see the same

level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1000"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Preparing to redirect /etc/localtime to /run/host/etc/localtime"
level=debug msg="/run/host/etc/localtime is a symbolic link"
level=debug msg="Redirecting /etc/localtime to /run/host/etc/localtime"
level=debug msg="Resolved /etc/localtime to /usr/share/zoneinfo/Europe/Copenhagen"
Error: /etc/localtime points to unknown location

Could this be caused by 4c9b80aee2b2ee3162b82e309718a23792d0e103 ?

I'm running into the same issue on Manjaro (arch based). Going through commits, I found b9a0bd5f0c2a2421ec15eea286ca20f03b7152d2 to be the culprit. 4c9b80aee2b2ee3162b82e309718a23792d0e103 appears to be the last one that works correctly.

Has anyone tried 0.0.98.1 on Arch Linux?

Out of curiosity, your Arch host systems don't have a /etc/localtime ?

Yes, Arch hosts does have /etc/localtime. It is a symlink into /usr/share/zoneinfo/.
Problem is that toolbox has:

        const zoneInfoRoot = "/run/host/usr/share/zoneinfo"

        if !strings.HasPrefix(localTimeEvaled, zoneInfoRoot) {
                return errors.New("/etc/localtime points to unknown location")
        }

If I change zoneInfoRoot to "/usr/share/zoneinfo", toolbox works nicely.

Before calling updateTimeZoneFromLocalTime(), the /etc/localtime link is updated to point to the same path as /run/host/etc/localtime points to.
I fail to see how that should make /etc/localtime point into /run/host, as host files doesn't know anything about toolbox.

So unless I miss something, maybe even obvious to anyone else, zoneInfoRoot should simply be changed/fixed. If there are both use-cases for /etc/localtime to point to /run/host/usr/share/zoneinfo/... and /usr/share/zoneinfo/... I guess we can check for both possibilities, and resolve timeZone based on the matching prefix.

Yes, Arch hosts does have /etc/localtime. It is a symlink into
/usr/share/zoneinfo/.

Ok, so it's not like the situation on Fedora CoreOS.

This should be addressed by https://github.com/containers/toolbox/pull/634

However, I must mention that both Arch and Ubuntu should be using relative symbolic links. If they did, this would continue to just work, because relative symlinks are able to handle chroot-like situations, which this is, better. eg., on Fedora Workstation we have:

$ ls -l /etc/localtime 
lrwxrwxrwx. 1 root root 35 Jan  5 18:20 /etc/localtime -> ../usr/share/zoneinfo/Europe/Prague

Whereas, on Ubuntu 18.04, I see:

$ ls -l /etc/localtime 
lrwxrwxrwx 1 root root 33 lis 11 16:21 /etc/localtime -> /usr/share/zoneinfo/Europe/Prague

But, if I use timedatectl then it gets fixed:

$ timedatectl set-timezone Europe/Prague
$ ls -l /etc/localtime 
lrwxrwxrwx 1 root root 35 led 12 20:29 /etc/localtime -> ../usr/share/zoneinfo/Europe/Prague

Yes, changing the /etc/localtime symlink to an absolute is a usable workaround. I will use this for now, thanks.

But I think #634 should do merged. I don't see much point in not supporting such absolute symlinks.

Closing. Feel to re-open or leave a comment if you think this still isn't fixed.

And thanks for testing Toolbox out.