Toolbox: sudo(8) inside toolbox containers asks for a password with Podman 2.0.5

I did some testing on Fedora Workstation 32 and Fedora Workstation 33 (on virtual machines) and I can confirm that on F32 was working fine, but on F33 I got the same issue.
So, this is an issue with the Fedora 33 itself, not just the Silverblue version.

After some testing I noticed some differences:

• The podman's version is different.
  
  
  
  • F32: 2.0.2
  
  
  • F33: 2.1.0-dev
  
  
• The files /etc/group and /etc/shadow are different.

At the toolbox container in Fedora 33 the user doesn't have its own group and it's not at the group wheel.
Also, the user doesn't have an entry at the /etc/shadow file and the rootuser has the password locked.

The diff for the /etc/group file (inside the container) between the Fedora 32 and the Fedora 33. The username for the user was vagrant:

--- f32-image-f33/group 2020-08-14 19:19:38.734363987 +0000
+++ f33-image-f33/group 2020-08-14 19:17:39.018504713 +0000
@@ -8,7 +8,7 @@
 lp:x:7:
 mem:x:8:
 kmem:x:9:
-wheel:x:10:vagrant
+wheel:x:10:
 cdrom:x:11:
 mail:x:12:
 man:x:15:
@@ -26,4 +26,3 @@
 utempter:x:35:
 ssh_keys:x:999:
 tcpdump:x:72:
-vagrant:x:1000:

The diff for the /etc/shadow:

--- f32-image-f33/shadow    2020-08-14 19:15:25.125242112 +0000
+++ f33-image-f33/shadow    2020-08-14 19:17:11.658920405 +0000
@@ -1,4 +1,4 @@
-root::18488:0:99999:7:::
+root:!locked::0:99999:7:::
 bin:*:18473:0:99999:7:::
 daemon:*:18473:0:99999:7:::
 adm:*:18473:0:99999:7:::
@@ -12,4 +12,3 @@
 ftp:*:18473:0:99999:7:::
 nobody:*:18473:0:99999:7:::
 tcpdump:!!:18481::::::
-vagrant::18488:0:99999:7:::

The problem is still present in rawhide (F33SB) with the exception of the message on entering toolbox (/usr/bin/id: cannot find name for group ID 1000) is no longer popping up.

I confirm that I too saw error (/usr/bin/id: cannot find name for group ID 1000) so i removed fedora-toolbox-33 image and re-created toolbox but now I cannot use sudo command. I am stuck now.

I follow up with the findings I did the other day and I manage to get the sudo working.

Steps:

(All this in a Fedora Workstation 33 in a VM)

1. Create the container.

[vagrant@ci-node-33 ~]$ toolbox create
Created container: fedora-toolbox-33
Enter with: toolbox enter

1. Enter into the container with toolbox and try the command sudo:

⬢[vagrant@toolbox ~]$ sudo ls

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for vagrant: 

It fails! :disappointed:

1. Enter into the container with podman:

[vagrant@ci-node-33 ~]$ podman exec -it fedora-toolbox-33 /bin/bash

1. As the user didn't have the right groups and it wasn't at the shadow file (as I showed at the previous comment), I deleted and re-created the user, trying to use the same commands and parameters that toolbox does (at the init-container command):

# Delete the user
⬢[root@toolbox /]# userdel --force vagrant

# Create the user
⬢[root@toolbox /]# useradd --home-dir /home/vagrant/ --no-create-home --shell /bin/bash --uid 1000 --groups wheel vagrant

# Check the user groups (this time are OK)
⬢[root@toolbox /]# id vagrant
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant),10(wheel)

# Delete the user password
⬢[root@toolbox /]# passwd --delete vagrant
Removing password for user vagrant.
passwd: Note: deleting a password also unlocks the password.
passwd: Success

# Check that the user is at the file /etc/shadow (this is important for PAM authentication and sudo)
⬢[root@toolbox /]# grep vagrant /etc/shadow
vagrant::18493:0:99999:7:::

# Logout from the container
⬢[root@toolbox /]# exit
[vagrant@ci-node-33 ~]$ 

1. Enter into the container using toolboxand try the sudo command:

[vagrant@ci-node-33 ~]$ toolbox enter
⬢[vagrant@toolbox vagrant]$ sudo id

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

uid=0(root) gid=0(root) groups=0(root)

Now it works! :smile:

Conclusion

Something seems to go wrong at the user generation, at the init-container command (somewhere around here).

Thank you for reporting this and trying to find the culprit! It seems that Podman in Rawhide changed the way it handles the --userns=keep-id option when creating containers. It creates now the user and the group. The first problem seems to be related to this functionality itself because the user group does not have a name despite having the correct GUID (I reported that upstream: https://github.com/containers/podman/issues/7389).

The other part (having to type password for sudo) is caused by the fact that the code path pointed out by @juanje is only triggered if the current user is not present in the container. That code takes care of creating the user, adding it to the correct groups and deleting passwords for the user and root. I think the code in the init-container command needs to be restructured a bit to be called even such cases.

It seems that Podman in Rawhide changed the way
it handles the --userns=keep-id option when creating
containers. It creates now the user and the group. The
first problem seems to be related to this functionality
itself because the user group does not have a name
despite having the correct GUID (I reported that upstream:
containers/podman#7389).

Seems like it's not actually creating the group. Otherwise you'd have a name. It's just creating the user.

I wonder if it's also creating a home directory. I hope not.

https://github.com/containers/podman/pull/6829 was the offending Podman change.

This is happening for me now on Silverblue 32 using fedora-toolbox:f32 image
toolbox version: 0.0.93
podman version: 2.0.5

This is happening for me now on Silverblue 32 using fedora-toolbox:f32 image
toolbox version: 0.0.93
podman version: 2.0.5

Yeah, I have the issue on Silverblue 32 now on any new toolbox instance of F32. With old toolbox instance, it still works, but not with new creations. So, whatever was done to "fix" it, broke it on F32.

Can confirm this issue on Fedora 32 SB with toolbox 0.0.93 and podman 2.0.5.

This is not related or confined to Fedora only, Arch with toolbox 0.0.94 and podman 2.0.5 has the same problem.

This is happening for me now on Silverblue 32 using fedora-toolbox:f32 image
toolbox version: 0.0.93
podman version: 2.0.5

That's because Podman 2.0.5 slipped into Fedora 32.

Is it possible to add some tests for podman or toolbox to catch regressions like this? Silverblue really encourages one to use Toolbox, and for that to happen Toolbox needs to be as reliable as gnome-terminal itself.

Is it possible to add some tests for podman or toolbox to
catch regressions like this? Silverblue really encourages
one to use Toolbox, and for that to happen Toolbox needs
to be as reliable as gnome-terminal itself.

It has proven to be a surprisingly uphill battle to get the Podman team to care about backwards compatibility or to check if some change breaks Toolbox or not. @HarryMichal is consistently chasing down breakages and pushing for tests, but progress is slow.

Is it possible to add some tests for podman or toolbox to
catch regressions like this? Silverblue really encourages
one to use Toolbox, and for that to happen Toolbox needs
to be as reliable as gnome-terminal itself.

It has proven to be a surprisingly uphill battle to get the Podman team to care about backwards compatibility or to check if some change breaks Toolbox or not. @HarryMichal is consistently chasing down breakages and pushing for tests, but progress is slow.

I guess one way to avoid this problem is for Silverblue to have a separate rpm repo for podman and only alllow updates that are passing regression tests against toolbox

When is it supposed to get to f32 updates?

The sooner it reach to 3 positive karma :)

You can track it yourself -> https://bodhi.fedoraproject.org/updates/FEDORA-2020-306addaac0

This karma system is new to me, how does it work?

You will need FAS id (Fedora account id) to login to bodhi system and provide your vote on the updates. See https://fedoraproject.org/wiki/Bodhi#Karma

On SilverBlue, can I pull only given package and test it? It seems it needs to be rebased to test a package. Any method of testing the package(including containers, if possible) without needing to touch my base system is fine.

On SilverBlue, can I pull only given package and test it?
It seems it needs to be rebased to test a package.
Any method of testing the package(including containers,
if possible) without needing to touch my base system is fine.

rpm-ostree override replace and rpm-ostree override reset are your friends.

Unfortunately, you can't test things like Podman and Toolbox inside a container.

Thanks so much for the workaround @juanje

# Create the user
⬢[root@toolbox /]# useradd --home-dir /home/vagrant/ --no-create-home --shell /bin/bash --uid 1000 --groups wheel vagrant

The only change I made for my user on a Silverblue host was making the home directory /var/home/<user>

Should this be reopened? Definitively still happening with 0.0.95. toolbox create seems to put the user into the guest's /etc/passwd alright, but forgets about copying the /etc/group entry. Something like echo 'martin:x:1000' | sudo tee -a /etc/group in the container fixes it.

Have recorded the same experience in https://github.com/containers/toolbox/issues/549#issuecomment-685740230 -- commented there as this doesn't (usually) break sudo.

Definitively still happening with 0.0.95. toolbox create seems to
put the user into the guest's /etc/passwd alright, but forgets about
copying the /etc/group entry. Something like
echo 'martin:x:1000' | sudo tee -a /etc/group in the container fixes it.

What's still happening? You mean you are seeing this error when entering a container:

/usr/bin/id: cannot find name for group ID 1000

That's https://github.com/containers/podman/issues/7389

I shied away from adding a similar workaround to Toolbox itself because it wouldn't correctly factor in things like /etc/login.defs.

Or did I misunderstand?

@debarshiray : Thanks for the podman issue pointer! That looks like the root cause indeed. In the meantime, there workaround above is easy enough.