Toolbox: Isolated development environments to fend against bugs and malware in code under development

You want a persistent user container without your /home? Just trying to understand the requirement/usecase better. - Say compared to just running a standard throwaway Fedora container?

Consider a Unix account which is used for both home and work. For a work container, I may prefer to mount only my work directory into the the container.

Consider an untrusted GUI app. It may only need access to a specific folder to load/save files. It doesn't need access to my .ssh folder and other unrelated files.

For improve security, Chrome OS does not share folders to containers by default from the host. There, if you want to provide data to the containers from your home directory or from a USB drive, you explicitly share it.

Containerization as a security feature loses significant value if you give the untrusted container all your data by default!

@juhp I see you work a good deal with Haskell packages. Presume that one of the Haskell packages you are install remotely has been compromised. Do Haskell packages need access to your .ssh folder or your Bitcoin wallet? Why share everything into your Haskell dev environment by default?

Fairly recently there was an NPM module that could still local Bitcoin if it was installed:
https://www.trendmicro.com/vinfo/nz/security/news/cybercrime-and-digital-threats/hacker-infects-node-js-package-to-steal-from-bitcoin-wallets

The module was popular and in fact my team had installed it into our tree. We didn't meet the the other criteria for it to trigger a bitcoin theft, but that risk would have been completely mitigated if our dev environment did not have complete access to our home directories by default. No employee would have chosen to share their Bitcoin wallet into this development environment.

This is totally valid but is going to be hard.

Practically speaking, I think the best path is to make it easy to run toolboxes as separate users (transiently create new uids, each with their own /home) - requires a privileged system service to do that.

Now if you want share some files e.g. read-only - that gets messier.

Is it possible to exclude files/dirs from a directory bind-mount?

To make sure I understand the challenge, I see that mounting the home directory happens in one line of code

    --volume "$HOME":"$HOME":rslave

But the challenge here is still making GUI apps and some other features work, because those features expect that the user inside and outside the container is the same?

This is actually a rather important feature. I think it's fine to mount home by default, but it should be possible to use a container in an untrusted manner. Mounting home into a container assumes you trust the container, which in a throwaway container often isn't the case. Even in simple cases like testing some NPM module out, I'd like to have the peace of mind that it isn't going to wreck my home folder.

@markstos maybe you can test what you can do without the home mount or if just mounting a specific dir (cwd)?

@juhp Actually this was showstopper for me-- I stopped evaluating Silverblue once I ran into this missing feature. If this addressed, I may give Silverblue a second look. I was also disappointed to find that it wasn't easy to figure out how to run a Ubuntu container, which is the OS I've been using before.

If Fedora wants to welcome users coming from other Linux distros, making it easy to run their old distro in a container would be a good place to start.

@markstos thanks - I am just another Toolbox user. :-)
I agree completely that extending Toolbox to support more OSes would be very useful indeed.
Note that Toolbox also works without Silverblue (ie on other Fedora editions).

@juhp Actually this was showstopper for me-- I stopped evaluating Silverblue once I ran into this missing feature

What did you do before? Is there some other tool/approach you were using that didn't work on Silverblue?

Nothing at all stops you from e.g. using a disposable VM (QubesOS like), or for that matter having a custom script which implements some of the suggestions from this thread. Arguably, we should be more opinionated and build in QubesOS-like functionality into Silverblue.

But anyways existing VM and container technology is there. In fact toolbox today is really a bunch of scripts that make podman intentionally blur with the host. If you don't want the integration, you can start with podman run... since that's the default.

@cgwalters I have started using Chrome OS on my personal laptop and have come to appreciate the security of having my development environment (and nearly everything else) running within a container in VM through the Crostini project. I like that it supports GUI apps as well as command line apps. I also like that containers are private by default. I have to explicitly share data folders or USB drives with them. On the other hand, sound and wayland support is automatically set up in those containers.
url
I've been evaluating similar solutions to use on my work laptop instead. One option would be Neverware's Cloudready-- an open source Chrome OS for PC hardware. Sometimes there are ports that Crostini crashes, data is lost and starting over is necessary. For that reason, I hesitate on doubling down on ChromeOS / Crostini now.

Silverblue also looked compelling until I found it doesn't appear to support Ubuntu containers out of the box and it over-shares my personal data with containers I don't intend to trust. I also looked at Clear Linux. It has same concept of running nearly everything in containers. I am not thrilled with the close ties to Intel and x86. It's also not primarily a desktop OS. A final (default?) option would be to stick with Ubuntu as a desktop and move more my development work into LXD containers, which what Chrome's Crostini is using. I should be able copy LXD containers between my work and personal laptops even though the host OS is different. Using LXD templates, I should be able to set up a template shares enough mounts into the containers for Wayland support.

side note: Thanks for all the years of maintaining Rhythmbox!

Perhaps I misunderstand the mission of toolbox. From the README:

.. Intention of these systems is to discourage installation of software on the host

Why? With the default is to share all your personal data into the container by default, I don't think improved security can be claimed.

The remaining potential benefit would isolation, so you could install conflicting versions of software side-by-side.

If the always-share behavior remains, it could be helpful to update the docs clarify that the toolbox shares your entire home directory into the containers, that the behavior can't be disabled and it should not be used with untrusted containers.

I get that I can use podman directly, but I was interested a container solution with GUI integration. When searching on how to accomplish that with podman, using toolbox is the recommended approach:

https://discussion.fedoraproject.org/t/how-to-run-a-containerized-gui-application/570

@markstos If you're interested, I've created PR #298 which builds an Ubuntu 19.04 image.

I think there's some confusion here.

Our README.md has grown and morphed a bit over the months, and is probably a bit harder to grasp. Earlier it used to simply say Hacking on OSTree-based Fedoras.

If you install Silverblue (or any other OSTree-based OS for that matter) the difficulty of debugging the OS or setting up a development environment becomes immediately obvious. Toolbox exists to solve that problem.

.. Intention of these systems is to discourage installation of software on the host

Why? With the default is to share all your personal data into the container by default, I
don't think improved security can be claimed.

Security is an overloaded word. Toolbox doesn't make any claims about it.

For many decades, any process running as your UID could look at your private cryptographic keys, documents, photos, etc. and even transmit them half way across the planet without you ever knowing. This is the status quo of free software client OSes.

Flatpak changes that by separating out each graphical application and the operating system into their own security domains. Silverblue solidifies this separation by making it difficult to install software directly inside the host OS image. So, for a smooth user experience, you really need to use applications shipped as Flatpaks.

However, as I mentioned at the top, this locked down OS image presents its own set of problems. How we solve them is a trade-off between convenience and security. The more radical the solution, the more difficult it would be for existing Linux users to adopt Silverblue.

At the moment, Toolbox leans towards adoption over security; because regardless of whether you are using Toolbox or not, Silverblue does make a quantum improvement in the state of free software client OSes and getting that into user's hands would be a positive step.

Plus, it's not just about security. It's also about stability.

It's very difficult to test a conventional Linux distribution. Packages and mirrors are always getting updated by random contributors on random mirrors all over the world - the combinatorial explosion can only be managed by an elaborate system of freezes. When things go wrong, and they do, it's also very difficult for a user to revert an update, and things like power cuts in the middle of an update can irretrievably break the system.

An OSTree-based OS changes all that.

I get that I can use podman directly, but I was interested a container solution with GUI
integration. When searching on how to accomplish that with podman, using toolbox is
the recommended approach:

https://discussion.fedoraproject.org/t/how-to-run-a-containerized-gui-application/570

The question is why do you want to use Podman to run graphical applications? :)

In general, Podman (or an OCI container) is a bad option for running GUI apps. That's the whole reason why Flatpak exists and Toolbox doesn't compete with that.

However, there is an overlap, in the sense that Toolbox containers do have some desktop integration, and there are some cases where the ability to run a non-Flatpaked GUI app is immensely useful in the immediate short-term. It could be that the application that you want hasn't been Flatpaked yet, or maybe the Flatpaked version is missing some features. It could be that you are working on some library that's used by graphical applications, and you want to quickly run a one-off test program to see if your library is working as expected.

Toolbox can, indeed, help in such cases, but that's different from saying that Toolbox's primary aim is to containerize graphical applications. Toolbox's primary aim is to let you get your hacking done on an OSTree-based OS.

Now, if you were using a container-native IDE like GNOME Builder, which is shipped as a Flatpak, and automatically sets up a container to build and run the software you are working on, then you wouldn't need Toolbox at all.

However, not everybody uses GNOME Builder, and the most popular IDEs like Visual Studio Code aren't container-native like that. Hence, Toolbox.

@debarshiray Thanks for the thoughtful and thorough response. The example I gave above was not for a graphical apps which might be covered by Flatpak. I gave the example of doing Node.js development. There was recently real malware in the Node.js dependency chain that could steal from a local crypto wallet. SSH keys could be stolen just as easily. While the README says the project targets developers, the share-with-containers-by-default does nothing secure developers from that kind of attack.

Toolbox does not provide safety from theft of personal files when doing CLI development with any untrusted dependencies. Given the complexity of modern open source software, there's bound to be some parts of the dependency tree that should be not trusted.

I find the current README to be misleading: Running "fully unprivileged in a container" sounds like security, but it is only security theater-- all personal files are accessible in the container. Above you clarify that Toolbox intends to make no claims about security. I think that would a useful statement to include in the README to clarify that the despite the use of containers, this is a tool for convenience not security.

@debarshiray Thanks for the thoughtful and thorough response. The example I gave
above was not for a graphical apps which might be covered by Flatpak. I gave the
example of doing Node.js development. There was recently real malware in the Node.js
dependency chain that could steal from a local crypto wallet. SSH keys could be stolen
just as easily. While the README says the project targets developers, the
share-with-containers-by-default does nothing secure developers from that kind of attack.

Yes, at the moment Toolbox merely tries to reproduce the traditional package-based environment within an image-based OS.

I find the current README to be misleading: Running "fully unprivileged in a container"
sounds like security, but it is only security theater-- all personal files are accessible in
the container. Above you clarify that Toolbox intends to make no claims about security.
I think that would a useful statement to include in the README to clarify that the despite
the use of containers, this is a tool for convenience not security.

Commit c047659c1d85ca982374da5c58ee7a24ba3847bd is where that line was added. I believe @cgwalters intended to say that you only have access to your own UID inside a toolbox container and the real root (ie., UID 0 on the host) is neither involved or available to you.

I too have the same problem. I installed Silverblue with the hope that I would be able to run software which I may not trust entirely without having to worry that it steals my SSH keys and other sensitive information.

The realisation that toolbox does not allow me to do this was a huge disappointment and have lead me to reconsider the use of Silverblue to begin with. Personally, I don't care about the main OS installation. I can always reinstall that. The information I want to truly keep segregated is located in my home directory.

Would it be possible to specify with some greater granularity precisely what parts of the home directory is shared? If one could whitelist just the directories that are needed by toolbox (probably the ones relating to desktop configuration, etc) this would solve a lot of issues.

Now, I'm perfectly aware of the issues with security. Since I've been using Qubes OS for several years, the problem of isolation is a difficult one. Even if you protect the home directory, that's not going to be enough since a program running inside a container can always take advantage of other means of communication between containers, such as the clipboard. I'm aware of this, and I am willing to accept some of those risks in exchange for the comvenience of something like toolbox compared to a full Qubes installation.

A workaround is published here.

The workaround exploits the fact that toolbox is implemented as a bash script which references the environment variable $HOME in all cases where the path of /home/user needs to be looked up. So by setting the environment variable $HOME for a particular call to toolbox causes a a directory other than your entire home directory full of valuable personal files to be shared with a potentially untrusted container.

Based on that, it appears you could share an empty directory that contains just the toolbox configuration by launching toolbox with a wrapper something like this:

#!/bin/bash
mkdir -p ~/toolboxes
HOME=~/toolboxes "$@"'

If this would script was named toolbox and put in your $PATH ahead of the system toolbox, then it appears it would address this concern. This solution is untested.

This feature is now working in the tlbx fork: https://gitlab.com/uppercat/tlbx/issues/3

I too have the same problem. I installed Silverblue with the hope that I would be able to
run software which I may not trust entirely without having to worry that it steals my SSH
keys and other sensitive information.

Umm... this seems misleading. If you want to run untrusted software as a user, then Flatpak already reduces your worries about losing sensitive information. If you want isolated development environments, then that's a different thing.

Also, none of this is Silverblue specific in any way. The problems have existed for decades, and the solutions aren't Silverblue-specific - they work on traditional package-based OSes too.

The workaround exploits the fact that toolbox is implemented as a bash script which
references the environment variable $HOME in all cases where the path of /home/user
needs to be looked up. So by setting the environment variable $HOME for a particular
call to toolbox causes a a directory other than your entire home directory full of valuable
personal files to be shared with a potentially untrusted container.

Yes, that's a neat hack. :)

However, the primary aim of the Toolbox project is to enable setting up a wide variety of development environments on image-based (particularly OSTree-based) operating systems with roughly the same amount of effort that's necessary on their package-based counterparts. Making these development environments more secure is definitely a valid desire, but it's not something that's on the immediate to-do list, because the problem isn't new and has existed since forever.

Also Silverblue isn't necessarily about security. I see it more as an improvement in the stability and robustness of the operating system. However, it does encourage the use of Flatpaks, so in that sense it does indirectly improve security.

Therefore, there's value in ensuring feature parity with package-based Linux distributions, even if it retains some of their existing problems. Solving some problems and making the solutions accessible to the existing user base is better than solving nothing. :)

I am going to close this issue so that the list of open issues doesn't balloon out of control, and continues to roughly reflect the items that are on the current to-do list. However, it will still be around for future reference.

I do agree that our README.md should be easier to read, though.

Hmm, I don't understand that argument. Just because a problem is old and has been there forever, just don't solve it?? That's not a good point of view, IMHO.
And as you say, flatpak does that: It step by step improves security and isolates applications. They also did not say "hmm, we've shipped apps as rpm packages forever, why should we solve this problem?", they just did solve it.

I see no reason why toolbox should not at least isolate the home dir too. So I guess the discussion can be continued in https://github.com/containers/toolbox/issues/348 anyway.

I think contributors and maintainers of toolbox are taking this thread as a bashing on the idea and the work that is put into the project. I think it should be taken as a feature that is required by many people excited by the idea. I don't get the resistance to working on a feature that allows you to mount a working directory other than $HOME. It is a feature that could greatly improve functionality and usecase which will also improve adoption and interest.

I was the one who suggested changing $HOME variable in #348 . It is a hacky solution because it causes problems when you want to work with two multiple containers with different $HOME directories. My fear is even this will not work when toolbox is rewritten with go.

Many people are asking for this feature and it has become a barrier to entry for many people to adopt toolboxes, rpm-ostree and silverblue. Even if it is not about security it is a feature that is going to open up a whole lot of possibilities for working with toolboxes.

I'm willing to help on the issue if I can and I think others will be willing too. I hope this feature doesn't become a point of contention and be left out of future toolbox releases. It is in my opinion the only thing that is currently missing in it. It would be like getting a dnf in a marathon because of quitting the last 10 meters. (pun intended :) )

It was a deal-breaker for me. I gave up on toolbox and Silverblue after
finding this flaw.

Just because a problem is old and has been there forever,
just don't solve it?? That's not a good point of view, IMHO.

Not having something on the immediate to-do list isn't the same thing as don't solve it. There's such a thing as priority.

And as you say, flatpak does that: It step by step improves security and
isolates applications. They also did not say "hmm, we've shipped apps as
rpm packages forever, why should we solve this problem?", they just did
solve it.

I love your use of the word they.

Okay, nice, so if I interpret that correctly, this may still be an idea for the future, just not on your "immediate to-do list "? (though I'd guess it's easy to implement, especially if you can just cherry-pick some patches from a fork)*

If so, maybe just keep this issue open and tag it as "help wanted" – though you don't really need help, dunno… what prevents you from implementing this? I guess you just need to say you accept PRs on this topic and it will be implemented.
I see nothing stopping you from that – and you can still achieve feature parity or so, whatever… it's really just a minor feature. :thinking:

* Yes I see and know this has been re-written in Go now, but maybe this still helps or so – dunno. At least it demonstrates that it is easy to implement.

Hijacking this issue a bit here in case any of the participants would like to check out and give feedback on my prototype at https://gitlab.com/bkhl/etui

It's meant as an alternative to Toolbox for when you want a more strictly contained development container.

@bkhl Thanks! Bookmarked.