Users and ssh keys are painful

[cgwalters]

We don't have one single answer for users and ssh keys in our default base image today.

This all started with https://github.com/CentOS/centos-bootc/blob/main/docs/cloud-agents.md

However, we still have the opportunity to rethink things here.

Short term possibilities

Default to cloud-init

We could choose to add cloud-init into our generic base image (moving it from the layered one); we'd need to change the Anaconda kickstarts to disable it. This is effectively what we did long ago with Atomic Host. It wouldn't work for every cloud or scenario, but it would level up the "baseline" such that we'd get ssh keys in e.g. AWS from the infrastructure by default. cloud-init is kind of a monstrosity (https://coreos.github.io/ignition/rationale/ touches on some problems) but...it's a known monstrosity 😄

Add afterburn

There's also https://github.com/coreos/afterburn/ which is a very targeted system designed to basically just do the bare minimum, and not itself do all the tons of extra things that cloud-init does like firstboot scripts, network reconfiguration, etc.

We'd have to do some nontrivial legwork though because afterburn is designed to pair with ignition, and Ignition really wants to discover the platform via an ignition.platform.id field, and we'd need to do a bit of work to change e.g. bootc-image-builder to do that.

(Inherently also this wouldn't work with the alongside replacement path in the cloud unless we also taught that how to do detection)

Another path is hacking afterburn to add some "detect the platform" code, I think if we just did that for the major public clouds we'd be fine.

Longer term

I think it would very much make sense to try to support "system extensions" for cloud agents - it would really work well for a lot of cases. However the systemd-sysext style thing only really works well with statically linked things, which things like cloud-init and the vmware agent etc. are definitely not.

[cgwalters]

I would say though if we go down the path of having an even bigger default base image, I would really like to reopen the path of shipping a much smaller tiny base image to start from too and also it's going to increase the desire for custom base image building as a more first class operation.

[cgwalters]

containers/bootc#296

[mvo5]

Thank you for starting this, I wanted to talk a bit about this topic :) I saw containers/bootc#296 and it looks like an interesting approach ,it has many upsides (as described). I would love to learn more about the vision for the future though. The description specifically talks about cloud environments and "bootc install --alongside" in a cloud environment. It seems to me for the cloud environments we have bootc-image-builder which can produce "native" images for ami/qcow2 (and we can add more easily). So I would assume that most users who want to use bootc in the cloud would do that via the "bib" route? (just to be clear, of course fine to have the option in bootc just double checking that our assumption about use-cases match :)

And I also wanted to ask about the wider user topic. Let me state my assumptions first:

1. root:root is the only thing we can assume is present in all bootc containers
2. any system user uid (1-999) can appear in random order and conflicts between layers may exist
3. regular users (uid >= 1000) shouldn't be part of any layers (and /etc/passwd and friends are owned by the user and /usr/etc/passwd is part of the layers and must not contain uids >= 1000)

If (1) is true then bib could support our root customizations (i.e. we would not need extra code).
If (3) is true (and I guess that is a huge if) we could support normal user customizations because there will be no conflicts on upgrades or duplication of uids. I tried to read up on the ostree history here (and I have my own history with this problem) but there is a lot and it's complicated so I'm sure I'm missing things. Happy to have a high bandwidth discussion about this.
if (3) is not true today, would it make sense to maybe add it as a constrain? i.e. lower layers cannot have uid>=1000 only the top layer can (toplayer so that users who want to derive containers via "FROM source-container\n RUN adduser my-user" are supported).

I'm sorry if this was discussed already and I missed it, feel free to ignore my questions if this is the case and I will do more own research :)

P.S. (2) looks pretty hard to solve, would love to learn the vision here too but it's most likely something we can ignore on bib by restricting the allowed range of UIDs for bootc (but interested for my own curiosity).

[edit: fixed some typos, thanks to achilleas-k for spotting them]

[cgwalters]

It seems to me for the cloud environments we have bootc-image-builder which can produce "native" images for ami/qcow2 (and we can add more easily). So I would assume that most users who want to use bootc in the cloud would do that via the "bib" route? (just to be clear, of course fine to have the option in bootc just double checking that our assumption about use-cases match :)

This is just an instinct...a "gut feeling" without hard numbers. But I think for the cloud, there will be some people who absolutely want disk images (in particular because it gets you a fast boot directly into your desired state). There will be others for whom if we have a sufficiently slick "take over existing AMI" flow (with something like bootc install alongside) they'd be happy with that.

The thing is that once you start making AMIs (or equivalent) you need to start considering garbage collection, versioning, etc. for them (which bib doesn't do today, but of course there are other tools). Whereas with the "takeover" path, the only thing you as the user need to think or care about in general are your container images.

[cgwalters]

Non-root users and groups are indeed a fantastically complex topic; previously we basically had "system users" owned by the image build, but that is no longer the case in general...I think the direction we need to go in general is towards entirely using systemd-users; see coreos/rpm-ostree#4680 and the earlier links.

However, yeah the problem with that is when people create content in the containers that have content owned by a potentially dynamic UID; the classic simple example is a plain RUN useradd someuser which would create a user with uid 1000 and say /home/someuser (owed by 1000). But then you later change your image to also have a RUN useradd otheruser before that...well, that now becomes 1000 and what happens when you upgrade?

See e.g. containers/podman#21251 (comment)

This one is going to need a lot more docs for sure.

[mvo5]

Thank you for your thoughts here, I understand the rational for the bootc install --alongside now a lot better.

As for the users (I'm trying to write a more succinct version of my earlier comment based on the most recent offline discussion):
I wonder if constraining the problem may help us solve it more easily. We could (at least as a starting point) say:

1. /etc/passwd is owned by the users and must only contain UIDs within the "normal" user range (>= 1000?)
2. /usr/etc/passwd is owned by the image, the UID range is system users only (0,999) the ordering of UIDs must never change and UIDs can never be removed. That is up to the image creator to ensure. I.e. if a derived image contains "FROM basecontainer\n RUN dnf install polkitd" and the polkitd package creates a UID in subsequent images this UID must be present with the same number (and we should have CI to enforce that).
3. If users want to create images with pre-build normal users that is a problem as it would violate (1) and (2). So either
   a) disallow it because users can always use "anaconda" or "bib" to install the image and setup their users there
   b) allow it but have a reserved range (e.g. UID>=100000) and have this range standardized if possible, put it into /usr/etc/passwd (CI and bootc upgrade would detect it and warn/error).
   c) try to be smart about it (track local UIDs and compare with the images /usr/etc/passwd and on conflicts rewrite/shift UIDs in the image (which sounds like a lot of work)

As for your example: about having RUN useradd someuser and then change the image to RUN useradd otheruser\nRUN useradd someuser (which is clearly incompatible with upgrades and IMHO an error on the image creation side). I wonder if we can:
a) provide default tooling that detects on image upload time to quay.io that the UID->name mapping changed and warn/fail
b) on bootc upgrade detect that the UID->name mapping changed and fail with an appropriate message (and hint what the user could do)

With the above constraints (1), (2), (3b) and detection of UID->name mapping changes at upgrade time, we should cover the various scenarios(?).

[mvo5]

[mvo5]

I see that I accidentally created a new thread, I will remove it and move the discussion back into the previous discussion