ignition.first parameter exists post deployment

[lethedata]

Bug Description

After Simplified Provisioner is provisioned with Ignition, the first boot contains the ignition.firstboot parameter causing broken tree mounts.

To Reproduce

1. Create butane file that adds a user with password (password_hash = pass)

variant: fiot
version: 1.0.0
passwd:
  users:
    - name: user
      password_hash: $y$j9T$xwWR4bvlM.oju.xHz5hYl0$z.vuMRdMTx.rLSw8LETGVS1MX0uHK0eUZ1j7J5dPIr.
      groups:
        - wheel

2. Convert to ignition
3. Host ignition file on accessible httpd server
4. Boot Simplified Provisioner

virt-install   --name "fedora-iot-device" \
               --os-variant "fedora40" \
               --boot uefi,loader.secure=false \
               --vcpus 1 --memory 3072 \
               --network=bridge=virbr0 \
               --disk pool=default,size=30 \
               --cdrom Fedora-IoT-provisioner-40.*.iso

5. Edit "Install Fedora 40" boot parameter appending ignition parameters to linux

coreos.inst.append=ignition.config.url=http://192.168.122.1/fiot.ign coreos.inst.append=rd.neednet=1

6. Boot system
7. Wait for provisioning to complete and system reboot
8. Login to the system - No home folder exists

Expected behavior

System boots without ignition.first after configuration allowing configured tree to mount with user's home folder existing.

Screenshots

localhost login: user 
Password: 
Last login: Wed Jun  5 18:18:43 on ttyS0
 -- user: /var/home/user: change directory failed: No such file or directory
Logging in with home = "/".
[user@localhost /]$ ls -lR / 2>/dev/null | grep var/home/user
[user@localhost /]$ ls -lR /sysroot/ 2>/dev/null | grep var/home/user
/sysroot/ostree/deploy/fedora/deploy/5bcc01335c19de1117841d2334062196fc40369bcbe6f71055ac3a2ddaa013f3.0/var/home/user:
[user@localhost /]$

OS version:

State: idle
BootedDeployment:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 40.20240422.3 (2024-04-22T19:43:59Z)
                   Commit: 5bcc01335c19de1117841d2334062196fc40369bcbe6f71055ac3a2ddaa013f3
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC

Additional context

• Running rpm-ostree kargs --delete-if-present='$ignition_firstboot' only works if ignition.firstboot is removed during the first boot. If ran after first boot, proper trees do not mount.

Work Around

1. Create butane file that adds a user with password (password_hash = pass)
2. Convert to ignition
3. Host ignition file on accessible httpd server
4. Boot Simplified Provisioner
5. Edit "Install Fedora 40" boot parameter appending ignition parameters to linux
6. Boot system
7. Wait for provisioning to complete and system reboot
8. Edit boot parameter removing ignition_firstboot during boot
9. login
10. run rpm-ostree kargs --delete-if-present='$ignition_firstboot'
11. reboot

[runcom]

$ignition_firstboot is a variable - and it's just empty on subsequent reboots, check /proc/cmdline - I can't see how this is gonna mess up mounts - if that variable is populated you'll get an error straight away on the second boot as ignition tries to run again so I think it's not that.
can you provide /proc/cmdline meanwhile and we'll get this bug scheduled asap
thanks for the report!

[lethedata]

/proc/cmdline

BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-9bcd9212246ae456c144c97977daf59b706493a1c791b52edb483dd838626bf2/vmlinuz-6.11.4-301.fc41.x86_64 luks.uuid=8ba00bc1-ef7a-481e-93f2-0c6d22625784 modprobe.blacklist=vc4 rw coreos.no_persist_ip ignition.platform.id=metal ignition.firstboot systemd.firstboot=off systemd.condition-first-boot=true root=LABEL=root ostree=/ostree/boot.1/fedora/9bcd9212246ae456c144c97977daf59b706493a1c791b52edb483dd838626bf2/0 ignition.config.url=http://192.168.199.1:9080/fiot.ign rd.neednet=1

[lethedata]

Replicated setup using Fedora-IoT-provisioner-41-20241027.0.x86_64.iso with same result. Looks like directories are not created at all, not just user home folders.

Butane Example:

variant: fiot
version: 1.0.0
passwd:
  users:
    - name: user
      groups:
        - wheel
      password_hash: ...
      ssh_authorized_keys:
        - ssh-ed25519 ...
storage:
  directories:
    - path: /var/srv/containers/config
  files:
# podman --userns=auto ns range enabling rootful podman to run containers in unique namespaces.
## Start Num: Unused UID range after systemd container UID ranges. Larger than before range.
## Range Num: Double default user range.
    - path: /etc/subuid
      append:
        - inline: |
            containers:1879048192:131072
    - path: /etc/subgid
      append:
        - inline: |
            containers:1879048192:131072

/var/srv/containers/config folder is not created and ssh_authorized_keys not created due to missing home folder.

[miabbott]

Looks like directories are not created at all, not just user home folders.

This could be related to #70

[runcom]

yes, David has been working on this and we have already hopefully came down to the problem :) cc @djach7 to comment with the findings

[djach7]

As Antonio mentioned I've been looking into this in relation to #70. I think it comes down to both the ssh-key-dir rpm and the ignition-edge rpm missing from the Fedora IoT 41 raw image, thus resulting in ssh keys not transferring and directories not being created.

I'm currently working to get these rpms added back and to see if they're the only issue.

[djach7]

Submitted a pr to add the packages necessary to fix this here: https://pagure.io/fedora-iot/ostree/pull-request/66

I was able to login with ssh successfully and both a home directory and a .ssh directory were created, so I believe this is sorted out. Going to wait to close until the pr is merged.

[djach7]

Closing now that PR is merged.

[djach7]

Just a clarifying comment here, I don't think the ignition.firstboot parameter was ever the real issue. The lack of directories (i.e. home and .ssh) seems to have been related to the missing packages above and was solved by adding the packages back to the commit.

[nullr0ute]

Can we also have a PR for the F-41 stable release please?

[miabbott]

Let's keep this open until we have a working compose with the fix.