alwaysInBeta Stable software is for the weak

Trying to work with bootc - moving from RH to Arch

2025-03-27

Background

As discussed in my previous post, I’m trying to see if I can use bootc to build container images of various Linux distros into bootable non-container machines. If you’re trying to follow along at home, please go back and read that post, as it discusses the big picture and prereqs, as well as following the simple happy path to make sure everything actually works before sprinting away from the beaten path.

Now it’s fine to use CentOS Stream images to figure out how to build images at all, but I have no desire to actually use them for anything. The only existing material I can find about bootc on anything other than Red Hat OSs is https://github.com/frap129/arch-bootc , which builds an Arch Linux image for bootc. I will therefore attempt to build and boot that, and we’ll see where things go from there.

Getting started

On my trusty openSUSE MicroOS box, I will (briefly drop into a distrobox container that has git and) clone the repo:

git clone --depth 1 https://github.com/frap129/arch-bootc.git
cd arch-bootc

and then poke around a bit.

The first problem I see is that this is building a couple packages on the host before building the container image (which installs these packages). This will be a problem, because this host isn’t running Arch Linux.

A few options present themselves:

  1. Build in an Arch container in distrobox
  2. Build in an Arch container not in distrobox
  3. Don’t build those packages

Option 1 is the obvious option, but I dislike the amount of manual work involved. I’m much more comfortable with the second option; I already have scripts on hand from attempting to build an automated AUR build pipeline in containers, which can probably be easily adapted to work here. That would benefit from being amenable to scripting, and steps that are scripted are easier to share with other people (I could even make a PR to try and convince frap129 to adopt my approach, so anyone could use their build.sh with no dependencies but podman.).

However…

On closer examination, the packages in question are bootc and bootupd. In an ideal system, we do want those because AIUI they help managing updates on the running system, but just to get something that works? The easiest option is to just not include them, and see if it does in fact break the build. In fairness, it might; I’m a little fuzzy on exactly where said tools are used.

Patching out packages and building

Starting from the checkout of arch-bootc.git,

$EDITOR Containerfile

find the lines (at this writing, lines 39-41)

# Install bootc and bootupd
COPY pkgbuilds /pkgbuilds
RUN pacman -U --noconfirm /pkgbuilds/bootupd/*.pkg.tar.zst /pkgbuilds/bootc/*.pkg.tar.zst && rm -rf /pkgbuilds

and just delete them.

Now instead of bothering with build.sh, just directly build:

sudo podman build . -t arch-bootc --net=host --cap-add sys_admin --cap-add mknod

and…

---snip---
[1/2] STEP 11/11: RUN pacstrap -c -P /mnt     base     btrfs-progs     linux     linux-firmware     linux-firmware-whence     intel-ucode     amd-ucode     grub     dracut     ostree     podman
==> Creating install root at /mnt
mount: /mnt/proc: cannot mount proc read-only.
       dmesg(1) may have more information after failed mount system call.
==> ERROR: failed to setup chroot /mnt
Error: building at STEP "RUN pacstrap -c -P /mnt     base     btrfs-progs     linux     linux-firmware     linux-firmware-whence     intel-ucode     amd-ucode     grub     dracut     ostree     podman": while running runtime: exit status 1

Huh.

I’d previously looked at https://gitlab.com/fedora/bootc/base-images to see how they build images, and I notice that their suggested build command is

podman build --security-opt=label=disable --cap-add=all \
  --device /dev/fuse -t localhost/fedora-bootc .

Now that’s apparently because they’re doing nested podmans, which is super cool, but that doesn’t mean I can’t rip off bits of it. (Note that this may not all be needed; I basically did a union of all options.)

sudo podman build . -t arch-bootc --net=host --cap-add all --security-opt=label=disable --device /dev/fuse

which gets much further before…

[2/2] STEP 6/9: RUN sed -i     -e 's|^#\(DBPath\s*=\s*\).*|\1/usr/lib/pacman|g'     -e 's|^#\(IgnoreGroup\s*=\s*\).*|\1modified|g'     "/etc/pacman.conf" &&     mv "/var/lib/
pacman" "/usr/lib/" &&     rm /var/cache/pacman/pkg/* &&
/bin/sh: -c: line 2: syntax error: unexpected end of file
Error: building at STEP "RUN sed -i     -e 's|^#\(DBPath\s*=\s*\).*|\1/usr/lib/pacman|g'     -e 's|^#\(IgnoreGroup\s*=\s*\).*|\1modified|g'     "/etc/pacman.conf" &&     mv "
/var/lib/pacman" "/usr/lib/" &&     rm /var/cache/pacman/pkg/* &&": while running runtime: exit status 2

I had to stare at this for a good long while before realizing that that was a sh command that just ended with &&. Open Containerfile in your editor, and scroll down to this part (this time with line numbers):

     47 # Move pacman db to /usr since /var will be a mount
     48 RUN sed -i \
     49     -e 's|^#\(DBPath\s*=\s*\).*|\1/usr/lib/pacman|g' \
     50     -e 's|^#\(IgnoreGroup\s*=\s*\).*|\1modified|g' \
     51     "/etc/pacman.conf" && \
     52     mv "/var/lib/pacman" "/usr/lib/" && \
     53     rm /var/cache/pacman/pkg/* &&
     54     find "/etc" -type s -exec rm {} \;

See line 53? It’s missing a trailing backslash. Just add a backslash:

     53     rm /var/cache/pacman/pkg/* && \

save, exit, and

sudo podman build . -t arch-bootc --net=host --cap-add all --security-opt=label=disable --device /dev/fuse

to hit an all-new error (this actually is progress):

[2/2] STEP 6/8: RUN sed -i     -e 's|^#\(DBPath\s*=\s*\).*|\1/usr/lib/pacman|g'     -e 's|^#\(IgnoreGroup\s*=\s*\).*|\1modified|g'     "/etc/pacman.conf" &&     mv "/var/lib/pacman" "/usr/lib/" &&     rm /var/cache/pacman/pkg/* &&     find "/etc" -type s -exec rm {} \;
rm: cannot remove '/var/cache/pacman/pkg/*': No such file or directory

which is weird? but whatever, just open Containerfile in your editor again, and add -f:

     53     rm -f /var/cache/pacman/pkg/* && \

, save, exit, build again:

sudo podman build . -t arch-bootc --net=host --cap-add all --security-opt=label=disable --device /dev/fuse

and BOOM!

Successfully tagged localhost/arch-bootc:latest
dfccd7aafc03fb3069a041d33338d388d964ebc90824f78059fae587b63e6a93

Success!

Well. We successfully built a container image. Let’s see if it can be turned into a qcow2 that actually boots a VM.

Take the bootc-image-builder command from last time, but skip config.toml:

mkdir output
sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type qcow2 \
    --use-librepo=True \
    localhost/arch-bootc

(honestly, I didn’t check that it doesn’t support a user config, but I don’t feel like messing with that yet)

[-] Manifest generation step
Message: Generating manifest manifest-qcow2.json
2025/03/27 21:33:17 error: cannot build manifest: cannot get rootfs type for container: failed to run bootc install print-configuration: exit status 127, stderr:
Error: crun: executable file `bootc` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Ah. So I guess those packages were doing something important.

Buuuut, actually that just says “cannot get rootfs type”. What if I just give it --rootfs?

sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type qcow2 \
    --use-librepo=True \
    --rootfs ext4 \
    localhost/arch-bootc

=>

[/] Manifest generation step
Message: Generating manifest manifest-qcow2.json
2025/03/27 21:36:13 error: cannot build manifest: missing VERSION_ID in os-release

Cool. (Again, the fact that we are getting a different error is genuinely a good sign.)

Now the freedesktop spec says that VERSION_ID is

A lower-case string (mostly numeric, no spaces or other characters outside of 0–9, a–z, “.”, “_” and “-”) identifying the operating system version, excluding any OS name information or release code name, and suitable for processing by scripts or usage in generated filenames. This field is optional.

I tend to think that this constitutes a bug in bootc-image-builder (and I will probably report it later) but for now we can just give it a value to read; edit Containerfile and insert this line (I put it right before the final LABEL line that ends the file):

RUN echo 'VERSION_ID=rolling' >> /etc/os-release

then

sudo podman build . -t arch-bootc --net=host --cap-add all --security-opt=label=disable --device /dev/fuse
sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type qcow2 \
    --use-librepo=True \
    --rootfs ext4 \
    localhost/arch-bootc

=>

Message: Generating manifest manifest-qcow2.json
2025/03/27 22:47:43 error: cannot build manifest: missing PLATFORM_ID in os-release

Eugh. This one is extra fun because the spec doesn’t even mention that being a thing, and indeed Fedora has a proposal to drop it which AFAICT says it was a local thing they invented and don’t use anymore. So again, probably a bug I should file, but for now I’m going to see that Fedora images set it to PLATFORM_ID="platform:f43" and work around via

RUN echo 'PLATFORM_ID="platform:rolling"' >> /etc/os-release

Run podman build, run bootc-image-builder.

[-] Manifest generation step
Message: Generating manifest manifest-qcow2.json
2025/03/27 22:53:27 error: cannot build manifest: initializing dnf in 349dc08b74e9b50c9a5af43d1c28a49bafffeb1a622977babf01026e4ba57318 container failed: exit status 127
output:
Error: crun: executable file `dnf` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Okay, now we’re done. If bootc-image-builder hardcodes the package manager… I’m not a developer. I’m not patching it. The best I can do from here is file bugs. (Which I will, of course; I’m not a monster;])

(Come to think of it, one of the issues I read did have someone talking about a patched version that used pacman, but Arch wasn’t my final goal so this is a blocker sooner or later.)

Conclusions

It doesn’t work.

So let me try to summarize the problems I hit. Not all of these are actual bugs, and some of them may or not be bugs.

In arch-bootc:

In bootc-image-builder: