Bare Metal Linux ISO (not PXE) Redux

Like many others before me, I'd like to use Packer to create Linux .iso images enabling the installation of my Packer-built Linux virtual environments on bare metal machines. As the three past requests about creating images for bare metal machines were closed without resolution, I'd like to re-open a feature request wherein we can:

1. Summarize and hopefully complete the solution fragments that have been presented in past issues
2. Discuss how and whether it makes sense to:
   a. integrate into an official Packer extension (builder, post-processor, provisioner)
   b. leave as an informal solution for users to implement in their Packer templates
   c. document these instructions in the official Packer documentation

Use Case Discussion/Limits

There are still many use cases for bare metal Linux installation, and here I am specifically targeting the use case of a (technically-proficient) development team that uses a Packer-built Linux virtual environment to develop a product with a bare metal deployment requirement, i.e. that the product be deployed as an .iso that can be installed, unattended, by (technically-non-proficient) customers onto arbitrary physical amd64 machines. Many past discussions focused on PXE-boot solutions, and although most of the underlying technology and mechanics are applicable here, please note that the PXE-boot use case is different enough to derail this issue like the rest, so please help me keep the discussion limited to creating installable iso media with Packer. Also, note that I'm specifically interested in Linux targets here; I know Packer supports building more OSes than Linux ones, but I don't want this discussion to be derailed by trying to find the perfect and general solution.

My use case has been building primarily Debian-based systems, and we're currently building virtualbox VM and docker containers with the same (mostly ansible) provisioning. Now add to this use case the need to create an iso that can be installed on bare metal that will be identical to the corresponding virtualbox VM.

Proposed Solution

For our needs, we believe "the Squashfs approach," originally hinted by @vtolstov, will work best as we're building primarily Debian machines. The Debian Live Project started including the Calamares Installer, which is a distro-independent linux installer infrastructure that enables the inclusion of an arbitrary squashed filesystem (squashfs) into the iso media, to boot into this squashfs "live linux" in non-persistent RAM, and notably, to conditionally copy the same squashfs into a device on the target system. All of this can be greatly customized, and debconf can be preseeded during installation just like a normal d-i install.

The Debian-Live Documentation describes the operation of "a special udeb called live-installer" in which "Installation will proceed in an identical fashion to the "normal" installation described above, but at the actual package installation stage, instead of using debootstrap to fetch and install packages, the live filesystem image is copied to the target."

From the above I hope it's obvious that this would provide an ideal platform for creating a Debian-based distributable appliance as a product which is identical to virtual images previously created by Packer, but as a slight aside, it would also seem to me that this approach could install any Linux.

It would work like this:

1. A Packer VM builder (in our case, virtualbox-iso) creates a normal virtual machine per the packer.json and our various provisioners (in our case, mostly ansible)
2. NEW provisioner steps are appended to the end, which will:
   • shell provisioner runs a script that:
     • install the squashfs tools
     • creates a squashfs file representing the machine's entire root partition, and store this in a known location in the VM
     • copies the kernel from the /boot to same location in VM
     • copies the initrd from the /boot partition to same location in VM
   • file provisioners then download the kernel, initrd, and root squashfs to the Packer host where they can be available for the subsequent post-processor step
3. A NEW final Shell post-processor step will then run the live-build commands required to build the iso, per the debian live project documentation. The output from this step will also be placed in the builds/ dir where it is automatically archived by the jenkins job.

To further illustrate, 2. above is implemented pretty much exactly as what @vtolstov originally proposed:

    {
      "type": "shell",
      "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
      "script": "scripts/squash-file-system.sh",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/vmlinuz"],
      "destination": "builds/squashfs/debian.vmlinuz",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/initrd.img"],
      "destination": "builds/squashfs/debian.initrd",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/filesystem.squashfs"],
      "destination": "builds/squashfs/root.squashfs",
      "only": ["virtualbox-iso"]
    }

where scripts/squash-file-system.sh is:

#!/bin/sh -ex

apt-get -y install squashfs-tools

mkdir -p /mnt/squashfs /squashfs
mount -o bind / /mnt/squashfs

mksquashfs /mnt/squashfs /squashfs/filesystem.squashfs -comp gzip -no-exports -xattrs -noappend -no-recovery -e /mnt/squashfs/squashfs/filesystem.squashfs
find /boot -name 'vmlinuz-*' -type f -exec cp {} /squashfs/vmlinuz \;
find /boot -name 'init*' -type f -exec cp {} /squashfs/initrd.img \;

We can soon post what we're doing for step 3, which is the Debian live-build step, but for now this is all we got.

Is anyone else using the new Debian Live (or another Calamares Installer) project as a Packer post-processor step to make isos?

Do you think this approach could / should be adopted as an official "Linux ISO" Packer post-processor?

Is there some way for the post-processor to ensure provisioning step 2 above is done, so that its inputs (root squashfs, kernel, and initrd) are made available for the post-processor? Or is the only way to create a new specialized builder that just does it all (builds and provisions the VM, generates squashfs, kernel, and initrd, AND builds the installer iso.)?

In this approach, the final live-build post-processor step which builds the iso, can't happen until after the VM is built. So we are trading time (it's not very time efficient this way) for versatility (can use this approach for ANY Packer machine that supports squashfs, regardless of what provisioners were used).

An alternative approach, which I was considering before discovering Debian's Live Installer/Calamares, would enable the bare metal iso to be built concurrently with the virtual machines. This uses the standard debian preseeded installer approach, modifying the initrd of an existing iso, embedding your preseed file, and the provisioning scripts referenced by the preseed file, inside it. In this approach, you just use the same provisioning scripts for the iso build process that you use for the packer template building the VM. But for this there's much more orchestration and management of scripts and processes than the former approach. So we choose to do the former for now.

Sorry for the dissertation, but it's not a simple topic, and I wanted to take care to thoroughly capture all the aspects of this specific problem. Thanks for your time!