flatcar Linux is a minimal Linux distribution tailored to running container workloads. It ships with docker as well as containerd (alone).

The update process if fully automated including reboots. These happen at random times during the night so there should not be a problem like two nodes rebooting and disturbing K8s.

For the time being we check the stable releases channel of flatcar linux manually and update if there is a new release:

update_engine_client -update
update_engine_client -status
# if CURRENT_OP=UPDATE_STATUS_UPDATED_NEED_REBOOT
sudo reboot; exit

Upload a version of flatcar_production_vmware_ova_image.vmdk to one of the VM hosts. Instead of current it might be better to upload a previous stable release and check that updating works. For example 3033.2.0 Remove any disks. Add the flatcar_production_vmware_ova_image.vmdk as the first disk (system, sda) and add a second thin provisionen 230 GB disk (container ephemeral, sdb).

You will find a flatcar_production_vmware_ova_image.vmdk.bz2. Use the filter box. Download that and unpack it. It is still in a special format not immedeiately usable by a ESXi host. After uploading to one VM host you have to copy the VMDK using vCenter to another host/data storage. The resulting copy is usable.

You can also copy the base vmdk of a snapshot of an already existing machine. Using the snapshot technique this is never touched. Note however that this of course does not get any updates so it might be really old and missing ignition configuration options found in the current docs. You can run an update in the running VM as the first task.

Per VM settings like networking can be passed to the VM using VMWare's guestinfo.ignition.* properties. These settings can be found in vCenter edit -> the VM Options -> Advanced -> Edit Configuration...

variant: flatcar
version: 1.1.0
passwd:
  users:
  - name: core
    ssh_authorized_keys:
      - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFBjtMTXcKtVlorGkFwpCs1XmHakFwk3ggShgmHZyJaAhjR81bAFFZVGURBIXUnmi4DpcAV1GaNDNdw0DJvPqRo= Omar Yubikey 5 nano
      - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAe26li9Rzcf+iBRftzE3WDddlDI8+HpCm+GFdep5tcWC6Q2Hzbi1GSqFTBSvpKAJRbyq49OkA3gd16ugmzImDU= Omar Yubikey 5 NFC
kernel_arguments:
  should_not_exist:
  - flatcar.autologin
storage:
  filesystems:
    - device: /dev/disk/by-label/OEM
      format: btrfs
      label: OEM
      path: /usr/share/oem
    - device: /dev/sdb
      format: xfs
      wipe_filesystem: true
      label: cont-ephemer
      with_mount_unit: true
      path: /var/lib/rancher
  files:
    - path: /usr/share/oem
      mode: 0644
      contents:
        inline: |
          set oem_id="vmware" # or something else like
          set linux_append=""
    # change IP and MAC addresses for other nodes!
    - path: /etc/systemd/network/10-ens1.network
      contents:
        inline: |     
          [Match]
          MACAddress=<MAC address from vCenter kubernetes_public>
          
          [Network]
          Address=10.3.6.xx/24
          Gateway=10.3.6.1
          DNS=192.168.1.250
          Domains=servants.priv
          NTP=192.168.1.250
    - path: /etc/systemd/network/10-ens2.network
      contents:
        inline: |         
          [Match]
          MACAddress=MAC address from vCenter kubernetes_internal>
          
          [Network]
          Address=10.6.16.xx/24
    - path: /etc/flatcar/update.conf
      content:
        inline: |
          REBOOT_STRATEGY="reboot"
          LOCKSMITHD_REBOOT_WINDOW_START="Thu 04:00"
          LOCKSMITHD_REBOOT_WINDOW_LENGTH="1h"
systemd:
  units:
    # Ensure docker starts automatically instead of being socket-activated
    - name: docker.socket
      enabled: false
    - name: update-engine.service
      enabled: true
    - name: locksmithd.service
      enabled: true
    - name: systemd-timesyncd.service
      enabled: true
    - name: docker.service
      enabled: false

This confioguration specified as YAML file has to be transpiledbutane:

cat acdh-clusterx.yaml | podman run --rm -i quay.io/coreos/butane:latest | gzip -c9 | base64 -w0 > acdh-clusterx.json+gz+base64.txt

Note: After configuring the guestinfo parameters, it is probably best to create a snapshot in ESXi (or vCenter, name it initial or Leer) and then boot. Guestinfo parameters are part of the snapshot! This configruation usually runs only once in an empty, unprovisioned VM. Like this we can easily go back to the initial empty system drive that is then set up using the guestinfo.ignition.config.data. When you start from scratch change the data after reverting to the snapshot. guestinfo.ignition.config.data is part of a snapshot as are the “UEFI” settings like boot order. If these are wrong you have to change them again.

Provisioning can be forced to (re)run using a boot parameter or a magic file touch /boot/flatcar/first_boot. Note: with the above settings that is destructive! The container ephemeral volume will be recreated.

The system boots with autologin for the first time. Reboot the system once until it only shows the correct IP addresses and a login prompt (there is no valid password for core).

This follows the Guide for flatcar on libvirt

mkdir -p /var/lib/libvirt/images/flatcar-linux
cd /var/lib/libvirt/images/flatcar-linux
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2{,.sig}
gpg --verify flatcar_production_qemu_image.img.bz2.sig
bunzip2 flatcar_production_qemu_image.img.bz2

Now create a qcow2 image snapshot using the command below:

cd /var/lib/libvirt/images/flatcar-linux
qemu-img create -f qcow2 -F qcow2 -b flatcar_production_qemu_image.img flatcar-linux1.qcow2
chown libvirt-qemu:kvm flatcar_production_qemu_image.img flatcar-linux1.qcow2