We use CentOS 8/9 as base distribution for running Rancher UI. Traditionally Rancher UI came in form of (docker) container images which conained the kubernetes cluster Rancher UI runs on. This is now phased out in favor of a solution that launches a small kubernetes distribution directly instead of packaging it in a container.

sudo modprobe ip_tables && echo 'ip_tables' | sudo tee -a /etc/modules-load.d/rancher.conf
sudo firewall-cmd --zone=public --permanent --add-service=http
sudo firewall-cmd --zone=public --permanent --add-service=https
sudo firewall-cmd --zone=public --add-service=http
sudo firewall-cmd --zone=public --add-service=https

We use certbot because this mimics the professional setup better which does not use letsencrypt but the acme api

sudo certbot certonly --standalone --preferred-challenges http -d rancher.machine-deck.jeffries-tube.at
sudo nano /etc/letsencrypt/renewal/rancher.machine-deck.jeffries-tube.at.conf
# Add
#[renewalparams]
#[...]
#pre_hook = systemctl stop rancher
#post_hook = systemctl restart rancher

Note: podman as root can not handle IPv6. A reverse proxy can handle IPv6 and then forward to IPv4.

sudo podman create --name rancher-2-a-b \
 -v /etc/letsencrypt/live/rancher.machine-deck.jeffries-tube.at/fullchain.pem:/etc/rancher/ssl/cert.pem \
 -v /etc/letsencrypt/live/rancher.machine-deck.jeffries-tube.at/privkey.pem:/etc/rancher/ssl/key.pem \
 -e HTTP_PROXY="http://192.168.1.250:8080" \
 -e HTTPS_PROXY="http://192.168.1.250:8080" \
 -e NO_PROXY="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,.svc,.cluster.local,192.168.1.0/24" \
 -e no_proxy="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,.svc,.cluster.local,192.168.1.0/24" \
 --privileged \
 -p 80:80 -p 443:443 --log-driver journald \
 rancher/rancher:v2.a.b --no-cacerts

Note: specify the latest version as a tag. Better not use latest because of potentioally unwanted updates.

Note: Rancher is now capable of running an ACME Let's encrypt client. So we use this and direct pass through TLS to have an official certifictate.

/etc/systemd/system/rancher.service

[Unit]
Description=Rancher Management Server
 
[Service]
Restart=on-failure
ExecStart=/usr/bin/podman start -a rancher-x-y-z
ExecStop=/usr/bin/podman stop -t 10 rancher-x-y-z
 
[Install]
WantedBy=multi-user.target

Note: There is a race condition in podman/cni port mapping setup . This is not fixed in CentOS 8.1.1911 Therefore this container waits for heketi before starting.

sudo systemctl enable rancher
sudo systemctl start rancher

Note: sudo podman stop rancher will result in an immediate restart. Use sudo systemctl stop rancher instead

Check the certbot timer

# watch
journalctl -f
# in another terminal
sudo certbot renew --standalone --dry-run
# This should stop and restart rancher
sudo systemctl enable --now certbot-renew.timer

See the official docs

Notes:

• change docker to sudo podman
• Set the environment the same as for the first install (HTTP_PROXY, HTTPS_PROXY, NO_PROXY)
• Workflow is the same as for Backup/Restore. Just update the container name (during restore) to rancher-2-5-9 (version) and image tag rancher/rancher:v2.5.9
• Rancher Releases can be found here

Make sure there is enough free space on the root volume. If you run out of space during this procedure the errors will be confusing and hard to fix. You can use prune commands to get rid of unused container parts. These commands delete everything but the parts of running containers.

sudo podman system prune --all
sudo podman volume prune
# also consider left over package management stuff
sudo dnf clean all

You should always do a backup before upgrading.

sudo systemctl stop rancher

In a rancher-backup directory in your home directory:

sudo podman run --rm --volumes-from rancher-2-y-z \
  -v $PWD:/backup:z busybox tar pzcvf /backup/rancher-data-backup-v2.y.z-JJJJ-MM-dd.tar.gz /var/lib/rancher

Probably have a look at the TAR file and get the size. Make sure you have enough free space on root. E.g.:

gzip -tl rancher-data-backup-v2.4.11-2021-02-08.tar.gz
#         compressed        uncompressed  ratio uncompressed_name
#         1352289033          3499342848  61.4% rancher-data-backup-v2.4.11-2021-02-08.tar
df -B1
# Dateisystem                         1B-Blöcke     Benutzt     Verfügbar  Verw% Eingehängt auf
# /dev/mapper/cl_acdh--rancher-root   17609785344   10596220928 7013564416   61% /
# 7013564416 > 3499342848

If you (must) use a proxy server it is important to configure the proxy server usage for the setup for every new rancher container. If this is not done correctly you will see hard to debug error messages like 403 on when contacting internal services on the nodes e.g. cattle service “reporting” 403 on 10.3.6.55:6443.

sudo podman create --name rancher-2-a-b \
 -v /etc/letsencrypt/live/rancher.machine-deck.jeffries-tube.at/fullchain.pem:/etc/rancher/ssl/cert.pem \
 -v /etc/letsencrypt/live/rancher.machine-deck.jeffries-tube.at/privkey.pem:/etc/rancher/ssl/key.pem \
 -e HTTP_PROXY="http://192.168.1.250:8080" \
 -e HTTPS_PROXY="http://192.168.1.250:8080" \
 -e NO_PROXY="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,.svc,.cluster.local,192.168.1.0/24" \
 -e no_proxy="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,.svc,.cluster.local,192.168.1.0/24" \
 --privileged \
 -p 80:80 -p 443:443 --log-driver journald \
 rancher/rancher:v2.a.b --no-cacerts

Now restore the backup of the entire configuration into the new container

sudo podman run --rm --volumes-from rancher-2-a-b -v $PWD:/backup \
  busybox sh -c "rm /var/lib/rancher/* -rf  && \
    tar pzxvf /backup/rancher-data-backup-v2.y.z-JJJJ-MM-dd.tar.gz"

Change /etc/systemd/system/rancher.service: replace rancher-2-y-z with rancher-2-a-b

sudo systemctl daemon-reload
sudo systemctl start rancher