diff --git a/README.md b/README.md index b99461f..092ef11 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # Checkmk im Container – Docker (Debian) & Podman (RHEL) mit NGINX Proxy Manager +*[English version: README_en.md](README_en.md)* + Vollständige, produktionstaugliche Container-Installation von **Checkmk** mit **NGINX Proxy Manager** als HTTPS-Terminierung (Let's Encrypt *oder* selbstsigniert), inklusive Werkzeugen für **Backup**, **Restore** und die **Migration bestehender @@ -75,7 +77,8 @@ werden müssen. ``` . -├── README.md +├── README.md # deutsche Dokumentation +├── README_en.md # englische Fassung ├── docker-compose.yml # für Docker UND Podman ├── .env.example # Vorlage -> .env ├── install.sh # Installation Docker (Debian) / Podman (RHEL) @@ -576,16 +579,13 @@ Zugang Der Port der Adminoberfläche folgt `NPM_ADMIN_PORT` aus der `.env`; die Ausgabe zeigt immer den tatsächlich eingestellten Wert. -``` -``` - Anmeldung an Checkmk: Benutzer `cmkadmin`, Passwort aus `CMK_PASSWORD`. --- ## 10. HTTPS einrichten – Let's Encrypt oder selbstsigniert -### 9.1 NGINX Proxy Manager vorbereiten +### 10.1 NGINX Proxy Manager vorbereiten 1. `http://:81` aufrufen – bzw. den in `NPM_ADMIN_PORT` gesetzten Port (siehe [Kapitel 12](#12-adminoberfläche-auf-einen-anderen-port-legen)); @@ -594,7 +594,7 @@ Anmeldung an Checkmk: Benutzer `cmkadmin`, Passwort aus `CMK_PASSWORD`. 2. Erstanmeldung: **admin@example.com** / **changeme** 3. Sofort E-Mail-Adresse und Passwort ändern (wird beim ersten Login erzwungen) -### 9.2 Proxy-Host für Checkmk anlegen +### 10.2 Proxy-Host für Checkmk anlegen **Hosts → Proxy Hosts → Add Proxy Host** @@ -617,7 +617,7 @@ proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ``` -### 9.3 Variante A – Let's Encrypt (öffentlich erreichbarer Server) +### 10.3 Variante A – Let's Encrypt (öffentlich erreichbarer Server) Voraussetzungen: der DNS-Name zeigt öffentlich auf diesen Server und **Port 80 ist aus dem Internet erreichbar** (HTTP-01-Challenge). @@ -637,7 +637,7 @@ gesichert. Bei interner DNS-Zone ohne öffentlichen Zugang: **DNS-Challenge** verwenden (SSL → *Use a DNS Challenge* → Provider auswählen, API-Token hinterlegen). -### 9.4 Variante B – Selbstsigniertes Zertifikat (interne Umgebung) +### 10.4 Variante B – Selbstsigniertes Zertifikat (interne Umgebung) Erzeugen: @@ -943,7 +943,7 @@ Proxy. Port **81** nur aus dem Administrationsnetz freigeben. Das Skript ist gleichzeitig Kommandozeilenwerkzeug **und** ncurses-Oberfläche. -### 11.1 ncurses-Oberfläche (wie im Midnight Commander) +### 14.1 ncurses-Oberfläche (wie im Midnight Commander) ```bash sudo ./scripts/cmk-manage.sh gui @@ -963,7 +963,7 @@ sudo ./scripts/cmk-manage.sh gui │ manage Backups verwalten (Liste/Details/Löschen) │ │ sites Sites im Container verwalten │ │ image Checkmk-Image beschaffen/auswählen │ - │ browser Terminal-Browser (Proxy Manager lokal) │ + │ browser Terminal-Browser (Proxy Manager lokal) │ │ cert Selbstsigniertes Zertifikat erzeugen │ │ config Einstellungen (.env) - menügeführt │ │ shell Root-Shell im Checkmk-Container │ @@ -977,7 +977,7 @@ Sites für ein Backup), **Enter** bestätigt, **ESC/Cancel** geht zurück. Es wird `dialog` bevorzugt; ist nur `whiptail` vorhanden, wird dieses genutzt. Eine Dateiauswahl mit Verzeichnisnavigation ist eingebaut. -### 11.2 Kommandozeile +### 14.2 Kommandozeile ```bash ./scripts/cmk-manage.sh --help @@ -1152,7 +1152,7 @@ der Site-Name wird dann aus dem Archiv gelesen. Ziel: eine bestehende Checkmk-Installation auf Blech/VM in den Container übernehmen – **ohne die Agents anzufassen**. -### 14.1 Warum die Agents weiterlaufen +### 17.1 Warum die Agents weiterlaufen Ein `omd backup` enthält die komplette Site inklusive `etc/ssl` (site-eigene CA und Zertifikate der registrierten Agents), @@ -1169,7 +1169,7 @@ Voraussetzung ist deshalb, dass das Zielsystem die **IP des Quellservers übernimmt** (bzw. der DNS-Name darauf umgestellt wird) – das ist im geplanten Vorgehen ohnehin so. -### 14.2 Schritt 1 – Skript auf den Quellserver kopieren +### 17.2 Schritt 1 – Skript auf den Quellserver kopieren ```bash scp scripts/migrate_creator.sh root@alt-server:/root/ @@ -1179,7 +1179,7 @@ ssh root@alt-server 'chmod +x /root/migrate_creator.sh' Das Skript ist **eigenständig** – es braucht auf dem Quellserver nur `bash`, `tar` und `omd` (für die Oberfläche zusätzlich `dialog` oder `whiptail`). -### 14.3 Schritt 2 – Bundle auf dem Quellserver erzeugen +### 17.3 Schritt 2 – Bundle auf dem Quellserver erzeugen Vorhandene Sites anzeigen: @@ -1224,7 +1224,7 @@ Checkmk-Version, Quell-Hostname, **Quell-IP-Adressen**, Quell-Betriebssystem und SHA256-Prüfsumme sowie ein Verzeichnis `siteinfo/` mit `omd config show`, `omd status` und der IP-Konfiguration des Altsystems zur späteren Kontrolle. -### 14.4 Schritt 3 – Zielsystem auf die richtige Version bringen +### 17.4 Schritt 3 – Zielsystem auf die richtige Version bringen `omd restore` kann eine Site **nur wiederherstellen, wenn genau deren Checkmk-Version im Zielsystem installiert ist.** Im Container kommt die Version @@ -1258,7 +1258,7 @@ ab, wenn die Version fehlt. > das passende `image pull`-Kommando. Enterprise-Images gibt es **nicht** auf > Docker Hub – siehe [Kapitel 8](#8-checkmk-image-beschaffen-raw-enterprise--co). -### 14.5 Schritt 4 – Bundle importieren +### 17.5 Schritt 4 – Bundle importieren Mit Oberfläche: @@ -1286,7 +1286,7 @@ Das Skript der Container die Site auch nach einem Reboot automatisch startet, 7. startet die Site und gibt die nächsten Schritte aus. -### 14.6 Schritt 5 – Umschalten und kontrollieren +### 17.6 Schritt 5 – Umschalten und kontrollieren ```bash # 1) Alten Server abschalten bzw. dessen Site stoppen (kein Doppel-Monitoring!) @@ -1315,7 +1315,7 @@ Danach in der Oberfläche prüfen: `cmk-agent-ctl status` – dort muss die Site `prod` mit dem Zielserver und „Connection: TLS“ erscheinen. -### 14.7 Checkliste Migration +### 17.7 Checkliste Migration - [ ] `migrate_creator.sh` auf den Quellserver kopiert - [ ] Bundle erzeugt (möglichst mit `--stop`) diff --git a/README_en.md b/README_en.md new file mode 100644 index 0000000..97570c0 --- /dev/null +++ b/README_en.md @@ -0,0 +1,1477 @@ +# Checkmk in a container – Docker (Debian) & Podman (RHEL) with NGINX Proxy Manager + +*[Deutsche Fassung: README.md](README.md)* + +A complete, production-ready container installation of **Checkmk** with +**NGINX Proxy Manager** for HTTPS termination (Let's Encrypt *or* self-signed), +including tooling for **backup**, **restore** and the **migration of existing +sites from a bare-metal installation** – without having to re-register the +agents. + +> **Note on language:** the scripts and their menus are in German. Literal +> program output and menu screenshots in this document are therefore shown +> unchanged; the surrounding text explains what they mean. + +--- + +## Table of contents + +1. [Overview & architecture](#1-overview--architecture) +2. [Directory layout](#2-directory-layout) +3. [Requirements & ports](#3-requirements--ports) +4. [Installation on Debian/Ubuntu (Docker)](#4-installation-on-debianubuntu-docker) +5. [Installation on RHEL/Rocky/Alma/Fedora (Podman)](#5-installation-on-rhelrockyalmafedora-podman) +6. [Starting automatically after a host reboot (Podman)](#6-starting-automatically-after-a-host-reboot-podman) +7. [Configuration (`.env`)](#7-configuration-env) +8. [Obtaining the Checkmk image (Raw, Enterprise & co.)](#8-obtaining-the-checkmk-image-raw-enterprise--co) +9. [First start](#9-first-start) +10. [Setting up HTTPS – Let's Encrypt or self-signed](#10-setting-up-https--lets-encrypt-or-self-signed) +11. [Configuring the Proxy Manager locally (terminal browser)](#11-configuring-the-proxy-manager-locally-terminal-browser) +12. [Moving the admin interface to a different port](#12-moving-the-admin-interface-to-a-different-port) +13. [Firewall](#13-firewall) +14. [Management: `cmk-manage.sh` (CLI and ncurses interface)](#14-management-cmk-managesh) +15. [Backup](#15-backup) +16. [Restore](#16-restore) +17. [Migrating a bare-metal installation (`migrate_creator.sh`)](#17-migrating-a-bare-metal-installation-migrate_creatorsh) +18. [Several sites in one container](#18-several-sites-in-one-container) +19. [Upgrading Checkmk](#19-upgrading-checkmk) +20. [Troubleshooting](#20-troubleshooting) +21. [Security notes](#21-security-notes) +22. [Uninstalling](#22-uninstalling) + +--- + +## 1. Overview & architecture + +``` + Internet / LAN + | + :80 :443 :81 + | + +-----------------------------------+ + | NGINX Proxy Manager (container) | network_mode: host + | - Let's Encrypt OR self-signed | + | - web interface on port 81 | + +-----------------------------------+ + | http://127.0.0.1:5000 + v + +-----------------------------------+ + | Checkmk / OMD (container) | network_mode: host + | :5000 web GUI (Apache) | + | :8000 agent receiver | <-- agents (cmk-agent-ctl) + | :6557 Livestatus (optional) | + +-----------------------------------+ + | + bind mount: ./data/checkmk/sites -> /omd/sites +``` + +**Deliberate design decisions** + +| Topic | Implementation | Rationale | +|---|---|---| +| Checkmk on the host network | `network_mode: host`, **no** `ports:` entries | agents, SNMP traps, ICMP and the agent receiver work without NAT; the source IP of the monitored hosts is preserved | +| NGINX Proxy Manager also on the host network | `network_mode: host` | reaches the Checkmk backend directly via `127.0.0.1:5000`; no bridge network, no `host-gateway` tricks – behaves identically under Docker **and** Podman | +| No named volumes | **bind mounts** into `./data/` only | all data sits visibly in the project directory, easy to back up, copy and inspect | +| One compose file for both | `docker compose` and `podman-compose` | identical operation on Debian and RHEL | +| `:Z` on the bind mounts | SELinux relabelling | required on RHEL; a no-op on Debian/Docker (not an error) | + +--- + +## 2. Directory layout + +``` +. +├── README.md # German documentation +├── README_en.md # this file +├── docker-compose.yml # for Docker AND Podman +├── .env.example # template -> .env +├── install.sh # installs Docker (Debian) / Podman (RHEL) +│ +├── scripts/ +│ ├── cmk-manage.sh # stack, backup, restore, import + ncurses GUI +│ ├── migrate_creator.sh # -> copy this to the SOURCE SERVER +│ └── lib/ +│ └── common.sh # shared functions + TUI helpers +│ +├── containers/ +│ └── browser/ +│ └── Containerfile # tools image: text browsers, curl, jq, openssl +│ +├── systemd/ +│ ├── checkmk-stack.service.example # compose-based unit (autostart) +│ └── quadlet/ +│ ├── checkmk.container # Podman Quadlet (alternative) +│ └── nginx-proxy-manager.container +│ +├── data/ # ALL payload data (bind mounts) +│ ├── checkmk/sites/ # -> /omd/sites (complete OMD sites) +│ └── npm/ +│ ├── data/ # -> /data (NPM config/DB) +│ ├── letsencrypt/ # -> /etc/letsencrypt (certificates) +│ └── custom-certs/ # self-signed certificates ready to upload +│ +├── backups/ # backups and migration bundles (*.tar.gz) +└── images/ # downloaded Checkmk image tarballs +``` + +--- + +## 3. Requirements & ports + +* 64-bit Linux, at least 2 CPU cores and 4 GB RAM (recommended: 4 cores, 8 GB) +* Disk space: rule of thumb 1 GB per 100 monitored hosts and year, plus backups +* root privileges +* Outbound internet access to download the images + +### Ports in use (host network!) + +| Port | Service | Reachable from | +|---|---|---| +| 80/tcp | NGINX Proxy Manager – HTTP, Let's Encrypt challenge | internet/LAN | +| 443/tcp | NGINX Proxy Manager – HTTPS | internet/LAN | +| 81/tcp | NGINX Proxy Manager – **admin interface** (default; changeable via [`npm-port`](#12-moving-the-admin-interface-to-a-different-port)) | admin network only! | +| 5000/tcp | Checkmk Apache (backend) | **localhost only** – do not expose | +| 8000/tcp | Checkmk agent receiver (TLS-registered agents) | monitored hosts | +| 6557/tcp | Livestatus (only with `CMK_LIVESTATUS_TCP=on`) | distributed sites | + +> **Important:** because both containers run on the host network, ports 80, 443 +> and 81 must not already be taken on the host (e.g. by an installed +> Apache/NGINX). Check with `ss -tlnp | grep -E ':(80|81|443|5000)\b'`. + +--- + +## 4. Installation on Debian/Ubuntu (Docker) + +```bash +# put the project where it should live +sudo mkdir -p /opt/checkmk-stack +sudo cp -r ./* ./.env.example /opt/checkmk-stack/ +cd /opt/checkmk-stack + +# install Docker CE + compose plugin, prepare the project +sudo ./install.sh --with-systemd +``` + +`install.sh` does the following: + +1. adds the Docker repository from `download.docker.com` (GPG key + sources list) +2. installs `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, + `docker-compose-plugin` +3. installs `dialog`, `openssl`, `tar`, `curl` (for the ncurses interface) +4. runs `systemctl enable --now docker` +5. creates the directories under `data/` and `backups/` and generates `.env` + from `.env.example` +6. optionally sets up the systemd unit `checkmk-stack.service` + +Then: + +```bash +sudo $EDITOR /opt/checkmk-stack/.env # at least CMK_PASSWORD and CMK_SITE_ID + +# choose the image (Raw needs no login, Enterprise see chapter 8) +sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest + +sudo ./scripts/cmk-manage.sh up +sudo ./scripts/cmk-manage.sh status +``` + +Under Docker, `restart: unless-stopped` already brings the containers back +after a reboot (the Docker daemon starts them). The systemd unit is still +worthwhile because it additionally starts **all** sites +(see [chapter 18](#18-several-sites-in-one-container)). + +--- + +## 5. Installation on RHEL/Rocky/Alma/Fedora (Podman) + +Docker is not available on Red Hat systems – **Podman** is used instead. The +compose file is the same one. + +```bash +sudo mkdir -p /opt/checkmk-stack +sudo cp -r ./* ./.env.example /opt/checkmk-stack/ +cd /opt/checkmk-stack + +sudo ./install.sh --with-systemd +``` + +`install.sh` installs: + +1. `podman`, `tar`, `dialog`, `openssl` +2. `podman-compose` – attempted in this order: + distribution repository → EPEL → `pip3 install podman-compose` +3. sets up the systemd unit (**always** with Podman, see the next chapter) + +Done manually, that is: + +```bash +sudo dnf install -y podman podman-compose tar dialog openssl +# if podman-compose is not in the repository: +sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm +sudo dnf install -y podman-compose +``` + +Starting up: + +```bash +sudo $EDITOR /opt/checkmk-stack/.env + +# obtain the image - for Enterprise with credentials, see chapter 8 +sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \ + --user --password + +sudo ./scripts/cmk-manage.sh up # automatically uses podman-compose +sudo ./scripts/cmk-manage.sh status +``` + +> On RHEL, do read [chapter 8.2](#8-obtaining-the-checkmk-image-raw-enterprise--co): +> short image names and `podman import` instead of `podman load` are the two +> causes of "image not known". + +### SELinux + +The bind mounts carry `:Z` in `docker-compose.yml`, so Podman automatically +applies the `container_file_t` context. Should access still fail: + +```bash +sudo semanage fcontext -a -t container_file_t "/opt/checkmk-stack/data(/.*)?" +sudo restorecon -Rv /opt/checkmk-stack/data +sudo ausearch -m avc -ts recent # inspect AVC messages +``` + +--- + +## 6. Starting automatically after a host reboot (Podman) + +> **This is the most important difference from Docker:** Podman has *no +> daemon*. `restart: unless-stopped` in the compose file therefore does **not** +> survive a reboot. Autostart has to go through systemd. There are two proven +> ways. + +### Option A – compose-based systemd unit (recommended, identical to Docker) + +Created and enabled automatically by `./install.sh --with-systemd`. Manually: + +```bash +sudo tee /etc/systemd/system/checkmk-stack.service >/dev/null <<'UNIT' +[Unit] +Description=Checkmk + NGINX Proxy Manager (container stack) +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +RemainAfterExit=yes +WorkingDirectory=/opt/checkmk-stack +EnvironmentFile=/opt/checkmk-stack/.env +TimeoutStartSec=0 +ExecStart=/bin/sh -c '/usr/bin/podman-compose up -d' +ExecStartPost=/opt/checkmk-stack/scripts/cmk-manage.sh start-all-sites +ExecStop=/bin/sh -c '/usr/bin/podman-compose down' + +[Install] +WantedBy=multi-user.target +UNIT + +sudo systemctl daemon-reload +sudo systemctl enable --now checkmk-stack.service +sudo systemctl status checkmk-stack.service +``` + +Verify that it survives a reboot: + +```bash +sudo systemctl is-enabled checkmk-stack # -> enabled +sudo reboot +# after the restart: +sudo podman ps +sudo /opt/checkmk-stack/scripts/cmk-manage.sh status +``` + +`ExecStartPost` calls `start-all-sites`. This is necessary because the Checkmk +entrypoint only starts the site named in `CMK_SITE_ID` – with several sites the +remaining ones would stay stopped. + +### Option B – Podman Quadlet (Podman 4.4 and newer, no compose) + +Quadlet turns simple description files into real systemd units. Advantages: no +`podman-compose` required, clean dependencies, native Podman. + +```bash +sudo ./install.sh --quadlet +``` + +Using the values from `.env`, this writes to `/etc/containers/systemd/`: + +* `checkmk.container` +* `nginx-proxy-manager.container` + +Then: + +```bash +sudo systemctl daemon-reload +sudo systemctl start checkmk nginx-proxy-manager +sudo systemctl status checkmk +``` + +`WantedBy=multi-user.target default.target` makes the units start +automatically at boot – a separate `systemctl enable` is neither needed nor +possible with Quadlet, because the units are generated. + +Templates live in [systemd/quadlet/](systemd/quadlet/). + +### Option C – rootless Podman (optional, more effort) + +```bash +sudo ./install.sh --rootless +sudo loginctl enable-linger checkmk # service account +# as a user unit: +mkdir -p ~/.config/systemd/user +cp systemd/checkmk-stack.service.example ~/.config/systemd/user/checkmk-stack.service +systemctl --user daemon-reload +systemctl --user enable --now checkmk-stack +``` + +`--rootless` sets `net.ipv4.ip_unprivileged_port_start=80` so that ports 80/443 +may be bound without root. `loginctl enable-linger` makes user units start +after a reboot even when nobody is logged in. + +> Note: in rootless mode the UIDs of the site users are mapped through user +> namespaces. Files under `data/checkmk/sites` then belong to subordinate UIDs +> (e.g. `165536`). That is normal, but it makes working with the files from the +> host awkward. For migrations, **rootful** operation is recommended. + +--- + +## 7. Configuration (`.env`) + +```bash +cp .env.example .env +chmod 600 .env +$EDITOR .env +``` + +The most important settings: + +| Variable | Meaning | +|---|---| +| `CMK_RUNTIME` | empty = auto-detect, otherwise `docker` or `podman` | +| `CMK_IMAGE` | **the image actually used**, always fully qualified – see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) | +| `CMK_EDITION`, `CMK_VERSION` | inputs for the image downloader only | +| `CMK_REGISTRY_USER/-PASSWORD` | login for `registry.checkmk.com` (Enterprise) | +| `CMK_DOWNLOAD_USER/-PASSWORD` | login for the Checkmk download portal | +| `CMK_SITE_ID` | the site the container creates and starts | +| `CMK_PASSWORD` | initial password for user `cmkadmin` (only on creation) | +| `CMK_LIVESTATUS_TCP` | `on` opens port 6557 for distributed monitoring | +| `MAIL_RELAY_HOST` | SMTP relay for notifications | +| `TZ` | time zone, e.g. `Europe/Berlin` | +| `NPM_ADMIN_PORT` | port of the Proxy Manager interface (default 81) – change only via `npm-port` or the menu | +| `NPM_ADMIN_URL` | start page of the terminal browser; empty = derived from `NPM_ADMIN_PORT` | +| `BACKUP_DIR` | location for backups/bundles (default `./backups`) | +| `CERT_DOMAIN`, `CERT_DAYS` | defaults for self-signed certificates | + +### 7.1 Who reads `.env` – and why does it exist alongside the script? + +`.env` is the **only** configuration file. There is deliberately no second +configuration inside the management script: + +``` + .env (single source of truth) + | + +---------------+----------------+------------------+ + | | | | +docker-compose systemd unit cmk-manage.sh migrate_creator.sh +(variable (EnvironmentFile) (source) (independent, runs on + substitution) the source server) +``` + +* **Read** by compose, by the systemd unit and by the script. +* **Written** by the script – either specifically (`image …` sets `CMK_IMAGE`, + `import --set-default-site` sets `CMK_SITE_ID`, `npm-port` sets + `NPM_ADMIN_PORT`) or through the settings menu / `config set` + (see [chapter 14](#14-management-cmk-managesh)). An editor is no longer + necessary, but remains available as a menu entry. +* The script itself only holds **fallback values** for the case that a key is + missing from `.env` (in `scripts/lib/common.sh`, function `load_env`). It + never overwrites a value that is set. + +In short: `.env` says *what* runs, the script *does* something with it. Nothing +has to be maintained twice. + +--- + +## 8. Obtaining the Checkmk image (Raw, Enterprise & co.) + +### 8.1 Which edition lives where? + +This is the most common pitfall: **the Enterprise edition is not on Docker +Hub.** The `checkmk` namespace on Docker Hub contains only: + +| Edition (`CMK_EDITION`) | Source | Login | +|---|---|---| +| `raw` (up to 2.4) | `docker.io/checkmk/check-mk-raw` | no | +| `community` (2.5 and later, successor of raw) | `docker.io/checkmk/check-mk-community` | no | +| `cloud` | `docker.io/checkmk/check-mk-cloud` | no | +| `managed` (up to 2.4) | `docker.io/checkmk/check-mk-managed` | no | +| `pro` (2.5 and later, successor of enterprise) | `docker.io/checkmk/check-mk-pro` | no | +| `ultimate`, `ultimatemt` (2.5 and later) | `docker.io/checkmk/check-mk-ultimate[mt]` | no | +| **`enterprise` (2.0 – 2.4)** | **`registry.checkmk.com/enterprise/check-mk-enterprise`** or an image tarball from the customer portal | **yes** | +| `free` (historic, up to 2.1) | `docker.io/checkmk/check-mk-free` | no | + +With Checkmk 2.5 the editions were renamed (Raw → Community, +Enterprise → Pro, Managed → Ultimate MT); those images are publicly available +on Docker Hub again. So if you run **2.5 or newer**, simply use `pro`. For +**2.2/2.3/2.4 Enterprise** the only routes are `registry.checkmk.com` or the +tarball. + +> Freely downloadable does not mean licence-free – the licence key is entered +> inside Checkmk afterwards. + +### 8.2 Two Podman pitfalls (the ones that actually bite) + +**1. Short image names.** Podman has no default registry. Short names such as +`checkmk/check-mk-enterprise:2.3.0p23` are resolved through +`unqualified-search-registries` in `/etc/containers/registries.conf` – and on +RHEL `registry.access.redhat.com` is listed **first**. Podman therefore looks +for the image at Red Hat, does not find it and reports `image not known`, or +asks interactively. That is why `.env` and `docker-compose.yml` consistently +use the **fully qualified** name: + +``` +CMK_IMAGE=registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p23 +``` + +Inspect the current resolution order: + +```bash +grep -A3 unqualified-search-registries /etc/containers/registries.conf +``` + +**2. `import` instead of `load`.** A downloaded image tarball is installed with +**`load`**: + +| Command | Expects | Result | +|---|---|---| +| `podman import file.tar.gz` | a **filesystem** archive (rootfs) | an image **without** entrypoint, environment or tag → unusable, "image unknown" | +| `podman load -i file.tar.gz` | an **image** archive (`docker save`/OCI) | a complete image including metadata and tag ✔ | + +Getting rid of a broken image created by `import`: + +```bash +podman images | grep '' +podman rmi +``` + +`cmk-manage.sh image load` checks the archive for `manifest.json` or +`oci-layout` beforehand and rejects a wrong archive with exactly this hint. + +### 8.3 Storing credentials in `.env` + +```bash +CMK_EDITION=enterprise +CMK_VERSION=2.3.0p23 + +# route 1 - private registry +CMK_REGISTRY=registry.checkmk.com +CMK_REGISTRY_USER= +CMK_REGISTRY_PASSWORD= + +# route 2 - tarball from the download portal +CMK_DOWNLOAD_BASEURL=https://download.checkmk.com/checkmk +CMK_DOWNLOAD_USER= +CMK_DOWNLOAD_PASSWORD= +``` + +The credentials come from the Checkmk customer portal (portal.checkmk.com). +`.env` should therefore be `chmod 600` – `install.sh` takes care of that. + +### 8.4 Fetching the image – three ways + +**a) From a registry (recommended)** + +```bash +# freely available, no login +sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest + +# Enterprise with a login to registry.checkmk.com +sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \ + --user --password +``` + +**b) Image tarball from the download portal (with credentials)** + +```bash +sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \ + --user --password +``` + +The downloader tries the usual file names under +`//` +(`check-mk-enterprise-.docker.tar.gz` and +`check-mk-enterprise-docker-.tar.gz`). Should Checkmk have changed the +scheme, the full URL can be given directly: + +```bash +sudo ./scripts/cmk-manage.sh image download \ + --url 'https://download.checkmk.com/checkmk/2.3.0p23/check-mk-enterprise-2.3.0p23.docker.tar.gz' \ + --user --password +``` + +The credentials are handed to `curl` through a temporary `.netrc` file and +therefore do **not** appear in the process list. + +**c) Loading a file you already downloaded** + +If you fetched the file from the customer portal with a browser: + +```bash +sudo ./scripts/cmk-manage.sh image load --file /root/check-mk-enterprise-2.3.0p23.docker.tar.gz +``` + +In all three cases the script determines the **actual** image name (for `load` +typically `localhost/checkmk/check-mk-enterprise:2.3.0p23`) and writes it to +`CMK_IMAGE` in `.env`. Afterwards: + +```bash +sudo ./scripts/cmk-manage.sh up +``` + +### 8.5 Checking what is in effect + +```bash +./scripts/cmk-manage.sh image status # configured image + local images +./scripts/cmk-manage.sh image list # local Checkmk images only +./scripts/cmk-manage.sh image use localhost/checkmk/check-mk-enterprise:2.3.0p23 +``` + +`image status` explicitly warns when `CMK_IMAGE` is a short name without a +registry. + +The ncurses interface offers the same workflow under the menu entry +*"Checkmk-Image beschaffen/auswählen"* (pick edition → version → credentials → +fetch). + +--- + +## 9. First start + +```bash +sudo ./scripts/cmk-manage.sh up +sudo ./scripts/cmk-manage.sh status +``` + +Output (example, German as printed by the tool): + +``` +Container + checkmk läuft + nginx-proxy-manager läuft + +Sites im Container + SITE VERSION STATUS + cmk 2.3.0p23.cre läuft + +Zugang + Checkmk (direkt) : http://10.0.0.5:5000/cmk/ + Checkmk (über Proxy) : https:///cmk/ + Agent-Receiver : 10.0.0.5:8000 + NGINX Proxy Manager : http://10.0.0.5:81/ (Erstlogin: admin@example.com / changeme) +``` + +(`läuft` = running, `gestoppt` = stopped, `Zugang` = access, `Erstlogin` = +initial login.) The port of the admin interface follows `NPM_ADMIN_PORT` from +`.env`; the output always shows the value actually configured. + +Logging in to Checkmk: user `cmkadmin`, password from `CMK_PASSWORD`. + +--- + +## 10. Setting up HTTPS – Let's Encrypt or self-signed + +### 10.1 Preparing the NGINX Proxy Manager + +1. open `http://:81` – or whichever port is set in `NPM_ADMIN_PORT` + (see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)); if + that port is blocked on the network, use the terminal browser from + [chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser) +2. initial login: **admin@example.com** / **changeme** +3. change the e-mail address and password immediately (enforced on first login) + +### 10.2 Creating a proxy host for Checkmk + +**Hosts → Proxy Hosts → Add Proxy Host** + +| Field | Value | +|---|---| +| Domain Names | `checkmk.example.com` (or your internal name) | +| Scheme | `http` | +| Forward Hostname / IP | `127.0.0.1` | +| Forward Port | `5000` | +| Block Common Exploits | on | +| Websockets Support | **on** (for the live data in the interface) | + +Tab **Advanced** (recommended, so Checkmk sees the real client IP and large +agent uploads work): + +```nginx +client_max_body_size 0; +proxy_read_timeout 300s; +proxy_set_header X-Forwarded-Proto $scheme; +proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +``` + +### 10.3 Option A – Let's Encrypt (publicly reachable server) + +Prerequisites: the DNS name resolves publicly to this server and **port 80 is +reachable from the internet** (HTTP-01 challenge). + +In the proxy host, tab **SSL**: + +* SSL Certificate → **Request a new SSL Certificate** +* *Force SSL* on +* *HTTP/2 Support* on +* *HSTS Enabled* as required +* enter an e-mail address, accept the terms → **Save** + +Renewal is handled by the Proxy Manager automatically; the certificates live in +the bind mount `data/npm/letsencrypt/` and are therefore kept inside the +project directory. + +For an internal DNS zone without public access, use the **DNS challenge** +(SSL → *Use a DNS Challenge* → select provider, store the API token). + +### 10.4 Option B – self-signed certificate (internal environment) + +Creating one: + +```bash +sudo ./scripts/cmk-manage.sh cert \ + --domain checkmk.intern.example \ + --alt 10.0.0.5 --alt checkmk \ + --days 3650 +``` + +Result in `data/npm/custom-certs/checkmk.intern.example/`: + +* `privkey.pem` – private key +* `fullchain.pem` – certificate (with SANs for every name/IP given) + +Installing it: + +1. **SSL Certificates → Add SSL Certificate → Custom** +2. give it a name, upload `privkey.pem` as *Certificate Key* and + `fullchain.pem` as *Certificate* +3. select this certificate in the proxy host under **SSL**, enable *Force SSL* + and *HTTP/2* + +> For Checkmk agents and browsers to trust the certificate, `fullchain.pem` +> must be installed as a trusted certificate on the clients (Windows: computer +> certificates → Trusted Root Certification Authorities; Linux: +> `/usr/local/share/ca-certificates/` + `update-ca-certificates`). +> For plain agent communication over port 8000 this is **not** needed – there +> Checkmk uses its own site-internal CA. + +The ncurses interface offers the same workflow under *"Selbstsigniertes +Zertifikat erzeugen"*. + +--- + +## 11. Configuring the Proxy Manager locally (terminal browser) + +When port 81 cannot be reached from the network, the NGINX Proxy Manager's +admin interface has to be operated **on the machine itself**. For that, +`cmk-manage.sh browser` starts another container on the host network – which +reaches `127.0.0.1:81` just like the Proxy Manager itself. + +```bash +sudo ./scripts/cmk-manage.sh browser +``` + +This is deliberately **not** a service in `docker-compose.yml`: a browser is an +interactive program with a terminal, not a background service. It is started +with `run --rm -it` and disappears without a trace when you quit. + +### 11.1 Why not `lynx` + +The Proxy Manager's admin interface is a JavaScript application. Classic text +browsers **cannot** render it – you get a blank page. There are therefore two +engines that render a full browser inside the terminal: + +| Engine | Based on | JavaScript | Proxy Manager usable | +|---|---|---|---| +| `browsh` | Firefox | yes | **yes** – default, maintained | +| `carbonyl` | Chromium | yes | **yes** – nicer rendering, last released 2023 | +| `links` (links2) | text browser | no | no | +| `w3m` | text browser | no | no | +| `lynx` | text browser | no | no | +| `shell` | Bash + curl/jq/openssl | – | for diagnostics and the API | + +The text browsers are included anyway because they are handy for quick checks. +If you pick one, the script warns explicitly. + +### 11.2 Usage + +```bash +# default: browsh on http://127.0.0.1:81 +sudo ./scripts/cmk-manage.sh browser + +# a different engine or a different page +sudo ./scripts/cmk-manage.sh browser --engine carbonyl +sudo ./scripts/cmk-manage.sh browser --engine links --url http://127.0.0.1:5000/cmk/ + +# shell with network tools on the host network +sudo ./scripts/cmk-manage.sh browser --engine shell +``` + +Key bindings in **browsh**: + +| Key | Action | +|---|---| +| `Ctrl+L` | address bar | +| `Ctrl+T` | new tab | +| `Ctrl+W` | close tab | +| `Ctrl+Q` | quit | + +The first time a text-browser or `shell` engine is used, the script builds a +small tools image (Debian slim with links2, w3m, lynx, curl, jq, openssl, +roughly 170 MB). Build it in advance – this works without a terminal, e.g. from +an installation script: + +```bash +sudo ./scripts/cmk-manage.sh browser --build-only +``` + +`browsh` and `carbonyl`, by contrast, are pulled as ready-made images; their +names are in `.env` (`BROWSH_IMAGE`, `CARBONYL_IMAGE`) and are fully qualified +so that Podman does not look for them at `registry.access.redhat.com`. + +In the ncurses interface everything lives under *"Terminal-Browser (Proxy +Manager lokal bedienen)"*. + +### 11.3 Limitation: file upload + +A terminal browser cannot open the operating system's file chooser. For +**Let's Encrypt** that does not matter (only form fields are filled in), but +uploading a **self-signed certificate** (*SSL Certificates → Custom*) does not +work reliably this way. + +For exactly this case, use the Proxy Manager's REST interface. Create the +certificate as described in +[chapter 10.4](#10-setting-up-https--lets-encrypt-or-self-signed), then: + +```bash +NPM=http://127.0.0.1:81 +CERTDIR=./data/npm/custom-certs/checkmk.intern.example + +# 1) log in and obtain a token +TOKEN=$(curl -s -X POST "$NPM/api/tokens" \ + -H 'Content-Type: application/json' \ + -d '{"identity":"admin@example.com","secret":""}' | jq -r .token) + +# 2) create the certificate entry (provider "other" = your own certificate) +CERT_ID=$(curl -s -X POST "$NPM/api/nginx/certificates" \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"provider":"other","nice_name":"checkmk-selfsigned","domain_names":["checkmk.intern.example"]}' \ + | jq -r .id) + +# 3) upload key and certificate +curl -s -X POST "$NPM/api/nginx/certificates/$CERT_ID/upload" \ + -H "Authorization: Bearer $TOKEN" \ + -F "certificate=@$CERTDIR/fullchain.pem" \ + -F "certificate_key=@$CERTDIR/privkey.pem" + +# 4) create the proxy host for Checkmk +curl -s -X POST "$NPM/api/nginx/proxy-hosts" \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d "{\"domain_names\":[\"checkmk.intern.example\"], + \"forward_scheme\":\"http\", + \"forward_host\":\"127.0.0.1\", + \"forward_port\":5000, + \"certificate_id\":$CERT_ID, + \"ssl_forced\":true, + \"http2_support\":true, + \"block_exploits\":true, + \"allow_websocket_upgrade\":true, + \"advanced_config\":\"client_max_body_size 0;\", + \"enabled\":true, + \"meta\":{}}" +``` + +`curl` and `jq` are part of the tools image in case the host lacks them: + +```bash +sudo ./scripts/cmk-manage.sh browser --engine shell +``` + +> On the very first login the Proxy Manager forces you to change the e-mail +> address and password. That is easiest in the browser +> (`cmk-manage.sh browser`); afterwards the API works with the new credentials. + +### 11.4 Alternative: SSH tunnel + +If SSH access to the machine is possible, a tunnel is the most convenient +route – you then use the interface in your usual browser: + +```bash +ssh -N -L 8081:127.0.0.1:81 user@checkmk-server +# then locally: http://127.0.0.1:8081 +``` + +This changes nothing about the firewall: port 81 stays closed from outside, +access runs through the existing SSH connection. + +--- + +## 12. Moving the admin interface to a different port + +Inside the image, the NGINX Proxy Manager's web interface listens on a +**hard-coded port 81** (`listen 81 default;` in +`/etc/nginx/conf.d/production.conf`). No environment variable for this exists +in the published 2.x versions – `NPM_ADMIN_PORT` is already prepared in the +development branch, but is not part of version 2.15.1. And because both +containers run on the host network, the port cannot be moved by port mapping +either. + +So if port 81 is blocked on your network: + +```bash +sudo ./scripts/cmk-manage.sh npm-port 10443 +``` + +### 12.1 What happens + +Within its `http{}` block, nginx includes an official extension point: + +``` +include /data/nginx/custom/http.conf; +``` + +`/data` is our bind mount `./data/npm/data`. The command places an additional +server there which listens on the desired port and forwards to the interface on +`127.0.0.1:81`: + +``` +./data/npm/data/nginx/custom/http.conf + +# >>> cmk-manage.sh: Admin-Port - nicht von Hand aendern >>> +server { + listen 10443; + listen [::]:10443; + server_name _; + client_max_body_size 0; + location / { + proxy_pass http://127.0.0.1:81; + ... + } +} +# <<< cmk-manage.sh: Admin-Port <<< +``` + +Only the region between the two markers is managed – your own additions to the +same file are left untouched. The script then restarts the Proxy Manager +container so nginx reads the file, and writes `NPM_ADMIN_PORT` and +`NPM_ADMIN_URL` into `.env`. The terminal browser from +[chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser) then +starts on the new port automatically. + +### 12.2 Important: port 81 stays alive locally + +The built-in server on port 81 **cannot** be switched off this way – it keeps +listening on all addresses. That is harmless as long as the firewall blocks it, +and that is exactly the recommendation: + +```bash +# firewalld +sudo firewall-cmd --permanent --add-port=10443/tcp +sudo firewall-cmd --permanent --remove-port=81/tcp 2>/dev/null +sudo firewall-cmd --reload + +# ufw +sudo ufw allow 10443/tcp +sudo ufw delete allow 81/tcp 2>/dev/null +``` + +### 12.3 Inspecting and reverting + +```bash +./scripts/cmk-manage.sh npm-port # show the current state +sudo ./scripts/cmk-manage.sh npm-port 81 # back to the default +``` + +With `81` the managed block is removed again; if the file is empty afterwards +it is deleted entirely. Only the configuration from the image then applies. + +The command rejects ports already taken by the stack (80, 443, 5000, 8000, +6557), validates the numeric range, and warns if another service is already +listening on the port you want. + +> As soon as an NPM release ships `NPM_ADMIN_PORT`, this detour becomes +> unnecessary – the environment variable will be enough. Until then, the +> extension point is the route that needs no change to the image. + +--- + +## 13. Firewall + +**firewalld (RHEL/Rocky/Alma):** + +```bash +sudo firewall-cmd --permanent --add-service=http +sudo firewall-cmd --permanent --add-service=https +sudo firewall-cmd --permanent --add-port=8000/tcp # agent receiver +sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.0.0.0/24 port port=81 protocol=tcp accept' +sudo firewall-cmd --reload + +# With a different admin port (e.g. 10443) open that number instead - +# "cmk-manage.sh npm-port " prints the matching command for you. +``` + +**ufw (Debian/Ubuntu):** + +```bash +sudo ufw allow 80/tcp +sudo ufw allow 443/tcp +sudo ufw allow 8000/tcp +sudo ufw allow from 10.0.0.0/24 to any port 81 proto tcp +sudo ufw enable +``` + +Port **5000 stays closed** – access goes exclusively through the proxy. Open +port **81** to the administration network only. + +--- + +## 14. Management: `cmk-manage.sh` + +The script is both a command-line tool **and** an ncurses interface. + +### 14.1 ncurses interface (Midnight Commander style) + +```bash +sudo ./scripts/cmk-manage.sh gui +``` + +The menu labels are German; the command table in +[14.2](#14-management-cmk-managesh) lists the same actions in English. + +``` + ┌──────────────── Checkmk Stack Manager ─────────────────┐ + │ Projekt : /opt/checkmk-stack │ + │ Runtime : podman Standard-Site: cmk │ + │ Backups : /opt/checkmk-stack/backups │ + │ │ + │ status Status von Containern und Sites │ + │ stack Stack steuern (Start/Stop/Logs/Update) │ + │ backup Backup erstellen │ + │ restore Backup wiederherstellen │ + │ import Site von Direktinstallation importieren │ + │ manage Backups verwalten (Liste/Details/Löschen) │ + │ sites Sites im Container verwalten │ + │ image Checkmk-Image beschaffen/auswählen │ + │ browser Terminal-Browser (Proxy Manager lokal) │ + │ cert Selbstsigniertes Zertifikat erzeugen │ + │ config Einstellungen (.env) - menügeführt │ + │ shell Root-Shell im Checkmk-Container │ + │ quit Beenden │ + │ < OK > │ + └────────────────────────────────────────────────────────┘ +``` + +Operation: arrow keys, **space** toggles entries in selection lists (e.g. +several sites for a backup), **Enter** confirms, **ESC/Cancel** goes back. +`dialog` is preferred; if only `whiptail` is present, that is used instead. +A file chooser with directory navigation is built in. + +### 14.2 Command line + +```bash +./scripts/cmk-manage.sh --help +``` + +| Command | Effect | +|---|---| +| `up` / `down` / `restart` / `pull` | control the stack | +| `status` | containers, sites, access addresses | +| `logs [-f]` | container logs | +| `sites` | list the sites in the container | +| `site-start SITE` / `site-stop SITE` | control a single site | +| `start-all-sites` | start all sites (invoked by systemd) | +| `backup …` | backup as a `.tar.gz` bundle | +| `list` / `verify` / `prune --keep N` | manage backups | +| `restore …` | restore a backup | +| `import …` | import a migration bundle | +| `image status\|list\|pull\|download\|load\|use` | obtain and select the Checkmk image | +| `browser [--engine …]` | terminal browser in a container – operate the Proxy Manager locally | +| `config list\|get\|set\|edit` | show and change settings in `.env` | +| `npm-port [PORT]` | set/show the port of the Proxy Manager interface | +| `cert …` | create a self-signed certificate | +| `shell` | root shell in the Checkmk container | +| `omd …` | any `omd` command inside the container | + +Global options: `--env-file FILE`, `-y/--yes` (no prompts, e.g. for cron jobs), +`-v/--verbose`. + +### 14.3 Changing settings – without an editor + +Every value in `.env` can be maintained through the menu. In the interface: +**Einstellungen (.env) – menügeführt**. The values are grouped by area: + +``` + ┌──────────────────── Einstellungen ─────────────────────┐ + │ Alle Werte der Datei .env - direkt hier änderbar. │ + │ Datei: /opt/checkmk-stack/.env │ + │ │ + │ runtime Runtime und Zeitzone │ + │ image Checkmk-Image und Zugangsdaten │ + │ checkmk Checkmk-Site und Container │ + │ npm NGINX Proxy Manager │ + │ browser Terminal-Browser │ + │ paths Verzeichnisse │ + │ cert Selbstsignierte Zertifikate │ + │ show Alle Werte auflisten │ + │ editor Datei .env im Editor öffnen │ + │ back zurück │ + └────────────────────────────────────────────────────────┘ +``` + +(runtime and time zone / Checkmk image and credentials / Checkmk site and +containers / NGINX Proxy Manager / terminal browser / directories / self-signed +certificates / list all values / open `.env` in an editor / back.) + +Each entry shows its current value and, when confirmed, opens the matching +input: free text, number, selection list or password field (passwords appear as +`********` in the overview and stay unchanged if you leave the field empty). +The editor remains available as a menu entry. + +The same values from the command line: + +```bash +./scripts/cmk-manage.sh config list # everything with current values +./scripts/cmk-manage.sh config list --category npm # one area only +./scripts/cmk-manage.sh config get CMK_SITE_ID +./scripts/cmk-manage.sh config set CMK_SITE_ID prod +./scripts/cmk-manage.sh config set NPM_ADMIN_PORT 10443 +``` + +**Validation happens when setting the value**, not at the next start: + +| Key | Check | +|---|---| +| `NPM_ADMIN_PORT` | number 1–65535, not 80/443/5000/8000/6557, warning if in use | +| `CMK_SITE_ID` | letters, digits and `_` only; at most 16 characters | +| `CMK_IMAGE`, `NPM_IMAGE`, … | must be fully qualified (the Podman pitfall) | +| `CMK_RUNTIME` | must be installed on this system | +| `CERT_DAYS` and others | numeric fields | +| URL fields | must start with `http://` or `https://` | + +Values containing spaces are quoted automatically so that `.env` remains +readable by compose, systemd **and** the shell. When you change +`NPM_ADMIN_PORT`, the script updates the additional nginx file along with it +(see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)). For +values that only take effect once the containers are recreated, the menu says +so explicitly. + +--- + +## 15. Backup + +Backups are created with `omd backup` and packed into a **bundle**: + +``` +cmkbackup___.tar.gz +└── cmkbundle/ + ├── MANIFEST site, version, source, checksum … + ├── INFO.txt human-readable summary + ├── site.tar.gz the actual omd backup + └── site.tar.gz.sha256 +``` + +This way the restore/import tool always knows **which site** and **which +Checkmk version** the archive contains, and can verify its integrity. + +```bash +# a single site +sudo ./scripts/cmk-manage.sh backup --site cmk + +# all sites, without log files, with a label +sudo ./scripts/cmk-manage.sh backup --all --no-logs --label vor-update + +# most consistent backup: the site is stopped briefly +sudo ./scripts/cmk-manage.sh backup --site cmk --stop + +# without performance data (considerably smaller) +sudo ./scripts/cmk-manage.sh backup --site cmk --no-rrds --dir /mnt/nas/checkmk +``` + +Listing and verifying: + +```bash +./scripts/cmk-manage.sh list +./scripts/cmk-manage.sh verify --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz +./scripts/cmk-manage.sh prune --keep 7 +``` + +### Nightly backup via cron + +```bash +sudo crontab -e +``` + +```cron +15 2 * * * /opt/checkmk-stack/scripts/cmk-manage.sh -y backup --all --no-logs >>/var/log/cmk-backup.log 2>&1 +30 3 * * 0 /opt/checkmk-stack/scripts/cmk-manage.sh -y prune --keep 14 >>/var/log/cmk-backup.log 2>&1 +``` + +--- + +## 16. Restore + +```bash +# simplest case: the site no longer exists +sudo ./scripts/cmk-manage.sh restore --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz + +# overwrite an existing site (terminating running processes) +sudo ./scripts/cmk-manage.sh restore --file backups/… --reuse --kill + +# restore under a different name (e.g. for a test) +sudo ./scripts/cmk-manage.sh restore --file backups/… --as-site test +``` + +What the script does: + +1. checks the archive structure and the SHA256 checksum +2. checks whether the **Checkmk version of the backup exists in the container** +3. prompts / requires `--reuse` for an existing site, stops the site +4. copies the archive into the container and runs `omd restore` +5. turns off `TMPFS` for the site (not usable inside the container) +6. starts the site and prints the next steps + +> **`--as-site` changes the site name** – registered agents would then point +> nowhere and would have to be re-registered. So do **not** use it for a +> migration. + +A raw `omd backup` archive (without the bundle wrapper) is accepted as well; +the site name is then read from the archive. + +--- + +## 17. Migrating a bare-metal installation (`migrate_creator.sh`) + +Goal: take over an existing Checkmk installation on hardware or a VM into the +container – **without touching the agents**. + +### 17.1 Why the agents keep working + +An `omd backup` contains the complete site including `etc/ssl` (the site's own +CA and the certificates of the registered agents), `etc/check_mk` (rules, +hosts, passwords) and the monitoring history. If the site is restored **under +the same name** and the server is reachable at **the same IP address**, then + +* push/pull agents (`cmk-agent-ctl`) find their registration again, +* the TLS connection to the agent receiver on **port 8000** stays valid, +* all URLs (`https:////check_mk/…`) remain correct, +* performance data and history are preserved. + +The prerequisite is therefore that the target system **takes over the source +server's IP** (or that the DNS name is repointed) – which is the plan anyway. + +### 17.2 Step 1 – copy the script to the source server + +```bash +scp scripts/migrate_creator.sh root@old-server:/root/ +ssh root@old-server 'chmod +x /root/migrate_creator.sh' +``` + +The script is **self-contained** – on the source server it only needs `bash`, +`tar` and `omd` (plus `dialog` or `whiptail` for the interface). + +### 17.3 Step 2 – create the bundle on the source server + +Show the existing sites: + +```bash +sudo ./migrate_creator.sh --list +``` + +``` +SITE VERSION STATUS GROESSE +prod 2.3.0p23.cre laeuft 4.7 GiB +test 2.3.0p23.cre gestoppt 318.2 MiB +``` + +(`GROESSE` = size, `laeuft` = running, `gestoppt` = stopped.) + +**With the interface** (select sites with the space bar): + +```bash +sudo ./migrate_creator.sh --gui +``` + +The dialog asks in turn for: sites → options (stop the site, skip RRDs/logs) → +target directory → label → optional transfer via `scp`. + +**With parameters:** + +```bash +# one site, stopping it during the backup (most consistent variant) +sudo ./migrate_creator.sh --site prod --stop --out /var/tmp + +# all sites, without logs, transferred straight to the target system +sudo ./migrate_creator.sh --all --no-logs \ + --scp root@10.0.0.5:/opt/checkmk-stack/backups/ +``` + +Result: + +``` +/var/tmp/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz +``` + +Besides the omd backup, the bundle contains a `MANIFEST` with the site name, +Checkmk version, source hostname, **source IP addresses**, source operating +system and SHA256 checksum, plus a `siteinfo/` directory with `omd config +show`, `omd status` and the IP configuration of the old system for later +reference. + +### 17.4 Step 3 – bring the target system to the right version + +`omd restore` can only restore a site **if exactly its Checkmk version is +installed on the target system.** In the container, the version comes from the +image – so the image tag has to match: + +```bash +tar -xzOf backups/cmkmigrate_prod_*.tar.gz cmkbundle/MANIFEST | grep CMK_VERSION +# CMK_VERSION=2.3.0p23.cre -> version 2.3.0p23, suffix .cre = Raw edition + +# fetch the matching image and have it written to CMK_IMAGE +sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p23 +sudo ./scripts/cmk-manage.sh up +``` + +For an Enterprise source (`.cee`) accordingly: + +```bash +sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \ + --user --password +# or, if the registry is unreachable: +sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \ + --user --password +``` + +The import script checks this itself and aborts with a corresponding message if +the version is missing. + +> **Mind the edition:** the version suffix reveals it – `.cre` = Raw, +> `.cee` = Enterprise, `.cce` = Cloud, `.cme` = Managed. The import script +> reads it from the manifest and, on failure, names the matching `image pull` +> command directly. Enterprise images are **not** on Docker Hub – see +> [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co). + +### 17.5 Step 4 – import the bundle + +With the interface: + +```bash +sudo ./scripts/cmk-manage.sh gui # -> "import" +``` + +With parameters: + +```bash +sudo ./scripts/cmk-manage.sh import \ + --file backups/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz \ + --set-default-site +``` + +The script + +1. recognises the migration bundle and shows the source host and source IP, +2. **warns if this host's IP differs from the source IP** (the agents would + then have to be adjusted), +3. verifies checksum, version and edition, +4. prompts if the site already exists (`--reuse` overwrites), +5. runs `omd restore` under the **unchanged site name**, +6. optionally sets `CMK_SITE_ID` in `.env` (`--set-default-site`) so the + container starts the site automatically after a reboot as well, +7. starts the site and prints the next steps. + +### 17.6 Step 5 – switch over and verify + +```bash +# 1) shut down the old server or stop its site (no double monitoring!) +ssh root@old-server 'omd stop prod' + +# 2) move the target system to the IP/DNS name of the old system +# (take over the static IP or repoint the DNS record) + +# 3) recreate the containers if CMK_SITE_ID was changed +sudo ./scripts/cmk-manage.sh up +sudo ./scripts/cmk-manage.sh status + +# 4) check inside Checkmk +sudo ./scripts/cmk-manage.sh omd su prod + OMD[prod]:~$ cmk -O # activate the configuration + OMD[prod]:~$ omd status + OMD[prod]:~$ exit +``` + +Then check in the interface: + +* **Monitor → All hosts** – are the agents reporting again? +* With TLS agents: the service *"Check_MK Agent"* must not report registration + errors. +* An agent can be tested from the monitored host with `cmk-agent-ctl status` – + the site `prod` must appear there with the target server and + "Connection: TLS". + +### 17.7 Migration checklist + +- [ ] `migrate_creator.sh` copied to the source server +- [ ] bundle created (preferably with `--stop`) +- [ ] bundle copied to the target system (`backups/`) +- [ ] matching image fetched (`image pull`/`image download`), `CMK_IMAGE` set, stack recreated +- [ ] `import --set-default-site` executed, site running +- [ ] old site stopped / old server shut down +- [ ] IP or DNS switched to the target system +- [ ] ports 80/443/8000 open in the firewall +- [ ] proxy host set up in the NGINX Proxy Manager with a certificate +- [ ] agents reporting, `cmk -O` executed +- [ ] backup of the new system set up (cron) + +--- + +## 18. Several sites in one container + +At startup the Checkmk entrypoint only creates – and only starts – the site +from `CMK_SITE_ID`. Further sites (e.g. several imported ones) exist in the +bind mount but are not brought up automatically after a restart. + +Hence: + +```bash +sudo ./scripts/cmk-manage.sh start-all-sites +``` + +This command is executed automatically by the systemd unit (`ExecStartPost`) +or the Quadlet unit after the container starts, and starts every site present. + +> **Recommendation:** for cleanly separated environments, run one project +> directory with its own `.env` per site. Note that because both containers run +> on the host network, only **one** Checkmk container per host can occupy ports +> 5000/8000 – for genuinely multiple instances use separate hosts or a +> bridge-network variant. + +--- + +## 19. Upgrading Checkmk + +Because `/omd/sites` is a bind mount, the data survives replacing the image. +The Checkmk software itself lives in the image under `/omd/versions`. + +```bash +# 1) ALWAYS back up first +sudo ./scripts/cmk-manage.sh backup --all --label vor-update + +# 2) fetch the new image (writes CMK_IMAGE into .env automatically) +sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p28 + +# 3) recreate the containers +sudo ./scripts/cmk-manage.sh up + +# 4) update the site (interactive, confirm conflicts) +sudo ./scripts/cmk-manage.sh omd stop cmk +sudo ./scripts/cmk-manage.sh omd update cmk +sudo ./scripts/cmk-manage.sh omd start cmk +``` + +For a major version jump (e.g. 2.2 → 2.3) be sure to read the Checkmk werks / +release notes beforehand. + +--- + +## 20. Troubleshooting + +| Symptom | Cause / solution | +|---|---| +| `Port 80/443 already in use` | a web server is already running on the host: `ss -tlnp \| grep ':80'` → `systemctl disable --now apache2 nginx httpd` | +| Containers do not run after a reboot (Podman) | systemd unit missing: `./install.sh --with-systemd` or `--quadlet`, then `systemctl is-enabled checkmk-stack` | +| `omd restore`: *version is not installed* | fetch the matching image: `cmk-manage.sh image pull --edition --version `, then `up` | +| Import aborts with an edition warning | an Enterprise backup needs an Enterprise image, see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) | +| `image not known` / Podman searches `registry.access.redhat.com` | short image name. Set `CMK_IMAGE` fully qualified: `cmk-manage.sh image use docker.io/…` or `registry.checkmk.com/enterprise/…` | +| `checkmk/check-mk-enterprise` not found on Docker Hub | correct – Enterprise is only on `registry.checkmk.com` or available as a tarball; from 2.5 the edition is called `pro` and is public on Docker Hub | +| Tarball installed with `podman import`, image "unknown" | `import` produces an image without metadata. Remove the broken image with `podman rmi ` and use `cmk-manage.sh image load --file …` (which uses `load`) | +| Download of the image tarball fails (401/404) | check the customer portal credentials, give the exact version (no `-latest`), if necessary `image download --url ''` | +| Proxy returns `502 Bad Gateway` | the Checkmk container is not running, or the forwarding does not point to `127.0.0.1:5000`: `./scripts/cmk-manage.sh status`, `curl -I http://127.0.0.1:5000/` | +| Let's Encrypt fails | port 80 not reachable from the internet, or wrong DNS; alternative: DNS challenge or a self-signed certificate | +| Agents do not report after the migration | site name changed (`--as-site`), IP not taken over, or port 8000 blocked; check on the agent: `cmk-agent-ctl status` | +| Permission denied on `data/` under RHEL | SELinux: `restorecon -Rv /opt/checkmk-stack/data`, check `:Z` on the mounts | +| `podman-compose` not found | enable EPEL or switch to Quadlet (`./install.sh --quadlet`) | +| Admin interface unreachable from the LAN | port 81 blocked. Move it to a free port: `cmk-manage.sh npm-port 10443` and open that in the firewall | +| New port does not answer after `npm-port` | the Proxy Manager was not restarted: `cmk-manage.sh restart`; check the file `data/npm/data/nginx/custom/http.conf` | +| `cmk-manage.sh` aborts with "CMK_RUNTIME … ist nicht installiert" | it now only warns and detects the runtime itself; set it cleanly with `config set CMK_RUNTIME auto` | +| Proxy Manager interface blank / only text fragments | a text browser without JavaScript was used. Use `cmk-manage.sh browser --engine browsh` (or `carbonyl`) | +| Self-signed certificate cannot be uploaded in the terminal browser | there are no file dialogs there – upload via the API, see [chapter 11.3](#11-configuring-the-proxy-manager-locally-terminal-browser) | +| `browser` reports "braucht ein echtes Terminal" (needs a real terminal) | called from a pipe/script. Call it on the console or over SSH; only the build runs headless, with `--build-only` | +| The TUI does not start | install `dialog`: `apt-get install -y dialog` or `dnf install -y dialog` | +| Site does not start, `tmp` errors | `./scripts/cmk-manage.sh omd config set TMPFS off`, then start the site | + +Useful commands: + +```bash +./scripts/cmk-manage.sh logs -f # container logs +./scripts/cmk-manage.sh omd status cmk # services of the site +./scripts/cmk-manage.sh shell # root shell in the container +docker exec -it checkmk omd su cmk # or podman exec … +``` + +--- + +## 21. Security notes + +* Change `CMK_PASSWORD` in the interface right after installation and remove it + from `.env` (it is only evaluated when the site is created); `chmod 600 .env` + is already set. +* The Proxy Manager's default login (`admin@example.com` / `changeme`) must be + changed on first login. +* Make port **81** (admin interface) reachable from the administration network + only, and never expose port **5000**. +* Backups contain **all** passwords and certificates of the site – store them + encrypted or restrict access tightly (`chmod 700 backups`). +* Schedule `./scripts/cmk-manage.sh pull` + `up` regularly for image security + updates. +* Host networking means the container shares the host's network stack. There is + no additional network isolation – protection comes from the host firewall. + +--- + +## 22. Uninstalling + +```bash +sudo ./scripts/cmk-manage.sh down + +# systemd +sudo systemctl disable --now checkmk-stack.service +sudo rm -f /etc/systemd/system/checkmk-stack.service +# or Quadlet +sudo rm -f /etc/containers/systemd/checkmk.container \ + /etc/containers/systemd/nginx-proxy-manager.container +sudo systemctl daemon-reload + +# remove the images (names from "cmk-manage.sh image list") +docker image rm docker.io/checkmk/check-mk-raw:2.3.0-latest docker.io/jc21/nginx-proxy-manager:2 +# podman image rm … + +# remove a stored registry login if present +podman logout registry.checkmk.com # or docker logout + +# data (CAUTION: this deletes the entire monitoring setup!) +sudo rm -rf /opt/checkmk-stack/data +``` + +--- + +## Licence / notes + +This project contains only configuration and helper scripts. +Checkmk (Raw Edition, GPLv2) and NGINX Proxy Manager (MIT) are subject to the +licence terms of their respective vendors.