Hacker-Software/docker-podman-installation-with-nginx-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
58308264c6daf76599894c2833798e1bb62e9487
docker-podman-installation-…/README_en.md
T
duffyduckandClaude Opus 5 58308264c6 Absturzschleife des Proxy Managers: IP-Bereich-Abruf abschaltbar 
Das Containerprotokoll eines Kunden zeigte, was hinter dem 502 steckte:
der Dienst stuerzt in einer Schleife ab.

  [IP Ranges] Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
  Uncaught Error: Client network socket disconnected before secure TLS
                  connection was established
  ... node --abort_on_uncaught_exception ... index.js
  Starting backend ...          [und das immer wieder]

Beim Start holt der Proxy Manager IP-Bereiche von AWS und Cloudflare -
gebraucht wird das nur fuer Zugriffslisten mit deren Adressbereichen,
auf einem internen Server also nie. Der Code faengt Fehler dabei ab,
dieser kommt aber asynchron vom TLS-Socket und laeuft am catch vorbei;
weil node mit --abort_on_uncaught_exception laeuft, bricht der Prozess
ab, s6 startet neu, und nginx davor antwortet durchgehend mit 502.

Am Abbild nachgesehen und ausprobiert:

  - IP_RANGES_FETCH_ENABLED=false verhindert den Abruf; im Protokoll
    steht dann "IP Ranges fetch is disabled by environment variable",
    das Backend laeuft.
  - Scheitert der Abruf sauber (DNS-Fehler, Verbindung abgelehnt), faengt
    der Code ihn ab und der Dienst laeuft weiter - der Absturz haengt
    also wirklich an der gekappten TLS-Verbindung.
  - initTimer() laeuft unabhaengig vom Schalter und versucht es alle
    sechs Stunden erneut. Das steht als Hinweis samt extra_hosts in der
    README.

Neu: NPM_IP_RANGES_FETCH, Vorgabe false. Und die Fehlermeldung erkennt
den Fall selbst - sie zaehlt Startversuche und Ausnahmen im
Containerprotokoll und nennt den Schalter, statt den Benutzer weiter auf
"startet noch" warten zu lassen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 00:21:10 +02:00

3491 lines
139 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Checkmk in a container – Docker (Debian) & Podman (RHEL) with NGINX Proxy Manager
*[Deutsche Fassung: README.md](README.md)*
A complete, production-ready container installation of **Checkmk** with
**NGINX Proxy Manager** for HTTPS termination (Let's Encrypt *or* self-signed),
including tooling for **backup**, **restore** and the **migration of existing
sites from a bare-metal installation** – without having to re-register the
agents.
> **Note on language:** the scripts and their menus are in German. Literal
> program output and menu screenshots in this document are therefore shown
> unchanged; the surrounding text explains what they mean.
---
## Table of contents
1. [Overview & architecture](#1-overview--architecture)
2. [Directory layout](#2-directory-layout)
3. [Requirements & ports](#3-requirements--ports)
4. [Installation on Debian/Ubuntu (Docker)](#4-installation-on-debianubuntu-docker)
5. [Installation on RHEL/Rocky/Alma/Fedora (Podman)](#5-installation-on-rhelrockyalmafedora-podman)
* [5.1 Unregistered RHEL: where the packages come from](#51-unregistered-rhel-where-the-packages-come-from)
6. [Starting automatically after a host reboot (Podman)](#6-starting-automatically-after-a-host-reboot-podman)
7. [Configuration (`.env`)](#7-configuration-env)
8. [Obtaining the Checkmk image (Raw, Enterprise & co.)](#8-obtaining-the-checkmk-image-raw-enterprise--co)
9. [First start](#9-first-start)
10. [Setting up HTTPS – Let's Encrypt or self-signed](#10-setting-up-https--lets-encrypt-or-self-signed)
11. [Configuring the Proxy Manager locally (terminal browser)](#11-configuring-the-proxy-manager-locally-terminal-browser)
12. [Moving the admin interface to a different port](#12-moving-the-admin-interface-to-a-different-port)
13. [Firewall](#13-firewall)
14. [Management: `cmk-manage.sh` (CLI and ncurses interface)](#14-management-cmk-managesh)
15. [Backup](#15-backup)
16. [Restore](#16-restore)
17. [Migrating a bare-metal installation (`migrate_creator.sh`)](#17-migrating-a-bare-metal-installation-migrate_creatorsh)
18. [Several sites in one container](#18-several-sites-in-one-container)
19. [Upgrading Checkmk](#19-upgrading-checkmk)
20. [Troubleshooting](#20-troubleshooting)
21. [Security notes](#21-security-notes)
22. [Uninstalling](#22-uninstalling)
---
## 1. Overview & architecture
```
Internet / LAN
|
:80 :443 :81
|
+-----------------------------------+
| NGINX Proxy Manager (container) | network_mode: host
| - Let's Encrypt OR self-signed |
| - web interface on port 81 |
+-----------------------------------+
| http://127.0.0.1:5000
v
+-----------------------------------+
| Checkmk / OMD (container) | network_mode: host
| :5000 web GUI (Apache) |
| :8000 agent receiver | <-- agents (cmk-agent-ctl)
| :6557 Livestatus (optional) |
+-----------------------------------+
|
bind mount: ./data/checkmk/sites -> /omd/sites
```
**Deliberate design decisions**
| Topic | Implementation | Rationale |
|---|---|---|
| Checkmk on the host network | `network_mode: host`, **no** `ports:` entries | agents, SNMP traps, ICMP and the agent receiver work without NAT; the source IP of the monitored hosts is preserved |
| NGINX Proxy Manager also on the host network | `network_mode: host` | reaches the Checkmk backend directly via `127.0.0.1:5000`; no bridge network, no `host-gateway` tricks – behaves identically under Docker **and** Podman |
| No named volumes | **bind mounts** into `./data/` only | all data sits visibly in the project directory, easy to back up, copy and inspect |
| One compose file for both | `docker compose` and `podman-compose` | identical operation on Debian and RHEL |
| `:Z` on the bind mounts | SELinux relabelling | required on RHEL; a no-op on Debian/Docker (not an error) |
---
## 2. Directory layout
```
.
├── README.md # German documentation
├── README_en.md # this file
├── docker-compose.yml # for Docker AND Podman
├── .env.example # template -> .env
├── install.sh # installs Docker (Debian) / Podman (RHEL)
│
├── scripts/
│ ├── cmk-manage.sh # stack, backup, restore, import + ncurses GUI
│ ├── migrate_creator.sh # -> copy this to the SOURCE SERVER
│ └── lib/
│ └── common.sh # shared functions + TUI helpers
│
├── containers/
│ └── browser/
│ └── Containerfile # tools image: text browsers, curl, jq, openssl
│
├── systemd/
│ ├── checkmk-stack.service.example # compose-based unit (autostart)
│ └── quadlet/
│ ├── checkmk.container # Podman Quadlet (alternative)
│ └── nginx-proxy-manager.container
│
├── data/ # ALL payload data (bind mounts)
│ ├── checkmk/sites/ # -> /omd/sites (complete OMD sites)
│ └── npm/
│ ├── data/ # -> /data (NPM config/DB)
│ ├── letsencrypt/ # -> /etc/letsencrypt (certificates)
│ └── custom-certs/ # self-signed certificates ready to upload
│
├── backups/ # backups and migration bundles (*.tar.gz)
└── images/ # downloaded Checkmk image tarballs
```
---
## 3. Requirements & ports
* 64-bit Linux, at least 2 CPU cores and 4 GB RAM (recommended: 4 cores, 8 GB)
* Disk space: rule of thumb 1 GB per 100 monitored hosts and year, plus backups
* root privileges
* Outbound internet access to download the images
### Ports in use (host network!)
| Port | Service | Reachable from |
|---|---|---|
| 80/tcp | NGINX Proxy Manager – HTTP, Let's Encrypt challenge | internet/LAN |
| 443/tcp | NGINX Proxy Manager – HTTPS | internet/LAN |
| 81/tcp | NGINX Proxy Manager – **admin interface** (default; changeable via [`npm-port`](#12-moving-the-admin-interface-to-a-different-port)) | admin network only! |
| 5000/tcp | Checkmk Apache (backend) | **localhost only** – do not expose. Exception: with the Proxy Manager turned off ([12.4](#124-turning-the-proxy-manager-off-entirely-external-nginx)), open it for the external proxy's IP only |
| 8000/tcp | Checkmk agent receiver (TLS-registered agents) | monitored hosts |
| 6557/tcp | Livestatus (only with `CMK_LIVESTATUS_TCP=on`) | distributed sites |
> **Important:** because both containers run on the host network, ports 80, 443
> and 81 must not already be taken on the host (e.g. by an installed
> Apache/NGINX). Check with `ss -tlnp | grep -E ':(80|81|443|5000)\b'`.
---
## 4. Installation on Debian/Ubuntu (Docker)
```bash
# put the project where it should live
sudo mkdir -p /opt/checkmk-stack
sudo cp -r ./* ./.env.example /opt/checkmk-stack/
cd /opt/checkmk-stack
# install Docker CE + compose plugin, prepare the project
sudo ./install.sh --with-systemd
```
`install.sh` does the following:
1. adds the Docker repository from `download.docker.com` (GPG key + sources list)
2. installs `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`,
`docker-compose-plugin`
3. installs `dialog`, `openssl`, `tar`, `curl` (for the ncurses interface)
4. runs `systemctl enable --now docker`
5. creates the directories under `data/` and `backups/` and generates `.env`
from `.env.example`
6. optionally sets up the systemd unit `checkmk-stack.service`
Then:
```bash
sudo $EDITOR /opt/checkmk-stack/.env # at least CMK_PASSWORD and CMK_SITE_ID
# choose the image (Raw needs no login, Enterprise see chapter 8)
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest
sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status
```
Under Docker, `restart: unless-stopped` already brings the containers back
after a reboot (the Docker daemon starts them). The systemd unit is still
worthwhile because it additionally starts **all** sites
(see [chapter 18](#18-several-sites-in-one-container)).
---
## 5. Installation on RHEL/Rocky/Alma/Fedora (Podman)
Docker is not available on Red Hat systems – **Podman** is used instead. The
compose file is the same one.
> ### Step 0 on RHEL: check the subscription
>
> **An unregistered RHEL installs no packages at all** – neither `podman` nor
> `git`, `dialog` or `tar`. `dnf` merely reports *"This system is not registered
> with an entitlement server"*. This is the most common reason why `install.sh`
> does not get through on a fresh RHEL.
>
> ```bash
> # is the system registered?
> sudo subscription-manager status # -> "Overall Status: Current" = fine
> sudo dnf repolist # -> must not be empty
>
> # if not: create a free account at https://developers.redhat.com
> # (Developer Subscription for Individuals, up to 16 systems) and then:
> sudo subscription-manager register --username <user>
> ```
>
> RHEL 9 uses *Simple Content Access*, so `subscription-manager attach` is **no
> longer** required. Without an account it also works from the installation DVD
> or by switching to Rocky/Alma – all three routes are covered in detail in
> [section 5.1](#51-unregistered-rhel-where-the-packages-come-from).
>
> **Rocky Linux, AlmaLinux, CentOS Stream and Fedora** need none of this – just
> read on.
```bash
sudo mkdir -p /opt/checkmk-stack
sudo cp -r ./* ./.env.example /opt/checkmk-stack/
cd /opt/checkmk-stack
sudo ./install.sh --with-systemd
```
`install.sh` installs:
1. first checks whether a usable repository exists at all, and otherwise names
the three routes from [section 5.1](#51-unregistered-rhel-where-the-packages-come-from)
2. `podman`, `tar`, `openssl`
3. `dialog` for the ncurses interface – if that fails, `newt` (`whiptail`) is
used instead; the scripts support either
4. `podman-compose` – attempted in this order:
distribution repository → EPEL → `pip3 install podman-compose`
5. sets up the systemd unit (**always** with Podman, see the next chapter)
Done manually, that is:
```bash
sudo dnf install -y podman podman-compose tar openssl
sudo dnf install -y dialog || sudo dnf install -y newt # dialog: AppStream, newt: BaseOS
# if podman-compose is not in the repository - add EPEL:
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo dnf install -y podman-compose
```
> **EPEL requires a working subscription.** The `epel-release` package only
> configures the EPEL source; its dependencies and everything installed from it
> afterwards still come from Red Hat's BaseOS/AppStream. On an unregistered
> system even the command above therefore fails – EPEL is **not** a substitute
> for registering.
>
> If `podman-compose` stays out of reach because of this, it is no obstacle:
> with **Quadlet** ([chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose))
> the stack runs entirely without compose, straight through systemd.
Starting up:
```bash
sudo $EDITOR /opt/checkmk-stack/.env
# obtain the image - for Enterprise with credentials, see chapter 8
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
sudo ./scripts/cmk-manage.sh up # automatically uses podman-compose
sudo ./scripts/cmk-manage.sh status
```
> On RHEL, do read [chapter 8.2](#82-two-podman-pitfalls-the-ones-that-actually-bite):
> short image names and `podman import` instead of `podman load` are the two
> causes of "image not known".
### 5.1 Unregistered RHEL: where the packages come from
A freshly installed Red Hat Enterprise Linux without a subscription hands out
nothing at all:
```
$ sudo dnf install -y podman
This system is not registered with an entitlement server.
You can use "rhc" or "subscription-manager" to register.
Error: There are no enabled repositories in "/etc/yum.repos.d".
```
This affects **every** package – `podman`, `git`, `dialog`, `tar`. There are
three ways out, and `install.sh` now detects the situation and names them.
> Only RHEL itself is affected. **Rocky Linux, AlmaLinux, CentOS Stream and
> Fedora** need no registration – `dnf` works there straight away.
#### Route 1 – free Developer Subscription (recommended)
Red Hat hands out the *Developer Subscription for Individuals* free of charge.
It covers **up to 16 systems**, physical or virtual, and explicitly allows
**small production use** alongside development and testing. It is granted to
the individual, not to a company.
1. Create a free account at <https://developers.redhat.com>
2. Register the server:
```bash
sudo subscription-manager register --username <user>
# you will be prompted for the password; alternatively --password <password>
```
RHEL 9 uses *Simple Content Access* – the repositories are enabled right away,
a `subscription-manager attach` is **no longer** necessary.
3. Verify:
```bash
sudo subscription-manager status # -> Overall Status: Current
sudo dnf repolist # -> rhel-9-for-x86_64-baseos-rpms etc.
sudo dnf install -y git podman
```
If individual repositories are missing:
```bash
sudo subscription-manager repos --list-enabled
sudo subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
```
With a corporate account and an activation key, use this instead:
```bash
sudo subscription-manager register --org <org-id> --activationkey <key>
```
#### Route 2 – installation DVD as a local repository (no account, offline)
Works without any registration and without internet access. You need the
**DVD ISO** (`rhel-9.x-x86_64-dvd.iso`, several GB) – the small `boot.iso`
contains no packages.
```bash
# mount the ISO permanently
sudo mkdir -p /mnt/rhel
sudo mount -o loop,ro /path/rhel-9.8-x86_64-dvd.iso /mnt/rhel
# mount it automatically at boot
echo '/path/rhel-9.8-x86_64-dvd.iso /mnt/rhel iso9660 loop,ro,nofail 0 0' \
| sudo tee -a /etc/fstab
# create the repository file
sudo tee /etc/yum.repos.d/rhel-local.repo >/dev/null <<'REPO'
[rhel-local-baseos]
name=RHEL local - BaseOS
baseurl=file:///mnt/rhel/BaseOS
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[rhel-local-appstream]
name=RHEL local - AppStream
baseurl=file:///mnt/rhel/AppStream
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
REPO
sudo dnf clean all
sudo dnf repolist
sudo dnf install -y podman tar openssl dialog git
```
> **Limitation:** the local repository holds exactly the state of the ISO –
> there are **no security updates**. For a permanently running monitoring
> server that is not acceptable; as a stopgap until registration is in place it
> is fine.
#### Route 3 – switch to Rocky Linux or AlmaLinux
Both are binary-compatible rebuilds of RHEL, permanently free and without
registration. This project runs there unchanged – `install.sh` recognises them
as part of the RHEL family.
```bash
# Rocky Linux
sudo dnf install -y git
sudo git clone https://github.com/rocky-linux/rocky-tools.git
sudo bash rocky-tools/migrate2rocky/migrate2rocky.sh -r
# AlmaLinux
curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
sudo bash almalinux-deploy.sh
```
> Take a backup or a snapshot first. The conversion replaces the base packages
> on a running system.
#### What this project actually needs
| Package | Purpose | Repository |
|---|---|---|
| `podman` | container runtime | AppStream |
| `tar` | backups and migration bundles | BaseOS |
| `openssl` | self-signed certificates | BaseOS |
| `dialog` | ncurses interface | AppStream |
| `newt` (`whiptail`) | substitute if `dialog` is missing | BaseOS |
| `curl` | image download | BaseOS |
| `podman-compose` | compose support | EPEL or `pip3` |
| `git` | only needed to clone this project | AppStream |
`dialog` is **optional**: if it is missing, the scripts automatically use
`whiptail` from the `newt` package, which lives in BaseOS. `install.sh` tries
exactly this order and therefore no longer aborts.
`git` is not strictly required – the project can just as well be transferred as
a ZIP file or via `scp`.
> **EPEL does not help here:** the EPEL repository depends on Red Hat's base
> repositories. Without a subscription or a local repository you cannot even
> install `epel-release` – and therefore no `podman-compose`. In that case the
> Quadlet route from
> [chapter 6](#6-starting-automatically-after-a-host-reboot-podman) is the
> answer: it needs no compose at all.
### 5.2 SELinux
The bind mounts carry `:Z` in `docker-compose.yml`, so Podman automatically
applies the `container_file_t` context. Should access still fail:
```bash
sudo semanage fcontext -a -t container_file_t "/opt/checkmk-stack/data(/.*)?"
sudo restorecon -Rv /opt/checkmk-stack/data
sudo ausearch -m avc -ts recent # inspect AVC messages
```
---
## 6. Starting automatically after a host reboot (Podman)
> **This is the most important difference from Docker:** Podman has *no
> daemon*. `restart: unless-stopped` in the compose file therefore does **not**
> survive a reboot. Autostart has to go through systemd. There are two proven
> ways.
### Option A – compose-based systemd unit (recommended, identical to Docker)
Created and enabled automatically by `./install.sh --with-systemd`. Manually:
```bash
sudo tee /etc/systemd/system/checkmk-stack.service >/dev/null <<'UNIT'
[Unit]
Description=Checkmk + NGINX Proxy Manager (container stack)
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/checkmk-stack
EnvironmentFile=/opt/checkmk-stack/.env
TimeoutStartSec=0
ExecStart=/bin/sh -c '/usr/bin/podman-compose up -d'
ExecStartPost=/opt/checkmk-stack/scripts/cmk-manage.sh start-all-sites
ExecStop=/bin/sh -c '/usr/bin/podman-compose down'
[Install]
WantedBy=multi-user.target
UNIT
sudo systemctl daemon-reload
sudo systemctl enable --now checkmk-stack.service
sudo systemctl status checkmk-stack.service
```
Verify that it survives a reboot:
```bash
sudo systemctl is-enabled checkmk-stack # -> enabled
sudo reboot
# after the restart:
sudo podman ps
sudo /opt/checkmk-stack/scripts/cmk-manage.sh status
```
`ExecStartPost` calls `start-all-sites`. This is necessary because the Checkmk
entrypoint only starts the site named in `CMK_SITE_ID` – with several sites the
remaining ones would stay stopped.
### Option B – Podman Quadlet (Podman 4.4 and newer, no compose)
Quadlet turns simple description files into real systemd units. Advantages: no
`podman-compose` required, clean dependencies, native Podman.
```bash
sudo ./install.sh --quadlet
```
Using the values from `.env`, this writes to `/etc/containers/systemd/`:
* `checkmk.container`
* `nginx-proxy-manager.container`
Then:
```bash
sudo systemctl daemon-reload
sudo systemctl start checkmk nginx-proxy-manager
sudo systemctl status checkmk
```
`WantedBy=multi-user.target default.target` makes the units start
automatically at boot – a separate `systemctl enable` is neither needed nor
possible with Quadlet, because the units are generated.
Templates live in [systemd/quadlet/](systemd/quadlet/).
### Option C – rootless Podman (optional, more effort)
```bash
sudo ./install.sh --rootless
sudo loginctl enable-linger checkmk # service account
# as a user unit:
mkdir -p ~/.config/systemd/user
cp systemd/checkmk-stack.service.example ~/.config/systemd/user/checkmk-stack.service
systemctl --user daemon-reload
systemctl --user enable --now checkmk-stack
```
`--rootless` sets `net.ipv4.ip_unprivileged_port_start=80` so that ports 80/443
may be bound without root. `loginctl enable-linger` makes user units start
after a reboot even when nobody is logged in.
> Note: in rootless mode the UIDs of the site users are mapped through user
> namespaces. Files under `data/checkmk/sites` then belong to subordinate UIDs
> (e.g. `165536`). That is normal, but it makes working with the files from the
> host awkward. For migrations, **rootful** operation is recommended.
---
## 7. Configuration (`.env`)
```bash
cp .env.example .env
chmod 600 .env
$EDITOR .env
```
The most important settings:
| Variable | Meaning |
|---|---|
| `CMK_RUNTIME` | empty = auto-detect, otherwise `docker` or `podman` |
| `CMK_IMAGE` | **the image actually used**, always fully qualified – see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) |
| `CMK_EDITION`, `CMK_VERSION` | inputs for the image downloader only |
| `CMK_IMAGE_SOURCE` | `auto`/`registry`/`download` – where the image is fetched from |
| `CMK_REGISTRY_USER/-PASSWORD` | login for `registry.checkmk.com` (Enterprise) |
| `CMK_DOWNLOAD_USER/-PASSWORD` | login for the Checkmk download portal |
| `CMK_SITE_ID` | the site the container creates and starts |
| `CMK_PASSWORD` | initial password for user `cmkadmin` (only on creation) |
| `CMK_LIVESTATUS_TCP` | `on` opens port 6557 for distributed monitoring |
| `MAIL_RELAY_HOST` | SMTP relay for notifications |
| `TZ` | time zone, e.g. `Europe/Berlin` |
| `NPM_ENABLED` | `on`/`off` – should the Proxy Manager container run at all? See [12.4](#124-turning-the-proxy-manager-off-entirely-external-nginx) |
| `NPM_ADMIN_PORT` | port of the Proxy Manager interface (default 81) – change only via `npm-port` or the menu |
| `NPM_API_HOST` | name the API is addressed by (`127.0.0.1` or `localhost`) – see [chapter 14.6](#146-networks-with-a-mandatory-proxy-an-api-error-that-is-none) |
| `NPM_IP_RANGES_FETCH` | fetch IP ranges from AWS/Cloudflare (default `false`) – see [chapter 10.1.2](#1012-the-service-crashes-in-a-loop) |
| `NPM_ADMIN_URL` | start page of the terminal browser; empty = derived from `NPM_ADMIN_PORT` |
| `BACKUP_DIR` | location for **manual** backups (default `./backups`); scheduled jobs have their own paths |
| `BACKUP_AUTO` | `on`/`off` – should the backup jobs run via cron? |
| `CERT_DOMAIN`, `CERT_DAYS` | defaults for self-signed certificates |
| `CERT_EXTERNAL_DIR`, `CERT_EXTERNAL_FULLCHAIN`, `CERT_EXTERNAL_KEY` | directory and file names of an externally delivered certificate |
| `CERT_EXTERNAL_WATCH` | check daily whether it was replaced |
| `FW_ROLE` | `single`/`central`/`remote` – role in distributed monitoring, drives the firewall check |
| `FW_EC_SYSLOG`, `FW_EC_SNMPTRAP`, `FW_EC_TCP` | Event Console: syslog (514), SNMP traps (162/udp), status over TCP (6558) |
| `FW_AGENT` | does a Checkmk agent run on this host in pull mode? (6556) |
| `FW_EXTRA_PORTS` | additional ports for the check, e.g. `3000/tcp=Grafana` |
### 7.1 Who reads `.env` – and why does it exist alongside the script?
`.env` is the **only** configuration file. There is deliberately no second
configuration inside the management script:
```
.env (single source of truth)
|
+---------------+----------------+------------------+
| | | |
docker-compose systemd unit cmk-manage.sh migrate_creator.sh
(variable (EnvironmentFile) (source) (independent, runs on
substitution) the source server)
```
* **Read** by compose, by the systemd unit and by the script.
* **Written** by the script – either specifically (`image …` sets `CMK_IMAGE`,
`import --set-default-site` sets `CMK_SITE_ID`, `npm-port` sets
`NPM_ADMIN_PORT`) or through the settings menu / `config set`
(see [chapter 14](#14-management-cmk-managesh)). An editor is no longer
necessary, but remains available as a menu entry.
* The script itself only holds **fallback values** for the case that a key is
missing from `.env` (in `scripts/lib/common.sh`, function `load_env`). It
never overwrites a value that is set.
In short: `.env` says *what* runs, the script *does* something with it. Nothing
has to be maintained twice.
---
## 8. Obtaining the Checkmk image (Raw, Enterprise & co.)
### 8.1 Which edition lives where?
This is the most common pitfall: **the Enterprise edition is not on Docker
Hub.** The `checkmk` namespace on Docker Hub contains only:
| Edition (`CMK_EDITION`) | Source | Login |
|---|---|---|
| `raw` (up to 2.4) | `docker.io/checkmk/check-mk-raw` | no |
| `community` (2.5 and later, successor of raw) | `docker.io/checkmk/check-mk-community` | no |
| `cloud` | `docker.io/checkmk/check-mk-cloud` | no |
| `managed` (up to 2.4) | `docker.io/checkmk/check-mk-managed` | no |
| `pro` (2.5 and later, successor of enterprise) | `docker.io/checkmk/check-mk-pro` | no |
| `ultimate`, `ultimatemt` (2.5 and later) | `docker.io/checkmk/check-mk-ultimate[mt]` | no |
| **`enterprise` (2.0 – 2.4)** | **`registry.checkmk.com/enterprise/check-mk-enterprise`** or an image tarball from the customer portal | **yes** |
| `free` (historic, up to 2.1) | `docker.io/checkmk/check-mk-free` | no |
With Checkmk 2.5 the editions were renamed; those images are publicly available
on Docker Hub again. So if you run **2.5 or newer**, simply use `pro`. For
**2.2/2.3/2.4 Enterprise** the only routes are `registry.checkmk.com` or the
tarball.
#### Beware of the names
The rename is the second big pitfall. **checkmk.com only shows the new names,
while files and repositories keep the old ones up to 2.4.** For 2.3.0p9
specifically – verified against the version archive:
| Choice on checkmk.com | `--edition` here | actual file name |
|---|---|---|
| Community | `raw` | `check-mk-raw-docker-2.3.0p9.tar.gz` |
| **Pro** | **`enterprise`** | **`check-mk-enterprise-docker-2.3.0p9.tar.gz`** |
| Ultimate | `cloud` | `check-mk-cloud-docker-2.3.0p9.tar.gz` |
| Ultimate MT | `managed` | `check-mk-managed-docker-2.3.0p9.tar.gz` |
Picking "Pro" on the website and deriving
`check-mk-pro-docker-2.3.0p9.tar.gz` from it yields an **HTTP 404** – that file
does not exist for 2.3.
The script takes care of this: it reads the real file name from the version
archive and additionally tries both naming variants. In the interface the
edition list shows both names.
> Freely downloadable does not mean licence-free – the licence key is entered
> inside Checkmk afterwards.
### 8.2 Two Podman pitfalls (the ones that actually bite)
**1. Short image names.** Podman has no default registry. Short names such as
`checkmk/check-mk-enterprise:2.3.0p23` are resolved through
`unqualified-search-registries` in `/etc/containers/registries.conf` – and on
RHEL `registry.access.redhat.com` is listed **first**. Podman therefore looks
for the image at Red Hat, does not find it and reports `image not known`, or
asks interactively. That is why `.env` and `docker-compose.yml` consistently
use the **fully qualified** name:
```
CMK_IMAGE=registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p23
```
Inspect the current resolution order:
```bash
grep -A3 unqualified-search-registries /etc/containers/registries.conf
```
**2. `import` instead of `load`.** A downloaded image tarball is installed with
**`load`**:
| Command | Expects | Result |
|---|---|---|
| `podman import file.tar.gz` | a **filesystem** archive (rootfs) | an image **without** entrypoint, environment or tag → unusable, "image unknown" |
| `podman load -i file.tar.gz` | an **image** archive (`docker save`/OCI) | a complete image including metadata and tag ✔ |
Getting rid of a broken image created by `import`:
```bash
podman images | grep '<none>'
podman rmi <image-id>
```
`cmk-manage.sh image load` checks the archive for `manifest.json` or
`oci-layout` beforehand and rejects a wrong archive with exactly this hint.
### 8.3 Storing credentials in `.env`
```bash
CMK_EDITION=enterprise
CMK_VERSION=2.3.0p23
# route 1 - private registry
CMK_REGISTRY=registry.checkmk.com
CMK_REGISTRY_USER=<customer-portal-user>
CMK_REGISTRY_PASSWORD=<password>
# route 2 - tarball from the download portal
CMK_DOWNLOAD_BASEURL=https://download.checkmk.com/checkmk
CMK_DOWNLOAD_USER=<user>
CMK_DOWNLOAD_PASSWORD=<password>
```
The credentials come from the Checkmk customer portal (portal.checkmk.com).
`.env` should therefore be `chmod 600` – `install.sh` takes care of that.
### 8.4 Testing the credentials first
Rather than having a multi-gigabyte download fail on a wrong login, both can be
checked in seconds:
```bash
./scripts/cmk-manage.sh image test --edition enterprise --version 2.3.0p23
```
Nothing is downloaded and no login is stored. What gets checked:
| Check | What it answers |
|---|---|
| registry reachable | network, proxy, DNS |
| login accepted | are the username and password correct? |
| access to the repository | does the account have the required entitlement (subscription)? |
| download portal | is the image tarball there and is the login accepted? |
Example with correct credentials from `.env`:
```
Registry registry.checkmk.com
[ OK ] erreichbar (HTTP 401)
[ OK ] Anmeldung akzeptiert
[ OK ] Zugriff auf 'enterprise/check-mk-enterprise' vorhanden
Downloadportal download.checkmk.com
[ OK ] Datei erreichbar (HTTP 200)
https://download.checkmk.com/checkmk/2.3.0p23/check-mk-enterprise-2.3.0p23.docker.tar.gz
[ OK ] Zugangsweg nutzbar.
```
With a wrong password:
```
Registry registry.checkmk.com
[ OK ] erreichbar (HTTP 401)
[FEHLER] Anmeldung abgelehnt (HTTP 401)
Benutzername/Kennwort des Checkmk-Kundenportals pruefen
```
(`erreichbar` = reachable, `Anmeldung akzeptiert` = login accepted,
`Anmeldung abgelehnt` = login rejected, `Zugangsweg nutzbar` = a route works.)
Useful distinctions the test makes:
* **Login accepted, but no access to the repository** – the credentials are
correct, but the subscription lacks the entitlement for this edition.
* **Login accepted, but HTTP 404 on the download portal** – username and
password are right, only the version or file name does not match. Give the
exact version (no `-latest`) or set `CMK_DOWNLOAD_URL`.
* For the freely available editions (`raw`, `pro`, `cloud`, …) the test reports
"public repository – no login required" and still verifies that the
repository can actually be reached.
Options:
```bash
# supply credentials once without writing them to .env
./scripts/cmk-manage.sh image test --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
# check only one route
./scripts/cmk-manage.sh image test --only registry
./scripts/cmk-manage.sh image test --only download
```
The credentials are passed to `curl` via `curl --config -` on standard input
and therefore do **not** appear in the process list.
> **Passwords with special characters:** this used to go through a temporary
> `.netrc` file. Its format separates fields by whitespace, so a password
> containing spaces, `#` or quotes arrived silently mangled and the login
> failed for no apparent reason. Measured against a test server: with a
> password containing a space, `#`, `"` and `$`, the new route returns HTTP
> 200 and the old one HTTP 401. If `curl` is missing, the registry check
falls back to `podman login` / `docker login` – into a throwaway file as well,
so the system's stored login is left untouched.
In the ncurses interface the test can be reached from two places:
* **main menu → image → "Zugangsdaten prüfen"**
* **Einstellungen → "Checkmk-Image und Zugangsdaten" →
">> Zugangsdaten jetzt prüfen"** – right where the username and password
are entered
After changing a credential field the menu also offers to run the test
straight away.
### 8.5 Listing the available versions
Instead of guessing version numbers:
```bash
./scripts/cmk-manage.sh image versions --edition enterprise --filter 2.3.0p
```
Up to three sources are queried:
| Source | `--source` | Credentials |
|---|---|---|
| version archive on checkmk.com | `web` | **no** |
| registry (Docker Hub or registry.checkmk.com) | `registry` | only for Enterprise |
| download portal (directory listing) | `download` | yes |
> **The registry and the download portal do not carry the same thing.** The
> registry usually keeps only the **most recent** patch levels – for `2.4.0`
> roughly `p31` upwards, plus `2.4.0-latest` and the daily builds. Older levels
> have rolled out of it. The download portal, by contrast, is an **archive** and
> still has them.
>
> This matters precisely when a **specific** version is required – during a
> migration, because `omd restore` demands exactly the version of the source
> site ([chapter 17.4](#174-step-3--bring-the-target-system-to-the-right-version)).
> For those cases the tarball is the right route, not the registry. `image why`
> points this out by itself when a tag is missing from the registry but the
> version exists in the archive.
Without `--source` all three are tried – so even without stored credentials you
get a usable list:
```
Versionsarchiv https://checkmk.com/download/archive
(oeffentlich, keine Zugangsdaten noetig)
2.3.0p50
2.3.0p49
2.3.0p48
...
Uebernehmen
aus der Registry : ./scripts/cmk-manage.sh image pull --edition enterprise --version <version>
als Tarball : ./scripts/cmk-manage.sh image download --edition enterprise --version <version>
nur eintragen : ./scripts/cmk-manage.sh config set CMK_VERSION <version>
```
Further options: `--limit N` (default 25), `--all`, `--filter TEXT` and
`--bare` for the plain list without any framing – handy for scripting:
```bash
./scripts/cmk-manage.sh image versions --source web --filter 2.3.0p --bare | head -1
```
In the interface there is **image → "Verfügbare Versionen auflisten"**; and
when fetching an image the version no longer has to be typed at all – a
selection list appears instead.
### 8.6 Fetching the image – one command, three routes
Once edition and version are set, this is enough:
```bash
sudo ./scripts/cmk-manage.sh image get
```
Where it fetches from is controlled by **`CMK_IMAGE_SOURCE`**:
| Value | Effect |
|---|---|
| `auto` (default) | registry first, then the download portal |
| `registry` | registry only |
| `download` | tarball from the download portal only |
Deviate once with `image get --source download`.
`auto` is the sensible default because registries only keep the most recent
patch levels – an older version needed for a migration then only exists in the
download portal. The switch happens automatically, without asking.
**`up` fetches automatically.** If the image is missing locally and edition and
version are set, `up` obtains it first through this route – otherwise compose
attempts a plain pull, which must fail for a version that only exists in the
archive:
```
[ INFO ] Das Image ist lokal nicht vorhanden - hole es (auto) ...
[ INFO ] Bezugsweg 'auto': zuerst die Registry ...
[ WARN ] Aus der Registry nicht zu bekommen - versuche das Downloadportal.
```
`up --no-fetch` suppresses it.
In the interface this is under **image → "Image holen"**. Only the source is
asked for there – the configured one is marked. To change edition or version,
pick **"Etwas anderes waehlen"** or the wizard directly.
#### The individual steps
> **In the interface this is a wizard:** **image → "Image beschaffen"** walks
> through edition, version and source in one go, then sets `CMK_EDITION`,
> `CMK_VERSION` **and** `CMK_IMAGE`, and does **not** ask again for credentials
> already stored in `.env`. It only asks if they are missing for the chosen
> route – and offers to save them. The same wizard is also available under
> **Einstellungen → Checkmk-Image und Zugangsdaten → ">> Image beschaffen"**.
On the command line the individual steps remain:
**a) From a registry (recommended)**
For the Enterprise edition this is exactly the following two steps – the script
performs them in one call and writes the result into `.env`:
```bash
podman login registry.checkmk.com # or docker login
podman pull registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p9
```
```bash
# freely available, no login
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest
# Enterprise with a login to registry.checkmk.com
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
```
**b) Image tarball from the download portal (with credentials)**
```bash
sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
```
The downloader tries the usual file names under
`<CMK_DOWNLOAD_BASEURL>/<version>/`
(`check-mk-enterprise-<version>.docker.tar.gz` and
`check-mk-enterprise-docker-<version>.tar.gz`). Should Checkmk have changed the
scheme, the full URL can be given directly:
```bash
sudo ./scripts/cmk-manage.sh image download \
--url 'https://download.checkmk.com/checkmk/2.3.0p23/check-mk-enterprise-2.3.0p23.docker.tar.gz' \
--user <user> --password <password>
```
The credentials are handed to `curl` via `curl --config -` and therefore do
not appear in the process list.
**c) Loading a file you already downloaded**
If you fetched the file from the customer portal with a browser:
```bash
sudo ./scripts/cmk-manage.sh image load --file /root/check-mk-enterprise-2.3.0p23.docker.tar.gz
```
In all three cases the script determines the **actual** image name (for `load`
typically `docker.io/checkmk/check-mk-enterprise:2.4.0p9`) and writes it to
`CMK_IMAGE` in `.env`. Afterwards:
```bash
sudo ./scripts/cmk-manage.sh up
```
### 8.7 `CMK_IMAGE` versus `CMK_EDITION`/`CMK_VERSION`
These three values have different jobs, and confusing them is a common
mistake:
| Value | Who reads it | What for |
|---|---|---|
| **`CMK_IMAGE`** | **compose and the systemd units** | the image that actually starts |
| `CMK_EDITION` | only `cmk-manage.sh image …` | input for the downloader |
| `CMK_VERSION` | only `cmk-manage.sh image …` | input for the downloader |
`CMK_IMAGE` can **not** be derived from edition and version – depending on how
you obtained it, the very same version looks completely different:
```
docker.io/checkmk/check-mk-raw:2.3.0-latest (Docker Hub)
registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p9 (registry)
docker.io/checkmk/check-mk-enterprise:2.4.0p9 (loaded from a tarball)
```
> **The name after `image download` is surprising.** The tarball carries the
> unqualified name `checkmk/check-mk-enterprise:<version>` internally, and
> Podman expands that to `docker.io/` when loading – even though the image does
> not exist there. That is correct and harmless: the image is local and `up`
> uses it. It just could not be **pulled** under that name. Should it ever go
> missing, `image get` fetches it from the download portal again – not via
> `podman pull`.
So `CMK_IMAGE` records what **actually applies**. It is written by
`image pull`, `image download`, `image load` and `image use` – you never have to
type anything there by hand. In the interface the three values are therefore not
editable individually: the wizard sets them together. On the command line they
remain individually settable (`config set …`).
**Recording it without downloading anything.** The reference can be built from
edition and version – no download required:
```bash
./scripts/cmk-manage.sh image use --edition enterprise --version 2.4.0p9
# -> CMK_IMAGE=registry.checkmk.com/enterprise/check-mk-enterprise:2.4.0p9
# CMK_EDITION=enterprise
# CMK_VERSION=2.4.0p9
```
The image itself is fetched by the runtime on the next `up`. In the wizard this
route is called **"Nur eintragen – geholt wird beim naechsten Start"**. If the
registry needs a login (Enterprise), that has to have happened once – either via
the "Registry holen" route or with `podman login registry.checkmk.com`.
> **All four wizard routes set `CMK_IMAGE`** – they only differ in whether
> something is fetched right away. The three values therefore stay consistent by
> themselves, and the mismatch warning no longer appears in normal operation.
On the command line, `image pull` and `image download` take `--remember` to
store the edition and version in `.env` as well:
```bash
./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p9 --remember
```
> **Consequence:** changing the edition or version does **not** start a
> different release. Only `image pull` (or `download`/`load`) writes the result
> to `CMK_IMAGE`, and only `up` recreates the container.
To stop these from drifting apart unnoticed, the script compares the three
values – in `image status`, in `status`, and in the settings menu right after
the edition or version is changed:
```
[ WARN ] CMK_IMAGE passt nicht zu den eingestellten Vorgaben.
verwendet wird : docker.io/checkmk/check-mk-raw:2.3.0-latest
Edition : eingestellt "enterprise", im Image steckt "raw"
...
./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0-latest
./scripts/cmk-manage.sh up
```
("CMK_IMAGE does not match the configured defaults / in use / configured … but
the image contains …".) Images loaded from a tarball (`localhost/…`) count as
consistent as long as name and tag match.
**If fetching fails**, `CMK_IMAGE` stays unchanged – deliberately, because the
old image keeps running. The wizard then says so explicitly ("Image wurde NICHT
geholt") instead of merely pointing at the mismatch. It still leaves a state in
which edition and version claim something different from the image in use.
There are two ways out, both in the menu underneath the three values:
| Route | Direction | Effect |
|---|---|---|
| **Assistent starten** (wizard) | – | walks through edition, version and source |
| **Image aus den Vorgaben setzen** (set from defaults) | edition/version → image | builds the reference and records it **without fetching anything**; the menu entry shows what it will be |
| **Vorgaben aus dem Image ableiten** (derive defaults) | image → edition/version | writes edition and version back from `CMK_IMAGE`; nothing about the running setup changes |
On the command line the second one is:
```bash
./scripts/cmk-manage.sh image use "$CMK_IMAGE" --remember
```
### 8.8 Finding the registry path
"Login Succeeded" does not yet mean the path is right. If the pull reports
`unable to copy from source docker://…`, the repository does not exist under
that name, or the subscription does not cover it:
```bash
./scripts/cmk-manage.sh image repos --edition enterprise
```
This first queries the catalog (`/v2/_catalog`) – which many registries block –
and then probes the usual paths:
```
Uebliche Pfade fuer "raw" durchprobiert
STATUS TAGS PFAD
200 = nutzbar, 404 = gibt es nicht, 401/403 = kein Zugriff
401 - raw/check-mk-raw (kein Zugriff - oder gibt es nicht)
200 2394 checkmk/check-mk-raw
...
[ OK ] Nutzbar: docker.io/checkmk/check-mk-raw
```
If an `image pull` against a private registry fails, this probe runs
automatically and shows what is actually there.
> **401 does not necessarily mean "it exists".** Many registries – Docker Hub
> among them – answer 401 for non-existent repositories too, so as not to
> reveal whether they exist. Only **200 = usable** and **404 = does not exist**
> are reliable.
If no path works, the tarball route remains. For Enterprise it is the
documented one anyway, and it works independently of the registry:
```bash
./scripts/cmk-manage.sh image download --edition enterprise --version 2.4.0p9
```
### 8.9 Checking what is in effect
```bash
./scripts/cmk-manage.sh image status # configured image + local images
./scripts/cmk-manage.sh image list # local Checkmk images only
./scripts/cmk-manage.sh image use localhost/checkmk/check-mk-enterprise:2.3.0p23
```
`image status` explicitly warns when `CMK_IMAGE` is a short name without a
registry.
The ncurses interface offers the same workflow under the menu entry
*"Checkmk-Image beschaffen/auswählen"* (pick edition → version → credentials →
fetch).
---
## 9. First start
```bash
sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status
```
Output (example, German as printed by the tool):
```
Container
checkmk läuft
nginx-proxy-manager läuft
Sites im Container
SITE VERSION STATUS
cmk 2.3.0p23.cre läuft
Zugang
Checkmk (direkt) : http://10.0.0.5:5000/cmk/
Checkmk (über Proxy) : https://<FQDN>/cmk/
Agent-Receiver : 10.0.0.5:8000
NGINX Proxy Manager : http://10.0.0.5:81/
```
(`läuft` = running, `gestoppt` = stopped, `Zugang` = access, `Erstlogin` =
initial login.) The port of the admin interface follows `NPM_ADMIN_PORT` from
`.env`; the output always shows the value actually configured.
Logging in to Checkmk: user `cmkadmin`, password from `CMK_PASSWORD`.
---
## 10. Setting up HTTPS – Let's Encrypt or self-signed
### 10.1 The administrator account
> **The former default credentials `admin@example.com` / `changeme` no longer
> exist as of version 2.15.** The container only creates the first account if
> `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD` are set at the **first**
> start – otherwise the browser shows a setup wizard. Verified in
> `backend/setup.js` of v2.15.1 and against a running container.
That is why the values live in `.env` and are passed through by
`docker-compose.yml`:
```bash
NPM_ADMIN_EMAIL=admin@example.com
NPM_ADMIN_PASSWORD=a-proper-password
```
Set both **before the first start**. The account then exists right away – and
with it the REST interface used by
[chapter 10.2](#102-creating-a-proxy-host--without-a-browser).
If the stack has already run and the account was created by hand through the
wizard, the variables no longer take effect (the database is no longer empty).
Simply enter the credentials you chose – the script logs in with those.
#### When the login is missing
If the credentials are absent, the script tells you what to do depending on the
situation – it checks whether `data/npm/data/database.sqlite` already exists:
| Situation | Route |
|---|---|
| **no database yet** | `config set NPM_ADMIN_EMAIL …` and `…_PASSWORD …`, then `up` – the container creates the account itself |
| **database present** | create the account in the browser (`cmk-manage.sh browser`), then enter the same credentials with `config set` |
To start over cleanly – in the interface under **Einstellungen → NGINX Proxy
Manager → ">> Datenbank zuruecksetzen"**, or from the **Proxy-Hosts** menu, or
on the command line:
```bash
sudo ./scripts/cmk-manage.sh npm-reset
```
This moves the database aside (renamed, not deleted) so the container recreates
the account from `NPM_ADMIN_EMAIL`/`NPM_ADMIN_PASSWORD` on the next `up`.
**Proxy hosts, redirections and access lists are gone** – the certificates
remain. The command shows exactly that beforehand and asks for confirmation.
The proxy hosts can be recreated with `proxy add` in a single call.
### 10.1.1 When the login fails
The message now names the cause itself – it shows the HTTP code, the API's
reply and, if nothing answered at all, what curl reported:
```
[FEHLER] Anmeldung am Proxy Manager fehlgeschlagen.
Aufgerufen : http://127.0.0.1:81/api/tokens
Konto : admin@example.com
HTTP-Code : 400
Antwort : Invalid email or password
Woran es liegen kann
Der Proxy Manager hat geantwortet und die Anmeldung abgelehnt.
Der Weg dorthin stimmt also - Port, Netz und Proxy sind in Ordnung.
```
That is half the battle: **if it answered, the network is not the problem.**
The other direction looks like this:
| HTTP code | meaning | next step |
|---|---|---|
| `400`/`401` | the Proxy Manager refuses | account/password in `.env`, or the account does not exist → `npm-reset` |
| `000` | no answer at all | container down? port free? – the command checks both and says which applies |
| `502` | two possible senders – see below | the `Antwortete:` line reveals which |
| `5xx` + HTML | something else answered | usually the HTTP proxy; the first line of the HTML reply is shown |
**A `502` has two senders**, and they feel identical. The `Server` header,
which the message prints, tells them apart:
| `Antwortete:` | who it was | what to do |
|---|---|---|
| `openresty` (no `Via:`) | **the Proxy Manager itself** – nginx is up, the service behind it is not | usually it is still starting: after a restart it takes about **ten seconds** for the API to answer. Otherwise `logs` or `proxy restart` |
| `squid`, `Via: …` etc. | **an HTTP proxy in front** | `noproxy` – the request never reached the Proxy Manager |
The second case is treacherous, because the proxy produces the 502 while trying
to reach `localhost:81` **from its own machine**. One call tells them apart:
```bash
curl -si --noproxy '*' http://localhost:81/api/tokens -X POST \
-H 'Content-Type: application/json' -d '{}' | head -3
```
If that differs from the result without `--noproxy`, the proxy was in between.
If HTML arrives instead of JSON, it says so explicitly – including the first
line, which reveals who answered:
```
Die Antwort ist HTML, kein JSON - sie stammt also nicht von der
Schnittstelle des Proxy Managers, sondern von etwas davor
(HTTP-Proxy, anderer Dienst auf dem Port). Erste Zeile:
<html><head><title>502 Proxy Error</title></head>...
```
### 10.1.2 The service crashes in a loop
If the container log looks like this, the `502` is explained:
```
[IP Ranges] › info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
Uncaught Error: Client network socket disconnected before secure TLS connection was established
at TLSSocket.emit (node:events:497:7)
... node --abort_on_uncaught_exception ... index.js
❯ Starting backend ...
[over and over]
```
At startup the Proxy Manager fetches IP ranges from AWS and Cloudflare – needed
**only** for access lists using those address ranges, so never on an internal
network. The code does catch errors there, but this one arrives asynchronously
on the TLS socket and bypasses the `catch`. Since node runs with
`--abort_on_uncaught_exception`, the process aborts, s6 restarts it – and nginx
in front answers `502` the whole time.
It is triggered by networks that **accept the connection and then cut it**
(TLS-intercepting proxies, firewalls). If the fetch fails cleanly – DNS error,
connection refused – nothing happens; the service keeps running.
The fix:
```bash
./scripts/cmk-manage.sh config set NPM_IP_RANGES_FETCH false
./scripts/cmk-manage.sh up
```
The log then says `IP Ranges fetch is disabled by environment variable`.
**This project defaults to `false`** – the fetch gains an internal monitoring
server nothing and can take it down.
> One remainder: `initTimer()` runs regardless of the switch and retries **every
> six hours**. To rule that out as well, point the names nowhere – then the
> fetch fails cleanly and is caught:
>
> ```yaml
> extra_hosts:
> - "ip-ranges.amazonaws.com:127.0.0.1"
> - "www.cloudflare.com:127.0.0.1"
> ```
The error message recognises this case by itself – it counts start attempts and
exceptions in the container log:
```
Der Dienst startet nicht nur - er stuerzt in einer Schleife ab.
Im Protokoll: 4 Startversuche, 4 unbehandelte Ausnahmen.
Die Ausnahmen kommen vom Abruf der IP-Bereiche bei AWS und
Cloudflare. ... Abschalten:
cmk-manage.sh config set NPM_IP_RANGES_FETCH false
```
### 10.2 Creating a proxy host – without a browser
The web interface is painful in a terminal browser, and forms with file uploads
do not work there at all. Since the Proxy Manager runs locally anyway, its
interface is the more reliable route:
```bash
# publish Checkmk at checkmk.example.com with a self-signed certificate
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --cert self
# or with Let's Encrypt (port 80 must be reachable from the internet)
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --cert letsencrypt
# forwarding to a different port instead of 5000
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --forward-port 8080
```
In one go the command: creates the certificate (or requests it from Let's
Encrypt), uploads it to the Proxy Manager, creates the proxy host, forces
HTTPS, enables HTTP/2 and websockets, and sets the recommended
`advanced_config` block.
| Command | Effect |
|---|---|
| `proxy status` | check the login, count hosts and certificates |
| `proxy list` | existing proxy hosts |
| `proxy certs` | existing certificates with expiry date |
| `proxy add …` | create a proxy host (`--update` replaces an existing one) |
| `proxy show --domain X` | details including additional paths |
| `proxy del --id N` | delete a proxy host |
| `proxy edit --id N …` | change an existing proxy host – only the fields you name |
| `proxy default --domain X` | enable HTTPS on the bare IP (mirrors X into a default server) |
| `proxy default --off` | remove that mirror again |
| `proxy admin --domain X` / `--root` | make the admin interface reachable – by name or via the root |
| `proxy check` | does something answer behind each path? (checked from inside the proxy container) |
| `proxy reload` | make nginx re-read its configuration – without interruption |
| `proxy restart` | restart the Proxy Manager container |
Options for `proxy add`:
| Option | Default | Meaning |
|---|---|---|
| `--domain FQDN` | `PROXY_DOMAIN` | public name |
| `--forward-host H` | `127.0.0.1` | forwarding target |
| `--forward-port P` | `5000` | target port – **freely selectable**, not fixed to Checkmk |
| `--scheme http\|https` | `http` | scheme towards the target |
| `--cert self\|letsencrypt\|none` | `self` | certificate |
| `--update` | – | replace an existing entry |
| `--advanced TEXT` | see below | custom nginx block |
| `--location /P=HOST:PORT` | – | another service under a path, repeatable – see [10.3](#103-several-services-behind-one-port-paths-instead-of-names) |
| `--no-force-ssl`, `--no-http2`, `--no-websockets`, `--no-block-exploits` | – | switches |
#### The `advanced_config` block
Without `--advanced`, `proxy add` sets this block – the same four lines you
would otherwise type into the **Advanced** tab by hand:
```nginx
client_max_body_size 0;
proxy_read_timeout 300s;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```
What each is for:
| Line | Reason |
|---|---|
| `client_max_body_size 0` | removes the size limit – large agent uploads and bakery packages fail otherwise |
| `proxy_read_timeout 300s` | long-running Checkmk views would otherwise time out |
| `X-Forwarded-Proto` | without it Checkmk generates `http://` links behind the HTTPS proxy |
| `X-Forwarded-For` | without it the Checkmk access log only shows the proxy address instead of the real client IP |
`--advanced` replaces the block (it does not append):
```bash
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com \
--advanced 'client_max_body_size 0;
proxy_read_timeout 600s;'
```
`--advanced ''` leaves the field empty. The wizard in the interface always uses
the default above; if you need something else, use the command line or adjust
the entry in the Proxy Manager afterwards.
In the interface the same flow is under **proxy → "Proxy-Host anlegen"**
(name → target port → certificate) and under **Einstellungen → NGINX Proxy
Manager → ">> Proxy-Hosts anlegen/verwalten"**.
> A self-signed certificate is **reused** on repeated runs instead of being
> created anew each time – otherwise dozens of entries with the same name pile
> up in the Proxy Manager.
### 10.3 Several services behind one port (paths instead of names)
The typical situation: only **one port** is reachable from outside, but the
server runs several interfaces – Checkmk, Grafana, maybe something else.
Instead of needing a separate name per service, they can be told apart by
**path**:
```
https://10.0.0.5/cmk/ -> Checkmk (127.0.0.1:5000)
https://10.0.0.5/grafana -> Grafana (127.0.0.1:3000)
https://10.0.0.5/other -> another service
```
This is exactly what the Proxy Manager is for. In one call:
```bash
sudo ./scripts/cmk-manage.sh proxy add \
--domain 10.0.0.5 \
--forward-port 5000 \
--cert self \
--location /grafana=127.0.0.1:3000 \
--location /other=127.0.0.1:8080
```
* `--domain` may be an **IP address** – a DNS name is not required. (The
interface accepts anything except a handful of special characters.)
* `--forward-port` is the target for everything **not** covered by a path –
Checkmk here.
* `--location` can be given any number of times. Syntax `/path=host:port`,
optionally with a scheme (`https://host:port`) and with a path prefixed at
the target (`/alt=host:3000/base`).
Inspect the result with:
```bash
./scripts/cmk-manage.sh proxy show --domain 10.0.0.5
```
In the interface the wizard asks about the certificate, whether further
services should live under the same name, and then takes path and target in
turn until you leave a field empty.
> **The path list is absolute.** `proxy add --update` replaces the entry
> including all paths. So when adding a path later, pass the existing ones
> again – `proxy show` tells you which those are.
#### What the services themselves need to know
A service behind a sub-path otherwise generates links to `/` that lead nowhere.
**Checkmk** is unaffected – it lives under `/<site>/` anyway. Others need a
setting:
**Grafana** (`/etc/grafana/grafana.ini` or `GF_*` environment variables):
```ini
[server]
domain = 10.0.0.5
root_url = %(protocol)s://%(domain)s/grafana/
serve_from_sub_path = true
```
As a container:
```yaml
environment:
GF_SERVER_ROOT_URL: https://10.0.0.5/grafana/
GF_SERVER_SERVE_FROM_SUB_PATH: "true"
```
Without it Grafana loads but looks for its JavaScript under
`https://10.0.0.5/public/...` instead of `/grafana/public/...` – the page stays
blank. The same applies to any other service: it has to know its base path.
#### Why the Proxy Manager at all
All containers run on the host network, so the services listen directly on
their ports (Checkmk 5000, Grafana 3000, …). You expose **none** of them –
only 443. Behind it the Proxy Manager distributes by path and handles HTTPS
termination with one certificate for all services.
```
Internet / LAN
|
:443 (the only open port)
|
NGINX Proxy Manager
| | |
/cmk/ /grafana /other
| | |
:5000 :3000 :8080 (localhost only)
```
### 10.4 If you do want the web interface
1. open `http://<server>:81` – or whichever port is set in `NPM_ADMIN_PORT`
(see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)); if
that port is blocked on the network, use the terminal browser from
[chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser)
2. log in with `NPM_ADMIN_EMAIL` / `NPM_ADMIN_PASSWORD`
### 10.5 Creating a proxy host in the browser
**Hosts → Proxy Hosts → Add Proxy Host**
| Field | Value |
|---|---|
| Domain Names | `checkmk.example.com` (or your internal name) |
| Scheme | `http` |
| Forward Hostname / IP | `127.0.0.1` |
| Forward Port | `5000` |
| Block Common Exploits | on |
| Websockets Support | **on** (for the live data in the interface) |
Tab **Advanced** (recommended, so Checkmk sees the real client IP and large
agent uploads work):
```nginx
client_max_body_size 0;
proxy_read_timeout 300s;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```
### 10.6 Option A – Let's Encrypt (publicly reachable server)
Prerequisites: the DNS name resolves publicly to this server and **port 80 is
reachable from the internet** (HTTP-01 challenge).
In the proxy host, tab **SSL**:
* SSL Certificate → **Request a new SSL Certificate**
* *Force SSL* on
* *HTTP/2 Support* on
* *HSTS Enabled* as required
* enter an e-mail address, accept the terms → **Save**
Renewal is handled by the Proxy Manager automatically; the certificates live in
the bind mount `data/npm/letsencrypt/` and are therefore kept inside the
project directory.
For an internal DNS zone without public access, use the **DNS challenge**
(SSL → *Use a DNS Challenge* → select provider, store the API token).
### 10.7 Option B – self-signed certificate (internal environment)
Creating one:
```bash
sudo ./scripts/cmk-manage.sh cert \
--domain checkmk.intern.example \
--alt 10.0.0.5 --alt checkmk \
--days 3650
```
Result in `data/npm/custom-certs/checkmk.intern.example/`:
* `privkey.pem` – private key
* `fullchain.pem` – certificate (with SANs for every name/IP given)
Installing it:
1. **SSL Certificates → Add SSL Certificate → Custom**
2. give it a name, upload `privkey.pem` as *Certificate Key* and
`fullchain.pem` as *Certificate*
3. select this certificate in the proxy host under **SSL**, enable *Force SSL*
and *HTTP/2*
> For Checkmk agents and browsers to trust the certificate, `fullchain.pem`
> must be installed as a trusted certificate on the clients (Windows: computer
> certificates → Trusted Root Certification Authorities; Linux:
> `/usr/local/share/ca-certificates/` + `update-ca-certificates`).
> For plain agent communication over port 8000 this is **not** needed – there
> Checkmk uses its own site-internal CA.
The ncurses interface offers the same workflow under *"Selbstsigniertes
Zertifikat erzeugen"*.
### 10.8 HTTPS on the bare IP
Opening the server as `https://172.0.250.82/cmk/` ends in
**`ERR_SSL_UNRECOGNIZED_NAME_ALERT`** in the browser – while `http://` on the
very same address works. This is not a certificate problem.
The reason is in TLS: the browser tells the server which name it wants to talk
to (*SNI*) while the connection is set up. RFC 6066 does not allow **IP
addresses** there, so for an IP the browser sends nothing at all. nginx then
cannot select the matching server block by name and falls back to its default
server. With NPM that is this block from `/etc/nginx/conf.d/default.conf`:
```nginx
server {
listen 443 ssl;
server_name localhost;
ssl_reject_handshake on; # <- exactly what the browser reports
return 444;
}
```
Over HTTP it does not show, because there nginx picks the server block by the
`Host` header – and that carries the IP.
The remedy is a default server of your own that accepts those nameless calls:
```bash
sudo ./scripts/cmk-manage.sh proxy default --domain checkmk.example.local --restart
```
The command reads the named proxy host through the API and mirrors it – target,
certificate and **all additional paths** – into a block carrying
`default_server`, written to the custom configuration. Calls by name are
untouched: nginx still selects by SNI first, and the default server only
applies when no name matches. None of the blocks NPM generates sets
`default_server` itself, so there is no conflict over it.
| Command | Effect |
|---|---|
| `proxy default` | shows whether it is set up, and which proxy host it mirrors |
| `proxy default --domain X` | mirror proxy host `X` |
| `proxy default --off` | remove the mirror |
| `--restart` | restart the Proxy Manager right away |
In the GUI this lives under **Proxy-Hosts → HTTPS über die blanke IP (ohne
Namen) ermöglichen**.
Two things to keep in mind:
- The **certificate must carry the IP as a subject alternative name**, otherwise
the browser reports a name mismatch (clickable, but ugly). `cmk-manage.sh
cert` and `proxy add --cert self` add the host's IP by themselves; with a
Let's Encrypt certificate this is impossible by design – there are no
certificates for IP addresses. The command checks this and says so.
- The custom configuration lives in the Proxy Manager's data directory, which
belongs to the container. So run it **with `sudo`**.
The clean alternative is still a name: a DNS record or one line in the clients'
`hosts` file is enough for the browser to send SNI, and the proxy host applies
as usual.
### 10.9 Changing a proxy host
`proxy add --update` rewrites the entry **completely** – using it to change
just a port silently loses the paths and the advanced configuration. That is
what `proxy edit` is for: you name only what should change, everything else is
left alone.
```bash
./scripts/cmk-manage.sh proxy edit --id 3 --forward-port 5001
./scripts/cmk-manage.sh proxy edit --domain checkmk.example.local --cert self
./scripts/cmk-manage.sh proxy edit --id 3 --location /grafana=127.0.0.1:3000
./scripts/cmk-manage.sh proxy edit --id 3 --del-location /grafana
./scripts/cmk-manage.sh proxy edit --id 3 # just look
```
The entry is selected with `--id` or `--domain`.
| Option | Changes |
|---|---|
| `--rename NAME` | the domain name |
| `--forward-host HOST`, `--forward-port PORT` | the forwarding target |
| `--scheme http\|https` | how the proxy talks to the target |
| `--cert self\|letsencrypt\|none\|ID` | the certificate |
| `--advanced 'TEXT'` | the advanced nginx configuration (`\n` separates lines) |
| `--force-ssl`, `--http2`, `--websockets`, `--block-exploits` | each `on`/`off` |
| `--enable`, `--disable` | switch the entry on or off |
| `--location /path=host:port` | add or replace a path |
| `--del-location /path` | remove a path |
Two things to know:
- The API only knows **paths** as a whole. `--location` and `--del-location`
therefore read the current list, change the one entry and write the list
back. If a replaced path had an advanced configuration of its own, you are
told beforehand.
- The Proxy Manager **quietly corrects some things**: without a certificate it
turns `http2_support` and `ssl_forced` off by itself. `proxy edit` compares
what was stored afterwards and reports a difference – otherwise you would
look for the fault in the wrong place later.
The GUI has the same under **Proxy-Hosts → Proxy-Host bearbeiten**; there the
fields come prefilled with the current values and the switches are offered as a
checklist.
> If the default server for IP access is set up
> ([chapter 10.8](#108-https-on-the-bare-ip)), it mirrors an earlier state.
> Rewrite it after content changes – `proxy edit` reminds you.
### 10.10 Applying changes
Whatever goes through the API – creating, changing, deleting proxy hosts – is
written by the Proxy Manager itself, which reloads nginx along the way. The
**custom configuration** however (the default server from
[chapter 10.8](#108-https-on-the-bare-ip), the admin port from
[chapter 12](#12-moving-the-admin-interface-to-a-different-port)) is a file in the
data directory and only takes effect on the next read:
```bash
./scripts/cmk-manage.sh proxy reload # re-read nginx config, no interruption
./scripts/cmk-manage.sh proxy restart # restart the container
```
`reload` checks the configuration **first** with `nginx -t` and only reloads if
it passes. If something is broken, the cause including the line number is right
there and nothing happens:
```
[FEHLER] Die nginx-Konfiguration ist fehlerhaft - es wird nichts neu geladen.
nginx: [emerg] invalid parameter "kaputt" in /data/nginx/custom/http.conf:1
```
`restart` is only needed when environment variables changed or nginx stopped
answering altogether. It then waits until nginx runs again **and** the admin
interface answers – nginx is back within a second, the service behind it takes
longer, and a "done" would otherwise be a lie.
The GUI has both under **Proxy-Hosts → Proxy Manager neu starten / nginx neu
laden**.
### 10.11 Making the Proxy Manager's admin interface reachable
It is a single-page application and loads its files over **absolute** paths:
```html
<link rel="preload" href="/images/logo-no-text.svg" ...>
```
Under a sub-path such as `/npm` only the start page arrives; everything else
lands on whatever sits at the root – a **blank page** in the browser. Measured
against a running Proxy Manager:
| Access | `/` | `/images/…` | login (`POST /api/tokens`) |
|---|---|---|---|
| sub-path `/npm` | 200 | **502** | – |
| own name | 200 | 200 | 200 |
| root | 200 | 200 | 200 |
Checkmk works under `/cmk/` only because it is built for a URL prefix and puts
it into every address it generates. The Proxy Manager cannot do that. Two ways
remain – **both work with the single free port**:
**Option A – own name** (if name resolution exists):
```bash
./scripts/cmk-manage.sh proxy admin --domain npm.example.local
./scripts/cmk-manage.sh proxy reload
```
Creates a proxy host pointing at `127.0.0.1:81` – the interface sits at `/`
there and everything works. Checkmk stays reachable under its own name or the
IP; which server block applies is decided by the name (SNI), not the port.
Without DNS, one line in the workstations' `hosts` file is enough.
**Option B – root** (if there is *no* name resolution):
```bash
./scripts/cmk-manage.sh proxy admin --root
./scripts/cmk-manage.sh proxy default --domain <name> --restart # if set up
```
The root of the existing proxy host then points at the admin interface, and the
previous target moves to `/cmk` (or `CMK_SITE_ID`):
```
https://172.0.250.82/ Proxy Manager admin interface
https://172.0.250.82/cmk/ Checkmk
```
This works because Checkmk only occupies its site path – the rest of the root
is free. To undo:
```bash
./scripts/cmk-manage.sh proxy admin --root --off
```
> With option B the **login form is exposed to the network**. For the duration
> of a setup that is acceptable; afterwards either undo it or restrict access
> to the administration network with the firewall.
Further options: `--id N` or `--host NAME` when there are several proxy hosts,
`--path /path` for a different site path, `--cert self|letsencrypt|none` with
option A. Without arguments `proxy admin` shows the current state.
The GUI has both under **Proxy-Hosts → Adminoberfläche erreichbar machen**.
### 10.12 Adding more services – Grafana as an example
A single free port is enough to keep several services apart by path:
```
https://172.0.250.82/ Proxy Manager admin interface
https://172.0.250.82/cmk/ Checkmk
https://172.0.250.82/grafana/ Grafana
```
Whether a service plays along is up to the service, not the proxy. It has to
know that it lives under a prefix and put that prefix into every address it
generates. Checkmk does this natively (the site path *is* the prefix), the
Proxy Manager's interface cannot do it at all
([chapter 10.11](#1011-making-the-proxy-managers-admin-interface-reachable)),
and **Grafana can, once you tell it**:
```yaml
# docker-compose.yml, service grafana
environment:
GF_SERVER_ROOT_URL: "https://172.0.250.82/grafana/"
GF_SERVER_SERVE_FROM_SUB_PATH: "true"
```
Equivalent in `grafana.ini`:
```ini
[server]
root_url = https://172.0.250.82/grafana/
serve_from_sub_path = true
```
With that Grafana puts a `<base href="/grafana/" />` into its pages and loads
every file relative to it. Without those two lines it redirects to `/login` –
the root, where something else lives entirely. Measured:
| Grafana setting | `/grafana/login` | page assets |
|---|---|---|
| without `serve_from_sub_path` | 302 to `/login` | go to the root |
| with `root_url` + `serve_from_sub_path` | 200 | `/grafana/public/…` → 200 |
On the proxy side it is one more path:
```bash
./scripts/cmk-manage.sh proxy edit --id 1 --location /grafana=127.0.0.1:3000
./scripts/cmk-manage.sh proxy default --domain <name> --restart # if set up
./scripts/cmk-manage.sh proxy check
```
`proxy check` queries every target **from inside the proxy container** – that
is where nginx opens the connection from, not your workstation:
```
Proxy-Host 1 stack.local
PFAD ZIEL ANTWORT
/ http://127.0.0.1:81 HTTP 200
/grafana http://127.0.0.1:3000 HTTP 302
/cmk http://127.0.0.1:9999 nicht erreichbar
```
That is how you find a typo in a target address before the browser shows
`502 Bad Gateway`. The GUI has it under **Proxy-Hosts → Ziele prüfen**.
> Port **3000 stays closed in the firewall**. Grafana is reached exclusively
> through the proxy, just like Checkmk on 5000. `firewall close` offers an
> accidentally opened port for closing again.
### 10.13 Using a certificate delivered from outside
When an external party drops the certificate into a directory and **replaces**
it on renewal – same file name, new content – the usual route does not fit: the
Proxy Manager copies uploaded certificates to `/data/custom_ssl/npm-<id>/` and
never notices a replaced file.
The directory is therefore **mounted read-only as `/certs`** into the
container, and the two files under `custom_ssl` are replaced by links to it:
```bash
./scripts/cmk-manage.sh config set CERT_EXTERNAL_DIR /etc/ssl/customer
./scripts/cmk-manage.sh up # recreate the container so it sees /certs
./scripts/cmk-manage.sh cert external --domain customer.example.com
```
| Setting | Meaning |
|---|---|
| `CERT_EXTERNAL_DIR` | directory on the host (default `./data/npm/external-certs`) |
| `CERT_EXTERNAL_FULLCHAIN` | file name of the chain (default `fullchain.pem`) |
| `CERT_EXTERNAL_KEY` | file name of the private key (default `privkey.pem`) |
| `CERT_EXTERNAL_WATCH` | check daily for a replacement (`on`/`off`) |
Before switching, both files are checked for readability, **whether certificate
and key belong together** (public keys compared) and how long it is still
valid:
```
[FEHLER] Zertifikat und Schluessel gehoeren nicht zusammen.
[ WARN ] Das Zertifikat laeuft in weniger als 14 Tagen ab (Sep 25 20:03:58 2026 GMT).
```
**Replacing the file alone is not enough.** nginx reads certificate files at
start and on reload – measured with a replaced certificate:
| | served |
|---|---|
| file replaced, no reload | the **old** certificate |
| after `nginx -s reload` | the new certificate |
Two ways to handle that:
```bash
./scripts/cmk-manage.sh cert external --refresh # once, reloads only on change
sudo ./scripts/cmk-manage.sh cert external --watch on # daily via cron
```
`--refresh` compares the SHA256 fingerprint with the last one seen
(`cert-external.state`) and reloads nginx only if it differs. The watcher writes
`/etc/cron.d/checkmk-cert-external`.
Back to an own copy – afterwards the Proxy Manager lives with its own files
again:
```bash
./scripts/cmk-manage.sh cert external --off --id 3
```
`cert external` without arguments shows the state: directory, expiry date,
issuer, whether `/certs` arrives in the container, which certificates point
there, and whether the watcher is installed. The GUI has all of it under
**Selbstsigniertes Zertifikat erzeugen → Extern geliefertes Zertifikat
verwenden**.
> If the certificate is uploaded again through the Proxy Manager's **web
> interface**, it replaces the links with copies. Just run
> `cert external --domain …` once more.
---
## 11. Configuring the Proxy Manager locally (terminal browser)
When port 81 cannot be reached from the network, the NGINX Proxy Manager's
admin interface has to be operated **on the machine itself**. For that,
`cmk-manage.sh browser` starts another container on the host network – which
reaches `127.0.0.1:81` just like the Proxy Manager itself.
```bash
sudo ./scripts/cmk-manage.sh browser
```
This is deliberately **not** a service in `docker-compose.yml`: a browser is an
interactive program with a terminal, not a background service. It is started
with `run --rm -it` and disappears without a trace when you quit.
### 11.1 Why not `lynx`
The Proxy Manager's admin interface is a JavaScript application. Classic text
browsers **cannot** render it – you get a blank page. There are therefore two
engines that render a full browser inside the terminal:
| Engine | Based on | JavaScript | Proxy Manager usable |
|---|---|---|---|
| `browsh` | Firefox | yes | **yes** – default, maintained |
| `carbonyl` | Chromium | yes | **yes** – nicer rendering, last released 2023 |
| `links` (links2) | text browser | no | no |
| `w3m` | text browser | no | no |
| `lynx` | text browser | no | no |
| `shell` | Bash + curl/jq/openssl | – | for diagnostics and the API |
The text browsers are included anyway because they are handy for quick checks.
If you pick one, the script warns explicitly.
### 11.2 Usage
```bash
# default: browsh on http://127.0.0.1:81
sudo ./scripts/cmk-manage.sh browser
# a different engine or a different page
sudo ./scripts/cmk-manage.sh browser --engine carbonyl
sudo ./scripts/cmk-manage.sh browser --engine links --url http://127.0.0.1:5000/cmk/
# shell with network tools on the host network
sudo ./scripts/cmk-manage.sh browser --engine shell
```
Key bindings in **browsh**:
| Key | Action |
|---|---|
| `Ctrl+L` | address bar |
| `Ctrl+T` | new tab |
| `Ctrl+W` | close tab |
| `Ctrl+Q` | quit |
The first time a text-browser or `shell` engine is used, the script builds a
small tools image (Debian slim with links2, w3m, lynx, curl, jq, openssl,
roughly 170 MB). Build it in advance – this works without a terminal, e.g. from
an installation script:
```bash
sudo ./scripts/cmk-manage.sh browser --build-only
```
`browsh` and `carbonyl`, by contrast, are pulled as ready-made images; their
names are in `.env` (`BROWSH_IMAGE`, `CARBONYL_IMAGE`) and are fully qualified
so that Podman does not look for them at `registry.access.redhat.com`.
In the ncurses interface everything lives under *"Terminal-Browser (Proxy
Manager lokal bedienen)"*.
### 11.3 Blank screen: the start page does not answer
If everything stays black after `Starting Browsh …` /
`Waiting for Firefox to connect…`, it is almost always because **nothing
answers on the start page** – not because of the browser. A terminal browser
shows no error page in that case, just nothing. You can tell that Browsh itself
is running because `Ctrl+Q` still returns you to the menu.
The script therefore checks the start page beforehand and names the cause:
```
[ WARN ] Auf http://127.0.0.1:81 antwortet nichts.
Ein Terminal-Browser zeigt dann nur eine schwarze Seite.
Das ist die Oberflaeche des NGINX Proxy Managers.
Er ist abgeschaltet (NPM_ENABLED=off):
./scripts/cmk-manage.sh npm-enable on
Trotzdem starten? [j/N]
```
("nothing answers on …", "a terminal browser then shows only a black page",
"it is switched off", "start anyway?") The cases it distinguishes:
| Situation | Message |
|---|---|
| Proxy Manager switched off | points to `npm-enable on` |
| container missing or stopped | points to `up` |
| container runs but does not answer | points to `logs` and `npm-port` |
| Checkmk interface (port 5000) dead | points to `up` or `sites` |
| external address | points to the host's network access |
`--no-check` skips the check. In the interface the same diagnosis appears as a
dialog offering to start anyway or to pick a different start page from the
menu.
### 11.4 Limitation: file upload
A terminal browser cannot open the operating system's file chooser. For
**Let's Encrypt** that does not matter (only form fields are filled in), but
uploading a **self-signed certificate** (*SSL Certificates → Custom*) does not
work reliably this way.
For exactly this case, use the Proxy Manager's REST interface. Create the
certificate as described in
[chapter 10.7](#107-option-b--self-signed-certificate-internal-environment), then:
```bash
NPM=http://127.0.0.1:81
CERTDIR=./data/npm/custom-certs/checkmk.intern.example
# 1) log in and obtain a token
TOKEN=$(curl -s -X POST "$NPM/api/tokens" \
-H 'Content-Type: application/json' \
-d '{"identity":"admin@example.com","secret":"<password>"}' | jq -r .token)
# 2) create the certificate entry (provider "other" = your own certificate)
CERT_ID=$(curl -s -X POST "$NPM/api/nginx/certificates" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"provider":"other","nice_name":"checkmk-selfsigned","domain_names":["checkmk.intern.example"]}' \
| jq -r .id)
# 3) upload key and certificate
curl -s -X POST "$NPM/api/nginx/certificates/$CERT_ID/upload" \
-H "Authorization: Bearer $TOKEN" \
-F "certificate=@$CERTDIR/fullchain.pem" \
-F "certificate_key=@$CERTDIR/privkey.pem"
# 4) create the proxy host for Checkmk
curl -s -X POST "$NPM/api/nginx/proxy-hosts" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d "{\"domain_names\":[\"checkmk.intern.example\"],
\"forward_scheme\":\"http\",
\"forward_host\":\"127.0.0.1\",
\"forward_port\":5000,
\"certificate_id\":$CERT_ID,
\"ssl_forced\":true,
\"http2_support\":true,
\"block_exploits\":true,
\"allow_websocket_upgrade\":true,
\"advanced_config\":\"client_max_body_size 0;\",
\"enabled\":true,
\"meta\":{}}"
```
`curl` and `jq` are part of the tools image in case the host lacks them:
```bash
sudo ./scripts/cmk-manage.sh browser --engine shell
```
> On the very first login the Proxy Manager forces you to change the e-mail
> address and password. That is easiest in the browser
> (`cmk-manage.sh browser`); afterwards the API works with the new credentials.
### 11.5 Alternative: SSH tunnel
If SSH access to the machine is possible, a tunnel is the most convenient
route – you then use the interface in your usual browser:
```bash
ssh -N -L 8081:127.0.0.1:81 user@checkmk-server
# then locally: http://127.0.0.1:8081
```
This changes nothing about the firewall: port 81 stays closed from outside,
access runs through the existing SSH connection.
---
## 12. Moving the admin interface to a different port
Inside the image, the NGINX Proxy Manager's web interface listens on a
**hard-coded port 81** (`listen 81 default;` in
`/etc/nginx/conf.d/production.conf`). No environment variable for this exists
in the published 2.x versions – `NPM_ADMIN_PORT` is already prepared in the
development branch, but is not part of version 2.15.1. And because both
containers run on the host network, the port cannot be moved by port mapping
either.
So if port 81 is blocked on your network:
```bash
sudo ./scripts/cmk-manage.sh npm-port 10443
```
### 12.1 What happens
Within its `http{}` block, nginx includes an official extension point:
```
include /data/nginx/custom/http.conf;
```
`/data` is our bind mount `./data/npm/data`. The command places an additional
server there which listens on the desired port and forwards to the interface on
`127.0.0.1:81`:
```
./data/npm/data/nginx/custom/http.conf
# >>> cmk-manage.sh: Admin-Port - nicht von Hand aendern >>>
server {
listen 10443;
listen [::]:10443;
server_name _;
client_max_body_size 0;
location / {
proxy_pass http://127.0.0.1:81;
...
}
}
# <<< cmk-manage.sh: Admin-Port <<<
```
Only the region between the two markers is managed – your own additions to the
same file are left untouched. The script then restarts the Proxy Manager
container so nginx reads the file, and writes `NPM_ADMIN_PORT` and
`NPM_ADMIN_URL` into `.env`. The terminal browser from
[chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser) then
starts on the new port automatically.
### 12.2 Important: port 81 stays alive locally
The built-in server on port 81 **cannot** be switched off this way – it keeps
listening on all addresses. That is harmless as long as the firewall blocks it,
and that is exactly the recommendation:
```bash
# firewalld
sudo firewall-cmd --permanent --add-port=10443/tcp
sudo firewall-cmd --permanent --remove-port=81/tcp 2>/dev/null
sudo firewall-cmd --reload
# ufw
sudo ufw allow 10443/tcp
sudo ufw delete allow 81/tcp 2>/dev/null
```
### 12.3 Inspecting and reverting
```bash
./scripts/cmk-manage.sh npm-port # show the current state
sudo ./scripts/cmk-manage.sh npm-port 81 # back to the default
```
With `81` the managed block is removed again; if the file is empty afterwards
it is deleted entirely. Only the configuration from the image then applies.
The command rejects ports already taken by the stack (80, 443, 5000, 8000,
6557), validates the numeric range, and warns if another service is already
listening on the port you want.
> As soon as an NPM release ships `NPM_ADMIN_PORT`, this detour becomes
> unnecessary – the environment variable will be enough. Until then, the
> extension point is the route that needs no change to the image.
### 12.4 Turning the Proxy Manager off entirely (external NGINX)
If HTTPS termination is already handled elsewhere – for instance by an NGINX on
an upstream system – the container is not needed at all. There is a switch for
that:
```bash
sudo ./scripts/cmk-manage.sh npm-enable off # turn off
sudo ./scripts/cmk-manage.sh npm-enable on # turn back on
./scripts/cmk-manage.sh npm-enable # show the current state
```
In the interface it is under **Einstellungen → NGINX Proxy Manager →
"Proxy Manager betreiben"**; in `.env` it is called `NPM_ENABLED`.
**What "off" means:** the container is stopped and removed, and is not created
by `up` in the first place – `compose` is only told about the Checkmk service.
So no image is pulled and no container is built:
```
NPM_ENABLED=off -> docker compose up -d checkmk
NPM_ENABLED=on -> docker compose up -d checkmk nginx-proxy-manager
```
**The data stays.** Configuration, database and certificates live in the bind
mounts `./data/npm/data` and `./data/npm/letsencrypt`, not inside the
container. Before turning it off, the command shows both with their sizes; when
you turn it back on, the existing configuration and certificates are picked up
unchanged.
**What to do afterwards:** the external proxy forwards to port 5000 of this
host. That port was meant for localhost only so far and now has to be reachable
for the proxy – as narrowly as possible:
```bash
# firewalld
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=<proxy-ip> port port=5000 protocol=tcp accept'
sudo firewall-cmd --reload
# ufw
sudo ufw allow from <proxy-ip> to any port 5000 proto tcp
```
In the external NGINX:
```nginx
location / {
proxy_pass http://<checkmk-host>:5000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 0;
proxy_read_timeout 300s;
}
```
`npm-enable off` prints the matching commands and this example configuration
for you.
> **The agent receiver is unaffected.** Agents talk to port 8000 of this host
> directly, not through the proxy – so it must stay reachable for the monitored
> systems.
> **With Quadlet** ([chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose))
> `install.sh --quadlet` removes the `nginx-proxy-manager.container` unit while
> the switch is `off`. If you turn it on later, simply run
> `sudo ./install.sh --quadlet` again.
---
## 13. Firewall
RHEL and Rocky ship firewalld enabled, and it normally allows SSH only. A stack
that runs perfectly is still unreachable from outside – agents cannot get
through and the browser shows nothing. The manager therefore checks this for
you:
```bash
./scripts/cmk-manage.sh firewall # report
./scripts/cmk-manage.sh firewall open # only print the commands
sudo ./scripts/cmk-manage.sh firewall open --apply
sudo ./scripts/cmk-manage.sh firewall open --apply --source 10.0.0.0/24
```
**firewalld**, **ufw**, **nftables** and **iptables** are detected in that
order; the first active one wins. For firewalld all *active zones* are
evaluated, not just the default zone: if two interfaces sit in different zones,
a port counts as open as soon as one of them allows it. Besides individual
ports, service names (`http`, `https`, `ssh`) and rich rules are taken into
account.
Which ports are needed depends on the configuration. Ports are handled as
`number/protocol` throughout – syslog and SNMP traps use UDP, and a TCP rule
does nothing for them.
**Always:**
| Port | Purpose | Needed |
|------|---------|--------|
| `80/tcp` | HTTP, Let's Encrypt challenge | only with `NPM_ENABLED=on` |
| `443/tcp` | HTTPS for Checkmk and other services | only with `NPM_ENABLED=on` |
| `NPM_ADMIN_PORT/tcp` (81) | Proxy Manager admin UI | administration network only |
| `5000/tcp` | Checkmk directly | **only** with `NPM_ENABLED=off`, keep closed otherwise |
| `8000/tcp` | agent receiver (registered agents) | always |
**Distributed monitoring** – driven by `FW_ROLE`:
| Port | Purpose | Needed |
|------|---------|--------|
| `6557/tcp` | Livestatus – the central site queries this one | with `FW_ROLE=remote`, otherwise only with `CMK_LIVESTATUS_TCP=on` |
| `443/tcp` | configuration sync and automation API of the central site | with `FW_ROLE=remote` |
The **central site** (`FW_ROLE=central`) opens the connections itself and
needs nothing extra inbound – but *outbound* 6557/tcp and 443/tcp to the
remote sites must be allowed. The report points this out, as well as the most
common trap: `FW_ROLE=remote` together with `CMK_LIVESTATUS_TCP=off` – the
port is open but nothing listens on it.
**Event Console and agent** – each has its own switch, because not every setup
needs them:
| Port | Purpose | Switch |
|------|---------|--------|
| `514/udp`, `514/tcp` | accept syslog messages | `FW_EC_SYSLOG=off\|udp\|tcp\|beide` |
| `162/udp` | accept SNMP traps | `FW_EC_SNMPTRAP=on` |
| `6558/tcp` | Event Console status and events over TCP | `FW_EC_TCP=on` |
| `6556/tcp` | Checkmk agent of this host (pull mode) | `FW_AGENT=on` |
Anything else – Grafana, NSCA, whatever else lives on the machine – goes into
`FW_EXTRA_PORTS`:
```bash
./scripts/cmk-manage.sh config set FW_EXTRA_PORTS "3000/tcp=Grafana,5667/tcp=NSCA"
```
The report flags the reverse case too: if 5000 is open while the Proxy Manager
is running, it reads *"open – should be closed"*. When something is missing,
`firewall` exits with status 1, so you can evaluate it in your own scripts.
Port ranges in existing rules are resolved: if you allowed `6556-6558/tcp`
(firewalld, nftables) or `6556:6558/tcp` (ufw, iptables), all three ports are
reported as open. With ufw a rule without a protocol – `ufw allow 514` –
counts for TCP **and** UDP, which is exactly how ufw creates it.
`--source` restricts every generated rule to a source network. That makes
sense for 8000, 6557 and 514 (only the monitored hosts or the central site),
but not for port 80: Let's Encrypt probes the domain from the internet, and a
restriction makes issuance fail. The report points this out.
### Closing them again
`firewall open` only knows one direction. Turn the Event Console off again or
drop an entry from `FW_EXTRA_PORTS`, and the rule stays – the port is open with
nothing listening behind it. Hence the opposite direction:
```bash
./scripts/cmk-manage.sh firewall close # only print the commands
sudo ./scripts/cmk-manage.sh firewall close --apply
```
Only what is **open, belongs to this setup and is not currently needed** is
offered. That covers the ports from the table above plus everything
`firewall open --apply` opened itself – recorded in `firewall-opened.conf`, so
a deleted `FW_EXTRA_PORTS` entry is not forgotten.
Three safeguards are built in:
| | |
|---|---|
| **SSH stays open** | port 22, the port `sshd` actually listens on (from `sshd -T`), and the port of the running session (`SSH_CONNECTION`) |
| **Foreign rules stay** | if you have 3306 open for your database, you keep it – only what this setup knows about is closed |
| **Collective rules stay** | a rule like `80,443/tcp` covers several ports; it is reported instead of deleted |
**The Proxy Manager's admin port is offered as well.** It is not a mandatory
port: the interface is reachable through `127.0.0.1` too – via the terminal
browser in the container or through an SSH tunnel. Exposed to the network it
has no business being open, which is exactly why it is up for discussion here:
```
PORT GEHOERT ZU ZUSTAND
10443/tcp Adminoberflaeche des Proxy Managers kann zu - Zugang dann nur lokal
81/tcp Adminoberflaeche - alter Vorgabeport kann zu
```
Which port that is comes from `NPM_ADMIN_PORT` – move it to 10443 and 10443 is
what gets offered. The default **81 is always in the catalogue as well**: after
moving the port it would otherwise stay open with nobody naming it any more. To
keep admin access from the network:
```bash
sudo ./scripts/cmk-manage.sh firewall close --apply --keep-admin
```
The GUI asks about it as soon as an admin port is affected.
With ufw the rules are deliberately **not** deleted by number: those shift on
the first deletion, and the next call would hit the wrong rule. Deletion goes
by rule specification instead (`ufw delete allow 514/udp`), with nftables by
handle, and with iptables using the same specification it was added with.
The GUI has this under **Einstellungen → Firewall → Nicht mehr benötigte Ports
schließen**.
> **Docker and ufw:** published container ports normally bypass ufw through the
> `DOCKER-USER` chain. Not here – both containers run in the host network, so
> the host rules apply unchanged.
The GUI offers the same report under **Firewall prüfen**, including a prompt to
set the missing rules right away. The switches live under **Einstellungen →
Firewall: was offen sein muss**, which also holds two actions:
| Menu entry | what it does |
|---|---|
| `>> Firewall jetzt pruefen` | the same report as `firewall check` |
| `>> Firewall-Regeln uebernehmen (Ports oeffnen)` | asks for a source network first, **then** shows the commands that will run verbatim, and applies them after confirmation |
The order is deliberate: the source network changes the commands, so it is
asked before they are shown – what you see in the window is what runs. Without
`root` the call is repeated through `sudo`. Afterwards the GUI shows the state
that resulted, so you do not have to guess whether it worked.
> If an older `.env` does not have the `FW_*` keys yet, the menu shows
> `<leer>` – the defaults apply then (`single`, everything off). As soon as a
> value is set, the key is written to `.env`.
### By hand
**firewalld (RHEL/Rocky/Alma):**
```bash
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-port=8000/tcp # agent receiver
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.0.0.0/24 port port=81 protocol=tcp accept'
# distributed monitoring: Livestatus for the central site only
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.0.0.5/32 port port=6557 protocol=tcp accept'
# Event Console: syslog and SNMP traps (UDP!)
sudo firewall-cmd --permanent --add-service=syslog # 514/udp
sudo firewall-cmd --permanent --add-service=snmptrap # 162/udp
sudo firewall-cmd --permanent --add-port=6558/tcp # status over TCP
sudo firewall-cmd --reload
# With a different admin port (e.g. 10443) open that number instead -
# "cmk-manage.sh npm-port <port>" prints the matching command for you.
```
**ufw (Debian/Ubuntu):**
```bash
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8000/tcp
sudo ufw allow from 10.0.0.0/24 to any port 81 proto tcp
sudo ufw allow from 10.0.0.5 to any port 6557 proto tcp # Livestatus for the central site
sudo ufw allow 514/udp # Event Console: syslog
sudo ufw allow 162/udp # Event Console: SNMP traps
sudo ufw enable
```
Port **5000 stays closed** – access goes exclusively through the proxy. Open
port **81** to the administration network only.
---
## 14. Management: `cmk-manage.sh`
The script is both a command-line tool **and** an ncurses interface.
### 14.1 ncurses interface (Midnight Commander style)
```bash
sudo ./scripts/cmk-manage.sh gui
```
The menu labels are German; the command table in
[14.2](#142-command-line) lists the same actions in English.
```
┌──────────────── Checkmk Stack Manager ─────────────────┐
│ Projekt : /opt/checkmk-stack │
│ Runtime : podman Standard-Site: cmk │
│ Backups : /opt/checkmk-stack/backups │
│ │
│ status Status von Containern und Sites │
│ stack Stack steuern (Start/Stop/Logs/Update) │
│ backup Backup erstellen / planen │
│ restore Backup wiederherstellen │
│ import Site von Direktinstallation importieren │
│ manage Backups verwalten (Liste/Details/Löschen) │
│ sites Sites im Container verwalten │
│ image Checkmk-Image beschaffen/auswählen │
│ proxy Proxy-Hosts anlegen/verwalten (HTTPS) │
│ browser Terminal-Browser (Proxy Manager lokal) │
│ cert Selbstsigniertes Zertifikat erzeugen │
│ firewall Firewall prüfen (benötigte Ports) │
│ config Einstellungen (.env) - menügeführt │
│ shell Root-Shell im Checkmk-Container │
│ quit Beenden │
│ < OK > <Cancel> │
└────────────────────────────────────────────────────────┘
```
Operation: arrow keys, **space** toggles entries in selection lists (e.g.
several sites for a backup), **Enter** confirms, **ESC/Cancel** goes back.
The window size adapts to the terminal size, the length of the text and the
number of entries – in a larger terminal more entries are visible at once.
This holds for **every** window, `migrate_creator.sh` included: all dialogs go
through three shared entry points in
[scripts/lib/common.sh](scripts/lib/common.sh) which compute the size
themselves.
### 14.4 Borders show up as `q`, `x`, `l`, `m` (PuTTY)
If the interface looks like this –
```
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Projekt : /opt/checkmk-stack x
```
– then the server lacks a **UTF-8 locale**. Without one, `dialog` and
`whiptail` draw the borders through the old VT100 switch `ESC(0`. Terminals in
UTF-8 mode – PuTTY by default – ignore that switch and show the raw letters
instead.
Measured: with `LC_ALL=C`, `whiptail` emits one `ESC(0` switch and **91 raw
`q`** characters; with a UTF-8 locale, none at all.
**The script fixes this in two ways.**
**1. UTF-8 locale.** When starting the interface it sets
`LC_CTYPE` to the first available UTF-8 locale (`C.UTF-8`, `en_US.UTF-8`, …) –
only `LC_CTYPE`, so sorting and number formats stay untouched. If `LC_ALL` is
set to a non-UTF-8 value it is removed, since it would otherwise override
`LC_CTYPE`. **2. Unicode instead of VT100.** That alone is not enough: **ncurses uses the
VT100 switch even with a UTF-8 locale** as long as the terminfo offers it –
which it does for `TERM=xterm`. So `NCURSES_NO_UTF8_ACS=1` is set as well,
making ncurses draw real Unicode box-drawing characters. Measured with
`dialog` under `LANG=C.UTF-8` and `TERM=xterm`:
| | `ESC(0` | UTF-8 box bytes | raw `q` |
|---|---|---|---|
| without the switch | 38 | 0 | **182** |
| **with the switch** | **0** | **220** | **0** |
`whiptail` does not need this – with a UTF-8 locale it draws Unicode by itself.
For the opposite case – a terminal that handles `ESC(0` but not UTF-8 –
`CMK_TUI_ACS=1` restores the old behaviour.
Verify with:
```bash
./scripts/cmk-manage.sh selftest # section "Zeichensatz fuer die Oberflaeche"
```
If the check reports that **no** UTF-8 locale exists, the language pack is
missing:
```bash
sudo dnf install -y glibc-langpack-en # RHEL/Rocky/Alma
sudo apt-get install -y locales # Debian/Ubuntu
```
#### PuTTY settings
So that the other end matches:
| PuTTY section | Setting |
|---|---|
| Window → Translation | **Remote character set: UTF-8** |
| Window → Translation | Handling of line drawing characters: **Use Unicode line drawing code points** |
| Connection → Data | Terminal-type string: **`xterm-256color`** (the default is just `xterm`) |
| Window → Appearance | a font with box-drawing glyphs, e.g. *Consolas* or *DejaVu Sans Mono* |
Save the session under *Session → Saved Sessions* to keep these settings.
#### If it still looks wrong: look instead of guessing
```bash
./scripts/cmk-manage.sh tui-check
```
This prints two test lines:
* **Line A** – real UTF-8 box-drawing characters (`┌──┐`)
* **Line B** – the old VT100 switch `ESC(0` with the letters `lqk`
The cause can be read off them:
| Observation | Meaning |
|---|---|
| A lines, B lines | everything is fine |
| A lines, **B shows `lqqqk`** | the terminal ignores the VT100 switch – **the PuTTY case**, see the settings above |
| A shows gibberish | the terminal is not in UTF-8 mode |
| both show letters | the font has no box-drawing glyphs |
The command names the matching remedy for each observation.
> **Installing `dialog` instead of `whiptail`** is the most effective move on
> this side – it handles the character set more reliably and is the only one
> that can draw plain ASCII borders:
>
> ```bash
> sudo dnf install -y dialog # RHEL/Rocky/Alma
> sudo apt-get install -y dialog # Debian/Ubuntu
> ```
>
> The script prefers `dialog` as soon as it is present. After that, plain
> borders can be forced if needed:
>
> ```bash
> CMK_TUI_ASCII=1 ./scripts/cmk-manage.sh gui
> ```
>
> Without `dialog` the script says so explicitly – `whiptail` has no such
> option.
### 14.5 Self-check
```bash
./scripts/cmk-manage.sh selftest
```
Checks the scripts for the pitfalls that turned out to be recurring here:
| Check | Why |
|---|---|
| syntax of all four scripts | typos surface immediately |
| dialogs go through the shared entry points | otherwise a new window would get a fixed size again |
| no hard-coded window sizes | same reason |
| no `[ … ] && VAR=…` | returns 1 when the condition is false and aborts the script under `set -e` – the cause of a silent abort three times |
| settings table complete | every key has six fields and appears in `.env.example` |
Worth running after adding your own menu entries.
`dialog` is preferred; if only `whiptail` is present, that is used instead.
A file chooser with directory navigation is built in.
### 14.2 Command line
```bash
./scripts/cmk-manage.sh --help
```
| Command | Effect |
|---|---|
| `up [--no-fetch]` / `down` / `restart` / `pull` | control the stack; `up` fetches a missing image itself |
| `status` | containers, sites, access addresses |
| `logs [-f]` | container logs |
| `sites` | list the sites in the container |
| `site-start SITE` / `site-stop SITE` | control a single site |
| `start-all-sites` | start all sites (invoked by systemd) |
| `site-create NAME` | create another site in the container – own ports (5001, 8001, …) |
| `site-delete NAME` | remove a site and all its data |
| `backup …` | backup as a `.tar.gz` bundle |
| `list` / `verify` / `prune --keep N` | manage backups |
| `job list\|add\|set\|del\|run\|auto` | scheduled backups (own target path per job) |
| `restore …` | restore a backup |
| `import …` | import a migration bundle |
| `image get` | fetch the image via the configured route |
| `image status\|list\|versions\|repos\|why\|test\|pull\|download\|load\|use` | obtain and select the Checkmk image, and explain failures |
| `browser [--engine …]` | terminal browser in a container – operate the Proxy Manager locally |
| `config list\|get\|set\|edit` | show and change settings in `.env` |
| `npm-port [PORT]` | set/show the port of the Proxy Manager interface |
| `npm-enable [on\|off]` | turn the Proxy Manager on/off (data is preserved) |
| `npm-reset` | move the database aside so the account is recreated |
| `proxy status\|list\|certs\|add\|del` | create and manage proxy hosts through the API |
| `cert …` | create a self-signed certificate |
| `cert external …` | use a certificate delivered from outside (link instead of copy) |
| `firewall [check\|open\|close]` | check, open, and close again what is no longer needed (firewalld, ufw, nftables, iptables); `close --keep-admin` keeps the admin port open |
| `shell` | root shell in the Checkmk container |
| `omd …` | any `omd` command inside the container |
| `noproxy [--export\|--fix]` | check whether local addresses bypass the HTTP proxy; write it to profile.d, `.bashrc`, `/etc/environment` or the systemd unit |
Global options: `--env-file FILE`, `-y/--yes` (no prompts, e.g. for cron jobs),
`-v/--verbose`.
### 14.3 Changing settings – without an editor
Every value in `.env` can be maintained through the menu. In the interface:
**Einstellungen (.env) – menügeführt**. The values are grouped by area:
```
┌──────────────────── Einstellungen ─────────────────────┐
│ Alle Werte der Datei .env - direkt hier änderbar. │
│ Datei: /opt/checkmk-stack/.env │
│ │
│ runtime Runtime und Zeitzone │
│ image Checkmk-Image und Zugangsdaten │
│ checkmk Checkmk-Site und Container │
│ npm NGINX Proxy Manager │
│ browser Terminal-Browser │
│ paths Verzeichnisse │
│ cert Selbstsignierte Zertifikate │
│ show Alle Werte auflisten │
│ editor Datei .env im Editor öffnen │
│ back zurück │
└────────────────────────────────────────────────────────┘
```
(runtime and time zone / Checkmk image and credentials / Checkmk site and
containers / NGINX Proxy Manager / terminal browser / directories / self-signed
certificates / list all values / open `.env` in an editor / back.)
Each entry shows its current value and, when confirmed, opens the matching
input: free text, number, selection list or password field (passwords appear as
`********` in the overview and stay unchanged if you leave the field empty).
The editor remains available as a menu entry.
The same values from the command line:
```bash
./scripts/cmk-manage.sh config list # everything with current values
./scripts/cmk-manage.sh config list --category npm # one area only
./scripts/cmk-manage.sh config get CMK_SITE_ID
./scripts/cmk-manage.sh config set CMK_SITE_ID prod
./scripts/cmk-manage.sh config set NPM_ADMIN_PORT 10443
```
**Validation happens when setting the value**, not at the next start:
| Key | Check |
|---|---|
| `NPM_ADMIN_PORT` | number 1–65535, not 80/443/5000/8000/6557, warning if in use |
| `CMK_SITE_ID` | letters, digits and `_` only; at most 16 characters |
| `CMK_IMAGE`, `NPM_IMAGE`, … | must be fully qualified (the Podman pitfall) |
| `CMK_RUNTIME` | must be installed on this system |
| `CERT_DAYS` and others | numeric fields |
| URL fields | must start with `http://` or `https://` |
Values containing spaces are quoted automatically so that `.env` remains
readable by compose, systemd **and** the shell. When you change
`NPM_ADMIN_PORT`, the script updates the additional nginx file along with it
(see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)). For
values that only take effect once the containers are recreated, the menu says
so explicitly.
### 14.6 Networks with a mandatory proxy: an `API error` that is none
In customer networks an HTTP proxy is often present in the environment
(`http_proxy`, `https_proxy`). If `no_proxy` does **not** exclude local
addresses, `curl` sends even a request to `127.0.0.1` there. The proxy does not
know that address – and the error surfaces somewhere entirely innocent:
```
[FEHLER] Anmeldung am Proxy Manager fehlgeschlagen.
```
That looks like wrong credentials or a broken API. It is neither. To check:
```bash
./scripts/cmk-manage.sh noproxy
```
```
HTTP-Proxy in der Umgebung
http_proxy http://proxy.customer.local:3128
no_proxy <leer>
Geht http://127.0.0.1:81/ daran vorbei?
[FEHLER] Nein - curl schickt auch diese Anfrage an den Proxy.
Einzutragen
no_proxy=localhost,127.0.0.1,::1,checkmk-host,172.0.250.82
```
This is not guessed but read from what `curl -v` reports itself
(`Uses proxy env variable http_proxy == …`) – nothing needs to be listening on
the target address for that.
### `no_proxy` matches names, not addresses
This is where the search takes longest: `no_proxy=localhost` does **not** cover
a request to `127.0.0.1` – and vice versa. Measured with curl 7.88:
| `no_proxy` | request to `127.0.0.1` | request to `localhost` |
|---|---|---|
| `localhost` | **goes to the proxy** | direct |
| `127.0.0.1` | direct | **goes to the proxy** |
| both | direct | direct |
Corporate networks often list only `localhost`. Then of all things the access
to the Proxy Manager's API goes through the proxy. `noproxy` therefore checks
**both** spellings and says which one is missing:
```
Gehen lokale Anfragen daran vorbei?
http://127.0.0.1:81/ geht an den Proxy
http://localhost:81/ direkt
[ WARN ] Nur 'localhost' ist ausgenommen, 127.0.0.1 nicht.
Bis das geaendert ist, hilft sofort:
cmk-manage.sh config set NPM_API_HOST localhost
```
`NPM_API_HOST` decides which name the API is addressed by – default
`127.0.0.1`, because that needs no name resolution and always points at IPv4.
If the exception list in the customer network cannot be changed, switching to
`localhost` is the shorter path.
**In the running session** – a program cannot change its parent shell's
environment, hence the detour via `eval`:
```bash
eval "$(./scripts/cmk-manage.sh noproxy --export)"
```
**Permanently** – the targets are not mutually exclusive; depending on the
machine you need several:
| Command | takes effect for |
|---|---|
| `sudo … noproxy --fix` | login shells (`/etc/profile.d/cmk-noproxy.sh`) |
| `… noproxy --fix --user` | this user only (`~/.bashrc`) |
| `sudo … noproxy --fix --environment` | logins without a shell (`/etc/environment`) |
| `sudo … noproxy --fix --service` | **system services** – systemd unit drop-in |
The last one is easily overlooked: **system services read neither `profile.d`
nor `.bashrc` nor `/etc/environment`.** If the stack runs under systemd,
`--service` writes a drop-in to
`/etc/systemd/system/checkmk-stack.service.d/cmk-noproxy.conf`:
```ini
[Service]
Environment=no_proxy=localhost,127.0.0.1,::1,checkmk-host,172.0.250.82
Environment=NO_PROXY=localhost,127.0.0.1,::1,checkmk-host,172.0.250.82
Environment=http_proxy=http://proxy.customer.local:3128
Environment=https_proxy=http://proxy.customer.local:3128
```
The proxy itself is carried over – without it the service could not reach the
internet to pull images. If no matching unit exists, the command prints the
section to paste by hand.
**Whatever is already there stays – and stays first.** The entry collects both
spellings (`no_proxy` *and* `NO_PROXY`; some systems only set the uppercase
one), appends the local addresses and drops duplicates:
| before | after |
|---|---|
| *(empty)* | `localhost,127.0.0.1,::1,host,172.0.250.82` |
| `no_proxy=company.internal,10.0.0.0/8` | `company.internal,10.0.0.0/8,localhost,127.0.0.1,…` |
| only `NO_PROXY=company.internal` | `company.internal,localhost,127.0.0.1,…` |
| both set, different | both kept, then ours |
| `127.0.0.1` already present | kept, only what is missing is added |
A second call writes nothing twice. The GUI has it under **Einstellungen →
Runtime und Zeitzone → Lokale Verbindungen prüfen**.
> **This tool's own calls are unaffected.** For requests to local addresses the
> proxy variables are removed for that call – deliberately *not* via
> `curl --noproxy`, because that option only exists from curl 7.19.4 on, and
> what is called `curl` on some machines is something else entirely. If the
> curl at hand knows the option, it is added as well; only that also overrides
> a proxy configured in `~/.curlrc`. Requests to the internet (registry,
> download portal) keep using the proxy. Which curl is in use and whether it
> knows the option is shown at the top of `noproxy`.
> **A curl quirk:** `no_proxy`/`NO_PROXY` and `https_proxy`/`HTTPS_PROXY` are
> honoured in both spellings, uppercase `HTTP_PROXY` is **not** – curl ignores
> it deliberately, because in CGI environments an incoming `Proxy:` header
> could set it. The report points this out when only that one is set.
> **Order of operations:** first `noproxy` to look, then write it, then apply it
> to the running session with `eval "$(… noproxy --export)"` – otherwise you
> will wonder why nothing changed.
---
## 15. Backup
Backups are created with `omd backup` and packed into a **bundle**:
```
cmkbackup_<site>_<version>_<timestamp>.tar.gz
└── cmkbundle/
├── MANIFEST site, version, source, checksum …
├── INFO.txt human-readable summary
├── site.tar.gz the actual omd backup
└── site.tar.gz.sha256
```
This way the restore/import tool always knows **which site** and **which
Checkmk version** the archive contains, and can verify its integrity.
```bash
# a single site
sudo ./scripts/cmk-manage.sh backup --site cmk
# all sites, without log files, with a label
sudo ./scripts/cmk-manage.sh backup --all --no-logs --label vor-update
# most consistent backup: the site is stopped briefly
sudo ./scripts/cmk-manage.sh backup --site cmk --stop
# without performance data (considerably smaller)
sudo ./scripts/cmk-manage.sh backup --site cmk --no-rrds --dir /mnt/nas/checkmk
```
Listing and verifying:
```bash
./scripts/cmk-manage.sh list
./scripts/cmk-manage.sh verify --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz
./scripts/cmk-manage.sh prune --keep 7
```
### 15.1 Scheduled backups (backup jobs)
Manual backups always go to `BACKUP_DIR` from `.env`. Scheduled jobs, by
contrast, bring **their own target directory** – so you can back up to a NAS
share daily and somewhere else weekly.
In the menu: **backup → "Backup erstellen / planen"** (create / schedule)
```
┌──────────── Backup erstellen / planen ─────────────────┐
│ Manuelle Backups gehen immer nach: │
│ /opt/checkmk-stack/backups (Einstellung BACKUP_DIR)│
│ │
│ Geplante Jobs haben ihr eigenes Zielverzeichnis. │
│ Automatische Ausfuehrung: aus │
│ │
│ manuell Backup jetzt erstellen │
│ jobs Backup-Jobs verwalten │
│ auto Automatische Ausfuehrung ein/aus │
│ status Zeitplan und Jobs anzeigen │
│ jetzt Einen Job sofort ausfuehren │
│ back zurueck │
└────────────────────────────────────────────────────────┘
```
(create now / manage jobs / automatic execution on-off / show schedule / run a
job now / back.) The same jobs are also reachable from
**Einstellungen → "Backup-Jobs"**.
#### Creating a job
In the menu a dialog walks through sites, schedule (daily / weekly / monthly /
free cron expression), target directory, retention and options. The same with
parameters:
```bash
# daily at 02:15, all sites, without logs, keep 14 backups
sudo ./scripts/cmk-manage.sh job add --name nachts \
--cron "15 2 * * *" --no-logs --keep 14
# Sundays at 03:30, two sites only, to the NAS, without RRDs, keep 8
sudo ./scripts/cmk-manage.sh job add --name nas-woche \
--cron "30 3 * * 0" --sites prod,test \
--dir /mnt/nas/checkmk --no-rrds --no-logs --keep 8
```
All commands:
| Command | Effect |
|---|---|
| `job list` | show the configured jobs |
| `job status` | schedule, cron file and switch |
| `job add --name N --cron "…"` | create a job |
| `job set NAME …` | modify a job (fields not given stay as they are) |
| `job del NAME` | delete a job (the backups remain) |
| `job enable\|disable NAME` | turn a job on/off |
| `job run NAME` | run a job right now |
| `job run-all` | run every active job |
| `job auto on\|off` | turn automatic execution on/off |
Options for `job add` / `job set`:
| Option | Meaning |
|---|---|
| `--sites ALL\|a,b` | sites to back up (default `ALL`) |
| `--cron "15 2 * * *"` | schedule, five cron fields |
| `--dir DIR` | this job's target directory (default `BACKUP_DIR`) |
| `--keep N` | how many backups stay in the target (`0` = all) |
| `--no-rrds`, `--no-logs`, `--stop` | as for `backup` |
| `--disabled` | create it but do not schedule it yet |
#### Turning automation on and off
```bash
sudo ./scripts/cmk-manage.sh job auto on # schedule
sudo ./scripts/cmk-manage.sh job auto off # remove the schedule
```
`on` writes `/etc/cron.d/checkmk-backup` from all **active** jobs – one line
per job with its own schedule:
```cron
15 2 * * * root /opt/checkmk-stack/scripts/cmk-manage.sh -y job run nachts >>/var/log/cmk-backup.log 2>&1
30 3 * * 0 root /opt/checkmk-stack/scripts/cmk-manage.sh -y job run nas-woche >>/var/log/cmk-backup.log 2>&1
```
This needs root. Without root the command prints the lines so you can add them
yourself. `off` removes the file again; the jobs themselves are kept. The
switch is stored as `BACKUP_AUTO` in `.env`.
After each run a job prunes its own target directory down to `--keep` – a
separate `prune` entry is not needed.
#### Where the jobs live
In `backup-jobs.conf` in the project directory, one line per job:
```
name|sites|cron|directory|options|keep|enabled
nachts|ALL|15 2 * * *|/opt/checkmk-stack/backups|--no-logs|14|on
nas-woche|prod,test|30 3 * * 0|/mnt/nas/checkmk|--no-rrds --no-logs|8|on
```
The file is deliberately readable and diffable, but does not belong in the
repository (it is in `.gitignore` because it contains system-specific paths).
---
## 16. Restore
```bash
# simplest case: the site no longer exists
sudo ./scripts/cmk-manage.sh restore --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz
# overwrite an existing site (terminating running processes)
sudo ./scripts/cmk-manage.sh restore --file backups/… --reuse --kill
# restore under a different name (e.g. for a test)
sudo ./scripts/cmk-manage.sh restore --file backups/… --as-site test
```
What the script does:
1. checks the archive structure and the SHA256 checksum
2. checks whether the **Checkmk version of the backup exists in the container**
3. prompts / requires `--reuse` for an existing site, stops the site
4. copies the archive into the container and runs `omd restore`
5. turns off `TMPFS` for the site (not usable inside the container)
6. starts the site and prints the next steps
> **`--as-site` changes the site name** – registered agents would then point
> nowhere and would have to be re-registered. So do **not** use it for a
> migration.
A raw `omd backup` archive (without the bundle wrapper) is accepted as well;
the site name is then read from the archive.
### 16.1 Why a restore failed after recreating the container
```
ERROR: apache user www-data must be member of group cmk
```
When a site is created, `omd create` adds the apache user to the site's group.
That membership lives in `/etc/group` – **inside the container**, not in the
bind mount. Recreate the container (every `up` with a changed image, a
`podman rm`, an update) and `/etc/group` is back to the state from the image,
while the site in the bind mount lives on unchanged.
The official entrypoint detects this case and recreates group and user – but
does **not** add `www-data` to the group:
```bash
if ! getent group "$CMK_SITE_ID" >/dev/null; then
groupadd -g 1000 "$CMK_SITE_ID"
fi
if ! getent passwd "$CMK_SITE_ID" >/dev/null; then
useradd -u 1000 ... -G omd -s /bin/bash "$CMK_SITE_ID"
fi
```
In daily operation this goes unnoticed: the site runs, the GUI answers, the
agent receiver accepts data. Only `omd restore --reuse` checks it
(`user_verify` in `omdlib/users_and_groups.py`) and aborts.
Right behind it waits the second obstacle:
```
ERROR: fstab entry for /omd/sites/cmk/tmp does not exist
```
`omd` explicitly tolerates a missing `/etc/fstab` (*"In case there is no fstab
(seen in some containers) assume everything is OK"*) – but the image ships a
file containing a comment line. So `/etc/fstab` exists, the exception does not
apply, and the entry is found missing.
Both are state of the **container**, not of the data. `cmk-manage.sh`
therefore restores them before every `restore` and `import` – and, for the same
reason, after every `up`, so a hand-written `omd` call does not trip over it:
```
[ INFO ] Apache-Benutzer www-data -> cmk in die Gruppe aufgenommen
(nach dem Neuerstellen des Containers verloren gegangen).
[ INFO ] fstab-Eintrag fuer /omd/sites/cmk/tmp ergaenzt - ohne ihn
verweigert 'omd restore' den Dienst.
```
The apache user and the command to add it are read from
`/omd/versions/default/share/omd/distro.info` – the same source `omd` itself
uses. By hand it would be:
```bash
docker exec checkmk gpasswd -a www-data cmk
docker exec checkmk sh -c 'echo "tmpfs /omd/sites/cmk/tmp tmpfs noauto,user,mode=751,uid=cmk,gid=cmk 0 0" >> /etc/fstab'
```
---
## 17. Migrating a bare-metal installation (`migrate_creator.sh`)
Goal: take over an existing Checkmk installation on hardware or a VM into the
container – **without touching the agents**.
### 17.1 Why the agents keep working
An `omd backup` contains the complete site including `etc/ssl` (the site's own
CA and the certificates of the registered agents), `etc/check_mk` (rules,
hosts, passwords) and the monitoring history. If the site is restored **under
the same name** and the server is reachable at **the same IP address**, then
* push/pull agents (`cmk-agent-ctl`) find their registration again,
* the TLS connection to the agent receiver on **port 8000** stays valid,
* all URLs (`https://<ip>/<site>/check_mk/…`) remain correct,
* performance data and history are preserved.
The prerequisite is therefore that the target system **takes over the source
server's IP** (or that the DNS name is repointed) – which is the plan anyway.
### 17.2 Step 1 – copy the script to the source server
```bash
scp scripts/migrate_creator.sh root@old-server:/root/
ssh root@old-server 'chmod +x /root/migrate_creator.sh'
```
The script is **self-contained** – on the source server it only needs `bash`,
`tar` and `omd` (plus `dialog` or `whiptail` for the interface).
### 17.3 Step 2 – create the bundle on the source server
Show the existing sites:
```bash
sudo ./migrate_creator.sh --list
```
```
SITE VERSION STATUS GROESSE
prod 2.3.0p23.cre laeuft 4.7 GiB
test 2.3.0p23.cre gestoppt 318.2 MiB
```
(`GROESSE` = size, `laeuft` = running, `gestoppt` = stopped.)
**With the interface** (select sites with the space bar):
```bash
sudo ./migrate_creator.sh --gui
```
The dialog asks in turn for: sites → options (stop the site, skip RRDs/logs) →
target directory → label → optional transfer via `scp`.
**With parameters:**
```bash
# one site, stopping it during the backup (most consistent variant)
sudo ./migrate_creator.sh --site prod --stop --out /var/tmp
# all sites, without logs, transferred straight to the target system
sudo ./migrate_creator.sh --all --no-logs \
--scp root@10.0.0.5:/opt/checkmk-stack/backups/
```
Result:
```
/var/tmp/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz
```
Besides the omd backup, the bundle contains a `MANIFEST` with the site name,
Checkmk version, source hostname, **source IP addresses**, source operating
system and SHA256 checksum, plus a `siteinfo/` directory with `omd config
show`, `omd status` and the IP configuration of the old system for later
reference.
### 17.4 Step 3 – bring the target system to the right version
`omd restore` can only restore a site **if exactly its Checkmk version is
installed on the target system.** In the container, the version comes from the
image – so the image tag has to match:
```bash
tar -xzOf backups/cmkmigrate_prod_*.tar.gz cmkbundle/MANIFEST | grep CMK_VERSION
# CMK_VERSION=2.3.0p23.cre -> version 2.3.0p23, suffix .cre = Raw edition
# fetch the matching image and have it written to CMK_IMAGE
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p23
sudo ./scripts/cmk-manage.sh up
```
For an Enterprise source (`.cee`) accordingly:
```bash
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
# or, if the registry is unreachable:
sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \
--user <user> --password <password>
```
The import script checks this itself and aborts with a corresponding message if
the version is missing.
> **Mind the edition:** the version suffix reveals it – `.cre` = Raw,
> `.cee` = Enterprise, `.cce` = Cloud, `.cme` = Managed. The import script
> reads it from the manifest and, on failure, names the matching `image pull`
> command directly. Enterprise images are **not** on Docker Hub – see
> [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co).
### 17.5 Step 4 – import the bundle
With the interface:
```bash
sudo ./scripts/cmk-manage.sh gui # -> "import"
```
With parameters:
```bash
sudo ./scripts/cmk-manage.sh import \
--file backups/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz \
--set-default-site
```
The script
1. recognises the migration bundle and shows the source host and source IP,
2. **warns if this host's IP differs from the source IP** (the agents would
then have to be adjusted),
3. verifies checksum, version and edition,
4. prompts if the site already exists (`--reuse` overwrites),
5. runs `omd restore` under the **unchanged site name**,
6. optionally sets `CMK_SITE_ID` in `.env` (`--set-default-site`) so the
container starts the site automatically after a reboot as well,
7. starts the site and prints the next steps.
### 17.6 Step 5 – switch over and verify
```bash
# 1) shut down the old server or stop its site (no double monitoring!)
ssh root@old-server 'omd stop prod'
# 2) move the target system to the IP/DNS name of the old system
# (take over the static IP or repoint the DNS record)
# 3) recreate the containers if CMK_SITE_ID was changed
sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status
# 4) check inside Checkmk
sudo ./scripts/cmk-manage.sh omd su prod
OMD[prod]:~$ cmk -O # activate the configuration
OMD[prod]:~$ omd status
OMD[prod]:~$ exit
```
Then check in the interface:
* **Monitor → All hosts** – are the agents reporting again?
* With TLS agents: the service *"Check_MK Agent"* must not report registration
errors.
* An agent can be tested from the monitored host with `cmk-agent-ctl status` –
the site `prod` must appear there with the target server and
"Connection: TLS".
### 17.7 Migration checklist
- [ ] `migrate_creator.sh` copied to the source server
- [ ] bundle created (preferably with `--stop`)
- [ ] bundle copied to the target system (`backups/`)
- [ ] matching image fetched (`image pull`/`image download`), `CMK_IMAGE` set, stack recreated
- [ ] `import --set-default-site` executed, site running
- [ ] old site stopped / old server shut down
- [ ] IP or DNS switched to the target system
- [ ] ports 80/443/8000 open in the firewall
- [ ] proxy host set up in the NGINX Proxy Manager with a certificate
- [ ] agents reporting, `cmk -O` executed
- [ ] backup of the new system set up (cron)
---
## 18. Several sites in one container
The image's entrypoint creates only the site from `CMK_SITE_ID` and starts only
that one. Further sites can be created, though – and with the proxy in front
they sit side by side behind a single port:
```bash
./scripts/cmk-manage.sh site-create zweite
./scripts/cmk-manage.sh site-create kunde2 --password secret123
./scripts/cmk-manage.sh sites
./scripts/cmk-manage.sh site-delete zweite
```
### 18.1 Each site gets its own ports
`omd` assigns the ports itself – everything else follows from that:
| | first site | second | third |
|---|---|---|---|
| web UI (Apache) | 5000 | 5001 | 5002 |
| **agent receiver** | 8000 | **8001** | **8002** |
| Livestatus | 6557 | 6558 | 6559 |
Two consequences:
1. **In the proxy** every site gets its path:
```bash
./scripts/cmk-manage.sh proxy edit --domain <name> --location /zweite=127.0.0.1:5001
./scripts/cmk-manage.sh proxy reload
```
It then sits next to the first one: `https://<ip>/cmk/` and
`https://<ip>/zweite/`. Checkmk copes with the prefix because the site name
is part of every address anyway – see
[chapter 10.12](#1012-adding-more-services--grafana-as-an-example).
2. **In the firewall** the agents of that site talk to a different port:
```bash
./scripts/cmk-manage.sh config set FW_EXTRA_PORTS "8001/tcp=Agent-Receiver zweite"
sudo ./scripts/cmk-manage.sh firewall open --apply
```
`site-create` prints exactly those two commands with the right port numbers,
plus a reminder about the default server for IP access if it is set up.
### 18.2 What happens during creation
Inside the container `omd create --no-tmpfs` runs – without `--no-tmpfs` it
fails, because no tmpfs can be mounted there. If the container does **not** run
in the host network, the site's Apache is additionally bound to `0.0.0.0`,
otherwise it would not be reachable from outside; in the host network
`127.0.0.1` is enough and the proxy still reaches it. Without `--password`,
`omd` generates one and prints it.
### 18.3 After a restart
```bash
sudo ./scripts/cmk-manage.sh start-all-sites
```
The entrypoint starts only the default site. This command is run automatically
by the systemd unit (`ExecStartPost`) or the Quadlet unit after the container
starts, and brings up all existing sites.
### 18.4 When separate projects are better
Several sites in one container share the image and version – one `omd update`
affects all of them. For cleanly separated environments (different Checkmk
versions, separate maintenance windows, different customers) a separate project
directory with its own `.env` per site is better. Since both containers run in
the host network, only **one** Checkmk container per host can occupy ports
5000/8000 – for real multiple instances use separate hosts or a bridge-network
variant.
The GUI has all of it under **Sites im Container verwalten**.
---
## 19. Upgrading Checkmk
Because `/omd/sites` is a bind mount, the data survives replacing the image.
The Checkmk software itself lives in the image under `/omd/versions`.
```bash
# 1) ALWAYS back up first
sudo ./scripts/cmk-manage.sh backup --all --label vor-update
# 2) fetch the new image (writes CMK_IMAGE into .env automatically)
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p28
# 3) recreate the containers
sudo ./scripts/cmk-manage.sh up
# 4) update the site (interactive, confirm conflicts)
sudo ./scripts/cmk-manage.sh omd stop cmk
sudo ./scripts/cmk-manage.sh omd update cmk
sudo ./scripts/cmk-manage.sh omd start cmk
```
For a major version jump (e.g. 2.2 → 2.3) be sure to read the Checkmk werks /
release notes beforehand.
---
## 20. Troubleshooting
| Symptom | Cause / solution |
|---|---|
| `ERR_SSL_UNRECOGNIZED_NAME_ALERT` when opening `https://<IP>/` | browsers send no SNI for an IP, nginx falls back to its default server and rejects. `cmk-manage.sh proxy default --domain <name>` – see [chapter 10.8](#108-https-on-the-bare-ip) |
| `omd restore`: *apache user www-data must be member of group `<site>`* | the container was recreated while the site lives in the bind mount. `cmk-manage.sh` repairs this on restore and on `up` – see [chapter 16.1](#161-why-a-restore-failed-after-recreating-the-container) |
| `omd restore`: *fstab entry for /omd/sites/…/tmp does not exist* | same cause, same fix |
| `Anmeldung am Proxy Manager fehlgeschlagen` despite correct credentials | mandatory proxy in the network: `no_proxy` does not exclude `127.0.0.1`. `cmk-manage.sh noproxy` – see [chapter 14.6](#146-networks-with-a-mandatory-proxy-an-api-error-that-is-none) |
| `Port 80/443 already in use` | a web server is already running on the host: `ss -tlnp \| grep ':80'` → `systemctl disable --now apache2 nginx httpd` |
| Containers do not run after a reboot (Podman) | systemd unit missing: `./install.sh --with-systemd` or `--quadlet`, then `systemctl is-enabled checkmk-stack` |
| `omd restore`: *version is not installed* | fetch the matching image: `cmk-manage.sh image pull --edition <ed> --version <v>`, then `up` |
| Import aborts with an edition warning | an Enterprise backup needs an Enterprise image, see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) |
| `manifest unknown` on pull | The repository is there, **the tag is not**. `cmk-manage.sh image why` lists the tags that do exist in that branch |
| `repository name unknown` / `Login Succeeded` but the pull fails | The repository path is wrong. `cmk-manage.sh image repos --edition enterprise` shows what the registry offers; otherwise use the tarball |
| `image not known` / Podman searches `registry.access.redhat.com` | short image name. Set `CMK_IMAGE` fully qualified: `cmk-manage.sh image use docker.io/…` or `registry.checkmk.com/enterprise/…` |
| `checkmk/check-mk-enterprise` not found on Docker Hub | correct – Enterprise is only on `registry.checkmk.com` or available as a tarball; from 2.5 the edition is called `pro` and is public on Docker Hub |
| Tarball installed with `podman import`, image "unknown" | `import` produces an image without metadata. Remove the broken image with `podman rmi <id>` and use `cmk-manage.sh image load --file …` (which uses `load`) |
| Download of the image tarball fails (401/404) | check the customer portal credentials, give the exact version (no `-latest`), if necessary `image download --url '<full URL>'` |
| Proxy returns `502 Bad Gateway` | the Checkmk container is not running, or the forwarding does not point to `127.0.0.1:5000`: `./scripts/cmk-manage.sh status`, `curl -I http://127.0.0.1:5000/` |
| Let's Encrypt fails | port 80 not reachable from the internet, or wrong DNS; alternative: DNS challenge or a self-signed certificate |
| Agents do not report after the migration | site name changed (`--as-site`), IP not taken over, or port 8000 blocked; check on the agent: `cmk-agent-ctl status` |
| Permission denied on `data/` under RHEL | SELinux: `restorecon -Rv /opt/checkmk-stack/data`, check `:Z` on the mounts |
| `podman-compose` not found | enable EPEL or switch to Quadlet (`./install.sh --quadlet`) |
| `This system is not registered with an entitlement server` | RHEL without a subscription – `dnf` then supplies no packages at all. Register, mount the installation DVD as a repository, or switch to Rocky/Alma: [section 5.1](#51-unregistered-rhel-where-the-packages-come-from) |
| `epel-release` cannot be installed | EPEL depends on Red Hat's base repositories and is no substitute for registering. Without a subscription, the Quadlet route without `podman-compose` gets you there: [chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose) |
| Proxy Manager container no longer starts | the switch is `off`: `cmk-manage.sh npm-enable on`. Check with `cmk-manage.sh npm-enable` |
| Admin interface unreachable from the LAN | port 81 blocked. Move it to a free port: `cmk-manage.sh npm-port 10443` and open that in the firewall |
| New port does not answer after `npm-port` | the Proxy Manager was not restarted: `cmk-manage.sh restart`; check the file `data/npm/data/nginx/custom/http.conf` |
| `cmk-manage.sh` aborts with "CMK_RUNTIME … ist nicht installiert" | it now only warns and detects the runtime itself; set it cleanly with `config set CMK_RUNTIME auto` |
| Browser stays black, "Waiting for Firefox to connect…" | The start page does not answer – usually the Proxy Manager is not running. The script now reports this beforehand, see [chapter 11.3](#113-blank-screen-the-start-page-does-not-answer) |
| Proxy Manager interface blank / only text fragments | a text browser without JavaScript was used. Use `cmk-manage.sh browser --engine browsh` (or `carbonyl`) |
| Self-signed certificate cannot be uploaded in the terminal browser | there are no file dialogs there – upload via the API, see [chapter 11.4](#114-limitation-file-upload) |
| `browser` reports "braucht ein echtes Terminal" (needs a real terminal) | called from a pipe/script. Call it on the console or over SSH; only the build runs headless, with `--build-only` |
| A menu entry drops out to the shell with an error | fixed: missing containers, sites or backups are now reported in a dialog and the interface stays open |
| `job auto on` reports missing permissions | `/etc/cron.d/` needs root: call it with `sudo`, or add the printed lines yourself |
| The Proxy Manager asks for a new account on first use | normal since 2.15. Set `NPM_ADMIN_EMAIL`/`NPM_ADMIN_PASSWORD` **before** the first start; if the account already exists, enter its credentials there |
| A JSON error appears for proxy hosts in the terminal browser | the interface is too heavy for it. Create proxy hosts with `cmk-manage.sh proxy add` instead ([chapter 10.2](#102-creating-a-proxy-host--without-a-browser)) |
| The TUI does not start | install `dialog`: `apt-get install -y dialog` or `dnf install -y dialog` |
| Site does not start, `tmp` errors | `./scripts/cmk-manage.sh omd config <site> set TMPFS off`, then start the site |
Useful commands:
```bash
./scripts/cmk-manage.sh logs -f # container logs
./scripts/cmk-manage.sh omd status cmk # services of the site
./scripts/cmk-manage.sh shell # root shell in the container
docker exec -it checkmk omd su cmk # or podman exec …
```
---
## 21. Security notes
* Change `CMK_PASSWORD` in the interface right after installation and remove it
from `.env` (it is only evaluated when the site is created); `chmod 600 .env`
is already set.
* Set `NPM_ADMIN_PASSWORD` to your own password before the first start – the
Proxy Manager's administrator account is created from it.
* Make port **81** (admin interface) reachable from the administration network
only, and never expose port **5000**.
* Backups contain **all** passwords and certificates of the site – store them
encrypted or restrict access tightly (`chmod 700 backups`).
* Schedule `./scripts/cmk-manage.sh pull` + `up` regularly for image security
updates.
* Host networking means the container shares the host's network stack. There is
no additional network isolation – protection comes from the host firewall.
---
## 22. Uninstalling
```bash
sudo ./scripts/cmk-manage.sh down
# systemd
sudo systemctl disable --now checkmk-stack.service
sudo rm -f /etc/systemd/system/checkmk-stack.service
# or Quadlet
sudo rm -f /etc/containers/systemd/checkmk.container \
/etc/containers/systemd/nginx-proxy-manager.container
sudo systemctl daemon-reload
# remove the images (names from "cmk-manage.sh image list")
docker image rm docker.io/checkmk/check-mk-raw:2.3.0-latest docker.io/jc21/nginx-proxy-manager:2
# podman image rm …
# remove a stored registry login if present
podman logout registry.checkmk.com # or docker logout
# data (CAUTION: this deletes the entire monitoring setup!)
sudo rm -rf /opt/checkmk-stack/data
```
---
## Licence / notes
This project contains only configuration and helper scripts.
Checkmk (Raw Edition, GPLv2) and NGINX Proxy Manager (MIT) are subject to the
licence terms of their respective vendors.
Reference in New Issue View Git Blame Copy Permalink