docker-podman-installation-with-nginx-manager/README_en.md

# Checkmk in a container – Docker (Debian) & Podman (RHEL) with NGINX Proxy Manager

*[Deutsche Fassung: README.md](README.md)*

A complete, production-ready container installation of **Checkmk** with
**NGINX Proxy Manager** for HTTPS termination (Let's Encrypt *or* self-signed),
including tooling for **backup**, **restore** and the **migration of existing
sites from a bare-metal installation** – without having to re-register the
agents.

> **Note on language:** the scripts and their menus are in German. Literal
> program output and menu screenshots in this document are therefore shown
> unchanged; the surrounding text explains what they mean.

---

## Table of contents

1. [Overview & architecture](#1-overview--architecture)
2. [Directory layout](#2-directory-layout)
3. [Requirements & ports](#3-requirements--ports)
4. [Installation on Debian/Ubuntu (Docker)](#4-installation-on-debianubuntu-docker)
5. [Installation on RHEL/Rocky/Alma/Fedora (Podman)](#5-installation-on-rhelrockyalmafedora-podman)
   * [5.1 Unregistered RHEL: where the packages come from](#51-unregistered-rhel-where-the-packages-come-from)
6. [Starting automatically after a host reboot (Podman)](#6-starting-automatically-after-a-host-reboot-podman)
7. [Configuration (`.env`)](#7-configuration-env)
8. [Obtaining the Checkmk image (Raw, Enterprise & co.)](#8-obtaining-the-checkmk-image-raw-enterprise--co)
9. [First start](#9-first-start)
10. [Setting up HTTPS – Let's Encrypt or self-signed](#10-setting-up-https--lets-encrypt-or-self-signed)
11. [Configuring the Proxy Manager locally (terminal browser)](#11-configuring-the-proxy-manager-locally-terminal-browser)
12. [Moving the admin interface to a different port](#12-moving-the-admin-interface-to-a-different-port)
13. [Firewall](#13-firewall)
14. [Management: `cmk-manage.sh` (CLI and ncurses interface)](#14-management-cmk-managesh)
15. [Backup](#15-backup)
16. [Restore](#16-restore)
17. [Migrating a bare-metal installation (`migrate_creator.sh`)](#17-migrating-a-bare-metal-installation-migrate_creatorsh)
18. [Several sites in one container](#18-several-sites-in-one-container)
19. [Upgrading Checkmk](#19-upgrading-checkmk)
20. [Troubleshooting](#20-troubleshooting)
21. [Security notes](#21-security-notes)
22. [Uninstalling](#22-uninstalling)

---

## 1. Overview & architecture

```
                    Internet / LAN
                          |
                     :80  :443  :81
                          |
        +-----------------------------------+
        |  NGINX Proxy Manager (container)  |   network_mode: host
        |  - Let's Encrypt OR self-signed   |
        |  - web interface on port 81       |
        +-----------------------------------+
                          |  http://127.0.0.1:5000
                          v
        +-----------------------------------+
        |  Checkmk / OMD (container)        |   network_mode: host
        |  :5000  web GUI (Apache)          |
        |  :8000  agent receiver            |   <-- agents (cmk-agent-ctl)
        |  :6557  Livestatus (optional)     |
        +-----------------------------------+
                          |
             bind mount: ./data/checkmk/sites  ->  /omd/sites
```

**Deliberate design decisions**

| Topic | Implementation | Rationale |
|---|---|---|
| Checkmk on the host network | `network_mode: host`, **no** `ports:` entries | agents, SNMP traps, ICMP and the agent receiver work without NAT; the source IP of the monitored hosts is preserved |
| NGINX Proxy Manager also on the host network | `network_mode: host` | reaches the Checkmk backend directly via `127.0.0.1:5000`; no bridge network, no `host-gateway` tricks – behaves identically under Docker **and** Podman |
| No named volumes | **bind mounts** into `./data/` only | all data sits visibly in the project directory, easy to back up, copy and inspect |
| One compose file for both | `docker compose` and `podman-compose` | identical operation on Debian and RHEL |
| `:Z` on the bind mounts | SELinux relabelling | required on RHEL; a no-op on Debian/Docker (not an error) |

---

## 2. Directory layout

```
.
├── README.md                        # German documentation
├── README_en.md                     # this file
├── docker-compose.yml               # for Docker AND Podman
├── .env.example                     # template -> .env
├── install.sh                       # installs Docker (Debian) / Podman (RHEL)
│
├── scripts/
│   ├── cmk-manage.sh                # stack, backup, restore, import + ncurses GUI
│   ├── migrate_creator.sh           # -> copy this to the SOURCE SERVER
│   └── lib/
│       └── common.sh                # shared functions + TUI helpers
│
├── containers/
│   └── browser/
│       └── Containerfile            # tools image: text browsers, curl, jq, openssl
│
├── systemd/
│   ├── checkmk-stack.service.example # compose-based unit (autostart)
│   └── quadlet/
│       ├── checkmk.container         # Podman Quadlet (alternative)
│       └── nginx-proxy-manager.container
│
├── data/                            # ALL payload data (bind mounts)
│   ├── checkmk/sites/               # -> /omd/sites   (complete OMD sites)
│   └── npm/
│       ├── data/                    # -> /data            (NPM config/DB)
│       ├── letsencrypt/             # -> /etc/letsencrypt (certificates)
│       └── custom-certs/            # self-signed certificates ready to upload
│
├── backups/                         # backups and migration bundles (*.tar.gz)
└── images/                          # downloaded Checkmk image tarballs
```

---

## 3. Requirements & ports

* 64-bit Linux, at least 2 CPU cores and 4 GB RAM (recommended: 4 cores, 8 GB)
* Disk space: rule of thumb 1 GB per 100 monitored hosts and year, plus backups
* root privileges
* Outbound internet access to download the images

### Ports in use (host network!)

| Port | Service | Reachable from |
|---|---|---|
| 80/tcp | NGINX Proxy Manager – HTTP, Let's Encrypt challenge | internet/LAN |
| 443/tcp | NGINX Proxy Manager – HTTPS | internet/LAN |
| 81/tcp | NGINX Proxy Manager – **admin interface** (default; changeable via [`npm-port`](#12-moving-the-admin-interface-to-a-different-port)) | admin network only! |
| 5000/tcp | Checkmk Apache (backend) | **localhost only** – do not expose. Exception: with the Proxy Manager turned off ([12.4](#124-turning-the-proxy-manager-off-entirely-external-nginx)), open it for the external proxy's IP only |
| 8000/tcp | Checkmk agent receiver (TLS-registered agents) | monitored hosts |
| 6557/tcp | Livestatus (only with `CMK_LIVESTATUS_TCP=on`) | distributed sites |

> **Important:** because both containers run on the host network, ports 80, 443
> and 81 must not already be taken on the host (e.g. by an installed
> Apache/NGINX). Check with `ss -tlnp | grep -E ':(80|81|443|5000)\b'`.

---

## 4. Installation on Debian/Ubuntu (Docker)

```bash
# put the project where it should live
sudo mkdir -p /opt/checkmk-stack
sudo cp -r ./* ./.env.example /opt/checkmk-stack/
cd /opt/checkmk-stack

# install Docker CE + compose plugin, prepare the project
sudo ./install.sh --with-systemd
```

`install.sh` does the following:

1. adds the Docker repository from `download.docker.com` (GPG key + sources list)
2. installs `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`,
   `docker-compose-plugin`
3. installs `dialog`, `openssl`, `tar`, `curl` (for the ncurses interface)
4. runs `systemctl enable --now docker`
5. creates the directories under `data/` and `backups/` and generates `.env`
   from `.env.example`
6. optionally sets up the systemd unit `checkmk-stack.service`

Then:

```bash
sudo $EDITOR /opt/checkmk-stack/.env     # at least CMK_PASSWORD and CMK_SITE_ID

# choose the image (Raw needs no login, Enterprise see chapter 8)
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest

sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status
```

Under Docker, `restart: unless-stopped` already brings the containers back
after a reboot (the Docker daemon starts them). The systemd unit is still
worthwhile because it additionally starts **all** sites
(see [chapter 18](#18-several-sites-in-one-container)).

---

## 5. Installation on RHEL/Rocky/Alma/Fedora (Podman)

Docker is not available on Red Hat systems – **Podman** is used instead. The
compose file is the same one.

> ### Step 0 on RHEL: check the subscription
>
> **An unregistered RHEL installs no packages at all** – neither `podman` nor
> `git`, `dialog` or `tar`. `dnf` merely reports *"This system is not registered
> with an entitlement server"*. This is the most common reason why `install.sh`
> does not get through on a fresh RHEL.
>
> ```bash
> # is the system registered?
> sudo subscription-manager status      # -> "Overall Status: Current" = fine
> sudo dnf repolist                     # -> must not be empty
>
> # if not: create a free account at https://developers.redhat.com
> # (Developer Subscription for Individuals, up to 16 systems) and then:
> sudo subscription-manager register --username <user>
> ```
>
> RHEL 9 uses *Simple Content Access*, so `subscription-manager attach` is **no
> longer** required. Without an account it also works from the installation DVD
> or by switching to Rocky/Alma – all three routes are covered in detail in
> [section 5.1](#51-unregistered-rhel-where-the-packages-come-from).
>
> **Rocky Linux, AlmaLinux, CentOS Stream and Fedora** need none of this – just
> read on.

```bash
sudo mkdir -p /opt/checkmk-stack
sudo cp -r ./* ./.env.example /opt/checkmk-stack/
cd /opt/checkmk-stack

sudo ./install.sh --with-systemd
```

`install.sh` installs:

1. first checks whether a usable repository exists at all, and otherwise names
   the three routes from [section 5.1](#51-unregistered-rhel-where-the-packages-come-from)
2. `podman`, `tar`, `openssl`
3. `dialog` for the ncurses interface – if that fails, `newt` (`whiptail`) is
   used instead; the scripts support either
4. `podman-compose` – attempted in this order:
   distribution repository → EPEL → `pip3 install podman-compose`
5. sets up the systemd unit (**always** with Podman, see the next chapter)

Done manually, that is:

```bash
sudo dnf install -y podman podman-compose tar openssl
sudo dnf install -y dialog || sudo dnf install -y newt   # dialog: AppStream, newt: BaseOS

# if podman-compose is not in the repository - add EPEL:
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
sudo dnf install -y podman-compose
```

> **EPEL requires a working subscription.** The `epel-release` package only
> configures the EPEL source; its dependencies and everything installed from it
> afterwards still come from Red Hat's BaseOS/AppStream. On an unregistered
> system even the command above therefore fails – EPEL is **not** a substitute
> for registering.
>
> If `podman-compose` stays out of reach because of this, it is no obstacle:
> with **Quadlet** ([chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose))
> the stack runs entirely without compose, straight through systemd.

Starting up:

```bash
sudo $EDITOR /opt/checkmk-stack/.env

# obtain the image - for Enterprise with credentials, see chapter 8
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>

sudo ./scripts/cmk-manage.sh up          # automatically uses podman-compose
sudo ./scripts/cmk-manage.sh status
```

> On RHEL, do read [chapter 8.2](#82-two-podman-pitfalls-the-ones-that-actually-bite):
> short image names and `podman import` instead of `podman load` are the two
> causes of "image not known".

### 5.1 Unregistered RHEL: where the packages come from

A freshly installed Red Hat Enterprise Linux without a subscription hands out
nothing at all:

```
$ sudo dnf install -y podman
This system is not registered with an entitlement server.
You can use "rhc" or "subscription-manager" to register.
Error: There are no enabled repositories in "/etc/yum.repos.d".
```

This affects **every** package – `podman`, `git`, `dialog`, `tar`. There are
three ways out, and `install.sh` now detects the situation and names them.

> Only RHEL itself is affected. **Rocky Linux, AlmaLinux, CentOS Stream and
> Fedora** need no registration – `dnf` works there straight away.

#### Route 1 – free Developer Subscription (recommended)

Red Hat hands out the *Developer Subscription for Individuals* free of charge.
It covers **up to 16 systems**, physical or virtual, and explicitly allows
**small production use** alongside development and testing. It is granted to
the individual, not to a company.

1. Create a free account at <https://developers.redhat.com>
2. Register the server:

```bash
sudo subscription-manager register --username <user>
# you will be prompted for the password; alternatively --password <password>
```

RHEL 9 uses *Simple Content Access* – the repositories are enabled right away,
a `subscription-manager attach` is **no longer** necessary.

3. Verify:

```bash
sudo subscription-manager status      # -> Overall Status: Current
sudo dnf repolist                     # -> rhel-9-for-x86_64-baseos-rpms etc.
sudo dnf install -y git podman
```

If individual repositories are missing:

```bash
sudo subscription-manager repos --list-enabled
sudo subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
```

With a corporate account and an activation key, use this instead:

```bash
sudo subscription-manager register --org <org-id> --activationkey <key>
```

#### Route 2 – installation DVD as a local repository (no account, offline)

Works without any registration and without internet access. You need the
**DVD ISO** (`rhel-9.x-x86_64-dvd.iso`, several GB) – the small `boot.iso`
contains no packages.

```bash
# mount the ISO permanently
sudo mkdir -p /mnt/rhel
sudo mount -o loop,ro /path/rhel-9.8-x86_64-dvd.iso /mnt/rhel

# mount it automatically at boot
echo '/path/rhel-9.8-x86_64-dvd.iso /mnt/rhel iso9660 loop,ro,nofail 0 0' \
  | sudo tee -a /etc/fstab

# create the repository file
sudo tee /etc/yum.repos.d/rhel-local.repo >/dev/null <<'REPO'
[rhel-local-baseos]
name=RHEL local - BaseOS
baseurl=file:///mnt/rhel/BaseOS
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[rhel-local-appstream]
name=RHEL local - AppStream
baseurl=file:///mnt/rhel/AppStream
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
REPO

sudo dnf clean all
sudo dnf repolist
sudo dnf install -y podman tar openssl dialog git
```

> **Limitation:** the local repository holds exactly the state of the ISO –
> there are **no security updates**. For a permanently running monitoring
> server that is not acceptable; as a stopgap until registration is in place it
> is fine.

#### Route 3 – switch to Rocky Linux or AlmaLinux

Both are binary-compatible rebuilds of RHEL, permanently free and without
registration. This project runs there unchanged – `install.sh` recognises them
as part of the RHEL family.

```bash
# Rocky Linux
sudo dnf install -y git
sudo git clone https://github.com/rocky-linux/rocky-tools.git
sudo bash rocky-tools/migrate2rocky/migrate2rocky.sh -r

# AlmaLinux
curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
sudo bash almalinux-deploy.sh
```

> Take a backup or a snapshot first. The conversion replaces the base packages
> on a running system.

#### What this project actually needs

| Package | Purpose | Repository |
|---|---|---|
| `podman` | container runtime | AppStream |
| `tar` | backups and migration bundles | BaseOS |
| `openssl` | self-signed certificates | BaseOS |
| `dialog` | ncurses interface | AppStream |
| `newt` (`whiptail`) | substitute if `dialog` is missing | BaseOS |
| `curl` | image download | BaseOS |
| `podman-compose` | compose support | EPEL or `pip3` |
| `git` | only needed to clone this project | AppStream |

`dialog` is **optional**: if it is missing, the scripts automatically use
`whiptail` from the `newt` package, which lives in BaseOS. `install.sh` tries
exactly this order and therefore no longer aborts.

`git` is not strictly required – the project can just as well be transferred as
a ZIP file or via `scp`.

> **EPEL does not help here:** the EPEL repository depends on Red Hat's base
> repositories. Without a subscription or a local repository you cannot even
> install `epel-release` – and therefore no `podman-compose`. In that case the
> Quadlet route from
> [chapter 6](#6-starting-automatically-after-a-host-reboot-podman) is the
> answer: it needs no compose at all.

### 5.2 SELinux

The bind mounts carry `:Z` in `docker-compose.yml`, so Podman automatically
applies the `container_file_t` context. Should access still fail:

```bash
sudo semanage fcontext -a -t container_file_t "/opt/checkmk-stack/data(/.*)?"
sudo restorecon -Rv /opt/checkmk-stack/data
sudo ausearch -m avc -ts recent          # inspect AVC messages
```

---

## 6. Starting automatically after a host reboot (Podman)

> **This is the most important difference from Docker:** Podman has *no
> daemon*. `restart: unless-stopped` in the compose file therefore does **not**
> survive a reboot. Autostart has to go through systemd. There are two proven
> ways.

### Option A – compose-based systemd unit (recommended, identical to Docker)

Created and enabled automatically by `./install.sh --with-systemd`. Manually:

```bash
sudo tee /etc/systemd/system/checkmk-stack.service >/dev/null <<'UNIT'
[Unit]
Description=Checkmk + NGINX Proxy Manager (container stack)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/checkmk-stack
EnvironmentFile=/opt/checkmk-stack/.env
TimeoutStartSec=0
ExecStart=/bin/sh -c '/usr/bin/podman-compose up -d'
ExecStartPost=/opt/checkmk-stack/scripts/cmk-manage.sh start-all-sites
ExecStop=/bin/sh -c '/usr/bin/podman-compose down'

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now checkmk-stack.service
sudo systemctl status checkmk-stack.service
```

Verify that it survives a reboot:

```bash
sudo systemctl is-enabled checkmk-stack     # -> enabled
sudo reboot
# after the restart:
sudo podman ps
sudo /opt/checkmk-stack/scripts/cmk-manage.sh status
```

`ExecStartPost` calls `start-all-sites`. This is necessary because the Checkmk
entrypoint only starts the site named in `CMK_SITE_ID` – with several sites the
remaining ones would stay stopped.

### Option B – Podman Quadlet (Podman 4.4 and newer, no compose)

Quadlet turns simple description files into real systemd units. Advantages: no
`podman-compose` required, clean dependencies, native Podman.

```bash
sudo ./install.sh --quadlet
```

Using the values from `.env`, this writes to `/etc/containers/systemd/`:

* `checkmk.container`
* `nginx-proxy-manager.container`

Then:

```bash
sudo systemctl daemon-reload
sudo systemctl start checkmk nginx-proxy-manager
sudo systemctl status checkmk
```

`WantedBy=multi-user.target default.target` makes the units start
automatically at boot – a separate `systemctl enable` is neither needed nor
possible with Quadlet, because the units are generated.

Templates live in [systemd/quadlet/](systemd/quadlet/).

### Option C – rootless Podman (optional, more effort)

```bash
sudo ./install.sh --rootless
sudo loginctl enable-linger checkmk           # service account
# as a user unit:
mkdir -p ~/.config/systemd/user
cp systemd/checkmk-stack.service.example ~/.config/systemd/user/checkmk-stack.service
systemctl --user daemon-reload
systemctl --user enable --now checkmk-stack
```

`--rootless` sets `net.ipv4.ip_unprivileged_port_start=80` so that ports 80/443
may be bound without root. `loginctl enable-linger` makes user units start
after a reboot even when nobody is logged in.

> Note: in rootless mode the UIDs of the site users are mapped through user
> namespaces. Files under `data/checkmk/sites` then belong to subordinate UIDs
> (e.g. `165536`). That is normal, but it makes working with the files from the
> host awkward. For migrations, **rootful** operation is recommended.

---

## 7. Configuration (`.env`)

```bash
cp .env.example .env
chmod 600 .env
$EDITOR .env
```

The most important settings:

| Variable | Meaning |
|---|---|
| `CMK_RUNTIME` | empty = auto-detect, otherwise `docker` or `podman` |
| `CMK_IMAGE` | **the image actually used**, always fully qualified – see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) |
| `CMK_EDITION`, `CMK_VERSION` | inputs for the image downloader only |
| `CMK_REGISTRY_USER/-PASSWORD` | login for `registry.checkmk.com` (Enterprise) |
| `CMK_DOWNLOAD_USER/-PASSWORD` | login for the Checkmk download portal |
| `CMK_SITE_ID` | the site the container creates and starts |
| `CMK_PASSWORD` | initial password for user `cmkadmin` (only on creation) |
| `CMK_LIVESTATUS_TCP` | `on` opens port 6557 for distributed monitoring |
| `MAIL_RELAY_HOST` | SMTP relay for notifications |
| `TZ` | time zone, e.g. `Europe/Berlin` |
| `NPM_ENABLED` | `on`/`off` – should the Proxy Manager container run at all? See [12.4](#124-turning-the-proxy-manager-off-entirely-external-nginx) |
| `NPM_ADMIN_PORT` | port of the Proxy Manager interface (default 81) – change only via `npm-port` or the menu |
| `NPM_ADMIN_URL` | start page of the terminal browser; empty = derived from `NPM_ADMIN_PORT` |
| `BACKUP_DIR` | location for **manual** backups (default `./backups`); scheduled jobs have their own paths |
| `BACKUP_AUTO` | `on`/`off` – should the backup jobs run via cron? |
| `CERT_DOMAIN`, `CERT_DAYS` | defaults for self-signed certificates |

### 7.1 Who reads `.env` – and why does it exist alongside the script?

`.env` is the **only** configuration file. There is deliberately no second
configuration inside the management script:

```
                    .env  (single source of truth)
                      |
      +---------------+----------------+------------------+
      |               |                |                  |
docker-compose    systemd unit    cmk-manage.sh      migrate_creator.sh
(variable         (EnvironmentFile)  (source)         (independent, runs on
 substitution)                                         the source server)
```

* **Read** by compose, by the systemd unit and by the script.
* **Written** by the script – either specifically (`image …` sets `CMK_IMAGE`,
  `import --set-default-site` sets `CMK_SITE_ID`, `npm-port` sets
  `NPM_ADMIN_PORT`) or through the settings menu / `config set`
  (see [chapter 14](#14-management-cmk-managesh)). An editor is no longer
  necessary, but remains available as a menu entry.
* The script itself only holds **fallback values** for the case that a key is
  missing from `.env` (in `scripts/lib/common.sh`, function `load_env`). It
  never overwrites a value that is set.

In short: `.env` says *what* runs, the script *does* something with it. Nothing
has to be maintained twice.

---

## 8. Obtaining the Checkmk image (Raw, Enterprise & co.)

### 8.1 Which edition lives where?

This is the most common pitfall: **the Enterprise edition is not on Docker
Hub.** The `checkmk` namespace on Docker Hub contains only:

| Edition (`CMK_EDITION`) | Source | Login |
|---|---|---|
| `raw` (up to 2.4) | `docker.io/checkmk/check-mk-raw` | no |
| `community` (2.5 and later, successor of raw) | `docker.io/checkmk/check-mk-community` | no |
| `cloud` | `docker.io/checkmk/check-mk-cloud` | no |
| `managed` (up to 2.4) | `docker.io/checkmk/check-mk-managed` | no |
| `pro` (2.5 and later, successor of enterprise) | `docker.io/checkmk/check-mk-pro` | no |
| `ultimate`, `ultimatemt` (2.5 and later) | `docker.io/checkmk/check-mk-ultimate[mt]` | no |
| **`enterprise` (2.0 – 2.4)** | **`registry.checkmk.com/enterprise/check-mk-enterprise`** or an image tarball from the customer portal | **yes** |
| `free` (historic, up to 2.1) | `docker.io/checkmk/check-mk-free` | no |

With Checkmk 2.5 the editions were renamed; those images are publicly available
on Docker Hub again. So if you run **2.5 or newer**, simply use `pro`. For
**2.2/2.3/2.4 Enterprise** the only routes are `registry.checkmk.com` or the
tarball.

#### Beware of the names

The rename is the second big pitfall. **checkmk.com only shows the new names,
while files and repositories keep the old ones up to 2.4.** For 2.3.0p9
specifically – verified against the version archive:

| Choice on checkmk.com | `--edition` here | actual file name |
|---|---|---|
| Community | `raw` | `check-mk-raw-docker-2.3.0p9.tar.gz` |
| **Pro** | **`enterprise`** | **`check-mk-enterprise-docker-2.3.0p9.tar.gz`** |
| Ultimate | `cloud` | `check-mk-cloud-docker-2.3.0p9.tar.gz` |
| Ultimate MT | `managed` | `check-mk-managed-docker-2.3.0p9.tar.gz` |

Picking "Pro" on the website and deriving
`check-mk-pro-docker-2.3.0p9.tar.gz` from it yields an **HTTP 404** – that file
does not exist for 2.3.

The script takes care of this: it reads the real file name from the version
archive and additionally tries both naming variants. In the interface the
edition list shows both names.

> Freely downloadable does not mean licence-free – the licence key is entered
> inside Checkmk afterwards.

### 8.2 Two Podman pitfalls (the ones that actually bite)

**1. Short image names.** Podman has no default registry. Short names such as
`checkmk/check-mk-enterprise:2.3.0p23` are resolved through
`unqualified-search-registries` in `/etc/containers/registries.conf` – and on
RHEL `registry.access.redhat.com` is listed **first**. Podman therefore looks
for the image at Red Hat, does not find it and reports `image not known`, or
asks interactively. That is why `.env` and `docker-compose.yml` consistently
use the **fully qualified** name:

```
CMK_IMAGE=registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p23
```

Inspect the current resolution order:

```bash
grep -A3 unqualified-search-registries /etc/containers/registries.conf
```

**2. `import` instead of `load`.** A downloaded image tarball is installed with
**`load`**:

| Command | Expects | Result |
|---|---|---|
| `podman import file.tar.gz` | a **filesystem** archive (rootfs) | an image **without** entrypoint, environment or tag → unusable, "image unknown" |
| `podman load -i file.tar.gz` | an **image** archive (`docker save`/OCI) | a complete image including metadata and tag ✔ |

Getting rid of a broken image created by `import`:

```bash
podman images | grep '<none>'
podman rmi <image-id>
```

`cmk-manage.sh image load` checks the archive for `manifest.json` or
`oci-layout` beforehand and rejects a wrong archive with exactly this hint.

### 8.3 Storing credentials in `.env`

```bash
CMK_EDITION=enterprise
CMK_VERSION=2.3.0p23

# route 1 - private registry
CMK_REGISTRY=registry.checkmk.com
CMK_REGISTRY_USER=<customer-portal-user>
CMK_REGISTRY_PASSWORD=<password>

# route 2 - tarball from the download portal
CMK_DOWNLOAD_BASEURL=https://download.checkmk.com/checkmk
CMK_DOWNLOAD_USER=<user>
CMK_DOWNLOAD_PASSWORD=<password>
```

The credentials come from the Checkmk customer portal (portal.checkmk.com).
`.env` should therefore be `chmod 600` – `install.sh` takes care of that.

### 8.4 Testing the credentials first

Rather than having a multi-gigabyte download fail on a wrong login, both can be
checked in seconds:

```bash
./scripts/cmk-manage.sh image test --edition enterprise --version 2.3.0p23
```

Nothing is downloaded and no login is stored. What gets checked:

| Check | What it answers |
|---|---|
| registry reachable | network, proxy, DNS |
| login accepted | are the username and password correct? |
| access to the repository | does the account have the required entitlement (subscription)? |
| download portal | is the image tarball there and is the login accepted? |

Example with correct credentials from `.env`:

```
Registry registry.checkmk.com
  [  OK  ] erreichbar (HTTP 401)
  [  OK  ] Anmeldung akzeptiert
  [  OK  ] Zugriff auf 'enterprise/check-mk-enterprise' vorhanden

Downloadportal download.checkmk.com
  [  OK  ] Datei erreichbar (HTTP 200)
           https://download.checkmk.com/checkmk/2.3.0p23/check-mk-enterprise-2.3.0p23.docker.tar.gz

[  OK  ] Zugangsweg nutzbar.
```

With a wrong password:

```
Registry registry.checkmk.com
  [  OK  ] erreichbar (HTTP 401)
  [FEHLER] Anmeldung abgelehnt (HTTP 401)
           Benutzername/Kennwort des Checkmk-Kundenportals pruefen
```

(`erreichbar` = reachable, `Anmeldung akzeptiert` = login accepted,
`Anmeldung abgelehnt` = login rejected, `Zugangsweg nutzbar` = a route works.)

Useful distinctions the test makes:

* **Login accepted, but no access to the repository** – the credentials are
  correct, but the subscription lacks the entitlement for this edition.
* **Login accepted, but HTTP 404 on the download portal** – username and
  password are right, only the version or file name does not match. Give the
  exact version (no `-latest`) or set `CMK_DOWNLOAD_URL`.
* For the freely available editions (`raw`, `pro`, `cloud`, …) the test reports
  "public repository – no login required" and still verifies that the
  repository can actually be reached.

Options:

```bash
# supply credentials once without writing them to .env
./scripts/cmk-manage.sh image test --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>

# check only one route
./scripts/cmk-manage.sh image test --only registry
./scripts/cmk-manage.sh image test --only download
```

The credentials are passed to `curl` through a temporary `.netrc` and therefore
do **not** appear in the process list. If `curl` is missing, the registry check
falls back to `podman login` / `docker login` – into a throwaway file as well,
so the system's stored login is left untouched.

In the ncurses interface the test can be reached from two places:

* **main menu → image → "Zugangsdaten prüfen"**
* **Einstellungen → "Checkmk-Image und Zugangsdaten" →
  ">> Zugangsdaten jetzt prüfen"** – right where the username and password
  are entered

After changing a credential field the menu also offers to run the test
straight away.

### 8.5 Listing the available versions

Instead of guessing version numbers:

```bash
./scripts/cmk-manage.sh image versions --edition enterprise --filter 2.3.0p
```

Up to three sources are queried:

| Source | `--source` | Credentials |
|---|---|---|
| version archive on checkmk.com | `web` | **no** |
| registry (Docker Hub or registry.checkmk.com) | `registry` | only for Enterprise |
| download portal (directory listing) | `download` | yes |

Without `--source` all three are tried – so even without stored credentials you
get a usable list:

```
Versionsarchiv  https://checkmk.com/download/archive
  (oeffentlich, keine Zugangsdaten noetig)
  2.3.0p50
  2.3.0p49
  2.3.0p48
  ...

Uebernehmen
  aus der Registry :  ./scripts/cmk-manage.sh image pull     --edition enterprise --version <version>
  als Tarball      :  ./scripts/cmk-manage.sh image download --edition enterprise --version <version>
  nur eintragen    :  ./scripts/cmk-manage.sh config set CMK_VERSION <version>
```

Further options: `--limit N` (default 25), `--all`, `--filter TEXT` and
`--bare` for the plain list without any framing – handy for scripting:

```bash
./scripts/cmk-manage.sh image versions --source web --filter 2.3.0p --bare | head -1
```

In the interface there is **image → "Verfügbare Versionen auflisten"**; and
when fetching an image the version no longer has to be typed at all – a
selection list appears instead.

### 8.6 Fetching the image – three ways

> **In the interface this is a wizard:** **image → "Image beschaffen"** walks
> through edition, version and source in one go, then sets `CMK_EDITION`,
> `CMK_VERSION` **and** `CMK_IMAGE`, and does **not** ask again for credentials
> already stored in `.env`. It only asks if they are missing for the chosen
> route – and offers to save them. The same wizard is also available under
> **Einstellungen → Checkmk-Image und Zugangsdaten → ">> Image beschaffen"**.

On the command line the individual steps remain:

**a) From a registry (recommended)**

For the Enterprise edition this is exactly the following two steps – the script
performs them in one call and writes the result into `.env`:

```bash
podman login registry.checkmk.com          # or docker login
podman pull registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p9
```


```bash
# freely available, no login
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0-latest

# Enterprise with a login to registry.checkmk.com
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>
```

**b) Image tarball from the download portal (with credentials)**

```bash
sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>
```

The downloader tries the usual file names under
`<CMK_DOWNLOAD_BASEURL>/<version>/`
(`check-mk-enterprise-<version>.docker.tar.gz` and
`check-mk-enterprise-docker-<version>.tar.gz`). Should Checkmk have changed the
scheme, the full URL can be given directly:

```bash
sudo ./scripts/cmk-manage.sh image download \
     --url 'https://download.checkmk.com/checkmk/2.3.0p23/check-mk-enterprise-2.3.0p23.docker.tar.gz' \
     --user <user> --password <password>
```

The credentials are handed to `curl` through a temporary `.netrc` file and
therefore do **not** appear in the process list.

**c) Loading a file you already downloaded**

If you fetched the file from the customer portal with a browser:

```bash
sudo ./scripts/cmk-manage.sh image load --file /root/check-mk-enterprise-2.3.0p23.docker.tar.gz
```

In all three cases the script determines the **actual** image name (for `load`
typically `localhost/checkmk/check-mk-enterprise:2.3.0p23`) and writes it to
`CMK_IMAGE` in `.env`. Afterwards:

```bash
sudo ./scripts/cmk-manage.sh up
```

### 8.7 `CMK_IMAGE` versus `CMK_EDITION`/`CMK_VERSION`

These three values have different jobs, and confusing them is a common
mistake:

| Value | Who reads it | What for |
|---|---|---|
| **`CMK_IMAGE`** | **compose and the systemd units** | the image that actually starts |
| `CMK_EDITION` | only `cmk-manage.sh image …` | input for the downloader |
| `CMK_VERSION` | only `cmk-manage.sh image …` | input for the downloader |

`CMK_IMAGE` can **not** be derived from edition and version – depending on how
you obtained it, the very same version looks completely different:

```
docker.io/checkmk/check-mk-raw:2.3.0-latest                 (Docker Hub)
registry.checkmk.com/enterprise/check-mk-enterprise:2.3.0p9 (registry)
localhost/checkmk/check-mk-enterprise:2.3.0p9               (loaded from a tarball)
```

So `CMK_IMAGE` records what is **actually there**, and it is written by
`image pull`, `image download`, `image load` and `image use` – you normally
never have to type anything there by hand. In the interface the three values
are therefore not editable individually: the wizard sets them together so they
cannot drift apart. On the command line they remain individually settable
(`config set …`).

On the command line, `image pull` and `image download` take `--remember` to
store the edition and version in `.env` as well:

```bash
./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p9 --remember
```

> **Consequence:** changing the edition or version does **not** start a
> different release. Only `image pull` (or `download`/`load`) writes the result
> to `CMK_IMAGE`, and only `up` recreates the container.

To stop these from drifting apart unnoticed, the script compares the three
values – in `image status`, in `status`, and in the settings menu right after
the edition or version is changed:

```
[ WARN ] CMK_IMAGE passt nicht zu den eingestellten Vorgaben.

  verwendet wird : docker.io/checkmk/check-mk-raw:2.3.0-latest
  Edition        : eingestellt "enterprise", im Image steckt "raw"
...
    ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0-latest
    ./scripts/cmk-manage.sh up
```

("CMK_IMAGE does not match the configured defaults / in use / configured … but
the image contains …".) Images loaded from a tarball (`localhost/…`) count as
consistent as long as name and tag match.

**If fetching fails**, `CMK_IMAGE` stays unchanged – deliberately, because the
old image keeps running. The wizard then says so explicitly ("Image wurde NICHT
geholt") instead of merely pointing at the mismatch. It still leaves a state in
which edition and version claim something different from the image in use.
There are two ways out, both in the menu underneath the three values:

| Route | Effect |
|---|---|
| **Assistent starten** (run the wizard) | fetches the wanted image; everything matches afterwards |
| **Vorgaben an das verwendete Image angleichen** (align the defaults) | writes edition and version back from `CMK_IMAGE` – the warning disappears, nothing about the running setup changes |

On the command line the second one is:

```bash
./scripts/cmk-manage.sh image use "$CMK_IMAGE" --remember
```

### 8.8 Finding the registry path

"Login Succeeded" does not yet mean the path is right. If the pull reports
`unable to copy from source docker://…`, the repository does not exist under
that name, or the subscription does not cover it:

```bash
./scripts/cmk-manage.sh image repos --edition enterprise
```

This first queries the catalog (`/v2/_catalog`) – which many registries block –
and then probes the usual paths:

```
Uebliche Pfade fuer "raw" durchprobiert
  STATUS TAGS     PFAD
  200 = nutzbar, 404 = gibt es nicht, 401/403 = kein Zugriff
  401    -        raw/check-mk-raw  (kein Zugriff - oder gibt es nicht)
  200    2394     checkmk/check-mk-raw
  ...
[  OK  ] Nutzbar: docker.io/checkmk/check-mk-raw
```

If an `image pull` against a private registry fails, this probe runs
automatically and shows what is actually there.

> **401 does not necessarily mean "it exists".** Many registries – Docker Hub
> among them – answer 401 for non-existent repositories too, so as not to
> reveal whether they exist. Only **200 = usable** and **404 = does not exist**
> are reliable.

If no path works, the tarball route remains. For Enterprise it is the
documented one anyway, and it works independently of the registry:

```bash
./scripts/cmk-manage.sh image download --edition enterprise --version 2.4.0p9
```

### 8.9 Checking what is in effect

```bash
./scripts/cmk-manage.sh image status     # configured image + local images
./scripts/cmk-manage.sh image list       # local Checkmk images only
./scripts/cmk-manage.sh image use localhost/checkmk/check-mk-enterprise:2.3.0p23
```

`image status` explicitly warns when `CMK_IMAGE` is a short name without a
registry.

The ncurses interface offers the same workflow under the menu entry
*"Checkmk-Image beschaffen/auswählen"* (pick edition → version → credentials →
fetch).

---

## 9. First start

```bash
sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status
```

Output (example, German as printed by the tool):

```
Container
  checkmk                  läuft
  nginx-proxy-manager      läuft

Sites im Container
  SITE                 VERSION            STATUS
  cmk                  2.3.0p23.cre       läuft

Zugang
  Checkmk (direkt)        : http://10.0.0.5:5000/cmk/
  Checkmk (über Proxy)    : https://<FQDN>/cmk/
  Agent-Receiver          : 10.0.0.5:8000
  NGINX Proxy Manager     : http://10.0.0.5:81/
```

(`läuft` = running, `gestoppt` = stopped, `Zugang` = access, `Erstlogin` =
initial login.) The port of the admin interface follows `NPM_ADMIN_PORT` from
`.env`; the output always shows the value actually configured.

Logging in to Checkmk: user `cmkadmin`, password from `CMK_PASSWORD`.

---

## 10. Setting up HTTPS – Let's Encrypt or self-signed

### 10.1 The administrator account

> **The former default credentials `admin@example.com` / `changeme` no longer
> exist as of version 2.15.** The container only creates the first account if
> `INITIAL_ADMIN_EMAIL` and `INITIAL_ADMIN_PASSWORD` are set at the **first**
> start – otherwise the browser shows a setup wizard. Verified in
> `backend/setup.js` of v2.15.1 and against a running container.

That is why the values live in `.env` and are passed through by
`docker-compose.yml`:

```bash
NPM_ADMIN_EMAIL=admin@example.com
NPM_ADMIN_PASSWORD=a-proper-password
```

Set both **before the first start**. The account then exists right away – and
with it the REST interface used by
[chapter 10.2](#102-creating-a-proxy-host--without-a-browser).

If the stack has already run and the account was created by hand through the
wizard, the variables no longer take effect (the database is no longer empty).
Simply enter the credentials you chose – the script logs in with those.

### 10.2 Creating a proxy host – without a browser

The web interface is painful in a terminal browser, and forms with file uploads
do not work there at all. Since the Proxy Manager runs locally anyway, its
interface is the more reliable route:

```bash
# publish Checkmk at checkmk.example.com with a self-signed certificate
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --cert self

# or with Let's Encrypt (port 80 must be reachable from the internet)
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --cert letsencrypt

# forwarding to a different port instead of 5000
sudo ./scripts/cmk-manage.sh proxy add --domain checkmk.example.com --forward-port 8080
```

In one go the command: creates the certificate (or requests it from Let's
Encrypt), uploads it to the Proxy Manager, creates the proxy host, forces
HTTPS, enables HTTP/2 and websockets, and sets the recommended
`advanced_config` block.

| Command | Effect |
|---|---|
| `proxy status` | check the login, count hosts and certificates |
| `proxy list` | existing proxy hosts |
| `proxy certs` | existing certificates with expiry date |
| `proxy add …` | create a proxy host (`--update` replaces an existing one) |
| `proxy del --id N` | delete a proxy host |

Options for `proxy add`:

| Option | Default | Meaning |
|---|---|---|
| `--domain FQDN` | `PROXY_DOMAIN` | public name |
| `--forward-host H` | `127.0.0.1` | forwarding target |
| `--forward-port P` | `5000` | target port – **freely selectable**, not fixed to Checkmk |
| `--scheme http\|https` | `http` | scheme towards the target |
| `--cert self\|letsencrypt\|none` | `self` | certificate |
| `--update` | – | replace an existing entry |
| `--advanced TEXT` | see below | custom nginx block |
| `--no-force-ssl`, `--no-http2`, `--no-websockets`, `--no-block-exploits` | – | switches |

In the interface the same flow is under **proxy → "Proxy-Host anlegen"**
(name → target port → certificate) and under **Einstellungen → NGINX Proxy
Manager → ">> Proxy-Hosts anlegen/verwalten"**.

> A self-signed certificate is **reused** on repeated runs instead of being
> created anew each time – otherwise dozens of entries with the same name pile
> up in the Proxy Manager.

### 10.3 If you do want the web interface

1. open `http://<server>:81` – or whichever port is set in `NPM_ADMIN_PORT`
   (see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)); if
   that port is blocked on the network, use the terminal browser from
   [chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser)
2. log in with `NPM_ADMIN_EMAIL` / `NPM_ADMIN_PASSWORD`

### 10.4 Creating a proxy host in the browser

**Hosts → Proxy Hosts → Add Proxy Host**

| Field | Value |
|---|---|
| Domain Names | `checkmk.example.com` (or your internal name) |
| Scheme | `http` |
| Forward Hostname / IP | `127.0.0.1` |
| Forward Port | `5000` |
| Block Common Exploits | on |
| Websockets Support | **on** (for the live data in the interface) |

Tab **Advanced** (recommended, so Checkmk sees the real client IP and large
agent uploads work):

```nginx
client_max_body_size 0;
proxy_read_timeout 300s;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```

### 10.5 Option A – Let's Encrypt (publicly reachable server)

Prerequisites: the DNS name resolves publicly to this server and **port 80 is
reachable from the internet** (HTTP-01 challenge).

In the proxy host, tab **SSL**:

* SSL Certificate → **Request a new SSL Certificate**
* *Force SSL* on
* *HTTP/2 Support* on
* *HSTS Enabled* as required
* enter an e-mail address, accept the terms → **Save**

Renewal is handled by the Proxy Manager automatically; the certificates live in
the bind mount `data/npm/letsencrypt/` and are therefore kept inside the
project directory.

For an internal DNS zone without public access, use the **DNS challenge**
(SSL → *Use a DNS Challenge* → select provider, store the API token).

### 10.6 Option B – self-signed certificate (internal environment)

Creating one:

```bash
sudo ./scripts/cmk-manage.sh cert \
     --domain checkmk.intern.example \
     --alt 10.0.0.5 --alt checkmk \
     --days 3650
```

Result in `data/npm/custom-certs/checkmk.intern.example/`:

* `privkey.pem` – private key
* `fullchain.pem` – certificate (with SANs for every name/IP given)

Installing it:

1. **SSL Certificates → Add SSL Certificate → Custom**
2. give it a name, upload `privkey.pem` as *Certificate Key* and
   `fullchain.pem` as *Certificate*
3. select this certificate in the proxy host under **SSL**, enable *Force SSL*
   and *HTTP/2*

> For Checkmk agents and browsers to trust the certificate, `fullchain.pem`
> must be installed as a trusted certificate on the clients (Windows: computer
> certificates → Trusted Root Certification Authorities; Linux:
> `/usr/local/share/ca-certificates/` + `update-ca-certificates`).
> For plain agent communication over port 8000 this is **not** needed – there
> Checkmk uses its own site-internal CA.

The ncurses interface offers the same workflow under *"Selbstsigniertes
Zertifikat erzeugen"*.

---

## 11. Configuring the Proxy Manager locally (terminal browser)

When port 81 cannot be reached from the network, the NGINX Proxy Manager's
admin interface has to be operated **on the machine itself**. For that,
`cmk-manage.sh browser` starts another container on the host network – which
reaches `127.0.0.1:81` just like the Proxy Manager itself.

```bash
sudo ./scripts/cmk-manage.sh browser
```

This is deliberately **not** a service in `docker-compose.yml`: a browser is an
interactive program with a terminal, not a background service. It is started
with `run --rm -it` and disappears without a trace when you quit.

### 11.1 Why not `lynx`

The Proxy Manager's admin interface is a JavaScript application. Classic text
browsers **cannot** render it – you get a blank page. There are therefore two
engines that render a full browser inside the terminal:

| Engine | Based on | JavaScript | Proxy Manager usable |
|---|---|---|---|
| `browsh` | Firefox | yes | **yes** – default, maintained |
| `carbonyl` | Chromium | yes | **yes** – nicer rendering, last released 2023 |
| `links` (links2) | text browser | no | no |
| `w3m` | text browser | no | no |
| `lynx` | text browser | no | no |
| `shell` | Bash + curl/jq/openssl | – | for diagnostics and the API |

The text browsers are included anyway because they are handy for quick checks.
If you pick one, the script warns explicitly.

### 11.2 Usage

```bash
# default: browsh on http://127.0.0.1:81
sudo ./scripts/cmk-manage.sh browser

# a different engine or a different page
sudo ./scripts/cmk-manage.sh browser --engine carbonyl
sudo ./scripts/cmk-manage.sh browser --engine links --url http://127.0.0.1:5000/cmk/

# shell with network tools on the host network
sudo ./scripts/cmk-manage.sh browser --engine shell
```

Key bindings in **browsh**:

| Key | Action |
|---|---|
| `Ctrl+L` | address bar |
| `Ctrl+T` | new tab |
| `Ctrl+W` | close tab |
| `Ctrl+Q` | quit |

The first time a text-browser or `shell` engine is used, the script builds a
small tools image (Debian slim with links2, w3m, lynx, curl, jq, openssl,
roughly 170 MB). Build it in advance – this works without a terminal, e.g. from
an installation script:

```bash
sudo ./scripts/cmk-manage.sh browser --build-only
```

`browsh` and `carbonyl`, by contrast, are pulled as ready-made images; their
names are in `.env` (`BROWSH_IMAGE`, `CARBONYL_IMAGE`) and are fully qualified
so that Podman does not look for them at `registry.access.redhat.com`.

In the ncurses interface everything lives under *"Terminal-Browser (Proxy
Manager lokal bedienen)"*.

### 11.3 Blank screen: the start page does not answer

If everything stays black after `Starting Browsh …` /
`Waiting for Firefox to connect…`, it is almost always because **nothing
answers on the start page** – not because of the browser. A terminal browser
shows no error page in that case, just nothing. You can tell that Browsh itself
is running because `Ctrl+Q` still returns you to the menu.

The script therefore checks the start page beforehand and names the cause:

```
[ WARN ] Auf http://127.0.0.1:81 antwortet nichts.
Ein Terminal-Browser zeigt dann nur eine schwarze Seite.

Das ist die Oberflaeche des NGINX Proxy Managers.
Er ist abgeschaltet (NPM_ENABLED=off):
    ./scripts/cmk-manage.sh npm-enable on

Trotzdem starten? [j/N]
```

("nothing answers on …", "a terminal browser then shows only a black page",
"it is switched off", "start anyway?") The cases it distinguishes:

| Situation | Message |
|---|---|
| Proxy Manager switched off | points to `npm-enable on` |
| container missing or stopped | points to `up` |
| container runs but does not answer | points to `logs` and `npm-port` |
| Checkmk interface (port 5000) dead | points to `up` or `sites` |
| external address | points to the host's network access |

`--no-check` skips the check. In the interface the same diagnosis appears as a
dialog offering to start anyway or to pick a different start page from the
menu.

### 11.4 Limitation: file upload

A terminal browser cannot open the operating system's file chooser. For
**Let's Encrypt** that does not matter (only form fields are filled in), but
uploading a **self-signed certificate** (*SSL Certificates → Custom*) does not
work reliably this way.

For exactly this case, use the Proxy Manager's REST interface. Create the
certificate as described in
[chapter 10.6](#106-option-b--self-signed-certificate-internal-environment), then:

```bash
NPM=http://127.0.0.1:81
CERTDIR=./data/npm/custom-certs/checkmk.intern.example

# 1) log in and obtain a token
TOKEN=$(curl -s -X POST "$NPM/api/tokens" \
  -H 'Content-Type: application/json' \
  -d '{"identity":"admin@example.com","secret":"<password>"}' | jq -r .token)

# 2) create the certificate entry (provider "other" = your own certificate)
CERT_ID=$(curl -s -X POST "$NPM/api/nginx/certificates" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"provider":"other","nice_name":"checkmk-selfsigned","domain_names":["checkmk.intern.example"]}' \
  | jq -r .id)

# 3) upload key and certificate
curl -s -X POST "$NPM/api/nginx/certificates/$CERT_ID/upload" \
  -H "Authorization: Bearer $TOKEN" \
  -F "certificate=@$CERTDIR/fullchain.pem" \
  -F "certificate_key=@$CERTDIR/privkey.pem"

# 4) create the proxy host for Checkmk
curl -s -X POST "$NPM/api/nginx/proxy-hosts" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d "{\"domain_names\":[\"checkmk.intern.example\"],
       \"forward_scheme\":\"http\",
       \"forward_host\":\"127.0.0.1\",
       \"forward_port\":5000,
       \"certificate_id\":$CERT_ID,
       \"ssl_forced\":true,
       \"http2_support\":true,
       \"block_exploits\":true,
       \"allow_websocket_upgrade\":true,
       \"advanced_config\":\"client_max_body_size 0;\",
       \"enabled\":true,
       \"meta\":{}}"
```

`curl` and `jq` are part of the tools image in case the host lacks them:

```bash
sudo ./scripts/cmk-manage.sh browser --engine shell
```

> On the very first login the Proxy Manager forces you to change the e-mail
> address and password. That is easiest in the browser
> (`cmk-manage.sh browser`); afterwards the API works with the new credentials.

### 11.5 Alternative: SSH tunnel

If SSH access to the machine is possible, a tunnel is the most convenient
route – you then use the interface in your usual browser:

```bash
ssh -N -L 8081:127.0.0.1:81 user@checkmk-server
# then locally:  http://127.0.0.1:8081
```

This changes nothing about the firewall: port 81 stays closed from outside,
access runs through the existing SSH connection.

---

## 12. Moving the admin interface to a different port

Inside the image, the NGINX Proxy Manager's web interface listens on a
**hard-coded port 81** (`listen 81 default;` in
`/etc/nginx/conf.d/production.conf`). No environment variable for this exists
in the published 2.x versions – `NPM_ADMIN_PORT` is already prepared in the
development branch, but is not part of version 2.15.1. And because both
containers run on the host network, the port cannot be moved by port mapping
either.

So if port 81 is blocked on your network:

```bash
sudo ./scripts/cmk-manage.sh npm-port 10443
```

### 12.1 What happens

Within its `http{}` block, nginx includes an official extension point:

```
include /data/nginx/custom/http.conf;
```

`/data` is our bind mount `./data/npm/data`. The command places an additional
server there which listens on the desired port and forwards to the interface on
`127.0.0.1:81`:

```
./data/npm/data/nginx/custom/http.conf

# >>> cmk-manage.sh: Admin-Port - nicht von Hand aendern >>>
server {
    listen 10443;
    listen [::]:10443;
    server_name _;
    client_max_body_size 0;
    location / {
        proxy_pass http://127.0.0.1:81;
        ...
    }
}
# <<< cmk-manage.sh: Admin-Port <<<
```

Only the region between the two markers is managed – your own additions to the
same file are left untouched. The script then restarts the Proxy Manager
container so nginx reads the file, and writes `NPM_ADMIN_PORT` and
`NPM_ADMIN_URL` into `.env`. The terminal browser from
[chapter 11](#11-configuring-the-proxy-manager-locally-terminal-browser) then
starts on the new port automatically.

### 12.2 Important: port 81 stays alive locally

The built-in server on port 81 **cannot** be switched off this way – it keeps
listening on all addresses. That is harmless as long as the firewall blocks it,
and that is exactly the recommendation:

```bash
# firewalld
sudo firewall-cmd --permanent --add-port=10443/tcp
sudo firewall-cmd --permanent --remove-port=81/tcp 2>/dev/null
sudo firewall-cmd --reload

# ufw
sudo ufw allow 10443/tcp
sudo ufw delete allow 81/tcp 2>/dev/null
```

### 12.3 Inspecting and reverting

```bash
./scripts/cmk-manage.sh npm-port          # show the current state
sudo ./scripts/cmk-manage.sh npm-port 81  # back to the default
```

With `81` the managed block is removed again; if the file is empty afterwards
it is deleted entirely. Only the configuration from the image then applies.

The command rejects ports already taken by the stack (80, 443, 5000, 8000,
6557), validates the numeric range, and warns if another service is already
listening on the port you want.

> As soon as an NPM release ships `NPM_ADMIN_PORT`, this detour becomes
> unnecessary – the environment variable will be enough. Until then, the
> extension point is the route that needs no change to the image.

### 12.4 Turning the Proxy Manager off entirely (external NGINX)

If HTTPS termination is already handled elsewhere – for instance by an NGINX on
an upstream system – the container is not needed at all. There is a switch for
that:

```bash
sudo ./scripts/cmk-manage.sh npm-enable off    # turn off
sudo ./scripts/cmk-manage.sh npm-enable on     # turn back on
./scripts/cmk-manage.sh npm-enable             # show the current state
```

In the interface it is under **Einstellungen → NGINX Proxy Manager →
"Proxy Manager betreiben"**; in `.env` it is called `NPM_ENABLED`.

**What "off" means:** the container is stopped and removed, and is not created
by `up` in the first place – `compose` is only told about the Checkmk service.
So no image is pulled and no container is built:

```
NPM_ENABLED=off  ->  docker compose up -d checkmk
NPM_ENABLED=on   ->  docker compose up -d checkmk nginx-proxy-manager
```

**The data stays.** Configuration, database and certificates live in the bind
mounts `./data/npm/data` and `./data/npm/letsencrypt`, not inside the
container. Before turning it off, the command shows both with their sizes; when
you turn it back on, the existing configuration and certificates are picked up
unchanged.

**What to do afterwards:** the external proxy forwards to port 5000 of this
host. That port was meant for localhost only so far and now has to be reachable
for the proxy – as narrowly as possible:

```bash
# firewalld
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=<proxy-ip> port port=5000 protocol=tcp accept'
sudo firewall-cmd --reload

# ufw
sudo ufw allow from <proxy-ip> to any port 5000 proto tcp
```

In the external NGINX:

```nginx
location / {
    proxy_pass http://<checkmk-host>:5000;
    proxy_set_header Host               $host;
    proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto  $scheme;
    client_max_body_size 0;
    proxy_read_timeout 300s;
}
```

`npm-enable off` prints the matching commands and this example configuration
for you.

> **The agent receiver is unaffected.** Agents talk to port 8000 of this host
> directly, not through the proxy – so it must stay reachable for the monitored
> systems.

> **With Quadlet** ([chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose))
> `install.sh --quadlet` removes the `nginx-proxy-manager.container` unit while
> the switch is `off`. If you turn it on later, simply run
> `sudo ./install.sh --quadlet` again.

---

## 13. Firewall

**firewalld (RHEL/Rocky/Alma):**

```bash
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-port=8000/tcp          # agent receiver
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=10.0.0.0/24 port port=81 protocol=tcp accept'
sudo firewall-cmd --reload

# With a different admin port (e.g. 10443) open that number instead -
# "cmk-manage.sh npm-port <port>" prints the matching command for you.
```

**ufw (Debian/Ubuntu):**

```bash
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8000/tcp
sudo ufw allow from 10.0.0.0/24 to any port 81 proto tcp
sudo ufw enable
```

Port **5000 stays closed** – access goes exclusively through the proxy. Open
port **81** to the administration network only.

---

## 14. Management: `cmk-manage.sh`

The script is both a command-line tool **and** an ncurses interface.

### 14.1 ncurses interface (Midnight Commander style)

```bash
sudo ./scripts/cmk-manage.sh gui
```

The menu labels are German; the command table in
[14.2](#142-command-line) lists the same actions in English.

```
 ┌──────────────── Checkmk Stack Manager ─────────────────┐
 │ Projekt : /opt/checkmk-stack                           │
 │ Runtime : podman     Standard-Site: cmk                │
 │ Backups : /opt/checkmk-stack/backups                   │
 │                                                        │
 │   status    Status von Containern und Sites            │
 │   stack     Stack steuern (Start/Stop/Logs/Update)     │
 │   backup    Backup erstellen                           │
 │   restore   Backup wiederherstellen                    │
 │   import    Site von Direktinstallation importieren    │
 │   manage    Backups verwalten (Liste/Details/Löschen)  │
 │   sites     Sites im Container verwalten               │
 │   image     Checkmk-Image beschaffen/auswählen         │
 │   browser   Terminal-Browser (Proxy Manager lokal)     │
 │   cert      Selbstsigniertes Zertifikat erzeugen       │
 │   config    Einstellungen (.env) - menügeführt         │
 │   shell     Root-Shell im Checkmk-Container            │
 │   quit      Beenden                                    │
 │                  <  OK  >   <Cancel>                   │
 └────────────────────────────────────────────────────────┘
```

Operation: arrow keys, **space** toggles entries in selection lists (e.g.
several sites for a backup), **Enter** confirms, **ESC/Cancel** goes back.
The window size adapts to the terminal size, the length of the text and the
number of entries – in a larger terminal more entries are visible at once.
`dialog` is preferred; if only `whiptail` is present, that is used instead.
A file chooser with directory navigation is built in.

### 14.2 Command line

```bash
./scripts/cmk-manage.sh --help
```

| Command | Effect |
|---|---|
| `up` / `down` / `restart` / `pull` | control the stack |
| `status` | containers, sites, access addresses |
| `logs [-f]` | container logs |
| `sites` | list the sites in the container |
| `site-start SITE` / `site-stop SITE` | control a single site |
| `start-all-sites` | start all sites (invoked by systemd) |
| `backup …` | backup as a `.tar.gz` bundle |
| `list` / `verify` / `prune --keep N` | manage backups |
| `job list\|add\|set\|del\|run\|auto` | scheduled backups (own target path per job) |
| `restore …` | restore a backup |
| `import …` | import a migration bundle |
| `image status\|list\|versions\|test\|pull\|download\|load\|use` | obtain and select the Checkmk image |
| `browser [--engine …]` | terminal browser in a container – operate the Proxy Manager locally |
| `config list\|get\|set\|edit` | show and change settings in `.env` |
| `npm-port [PORT]` | set/show the port of the Proxy Manager interface |
| `npm-enable [on\|off]` | turn the Proxy Manager on/off (data is preserved) |
| `proxy status\|list\|certs\|add\|del` | create and manage proxy hosts through the API |
| `cert …` | create a self-signed certificate |
| `shell` | root shell in the Checkmk container |
| `omd …` | any `omd` command inside the container |

Global options: `--env-file FILE`, `-y/--yes` (no prompts, e.g. for cron jobs),
`-v/--verbose`.

### 14.3 Changing settings – without an editor

Every value in `.env` can be maintained through the menu. In the interface:
**Einstellungen (.env) – menügeführt**. The values are grouped by area:

```
 ┌──────────────────── Einstellungen ─────────────────────┐
 │ Alle Werte der Datei .env - direkt hier änderbar.      │
 │ Datei: /opt/checkmk-stack/.env                         │
 │                                                        │
 │   runtime   Runtime und Zeitzone                       │
 │   image     Checkmk-Image und Zugangsdaten             │
 │   checkmk   Checkmk-Site und Container                 │
 │   npm       NGINX Proxy Manager                        │
 │   browser   Terminal-Browser                           │
 │   paths     Verzeichnisse                              │
 │   cert      Selbstsignierte Zertifikate                │
 │   show      Alle Werte auflisten                       │
 │   editor    Datei .env im Editor öffnen                │
 │   back      zurück                                     │
 └────────────────────────────────────────────────────────┘
```

(runtime and time zone / Checkmk image and credentials / Checkmk site and
containers / NGINX Proxy Manager / terminal browser / directories / self-signed
certificates / list all values / open `.env` in an editor / back.)

Each entry shows its current value and, when confirmed, opens the matching
input: free text, number, selection list or password field (passwords appear as
`********` in the overview and stay unchanged if you leave the field empty).
The editor remains available as a menu entry.

The same values from the command line:

```bash
./scripts/cmk-manage.sh config list                  # everything with current values
./scripts/cmk-manage.sh config list --category npm   # one area only
./scripts/cmk-manage.sh config get CMK_SITE_ID
./scripts/cmk-manage.sh config set CMK_SITE_ID prod
./scripts/cmk-manage.sh config set NPM_ADMIN_PORT 10443
```

**Validation happens when setting the value**, not at the next start:

| Key | Check |
|---|---|
| `NPM_ADMIN_PORT` | number 1–65535, not 80/443/5000/8000/6557, warning if in use |
| `CMK_SITE_ID` | letters, digits and `_` only; at most 16 characters |
| `CMK_IMAGE`, `NPM_IMAGE`, … | must be fully qualified (the Podman pitfall) |
| `CMK_RUNTIME` | must be installed on this system |
| `CERT_DAYS` and others | numeric fields |
| URL fields | must start with `http://` or `https://` |

Values containing spaces are quoted automatically so that `.env` remains
readable by compose, systemd **and** the shell. When you change
`NPM_ADMIN_PORT`, the script updates the additional nginx file along with it
(see [chapter 12](#12-moving-the-admin-interface-to-a-different-port)). For
values that only take effect once the containers are recreated, the menu says
so explicitly.

---

## 15. Backup

Backups are created with `omd backup` and packed into a **bundle**:

```
cmkbackup_<site>_<version>_<timestamp>.tar.gz
└── cmkbundle/
    ├── MANIFEST              site, version, source, checksum …
    ├── INFO.txt              human-readable summary
    ├── site.tar.gz           the actual omd backup
    └── site.tar.gz.sha256
```

This way the restore/import tool always knows **which site** and **which
Checkmk version** the archive contains, and can verify its integrity.

```bash
# a single site
sudo ./scripts/cmk-manage.sh backup --site cmk

# all sites, without log files, with a label
sudo ./scripts/cmk-manage.sh backup --all --no-logs --label vor-update

# most consistent backup: the site is stopped briefly
sudo ./scripts/cmk-manage.sh backup --site cmk --stop

# without performance data (considerably smaller)
sudo ./scripts/cmk-manage.sh backup --site cmk --no-rrds --dir /mnt/nas/checkmk
```

Listing and verifying:

```bash
./scripts/cmk-manage.sh list
./scripts/cmk-manage.sh verify --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz
./scripts/cmk-manage.sh prune --keep 7
```

### 15.1 Scheduled backups (backup jobs)

Manual backups always go to `BACKUP_DIR` from `.env`. Scheduled jobs, by
contrast, bring **their own target directory** – so you can back up to a NAS
share daily and somewhere else weekly.

In the menu: **backup → "Backup erstellen / planen"** (create / schedule)

```
 ┌──────────── Backup erstellen / planen ─────────────────┐
 │ Manuelle Backups gehen immer nach:                     │
 │   /opt/checkmk-stack/backups   (Einstellung BACKUP_DIR)│
 │                                                        │
 │ Geplante Jobs haben ihr eigenes Zielverzeichnis.       │
 │ Automatische Ausfuehrung: aus                          │
 │                                                        │
 │   manuell   Backup jetzt erstellen                     │
 │   jobs      Backup-Jobs verwalten                      │
 │   auto      Automatische Ausfuehrung ein/aus           │
 │   status    Zeitplan und Jobs anzeigen                 │
 │   jetzt     Einen Job sofort ausfuehren                │
 │   back      zurueck                                    │
 └────────────────────────────────────────────────────────┘
```

(create now / manage jobs / automatic execution on-off / show schedule / run a
job now / back.) The same jobs are also reachable from
**Einstellungen → "Backup-Jobs"**.

#### Creating a job

In the menu a dialog walks through sites, schedule (daily / weekly / monthly /
free cron expression), target directory, retention and options. The same with
parameters:

```bash
# daily at 02:15, all sites, without logs, keep 14 backups
sudo ./scripts/cmk-manage.sh job add --name nachts \
     --cron "15 2 * * *" --no-logs --keep 14

# Sundays at 03:30, two sites only, to the NAS, without RRDs, keep 8
sudo ./scripts/cmk-manage.sh job add --name nas-woche \
     --cron "30 3 * * 0" --sites prod,test \
     --dir /mnt/nas/checkmk --no-rrds --no-logs --keep 8
```

All commands:

| Command | Effect |
|---|---|
| `job list` | show the configured jobs |
| `job status` | schedule, cron file and switch |
| `job add --name N --cron "…"` | create a job |
| `job set NAME …` | modify a job (fields not given stay as they are) |
| `job del NAME` | delete a job (the backups remain) |
| `job enable\|disable NAME` | turn a job on/off |
| `job run NAME` | run a job right now |
| `job run-all` | run every active job |
| `job auto on\|off` | turn automatic execution on/off |

Options for `job add` / `job set`:

| Option | Meaning |
|---|---|
| `--sites ALL\|a,b` | sites to back up (default `ALL`) |
| `--cron "15 2 * * *"` | schedule, five cron fields |
| `--dir DIR` | this job's target directory (default `BACKUP_DIR`) |
| `--keep N` | how many backups stay in the target (`0` = all) |
| `--no-rrds`, `--no-logs`, `--stop` | as for `backup` |
| `--disabled` | create it but do not schedule it yet |

#### Turning automation on and off

```bash
sudo ./scripts/cmk-manage.sh job auto on     # schedule
sudo ./scripts/cmk-manage.sh job auto off    # remove the schedule
```

`on` writes `/etc/cron.d/checkmk-backup` from all **active** jobs – one line
per job with its own schedule:

```cron
15 2 * * * root /opt/checkmk-stack/scripts/cmk-manage.sh -y job run nachts    >>/var/log/cmk-backup.log 2>&1
30 3 * * 0 root /opt/checkmk-stack/scripts/cmk-manage.sh -y job run nas-woche >>/var/log/cmk-backup.log 2>&1
```

This needs root. Without root the command prints the lines so you can add them
yourself. `off` removes the file again; the jobs themselves are kept. The
switch is stored as `BACKUP_AUTO` in `.env`.

After each run a job prunes its own target directory down to `--keep` – a
separate `prune` entry is not needed.

#### Where the jobs live

In `backup-jobs.conf` in the project directory, one line per job:

```
name|sites|cron|directory|options|keep|enabled
nachts|ALL|15 2 * * *|/opt/checkmk-stack/backups|--no-logs|14|on
nas-woche|prod,test|30 3 * * 0|/mnt/nas/checkmk|--no-rrds --no-logs|8|on
```

The file is deliberately readable and diffable, but does not belong in the
repository (it is in `.gitignore` because it contains system-specific paths).

---

## 16. Restore

```bash
# simplest case: the site no longer exists
sudo ./scripts/cmk-manage.sh restore --file backups/cmkbackup_cmk_2.3.0p23.cre_20260820-030000.tar.gz

# overwrite an existing site (terminating running processes)
sudo ./scripts/cmk-manage.sh restore --file backups/… --reuse --kill

# restore under a different name (e.g. for a test)
sudo ./scripts/cmk-manage.sh restore --file backups/… --as-site test
```

What the script does:

1. checks the archive structure and the SHA256 checksum
2. checks whether the **Checkmk version of the backup exists in the container**
3. prompts / requires `--reuse` for an existing site, stops the site
4. copies the archive into the container and runs `omd restore`
5. turns off `TMPFS` for the site (not usable inside the container)
6. starts the site and prints the next steps

> **`--as-site` changes the site name** – registered agents would then point
> nowhere and would have to be re-registered. So do **not** use it for a
> migration.

A raw `omd backup` archive (without the bundle wrapper) is accepted as well;
the site name is then read from the archive.

---

## 17. Migrating a bare-metal installation (`migrate_creator.sh`)

Goal: take over an existing Checkmk installation on hardware or a VM into the
container – **without touching the agents**.

### 17.1 Why the agents keep working

An `omd backup` contains the complete site including `etc/ssl` (the site's own
CA and the certificates of the registered agents), `etc/check_mk` (rules,
hosts, passwords) and the monitoring history. If the site is restored **under
the same name** and the server is reachable at **the same IP address**, then

* push/pull agents (`cmk-agent-ctl`) find their registration again,
* the TLS connection to the agent receiver on **port 8000** stays valid,
* all URLs (`https://<ip>/<site>/check_mk/…`) remain correct,
* performance data and history are preserved.

The prerequisite is therefore that the target system **takes over the source
server's IP** (or that the DNS name is repointed) – which is the plan anyway.

### 17.2 Step 1 – copy the script to the source server

```bash
scp scripts/migrate_creator.sh root@old-server:/root/
ssh root@old-server 'chmod +x /root/migrate_creator.sh'
```

The script is **self-contained** – on the source server it only needs `bash`,
`tar` and `omd` (plus `dialog` or `whiptail` for the interface).

### 17.3 Step 2 – create the bundle on the source server

Show the existing sites:

```bash
sudo ./migrate_creator.sh --list
```

```
SITE                 VERSION            STATUS       GROESSE
prod                 2.3.0p23.cre       laeuft       4.7 GiB
test                 2.3.0p23.cre       gestoppt     318.2 MiB
```

(`GROESSE` = size, `laeuft` = running, `gestoppt` = stopped.)

**With the interface** (select sites with the space bar):

```bash
sudo ./migrate_creator.sh --gui
```

The dialog asks in turn for: sites → options (stop the site, skip RRDs/logs) →
target directory → label → optional transfer via `scp`.

**With parameters:**

```bash
# one site, stopping it during the backup (most consistent variant)
sudo ./migrate_creator.sh --site prod --stop --out /var/tmp

# all sites, without logs, transferred straight to the target system
sudo ./migrate_creator.sh --all --no-logs \
     --scp root@10.0.0.5:/opt/checkmk-stack/backups/
```

Result:

```
/var/tmp/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz
```

Besides the omd backup, the bundle contains a `MANIFEST` with the site name,
Checkmk version, source hostname, **source IP addresses**, source operating
system and SHA256 checksum, plus a `siteinfo/` directory with `omd config
show`, `omd status` and the IP configuration of the old system for later
reference.

### 17.4 Step 3 – bring the target system to the right version

`omd restore` can only restore a site **if exactly its Checkmk version is
installed on the target system.** In the container, the version comes from the
image – so the image tag has to match:

```bash
tar -xzOf backups/cmkmigrate_prod_*.tar.gz cmkbundle/MANIFEST | grep CMK_VERSION
# CMK_VERSION=2.3.0p23.cre   ->  version 2.3.0p23, suffix .cre = Raw edition

# fetch the matching image and have it written to CMK_IMAGE
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p23
sudo ./scripts/cmk-manage.sh up
```

For an Enterprise source (`.cee`) accordingly:

```bash
sudo ./scripts/cmk-manage.sh image pull --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>
# or, if the registry is unreachable:
sudo ./scripts/cmk-manage.sh image download --edition enterprise --version 2.3.0p23 \
     --user <user> --password <password>
```

The import script checks this itself and aborts with a corresponding message if
the version is missing.

> **Mind the edition:** the version suffix reveals it – `.cre` = Raw,
> `.cee` = Enterprise, `.cce` = Cloud, `.cme` = Managed. The import script
> reads it from the manifest and, on failure, names the matching `image pull`
> command directly. Enterprise images are **not** on Docker Hub – see
> [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co).

### 17.5 Step 4 – import the bundle

With the interface:

```bash
sudo ./scripts/cmk-manage.sh gui      # -> "import"
```

With parameters:

```bash
sudo ./scripts/cmk-manage.sh import \
     --file backups/cmkmigrate_prod_2.3.0p23.cre_20260820-101500.tar.gz \
     --set-default-site
```

The script

1. recognises the migration bundle and shows the source host and source IP,
2. **warns if this host's IP differs from the source IP** (the agents would
   then have to be adjusted),
3. verifies checksum, version and edition,
4. prompts if the site already exists (`--reuse` overwrites),
5. runs `omd restore` under the **unchanged site name**,
6. optionally sets `CMK_SITE_ID` in `.env` (`--set-default-site`) so the
   container starts the site automatically after a reboot as well,
7. starts the site and prints the next steps.

### 17.6 Step 5 – switch over and verify

```bash
# 1) shut down the old server or stop its site (no double monitoring!)
ssh root@old-server 'omd stop prod'

# 2) move the target system to the IP/DNS name of the old system
#    (take over the static IP or repoint the DNS record)

# 3) recreate the containers if CMK_SITE_ID was changed
sudo ./scripts/cmk-manage.sh up
sudo ./scripts/cmk-manage.sh status

# 4) check inside Checkmk
sudo ./scripts/cmk-manage.sh omd su prod
   OMD[prod]:~$ cmk -O                # activate the configuration
   OMD[prod]:~$ omd status
   OMD[prod]:~$ exit
```

Then check in the interface:

* **Monitor → All hosts** – are the agents reporting again?
* With TLS agents: the service *"Check_MK Agent"* must not report registration
  errors.
* An agent can be tested from the monitored host with `cmk-agent-ctl status` –
  the site `prod` must appear there with the target server and
  "Connection: TLS".

### 17.7 Migration checklist

- [ ] `migrate_creator.sh` copied to the source server
- [ ] bundle created (preferably with `--stop`)
- [ ] bundle copied to the target system (`backups/`)
- [ ] matching image fetched (`image pull`/`image download`), `CMK_IMAGE` set, stack recreated
- [ ] `import --set-default-site` executed, site running
- [ ] old site stopped / old server shut down
- [ ] IP or DNS switched to the target system
- [ ] ports 80/443/8000 open in the firewall
- [ ] proxy host set up in the NGINX Proxy Manager with a certificate
- [ ] agents reporting, `cmk -O` executed
- [ ] backup of the new system set up (cron)

---

## 18. Several sites in one container

At startup the Checkmk entrypoint only creates – and only starts – the site
from `CMK_SITE_ID`. Further sites (e.g. several imported ones) exist in the
bind mount but are not brought up automatically after a restart.

Hence:

```bash
sudo ./scripts/cmk-manage.sh start-all-sites
```

This command is executed automatically by the systemd unit (`ExecStartPost`)
or the Quadlet unit after the container starts, and starts every site present.

> **Recommendation:** for cleanly separated environments, run one project
> directory with its own `.env` per site. Note that because both containers run
> on the host network, only **one** Checkmk container per host can occupy ports
> 5000/8000 – for genuinely multiple instances use separate hosts or a
> bridge-network variant.

---

## 19. Upgrading Checkmk

Because `/omd/sites` is a bind mount, the data survives replacing the image.
The Checkmk software itself lives in the image under `/omd/versions`.

```bash
# 1) ALWAYS back up first
sudo ./scripts/cmk-manage.sh backup --all --label vor-update

# 2) fetch the new image (writes CMK_IMAGE into .env automatically)
sudo ./scripts/cmk-manage.sh image pull --edition raw --version 2.3.0p28

# 3) recreate the containers
sudo ./scripts/cmk-manage.sh up

# 4) update the site (interactive, confirm conflicts)
sudo ./scripts/cmk-manage.sh omd stop cmk
sudo ./scripts/cmk-manage.sh omd update cmk
sudo ./scripts/cmk-manage.sh omd start cmk
```

For a major version jump (e.g. 2.2 → 2.3) be sure to read the Checkmk werks /
release notes beforehand.

---

## 20. Troubleshooting

| Symptom | Cause / solution |
|---|---|
| `Port 80/443 already in use` | a web server is already running on the host: `ss -tlnp \| grep ':80'` → `systemctl disable --now apache2 nginx httpd` |
| Containers do not run after a reboot (Podman) | systemd unit missing: `./install.sh --with-systemd` or `--quadlet`, then `systemctl is-enabled checkmk-stack` |
| `omd restore`: *version is not installed* | fetch the matching image: `cmk-manage.sh image pull --edition <ed> --version <v>`, then `up` |
| Import aborts with an edition warning | an Enterprise backup needs an Enterprise image, see [chapter 8](#8-obtaining-the-checkmk-image-raw-enterprise--co) |
| `Login Succeeded` but the pull fails with `unable to copy from source` | The repository path or the tag is wrong. `cmk-manage.sh image repos --edition enterprise` shows what the registry offers; otherwise use the tarball |
| `image not known` / Podman searches `registry.access.redhat.com` | short image name. Set `CMK_IMAGE` fully qualified: `cmk-manage.sh image use docker.io/…` or `registry.checkmk.com/enterprise/…` |
| `checkmk/check-mk-enterprise` not found on Docker Hub | correct – Enterprise is only on `registry.checkmk.com` or available as a tarball; from 2.5 the edition is called `pro` and is public on Docker Hub |
| Tarball installed with `podman import`, image "unknown" | `import` produces an image without metadata. Remove the broken image with `podman rmi <id>` and use `cmk-manage.sh image load --file …` (which uses `load`) |
| Download of the image tarball fails (401/404) | check the customer portal credentials, give the exact version (no `-latest`), if necessary `image download --url '<full URL>'` |
| Proxy returns `502 Bad Gateway` | the Checkmk container is not running, or the forwarding does not point to `127.0.0.1:5000`: `./scripts/cmk-manage.sh status`, `curl -I http://127.0.0.1:5000/` |
| Let's Encrypt fails | port 80 not reachable from the internet, or wrong DNS; alternative: DNS challenge or a self-signed certificate |
| Agents do not report after the migration | site name changed (`--as-site`), IP not taken over, or port 8000 blocked; check on the agent: `cmk-agent-ctl status` |
| Permission denied on `data/` under RHEL | SELinux: `restorecon -Rv /opt/checkmk-stack/data`, check `:Z` on the mounts |
| `podman-compose` not found | enable EPEL or switch to Quadlet (`./install.sh --quadlet`) |
| `This system is not registered with an entitlement server` | RHEL without a subscription – `dnf` then supplies no packages at all. Register, mount the installation DVD as a repository, or switch to Rocky/Alma: [section 5.1](#51-unregistered-rhel-where-the-packages-come-from) |
| `epel-release` cannot be installed | EPEL depends on Red Hat's base repositories and is no substitute for registering. Without a subscription, the Quadlet route without `podman-compose` gets you there: [chapter 6, option B](#option-b--podman-quadlet-podman-44-and-newer-no-compose) |
| Proxy Manager container no longer starts | the switch is `off`: `cmk-manage.sh npm-enable on`. Check with `cmk-manage.sh npm-enable` |
| Admin interface unreachable from the LAN | port 81 blocked. Move it to a free port: `cmk-manage.sh npm-port 10443` and open that in the firewall |
| New port does not answer after `npm-port` | the Proxy Manager was not restarted: `cmk-manage.sh restart`; check the file `data/npm/data/nginx/custom/http.conf` |
| `cmk-manage.sh` aborts with "CMK_RUNTIME … ist nicht installiert" | it now only warns and detects the runtime itself; set it cleanly with `config set CMK_RUNTIME auto` |
| Browser stays black, "Waiting for Firefox to connect…" | The start page does not answer – usually the Proxy Manager is not running. The script now reports this beforehand, see [chapter 11.3](#113-blank-screen-the-start-page-does-not-answer) |
| Proxy Manager interface blank / only text fragments | a text browser without JavaScript was used. Use `cmk-manage.sh browser --engine browsh` (or `carbonyl`) |
| Self-signed certificate cannot be uploaded in the terminal browser | there are no file dialogs there – upload via the API, see [chapter 11.4](#114-limitation-file-upload) |
| `browser` reports "braucht ein echtes Terminal" (needs a real terminal) | called from a pipe/script. Call it on the console or over SSH; only the build runs headless, with `--build-only` |
| A menu entry drops out to the shell with an error | fixed: missing containers, sites or backups are now reported in a dialog and the interface stays open |
| `job auto on` reports missing permissions | `/etc/cron.d/` needs root: call it with `sudo`, or add the printed lines yourself |
| The Proxy Manager asks for a new account on first use | normal since 2.15. Set `NPM_ADMIN_EMAIL`/`NPM_ADMIN_PASSWORD` **before** the first start; if the account already exists, enter its credentials there |
| A JSON error appears for proxy hosts in the terminal browser | the interface is too heavy for it. Create proxy hosts with `cmk-manage.sh proxy add` instead ([chapter 10.2](#102-creating-a-proxy-host--without-a-browser)) |
| The TUI does not start | install `dialog`: `apt-get install -y dialog` or `dnf install -y dialog` |
| Site does not start, `tmp` errors | `./scripts/cmk-manage.sh omd config <site> set TMPFS off`, then start the site |

Useful commands:

```bash
./scripts/cmk-manage.sh logs -f                  # container logs
./scripts/cmk-manage.sh omd status cmk           # services of the site
./scripts/cmk-manage.sh shell                    # root shell in the container
docker exec -it checkmk omd su cmk               # or podman exec …
```

---

## 21. Security notes

* Change `CMK_PASSWORD` in the interface right after installation and remove it
  from `.env` (it is only evaluated when the site is created); `chmod 600 .env`
  is already set.
* Set `NPM_ADMIN_PASSWORD` to your own password before the first start – the
  Proxy Manager's administrator account is created from it.
* Make port **81** (admin interface) reachable from the administration network
  only, and never expose port **5000**.
* Backups contain **all** passwords and certificates of the site – store them
  encrypted or restrict access tightly (`chmod 700 backups`).
* Schedule `./scripts/cmk-manage.sh pull` + `up` regularly for image security
  updates.
* Host networking means the container shares the host's network stack. There is
  no additional network isolation – protection comes from the host firewall.

---

## 22. Uninstalling

```bash
sudo ./scripts/cmk-manage.sh down

# systemd
sudo systemctl disable --now checkmk-stack.service
sudo rm -f /etc/systemd/system/checkmk-stack.service
# or Quadlet
sudo rm -f /etc/containers/systemd/checkmk.container \
           /etc/containers/systemd/nginx-proxy-manager.container
sudo systemctl daemon-reload

# remove the images (names from "cmk-manage.sh image list")
docker image rm docker.io/checkmk/check-mk-raw:2.3.0-latest docker.io/jc21/nginx-proxy-manager:2
# podman image rm …

# remove a stored registry login if present
podman logout registry.checkmk.com   # or docker logout

# data (CAUTION: this deletes the entire monitoring setup!)
sudo rm -rf /opt/checkmk-stack/data
```

---

## Licence / notes

This project contains only configuration and helper scripts.
Checkmk (Raw Edition, GPLv2) and NGINX Proxy Manager (MIT) are subject to the
licence terms of their respective vendors.