Prometheus + Grafana

Monitoring rootless 2 containers
Open in Generator
prometheus docker.io/prom/prometheus:latest · 9090:9090
grafana docker.io/grafana/grafana:latest · 3000:3000

Deployment rootless

1. Prepare the host

Operating System:
!

Prerequisites once as root

Install Podman (needs podman 4.3 or newer; Ubuntu 22.04 ships 3.4 and is too old, use 24.04)

apt update && apt install -y podman

Create user (if not existing)

useradd -m -s /bin/bash monitoring
passwd monitoring

Enable linger (services keep running after logout and start after reboot)

loginctl enable-linger monitoring
↪

Switch to deployment user

su - monitoring

Use su - with the dash — this creates a proper login shell as monitoring. Without the dash, root environment variables carry over and rootless Podman will not work correctly.

!

Fix environment for rootless systemd (new users only — skip if already set)

Required once so systemctl --user works in non-login sessions.

cat << 'EOF' >> ~/.bashrc

# Fix for systemd user services (rootless Podman/systemctl)
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
EOF
source ~/.bashrc

2. Choose a format and deploy

The same pod in four forms. Pick one and follow only its steps.

Each tab says which podman it needs. Check yours with podman --version: Debian 12 ships 4.3, Ubuntu 24.04 4.9, Debian 13 5.4, Fedora 5.8, Arch 6.x.

Kubernetes YAML for podman kube play, started by systemd through a Quadlet .kube unit (podman 4.4+; older podman like Debian 12 uses podman-kube@, see the version check below).

Kubernetes YAML 

apiVersion: v1
kind: Pod
metadata:
  name: monitoring
  labels:
    app: monitoring
spec:
  restartPolicy: Always
  containers:
  - name: prometheus
    image: docker.io/prom/prometheus:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 9090
      hostPort: 9090
    volumeMounts:
    - name: vol-0
      mountPath: /etc/prometheus
    - name: vol-1
      mountPath: /prometheus
    args:
    - --config.file=/etc/prometheus/prometheus.yml
    - --storage.tsdb.path=/prometheus
    - --web.console.libraries=/usr/share/prometheus/console_libraries
    - --web.console.templates=/usr/share/prometheus/consoles
    securityContext:
      runAsUser: 0
      runAsGroup: 0
  - name: grafana
    image: docker.io/grafana/grafana:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 3000
      hostPort: 3000
    env:
    - name: GF_SECURITY_ADMIN_USER
      value: admin
    - name: GF_SECURITY_ADMIN_PASSWORD
      value: Uz8L33aoe5xTGjQCSByZ2w
    - name: GF_USERS_ALLOW_SIGN_UP
      value: 'false'
    volumeMounts:
    - name: vol-2
      mountPath: /var/lib/grafana
    securityContext:
      runAsUser: 0
      runAsGroup: 0
  volumes:
  - name: vol-0
    persistentVolumeClaim:
      claimName: monitoring-prometheus
  - name: vol-1
    persistentVolumeClaim:
      claimName: monitoring-prometheus-data
  - name: vol-2
    persistentVolumeClaim:
      claimName: monitoring-grafana
1

Save the YAML file

mkdir -p ~/.config/containers/
nano ~/.config/containers/monitoring.yaml
2

Test the pod (without autostart)

podman kube play ~/.config/containers/monitoring.yaml

# Check status:
podman pod ps && podman ps

# Stop:
podman kube down ~/.config/containers/monitoring.yaml

Check your podman version

podman --version

4.4 or newer: continue with step 3 below.

Older, e.g. 4.3 on Debian 12? There is no Quadlet yet: use podman's own podman-kube@ service instead of steps 3–5 
systemctl --user enable --now podman-kube@$(systemd-escape ~/.config/containers/monitoring.yaml).service

# Status & logs:
systemctl --user status podman-kube@$(systemd-escape ~/.config/containers/monitoring.yaml).service
journalctl --user -u podman-kube@$(systemd-escape ~/.config/containers/monitoring.yaml).service -f

The day-to-day commands below have a block for this service.

3

Create Quadlet .kube file

monitoring.kube 
[Unit]
Description=monitoring pod
After=network-online.target
Wants=network-online.target

[Kube]
Yaml=%h/.config/containers/monitoring.yaml

[Service]
Restart=always

[Install]
WantedBy=default.target

%h is a systemd home-directory specifier — do not replace it with ~/.

Save to ~/.config/containers/systemd/:

mkdir -p ~/.config/containers/systemd/
nano ~/.config/containers/systemd/monitoring.kube
4

Start systemd service

systemctl --user daemon-reload
systemctl --user start monitoring.service
5

Status & Logs

systemctl --user status monitoring.service
journalctl --user -u monitoring.service -f
podman pod ps
podman ps
6

Enable automatic image updates optional

Requires AutoUpdate=registry in the .kube file. Podman then checks for new images and restarts the pod automatically:

systemctl --user enable --now podman.socket
systemctl --user daemon-reload
systemctl --user enable --now podman-auto-update.timer

# Check status:
systemctl --user status podman-auto-update.timer

# Trigger manually:
podman auto-update
Day-to-day logs, restart, update, remove (click to open)

Logs & shell into a container

podman logs -f monitoring-prometheus
podman logs -f monitoring-grafana
podman exec -it monitoring-prometheus /bin/sh

Restart

systemctl --user restart monitoring.service

Update an image by hand

Pull the new version, the restart recreates the containers from it:

podman pull <image>:<tag>
systemctl --user restart monitoring.service

Stop & remove

Stop through systemd, otherwise Restart=always brings the pod back. To remove it for good, delete the unit files and reload:

systemctl --user stop monitoring.service
rm ~/.config/containers/systemd/monitoring.kube
systemctl --user daemon-reload

podman older than 4.4 (podman-kube@ from step 3)

systemctl --user restart podman-kube@$(systemd-escape ~/.config/containers/monitoring.yaml).service

# Update an image: pull, then restart as above
podman pull <image>:<tag>

# Stop and remove from autostart:
systemctl --user disable --now podman-kube@$(systemd-escape ~/.config/containers/monitoring.yaml).service

Delete the data — permanent, only after stop & remove

podman volume rm monitoring-prometheus
podman volume rm monitoring-prometheus-data
podman volume rm monitoring-grafana

Native systemd units without the Kubernetes YAML: one .pod plus one .container per container. Each container becomes its own systemd service (restart, logs and ordering per container). Needs podman 5.0 or newer — Debian 12 ships 4.3, use the Kube YAML tab there.

1

Save the unit files

monitoring.pod 
[Unit]
Description=monitoring pod

[Pod]
PodName=monitoring
PublishPort=9090:9090
PublishPort=3000:3000
monitoring-prometheus.container 
[Unit]
Description=monitoring-prometheus container

[Container]
Image=docker.io/prom/prometheus:latest
Pod=monitoring.pod
ContainerName=monitoring-prometheus
Pull=always
Volume=monitoring-prometheus:/etc/prometheus:Z
Volume=monitoring-prometheus-data:/prometheus:Z
User=0
Group=0
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles

[Service]
Restart=always

[Install]
WantedBy=default.target
monitoring-grafana.container 
[Unit]
Description=monitoring-grafana container

[Container]
Image=docker.io/grafana/grafana:latest
Pod=monitoring.pod
ContainerName=monitoring-grafana
Pull=always
Environment=GF_SECURITY_ADMIN_USER=admin
Environment=GF_SECURITY_ADMIN_PASSWORD=Uz8L33aoe5xTGjQCSByZ2w
Environment=GF_USERS_ALLOW_SIGN_UP=false
Volume=monitoring-grafana:/var/lib/grafana:Z
User=0
Group=0

[Service]
Restart=always

[Install]
WantedBy=default.target

Save all of them to ~/.config/containers/systemd/:

mkdir -p ~/.config/containers/systemd/
nano ~/.config/containers/systemd/monitoring.pod
nano ~/.config/containers/systemd/monitoring-prometheus.container
nano ~/.config/containers/systemd/monitoring-grafana.container
2

Start the pod

The pod service pulls in every container service. Autostart at boot comes from the [Install] section of the units, no enable needed.

systemctl --user daemon-reload
systemctl --user start monitoring-pod.service
3

Status & Logs

systemctl --user status monitoring-pod.service
journalctl --user -u monitoring-prometheus.service -f
journalctl --user -u monitoring-grafana.service -f
podman pod ps && podman ps
Day-to-day logs, restart, update, remove (click to open)

Logs & shell into a container

podman logs -f monitoring-prometheus
podman logs -f monitoring-grafana
podman exec -it monitoring-prometheus /bin/sh

Restart

systemctl --user restart monitoring-pod.service

Update an image by hand

Pull the new version, the restart recreates the containers from it:

podman pull <image>:<tag>
systemctl --user restart monitoring-pod.service

Stop & remove

Stop through systemd, otherwise Restart=always brings the pod back. To remove it for good, delete the unit files and reload:

systemctl --user stop monitoring-pod.service
rm ~/.config/containers/systemd/monitoring.pod
rm ~/.config/containers/systemd/monitoring-prometheus.container
rm ~/.config/containers/systemd/monitoring-grafana.container
systemctl --user daemon-reload

Delete the data — permanent, only after stop & remove

podman volume rm monitoring-prometheus
podman volume rm monitoring-prometheus-data
podman volume rm monitoring-grafana

A docker-compose.yml for podman-compose. Handy if you already work with Compose; there are no per-container systemd services.

1

Install podman-compose as root

Run this as root, e.g. together with the prerequisites before su - monitoring.

apt install -y podman-compose
2

Save the compose file

docker-compose.yml 
services:
  prometheus:
    image: docker.io/prom/prometheus:latest
    pull_policy: always
    restart: always
    ports:
    - 9090:9090
    - 3000:3000
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    - --storage.tsdb.path=/prometheus
    - --web.console.libraries=/usr/share/prometheus/console_libraries
    - --web.console.templates=/usr/share/prometheus/consoles
    user: 0:0
    volumes:
    - monitoring-prometheus:/etc/prometheus:Z
    - monitoring-prometheus-data:/prometheus:Z
  grafana:
    image: docker.io/grafana/grafana:latest
    pull_policy: always
    restart: always
    environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=Uz8L33aoe5xTGjQCSByZ2w
    - GF_USERS_ALLOW_SIGN_UP=false
    user: 0:0
    volumes:
    - monitoring-grafana:/var/lib/grafana:Z
    network_mode: service:prometheus
    depends_on:
      prometheus:
        condition: service_started
volumes:
  monitoring-prometheus: null
  monitoring-prometheus-data: null
  monitoring-grafana: null

mkdir -p ~/monitoring
nano ~/monitoring/docker-compose.yml
3

Start, status & logs

cd ~/monitoring
podman-compose up -d
podman-compose ps
podman-compose logs -f

# Stop:
podman-compose down
4

Start after reboot

podman-restart.service starts every container with restart: always at boot:

systemctl --user enable podman-restart.service
5

Update images

cd ~/monitoring
podman-compose pull
podman-compose up -d
Day-to-day logs, restart, update, remove (click to open)

Logs & shell into a container

podman logs -f monitoring_prometheus_1
podman logs -f monitoring_grafana_1
podman exec -it monitoring_prometheus_1 /bin/sh

Restart

cd ~/monitoring
podman-compose restart

Stop & remove

cd ~/monitoring
podman-compose down

Delete the data — permanent, only after stop & remove

podman volume rm monitoring_monitoring-prometheus
podman volume rm monitoring_monitoring-prometheus-data
podman volume rm monitoring_monitoring-grafana

Plain podman pod create / podman create commands. Good for a quick test or to see what happens under the hood.

1

Save the script

monitoring.sh 
#!/bin/bash
set -e

# ── Create named volumes ───────────────────────────────────────────
# Named volumes are created automatically — no UID/GID permission adjustment needed.
podman volume exists monitoring-prometheus || podman volume create monitoring-prometheus
podman volume exists monitoring-prometheus-data || podman volume create monitoring-prometheus-data
podman volume exists monitoring-grafana || podman volume create monitoring-grafana

# ── Create pod ─────────────────────────────────────────────────────
podman pod create \
  --name monitoring \
  -p 9090:9090 \
  -p 3000:3000

# ── Container: monitoring-prometheus ─────────────────────────────────────────────
podman create \
  --pod monitoring \
  --name monitoring-prometheus \
  --restart always \
  --pull always \
  -v monitoring-prometheus:/etc/prometheus:Z \
  -v monitoring-prometheus-data:/prometheus:Z \
  --user 0:0 \
  docker.io/prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles

# ── Container: monitoring-grafana ─────────────────────────────────────────────
podman create \
  --pod monitoring \
  --name monitoring-grafana \
  --restart always \
  --pull always \
  -e GF_SECURITY_ADMIN_USER='admin' \
  -e GF_SECURITY_ADMIN_PASSWORD='Uz8L33aoe5xTGjQCSByZ2w' \
  -e GF_USERS_ALLOW_SIGN_UP='false' \
  -v monitoring-grafana:/var/lib/grafana:Z \
  --user 0:0 \
  docker.io/grafana/grafana:latest

# ── Start pod ──────────────────────────────────────────────────────
podman pod start monitoring

# ── Stop & clean up ───────────────────────────────────────────────
# podman pod stop monitoring
# podman pod rm monitoring
nano monitoring.sh
2

Run it, status & logs

bash monitoring.sh
podman pod ps && podman ps
podman logs -f <container-name>
3

Start after reboot

podman-restart.service starts every container with --restart always at boot:

systemctl --user enable podman-restart.service
4

Remove or rebuild

The script creates the pod once. To apply changes or new images, remove the pod and run the script again (named volumes stay):

podman pod rm -f monitoring
bash monitoring.sh
Day-to-day logs, restart, update, remove (click to open)

Logs & shell into a container

podman logs -f monitoring-prometheus
podman logs -f monitoring-grafana
podman exec -it monitoring-prometheus /bin/sh

Restart

podman pod stop monitoring
podman pod start monitoring

Stop & remove

podman pod rm -f monitoring

Delete the data — permanent, only after stop & remove

podman volume rm monitoring-prometheus
podman volume rm monitoring-prometheus-data
podman volume rm monitoring-grafana
Useful Tips

Ports < 1024 (e.g. 80, 443)

Rootless cannot open privileged ports. Solution:

sysctl net.ipv4.ip_unprivileged_port_start=80

Make persistent in /etc/sysctl.d/99-podman.conf.

Containers communicate via localhost

All containers in the pod share the same network namespace. Always use localhost, not container names.

# Correct (e.g. app → db):
localhost:5432

# Wrong (doesn't work in a pod):
db-container:5432

List open ports

Which ports are the running containers publishing? See the PORTS column:

podman ps

Custom DNS for the pod

Set a custom DNS server (e.g. local Pi-hole) in the generator under DNS server. Every format picks it up.

Set volume ownership

Fix permission errors by adjusting UID/GID in the user namespace:

podman unshare chown 1000:1000 /path/to/volume

SELinux volume labels

On SELinux systems (RHEL, Fedora) set the volume suffix:

/host/path:/container/path:Z   # private
/host/path:/container/path:z   # shared

List all volumes

podman volume ls
podman volume inspect <volume-name>

Volume backup

Back up data from a named volume:

podman run --rm \
  -v <volume-name>:/data:ro \
  -v $(pwd):/backup \
  busybox tar czf /backup/backup.tar.gz /data

Cleanup

Remove unused images, containers and volumes:

podman system prune -f        # containers + images
podman image prune -f         # untagged images only
podman volume prune -f        # unused volumes

Check for outdated images

List local images; auto-update only covers units with AutoUpdate= (Kube and Quadlet tabs):

podman images
podman auto-update --dry-run

All containers, including stopped ones

podman pod ps
podman ps -a

Resource usage

podman stats --no-stream
Customize this stack
Change ports, volumes, environment variables and more in the generator.
Open in Generator