SKYLINE Knowledge Base

Linux Basics

Linux Server Monitoring with Prometheus, node_exporter and Alertmanager on Debian and Ubuntu

Set up Prometheus, node_exporter and Alertmanager from Debian/Ubuntu packages, keep them off the internet, and alert on down hosts, full disks and low memory.

Published
Reading time
7 min read

A usage chart tells you what happened; an alert tells you while there is still time to act. Prometheus, its node_exporter and Alertmanager are a common open-source stack for this on Linux servers, and all three are packaged in Debian and Ubuntu. This guide sets them up on one server, keeps their web interfaces off the internet, and adds three alerts most servers need: host down, disk filling up and memory running low.

The three pieces

  • node_exporter exposes machine metrics (CPU, memory, disks, network) over HTTP on port 9100. Its metric names start with node_, for example node_filesystem_avail_bytes, the space available to non-root users (node exporter guide).
  • Prometheus scrapes those metrics on a schedule, stores them (15 days by default, according to the --storage.tsdb.retention.time description in prometheus(1)), evaluates alert rules and serves a web UI on port 9090.
  • Alertmanager receives firing alerts from Prometheus, groups them and sends notifications such as email. It listens on port 9093.

An alert therefore travels in four steps: node_exporter reports a value, Prometheus stores it at every scrape, a rule compares it with a threshold at every evaluation, and once the condition has held for the configured time, Alertmanager decides who is told and how often. Each step can fail on its own, which is why the last section of this guide is about testing the whole chain rather than any single part.

Install from the distribution

sudo apt update
sudo apt install prometheus prometheus-node-exporter prometheus-alertmanager

Debian 13 (trixie) packages Prometheus 2.53.3, node_exporter 1.9.0 and Alertmanager 0.28.1; Debian 12 (bookworm) has 2.42.0, 1.5.0 and 0.25.0 (package tracker). The online documentation follows the latest upstream release, so check a setting against your version before relying on it. The configuration below sticks to long-standing basics.

The packages run each service as the prometheus user and read extra command-line options from ARGS in /etc/default/prometheus, /etc/default/prometheus-node-exporter and /etc/default/prometheus-alertmanager. The main configuration files are /etc/prometheus/prometheus.yml and /etc/prometheus/alertmanager.yml.

Keep the web interfaces private

Prometheus's security model presumes that anyone who can reach its HTTP endpoint can read all stored time series. For Alertmanager it warns that anyone with access to the endpoint can create and resolve alerts and create or delete silences. None of these ports should be open to the internet. On a single server, bind them to loopback:

# /etc/default/prometheus-node-exporter
ARGS="--web.listen-address=127.0.0.1:9100"

# /etc/default/prometheus
ARGS="--web.listen-address=127.0.0.1:9090"

# /etc/default/prometheus-alertmanager
ARGS="--web.listen-address=127.0.0.1:9093"

sudo systemctl restart prometheus-node-exporter prometheus prometheus-alertmanager
sudo ss -tlnp | grep -E ':(9090|9093|9100)'

Reach the UI through an SSH tunnel: ssh -L 9090:127.0.0.1:9090 user@server, then open http://localhost:9090. If Prometheus scrapes other servers, let node_exporter listen on a private address and allow port 9100 only from the Prometheus server. Prometheus and its exporters support TLS and basic authentication through a web configuration file (--web.config.file); the documentation notes that basic authentication without TLS sends credentials in clear text.

Scrape and alert configuration

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['127.0.0.1:9093']

rule_files:
  - /etc/prometheus/node-alerts.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['127.0.0.1:9090']
  - job_name: node
    static_configs:
      - targets: ['127.0.0.1:9100']

Three alerts worth having on every server

The rule format follows the alerting rules documentation; for makes a condition hold for a while before the alert fires, which avoids noise from short spikes.

# /etc/prometheus/node-alerts.yml
groups:
  - name: node
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} has been unreachable for 5 minutes"

      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes < 0.10
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Less than 10% free on {{ $labels.mountpoint }} ({{ $labels.instance }})"

      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
        for: 30m
        labels:
          severity: warning

      - alert: MemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels:
          severity: warning

predict_linear extrapolates the last six hours of free space; it catches a runaway log before the disk is actually full. Check the files and reload:

promtool check rules /etc/prometheus/node-alerts.yml
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Sending the alerts by email

# /etc/prometheus/alertmanager.yml
global:
  smtp_smarthost: 'mail.example.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: 'REPLACE-ME'

route:
  receiver: ops-email
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: ops-email
    email_configs:
      - to: 'ops@example.com'
        send_resolved: true

The field names come from the Alertmanager configuration reference. The file holds a password, so keep it readable only by root and the prometheus group. Restart with sudo systemctl restart prometheus-alertmanager and check journalctl -u prometheus-alertmanager for errors.

Common pitfalls

  • Monitoring a server only from itself: if the server goes down, so does the alert. Run Prometheus on a separate machine, or add an outside uptime check that does not depend on the server.
  • Ports 9090, 9093 and 9100 left open to the internet, exposing system details and alert controls.
  • Alerts that fire constantly. Tune thresholds and for durations until every alert means "act now".
  • Never testing delivery. Stop node_exporter for five minutes in a quiet hour and confirm the email arrives.

What you should do

  • Install the three packages, bind every web interface to loopback or a private address, and confirm with ss -tlnp.
  • Add the down, disk and memory rules, validate them with promtool, and test notification delivery end to end.
  • Watch Prometheus's own disk use and choose a retention period that fits.
  • Keep the packages updated with the rest of the system, and plan a second vantage point so a dead server still triggers an alert.

Sources

Topics

SKYLINE Engineering

@skyline

The engineering team at SKYLINE Industrial Solutions. We publish field-tested guides drawn from real KSA and GCC deployments.

See author profile
More in Linux Basics

Related: Skyline Nexus ERP guides

Nexus knowledge base →
Skyline Nexus ERP

Gulf & Middle East tax and e-invoicing guides

VAT, ZATCA e-invoicing and corporate tax explained for Saudi Arabia, the UAE and the wider GCC.

 Skyline Nexus ERP

Skyline Nexus ERP — cloud accounting & ZATCA e-invoicing

Our cloud ERP: accounting, invoicing, inventory and POS in one place. First month free.
SKYLINE engineering services

Need this implemented for you?

Reading is free — building it right takes a team. SKYLINE engineers ship Linux Basics for Aramco vendors, banks, hospitals and government agencies across Saudi Arabia. Talk to us before you start.

Server Infrastructure Dell, HPE, Huawei, VMware, Hyper-V, HCI IT AMC Annual maintenance, 24/7 SLA, Bronze→Platinum
Talk to a SKYLINE engineer Call +966 50 993 9334
Aramco Approved Contractor ISO 9001 · ISO 27001 SAMA CSF aligned NCA ECC ready 247+ KSA clients

Comments

0 total · 0 threads
Be the first to leave a comment.