Monitoring with Prometheus

Prometheus collects data from metrics endpoints.

See Prometheus getting started

One easy way is to run Prometheus in Docker

docker run -d -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
[$[Get Code]]

Query Data

Once you connected targets you can query them as well. The following examples use the server load as exported by the node exporter (see below)

Just get for each configured server one value (at one point in time). This is called in Prometheus a "Instant vector"

node_load1
[$[Get Code]]

You can also get the value as it was some time ago

node_load1 offset 1h
[$[Get Code]]

Filter on servers that have "rabbit" in the name (~ activates regexp).

node_load1{instance=~".*rabbit.*"}
[$[Get Code]]

Can also have multiple conditions (contains rabbit but not rabbit 02)

node_load1{instance=~".*rabbit.*", instance!~".*rabbit.*-02.*"}
[$[Get Code]]

Aggregate values

COUNT(node_load1)
avg(node_load1{instance=~".*rabbit.*"})
MAX(node_load1{instance=~".*rabbit.*"})
[$[Get Code]]

Highest (lowest) 5 values per moment in time

topk(4, node_load1)
bottomk(4, node_load1)
[$[Get Code]]
Do math WITH VALUES
node_load1-(node_load1 offset 60m)
[$[Get Code]]

Instead of getting only one value per target per moment in time, you can also get all the values within a time range. For example all values in the last 5 minutes. This is called in Prometheus a "Range vector".

node_load1[5m] offset 4h
[$[Get Code]]

You can also get 5 minutes from 4 hours ago

node_load1[5m] offset 4h
[$[Get Code]]

Compare values to the average values

node_load5{instance=~".*rabbit.*"} - ignoring(job, instance) group_left avg WITHOUT (job, instance)(node_load5{instance=~".*rabbit.*"})
[$[Get Code]]

Targets

Each target is a metrics endpoint that Prometheus can collect data from. Once configured in Prometheus they should be listed in your Prometheus server http://my-prometheus-server.example.com:9090/targets

Random

git clone https://github.com/prometheus/client_golang.git
cd client_golang/examples/random
go get -d
go build

# Start 3 example targets in separate terminals:
./random -listen-address=:8080
./random -listen-address=:8081
./random -listen-address=:8082
[$[Get Code]]

Node exporter

The Prometheus node exporter exports from the current computer some general stats, like for example CPU load.

apt-get install prometheus-node-exporter
[$[Get Code]]

Test if with

curl http://localhost:9100/metrics
curl https://localhost:9100/metrics
[$[Get Code]]

MongoDB

docker run -p 9104:9104 --name mongo_exporter -d eses/mongodb_exporter -mongodb.uri mongodb://my-mongo-server.example.com:27017
[$[Get Code]]

Metrics endpoint is per default at port 9104

Aerospike

docker run -p 9145:9145 -d --name aerospike_prometheus_exporter ysde/docker-aerospike-exporter --node my-aerospike-server.example.com:3000
[$[Get Code]]

HAProxy

Recent versions of haproxy come with a metrics endoint. Otherwise there is also a Debian package for an exporter

apt-get install prometheus-haproxy-exporter
[$[Get Code]]

Or you run it in Docker

docker run -p 9101:9101 -d --name prometheus-haproxy-exporter quay.io/prometheus/haproxy-exporter --haproxy.scrape-uri="http://user:secret@myserver.example.com:80/stats;csv"
[$[Get Code]]

Docker

Docker can expose its stats to Prometheus

Alerting

prometheus.yml

global:
  scrape_interval:     30s

rule_files:
  - 'alerts/*.yml'

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "my-alerting-server.example.com:9093"
[$[Get Code]]

Example for an alert file

groups:
  - name: alerting_rules_node_exporter
    rules:
      - alert: OutOfMemory
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Out of memory (instance {{ $labels.instance }})"
          description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
- alert: HAProxyExporterBadHttpCodes
        expr:  sum(rate(haproxy_backend_http_responses_total{code!="2xx"}[5m])) BY (backend) * 100 / ( sum(rate(haproxy_backend_http_responses_total[5m])) BY (backend) + 1) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "HAProxy too many bad http code (instance {{ $labels.instance }})"
          description: "HAProxy bad http code rate is more than 5%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
[$[Get Code]]

Dashboards with Grafana

docker volume create grafana-storage
docker run -d -p 3000:3000 --name=grafana -v grafana-storage:/var/lib/grafana grafana/grafana
[$[Get Code]]