Prometheus Cheat Sheet
There are some amazing resources available to learn about Prometheus. This post is a cheat sheet with examples that serve as a quick reference and record of interesting things.
PromQL Queries
Use API
Use the query endpoint,
curl -s 'http://prom.server:9090/api/v1/query?query=rate(node_cpu_seconds_total%5B1m%5D)' | jq .
Use the query_range endpoint,
curl -s -X POST -F 'query=rate(node_cpu_seconds_total[1m])' -F 'start=2023-02-22T04:33:46.167Z' -F'end=2023-02-22T04:53:46.167Z' -F 'step=15s' 'http://prom.server:9090/api/v1/query_range' | jq .
Note how in query you need to use HTML URL encoding for some things but not so in query_range.
CPU Usage
CPU usage for all kinds or modes,
node_cpu_seconds_total
Use the rate function to calculate the CPU usage per mode over time,
rate(node_cpu_seconds_total[1m])
Calculate the total percentage of CPU usage per core per instance by summing up rate of all modes except the idle mode,
(sum by (instance, cpu) (rate(node_cpu_seconds_total{mode!="idle"}[30s])))*100
The above can be modified to sum by (instance) but that will give a combined
total for all cores which can be more than 100%. An
alternative
is below,
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[30s])) * 100)
Memory Usage
Query for total memory use,
((node_memory_MemFree_bytes / node_memory_MemTotal_bytes ) * 100)