Back to blog
January 2026 5 min read

Wiring Up Observability: Prometheus, Grafana, and Every Exporter In Between

A field-tested rundown of building a monitoring stack from scratch — Prometheus and Grafana via Docker, plus exporters for MySQL, PHP-FPM, VMware, Blackbox probing, cAdvisor, Pinpoint APM, and node-level metrics.

Prometheus Grafana Monitoring Observability DevOps Docker

A monitoring stack is only as good as its exporters — Prometheus and Grafana are the easy part. The real work is getting every layer of your infrastructure (databases, PHP pools, hypervisors, containers, external endpoints) to actually expose metrics in a format Prometheus can scrape.

Core Stack: Prometheus + Grafana

Prometheus

docker run -d \
  -p 9090:9090 \
  --name prometheus \
  -v /root/workdir/prometheus:/etc/prometheus \
  prom/prometheus

Mount your prometheus.yml scrape config into /etc/prometheus so Prometheus knows which exporters to poll and how often.

Grafana

docker run -d \
  -p 3000:3000 \
  --name=grafana \
  --user "$(id -u)" \
  -e "GF_FEATURE_TOGGLES_ENABLE=publicDashboards" \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel" \
  --volume "$PWD/grafana:/var/lib/grafana" \
  grafana/grafana-enterprise

--user "$(id -u)" avoids permission issues on the mounted volume by running Grafana as your host user instead of the container's default. GF_INSTALL_PLUGINS accepts a comma-separated list if you need more than one plugin — just don't repeat the env var multiple times, only the last one takes effect.

MySQL Exporter

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.15.0/mysqld_exporter-0.15.0.linux-amd64.tar.gz
tar xvfz mysqld_exporter-0.15.0.linux-amd64.tar.gz
cd mysqld_exporter-0.15.0.linux-amd64

Create a dedicated read-only MySQL user scoped just for metrics collection:

CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<strong-password>' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

Point the exporter at it:

export DATA_SOURCE_NAME='exporter:<strong-password>@(<mysql-host>:3306)/'

Move the binary somewhere on $PATH and run it as a systemd service:

# /etc/systemd/system/mysqld_exporter.service
[Unit]
Description=mysqld_exporter
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/mysqld_exporter --config.my-cnf /root/.my.cnf
Restart=on-failure

[Install]
WantedBy=multi-user.target

Reference: Grafana's MySQL exporter installation docs

Useful Query: Slow Queries Rate

sum(rate(mysql_global_status_slow_queries{job=~"$job", instance=~"$instance"}[$__interval]))

Scoped to a specific instance:

sum(rate(mysql_global_status_slow_queries{instance="<mysql-host>:9104", job="mysql-exporter"}[1m]))

PHP-FPM Exporter

Using hipages/php-fpm_exporter:

yum install php71-php-fpm
sudo systemctl start php71-php-fpm
sudo systemctl enable php71-php-fpm

Enable the FPM status endpoint in the pool config:

; /etc/opt/remi/php71/php-fpm.d/www.conf
[www]
...
pm.status_path = /status
...

Uncomment the syslog identifier in the main FPM config:

; /etc/opt/remi/php71/php-fpm.conf
syslog.ident = php-fpm

Run as a systemd service:

# /etc/systemd/system/php-fpm-exporter.service
[Unit]
Description=PHP-FPM Exporter

[Service]
ExecStart=/usr/local/bin/php-fpm_exporter server --phpfpm.scrape-uri=tcp://127.0.0.1:9000/status
Restart=always

[Install]
WantedBy=multi-user.target

Or run it in Docker instead, scraping multiple pools at once:

docker run -d --rm \
  -e PHP_FPM_SCRAPE_URI="tcp://127.0.0.1:9000/status,tcp://127.0.0.1:9001/status" \
  -p 9253:9253 \
  --name=php_exporter \
  hipages/php-fpm_exporter

VMware Exporter

For hypervisor-level metrics from vSphere:

docker run -d --rm \
  -p 9272:9272 \
  -e VSPHERE_USER=root \
  -e VSPHERE_PASSWORD=<vsphere-password> \
  -e VSPHERE_HOST=<vsphere-host> \
  -e VSPHERE_IGNORE_SSL=True \
  -e VSPHERE_SPECS_SIZE=2000 \
  --name vmware_exporter \
  pryorda/vmware_exporter

VSPHERE_IGNORE_SSL=True is convenient for internal/self-signed vCenter certs, but skip it if your vCenter has a properly trusted cert chain.

Blackbox Exporter (Endpoint / Uptime Probing)

Download the release binary from prometheus.io/downloads, extract it, and wire it into systemd with a probe config:

# /etc/systemd/system/blackbox_exporter.service
[Unit]
Description=blackbox_exporter
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/root/workdir/blackbox/blackbox_exporter-0.24.0.linux-amd64/blackbox.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

Blackbox is what you'd point at public endpoints for HTTP/TCP/ICMP probing — useful for tracking external uptime and TLS cert expiry separately from internal service health.

cAdvisor (Container-Level Metrics)

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8000:8080 \
  --detach=true \
  --name=cadvisor \
  gcr.io/cadvisor/cadvisor:latest

cAdvisor exposes per-container CPU, memory, filesystem, and network metrics — the read-only volume mounts give it visibility into the host's container runtime without write access.

Pinpoint APM

For distributed tracing alongside metrics, Pinpoint pairs a collector agent with language-specific instrumentation.

Collector Agent

docker run -d -p 9999:9999 \
  --name pinpoint_c_collector_agent \
  --restart always \
  --env-file ./env.list \
  ghcr.io/pinpoint-apm/pinpoint-c-agent/collector-agent:latest

PHP Instrumentation

extension=pinpoint_php.so

; Must match the Collector-Agent's PP_ADDRESS from the step above
pinpoint_php.CollectorHost=Tcp:<collector-ip>:<collector-port>
pinpoint_php.SendSpanTimeOutMs=0 ; 0 is recommended
pinpoint_php.TraceLimit=-1 ; negative = no limit on capture duration

; Debug only — also requires PHP's error_reporting/log_errors enabled
;pinpoint_php.DebugReport=true
;error_reporting = E_ALL
;log_errors = On
;error_log = /tmp/php_fpm_error.log

Add the AOP plugin via Composer for automatic instrumentation:

composer require "pinpoint-apm/pinpoint-php-aop:^0.3"

Reference: Pinpoint PHP agent readme

Node Exporter (Host-Level Metrics)

The baseline exporter for CPU, memory, disk, and network stats on any Linux host:

wget https://github.com/prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.linux-amd64.tar.gz
tar -xzvf node_exporter-1.10.2.linux-amd64.tar.gz
cd node_exporter-1.10.2.linux-amd64

cp node_exporter /usr/local/bin/node_exporter

useradd --no-create-home --shell /bin/false node_exporter
chown node_exporter:node_exporter /usr/local/bin/node_exporter

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

Running node_exporter as a dedicated, shell-less system user is worth keeping consistent across every exporter here — none of them need root, and a compromised metrics endpoint shouldn't hand over a usable shell.

Summary

| Exporter | Scrapes | Default Port | |---|---|---|

  • Prometheus | itself + all configured targets | 9090 |
  • Grafana | dashboards / visualization | 3000 |
  • mysqld_exporter | MySQL global status, slow queries | 9104 |
  • php-fpm_exporter | PHP-FPM pool status | 9253 |
  • vmware_exporter | vSphere/vCenter | 9272 |
  • blackbox_exporter | HTTP/TCP/ICMP probes | 9115 |
  • cAdvisor | per-container resource usage | 8080 |
  • node_exporter | host-level system metrics | 9100 |

Golden rule: every exporter here should run as its own unprivileged systemd user, and every credential (DB passwords, vSphere creds, collector addresses) belongs in a secrets manager or .my.cnf-style restricted file — never hardcoded into a long-running ExecStart line or committed to a repo.