Remote-Pulse Prometheus Integration Runbook¶

Overview¶

This runbook describes how to integrate the Remote-Pulse server /metrics endpoint with the existing Prometheus/Grafana observability stack (VM 208).

Prerequisites¶

Remote-Pulse server running on LXC 280 (192.168.0.196:8080)
Prometheus server accessible (typically VM 208)
Grafana server accessible (typically VM 208)

Step 1: Configure Prometheus Scrape Job¶

Locate Prometheus Configuration¶

The Prometheus configuration file is typically located at: - /etc/prometheus/prometheus.yml (standard install) - Or within a Docker volume if containerized

Add Scrape Job¶

Add the following scrape job to the scrape_configs section of prometheus.yml:

scrape_configs:
  # ... existing jobs ...

  - job_name: 'remote-pulse-server'
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['192.168.0.196:8080']
        labels:
          service: 'remote-pulse'
          environment: 'homelab'
          instance: 'rp-server-lxc280'

Reference file: ansible/roles/observability_collector/files/prometheus/remote-pulse-scrape.yml

Reload Prometheus¶

After adding the scrape job, reload Prometheus configuration:

# If systemd service
sudo systemctl reload prometheus

# If Docker container
docker exec prometheus-container kill -HUP 1

# Or via HTTP API
curl -X POST http://localhost:9090/-/reload

Verify Scrape Target¶

Access Prometheus UI: http://<prometheus-host>:9090
Navigate to Status → Targets
Verify remote-pulse-server target shows as UP
Check Last Scrape column shows recent timestamp

Step 2: Install Grafana Dashboard¶

Option A: Manual Import (Grafana UI)¶

Access Grafana UI: http://<grafana-host>:3000
Navigate to Dashboards → Import
Upload the JSON file: ansible/roles/observability_collector/files/grafana/dashboards/remote-pulse-fleet.json
Select Prometheus datasource when prompted
Click Import

Option B: Provisioned Dashboard (Recommended)¶

If Grafana uses provisioned dashboards:

Copy dashboard JSON to Grafana provisioning directory:

sudo cp ansible/roles/observability_collector/files/grafana/dashboards/remote-pulse-fleet.json \
    /etc/grafana/provisioning/dashboards/

Ensure dashboard provisioning is enabled in /etc/grafana/grafana.ini or /etc/grafana/provisioning/dashboards/dashboard.yml:

apiVersion: 1
providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards

Reload Grafana:
```
sudo systemctl reload grafana-server
```

Verify Dashboard¶

Navigate to Dashboards → Browse
Search for "Remote-Pulse Fleet"
Open dashboard and verify panels populate with data

Step 3: Verify Metrics¶

Test Metrics Endpoint¶

# Direct curl to server
curl http://192.168.0.196:8080/metrics | head -30

# Expected output includes:
# - rp_host_up{host_id="...",hostname="...",group="..."}
# - rp_host_cpu_pct{...}
# - rp_heartbeat_total{group="..."}
# - rp_enroll_total{group="...",result="..."}

Query Prometheus¶

# Via Prometheus UI or API
curl 'http://<prometheus-host>:9090/api/v1/query?query=rp_host_up'

# Should return JSON with metrics for all enrolled hosts

Dashboard Panels Overview¶

The Remote-Pulse Fleet dashboard includes 17 panels across 9 rows:

Row 1: Fleet Overview (4 stat panels)¶

Total Hosts - Count of all registered hosts
Hosts Up - Count of hosts with last_seen < 180s
Hosts Down - Count of hosts with last_seen > 180s
Agent Versions - Count of distinct agent versions

Row 2: Per-Host CPU (repeated panel)¶

CPU Usage sparkline per host (filtered by $host_id variable)

Row 3: Per-Host Memory (repeated panel)¶

Memory Usage sparkline per host

Row 4: Per-Host Load (repeated panel)¶

Load Average sparkline per host

Row 5: Per-Host Last Seen (repeated panel)¶

Last Seen age with threshold colors (green <60s, yellow <180s, red >180s)

Row 6: Rates¶

Heartbeat Rate by Group (time series)
Command Execution Rate by type/group/result (time series)

Row 7: Security & Errors¶

Signature Invalid Attempts (stat, red threshold)
Local Policy Denies (time series)

Row 8: Performance¶

HTTP Latency p95 (time series)

Row 9: Server Health¶

DB Connections Active (stat)
WebSocket Connections (time series)
SSH Keys Active (stat)
Enrollment Tokens Active (stat)

Variables¶

The dashboard includes two template variables:

$host_id - Multi-select dropdown of all host IDs (from rp_host_up labels)
$group - Multi-select dropdown of all groups (from rp_host_up labels)

Troubleshooting¶

Metrics Not Appearing¶

Check server is running:
```
curl http://192.168.0.196:8080/health
```
Check /metrics endpoint responds:
```
curl http://192.168.0.196:8080/metrics
```
Check Prometheus target status (Status → Targets in UI)
Check Prometheus logs for scrape errors:
```
sudo journalctl -u prometheus -f
```

Dashboard Shows "No Data"¶

Verify Prometheus datasource configured in Grafana (Configuration → Data Sources)
Check time range selector (default is last 30 minutes)
Verify at least one agent has enrolled and sent heartbeat
Test query directly in Prometheus UI

Stale Metrics¶

Prometheus scrapes every 30s. If dashboard shows stale data: 1. Check Prometheus is actively scraping (Status → Targets) 2. Verify server timestamp is recent in metrics output 3. Check for network connectivity issues between Prometheus and LXC 280

Future Enhancements (F8)¶

Add IP allowlist for /metrics endpoint
Token-based authentication for scrape endpoint
Alert rules for host down, signature failures, high latency
Long-term metrics retention policy (Prometheus retention vs external TSDB)