Container Monitoring Operations Runbook¶

⚠️ DOCUMENTO LEGACY / HISTÓRICO (revisado 2026-07-12) Este runbook describe un sistema de monitorización antiguo basado en el alert-coordinator (bash + SQLite, "ML severity") y notificaciones a ntfy (topic media-alerts). Ese sistema quedó superado por Prometheus Alertmanager como componente central de alertas — ver alerting-flow.md. Se conserva por valor histórico; no lo uses como referencia operativa actual.

🚀 Emergency System Status¶

Alert Coordinator: ✅ OPERATIONAL (Fixed Feb 22, 2026) Enhanced Monitor: ✅ DEPLOYED - 67 containers monitored (1,300% improvement) Coverage: 67/76 running containers (88% coverage, up from 7%) Zero Silent Failures: ✅ ACHIEVED - alerts within 2 minutes

📊 System Overview¶

Components¶

Alert Coordinator (/home/monxas/alert-system/)
Intelligent monitoring with ML-based severity classification
Database: data/alerts.db (fixed with VACUUM operation)
Service: systemctl --user status alert-coordinator
Enhanced Container Monitor (/home/monxas/scripts/enhanced-critical-containers-monitor.sh)
Auto-discovery of all running containers
2-minute monitoring intervals
Multi-layer escalation (log → ntfy → telegram → restart)
Configuration (/home/monxas/scripts/monitor-config.yaml)
Health endpoints for HTTP checks
Escalation rules and thresholds
Blacklist for non-critical containers

🔧 Daily Operations¶

Check System Health¶

# Alert Coordinator Status
ssh [email protected] 'systemctl --user status alert-coordinator'

# Enhanced Monitor Status  
ssh [email protected] 'tail -5 /home/monxas/scripts/container-monitor.jsonl'

# Current Coverage
ssh [email protected] 'sudo docker ps | wc -l && cat /home/monxas/scripts/discovered-containers.json | jq ".containers | length"'

# Active Issues
ssh [email protected] 'ls -la /tmp/*alert* 2>/dev/null || echo "No active alerts"'

Monitor Logs¶

# Real-time monitoring logs
ssh [email protected] 'tail -f /home/monxas/scripts/container-monitor.jsonl'

# Cron execution logs
ssh [email protected] 'tail -f /home/monxas/scripts/container-monitor-cron.log'

# Alert Coordinator logs
ssh [email protected] 'tail -f /home/monxas/alert-system/data/coordinator.log'

🚨 Escalation Levels¶

Level 1: Monitoring¶

Trigger: First container failure detected
Action: Log entry created, monitoring begins
Alert: None (silent monitoring)

Level 2: ntfy Alert¶

Trigger: First failure (immediate alert for zero silent failures)
Action: ntfy notification sent to media-alerts topic
Alert: ⚠️ Warning notification

Level 3: Telegram + Auto-Restart¶

Trigger: 3rd failure within 10 minutes
Action: Telegram alert + automatic container restart attempt
Alert: 🚨 Critical escalation

Level 4: Emergency Escalation¶

Trigger: Sustained failure >10 minutes
Action: Emergency telegram alert, manual intervention required
Alert: ⚡ EMERGENCY status

📦 Adding New Services¶

Automatic Inclusion (Default)¶

New containers are automatically discovered and included within 2 minutes of deployment. No manual intervention required.

Custom Health Checks¶

Edit /home/monxas/scripts/monitor-config.yaml:

containers:
  your-new-service:
    health_endpoint: "http://localhost:PORT/health"
    critical: true
    auto_restart: true

Blacklist Non-Critical Services¶

Add to the blacklist section in monitor-config.yaml:

blacklist:
  - "your-non-critical-service"
  - "*-temp"  # Pattern matching supported

⚙️ Threshold Adjustments¶

Modify Alerting Sensitivity¶

Edit /home/monxas/scripts/enhanced-critical-containers-monitor.sh:

# Current settings (zero silent failures)
FAILURES_BEFORE_ALERT=1      # Alert on first failure
FAILURES_BEFORE_TELEGRAM=3   # Escalate after 3 failures  
FAILURE_WINDOW_MINUTES=10    # Time window for counting failures

Change Monitoring Frequency¶

# Edit crontab
ssh [email protected] 'crontab -e'

# Current: every 2 minutes
*/2 * * * * /home/monxas/scripts/enhanced-critical-containers-monitor.sh

# Change to 1 minute for ultra-fast detection:
*/1 * * * * /home/monxas/scripts/enhanced-critical-containers-monitor.sh

# Change to 5 minutes for less aggressive monitoring:
*/5 * * * * /home/monxas/scripts/enhanced-critical-containers-monitor.sh

🧪 Testing & Validation¶

Test Alert System¶

# Temporarily stop a critical container to test alerting
ssh [email protected] 'sudo docker stop sonarr'

# Wait 2-4 minutes, verify alert received via ntfy
# Check logs for escalation
ssh [email protected] 'tail -10 /home/monxas/scripts/container-monitor.jsonl'

# Restart container 
ssh [email protected] 'sudo docker start sonarr'

Validate Discovery¶

# Force cache refresh and check discovery
ssh [email protected] 'rm -f /home/monxas/scripts/discovered-containers.json'
ssh [email protected] 'cd /home/monxas/scripts && ./enhanced-critical-containers-monitor.sh'
ssh [email protected] 'cat /home/monxas/scripts/discovered-containers.json | jq ".containers | length"'

Test Recovery¶

# Simulate container restart failure
ssh [email protected] 'sudo docker stop radarr && sudo docker rename radarr radarr-broken'

# Wait for escalation to auto-restart level
# Verify telegram alert and auto-restart attempt
# Clean up test
ssh [email protected] 'sudo docker rename radarr-broken radarr && sudo docker start radarr'

🛠️ Troubleshooting¶

Alert Coordinator Issues¶

# Database corruption
ssh [email protected] 'systemctl --user stop alert-coordinator'
ssh [email protected] 'cd /home/monxas/alert-system/data && sqlite3 alerts.db "VACUUM;"'
ssh [email protected] 'systemctl --user start alert-coordinator'

# Service won't start
ssh [email protected] 'journalctl --user -u alert-coordinator -f'

Enhanced Monitor Issues¶

# Check script permissions
ssh [email protected] 'ls -la /home/monxas/scripts/enhanced-critical-containers-monitor.sh'

# Verify dependencies
ssh [email protected] 'which jq yq curl sudo'

# Test manual run
ssh [email protected] 'cd /home/monxas/scripts && ./enhanced-critical-containers-monitor.sh'

Missing Containers¶

# Check blacklist
grep -A 20 "BLACKLIST_CONTAINERS" /home/monxas/scripts/enhanced-critical-containers-monitor.sh

# Force rediscovery
ssh [email protected] 'rm -f /home/monxas/scripts/discovered-containers.json'

📈 Performance Metrics¶

Target SLAs¶

Detection Time: < 2 minutes for any container failure
Coverage: > 85% of running containers monitored
False Positives: < 5% of alerts
Auto-Recovery Rate: > 70% of issues resolved without manual intervention

Current Performance (as of Feb 22, 2026)¶

✅ Detection Time: 2 minutes (cron interval)
✅ Coverage: 88% (67/76 containers)
✅ Auto-Discovery: 100% of new containers included within 2 minutes
✅ Zero Silent Failures: ACHIEVED

🔄 Maintenance Schedule¶

Daily¶

Review alert logs for patterns
Verify coverage remains >85%

Weekly¶

Check Alert Coordinator database size and performance
Review escalation metrics
Update blacklist if needed

Monthly¶

Backup monitoring configuration
Review and tune alerting thresholds
Validate all health check endpoints

📞 Emergency Contacts¶

System Owner: Mon (Telegram alerts configured)
Backup: Check /tmp/telegram-alert-for-openclaw.json for fallback alerts Logs Location: /home/monxas/scripts/container-monitor*.log

Updated: February 22, 2026 - Emergency monitoring system overhaul complete