Skip to content

ADR-0013: WP-Pulse operator QoL — daily ergonomics, fleet health, surgical recovery, bulk ops

Status: Proposed Date: 2026-05-30 Deciders: Ramón Kamibayashi + Claude Opus 4.7 Related: ADR-0010 WP-Pulse, ADR-0011 Snapshot model, ADR-0012 Web UI redesign

Technical story: ADR-0010 shipped the engine (pull/push/version/diff/schedule/preview). ADR-0011 collapsed the data model. ADR-0012 redesigned the UI around operator verbs. What is missing now is the everyday operator experience: starting a work session is still 5 clicks, fleet-wide signals require visiting each site, recovery from a bad push is "find a version and hope", and cross-fleet operations (search-replace, upgrades) are manual repetition. This ADR fixes that with 13 features in 7 phases, each one selected from a brainstorm against a single operator persona (Ramón) running ~10 sites mixed legacy + modern.

Context

Operator persona (single-tenant assumption)

  • 1 operator, ~10 WP sites, mix of WP 4.6/PHP 5.6 (hotelaldamagolf, SiteGround hosting) through WP 6.4/PHP 8.x.
  • Customer-facing: one site is a live hotel; downtime is visible.
  • Work mode: episodic. Operator opens the tool, does 30–90 min of work on 1–3 sites, closes it.
  • Existing automation: schedules pull at night; visual-diffs are captured on every pull; pushes auto-version pre/post; rollback banner lives 10 min after a destructive op.

What is already shipped (post ADR-0010/11/12)

  • Pull / push / version / load_version / visual-diff / schedule / preview, plus the new single-page site layout (SiteHero + ActionSidebar + ActivityFeed + collapsible panels), dark mode, keyboard shortcuts (g h/f/s/a/o/S, /, ?).
  • 11 alembic migrations applied. Web on LXC 281, agents on each WP host.

What still hurts (audited from the 2026-05-30 E2E session)

# Pain Where it hurts
1 "Start working on site X" is a 5-step ritual (open → check stale → pull → start stack → open preview) Friction every single session
2 No way to reach any action without scanning the UI New muscle memory not formed
3 Site-specific gotchas (don't touch plugin X, theme custom in footer line 47) live in operator's head Knowledge loss on context-switch
4 Experimental work creates noisy versions Version list pollution
5 Fleet-wide drift (WP/PHP/SSL/DB-size) requires opening each site Reactive instead of proactive
6 "What changed between v_a and v_b?" → no answer Forensic blind
7 "Bring back yesterday's site as a sandbox" → no answer Cannot demo "before" to a client
8 A plugin causes fatal in prod → manual SSH + edit Slow under pressure
9 Same string typo across the fleet → manual per site Doesn't scale
10 Upgrading PHP/WP version on a legacy site → terrifying, no rehearsal Risk avoidance = stuck on old stacks
11 wp-content/uploads/ grows unbounded Storage + page weight

Out of scope (intentionally rejected from the brainstorm)

Listed for the record so they don't sneak back later:

  • Email deliverability monitor, Core Web Vitals tracker, 404/5xx aggregator, login attempts heatmap — value is real but not this operator's biggest friction today.
  • Bulk plugin updater fleet-wide, auto-minor WP-core, stale-plugin sniper, license audit — fleet is small enough to do manually; canary + rollback infra would be over-built.
  • All client-billing / status-page / monthly report / approval-via-Telegram features (section D of brainstorm) — operator is solo, doesn't need to externalize anything yet.
  • DR drill automation, smart rollback target, malware scanner, file integrity monitoring — current rollback + version model is already a strong safety net for the threat model.

Decision

Ship 13 features in 7 phases over an estimated 10–15 days of paralleled agent work.

Phase Features Effort Independence
F1 — Ergonomía diaria (1) Start session, (3) Notebook, (4) Scratch mode, (28) Quarantine plugin S Parallel-safe
F2 — Command palette (2) ⌘K S Parallel-safe
F3 — Fleet health (6) Version radar, (7) SSL expiry, (8) DB size trend M Parallel-safe
F4 — Recovery quirúrgico (26) Cross-version diff, (25) Time-travel preview M Depends on F1 (notebook) for nothing else
F5 — Media optimizer (38) Bulk media optimizer S/M Independent
F6 — Mass search-replace (14) Cross-fleet S&R M Depends on local stack readiness primitives from F1
F7 — Upgrade rehearsal (17) PHP/WP rehearsal lane L Hardest; build last, with everything else as scaffolding

F1/F2/F3 can ship in a single session as three parallel sub-agents (proven pattern from ADR-0012). F4 and F5 in a second session. F6 alone. F7 last because it depends on rehearsal-stack primitives that F4's time-travel preview already proves out.

F1 — Ergonomía diaria

Four small, high-frequency wins.

F1.1 — "Start session" button (idea #1)

Goal: From the site card or the new SiteHero, one click that takes the operator from cold to "browser pointed at preview, ready to edit".

Flow: 

operator clicks "Open & work"
  ↓
client-side orchestrator (no new backend state):
  1. GET  /api/v1/sites/{slug}/stack/status
  2. if not running → POST /api/v1/sites/{slug}/stack/start, poll until ready
  3. if last_pull older than session_fresh_pull_hours → POST /api/v1/sites/{slug}/pulls {auto_load: true}
     subscribe to /api/v1/sites/{slug}/events (SSE from F4) for progress
  4. once worktree ready → window.open(preview_url, "_blank")

New / changed: - POST /api/v1/sites/{slug}/pulls already exists; add optional auto_load: boolean param that internally calls load_version on the newly-created version after fetch completes. No new endpoint. - New component web/components/app/StartSessionButton.tsx — client component. Single primary-color button. Tooltip describes the steps. Shows inline progress chip with current step. - New setting session_fresh_pull_hours (default 24) in server_settings table; surfaced in /settings UI. - Mount in SiteHero top-right and in fleet site cards.

Edge cases: - Stack start fails (port conflict, image missing) → fall back to "show me the LocalStackPanel". Don't block the operator. - Pull is already in progress → join the SSE stream instead of starting a new pull. - Operator clicks again mid-flow → idempotent (button shows disabled while session orchestration in flight; per-tab in-memory lock keyed by slug).

Telemetry: log session_started event into activity_feed so it shows up in ActivityFeed.

Estimated LOC: ~250 (mostly new component + auto_load wiring in pulls.py).

F1.2 — Per-site notebook (idea #3)

Goal: A markdown scratchpad attached to each site. "Don't update plugin X. The custom footer.php logo override is at line 47. The client prefers Mondays for deploys."

Data model: - Alembic migration 012: 

op.add_column('sites', sa.Column('notes_markdown', sa.Text(), nullable=False, server_default=''))
op.add_column('sites', sa.Column('notes_updated_at', sa.DateTime(timezone=True), nullable=True))
- No history table for v1 (KISS). If we want history later, we add a site_notes_history table; the column can be promoted to "latest" then.

API: - GET /api/v1/sites/{slug}/notes → {markdown: string, updated_at: datetime | null}. - PUT /api/v1/sites/{slug}/notes body {markdown: string} → returns same shape. Max 64 KB per note.

UI: - New <NotebookPanel> mounted as a CollapsiblePanel in the new ADR-0012 site layout. Default expanded if notes non-empty. - Two modes: render (markdown → safe HTML via marked + dompurify if not already on deps; or react-markdown which is already common) and edit (textarea). Edit toggle button. Save on blur + explicit Save button. - Hero shows a 1-line snippet badge "📝 has notes" if non-empty; clicking jumps to the panel.

Edge cases: - Concurrent edit (operator on two tabs): last write wins. Surfaced in notes_updated_at so a second tab's "Save" can warn "this was updated 30s ago — overwrite?" via an If-Unmodified-Since style precondition. Optional; v1 just last-write-wins.

Estimated LOC: ~200.

F1.3 — Scratch mode (idea #4)

Goal: Toggle on the site detail. When ON, the local worktree is marked scratch: - No version is created pre/post any operation - Push button is disabled with a clear tooltip - Stop & discard is one click

Data model: - Worktree metadata file /var/lib/wp-pulse/sites/{slug}/worktree.meta.json already exists (managed by the stack module). Add field scratch: boolean defaulting to false. - No DB change.

API: - POST /api/v1/sites/{slug}/worktree/scratch body {enabled: boolean}. When enabling, requires worktree clean OR explicit discard_pending: true. When disabling, same. - Existing push endpoint reads the flag; rejects with 409 {error: "scratch_mode_active"} if true. - Existing pull/load endpoints respect the flag: skip auto-version-pre and auto-version-post.

UI: - Switch in ActionSidebar footer: "Scratch mode" with tooltip "Experiments only — no version saved, push disabled". - When ON, ActionSidebar Push button greyed out with red border; tooltip explains. - When ON, SiteHero shows orange banner: "Scratch mode — changes will be discarded on next pull."

Edge cases: - Operator toggles OFF with pending changes → prompt: "Discard pending changes? (Yes / Save as version then disable / Cancel)". - Schedule fires while scratch mode is on → schedule is paused for that site until off. Schedule run records skipped_reason: "scratch_mode".

Estimated LOC: ~180.

F1.4 — Quarantine plugin (idea #28)

Goal: A plugin is causing a fatal in prod (Fatal error: Cannot redeclare …). The operator needs it disabled NOW, without restoring an older version, without SSH.

Mechanism: Push a mu-plugin that filters active_plugins to remove the offender. WordPress evaluates mu-plugins before regular plugins, so the offender never loads. No DB write needed.

Generated mu-plugin file at wp-content/mu-plugins/wpp-quarantine.php: 

<?php
/*
 * Plugin Name: WP-Pulse Quarantine
 * Description: Managed by wp-pulse-server. Do not edit by hand.
 */
$wpp_quarantined = [{slug_list}];
add_filter('option_active_plugins', function($plugins) use ($wpp_quarantined) {
    return array_values(array_diff($plugins, $wpp_quarantined));
});
add_filter('site_option_active_sitewide_plugins', function($plugins) use ($wpp_quarantined) {
    foreach ($wpp_quarantined as $q) unset($plugins[$q]);
    return $plugins;
});

$wpp_quarantined is a list of "plugin-slug/plugin-slug.php" strings (the exact format WP stores in active_plugins).

Data model: - Alembic 012 (same migration as notebook): 

op.add_column('sites', sa.Column('quarantined_plugins', postgresql.JSONB(), nullable=False, server_default='[]'))

API: - POST /api/v1/sites/{slug}/quarantine body {plugin_path: "redux-framework/redux-framework.php"} → regenerates the mu-plugin, triggers a push of just that file, updates quarantined_plugins column. Returns {push_id, quarantined_plugins}. - DELETE /api/v1/sites/{slug}/quarantine/{plugin_path:path} → removes from list, regenerates or deletes mu-plugin, pushes. (If list becomes empty, delete the mu-plugin entirely.) - GET /api/v1/sites/{slug}/quarantine → current list + history (from pushes table).

UI: - ActionSidebar new button "Quarantine plugin" (always visible, low priority position). Click → modal lists active plugins from the latest manifest with toggle switches showing current quarantine state. Apply → preview the diff (single file change) → confirm → push. - After successful quarantine, ActivityFeed row: "🚫 Quarantined plugin: redux-framework". Sticky banner offers Unquarantine (calls DELETE). - If site has any quarantined plugins, SiteHero shows a red "🚫 N plugins quarantined" badge.

Edge cases: - Multisite: filter both option_active_plugins and site_option_active_sitewide_plugins. The template above does both. - Plugin uses register_activation_hook for required tables → quarantine only stops it from loading, not unwinding DB state. That's fine; tables remain idle. - Operator tries to quarantine WP-Pulse agent itself → reject with 400.

Estimated LOC: ~250.

F2 — Command palette ⌘K (idea #2)

Goal: ⌘K (Ctrl+K on non-Mac) opens a modal. Fuzzy-search across sites, actions, navigation, recent activity, and (advanced) wp-cli commands. Selecting an item executes the action or navigates.

Library

Use cmdk (MIT, by pacocoursey, used by Vercel/Linear/Raycast). ~8 KB gzipped, headless React component. No alternative is as battle-tested.

Add to package.json dependencies. License compliant.

Source set

Indexed in-memory on every palette open (refetched if older than 30 s):

type CmdItem =
  | { type: "site"; slug: string; label: string; status: SiteStatus }
  | { type: "action"; siteSlug?: string; label: string; verb: ActionVerb; available: boolean }
  | { type: "nav"; label: string; href: string }
  | { type: "recent"; label: string; href: string; timestamp: string }
  | { type: "wpcli"; label: string; command: string; siteSlug: string };
  • Sites: from /api/v1/fleet/sites/lite (slug + status only, no manifest).
  • Actions (context-aware): if a site detail page is in view, add per-site actions (Pull, Push, Save, Restore, Quarantine, Open preview, Schedule). Otherwise add global actions (New site, Open fleet, Refresh all).
  • Nav: static list — Home, Fleet, Onboard, Settings, Schedules tab, Activity tab.
  • Recent: last 10 entries from the global activity feed (deduped by site).
  • WP-CLI mode: prefix > switches to CLI mode. While site context is active, type > option get blogname, hit Enter → executes against local stack, output in a result panel below.

WP-CLI security

  • New endpoint POST /api/v1/sites/{slug}/wpcli body {command: string} → {stdout, stderr, exit_code, duration_ms}.
  • Whitelist enforced in api/utils/wpcli_safelist.py:
  • Allowed verbs: option get, user list, user get, post list, post get, plugin list, plugin status, theme list, theme status, cron event list, cache flush, transient delete, core version, db size, eval-file --dry-run.
  • Allowed flags: --format, --field, --fields, --user, --allow-root (always added).
  • Blocked: eval, db query, db cli, db export, db import, cli alias, package install, --require, --exec, shell, any path starting with /.
  • Anything not on the allowlist → 403 with the message.
  • Destructive commands inside the allowlist (cache flush, transient delete) require X-Confirm: yes header (palette UI adds an "Are you sure?" step).
  • Every executed command is logged to a new audit table wpcli_audit (id, site_uuid, command, exit_code, stdout_b64, stderr_b64, executed_at, operator).

UI

  • New component web/components/app/CommandPalette.tsx. Client component. Mounted in root layout.
  • Keyboard shortcut: hook the existing useKeyboardShortcuts hook (extends Phase 6's binding). Cmd+K / Ctrl+K toggles open.
  • Two modes:
  • Default: fuzzy search across the source set above. Up/down arrows to navigate, Enter to execute, Esc closes.
  • CLI mode: triggered by > prefix. Different styling (mono font, prompt). Result rendered inline.
  • Recent items appear at top of empty palette.
  • Highlight match characters in result labels.

Edge cases

  • Site label collisions (hotelaldamagolf-com vs hotelaldamagolf-com-staging): sort by status (active first) and recency.
  • Action items are filtered by availability (e.g. "Restore" only if versions exist).
  • Network failure on fetch: show cached set with a "stale" indicator.
  • Palette stays open for chained actions? No — closes on execute. (Open again if needed; faster than persistent mode.)

Estimated LOC: ~600 (component + safelist + endpoint + tests).

F3 — Fleet health (three sub-features, one tab)

Adds a new "Health" tab to the Fleet page (alongside Activity, Schedules, Batches, etc.) showing one row per site with version, SSL, and DB-size signals.

F3.1 — WP/PHP version radar (idea #6)

Agent contract: The agent's heartbeat payload already includes wp_version, php_version. Verify by reading agent_heartbeat.py; if missing, add. Server stores in sites.wp_version, sites.php_version, sites.mysql_version (add column if missing) + sites.versions_last_seen_at.

Reference data: - New cron latest_versions_sync.py daily at 03:00: - Fetches https://api.wordpress.org/core/version-check/1.7/ → latest WP stable version. - Fetches PHP supported branches from https://www.php.net/supported-versions.json (or the human-readable page parsed; the JSON one is community-maintained, prefer the static list maintained in code with manual updates). - Stores in version_reference table: 

create table version_reference (
  key text primary key,         -- "wp_latest", "php_supported", "mysql_supported"
  value jsonb not null,
  updated_at timestamptz not null
);

Staleness rules (computed server-side): - WP: green if current == latest_stable, amber if latest_minor_behind, red if latest_major_behind. - PHP: green if >= 8.1, amber if 8.0, red if 7.x supported, black if <= 7.3 (EOL'd). - MySQL: green if >= 8.0 or >= MariaDB 10.6, amber if 5.7, red otherwise.

API: GET /api/v1/fleet/health → array per site: 

{
  "slug": "hotelaldamagolf-com",
  "wp_version": "4.6.17",
  "wp_status": "red",
  "wp_latest_stable": "6.4.7",
  "php_version": "7.4.33",
  "php_status": "red",
  "mysql_version": "5.7.41",
  "mysql_status": "amber",
  "versions_last_seen_at": "2026-05-30T11:42:13Z",
  "ssl_expires_at": "2026-08-17T00:00:00Z",
  "ssl_status": "green",
  "db_size_bytes": 1284513792,
  "db_size_status": "green",
  "db_size_growth_pct_7d": 4.2
}

F3.2 — SSL expiry tracker (idea #7)

  • New cron ssl_check.py daily at 04:00.
  • For each site, Python: 
    import ssl, socket
ctx = ssl.create_default_context()
with socket.create_connection((host, 443), timeout=10) as sock:
    with ctx.wrap_socket(sock, server_hostname=host) as ssock:
        cert = ssock.getpeercert()
expires = datetime.strptime(cert['notAfter'], '%b %d %H:%M:%S %Y %Z')
  • Store in sites.ssl_expires_at, sites.ssl_last_checked_at, sites.ssl_issuer.
  • Status: green > 30d, amber 14–30d, red < 14d, black: expired.
  • Edge case: site behind CF/Cloudflare → returns CF cert, not origin. New sites.ssl_check_target column: auto (host + 443), host:port, origin_ip:443 (when operator provides explicit origin IP override). Default auto.
  • Alert: when status transitions to red, raise an event of type ssl_expiring in the activity feed and dispatch the existing Telegram alert.

F3.3 — DB size trend (idea #8)

Data ingestion: - Agent extends heartbeat with db_size_bytes (use INFORMATION_SCHEMA.TABLES SUM(data_length + index_length) for the site's DB). - Server stores latest value on sites.db_size_bytes. - Daily rollup: new table 

create table site_metrics_daily (
  site_uuid uuid not null references sites(uuid) on delete cascade,
  day date not null,
  db_size_bytes bigint,
  file_size_bytes bigint,
  plugin_count int,
  primary key (site_uuid, day)
);
- Cron metrics_rollup.py runs at 00:30 daily, snapshots current values into yesterday's row. - Retention: 365 days (delete older).

API: - GET /api/v1/sites/{slug}/metrics?days=90 → {db_size: [{day, bytes}, ...], file_size: [...], plugin_count: [...]} - Included in /fleet/health response: db_size_growth_pct_7d (last value vs 7 days ago).

UI for all three:

  • New FleetHealthTab mounted as 6th tab in the fleet page tabs. Table: slug | WP | PHP | MySQL | SSL expiry | DB size + sparkline | actions.
  • Color chips per cell using the existing semantic tokens (bg-success, bg-warning, bg-danger).
  • Sortable by each column; default sort: worst-status-first (any red bubbles to top).
  • Sparkline for DB size: inline SVG, no chart library — 60-day series, 80×20 px.
  • Click any row → site detail.

Estimated LOC F3 total: ~900 (3 features, cron jobs, alembic, fleet tab UI).

F4 — Recovery quirúrgico

F4.1 — Cross-version diff (idea #26)

Goal: Pick v_a and v_b. See exactly what differs. Files added/removed/modified, DB rows changed (focus on wp_options, wp_posts, wp_postmeta, wp_users).

Implementation:

  • New async endpoint: 
    POST /api/v1/sites/{slug}/versions/diff
body: {version_a: uuid, version_b: uuid, scope?: ["files","db","both"]}
→ 202 {job_id}

GET  /api/v1/sites/{slug}/versions/diff/{job_id}
→ {status, progress, result?, error?}
  • Job runner: extracts both version tarballs into /tmp/wpp-diff-{job}/{a,b}/ (uses a worker pool, max 2 concurrent jobs server-wide).
  • Files diff:
  • diff -r --brief a b → list of added/removed/modified paths.
  • For each modified file ≤ 256 KB and not binary, run diff -u and capture hunks (cap 100 lines per file).
  • Cap modified-file detail at 50 entries; the rest are summarised by path.
  • Skip wp-content/uploads/ from byte-diff (just report size delta + count).
  • DB diff:
  • Don't actually load both into mysql. Parse the .sql.gz dumps directly:
    • Walk INSERT statements, build dicts keyed by primary key per table.
    • For wp_options: key by option_name, surface every value change (option_value full text up to 4 KB).
    • For wp_posts: key by ID, surface action (added/removed/modified), title, post_status, modified date.
    • For wp_postmeta: aggregate (post_id, meta_key) change counts, don't dump full values unless ≤ 2 KB.
    • For other tables: just row-count delta.
  • Result shape: 
    {
  "files": {
    "added": ["wp-content/themes/melinda/style.css", ...],
    "removed": [...],
    "modified": [
      {"path": "wp-config.php", "size_before": 4120, "size_after": 4180, "diff_unified": "@@ -42,3 +42,4 @@\n..."},
      ...
    ],
    "summary": {"added": 3, "removed": 1, "modified": 12, "binary_modified": 4}
  },
  "db": {
    "options_changed": [{"option_name": "blogname", "before": "Old", "after": "New"}, ...],
    "posts_changed": [{"id": 142, "title": "...", "action": "modified"}],
    "table_row_deltas": {"wp_posts": {"before": 421, "after": 425, "delta": 4}, ...}
  },
  "stats": {"duration_ms": 12450, "extracted_bytes": 8500000000}
}
  • Cleanup: temp dirs removed at job end, also a sweeper cron in case a job crashes.

UI:

  • New <VersionDiffView> component reached via VersionsPanel: each version row gets a "Compare with…" action that opens a modal with a second-version picker.
  • After diff completes (10–60 s typical), render in three tabs: Files / Database / Summary.
  • Files tab: tree view of changed paths with status icons. Click a file to see the unified diff in a code viewer (use react-diff-viewer-continued if not heavy, else simple <pre> styled).
  • Database tab: separate sub-sections per table. Options table is most useful — render as before/after side-by-side.

Edge cases: - Versions > 5 GB: refuse with 413, suggest "diff only one scope". Or run in files-only mode automatically. - Operator runs many diffs at once: queue them, max 2 concurrent. - Same version twice: short-circuit, return empty diff.

Estimated LOC: ~800.

F4.2 — Time-travel preview (idea #25)

Goal: "Show me the site as it was on March 12 at 15:00." Spin up an ephemeral stack from version v_X, attached to a unique hostname tt-{slug}-{shortid}.preview.monxas.casa, valid for 1 hour.

Implementation:

  • New endpoint: 

    POST /api/v1/sites/{slug}/versions/{version_id}/preview
body: {ttl_minutes?: number = 60}
→ 202 {preview_id, status: "provisioning", expires_at?}

GET  /api/v1/version-previews/{preview_id}
→ {status, hostname, url, expires_at, error?}

DELETE /api/v1/version-previews/{preview_id}
→ 204 (early teardown)

  • New DB table: 

    create table version_previews (
  id uuid primary key default gen_random_uuid(),
  site_uuid uuid not null references sites(uuid) on delete cascade,
  version_id uuid not null references site_versions(id),
  hostname text not null unique,
  port int not null,
  compose_project text not null,
  status text not null,        -- provisioning | running | expired | failed | destroyed
  started_at timestamptz default now(),
  expires_at timestamptz not null,
  destroyed_at timestamptz,
  error text
);
create index on version_previews(status, expires_at);

  • Provisioning flow:

  • Allocate port from pool 9100–9200 (skip ports already in version_previews where status in ('provisioning','running')).
  • Generate compose project name wpp-tt-{shortid}.
  • Create a docker-compose.yaml in /var/lib/wp-pulse/previews/{preview_id}/ with the standard stack (php-fpm + nginx + mariadb) — same templates as the main local stack module.
  • Load version v_X (extract tarball + import SQL).
  • Run wp search-replace from original prod URL → ephemeral hostname.
  • Bring up docker compose up -d. Wait for health (HTTP 200 on /).
  • Write a Caddy snippet on LXC 270/271 reverse-proxying tt-{slug}-{shortid}.preview.monxas.casa → localhost:{port}.
    • Reuse the same Caddy snippet generator + homelab-ctl.py deploy used by the existing per-site preview module (see api/utils/preview.py).

  • Mark status running.

  • Teardown cron cleanup_previews.py every 5 min:

  • For each row where status='running' and expires_at <= now():
    • docker compose -p {project} down -v
    • Remove Caddy snippet, redeploy LXCs
    • Set status='destroyed', destroyed_at=now().

  • For stuck provisioning older than 10 min: mark failed, clean up.

  • Concurrency cap: max 3 active previews server-wide. New request when at cap → 429 with retry-after.

UI: - VersionsPanel: each row gets a clock icon "Time-travel preview". Click → modal "Spin up a sandbox of this version for 1 hour?" → confirm → progress (provisioning… extracting… starting… 30–90 s) → URL ready → "Open" button. - Active previews surfaced as a sticky banner at the top of site detail: "🕰️ Sandbox open: tt-…preview.monxas.casa — expires in 47m [Open] [Destroy now]". - A new global page /previews lists active previews fleet-wide.

Edge cases: - Port exhaustion: refuse with 503, explain "max 3 sandboxes server-wide". - Caddy snippet deploy fails (LXC unreachable): keep the container running, surface the localhost port + a "deploy snippet manually" hint. Don't fail the whole request. - DNS: *.preview.monxas.casa already wildcard-resolves to LXC 270/271 (verify; if not, add to DNS). - Cleanup race: if cleanup cron runs during a manual DELETE, the second wins; both are idempotent on docker compose down.

Security: - Sandbox hostnames are randomized so they're not guessable. - Caddy snippet adds the same PocketID forward_auth as production preview hostnames — only the operator can reach it. - Stack image is the project's standard image (no untrusted code loaded), but mariadb has the version's full DB contents — treat as sensitive.

Estimated LOC: ~700.

F5 — Bulk media optimizer (idea #38)

Goal: After a pull, scan wp-content/uploads/, downsize over-large images, re-encode at sensible quality, report MB saved. Operator reviews before pushing optimized files.

Implementation:

  • New endpoint: 

    POST /api/v1/sites/{slug}/media/optimize
body: {
  dry_run?: boolean = true,
  max_width?: int = 2000,
  jpeg_quality?: int = 85,
  convert_png_to_webp?: boolean = false,
  skip_if_saving_below_pct?: int = 10
}
→ 202 {job_id}

GET /api/v1/sites/{slug}/media/optimize/{job_id}
→ {status, progress, summary?, candidates?}

  • Pre-req tools in the wpcli docker image: imagemagick, cwebp, mozjpeg (or jpegoptim), optipng.

  • Worker walks wp-content/uploads/ in the worktree:
  • For each .jpg/.jpeg:
    • If width > max_width → resize.
    • Re-encode with mozjpeg at jpeg_quality.
    • Compare new size; if savings < skip_if_saving_below_pct% → skip (keep original).
  • For each .png:
    • Run optipng -o2 (lossless).
    • If convert_png_to_webp and image has no transparency → convert to .webp, only if the .png file is not referenced by exact-extension links (grep through wp-content/). Default OFF — risky.
  • Skip .gif, .svg, .pdf, animated formats.
  • Output: a candidates list per file with (path, size_before, size_after, action, savings_pct). On dry_run, no files are modified — only the candidates list is returned.
  • Apply mode (dry_run: false): writes changes in-place to the worktree.

UI: - New action sidebar button "Optimize media". Click → modal with knobs (sliders for max width, jpeg quality, toggles). "Preview" → dry-run, render results table with checkboxes per candidate (default all checked). "Apply selected" → re-run with dry_run: false for the selected paths. - After apply: summary toast "Saved 412 MB across 824 files. [Push to prod] [Discard]". Push button opens the existing PendingChangesPanel with all modified images staged.

Edge cases: - Re-running on already-optimized media → most files skip via the savings threshold. - File permission issues in worktree → log and continue, surface failed paths in the summary. - Memory: process one image at a time. Total job memory bounded. - Large library (>10 GB uploads): job streams; progress reported every 100 files.

Estimated LOC: ~500.

F6 — Mass search-replace (idea #14)

Goal: Change a phone number, a URL, a copyright year across every site that has it. Preview matches per site, apply with canary.

Implementation:

Preview phase

  • POST /api/v1/fleet/search-replace/preview body: 
    {
  "search": "(555) 123-4567",
  "replace": "(555) 987-6543",
  "sites": ["all"] | ["slug1", "slug2"],
  "tables": ["wp_posts", "wp_postmeta", "wp_options"] // optional, default = all
}
→ 202 {job_id}
  • Worker (orchestrator runs sequentially, max 1 concurrent fleet job):
  • For each site: ensure local stack is running (start if not). Skip sites in scratch mode (warn).
  • On each stack, execute wp search-replace "..." "..." --dry-run --report-changed-only --format=json. Parse output.
  • Aggregate into a matrix: 
    {
  "results": [
    {"slug": "hotelaldamagolf-com", "stack_ready": true, "matches_per_table": {"wp_posts": 3, "wp_options": 1}, "total": 4},
    {"slug": "site-b", "stack_ready": false, "skipped": true, "reason": "scratch_mode"},
    ...
  ],
  "stats": {"sites_scanned": 8, "sites_with_matches": 5, "total_matches": 47}
}
  • Cap: max 30 sites per preview job.

Apply phase

  • POST /api/v1/fleet/search-replace/apply body: 
    {
  "search": "...",
  "replace": "...",
  "sites": ["slug1", "slug2", ...],
  "canary": true,
  "abort_on_http_5xx": true,
  "abort_on_fatal_log": true
}
→ 202 {job_id}
  • Per-site sequence:
  • Pick canary = first in list. For each subsequent site after the first canary succeeds, treat as normal.
  • wp search-replace (no --dry-run) on local stack.
  • Save version "Pre mass-search-replace: ".
  • Push to prod.
  • Health check: GET prod_url → must be 200, no fatal in tail of error_log. If abort_on_http_5xx and check fails → rollback the push, mark this site failed, abort the whole batch.
  • Continue to next site.
  • After full apply: emit a fleet-level mass_search_replace event in each affected site's activity feed.

UI

  • New page /fleet/tools/search-replace:
  • Form: search input, replace input, site multi-picker (default all), table filter dropdown.
  • "Preview" button → spinner → results matrix with per-site match counts. Each row has a checkbox; default all-with-matches checked.
  • "Apply to selected" → confirm modal "This will push changes to N sites. The first site is the canary; if it fails, the rest are skipped. Continue?".
  • Progress view: per-site status as the batch runs (pending / in-progress / done / failed / aborted).
  • Reachable from command palette via > sr shortcut.

Edge cases

  • Site's local stack can't start → mark skipped with reason.
  • A site's database has the string in serialized PHP data (a:N:{s:M:"…"}) — wp search-replace already handles this correctly. Do not implement raw SQL REPLACE() — would corrupt serialized data.
  • Two sites have very different match counts (one has 1, one has 200) — surface clearly; operator can deselect the noisy one.
  • Operator wants regex: NOT in v1 (foot-gun). v1 is exact-string only. Add --regex flag in v2 with explicit double-confirmation.

Estimated LOC: ~700.

F7 — Upgrade rehearsal lane (idea #17)

Goal: "Rehearse upgrading hotelaldamagolf-com from WP 4.6 / PHP 7.4 to WP 6.4 / PHP 8.1." System clones the worktree into a rehearsal stack with the target images, runs the upgrade, runs smoke tests, produces a compatibility report. Operator decides whether to promote.

This is the biggest feature. Build last; reuse primitives from F4 (ephemeral compose stacks) and F5 (worker pool).

Inputs

POST /api/v1/sites/{slug}/rehearsals
body: {
  target_wp_version?: string,        // optional; default = latest_stable
  target_php_version?: string,       // optional; one of: "7.4","8.0","8.1","8.2","8.3"
  target_mysql_version?: string,     // optional; "5.7","8.0","10.6","10.11"
  smoke_urls?: string[],             // additional URLs to test beyond defaults
  preserve_until?: timestamp         // how long to keep the rehearsal stack
}
→ 202 {rehearsal_id}

Data model

create table rehearsals (
  id uuid primary key default gen_random_uuid(),
  site_uuid uuid not null references sites(uuid),
  source_version_id uuid references site_versions(id),
  target jsonb not null,        -- {wp, php, mysql}
  status text not null,         -- queued | provisioning | upgrading | testing | reporting | ready | failed | promoted | discarded
  started_at timestamptz default now(),
  finished_at timestamptz,
  report jsonb,                 -- full report
  rehearsal_hostname text,
  rehearsal_port int,
  compose_project text,
  expires_at timestamptz,
  error text
);

Flow (state machine)

queued
  ↓
provisioning   (allocate port, compose project, image-pin php/mysql to target,
                fresh stack with current site state imported)
  ↓
upgrading      (steps run inside the rehearsal stack:
                1. Take baseline snapshot inside the stack
                2. wp core update --version={target_wp}
                3. wp core update-db
                4. wp plugin update --all
                5. Each step's output captured)
  ↓
testing        (run smoke suite — see below)
  ↓
reporting      (compile report JSON, render PDF/markdown)
  ↓
ready          (operator can browse the rehearsal at rehearsal_hostname, promote, or discard)

Smoke test suite

Implemented as Python script api/services/rehearsal/smoke.py. Each check returns {name, status: pass|warn|fail, evidence}.

  1. HTTP homepage 200: GET / returns 200, < 5 s.
  2. HTTP admin 200: GET /wp-admin/index.php (with auth) returns 200.
  3. No PHP fatals: tail wp-content/debug.log and php-fpm logs, scan for Fatal error|Parse error.
  4. Plugin parity: wp plugin list --status=active --format=json before & after — same plugins active.
  5. Theme parity: same theme active.
  6. Database integrity: wp db check returns OK.
  7. Cron parity: same number of registered cron events ± 5%.
  8. Top-5 pages 200: pull the 5 most-trafficked URLs from a small in-app counter, hit each.
  9. Visual diff homepage (vs baseline rehearsal screenshot): ratio ≤ 10%.
  10. Custom smoke URLs: each URL the operator passed in must return 200.

Each check has a criticality: critical (fail aborts), warning (fail surfaces but continues).

Report

{
  "summary": {"status": "pass" | "warn" | "fail", "checks_total": 10, "checks_pass": 9, "checks_warn": 1, "checks_fail": 0},
  "target": {"wp": "6.4.7", "php": "8.1", "mysql": "10.6"},
  "source": {"wp": "4.6.17", "php": "7.4", "mysql": "5.7"},
  "checks": [
    {"name": "http_homepage", "status": "pass", "duration_ms": 1230, "evidence": "200"},
    {"name": "no_php_fatals", "status": "warn", "evidence": "1 deprecation: Function each() is deprecated in plugin redux-framework/redux-core/inc/extensions/wbc_importer/extension_wbc_importer.php:158"},
    ...
  ],
  "upgrade_log": [
    {"step": "wp core update", "exit_code": 0, "stdout": "...", "stderr": ""},
    ...
  ],
  "visual_diff_baseline": "<base64 png>",
  "visual_diff_after": "<base64 png>",
  "visual_diff_ratio": 0.043
}

Promote

  • POST /api/v1/sites/{slug}/rehearsals/{id}/promote →
  • Save a version of the current (pre-rehearsal) state on the main stack.
  • Copy the rehearsal worktree contents over the main worktree.
  • Tear down the rehearsal stack.
  • Mark rehearsal promoted.
  • The operator now sees the upgraded state on the main stack, ready for normal push to prod via existing flow.

Critical: promote does NOT push to prod automatically. It promotes to the local stack only. The operator still uses the normal push-with-diff flow against prod. This keeps the existing safety net.

UI

  • New page /sites/[slug]/rehearse:
  • Form: target picker (WP version dropdown, PHP version dropdown, MySQL version dropdown, custom smoke URLs textarea).
  • "Start rehearsal" → progress view (SSE-driven, like Op Progress from F4 of ADR-0012).
  • When ready: render the report (collapsible per-check), embedded visual diff slider, link to the rehearsal hostname (with PocketID gate).
  • Two big buttons: Promote (primary if all checks pass) and Discard.
  • ActionSidebar button: "Rehearse upgrade" (lower priority, always visible).

Edge cases

  • Rehearsal stack runs out of memory on import (huge sites): allocate higher mem-limit for rehearsal stacks; document in settings.
  • Plugins that hard-die on PHP 8 (e.g. Visual Composer < 6) → smoke catches fatal, report shows it, operator knows to update VC before rehearsing again.
  • Upgrade-script timeouts: cap each wp command at 5 min; surface in log.
  • Rehearsal stack lifecycle: default preserve_until = now + 24h. Cron cleanup_rehearsals.py tears down. Promoted ones are torn down immediately on promote.
  • Concurrency: max 1 active rehearsal per site, max 2 server-wide.

Estimated LOC: ~2000 (largest single feature; counts both backend job orchestration and UI).

Consolidated data model changes

Alembic 012 — F1 + F2 columns

# F1.2 Notebook + F1.4 Quarantine
op.add_column('sites', sa.Column('notes_markdown', sa.Text(), nullable=False, server_default=''))
op.add_column('sites', sa.Column('notes_updated_at', sa.DateTime(timezone=True), nullable=True))
op.add_column('sites', sa.Column('quarantined_plugins', postgresql.JSONB(), nullable=False, server_default='[]'))

# F2 WP-CLI audit
op.create_table('wpcli_audit',
    sa.Column('id', postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text('gen_random_uuid()')),
    sa.Column('site_uuid', postgresql.UUID(as_uuid=True), sa.ForeignKey('sites.uuid', ondelete='CASCADE'), nullable=False),
    sa.Column('command', sa.Text(), nullable=False),
    sa.Column('exit_code', sa.Integer()),
    sa.Column('stdout_b64', sa.Text()),
    sa.Column('stderr_b64', sa.Text()),
    sa.Column('duration_ms', sa.Integer()),
    sa.Column('executed_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
)
op.create_index('idx_wpcli_audit_site_time', 'wpcli_audit', ['site_uuid', 'executed_at'])

Alembic 013 — F3 health

op.add_column('sites', sa.Column('mysql_version', sa.Text()))
op.add_column('sites', sa.Column('versions_last_seen_at', sa.DateTime(timezone=True)))
op.add_column('sites', sa.Column('ssl_expires_at', sa.DateTime(timezone=True)))
op.add_column('sites', sa.Column('ssl_last_checked_at', sa.DateTime(timezone=True)))
op.add_column('sites', sa.Column('ssl_issuer', sa.Text()))
op.add_column('sites', sa.Column('ssl_check_target', sa.Text(), nullable=False, server_default='auto'))
op.add_column('sites', sa.Column('db_size_bytes', sa.BigInteger()))

op.create_table('version_reference',
    sa.Column('key', sa.Text(), primary_key=True),
    sa.Column('value', postgresql.JSONB(), nullable=False),
    sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
)

op.create_table('site_metrics_daily',
    sa.Column('site_uuid', postgresql.UUID(as_uuid=True), sa.ForeignKey('sites.uuid', ondelete='CASCADE'), nullable=False),
    sa.Column('day', sa.Date(), nullable=False),
    sa.Column('db_size_bytes', sa.BigInteger()),
    sa.Column('file_size_bytes', sa.BigInteger()),
    sa.Column('plugin_count', sa.Integer()),
    sa.PrimaryKeyConstraint('site_uuid', 'day'),
)

Alembic 014 — F4 previews + diffs

op.create_table('version_previews',
    sa.Column('id', postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text('gen_random_uuid()')),
    sa.Column('site_uuid', postgresql.UUID(as_uuid=True), sa.ForeignKey('sites.uuid', ondelete='CASCADE'), nullable=False),
    sa.Column('version_id', postgresql.UUID(as_uuid=True), sa.ForeignKey('site_versions.id'), nullable=False),
    sa.Column('hostname', sa.Text(), unique=True, nullable=False),
    sa.Column('port', sa.Integer(), nullable=False),
    sa.Column('compose_project', sa.Text(), nullable=False),
    sa.Column('status', sa.Text(), nullable=False),
    sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
    sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
    sa.Column('destroyed_at', sa.DateTime(timezone=True)),
    sa.Column('error', sa.Text()),
)
op.create_index('idx_previews_status_expires', 'version_previews', ['status', 'expires_at'])

# diff jobs persisted only briefly; in-memory dict + Redis-style TTL acceptable.
# If a table is preferred:
op.create_table('version_diff_jobs',
    sa.Column('id', postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text('gen_random_uuid()')),
    sa.Column('site_uuid', postgresql.UUID(as_uuid=True), sa.ForeignKey('sites.uuid', ondelete='CASCADE')),
    sa.Column('version_a', postgresql.UUID(as_uuid=True)),
    sa.Column('version_b', postgresql.UUID(as_uuid=True)),
    sa.Column('status', sa.Text(), nullable=False),
    sa.Column('result', postgresql.JSONB()),
    sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
    sa.Column('finished_at', sa.DateTime(timezone=True)),
    sa.Column('error', sa.Text()),
)

Alembic 015 — F7 rehearsals

op.create_table('rehearsals',
    sa.Column('id', postgresql.UUID(as_uuid=True), primary_key=True, server_default=sa.text('gen_random_uuid()')),
    sa.Column('site_uuid', postgresql.UUID(as_uuid=True), sa.ForeignKey('sites.uuid'), nullable=False),
    sa.Column('source_version_id', postgresql.UUID(as_uuid=True), sa.ForeignKey('site_versions.id')),
    sa.Column('target', postgresql.JSONB(), nullable=False),
    sa.Column('status', sa.Text(), nullable=False),
    sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
    sa.Column('finished_at', sa.DateTime(timezone=True)),
    sa.Column('report', postgresql.JSONB()),
    sa.Column('rehearsal_hostname', sa.Text()),
    sa.Column('rehearsal_port', sa.Integer()),
    sa.Column('compose_project', sa.Text()),
    sa.Column('expires_at', sa.DateTime(timezone=True)),
    sa.Column('error', sa.Text()),
)

Security considerations

  • WP-CLI palette (F2): allowlist enforced before any subprocess call. All executions logged. The operator runs as wp-pulse user inside the container; can't reach the host filesystem. Treat as "I trust the operator with the local stack, but I want an audit trail".
  • Quarantine (F1.4): the mu-plugin can disable other plugins. Cannot disable the WP-Pulse agent (blocked at API level). The pushed file is signed by the existing Ed25519 chain, no new trust surface.
  • Time-travel preview (F4.2) + rehearsal (F7): the ephemeral stacks contain a real copy of customer data and are reachable via Caddy. Snippets reuse PocketID forward_auth from the existing preview infrastructure → only the operator (authenticated via SSO) reaches them. Hostnames include a random shortid to defeat enumeration.
  • Mass search-replace (F6): the destructive variant is canary'd + reversible per-site via rollback. The aggregate operation can be aborted mid-batch.
  • Daily crons (SSL check, version reference): outbound only; no inbound surface added.
  • CMD palette tool search injection: the palette renders user-typed strings only inside controlled DOM (no dangerouslySetInnerHTML). The wp-cli result is rendered in a <pre> with HTML-escaped content.

Rollout sequence

Recommended order (matches "build with confidence on previous proof"):

  1. Day 1: F1 (Ergonomía) + F2 (palette) + F3 (Fleet health) in parallel. 3 sub-agents, ADR-0012 pattern.
  2. Day 2: F4 (Recovery). Cross-version diff first (no new infra) then time-travel preview (validates ephemeral stack pattern needed for F7).
  3. Day 3: F5 (Media optimizer) — independent, can also run in parallel with day 2.
  4. Day 4: F6 (Mass search-replace) — depends on stack readiness primitives proven by F1.
  5. Day 5–7: F7 (Upgrade rehearsal) — biggest, uses everything above.

Each phase ends with a smoke test on hotelaldamagolf-com (the live customer site). Promotion of features to "default-on" is gated by that.

Open questions

  1. wp-cli palette destructive commands (cache flush, transient delete): ship with X-Confirm gate, or ship without and only allow read commands in v1? Recommended: read-only in v1; destructive needs F2.1 follow-up with the confirm modal.
  2. Time-travel preview vs production isolation: the rehearsal stack imports prod DB. Should it also block outbound traffic (so the cloned site can't send transactional emails)? Recommended yes — add network_mode: none for the rehearsal stack's outbound, route only the inbound through Caddy.
  3. Notebook history: skip in v1 or add a tiny append-only history table? Recommended skip; if needed, retroactively populate from updated_at deltas.
  4. Bulk media optimizer extension list: ship with WebP conversion default OFF (safe) or ON for PNG-no-alpha (storage win)? Recommended OFF; add explicit toggle and document the URL-rewrite caveat.
  5. Rehearsal report formats: render PDF or only markdown/HTML? Recommended markdown only; PDF if a real need surfaces.

Consequences

Positive: - Operator's daily session ritual collapses to one click. - Fleet drift is visible at-a-glance instead of latent. - Recovery becomes surgical, not nuclear. - Upgrading legacy sites becomes routine instead of risky. - Mass-edit operations stop being manual.

Negative / debt: - ~5 new alembic migrations (manageable; downgrade scripts mandatory). - More cron jobs to monitor (SSL check, metrics rollup, latest_versions sync, preview cleanup, rehearsal cleanup). Add to existing Healthchecks.io. - Storage growth: each ephemeral stack consumes 100–500 MB during life. Cleanup cron must be reliable. - Increased blast radius if the rehearsal stack misroutes — Caddy snippet must always include the PocketID gate.

References

  • ADR-0010 WP-Pulse phases F0–F6
  • ADR-0011 Snapshot model consolidation
  • ADR-0012 Web UI redesign
  • Brainstorm (in-session 2026-05-30): 45 ideas → 13 selected
  • cmdk library: https://cmdk.paco.me/
  • WP.org core version-check API: https://api.wordpress.org/core/version-check/1.7/