Skip to content

E6 — Cross-Cutting E2E + Fix-Wave Verification (LXC 281)

Date: 2026-05-31 Target: http://192.168.0.211:8080 (Docker-in-LXC, wp-pulse-server.service) Site under test: hotelaldamagolf-com Branch verified: Fix-Wave (R6 + ADR-0013 + ADR-0017 hardening through 2026-05-31) Outcome: 24 / 25 PASS, 1 PARTIAL, 0 FAIL.

Summary

This run executed the full 25-step cross-cutting matrix against the live control-plane at LXC 281. Every Fix-Wave concern (steps 12–25) passed cleanly. The single non-perfect result is step 7: the ⌘K command palette indexes 15 of 18 advertised site-action routes — manifest, snapshots, and the per-site /settings page exist as Next.js routes (/web/app/sites/[slug]/{manifest,snapshots,settings}) but are not present in components/app/CommandPalette.tsx. Fleet-level nav exposes 8 entries (≥6 required), so step 8 passes.

The headline Fix-Wave verifications all hold:

  • Alembic 028 (head) confirmed (step 12).
  • rehearsals.source_version_id FK is ON DELETE SET NULL (confdeltype='n', step 13).
  • Two partial indexes present (ix_pending_db_changes_site_created_partial, ix_pending_user_actions_site_queued_partial, step 14).
  • Security preview-first contract holds end-to-end: POST /login-failures/block and POST /2fa-enforce both wrote to worktree / pending_user_actions and did not create a pushes_v2 row (pushes count stayed at 26 across the call, steps 15–17, 19–20).
  • TOCTOU subprocess cap enforced under 5-way concurrency: 2×202 + 3×429 with distinct ports 9100/9101/9102 (3rd slot already occupied by stale row; new POSTs filled the gap then rejected). No port collisions, no half-baked rows (step 21).
  • Pull completed in ~99 s end-to-end (well under 3 min timeout, step 22).
  • CSRF on migration ZIP import rejects cross-Origin POST with 403 "Forbidden origin" (step 23).
  • /api/v1/imports prefix collision fixed — the new path serves a 200 with the job list, and the old /api/v1/sites/_all-imports is 404 (steps 10–11).
  • Symlink /srv/wp-pulse/api → /opt/wp-pulse-server/api verified (step 25). The api.bak.20260531/ and alembic.bak.20260531/ shadow dirs sit alongside, confirming the morning's symlink cutover.
  • Unified pending-changes includes all three categories (files, quarantine, db) and apply-all-to-prod populates pre_apply_version_id on every push row with a log_tail describing the DB ops — the main Fix-Wave bug (db category missing from the response) is resolved (steps 1–5).
  • formatError / errors.ts deployed across the UI surface (10 components including QuarantineModal.tsx, PendingChangesModal.tsx, OptionsTable.tsx, step 24).

One side-effect from the parallel-preview test (step 21) — a stale "running" row with a caddy-deploy partial error from a prior session was visible at the start; a follow-up DELETE cleared it. The TOCTOU fix is therefore validated against an already-saturated queue, not just a cold start.

Cleanup left the site at total_count=0 across all three pending categories, zero active previews, and the smoke quarantine entry removed from both preview and prod lists. Two synthetic pending_db_changes rows (the htaccess_rule/create entries created by steps 15+17) and one pending_user_actions row (step 19's 2FA nudge) were deleted directly from postgres since /discard is scoped to worktree + quarantine column only.

Per-item results

Unified pending-changes + apply-all-to-prod (ADR-0013 R6)

# Step Result Notes
1 GET pending-changes returns files, quarantine, db PASS All 3 keys present. db.count=4 at baseline (pre-existing).
2 POST /quarantine smoke-e6/x.php preview then pending-changes shows drift PASS quarantine.count=1, to_add=["smoke-e6/x.php"].
3 db.count >= 1 and breakdown surfaces it PASS db.count=6, breakdown lists htaccess_rules.added=1, two_fa_nudges.queued=1, users, options.
4 apply-all-to-prod queues all 3 categories PASS Queued: quarantine (64bb23f40a0f), db (d777f4ab9623); skipped: files (nothing pending).
5 pre_apply_version_id on response + every push row PASS Both pushes_v2 rows show pre_apply_version_id = e6f14481-.... db row log_tail = db-only push; 5 db_change row(s), 1 user_action row(s): htaccess_rule/create 192.0.2.99, user/create 3, user/delete 3, option/update blogname, option/update blogname, two_fa_nudge.
6 POST /discard → all 3 categories zero PASS total_count=0 post-discard. (Quarantine reconciled into prod via apply, so no drift, not a bug.)

⌘K Command palette + SiteHero (R6 I1)

# Step Result Notes
7 18 site-action route refs in palette PARTIAL 15/18 found in CommandPalette.tsx: plugins, themes, users, posts, options, config, db, security, performance, staging, approvals, reports, feedback, status-page, rehearse. Missing: settings (per-site), manifest, snapshots. Pages exist on disk. Tracked as palette-index gap, not a blocker.
8 6 fleet-level nav entries PASS 8 unique fleet paths in palette: /fleet, /fleet/tools, /fleet?tab={activity,batches,fleet_health,performance,schedules,security}.
9 3 cluster headings (Content / Config / Ops) + ≥ 15 chip labels in SiteHero PASS All three headings present in SiteHero.tsx; 17 chip labels.

Imports prefix collision (R6 I3)

# Step Result Notes
10 GET /api/v1/imports → 200 list PASS Returns {jobs:[…]} with one historical failed import.
11 GET /api/v1/sites/_all-imports → 404 (old path gone) PASS 404 "site not found" — handled by sites router treating _all-imports as a slug.

Alembic + DB constraints (Fix-Wave F4)

# Step Result Notes
12 alembic current → 028 (head) PASS Exact match.
13 rehearsals.source_version_id FK = confdeltype='n' (SET NULL) PASS rehearsals_source_version_id_fkey returned n.
14 2 partial indexes present PASS ix_pending_db_changes_site_created_partial + ix_pending_user_actions_site_queued_partial. (One unrelated continuous_agg_partial_view_…_key belongs to TimescaleDB.)

Security preview-first (ADR-0017 F1)

# Step Result Notes
15 POST /login-failures/block returns appended_to_worktree:true, pending_logged:true PASS First call IP 192.0.2.50, second IP 192.0.2.51. Both 200.
16 No new pushes_v2 row PASS Count stayed at 26 before AND after the call.
17 .htaccess in worktree has Deny line PASS /srv/wp-pulse/sites/hotelaldamagolf-com/_stack/hotelaldamagolf-com/www/wp-content/.htaccess contains Deny from 192.0.2.51 with the WP-Pulse marker comment. (First attempt raced with a concurrent discard; second call landed cleanly.)
18 POST /discard restores PASS 200, worktree reset to HEAD.
19 POST /2fa-enforce {"user_ids":[1]} returns pending_logged:true PASS queued:1, pending_action_id:34ef4fb7-….
20 pending_user_actions row with action=two_fa_nudge, applied_at IS NULL PASS Row confirmed via psql; applied_at empty.

Subprocess hardening + TOCTOU (Fix-Wave F2)

# Step Result Notes
21 5 parallel previews → cap=3 enforced, distinct ports PASS 2×202 (preview_id 4517…, 536d…) + 3×429. Ports allocated: 9100 (pre-existing stale), 9101, 9102 — all distinct, no collision. Tear-down via DELETE returned 204 for all three.
22 Pull completes ≤ 3 min PASS 5bb61471-… reached DONE in 99 s (states: COMMANDS_ISSUED → STREAMING_FILES → RESTORING → DONE).

CSRF + frontend (Fix-Wave F3)

# Step Result Notes
23 Cross-Origin POST /api/internal/migration/import-zip → 403 PASS {"error":"Forbidden origin"} via requireSameOrigin() in web/app/api/internal/migration/import-zip/route.ts.
24 errors.ts / formatError deployed PASS Used in 10 components: QuarantineModal, PendingChangesModal, RevertModal, DbHealthCard, DnsSetupStep, FleetDashboard, DbQueryConsole, WpConfigEditor, OptionsTable, ImportZipDropzone.
# Step Result Notes
25 /srv/wp-pulse/api symlink to /opt/wp-pulse-server/api PASS lrwxrwxrwx … /srv/wp-pulse/api -> /opt/wp-pulse-server/api. Same applies to /srv/wp-pulse/alembic. api.bak.20260531 and alembic.bak.20260531 snapshots sit beside the symlinks.

Bugs / observations

  1. Palette indexing gap (step 7, minor). CommandPalette.tsx is missing 3 site-action entries that have backing pages: manifest, snapshots, and per-site settings. Recommend adding three more Command.Item rows in the per-site cluster in web/components/app/CommandPalette.tsx.

  2. Discard scope (observation, not a bug). POST /discard resets worktree + reconciles sites.quarantined_plugins, but does not touch pending_db_changes or pending_user_actions. This is by design (those rows are intended-future-actions awaiting an apply), but operators running ad-hoc smoke tests should know they need to either apply-all-to-prod or delete from psql to fully zero the breakdown. Documentation candidate.

  3. DELETE quarantine plugin_path parsing (minor). DELETE /api/v1/sites/{slug}/quarantine/{plugin_path:path} with a slug containing a / (e.g. smoke-e6/x.php) only works when the slash is URL-encoded as %2F when targeting target=prod. The unencoded path was parsed correctly for the preview list but produced a 404 against the prod list on the first attempt. Worth a small consistency note in the OpenAPI summary.

  4. Stale preview row at start of step 21 (observation). The fleet preview-list contained a 1-hour-old row in status=running with destroyed_at already set and a caddy-deploy partial error. This means the periodic sweeper in health_cron either hasn't run since the failure or the sweeper isn't covering rows whose docker side was torn down out-of-band. Worth a follow-up.

  5. appended_to_worktree:true returned before file exists (step 17, race). The first call to POST /login-failures/block returned appended_to_worktree:true but the .htaccess file was absent from disk a few seconds later — likely because a concurrent discard racing in another tab wiped the worktree. The response is technically truthful (the write succeeded at the moment), but operators chaining block → inspect-worktree could see a transient absence. Acceptable as-is; documenting for the operator runbook.

Cleanup performed

  • POST /discard on hotelaldamagolf-com (final).
  • DELETE /api/v1/sites/hotelaldamagolf-com/quarantine/smoke-e6%2Fx.php?target=prod → both lists empty.
  • DELETE on all 3 parallel-preview rows from step 21.
  • psql DELETE of 2 synthetic pending_db_changes rows (kind=htaccess_rule, created during steps 15+17) + 1 pending_user_actions row (id 34ef4fb7-…, from step 19).
  • Final GET pending-changes: total_count=0, all three categories empty, quarantine.preview_list=[], quarantine.prod_list=[].
  • Final GET /api/v1/version-previews: {previews:[]}.

Methodology notes

  • Auth header: X-Admin-Token: <token> (not Authorization: Bearer). The require_admin_token dep in api/deps/session.py reads the X-Admin-Token header (with constant-time comparison) and supports an internal-only X-WPP-Internal-Call bypass for the ASGITransport loopback path.
  • API reached over plaintext HTTP at localhost:8080 from inside LXC 281 (the public ingress is via Caddy LXC 270/271 → Cloudflare → wp-pulse-app.monxas.casa).
  • All psql probes executed as postgres user inside the postgres container's host context (Postgres listens locally on the LXC). Database name: wp_pulse.
  • The tests/ directory in the repo would normally exercise much of this automatically; this run was a live-system smoke against the deployed binary.