Build optimization: pre-built PM3 binaries, ARM64 CI, base image caching

Replace PM3 compile-from-source in pi-gen with pre-built tarball extraction (saves 43-58 min). Merge stagePM3 into stageDangerousPi as 02-pm3-install substage, renumber all subsequent substages. Switch CI PM3 build to native ARM64 runner (ubuntu-24.04-arm64) eliminating QEMU overhead. Add weekly base-image workflow for pre-baking stages 0-2. Support PM3_TARBALL, BASE_IMAGE, and APT_PROXY env vars in build-image.sh. Also includes prior Phase 5 work: theme system, design system integration, component update system, OS updates, CI build pipeline, and test results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 12:01:01 -08:00
parent 2ec89041ef
commit a9acdb85ce
163 changed files with 8124 additions and 921 deletions
--- a/docs/COMPONENT_UPDATE_SYSTEM.md
+++ b/docs/COMPONENT_UPDATE_SYSTEM.md
@@ -0,0 +1,339 @@
+# Component-Based Update System
+
+## Context
+
+The current update system is monolithic — a single `.tar.gz` downloaded from GitHub Releases, extracted to `/opt/dangerous-pi`, followed by `make clean && make` to rebuild PM3 on-device. This is slow, resource-heavy, and fragile on a Pi. The goal is to:
+
+1. Ship **pre-built PM3 binaries** (cross-compiled in CI) instead of compiling on-device
+2. Support **per-component updates** (update just the frontend, just the PM3 client, etc.)
+3. Have the update system **auto-manage requirements** like Python version matching for SWIG `.so` files
+
+## Component Model
+
+Four independently-updateable components. 3rd-party themes use the plugin system instead.
+
+| Component | Artifacts | Install Path | Platform Constraints |
+|-----------|-----------|-------------|---------------------|
+| `pm3` | `proxmark3` binary, `libpm3rrg_rdv4.so`, `_pm3.so`, `pm3.py`, `pm3-flash-*`, patches, `bootrom.elf`, `fullimage.elf` | `$HOME/.pm3/proxmark3/` | aarch64-linux, Python version must match |
+| `frontend` | Remix build output (`build/client/`, `build/server/`), app components, routes | `/opt/dangerous-pi/app/frontend/build/` | None |
+| `backend` | Python source, `requirements.txt` | `/opt/dangerous-pi/app/backend/` | Python 3.11+ |
+| `theme` | Generated token CSS from `@dangerousthings/tokens` + `themes.json` registry | `/opt/dangerous-pi/app/frontend/themes/` | None |
+
+The PM3 client and firmware are always distributed together — they're built from the same source tree with the same patches, and a version mismatch between client and firmware could cause flashing issues.
+
+### Theme Distribution (two channels)
+
+**Source of truth:** The `dt-design-system` monorepo at `/home/work/dt-design-system/` (Turborepo + npm workspaces).
+
+**Monorepo packages:**
+- `@dangerousthings/tokens` — TypeScript token definitions → generates `dist/css/{dt,classic,supra}.css`
+- `@dangerousthings/web` — Structural CSS (bevels, glows, elevation, fonts) + imports token CSS
+- `@dangerousthings/react-native` — RN themed components (DTButton, DTCard, etc.)
+
+**Currently:** The frontend imports `@dangerousthings/web/dist/index.css` which bundles *everything* (tokens + structural CSS) at Remix build time. This means a theme color change requires a full frontend rebuild.
+
+**New approach:** Split token CSS (brand colors, typography, shapes) from structural CSS (bevels, glows, elevation):
+- **Structural CSS** stays bundled with the frontend build (changes rarely, tied to component code)
+- **Token CSS** (`dt.css`, `classic.css`, `supra.css`) is served dynamically from a `themes/` directory and loaded via `<link>` tags — can be updated without rebuilding the frontend
+
+**Built-in themes** (dt, classic, supra) are distributed as the `theme` component via the update system. They ship with tested releases and are always available.
+
+**3rd-party themes** are distributed as plugins via the existing plugin system. A theme plugin provides token CSS + `theme.json` and registers via the `theme_register` plugin hook.
+
+Both channels write to the same dynamic theme registry, which replaces the current hardcoded `themes[]` array in `ThemeContext.tsx`.
+
+**Theme package structure** (same format for both channels):
+```
+{theme-id}/
+  theme.json          # id, name, description, supportsModes, defaultMode, author
+  tokens.css          # CSS custom properties (--color-*, --radius-*, --font-*, --shape-*)
+```
+
+## Implementation Phases
+
+### Phase 1: Manifest System (Foundation)
+
+Add component tracking without changing update behavior yet.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py` — Add dataclasses and manifest I/O
+
+**New dataclasses:**
+```python
+class ComponentId(str, Enum):
+    PM3 = "pm3"
+    FRONTEND = "frontend"
+    BACKEND = "backend"
+    THEME = "theme"
+
+@dataclass
+class ComponentInfo:
+    component_id: str
+    version: str
+    installed_at: str
+    checksum: Optional[str] = None
+    install_path: str = ""
+    platform: Optional[str] = None
+    python_version: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+@dataclass
+class SystemManifest:
+    schema_version: int = 1
+    system_version: str = "0.1.0"
+    components: Dict[str, ComponentInfo] = field(default_factory=dict)
+    last_updated: str = ""
+```
+
+**New file:** `/opt/dangerous-pi/component-manifest.json` — replaces the single `VERSION` file.
+
+**Migration:** On first startup with no manifest, `_migrate_from_legacy()` reads the existing `VERSION` file and PM3 `VERSION.txt` to populate a manifest with current component versions.
+
+---
+
+### Phase 2: CI/CD Build Pipeline
+
+Cross-compile PM3 and package all components in GitHub Actions.
+
+**New files:**
+- `.github/workflows/build-release.yml` — Main workflow, triggered on tag push (`v*`)
+- `ci/build-pm3.sh` — Runs in aarch64 Docker container (QEMU), reproduces the build from `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`:
+  - Clone RfidResearchGroup/proxmark3
+  - Apply `led-pwm-control.patch` + `hf-booster-detection.patch`
+  - Apply DangerousPi branding
+  - Set `PLATFORM=PM3GENERIC`, `LED_ORDER=PM3EASY`
+  - Build firmware (`bootrom.elf`, `fullimage.elf`) + client + SWIG bindings in one pass
+  - `cmake -DBUILD_PYTHON_LIB=ON`, SWIG bindings
+  - Matrix build for Python 3.11 + 3.12
+- `ci/build-frontend.sh` — `npm ci && npm run build`, package `build/`
+- `ci/build-backend.sh` — Package `app/backend/` + `requirements.txt`
+- `ci/build-theme.sh` — Builds `@dangerousthings/tokens` in the `dt-design-system` monorepo, packages the generated `dist/css/{dt,classic,supra}.css` + a `themes.json` registry
+- `ci/generate-manifest.py` — Generates `release-manifest.json` with checksums
+
+**Release asset naming:**
+```
+release-manifest.json
+pm3-{ver}-aarch64-linux-cp312.tar.gz
+pm3-{ver}-aarch64-linux-cp311.tar.gz
+frontend-{ver}.tar.gz
+backend-{ver}.tar.gz
+theme-{ver}.tar.gz
+```
+
+**`release-manifest.json` structure:**
+```json
+{
+  "schema_version": 1,
+  "release_version": "0.2.0",
+  "components": {
+    "pm3": {
+      "version": "4.19544-dp1",
+      "assets": {
+        "aarch64-linux-cp312": {
+          "filename": "pm3-4.19544-dp1-aarch64-linux-cp312.tar.gz",
+          "checksum_sha256": "...",
+          "size": 12345678,
+          "python_version": "3.12"
+        }
+      },
+      "changelog": "..."
+    }
+  }
+}
+```
+
+---
+
+### Phase 3: Component-Aware Update Logic
+
+Refactor `UpdateManager` to download/install individual components.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py`
+
+**Key changes:**
+1. `_fetch_latest_release()` → fetch and parse `release-manifest.json` from GitHub release assets (fall back to legacy `.tar.gz` scanning for old releases)
+2. `_select_pm3_asset()` → auto-detect Pi's Python version (`sys.version_info`) and architecture (`platform.machine()`) to pick the correct pm3 variant
+3. `check_for_updates()` → return per-component availability with compatibility info
+4. New `download_component(component_id)` → download a single component tarball
+5. New `install_component(component_id)` → component-specific install logic:
+   - `pm3`: copy client + firmware to `$HOME/.pm3/proxmark3/`, create `_pm3.so` symlink, verify arch
+   - `frontend`: replace `build/` directory
+   - `backend`: replace source, run `pip install -r requirements.txt` in venv
+   - `theme`: extract to `/opt/dangerous-pi/app/frontend/themes/`, update theme registry
+6. **Delete `_rebuild_pm3_client()`** — no more on-device compilation
+7. New `_backup_component()` / `rollback_component()` → per-component backups at `/opt/dangerous-pi/backups/{component_id}/`
+
+**New status:** `ROLLING_BACK` added to `UpdateStatus` enum.
+
+**New progress tracking:**
+```python
+@dataclass
+class ComponentProgress:
+    component_id: str
+    status: UpdateStatus
+    download_progress: float = 0.0
+    error_message: Optional[str] = None
+```
+
+Added as `component_progress: Dict[str, ComponentProgress]` to `UpdateProgress`.
+
+---
+
+### Phase 4: Unified Plugin Update Checking
+
+Integrate plugin update checks into the same update flow so "Check for Updates" covers everything.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py` — Add plugin update checking alongside component checks
+- `app/backend/managers/plugin_manager.py` — Add `check_plugin_updates()` method that queries each installed plugin's GitHub repo for newer releases
+
+**How it works:**
+1. `check_for_updates()` calls both:
+   - Core component check (from `release-manifest.json` on the main repo)
+   - Plugin update check (queries each plugin's source repo for newer tags)
+2. Plugin update info is returned alongside component updates in the same response
+3. Plugin download/install still uses the existing `plugin_manager.py` install logic — the update manager just orchestrates the version check
+4. Plugin versions are tracked in `component-manifest.json` under a `plugins` key (separate from `components`)
+
+**Manifest extension:**
+```json
+{
+  "components": { ... },
+  "plugins": {
+    "hello_world": {
+      "version": "1.0.0",
+      "source_repo": "dangerous-tacos/dp-plugin-hello-world",
+      "installed_at": "2026-03-03T10:00:00Z"
+    }
+  }
+}
+```
+
+**Update check response includes plugins:**
+```python
+class PluginUpdateInfo(BaseModel):
+    plugin_id: str
+    current_version: str
+    available_version: str
+    changelog: str
+    download_size: Optional[int]
+```
+
+The `/api/updates/check` response adds `plugins: List[PluginUpdateInfo]` alongside `components`.
+
+---
+
+### Phase 5: API & Service Layer (includes plugin update endpoints)
+
+**Files to modify:**
+- `app/backend/services/update_service.py` — Add `download_component()`, `install_component()`, `rollback_component()`, `get_installed_components()`
+- `app/backend/api/updates.py` — New endpoints + extended response models
+
+**New endpoints:**
+```
+GET  /api/updates/components                    — installed component manifest
+POST /api/updates/components/{id}/download      — download one component
+POST /api/updates/components/{id}/install       — install one component
+POST /api/updates/components/{id}/rollback      — rollback one component
+```
+
+**Modified endpoints (backward-compatible):**
+- `GET /check` — adds `components: List[ComponentUpdateInfo]` to response
+- `POST /download` — accepts optional `{"components": ["frontend"]}` body; no body = all
+- `POST /install` — same optional body pattern
+- `GET /progress` — adds `active_component` and `components` list to response
+
+---
+
+### Phase 5: Dynamic Theme Registry
+
+Decouple token CSS from the frontend build so themes can be updated independently.
+
+**Current flow (bundled):**
+`@dangerousthings/web/dist/index.css` → imports token CSS + structural CSS → bundled by Vite into Remix build
+
+**New flow (split):**
+- Structural CSS (`base.css`, `bevels.css`, `glows.css`, `elevation.css`, fonts) stays in the Remix build via `@dangerousthings/web`
+- Token CSS (`dt.css`, `classic.css`, `supra.css`) served from `/themes/{brand}/tokens.css` and loaded dynamically
+
+**Files to modify:**
+- `app/frontend/app/lib/ThemeContext.tsx` — Replace hardcoded `themes[]` array with dynamic loading from `GET /api/themes`; inject `<link>` tag for active brand's token CSS
+- `app/frontend/app/root.tsx` — Remove token CSS from the static import of `@dangerousthings/web/dist/index.css`; keep only structural CSS. Add dynamic `<link>` for token CSS.
+- `app/backend/api/system.py` — Add `GET /api/themes` endpoint that reads available themes from disk
+- `app/backend/main.py` — Mount `/themes/` as a static file directory
+
+**Files to create:**
+- `app/frontend/themes/` — Directory for installed theme packages (in dev)
+- `/opt/dangerous-pi/app/frontend/themes/` — Same, in production
+- Each theme: `themes/{theme-id}/theme.json` + `tokens.css`
+
+**How it works:**
+1. Backend scans `themes/` directory + plugin-registered themes → serves merged list at `GET /api/themes`
+2. `ThemeContext` fetches available themes on mount (with hardcoded fallback for SSR/offline)
+3. On brand switch, `ThemeContext` updates `<link href="/themes/{brand}/tokens.css">` in the document head
+4. Plugin themes register via `theme_register` hook, providing their CSS path + `theme.json`
+
+**Migration from `@dangerousthings/web`:**
+- `@dangerousthings/web` needs a new export that excludes token CSS: `@dangerousthings/web/dist/structural.css` (or the `index.css` is refactored to not import token CSS)
+- The CI `build-theme.sh` step copies `@dangerousthings/tokens/dist/css/{dt,classic,supra}.css` into the theme tarball as `{brand}/tokens.css`
+
+**Theme plugin hook** (added to plugin_manager.py):
+```python
+# Plugin calls this in on_enable():
+self.register_hook("theme_register", self._register_theme)
+
+def _register_theme(self):
+    return {
+        "id": "my-theme",
+        "name": "My Custom Theme",
+        "css_path": self.plugin_dir / "tokens.css",
+        "theme_json": self.plugin_dir / "theme.json"
+    }
+```
+
+---
+
+### Phase 6: Frontend UI
+
+**File to modify:**
+- `app/frontend/app/routes/updates.tsx`
+
+**Layout:**
+1. **System overview card** — system version, last check, "Check for Updates" + "Update All" buttons
+2. **Component cards** — one per component showing:
+   - Name, installed version, available version
+   - Compatibility badge (for pm3 Python version match)
+   - Collapsible per-component changelog
+   - Individual Download / Install / Rollback buttons
+   - Per-component progress bar
+3. **Plugin update cards** — shown for any installed plugins with updates available, using the same card layout. Download/install delegates to the existing plugin manager.
+4. **Status states per card:** up-to-date (muted), available (accent), downloading (progress bar), installing (spinner), complete (success), failed (danger + rollback button), incompatible (warning + reason)
+
+---
+
+### Phase 7: Testing & Polish
+
+- Test legacy migration (no manifest → auto-generated manifest)
+- Test partial update (update only frontend, leave PM3 alone)
+- Test Python version mismatch (pm3-client shows incompatible)
+- Test rollback per component
+- Test disk space check before download
+- Test backward compat with a legacy monolithic release (no `release-manifest.json`)
+- Test theme update (update `theme` component without touching frontend)
+- Test 3rd-party theme plugin (install, register, appear in theme selector, uninstall)
+- Test plugin update check (installed plugin with newer release shows in update check response)
+- Test plugin update install (update delegates to plugin_manager install logic)
+
+## Verification
+
+1. **Manifest migration**: Start backend with no `component-manifest.json` — verify it auto-creates from `VERSION` files
+2. **CI pipeline**: Push a test tag — verify all component tarballs + `release-manifest.json` are produced
+3. **Component check**: `GET /api/updates/check` — verify per-component version comparison
+4. **Selective update**: Download + install only `frontend` — verify only frontend files change
+5. **PM3 binary**: Install `pm3-client` — verify binary runs, `_pm3.so` symlink works, no compilation triggered
+6. **Rollback**: Install a component, then `POST /rollback` — verify files restored and manifest updated
+7. **Python version matching**: On a Pi with Python 3.12, verify `cp312` asset is selected; `cp311` shown as incompatible
+8. **Theme update**: Update `theme` component — verify new CSS applied without frontend rebuild
+9. **Theme plugin**: Install a theme plugin — verify it appears in theme selector alongside built-in themes
+10. **Plugin updates**: Install a plugin, publish a newer release to its repo — verify `GET /api/updates/check` includes it in the response and install works via existing plugin manager
--- a/docs/ON_DEVICE_TESTING.md
+++ b/docs/ON_DEVICE_TESTING.md
@@ -0,0 +1,330 @@
+# On-Device Testing & Iteration Guide
+
+How to build, deploy, and test Dangerous Pi components on a live Pi Zero 2 W from a development laptop.
+
+## Prerequisites
+
+- **Pi connectivity**: WiFi (`dangerous-pi.local` / IP) or Ethernet
+- **Serial console**: USB-to-serial adapter at `/dev/ttyUSB0` (115200 baud)
+- **Docker**: With QEMU binfmt support for `linux/arm64`
+- **ARM toolchain**: `arm-none-eabi-gcc` (firmware only)
+- **Python venv**: `.venv/` with `httpx`, `pyserial`, `websockets`
+
+## Connectivity
+
+| Method | Address | Use |
+|--------|---------|-----|
+| HTTP API | `http://dangerous-pi.local:8000/api/` | Backend testing |
+| HTTP UI | `http://dangerous-pi.local/` | Frontend (via nginx) |
+| HTTPS | `https://dangerous-pi.local/` | If HTTPS enabled |
+| Serial | `/dev/ttyUSB0` @ 115200 | System commands, logs, deployment |
+| Bluetooth | Local adapter | BLE scan testing |
+
+---
+
+## 1. Cross-Compiling Proxmark3
+
+The PM3 has two components that must match versions: **firmware** (ARM7TDMI, runs on PM3 hardware) and **client** (aarch64, runs on Pi).
+
+### Source Setup
+
+```bash
+cd /home/work/dangerous-pi/.pm3-test/proxmark3
+
+# Keep in sync with upstream
+git fetch origin
+git pull origin master
+
+# Configure platform
+cat > Makefile.platform << 'EOF'
+PLATFORM=PM3GENERIC
+LED_ORDER=PM3EASY
+EOF
+```
+
+### Build Firmware (Native, Fast)
+
+Firmware targets ARM7TDMI (the PM3's own chip), so `arm-none-eabi-gcc` builds it natively on x86:
+
+```bash
+cd /home/work/dangerous-pi/.pm3-test/proxmark3
+SKIPQT=1 make -j$(nproc) bootrom fullimage
+```
+
+Output:
+- `bootrom/obj/bootrom.elf` (~10KB)
+- `armsrc/obj/fullimage.elf` (~370KB)
+
+Build time: ~1 minute on x86.
+
+### Build Client (Docker aarch64, Slow)
+
+The client runs on the Pi (aarch64), so we cross-compile using Docker with QEMU emulation:
+
+```bash
+# Build the Docker image (one-time)
+cd /home/work/dangerous-pi/.pm3-test
+docker build --platform linux/arm64 -t pm3-client-builder -f Dockerfile.pm3-client .
+
+# IMPORTANT: Clean x86 build artifacts first
+cd proxmark3
+make client/clean
+
+# Build the aarch64 client
+docker run --rm --platform linux/arm64 \
+  -v "$(pwd):/build/proxmark3" \
+  pm3-client-builder \
+  bash -c "cd /build/proxmark3 && SKIPQT=1 make client"
+```
+
+Output:
+- `client/proxmark3` (aarch64 ELF, ~5.7MB)
+
+Build time: ~25 minutes (QEMU emulation is slow).
+
+Verify architecture:
+```bash
+file client/proxmark3
+# Should show: ELF 64-bit LSB pie executable, ARM aarch64
+```
+
+### Dockerfile
+
+Located at `.pm3-test/Dockerfile.pm3-client`:
+
+```dockerfile
+FROM --platform=linux/arm64 debian:trixie-slim
+RUN apt-get update && apt-get install -y \
+    build-essential pkg-config libreadline-dev libbz2-dev \
+    liblz4-dev libgd-dev libjansson-dev libbluetooth-dev \
+    python3-dev swig cmake \
+    && rm -rf /var/lib/apt/lists/*
+WORKDIR /build/proxmark3
+```
+
+### Common Pitfalls
+
+- **Stale x86 objects**: If you built on the host first, `.o` files in the volume mount will be x86_64. Docker can't link them. Always `make client/clean` before the Docker build.
+- **Merge conflicts with LED patch**: When pulling upstream, our `cmdhw.c` and `pm3_cmd.h` changes may conflict. Resolve by keeping our `CMD_LED_CONTROL` define and `led` command while taking upstream's new code.
+
+---
+
+## 2. Deploying to Pi
+
+### Transfer Files (HTTP Server Method)
+
+From the laptop, serve the built files:
+
+```bash
+cd /home/work/dangerous-pi/.pm3-test/proxmark3
+python3 -m http.server 9999 --bind $(hostname -I | awk '{print $1}')
+```
+
+From the Pi (via serial console or SSH):
+
+```bash
+LAPTOP_IP=10.0.0.202  # adjust to your laptop's IP
+mkdir -p ~/pm3-update
+
+# Download client and firmware
+curl -o ~/pm3-update/proxmark3 http://$LAPTOP_IP:9999/client/proxmark3
+curl -o ~/pm3-update/fullimage.elf http://$LAPTOP_IP:9999/armsrc/obj/fullimage.elf
+curl -o ~/pm3-update/bootrom.elf http://$LAPTOP_IP:9999/bootrom/obj/bootrom.elf
+
+chmod +x ~/pm3-update/proxmark3
+```
+
+### Deploy Binaries
+
+```bash
+# Stop the service first
+sudo systemctl stop dangerous-pi
+
+# Backup current binaries
+cp ~/.pm3/proxmark3/client/proxmark3 ~/.pm3/proxmark3/client/proxmark3.bak
+cp ~/.pm3/proxmark3/firmware/fullimage.elf ~/.pm3/proxmark3/firmware/fullimage.elf.bak
+
+# Deploy new binaries
+cp ~/pm3-update/proxmark3 ~/.pm3/proxmark3/client/
+cp ~/pm3-update/fullimage.elf ~/.pm3/proxmark3/firmware/
+cp ~/pm3-update/bootrom.elf ~/.pm3/proxmark3/firmware/
+```
+
+### Flash Firmware
+
+```bash
+cd ~/.pm3/proxmark3
+./pm3-flash-all
+```
+
+After flashing, the PM3 resets. If it doesn't re-enumerate on USB (`/dev/ttyACM0`), physically unplug and replug the USB cable.
+
+### Restart Service
+
+```bash
+sudo systemctl start dangerous-pi
+```
+
+---
+
+## 3. Deploying Backend Changes
+
+For Python backend changes (e.g., `ble_manager.py`, `system.py`):
+
+### Quick Inline Fix (via serial sed)
+
+For small changes, use `sed -i` over serial:
+
+```bash
+# Example: fix a string match in system.py
+sudo sed -i 's/"subjectAltName"/"Subject Alternative Name"/' \
+  /opt/dangerous-pi/app/backend/api/system.py
+sudo systemctl restart dangerous-pi
+```
+
+### Full File Transfer (HTTP method)
+
+```bash
+# On laptop - serve the file
+cd /home/work/dangerous-pi
+python3 -m http.server 9999 --bind $(hostname -I | awk '{print $1}')
+
+# On Pi - download and deploy
+curl -o /tmp/ble_manager.py http://$LAPTOP_IP:9999/app/backend/managers/ble_manager.py
+sudo cp /tmp/ble_manager.py /opt/dangerous-pi/app/backend/managers/ble_manager.py
+sudo systemctl restart dangerous-pi
+```
+
+### Deploying nginx Config
+
+```bash
+# On Pi
+curl -o /tmp/dangerous-pi-https.conf http://$LAPTOP_IP:9999/nginx/dangerous-pi-https.conf
+sudo cp /tmp/dangerous-pi-https.conf /opt/dangerous-pi/nginx/
+sudo /opt/dangerous-pi/scripts/configure-nginx.sh
+sudo systemctl reload nginx
+```
+
+---
+
+## 4. Running the Hardware Test Suite
+
+The test suite at `test_hardware_2026-03-03.py` validates everything against the live Pi.
+
+### Setup
+
+```bash
+source .venv/bin/activate
+pip install httpx pyserial websockets
+```
+
+### Run All Tests
+
+```bash
+python test_hardware_2026-03-03.py
+```
+
+### Run Specific Phase
+
+```bash
+python test_hardware_2026-03-03.py --phase 1   # Regression (Jan 23 issues)
+python test_hardware_2026-03-03.py --phase 2   # Core health endpoints
+python test_hardware_2026-03-03.py --phase 3   # PM3 hardware
+python test_hardware_2026-03-03.py --phase 4   # WiFi manager
+python test_hardware_2026-03-03.py --phase 5   # HTTPS/SSL
+python test_hardware_2026-03-03.py --phase 6   # Auth & WebSocket
+python test_hardware_2026-03-03.py --phase 7   # Plugins
+python test_hardware_2026-03-03.py --phase 8   # BLE & UPS
+python test_hardware_2026-03-03.py --phase 9   # Frontend & performance
+```
+
+### Safe Mode (Skip Destructive Tests)
+
+```bash
+python test_hardware_2026-03-03.py --safe-only
+```
+
+Results are written to `TEST_RESULTS_2026-03-03_vN.md`.
+
+### Test Phases Overview
+
+| Phase | Tests | What It Covers |
+|-------|-------|----------------|
+| 1 | 4 | Regression of Jan 2026 issues (WiFi, UPS, BLE, Pydantic) |
+| 2 | 9 | Health, readiness, system info, config, response times |
+| 3 | 14 | PM3 device discovery, sessions, command execution |
+| 4 | 6 | WiFi status, scanning, saved networks |
+| 5 | 8 | SSL cert info, HTTPS access, cert scripts |
+| 6 | 10 | Auth enable/disable, token flow, WebSocket |
+| 7 | 10 | Plugin discover, load, enable, hooks, widgets |
+| 8 | 10 | BLE status/advertising, UPS battery/thresholds |
+| 9 | 8 | Frontend pages, response times, memory, CPU temp |
+
+---
+
+## 5. Useful Serial Console Commands
+
+```bash
+# Service management
+sudo systemctl status dangerous-pi
+sudo systemctl restart dangerous-pi
+journalctl -u dangerous-pi -f        # Follow logs
+
+# Check PM3
+ls /dev/ttyACM*                       # USB device present?
+dmesg | tail -20                      # USB errors?
+~/.pm3/proxmark3/client/proxmark3 -p /dev/ttyACM0 -c "hw version"
+
+# Check networking
+ip addr show wlan0
+nmcli connection show --active
+curl -s http://localhost:8000/api/health
+
+# Check SSL
+ls -la /opt/dangerous-pi/ssl/
+openssl x509 -in /opt/dangerous-pi/ssl/dangerous-pi.crt -text -noout | head -20
+
+# Check BLE
+journalctl -u dangerous-pi --no-pager | grep -i "ble\|bluetooth"
+
+# Check UPS
+journalctl -u dangerous-pi --no-pager | grep -i "ups\|pisugar\|battery"
+```
+
+---
+
+## 6. Known Issues & Workarounds
+
+### PM3 USB Enumeration After Flash
+
+After `pm3-flash-all`, the PM3 sometimes fails to re-enumerate on USB (dmesg shows `error -71`). Fix: physically unplug and replug the USB cable.
+
+### PM3 Client Exit Code 1
+
+When client and firmware versions don't match, `pm3` exits with code 1 (despite producing valid output). The `SubprocessPM3Worker` treats this as failure, putting output in the `error` field instead of `output`. Fix: ensure client and firmware are built from the same commit.
+
+### SWIG Library on Pi Zero
+
+Building the SWIG Python bindings (`cmake -DBUILD_PYTHON_LIB=ON`) can OOM on Pi Zero 2 W (512MB RAM). The pi-gen chroot build handles this with swap, but on-device rebuilds may fail. The `SubprocessPM3Worker` (used by default) doesn't need SWIG — it shells out to the `pm3` CLI.
+
+### SSL Directory Permissions
+
+`/opt/dangerous-pi/ssl/` must be readable by the `dt` user (backend runs as `dt`):
+```bash
+sudo chmod 755 /opt/dangerous-pi/ssl/
+```
+
+---
+
+## 7. Pi-gen vs Cross-Compilation
+
+| Aspect | pi-gen (image build) | Cross-compile (dev iteration) |
+|--------|---------------------|-------------------------------|
+| When | Building release images | Day-to-day development |
+| Where | Docker chroot (aarch64) | Laptop + Docker for client |
+| Speed | Full build: hours | Firmware: 1min, Client: 25min |
+| SWIG | Built in chroot | Not needed (SubprocessPM3Worker) |
+| LED Patch | Applied via `led-pwm-control.patch` | Maintained in git (merge conflicts) |
+| Deploy | Baked into `.img` | HTTP transfer + manual copy |
+
+The pi-gen pipeline (`stagePM3/01-proxmark3/00-run-chroot.sh`) builds everything inside an aarch64 chroot — firmware, client, and SWIG bindings. For development iteration, cross-compilation from the laptop is faster and doesn't require a full image rebuild.