Build optimization: pre-built PM3 binaries, ARM64 CI, base image caching

Replace PM3 compile-from-source in pi-gen with pre-built tarball extraction (saves 43-58 min). Merge stagePM3 into stageDangerousPi as 02-pm3-install substage, renumber all subsequent substages. Switch CI PM3 build to native ARM64 runner (ubuntu-24.04-arm64) eliminating QEMU overhead. Add weekly base-image workflow for pre-baking stages 0-2. Support PM3_TARBALL, BASE_IMAGE, and APT_PROXY env vars in build-image.sh. Also includes prior Phase 5 work: theme system, design system integration, component update system, OS updates, CI build pipeline, and test results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 12:01:01 -08:00
parent 2ec89041ef
commit a9acdb85ce
163 changed files with 8124 additions and 921 deletions
--- a/docs/COMPONENT_UPDATE_SYSTEM.md
+++ b/docs/COMPONENT_UPDATE_SYSTEM.md
@@ -0,0 +1,339 @@
+# Component-Based Update System
+
+## Context
+
+The current update system is monolithic — a single `.tar.gz` downloaded from GitHub Releases, extracted to `/opt/dangerous-pi`, followed by `make clean && make` to rebuild PM3 on-device. This is slow, resource-heavy, and fragile on a Pi. The goal is to:
+
+1. Ship **pre-built PM3 binaries** (cross-compiled in CI) instead of compiling on-device
+2. Support **per-component updates** (update just the frontend, just the PM3 client, etc.)
+3. Have the update system **auto-manage requirements** like Python version matching for SWIG `.so` files
+
+## Component Model
+
+Four independently-updateable components. 3rd-party themes use the plugin system instead.
+
+| Component | Artifacts | Install Path | Platform Constraints |
+|-----------|-----------|-------------|---------------------|
+| `pm3` | `proxmark3` binary, `libpm3rrg_rdv4.so`, `_pm3.so`, `pm3.py`, `pm3-flash-*`, patches, `bootrom.elf`, `fullimage.elf` | `$HOME/.pm3/proxmark3/` | aarch64-linux, Python version must match |
+| `frontend` | Remix build output (`build/client/`, `build/server/`), app components, routes | `/opt/dangerous-pi/app/frontend/build/` | None |
+| `backend` | Python source, `requirements.txt` | `/opt/dangerous-pi/app/backend/` | Python 3.11+ |
+| `theme` | Generated token CSS from `@dangerousthings/tokens` + `themes.json` registry | `/opt/dangerous-pi/app/frontend/themes/` | None |
+
+The PM3 client and firmware are always distributed together — they're built from the same source tree with the same patches, and a version mismatch between client and firmware could cause flashing issues.
+
+### Theme Distribution (two channels)
+
+**Source of truth:** The `dt-design-system` monorepo at `/home/work/dt-design-system/` (Turborepo + npm workspaces).
+
+**Monorepo packages:**
+- `@dangerousthings/tokens` — TypeScript token definitions → generates `dist/css/{dt,classic,supra}.css`
+- `@dangerousthings/web` — Structural CSS (bevels, glows, elevation, fonts) + imports token CSS
+- `@dangerousthings/react-native` — RN themed components (DTButton, DTCard, etc.)
+
+**Currently:** The frontend imports `@dangerousthings/web/dist/index.css` which bundles *everything* (tokens + structural CSS) at Remix build time. This means a theme color change requires a full frontend rebuild.
+
+**New approach:** Split token CSS (brand colors, typography, shapes) from structural CSS (bevels, glows, elevation):
+- **Structural CSS** stays bundled with the frontend build (changes rarely, tied to component code)
+- **Token CSS** (`dt.css`, `classic.css`, `supra.css`) is served dynamically from a `themes/` directory and loaded via `<link>` tags — can be updated without rebuilding the frontend
+
+**Built-in themes** (dt, classic, supra) are distributed as the `theme` component via the update system. They ship with tested releases and are always available.
+
+**3rd-party themes** are distributed as plugins via the existing plugin system. A theme plugin provides token CSS + `theme.json` and registers via the `theme_register` plugin hook.
+
+Both channels write to the same dynamic theme registry, which replaces the current hardcoded `themes[]` array in `ThemeContext.tsx`.
+
+**Theme package structure** (same format for both channels):
+```
+{theme-id}/
+  theme.json          # id, name, description, supportsModes, defaultMode, author
+  tokens.css          # CSS custom properties (--color-*, --radius-*, --font-*, --shape-*)
+```
+
+## Implementation Phases
+
+### Phase 1: Manifest System (Foundation)
+
+Add component tracking without changing update behavior yet.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py` — Add dataclasses and manifest I/O
+
+**New dataclasses:**
+```python
+class ComponentId(str, Enum):
+    PM3 = "pm3"
+    FRONTEND = "frontend"
+    BACKEND = "backend"
+    THEME = "theme"
+
+@dataclass
+class ComponentInfo:
+    component_id: str
+    version: str
+    installed_at: str
+    checksum: Optional[str] = None
+    install_path: str = ""
+    platform: Optional[str] = None
+    python_version: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+@dataclass
+class SystemManifest:
+    schema_version: int = 1
+    system_version: str = "0.1.0"
+    components: Dict[str, ComponentInfo] = field(default_factory=dict)
+    last_updated: str = ""
+```
+
+**New file:** `/opt/dangerous-pi/component-manifest.json` — replaces the single `VERSION` file.
+
+**Migration:** On first startup with no manifest, `_migrate_from_legacy()` reads the existing `VERSION` file and PM3 `VERSION.txt` to populate a manifest with current component versions.
+
+---
+
+### Phase 2: CI/CD Build Pipeline
+
+Cross-compile PM3 and package all components in GitHub Actions.
+
+**New files:**
+- `.github/workflows/build-release.yml` — Main workflow, triggered on tag push (`v*`)
+- `ci/build-pm3.sh` — Runs in aarch64 Docker container (QEMU), reproduces the build from `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`:
+  - Clone RfidResearchGroup/proxmark3
+  - Apply `led-pwm-control.patch` + `hf-booster-detection.patch`
+  - Apply DangerousPi branding
+  - Set `PLATFORM=PM3GENERIC`, `LED_ORDER=PM3EASY`
+  - Build firmware (`bootrom.elf`, `fullimage.elf`) + client + SWIG bindings in one pass
+  - `cmake -DBUILD_PYTHON_LIB=ON`, SWIG bindings
+  - Matrix build for Python 3.11 + 3.12
+- `ci/build-frontend.sh` — `npm ci && npm run build`, package `build/`
+- `ci/build-backend.sh` — Package `app/backend/` + `requirements.txt`
+- `ci/build-theme.sh` — Builds `@dangerousthings/tokens` in the `dt-design-system` monorepo, packages the generated `dist/css/{dt,classic,supra}.css` + a `themes.json` registry
+- `ci/generate-manifest.py` — Generates `release-manifest.json` with checksums
+
+**Release asset naming:**
+```
+release-manifest.json
+pm3-{ver}-aarch64-linux-cp312.tar.gz
+pm3-{ver}-aarch64-linux-cp311.tar.gz
+frontend-{ver}.tar.gz
+backend-{ver}.tar.gz
+theme-{ver}.tar.gz
+```
+
+**`release-manifest.json` structure:**
+```json
+{
+  "schema_version": 1,
+  "release_version": "0.2.0",
+  "components": {
+    "pm3": {
+      "version": "4.19544-dp1",
+      "assets": {
+        "aarch64-linux-cp312": {
+          "filename": "pm3-4.19544-dp1-aarch64-linux-cp312.tar.gz",
+          "checksum_sha256": "...",
+          "size": 12345678,
+          "python_version": "3.12"
+        }
+      },
+      "changelog": "..."
+    }
+  }
+}
+```
+
+---
+
+### Phase 3: Component-Aware Update Logic
+
+Refactor `UpdateManager` to download/install individual components.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py`
+
+**Key changes:**
+1. `_fetch_latest_release()` → fetch and parse `release-manifest.json` from GitHub release assets (fall back to legacy `.tar.gz` scanning for old releases)
+2. `_select_pm3_asset()` → auto-detect Pi's Python version (`sys.version_info`) and architecture (`platform.machine()`) to pick the correct pm3 variant
+3. `check_for_updates()` → return per-component availability with compatibility info
+4. New `download_component(component_id)` → download a single component tarball
+5. New `install_component(component_id)` → component-specific install logic:
+   - `pm3`: copy client + firmware to `$HOME/.pm3/proxmark3/`, create `_pm3.so` symlink, verify arch
+   - `frontend`: replace `build/` directory
+   - `backend`: replace source, run `pip install -r requirements.txt` in venv
+   - `theme`: extract to `/opt/dangerous-pi/app/frontend/themes/`, update theme registry
+6. **Delete `_rebuild_pm3_client()`** — no more on-device compilation
+7. New `_backup_component()` / `rollback_component()` → per-component backups at `/opt/dangerous-pi/backups/{component_id}/`
+
+**New status:** `ROLLING_BACK` added to `UpdateStatus` enum.
+
+**New progress tracking:**
+```python
+@dataclass
+class ComponentProgress:
+    component_id: str
+    status: UpdateStatus
+    download_progress: float = 0.0
+    error_message: Optional[str] = None
+```
+
+Added as `component_progress: Dict[str, ComponentProgress]` to `UpdateProgress`.
+
+---
+
+### Phase 4: Unified Plugin Update Checking
+
+Integrate plugin update checks into the same update flow so "Check for Updates" covers everything.
+
+**Files to modify:**
+- `app/backend/managers/update_manager.py` — Add plugin update checking alongside component checks
+- `app/backend/managers/plugin_manager.py` — Add `check_plugin_updates()` method that queries each installed plugin's GitHub repo for newer releases
+
+**How it works:**
+1. `check_for_updates()` calls both:
+   - Core component check (from `release-manifest.json` on the main repo)
+   - Plugin update check (queries each plugin's source repo for newer tags)
+2. Plugin update info is returned alongside component updates in the same response
+3. Plugin download/install still uses the existing `plugin_manager.py` install logic — the update manager just orchestrates the version check
+4. Plugin versions are tracked in `component-manifest.json` under a `plugins` key (separate from `components`)
+
+**Manifest extension:**
+```json
+{
+  "components": { ... },
+  "plugins": {
+    "hello_world": {
+      "version": "1.0.0",
+      "source_repo": "dangerous-tacos/dp-plugin-hello-world",
+      "installed_at": "2026-03-03T10:00:00Z"
+    }
+  }
+}
+```
+
+**Update check response includes plugins:**
+```python
+class PluginUpdateInfo(BaseModel):
+    plugin_id: str
+    current_version: str
+    available_version: str
+    changelog: str
+    download_size: Optional[int]
+```
+
+The `/api/updates/check` response adds `plugins: List[PluginUpdateInfo]` alongside `components`.
+
+---
+
+### Phase 5: API & Service Layer (includes plugin update endpoints)
+
+**Files to modify:**
+- `app/backend/services/update_service.py` — Add `download_component()`, `install_component()`, `rollback_component()`, `get_installed_components()`
+- `app/backend/api/updates.py` — New endpoints + extended response models
+
+**New endpoints:**
+```
+GET  /api/updates/components                    — installed component manifest
+POST /api/updates/components/{id}/download      — download one component
+POST /api/updates/components/{id}/install       — install one component
+POST /api/updates/components/{id}/rollback      — rollback one component
+```
+
+**Modified endpoints (backward-compatible):**
+- `GET /check` — adds `components: List[ComponentUpdateInfo]` to response
+- `POST /download` — accepts optional `{"components": ["frontend"]}` body; no body = all
+- `POST /install` — same optional body pattern
+- `GET /progress` — adds `active_component` and `components` list to response
+
+---
+
+### Phase 5: Dynamic Theme Registry
+
+Decouple token CSS from the frontend build so themes can be updated independently.
+
+**Current flow (bundled):**
+`@dangerousthings/web/dist/index.css` → imports token CSS + structural CSS → bundled by Vite into Remix build
+
+**New flow (split):**
+- Structural CSS (`base.css`, `bevels.css`, `glows.css`, `elevation.css`, fonts) stays in the Remix build via `@dangerousthings/web`
+- Token CSS (`dt.css`, `classic.css`, `supra.css`) served from `/themes/{brand}/tokens.css` and loaded dynamically
+
+**Files to modify:**
+- `app/frontend/app/lib/ThemeContext.tsx` — Replace hardcoded `themes[]` array with dynamic loading from `GET /api/themes`; inject `<link>` tag for active brand's token CSS
+- `app/frontend/app/root.tsx` — Remove token CSS from the static import of `@dangerousthings/web/dist/index.css`; keep only structural CSS. Add dynamic `<link>` for token CSS.
+- `app/backend/api/system.py` — Add `GET /api/themes` endpoint that reads available themes from disk
+- `app/backend/main.py` — Mount `/themes/` as a static file directory
+
+**Files to create:**
+- `app/frontend/themes/` — Directory for installed theme packages (in dev)
+- `/opt/dangerous-pi/app/frontend/themes/` — Same, in production
+- Each theme: `themes/{theme-id}/theme.json` + `tokens.css`
+
+**How it works:**
+1. Backend scans `themes/` directory + plugin-registered themes → serves merged list at `GET /api/themes`
+2. `ThemeContext` fetches available themes on mount (with hardcoded fallback for SSR/offline)
+3. On brand switch, `ThemeContext` updates `<link href="/themes/{brand}/tokens.css">` in the document head
+4. Plugin themes register via `theme_register` hook, providing their CSS path + `theme.json`
+
+**Migration from `@dangerousthings/web`:**
+- `@dangerousthings/web` needs a new export that excludes token CSS: `@dangerousthings/web/dist/structural.css` (or the `index.css` is refactored to not import token CSS)
+- The CI `build-theme.sh` step copies `@dangerousthings/tokens/dist/css/{dt,classic,supra}.css` into the theme tarball as `{brand}/tokens.css`
+
+**Theme plugin hook** (added to plugin_manager.py):
+```python
+# Plugin calls this in on_enable():
+self.register_hook("theme_register", self._register_theme)
+
+def _register_theme(self):
+    return {
+        "id": "my-theme",
+        "name": "My Custom Theme",
+        "css_path": self.plugin_dir / "tokens.css",
+        "theme_json": self.plugin_dir / "theme.json"
+    }
+```
+
+---
+
+### Phase 6: Frontend UI
+
+**File to modify:**
+- `app/frontend/app/routes/updates.tsx`
+
+**Layout:**
+1. **System overview card** — system version, last check, "Check for Updates" + "Update All" buttons
+2. **Component cards** — one per component showing:
+   - Name, installed version, available version
+   - Compatibility badge (for pm3 Python version match)
+   - Collapsible per-component changelog
+   - Individual Download / Install / Rollback buttons
+   - Per-component progress bar
+3. **Plugin update cards** — shown for any installed plugins with updates available, using the same card layout. Download/install delegates to the existing plugin manager.
+4. **Status states per card:** up-to-date (muted), available (accent), downloading (progress bar), installing (spinner), complete (success), failed (danger + rollback button), incompatible (warning + reason)
+
+---
+
+### Phase 7: Testing & Polish
+
+- Test legacy migration (no manifest → auto-generated manifest)
+- Test partial update (update only frontend, leave PM3 alone)
+- Test Python version mismatch (pm3-client shows incompatible)
+- Test rollback per component
+- Test disk space check before download
+- Test backward compat with a legacy monolithic release (no `release-manifest.json`)
+- Test theme update (update `theme` component without touching frontend)
+- Test 3rd-party theme plugin (install, register, appear in theme selector, uninstall)
+- Test plugin update check (installed plugin with newer release shows in update check response)
+- Test plugin update install (update delegates to plugin_manager install logic)
+
+## Verification
+
+1. **Manifest migration**: Start backend with no `component-manifest.json` — verify it auto-creates from `VERSION` files
+2. **CI pipeline**: Push a test tag — verify all component tarballs + `release-manifest.json` are produced
+3. **Component check**: `GET /api/updates/check` — verify per-component version comparison
+4. **Selective update**: Download + install only `frontend` — verify only frontend files change
+5. **PM3 binary**: Install `pm3-client` — verify binary runs, `_pm3.so` symlink works, no compilation triggered
+6. **Rollback**: Install a component, then `POST /rollback` — verify files restored and manifest updated
+7. **Python version matching**: On a Pi with Python 3.12, verify `cp312` asset is selected; `cp311` shown as incompatible
+8. **Theme update**: Update `theme` component — verify new CSS applied without frontend rebuild
+9. **Theme plugin**: Install a theme plugin — verify it appears in theme selector alongside built-in themes
+10. **Plugin updates**: Install a plugin, publish a newer release to its repo — verify `GET /api/updates/check` includes it in the response and install works via existing plugin manager