MikeFaith/pi-pm3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build optimization: pre-built PM3 binaries, ARM64 CI, base image caching

Browse Source 
Replace PM3 compile-from-source in pi-gen with pre-built tarball extraction
(saves 43-58 min). Merge stagePM3 into stageDangerousPi as 02-pm3-install
substage, renumber all subsequent substages. Switch CI PM3 build to native
ARM64 runner (ubuntu-24.04-arm64) eliminating QEMU overhead. Add weekly
base-image workflow for pre-baking stages 0-2. Support PM3_TARBALL,
BASE_IMAGE, and APT_PROXY env vars in build-image.sh.

Also includes prior Phase 5 work: theme system, design system integration,
component update system, OS updates, CI build pipeline, and test results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
michael
2026-03-04 12:01:01 -08:00
parent 2ec89041ef
commit a9acdb85ce
163 changed files with 8124 additions and 921 deletions
Download Patch File Download Diff File Expand all files Collapse all files

282
.plan Normal file
View File

@@ -0,0 +1,282 @@
# Plan: APT Optimization & PM3 Build Caching
## Goal
Reduce redundant network requests and unnecessary rebuilds during pi-gen image builds.
---
## Part 1: Consolidate APT operations in stageDangerousPi
### 1A. Create a unified package list file for stageDangerousPi
**File:** `pi-gen/stageDangerousPi/00-packages` (new)
Move all apt packages currently installed inline across substages into a single
pi-gen `00-packages` file at the stage root level. Pi-gen's `run_sub_stage()`
already handles these automatically — one `apt-get install` call, one
`Reading package lists` parse.
**Problem:** pi-gen processes `XX-packages` files *per sub-stage directory*, not
at the stage level. The `run_stage()` iterates over `${STAGE_DIR}/*` sub-stage
directories, then `run_sub_stage()` looks for `{00..99}-packages` inside each.
There is no stage-level package consolidation built in.
**Revised approach:** Add a new sub-stage `00-apt-setup` that runs first (before
`01-Wireless-AP`) and handles all package installation for the entire
stageDangerousPi in one shot.
Create: `pi-gen/stageDangerousPi/00-apt-setup/00-packages`
```
# All packages needed across stageDangerousPi substages
# (consolidated to avoid redundant apt-get calls)
# 01-Wireless-AP
hostapd
dnsmasq
nftables
nginx
# 03-dangerous-pi
nodejs
npm
lrzsz
# 04-pisugar
i2c-tools
```
Note: `python3-pip` is omitted because it's already present from stage2 (Python 3.13
is installed in stage1, pip comes with it). The conditional `if ! command -v pip3`
in 03-dangerous-pi is dead code.
### 1B. Remove inline `apt-get install` calls from substage scripts
**Files to modify:**
1. `pi-gen/stageDangerousPi/01-Wireless-AP/00-run-chroot.sh`
- Remove lines 12-16 (`apt-get install -y hostapd dnsmasq nftables nginx`)
- Add a comment: `# Packages installed by 00-apt-setup stage`
2. `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh`
- Remove lines 17-25 (the conditional pip3 check + `apt-get update` + both
`apt-get install` calls)
- Add a comment noting packages come from 00-apt-setup
3. `pi-gen/stageDangerousPi/04-pisugar/00-run-chroot.sh`
- Remove lines 49-50 (`apt-get update` + `apt-get install -y i2c-tools`)
- Add a comment noting i2c-tools comes from 00-apt-setup
### 1C. Remove the redundant `apt-get update` from stagePM3
**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`
- Remove line 11 (`apt-get update`)
- The metadata was just refreshed by stage0's `00-configure-apt`. Between
stage0 and stagePM3, `copy_previous()` preserves `/var/lib/apt/lists/`
(only `/var/cache/apt/archives` is excluded). The index is still valid.
**Risk assessment:** Low. The build runs in a single session — the metadata
fetched minutes ago in stage0 hasn't changed. The build log confirms all 4
repos returned `Hit:` (cache valid) when stagePM3 ran `apt-get update`.
### Estimated savings
- 2-3 redundant `apt-get update` round-trips eliminated: ~10-15s
- ~5 fewer `Reading package lists` parses under QEMU: ~25-40s
- Fewer apt-get install invocations (3→0 in stageDangerousPi): ~15-20s
- **Total: ~50-75s per build**
---
## Part 2: PM3 build caching (skip rebuild when unchanged)
### Problem
The PM3 build (clone + compile firmware + client + SWIG bindings) takes
~45-60 minutes under QEMU. Currently `from-pm3` always rebuilds it, even
if nothing changed (same upstream commit, same patch, same build deps).
### Approach: Content-addressed cache key
Create a manifest file that captures everything that affects the PM3 build
output. If the manifest matches what was used for the cached build, skip
the entire PM3 compilation.
**Cache key inputs:**
1. The LED PWM control patch file hash
2. The Proxmark3 upstream git commit (pinned or HEAD)
3. The branding sed command in the build script
4. The build script itself (00-run-chroot.sh)
### Implementation
#### 2A. Pin the Proxmark3 upstream commit
**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`
Currently clones `HEAD` of master — non-deterministic. Change to clone a
specific tag or commit:
```bash
PM3_VERSION="v4.20728" # Pin to known-good release
PM3_COMMIT="" # Optional: specific commit hash
git clone https://github.com/RfidResearchGroup/proxmark3
cd proxmark3
if [ -n "$PM3_COMMIT" ]; then
git checkout "$PM3_COMMIT"
fi
```
Add a small config file `pi-gen/stagePM3/pm3-version.conf`:
```bash
PM3_REPO="https://github.com/RfidResearchGroup/proxmark3"
PM3_COMMIT="ef82d5ba1" # from the build log version string
```
#### 2B. Generate and check a build manifest
**File:** `pi-gen/stagePM3/01-proxmark3/00-run.sh` (pre-chroot, runs on host)
Add manifest generation and comparison logic:
```bash
# Generate cache key from all PM3 build inputs
generate_pm3_manifest() {
{
# Hash of the build script itself
sha256sum "${SCRIPT_DIR}/00-run-chroot.sh"
# Hash of the LED patch
sha256sum "${SCRIPT_DIR}/led-pwm-control.patch"
# PM3 version config
cat "${SCRIPT_DIR}/../pm3-version.conf" 2>/dev/null || echo "HEAD"
} | sha256sum | cut -d' ' -f1
}
MANIFEST=$(generate_pm3_manifest)
CACHED_MANIFEST="${ROOTFS_DIR}/opt/dangerous-pi/.pm3-build-manifest"
if [ -f "$CACHED_MANIFEST" ] && [ "$(cat "$CACHED_MANIFEST")" = "$MANIFEST" ]; then
echo "PM3 build manifest unchanged — skipping rebuild"
# Create a SKIP file for the chroot script
touch "${SCRIPT_DIR}/SKIP_PM3_BUILD"
else
echo "PM3 build manifest changed — will rebuild"
echo " New: $MANIFEST"
[ -f "$CACHED_MANIFEST" ] && echo " Old: $(cat "$CACHED_MANIFEST")"
rm -f "${SCRIPT_DIR}/SKIP_PM3_BUILD"
fi
```
#### 2C. Honor the skip flag in the chroot build script
**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`
Wrap the entire build in a check:
```bash
if [ -f /tmp/SKIP_PM3_BUILD ]; then
echo "=== Skipping Proxmark3 build (cached, manifest unchanged) ==="
# Still verify the installation is intact
DEFAULT_USER=$(awk -F: '$3 >= 1000 && $3 < 65534 {print $1; exit}' /etc/passwd)
DEFAULT_HOME=$(eval echo ~${DEFAULT_USER:-pi})
if [ -f "$DEFAULT_HOME/.pm3/proxmark3/client/proxmark3" ]; then
echo "✓ PM3 client binary present"
exit 0
else
echo "✗ PM3 binary missing despite manifest match — rebuilding"
fi
fi
# ... existing build logic ...
```
And at the end of a successful build, save the manifest:
```bash
# Save build manifest for future cache checks
mkdir -p /opt/dangerous-pi
# (manifest value is passed via a file from the pre-chroot script)
if [ -f /tmp/pm3-build-manifest ]; then
cp /tmp/pm3-build-manifest /opt/dangerous-pi/.pm3-build-manifest
fi
```
#### 2D. Pass files between pre-chroot and chroot
**File:** `pi-gen/stagePM3/01-proxmark3/00-run.sh`
The pre-chroot script already copies the LED patch into `${ROOTFS_DIR}/tmp/`.
Extend it to also copy the skip flag and manifest value:
```bash
# Copy skip flag if set
if [ -f "${SCRIPT_DIR}/SKIP_PM3_BUILD" ]; then
touch "${ROOTFS_DIR}/tmp/SKIP_PM3_BUILD"
rm -f "${SCRIPT_DIR}/SKIP_PM3_BUILD"
fi
# Always write current manifest for the chroot to save on success
echo "$MANIFEST" > "${ROOTFS_DIR}/tmp/pm3-build-manifest"
```
#### 2E. Update build-image.sh for clarity
Add a `--force-pm3` flag to `from-pm3` mode that removes the cached manifest,
forcing a rebuild even if the inputs haven't changed:
```bash
from-pm3)
# ... existing logic ...
if [ "$FORCE_PM3" = "1" ]; then
echo "Forcing PM3 rebuild (--force-pm3 flag)"
# Remove cached manifest from the work directory rootfs
rm -f "${PI_GEN_DIR}/work/${IMG_NAME}/stagePM3/rootfs/opt/dangerous-pi/.pm3-build-manifest"
fi
```
And update the usage text to document the new behavior.
---
## Part 3: File changes summary
### New files
| File | Purpose |
|------|---------|
| `pi-gen/stageDangerousPi/00-apt-setup/00-packages` | Consolidated package list |
| `pi-gen/stagePM3/pm3-version.conf` | Pinned PM3 repo + commit |
### Modified files
| File | Change |
|------|--------|
| `pi-gen/stagePM3/01-proxmark3/00-run.sh` | Add manifest generation, skip flag, manifest passthrough |
| `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh` | Remove `apt-get update`; add skip check at top; save manifest on success; pin git clone to commit |
| `pi-gen/stageDangerousPi/01-Wireless-AP/00-run-chroot.sh` | Remove inline `apt-get install` |
| `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh` | Remove conditional pip check, `apt-get update`, both `apt-get install` calls |
| `pi-gen/stageDangerousPi/04-pisugar/00-run-chroot.sh` | Remove `apt-get update` + `apt-get install` |
| `build-image.sh` | Add `--force-pm3` flag parsing and usage docs |
### Not modified (left as-is)
| File | Reason |
|------|--------|
| `pi-gen/stageDangerousPi/06-os-updates/00-packages` | Already uses file-based packages correctly |
| `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh` cleanup section | `apt-get clean/autoclean/autoremove` at bottom is still useful |
| `pi-gen/stageDangerousPi/05-https-support/00-run-chroot.sh` | No apt calls to change |
---
## Risk assessment
| Change | Risk | Mitigation |
|--------|------|------------|
| Removing `apt-get update` from stagePM3 | Low — metadata inherited from stage0, same build session | Build log proves `Hit:` on all repos |
| Consolidating packages into 00-apt-setup | Low — same packages, just installed earlier | Substage ordering (00- before 01-) guarantees execution order |
| Removing conditional pip3 install | Low — pip3 always present from stage1/2 Python install | Verified in stage2 package lists |
| PM3 skip logic | Medium — if manifest check has a bug, stale binary ships | Verification step checks binary exists; `--force-pm3` escape hatch |
| Pinning PM3 commit | Low — explicit is better than implicit HEAD | Documented in pm3-version.conf; easy to bump |
## Testing approach
1. Run `./build-image.sh test` to syntax-check all modified scripts
2. Run `./build-image.sh from-dtpi` to verify APT consolidation works (Part 1)
3. Run `./build-image.sh from-pm3` twice — second run should skip PM3 build (Part 2)
4. Modify `led-pwm-control.patch` slightly, rebuild — should trigger PM3 rebuild
5. Run `./build-image.sh from-pm3 --force-pm3` — should always rebuild