pi-pm3/.plan

# Plan: APT Optimization & PM3 Build Caching

## Goal
Reduce redundant network requests and unnecessary rebuilds during pi-gen image builds.

---

## Part 1: Consolidate APT operations in stageDangerousPi

### 1A. Create a unified package list file for stageDangerousPi

**File:** `pi-gen/stageDangerousPi/00-packages` (new)

Move all apt packages currently installed inline across substages into a single
pi-gen `00-packages` file at the stage root level. Pi-gen's `run_sub_stage()`
already handles these automatically — one `apt-get install` call, one
`Reading package lists` parse.

**Problem:** pi-gen processes `XX-packages` files *per sub-stage directory*, not
at the stage level. The `run_stage()` iterates over `${STAGE_DIR}/*` sub-stage
directories, then `run_sub_stage()` looks for `{00..99}-packages` inside each.
There is no stage-level package consolidation built in.

**Revised approach:** Add a new sub-stage `00-apt-setup` that runs first (before
`01-Wireless-AP`) and handles all package installation for the entire
stageDangerousPi in one shot.

Create: `pi-gen/stageDangerousPi/00-apt-setup/00-packages`
```
# All packages needed across stageDangerousPi substages
# (consolidated to avoid redundant apt-get calls)

# 01-Wireless-AP
hostapd
dnsmasq
nftables
nginx

# 03-dangerous-pi
nodejs
npm
lrzsz

# 04-pisugar
i2c-tools
```

Note: `python3-pip` is omitted because it's already present from stage2 (Python 3.13
is installed in stage1, pip comes with it). The conditional `if ! command -v pip3`
in 03-dangerous-pi is dead code.

### 1B. Remove inline `apt-get install` calls from substage scripts

**Files to modify:**

1. `pi-gen/stageDangerousPi/01-Wireless-AP/00-run-chroot.sh`
   - Remove lines 12-16 (`apt-get install -y hostapd dnsmasq nftables nginx`)
   - Add a comment: `# Packages installed by 00-apt-setup stage`

2. `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh`
   - Remove lines 17-25 (the conditional pip3 check + `apt-get update` + both
     `apt-get install` calls)
   - Add a comment noting packages come from 00-apt-setup

3. `pi-gen/stageDangerousPi/04-pisugar/00-run-chroot.sh`
   - Remove lines 49-50 (`apt-get update` + `apt-get install -y i2c-tools`)
   - Add a comment noting i2c-tools comes from 00-apt-setup

### 1C. Remove the redundant `apt-get update` from stagePM3

**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`
- Remove line 11 (`apt-get update`)
- The metadata was just refreshed by stage0's `00-configure-apt`. Between
  stage0 and stagePM3, `copy_previous()` preserves `/var/lib/apt/lists/`
  (only `/var/cache/apt/archives` is excluded). The index is still valid.

**Risk assessment:** Low. The build runs in a single session — the metadata
fetched minutes ago in stage0 hasn't changed. The build log confirms all 4
repos returned `Hit:` (cache valid) when stagePM3 ran `apt-get update`.

### Estimated savings
- 2-3 redundant `apt-get update` round-trips eliminated: ~10-15s
- ~5 fewer `Reading package lists` parses under QEMU: ~25-40s
- Fewer apt-get install invocations (3→0 in stageDangerousPi): ~15-20s
- **Total: ~50-75s per build**

---

## Part 2: PM3 build caching (skip rebuild when unchanged)

### Problem
The PM3 build (clone + compile firmware + client + SWIG bindings) takes
~45-60 minutes under QEMU. Currently `from-pm3` always rebuilds it, even
if nothing changed (same upstream commit, same patch, same build deps).

### Approach: Content-addressed cache key

Create a manifest file that captures everything that affects the PM3 build
output. If the manifest matches what was used for the cached build, skip
the entire PM3 compilation.

**Cache key inputs:**
1. The LED PWM control patch file hash
2. The Proxmark3 upstream git commit (pinned or HEAD)
3. The branding sed command in the build script
4. The build script itself (00-run-chroot.sh)

### Implementation

#### 2A. Pin the Proxmark3 upstream commit

**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`

Currently clones `HEAD` of master — non-deterministic. Change to clone a
specific tag or commit:

```bash
PM3_VERSION="v4.20728"  # Pin to known-good release
PM3_COMMIT=""           # Optional: specific commit hash

git clone https://github.com/RfidResearchGroup/proxmark3
cd proxmark3
if [ -n "$PM3_COMMIT" ]; then
    git checkout "$PM3_COMMIT"
fi
```

Add a small config file `pi-gen/stagePM3/pm3-version.conf`:
```bash
PM3_REPO="https://github.com/RfidResearchGroup/proxmark3"
PM3_COMMIT="ef82d5ba1"  # from the build log version string
```

#### 2B. Generate and check a build manifest

**File:** `pi-gen/stagePM3/01-proxmark3/00-run.sh` (pre-chroot, runs on host)

Add manifest generation and comparison logic:

```bash
# Generate cache key from all PM3 build inputs
generate_pm3_manifest() {
    {
        # Hash of the build script itself
        sha256sum "${SCRIPT_DIR}/00-run-chroot.sh"
        # Hash of the LED patch
        sha256sum "${SCRIPT_DIR}/led-pwm-control.patch"
        # PM3 version config
        cat "${SCRIPT_DIR}/../pm3-version.conf" 2>/dev/null || echo "HEAD"
    } | sha256sum | cut -d' ' -f1
}

MANIFEST=$(generate_pm3_manifest)
CACHED_MANIFEST="${ROOTFS_DIR}/opt/dangerous-pi/.pm3-build-manifest"

if [ -f "$CACHED_MANIFEST" ] && [ "$(cat "$CACHED_MANIFEST")" = "$MANIFEST" ]; then
    echo "PM3 build manifest unchanged — skipping rebuild"
    # Create a SKIP file for the chroot script
    touch "${SCRIPT_DIR}/SKIP_PM3_BUILD"
else
    echo "PM3 build manifest changed — will rebuild"
    echo "  New: $MANIFEST"
    [ -f "$CACHED_MANIFEST" ] && echo "  Old: $(cat "$CACHED_MANIFEST")"
    rm -f "${SCRIPT_DIR}/SKIP_PM3_BUILD"
fi
```

#### 2C. Honor the skip flag in the chroot build script

**File:** `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh`

Wrap the entire build in a check:

```bash
if [ -f /tmp/SKIP_PM3_BUILD ]; then
    echo "=== Skipping Proxmark3 build (cached, manifest unchanged) ==="
    # Still verify the installation is intact
    DEFAULT_USER=$(awk -F: '$3 >= 1000 && $3 < 65534 {print $1; exit}' /etc/passwd)
    DEFAULT_HOME=$(eval echo ~${DEFAULT_USER:-pi})
    if [ -f "$DEFAULT_HOME/.pm3/proxmark3/client/proxmark3" ]; then
        echo "✓ PM3 client binary present"
        exit 0
    else
        echo "✗ PM3 binary missing despite manifest match — rebuilding"
    fi
fi

# ... existing build logic ...
```

And at the end of a successful build, save the manifest:

```bash
# Save build manifest for future cache checks
mkdir -p /opt/dangerous-pi
# (manifest value is passed via a file from the pre-chroot script)
if [ -f /tmp/pm3-build-manifest ]; then
    cp /tmp/pm3-build-manifest /opt/dangerous-pi/.pm3-build-manifest
fi
```

#### 2D. Pass files between pre-chroot and chroot

**File:** `pi-gen/stagePM3/01-proxmark3/00-run.sh`

The pre-chroot script already copies the LED patch into `${ROOTFS_DIR}/tmp/`.
Extend it to also copy the skip flag and manifest value:

```bash
# Copy skip flag if set
if [ -f "${SCRIPT_DIR}/SKIP_PM3_BUILD" ]; then
    touch "${ROOTFS_DIR}/tmp/SKIP_PM3_BUILD"
    rm -f "${SCRIPT_DIR}/SKIP_PM3_BUILD"
fi

# Always write current manifest for the chroot to save on success
echo "$MANIFEST" > "${ROOTFS_DIR}/tmp/pm3-build-manifest"
```

#### 2E. Update build-image.sh for clarity

Add a `--force-pm3` flag to `from-pm3` mode that removes the cached manifest,
forcing a rebuild even if the inputs haven't changed:

```bash
from-pm3)
    # ... existing logic ...
    if [ "$FORCE_PM3" = "1" ]; then
        echo "Forcing PM3 rebuild (--force-pm3 flag)"
        # Remove cached manifest from the work directory rootfs
        rm -f "${PI_GEN_DIR}/work/${IMG_NAME}/stagePM3/rootfs/opt/dangerous-pi/.pm3-build-manifest"
    fi
```

And update the usage text to document the new behavior.

---

## Part 3: File changes summary

### New files
| File | Purpose |
|------|---------|
| `pi-gen/stageDangerousPi/00-apt-setup/00-packages` | Consolidated package list |
| `pi-gen/stagePM3/pm3-version.conf` | Pinned PM3 repo + commit |

### Modified files
| File | Change |
|------|--------|
| `pi-gen/stagePM3/01-proxmark3/00-run.sh` | Add manifest generation, skip flag, manifest passthrough |
| `pi-gen/stagePM3/01-proxmark3/00-run-chroot.sh` | Remove `apt-get update`; add skip check at top; save manifest on success; pin git clone to commit |
| `pi-gen/stageDangerousPi/01-Wireless-AP/00-run-chroot.sh` | Remove inline `apt-get install` |
| `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh` | Remove conditional pip check, `apt-get update`, both `apt-get install` calls |
| `pi-gen/stageDangerousPi/04-pisugar/00-run-chroot.sh` | Remove `apt-get update` + `apt-get install` |
| `build-image.sh` | Add `--force-pm3` flag parsing and usage docs |

### Not modified (left as-is)
| File | Reason |
|------|--------|
| `pi-gen/stageDangerousPi/06-os-updates/00-packages` | Already uses file-based packages correctly |
| `pi-gen/stageDangerousPi/03-dangerous-pi/00-run-chroot.sh` cleanup section | `apt-get clean/autoclean/autoremove` at bottom is still useful |
| `pi-gen/stageDangerousPi/05-https-support/00-run-chroot.sh` | No apt calls to change |

---

## Risk assessment

| Change | Risk | Mitigation |
|--------|------|------------|
| Removing `apt-get update` from stagePM3 | Low — metadata inherited from stage0, same build session | Build log proves `Hit:` on all repos |
| Consolidating packages into 00-apt-setup | Low — same packages, just installed earlier | Substage ordering (00- before 01-) guarantees execution order |
| Removing conditional pip3 install | Low — pip3 always present from stage1/2 Python install | Verified in stage2 package lists |
| PM3 skip logic | Medium — if manifest check has a bug, stale binary ships | Verification step checks binary exists; `--force-pm3` escape hatch |
| Pinning PM3 commit | Low — explicit is better than implicit HEAD | Documented in pm3-version.conf; easy to bump |

## Testing approach

1. Run `./build-image.sh test` to syntax-check all modified scripts
2. Run `./build-image.sh from-dtpi` to verify APT consolidation works (Part 1)
3. Run `./build-image.sh from-pm3` twice — second run should skip PM3 build (Part 2)
4. Modify `led-pwm-control.patch` slightly, rebuild — should trigger PM3 rebuild
5. Run `./build-image.sh from-pm3 --force-pm3` — should always rebuild