pi-pm3/BUILD_CONSTRAINTS.md

# Build Constraints and Optimization Strategy

**Date:** 2025-11-27
**Critical Context:** This project has severe bandwidth constraints that make build optimization MANDATORY, not optional.

---

## Primary Constraint: Slow Network

**Measured download speed: ~265 kB/s (0.26 MB/s)**

### Impact on Build Times

| Stage | Download Size | Download Time | Total Time | Notes |
|-------|---------------|---------------|------------|-------|
| **stage0** | ~200 MB | 13 min | 18 min | Base Debian system |
| **stage1** | ~450 MB | 29 min | 37 min | Kernel + firmware |
| **stage2** | ~850 MB | 55 min | 65 min | Desktop + network tools |
| **stagePM3** | ~220 MB | 14 min | 59 min | PM3 build (downloads + compile) |
| **stageDangerousPi** | ~140 MB | 9 min | 12 min | Custom packages |
| **TOTAL** | **~1.86 GB** | **~120 min** | **~191 min** | **3+ hours** |

### Critical Implications

1. **Every failed build wastes 3+ hours**
2. **Stage0-2 alone = 2 hours of downloading**
3. **Cannot afford trial-and-error debugging**
4. **Pre-flight validation MUST be perfect**
5. **Incremental builds are MANDATORY for iteration**

---

## Build Optimization Strategy

### 1. QCOW2 Caching (MANDATORY)

```bash
# In pi-gen/config
USE_QCOW2=1  # NEVER disable this
```

**Why:** Enables copy-on-write snapshots. After stage2 completes once (2 hours), we can rebuild from that point in ~70 minutes instead of 3+ hours.

**Cache Location:** `/home/work/pi-gen-builder/work/*/stage*.qcow2`

### 2. Three-Tier Build Strategy

```bash
# Full build: Clean slate (3+ hours)
./build-image.sh full

# From PM3: Rebuild PM3 + customizations (70 min)
./build-image.sh from-pm3

# From DangerousPi: Rebuild only customizations (5 min)
./build-image.sh from-dtpi
```

**Daily workflow:**
- First build: `full` (one-time 3hr investment)
- Iterating on PM3 changes: `from-pm3` (70 min)
- Iterating on app/WiFi/config: `from-dtpi` (5 min)

### 3. Container vs Volume Management

**CRITICAL DISTINCTION:**
- **Container** (`docker rm pigen_work`): Safe to remove, just metadata
- **Volume** (`docker rm -v pigen_work`): ❌ NEVER DO THIS - nukes 2hr download cache

**Build script behavior:**
- `full` mode: Removes container AND volume (fresh start)
- `from-pm3` mode: Preserves cache, removes container
- `from-dtpi` mode: Preserves cache, removes container

---

## Pre-Flight Validation Requirements

### Philosophy

> **"Fail fast, fail loud, fail BEFORE downloading anything"**

With 3+ hour builds, we CANNOT afford to discover errors after stage0-2 completes.

### Current Checks (52 total)

✅ Pi-gen directory exists
✅ Docker available and accessible
✅ qemu-aarch64 registration
✅ Config file validation
✅ Stage structure (prerun.sh, EXPORT_IMAGE, substages)
✅ Build script syntax validation
✅ Dependency analysis (header → package mapping)
✅ Source file availability
✅ Build script validation

### Required Additions

The following checks MUST be added to prevent wasted builds:

#### A. Pre-Download Validation
- [ ] **Disk space check:** Require 15GB free before starting
- [ ] **Network connectivity:** Verify can reach deb.debian.org
- [ ] **Git repo accessibility:** Test clone URLs without cloning
- [ ] **Stage size estimation:** Warn if total download > threshold

#### B. Deep Script Analysis
- [ ] **File reference validation:** Grep for all file paths in scripts, verify they exist
- [ ] **Service file syntax:** Validate systemd .service files
- [ ] **Environment variables:** Check for undefined vars in scripts
- [ ] **Path existence:** Verify all mkdir -p targets don't conflict
- [ ] **Port conflicts:** Check for duplicate port assignments

#### C. Configuration Validation
- [ ] **STAGE_LIST ordering:** Ensure stages are in dependency order
- [ ] **EXPORT_IMAGE syntax:** Validate all EXPORT_IMAGE files
- [ ] **Duplicate substages:** Check for numbering conflicts (e.g., two 00-run.sh in same stage)
- [ ] **Username consistency:** Verify same user referenced across stages

#### D. Build Artifact Validation
- [ ] **Post-stage verification:** After each stage, check expected files exist
- [ ] **Binary executability:** Test compiled binaries actually run
- [ ] **Python import test:** Verify Python modules can be imported
- [ ] **Service enable test:** Verify systemd services are enabled

---

## Development Workflow Optimizations

### 1. Test Changes Locally First

Before modifying stage scripts, test commands locally:

```bash
# Test PM3 dependency installation
docker run --rm -it debian:trixie bash
> apt-get update && apt-get install -y gcc-arm-none-eabi
> which arm-none-eabi-gcc

# Test Python package installation
docker run --rm -it debian:trixie bash
> apt-get update && apt-get install -y python3-pip
> pip3 install --break-system-packages fastapi==0.115.0
```

**Saves:** Hours of build time by catching errors early

### 2. Use Build Log Analysis

During builds, monitor for early warning signs:

```bash
# Watch for errors in real-time
docker logs -f pigen_work 2>&1 | grep -i "error\|fail\|fatal"

# Check what stage we're on
./scripts/build-status.sh

# Estimate completion based on download speed
docker logs pigen_work 2>&1 | grep "Fetched.*in.*(" | tail -5
```

### 3. Parallel Development

While a build runs, prepare next iteration:

```bash
# Terminal 1: Build running
./build-image.sh full

# Terminal 2: Test next changes
cd /tmp/test-changes
# Test scripts, validate syntax, etc.
```

---

## Failure Recovery Procedures

### If Build Fails During stagePM3 or Later

**DO NOT KILL CONTAINER**

1. Let it fail completely
2. Check what stage completed:
   ```bash
   ls -la /home/work/pi-gen-builder/work/*/stage*.qcow2
   ```
3. If stage2.qcow2 exists, you have cache:
   ```bash
   ./build-image.sh from-pm3
   ```

### If Build Fails During stage0-2

**Only kill if you know stage will fail:**

```bash
# Check if we have partial cache
ls -la /home/work/pi-gen-builder/work/*/stage*.qcow2

# If no .qcow2 files, killing wastes nothing
# If stage1.qcow2 exists, let it finish stage2
```

---

## Cost-Benefit Analysis

### Why Pre-Flight Validation Matters

**Without perfect validation:**
- Change PM3 script
- Start 3hr build
- Discover typo at hour 2.5
- Fix typo
- Restart 3hr build
- **Total time wasted: 6 hours**

**With perfect validation:**
- Change PM3 script
- Run pre-flight (30 seconds)
- Catch typo immediately
- Fix typo
- Run pre-flight again (30 seconds)
- Start build with confidence
- **Total time wasted: 60 seconds**

**ROI: 360x time savings**

---

## Questions for Future Optimization

1. **Network caching:** Should we set up apt-cacher-ng to cache Debian packages locally?
   - Pro: Rebuilds wouldn't re-download same packages
   - Con: Setup complexity, disk space (20GB+)

2. **Disk space monitoring:** Add alerts when cache disk < 10GB free?

3. **Build checkpointing:** Should we export stage2.qcow2 as a baseline artifact?

4. **Parallel builds:** Can we test multiple variants simultaneously?

5. **Common errors database:** Should we maintain a list of known failure patterns to check for?

---

## Action Items

### Immediate (Before Next Build)
- [ ] Add disk space check to pre-flight
- [ ] Add network connectivity check to pre-flight
- [ ] Document common failure patterns we've seen
- [ ] Test local command validation workflow

### Short-term (This Week)
- [ ] Add deep script validation (file references, env vars)
- [ ] Create build artifact verification
- [ ] Set up build time estimation based on network speed
- [ ] Document recovery procedures for each failure type

### Long-term (Future Optimization)
- [ ] Consider apt-cacher-ng for package caching
- [ ] Automated build artifact testing
- [ ] Build time alerting/monitoring
- [ ] Stage baseline export/import

---

## Key Takeaway

> **Every validation check we add to pre-flight saves potentially 3+ hours of wasted build time.**
>
> **Spending 5 minutes improving pre-flight can save hours of debugging.**
>
> **This is not premature optimization - it's mandatory survival strategy.**