pi-pm3/BUILD_CONSTRAINTS.md

Build Constraints and Optimization Strategy

Date: 2025-11-27 Critical Context: This project has severe bandwidth constraints that make build optimization MANDATORY, not optional.

---

Primary Constraint: Slow Network

Measured download speed: ~265 kB/s (0.26 MB/s)

Impact on Build Times

Stage	Download Size	Download Time	Total Time	Notes
stage0	~200 MB	13 min	18 min	Base Debian system
stage1	~450 MB	29 min	37 min	Kernel + firmware
stage2	~850 MB	55 min	65 min	Desktop + network tools
stagePM3	~220 MB	14 min	59 min	PM3 build (downloads + compile)
stageDangerousPi	~140 MB	9 min	12 min	Custom packages
TOTAL	~1.86 GB	~120 min	~191 min	3+ hours

Critical Implications

1. Every failed build wastes 3+ hours
2. Stage0-2 alone = 2 hours of downloading
3. Cannot afford trial-and-error debugging
4. Pre-flight validation MUST be perfect
5. Incremental builds are MANDATORY for iteration

---

Build Optimization Strategy

1. QCOW2 Caching (MANDATORY)

# In pi-gen/config
USE_QCOW2=1  # NEVER disable this

Why: Enables copy-on-write snapshots. After stage2 completes once (2 hours), we can rebuild from that point in ~70 minutes instead of 3+ hours.

Cache Location: /home/work/pi-gen-builder/work/*/stage*.qcow2

2. Three-Tier Build Strategy

# Full build: Clean slate (3+ hours)
./build-image.sh full

# From PM3: Rebuild PM3 + customizations (70 min)
./build-image.sh from-pm3

# From DangerousPi: Rebuild only customizations (5 min)
./build-image.sh from-dtpi

Daily workflow:

• First build: full (one-time 3hr investment)
• Iterating on PM3 changes: from-pm3 (70 min)
• Iterating on app/WiFi/config: from-dtpi (5 min)

3. Container vs Volume Management

CRITICAL DISTINCTION:

• Container (docker rm pigen_work): Safe to remove, just metadata
• Volume (docker rm -v pigen_work): ❌ NEVER DO THIS - nukes 2hr download cache

Build script behavior:

• full mode: Removes container AND volume (fresh start)
• from-pm3 mode: Preserves cache, removes container
• from-dtpi mode: Preserves cache, removes container

---

Pre-Flight Validation Requirements

Philosophy

"Fail fast, fail loud, fail BEFORE downloading anything"

With 3+ hour builds, we CANNOT afford to discover errors after stage0-2 completes.

Current Checks (52 total)

✅ Pi-gen directory exists ✅ Docker available and accessible ✅ qemu-aarch64 registration ✅ Config file validation ✅ Stage structure (prerun.sh, EXPORT_IMAGE, substages) ✅ Build script syntax validation ✅ Dependency analysis (header → package mapping) ✅ Source file availability ✅ Build script validation

Required Additions

The following checks MUST be added to prevent wasted builds:

A. Pre-Download Validation

• Disk space check: Require 15GB free before starting
• Network connectivity: Verify can reach deb.debian.org
• Git repo accessibility: Test clone URLs without cloning
• Stage size estimation: Warn if total download > threshold

B. Deep Script Analysis

• File reference validation: Grep for all file paths in scripts, verify they exist
• Service file syntax: Validate systemd .service files
• Environment variables: Check for undefined vars in scripts
• Path existence: Verify all mkdir -p targets don't conflict
• Port conflicts: Check for duplicate port assignments

C. Configuration Validation

• STAGE_LIST ordering: Ensure stages are in dependency order
• EXPORT_IMAGE syntax: Validate all EXPORT_IMAGE files
• Duplicate substages: Check for numbering conflicts (e.g., two 00-run.sh in same stage)
• Username consistency: Verify same user referenced across stages

D. Build Artifact Validation

• Post-stage verification: After each stage, check expected files exist
• Binary executability: Test compiled binaries actually run
• Python import test: Verify Python modules can be imported
• Service enable test: Verify systemd services are enabled

---

Development Workflow Optimizations

1. Test Changes Locally First

Before modifying stage scripts, test commands locally:

# Test PM3 dependency installation
docker run --rm -it debian:trixie bash
> apt-get update && apt-get install -y gcc-arm-none-eabi
> which arm-none-eabi-gcc

# Test Python package installation
docker run --rm -it debian:trixie bash
> apt-get update && apt-get install -y python3-pip
> pip3 install --break-system-packages fastapi==0.115.0

Saves: Hours of build time by catching errors early

2. Use Build Log Analysis

During builds, monitor for early warning signs:

# Watch for errors in real-time
docker logs -f pigen_work 2>&1 | grep -i "error\|fail\|fatal"

# Check what stage we're on
./scripts/build-status.sh

# Estimate completion based on download speed
docker logs pigen_work 2>&1 | grep "Fetched.*in.*(" | tail -5

3. Parallel Development

While a build runs, prepare next iteration:

# Terminal 1: Build running
./build-image.sh full

# Terminal 2: Test next changes
cd /tmp/test-changes
# Test scripts, validate syntax, etc.

---

Failure Recovery Procedures

If Build Fails During stagePM3 or Later

DO NOT KILL CONTAINER

1. Let it fail completely
2. Check what stage completed:

   ls -la /home/work/pi-gen-builder/work/*/stage*.qcow2
   

3. If stage2.qcow2 exists, you have cache:

   ./build-image.sh from-pm3
   

If Build Fails During stage0-2

Only kill if you know stage will fail:

# Check if we have partial cache
ls -la /home/work/pi-gen-builder/work/*/stage*.qcow2

# If no .qcow2 files, killing wastes nothing
# If stage1.qcow2 exists, let it finish stage2

---

Cost-Benefit Analysis

Why Pre-Flight Validation Matters

Without perfect validation:

• Change PM3 script
• Start 3hr build
• Discover typo at hour 2.5
• Fix typo
• Restart 3hr build
• Total time wasted: 6 hours

With perfect validation:

• Change PM3 script
• Run pre-flight (30 seconds)
• Catch typo immediately
• Fix typo
• Run pre-flight again (30 seconds)
• Start build with confidence
• Total time wasted: 60 seconds

ROI: 360x time savings

---

Questions for Future Optimization

1. Network caching: Should we set up apt-cacher-ng to cache Debian packages locally?

   • Pro: Rebuilds wouldn't re-download same packages
   • Con: Setup complexity, disk space (20GB+)

2. Disk space monitoring: Add alerts when cache disk < 10GB free?

3. Build checkpointing: Should we export stage2.qcow2 as a baseline artifact?

4. Parallel builds: Can we test multiple variants simultaneously?

5. Common errors database: Should we maintain a list of known failure patterns to check for?

---

Action Items

Immediate (Before Next Build)

• Add disk space check to pre-flight
• Add network connectivity check to pre-flight
• Document common failure patterns we've seen
• Test local command validation workflow

Short-term (This Week)

• Add deep script validation (file references, env vars)
• Create build artifact verification
• Set up build time estimation based on network speed
• Document recovery procedures for each failure type

Long-term (Future Optimization)

• Consider apt-cacher-ng for package caching
• Automated build artifact testing
• Build time alerting/monitoring
• Stage baseline export/import

---

Key Takeaway

Every validation check we add to pre-flight saves potentially 3+ hours of wasted build time.

Spending 5 minutes improving pre-flight can save hours of debugging.

This is not premature optimization - it's mandatory survival strategy.