pi-pm3/PREFLIGHT_ENHANCEMENTS.md

Pre-Flight Validation Enhancements

Date: 2025-11-27 Context: Added comprehensive validation to catch errors BEFORE wasting 3+ hours on failed builds

---

Summary

Added 6 new validation sections to scripts/preflight-check.sh:

#	Section	Purpose	Time Impact
9	Package Name Validation	Prevents hallucinated/typo package names	Saves 2+ hours
10	File Reference Validation	Catches missing source files	Saves 1+ hours
11	Service File Syntax	Validates systemd .service files	Saves 3+ hours
12	Environment Variable Checks	Detects undefined variables	Saves 2+ hours
13	Disk Space Check	Ensures 15GB available	Prevents mid-build failures
14	Network Connectivity	Tests repos before downloading	Saves 3+ hours

Total new checks: 6 sections Previous total: 52 checks New estimate: 100+ checks

---

1. Package Name Validation (Section 9)

Problem It Solves

Past mistakes:

• python-swig instead of swig
• libbluez-dev instead of libbluetooth-dev
• Typos in package names that fail 2 hours into build

How It Works

# Spins up temporary Debian:trixie container
# Runs apt-get update inside container
# Extracts ALL package names from stage scripts
# Queries apt-cache show <package> for each one
# Reports EXACT package names that don't exist

Output Example

✓ Package exists: gcc-arm-none-eabi
✓ Package exists: libbluetooth-dev
✗ ERROR: Package NOT FOUND in Debian repos: python-swig
    Check for typos or hallucinated package names!
    Common mistakes: python-swig (should be: swig)

Performance

• Time: ~30-60 seconds (depends on package count)
• Network: Minimal (only package index download)
• Accuracy: 100% - queries actual Debian repository

---

2. File Reference Validation (Section 10)

Problem It Solves

Scripts reference files that don't exist:

cp "${SCRIPT_DIR}/files/app" ...  # files/ directory missing!

How It Works

# Scans all 00-run.sh scripts (outside chroot)
# Extracts 'cp' commands
# Checks if source files/directories exist
# Warns about missing references

Output Example

✓ Found: files/app (in 03-dangerous-pi)
✓ Found: files/systemd (in 03-dangerous-pi)
⚠ WARNING: Missing file reference: files/firmware in stagePM3/01-proxmark3/00-run.sh

Limitations

• Only checks static paths (not variables like $ROOTFS_DIR)
• Only checks 00-run.sh (files outside chroot)
• Doesn't validate paths inside chroot environment

---

3. Service File Syntax Validation (Section 11)

Problem It Solves

Malformed systemd service files that fail silently or at boot:

[Unit]
# Missing [Service] section!
[Install]
WantedBy=multi-user.target

How It Works

# Finds all .service files in stages and systemd/
# Checks for required sections: [Unit], [Service], [Install]
# Verifies ExecStart= exists
# Detects un-substituted __PLACEHOLDERS__

Output Example

✓ Service file structure valid: dangerous-pi.service
✓ Has ExecStart: dangerous-pi.service
⚠ WARNING: Found un-substituted placeholder in: ttyd.service
    ExecStart=/usr/bin/ttyd -u __USER_UID__

What It Catches

• Missing required sections
• Missing ExecStart command
• Un-replaced template variables (like __USER__)

---

4. Environment Variable Checks (Section 12)

Problem It Solves

Scripts use variables that aren't defined:

mkdir -p $DATA_DIR/logs  # $DATA_DIR never set!

How It Works

# Scans all stage scripts for $VARIABLE usage
# Checks if variable is defined in same script
# Allows known pi-gen variables (ROOTFS_DIR, etc.)
# Warns about potentially undefined variables

Output Example

⚠ WARNING: Potentially undefined variable: $DATA_DIR in 00-run-chroot.sh
⚠ WARNING: Potentially undefined variable: $INSTALL_PATH in 01-run-chroot.sh

Known Safe Variables (Allowed)

• ROOTFS_DIR (pi-gen provided)
• SCRIPT_DIR (pi-gen provided)
• DEFAULT_USER (defined in our scripts)
• DEFAULT_HOME (defined in our scripts)
• Standard: PATH, HOME, USER

---

5. Disk Space Check (Section 13)

Problem It Solves

Build fails mid-download when disk fills up:

E: Failed to fetch ... No space left on device

How It Works

# Checks /home/work/pi-gen-builder available space
# Requires 15GB minimum
# Fails pre-flight if insufficient

Output Example

✓ Sufficient disk space: 47GB available (15GB required)

Or:

✗ ERROR: Insufficient disk space: 8GB available (15GB required)
    Pi-gen builds require ~15GB for image + cache
    Free up space or build will fail mid-download

Why 15GB?

• Base image: ~3GB
• Stage downloads: ~2GB
• QCOW2 cache: ~5GB
• Temporary files: ~2GB
• Safety margin: ~3GB

---

6. Network Connectivity Check (Section 14)

Problem It Solves

Network issues discovered 30 minutes into download phase:

Err:1 http://deb.debian.org/debian trixie InRelease
  Could not resolve 'deb.debian.org'

How It Works

# Tests each required repository:
curl -s http://deb.debian.org/debian/dists/trixie/Release
curl -s http://archive.raspberrypi.com/debian/dists/trixie/Release
curl -s https://github.com
git ls-remote https://github.com/RfidResearchGroup/proxmark3

Output Example

✓ Can reach deb.debian.org
✓ Can reach archive.raspberrypi.com
✓ Can reach github.com
✓ Proxmark3 repository is accessible

Or:

✗ ERROR: Cannot reach deb.debian.org (Debian package repository)
    Check internet connection or firewall

What It Prevents

• Starting 3+ hour build with no internet
• Discovering DNS issues after stage0 downloads
• GitHub access problems (SSH keys, firewall)
• Repository downtime/unavailability

---

Performance Impact

Pre-Enhancement

• Run time: 30 seconds
• Checks: 52
• False negatives: Common (missed errors)

Post-Enhancement

• Run time: 60-90 seconds
• Checks: 100+
• False negatives: Rare

Cost-benefit: 60 seconds validation vs 3+ hours wasted build = 180x ROI

---

Known Limitations

1. Package Validation

• Docker required: Needs Docker to spin up test container
• Network required: Must download package index
• Slow: Takes 30-60s depending on package count
• Workaround: Could cache results between runs

2. File Reference Validation

• Static paths only: Doesn't handle $VARIABLES in paths
• Outside chroot only: Can't validate files inside image
• Limited coverage: Only checks cp commands in 00-run.sh

3. Service File Validation

• Basic syntax only: Doesn't run systemd-analyze verify
• Template detection: May miss some placeholder patterns
• No runtime check: Can't verify service actually starts

4. Environment Variable Checks

• Conservative: May warn about legitimately inherited vars
• Whitelist-based: Must manually add known-safe variables
• No scope analysis: Doesn't track where variables are set

5. Disk Space Check

• Fixed threshold: 15GB may be too much or too little
• Doesn't track growth: Can't predict final image size
• Single check: Doesn't monitor during build

6. Network Connectivity

• Snapshot in time: Network may fail during build
• Timeout issues: 5s timeout may be too short for slow connections
• No bandwidth test: Doesn't measure actual speed

---

Future Enhancements

Priority 1: Immediate Value

• Cache package validation results (avoid re-querying same packages)
• Add Python package validation (check PyPI for pip3 install packages)
• Validate Debian package versions (some packages may be Trixie-specific)
• Test binary downloads (ttyd, other wget/curl downloads)

Priority 2: Nice to Have

• Run systemd-analyze verify (requires systemd in test container)
• Variable scope tracking (track where variables are defined)
• Disk space monitoring (warn if low during build)
• Network speed test (estimate build time based on bandwidth)

Priority 3: Advanced

• Parallel validation (run multiple checks concurrently)
• JSON output mode (for CI/CD integration)
• Fix suggestions (auto-suggest corrections for common errors)
• Incremental validation (only check changed files)

---

Usage

Run Full Validation

./scripts/preflight-check.sh

Expected Runtime

• Fast connection: 60-90 seconds
• Slow connection: 90-120 seconds

Exit Codes

• 0 - All checks passed or warnings only
• 1 - Errors detected, build will fail

Skip Sections (for debugging)

Not currently supported. To skip specific sections, comment them out in the script.

---

Testing Validation

Test Package Validation

# Add fake package to test script
echo 'apt-get install -y fake-package-xyz' >> /tmp/test.sh
# Run validation - should catch it

Test File Reference Validation

# Reference non-existent file
echo 'cp files/missing.txt .' >> pi-gen/stage*/*/00-run.sh
# Run validation - should warn

Test Service File Validation

# Create malformed service
cat > /tmp/broken.service << EOF
[Unit]
Description=Test
# Missing [Service] and [Install]
EOF
# Copy to systemd/ - should error

Test Disk Space Check

# Temporarily fill disk (BE CAREFUL!)
dd if=/dev/zero of=/tmp/fillfile bs=1G count=50
# Run validation - should error
rm /tmp/fillfile  # Clean up!

Test Network Check

# Temporarily block network (requires sudo)
sudo iptables -A OUTPUT -d deb.debian.org -j DROP
# Run validation - should error
sudo iptables -D OUTPUT -d deb.debian.org -j DROP  # Restore!

---

Key Takeaway

These 6 enhancements transform pre-flight from "basic sanity check" to "comprehensive build readiness validation"

Every minute spent on validation saves hours of wasted build time

With slow network (265 kB/s), this is not optional - it's survival