pi-pm3/DOCKER_QCOW2_SETUP.md

# Docker + QCOW2 Setup Requirements

**Critical:** This document explains how to enable QCOW2 caching in pi-gen Docker builds. **Skipping this will cost you hours of wasted build time.**

---

## The Problem

Pi-gen's `USE_QCOW2=1` config option **does not work out-of-the-box** with Docker mode. If you don't set this up correctly:

❌ **No .qcow2 snapshot files are created**
❌ **Builds cannot be cached or resumed**
❌ **Every build takes full 3+ hours, even after failures**
❌ **No incremental `from-pm3` or `from-dtpi` builds**

**This problem cost us an 82-minute build** when it failed at stagePM3 with zero cache preserved.

---

## Root Cause

QCOW2 support requires:
1. `qemu-utils` package in Docker image (provides `qemu-nbd`)
2. NBD kernel module loaded on **host system**
3. `/dev/nbd*` devices available to Docker container

**Pi-gen's default Dockerfile is missing `qemu-utils`**, so QCOW2 silently fails.

---

## The Fix (One-Time Setup)

### Step 1: Load NBD Kernel Module on Host

**Requires root/sudo access:**

```bash
# Load NBD module
sudo modprobe nbd max_part=8

# Verify it loaded
lsmod | grep nbd
ls /dev/nbd*  # Should show /dev/nbd0, /dev/nbd1, etc.
```

**Make it persistent across reboots:**

```bash
# Auto-load on boot
echo "nbd" | sudo tee /etc/modules-load.d/nbd.conf

# Set max_part option
echo "options nbd max_part=8" | sudo tee /etc/modprobe.d/nbd.conf
```

### Step 2: Fix Pi-Gen Dockerfile

Edit `/home/work/pi-gen-builder/Dockerfile` and add `qemu-utils`:

```dockerfile
RUN apt-get -y update && \
    apt-get -y install --no-install-recommends \
        git vim parted \
        quilt coreutils qemu-user-static qemu-utils debootstrap zerofree zip dosfstools e2fsprogs\
        libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc \
        binfmt-support ca-certificates fdisk gpg pigz arch-test \
    && rm -rf /var/lib/apt/lists/*
```

**Key change:** Added `qemu-utils` after `qemu-user-static` on line 9.

### Step 3: Rebuild Pi-Gen Image

```bash
cd /home/work/pi-gen-builder
docker build -t pi-gen .
```

**Expected output:**
```
Successfully built <image-id>
Successfully tagged pi-gen:latest
```

### Step 4: Verify Setup

```bash
# Check NBD on host
lsmod | grep nbd

# Check qemu-nbd in container
docker run --rm pi-gen which qemu-nbd
# Should output: /usr/bin/qemu-nbd
```

---

## Verification: QCOW2 Works

After setup, run a build and verify .qcow2 files are created:

```bash
# Start a build
./build-image.sh full

# In another terminal, check for .qcow2 files after stage0 completes
docker exec pigen_work find /pi-gen/work -name "*.qcow2"

# Should see files like:
# /pi-gen/work/Proxmark3/stage0/EXPORT_IMAGE.qcow2
# /pi-gen/work/Proxmark3/stage1/EXPORT_IMAGE.qcow2
# etc.
```

If you see .qcow2 files, **caching is working!**

---

## What Happens Without This Setup

### Symptom: Silent Failure

When QCOW2 is not properly configured:
1. Config has `USE_QCOW2=1` ✅
2. Build appears to run normally ✅
3. **But .qcow2 files are never created** ❌
4. Build completes or fails with no cache ❌
5. `from-pm3` and `from-dtpi` modes see "No cached stages found" ❌

### Example: What We Experienced

```bash
$ ./build-image.sh full
# ... 82 minutes later ...
[21:14:41] Build failed

$ ls /home/work/pi-gen-builder/work/
# Nothing - directory doesn't exist on host (it's in Docker volume)

$ docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"
# No output - no .qcow2 files exist!

$ ./build-image.sh from-pm3
No cached stages found - will build all stages
# Another 3 hours wasted!
```

**Total time wasted:** 5+ hours across 2 builds
**Total data downloaded:** 3.6GB+ (twice)
**Cause:** Missing 10MB `qemu-utils` package and NBD module

---

## Why NBD Module Is Required

QCOW2 (QEMU Copy-On-Write version 2) requires Network Block Device (NBD) to mount images:

1. **qemu-nbd** creates a virtual block device from .qcow2 file
2. **NBD kernel module** provides /dev/nbd* devices
3. **Docker container** accesses these devices from host kernel

**Key point:** Kernel modules **cannot** be loaded from inside Docker. They must be loaded on the host system by a privileged user.

---

## Troubleshooting

### Problem: "No cached stages found" after build

**Diagnosis:**
```bash
# Check if .qcow2 files exist in Docker volume
docker ps -a | grep pigen_work  # Get container ID
docker inspect <container-id> | grep -A20 "Mounts"  # Find volume ID
docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"
```

**If no .qcow2 files found:**
- NBD module not loaded on host
- `qemu-utils` not in Docker image
- QCOW2 creation failed silently

### Problem: NBD module won't load

```bash
$ sudo modprobe nbd
modprobe: ERROR: could not insert 'nbd': Operation not permitted
```

**Causes:**
- Running in a VM or container without kernel module support
- Kernel doesn't have NBD compiled in
- Secure boot preventing module loading

**Solutions:**
- Check `grep NBD /boot/config-$(uname -r)` - should show `CONFIG_BLK_DEV_NBD=m` or `=y`
- If VM: Enable NBD in host kernel, not guest
- If container: Use host's Docker, not Docker-in-Docker

### Problem: Build fails with "nbd: device nbd0 not found"

**Cause:** NBD module loaded but Docker container can't access /dev/nbd* devices

**Solution:**
- Ensure Docker has access to host devices (usually automatic)
- Try `--privileged` mode (pi-gen already uses this)
- Verify `/dev/nbd*` exist on host

---

## Performance Impact

### Without QCOW2 (Before Fix)

| Build Type | Time | Downloads | Cache |
|------------|------|-----------|-------|
| **Full build** | 3+ hours | 1.8GB | None |
| **After failure** | 3+ hours | 1.8GB | None |
| **Fix typo** | 3+ hours | 1.8GB | None |

**Every change = 3+ hours + 1.8GB downloads**

### With QCOW2 (After Fix)

| Build Type | Time | Downloads | Cache |
|------------|------|-----------|-------|
| **Full build (first time)** | 3+ hours | 1.8GB | stage0-2 cached |
| **Rebuild from PM3** | ~70 min | ~220MB | Reuses stage0-2 |
| **Rebuild from DTPI** | ~5 min | ~140MB | Reuses stage0-PM3 |

**ROI: 70-180 minutes saved per rebuild**

---

## Quick Setup Script

Save this as `scripts/setup-docker-qcow2.sh`:

```bash
#!/bin/bash
# One-time setup for QCOW2 support in pi-gen Docker builds
set -e

echo "=== Setting up QCOW2 support for pi-gen Docker builds ==="

# Check if running as root
if [ "$EUID" -ne 0 ]; then
    echo "ERROR: This script must be run as root (sudo)"
    echo "Usage: sudo ./scripts/setup-docker-qcow2.sh"
    exit 1
fi

# Load NBD module
echo "Loading NBD kernel module..."
modprobe nbd max_part=8

# Make persistent
echo "Making NBD module persistent..."
echo "nbd" > /etc/modules-load.d/nbd.conf
echo "options nbd max_part=8" > /etc/modprobe.d/nbd.conf

# Verify
if lsmod | grep -q nbd; then
    echo "✓ NBD module loaded successfully"
    echo "  Available devices:"
    ls /dev/nbd* | head -5
else
    echo "✗ ERROR: NBD module failed to load"
    exit 1
fi

echo ""
echo "✓ NBD setup complete!"
echo ""
echo "Next steps (run as regular user):"
echo "  1. Edit /home/work/pi-gen-builder/Dockerfile"
echo "  2. Add 'qemu-utils' to the apt-get install line"
echo "  3. Run: cd /home/work/pi-gen-builder && docker build -t pi-gen ."
echo "  4. Verify: docker run --rm pi-gen which qemu-nbd"
echo ""