pi-pm3/DOCKER_QCOW2_SETUP.md

Docker + QCOW2 Setup Requirements

Critical: This document explains how to enable QCOW2 caching in pi-gen Docker builds. Skipping this will cost you hours of wasted build time.

---

The Problem

Pi-gen's USE_QCOW2=1 config option does not work out-of-the-box with Docker mode. If you don't set this up correctly:

❌ No .qcow2 snapshot files are created ❌ Builds cannot be cached or resumed ❌ Every build takes full 3+ hours, even after failures ❌ No incremental from-pm3 or from-dtpi builds

This problem cost us an 82-minute build when it failed at stagePM3 with zero cache preserved.

---

Root Cause

QCOW2 support requires:

1. qemu-utils package in Docker image (provides qemu-nbd)
2. NBD kernel module loaded on host system
3. /dev/nbd* devices available to Docker container

Pi-gen's default Dockerfile is missing qemu-utils, so QCOW2 silently fails.

---

The Fix (One-Time Setup)

Step 1: Load NBD Kernel Module on Host

Requires root/sudo access:

# Load NBD module
sudo modprobe nbd max_part=8

# Verify it loaded
lsmod | grep nbd
ls /dev/nbd*  # Should show /dev/nbd0, /dev/nbd1, etc.

Make it persistent across reboots:

# Auto-load on boot
echo "nbd" | sudo tee /etc/modules-load.d/nbd.conf

# Set max_part option
echo "options nbd max_part=8" | sudo tee /etc/modprobe.d/nbd.conf

Step 2: Fix Pi-Gen Dockerfile

Edit /home/work/pi-gen-builder/Dockerfile and add qemu-utils:

RUN apt-get -y update && \
    apt-get -y install --no-install-recommends \
        git vim parted \
        quilt coreutils qemu-user-static qemu-utils debootstrap zerofree zip dosfstools e2fsprogs\
        libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc \
        binfmt-support ca-certificates fdisk gpg pigz arch-test \
    && rm -rf /var/lib/apt/lists/*

Key change: Added qemu-utils after qemu-user-static on line 9.

Step 3: Rebuild Pi-Gen Image

cd /home/work/pi-gen-builder
docker build -t pi-gen .

Expected output:

Successfully built <image-id>
Successfully tagged pi-gen:latest

Step 4: Verify Setup

# Check NBD on host
lsmod | grep nbd

# Check qemu-nbd in container
docker run --rm pi-gen which qemu-nbd
# Should output: /usr/bin/qemu-nbd

---

Verification: QCOW2 Works

After setup, run a build and verify .qcow2 files are created:

# Start a build
./build-image.sh full

# In another terminal, check for .qcow2 files after stage0 completes
docker exec pigen_work find /pi-gen/work -name "*.qcow2"

# Should see files like:
# /pi-gen/work/Proxmark3/stage0/EXPORT_IMAGE.qcow2
# /pi-gen/work/Proxmark3/stage1/EXPORT_IMAGE.qcow2
# etc.

If you see .qcow2 files, caching is working!

---

What Happens Without This Setup

Symptom: Silent Failure

When QCOW2 is not properly configured:

1. Config has USE_QCOW2=1 ✅
2. Build appears to run normally ✅
3. But .qcow2 files are never created ❌
4. Build completes or fails with no cache ❌
5. from-pm3 and from-dtpi modes see "No cached stages found" ❌

Example: What We Experienced

$ ./build-image.sh full
# ... 82 minutes later ...
[21:14:41] Build failed

$ ls /home/work/pi-gen-builder/work/
# Nothing - directory doesn't exist on host (it's in Docker volume)

$ docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"
# No output - no .qcow2 files exist!

$ ./build-image.sh from-pm3
No cached stages found - will build all stages
# Another 3 hours wasted!

Total time wasted: 5+ hours across 2 builds Total data downloaded: 3.6GB+ (twice) Cause: Missing 10MB qemu-utils package and NBD module

---

Why NBD Module Is Required

QCOW2 (QEMU Copy-On-Write version 2) requires Network Block Device (NBD) to mount images:

1. qemu-nbd creates a virtual block device from .qcow2 file
2. NBD kernel module provides /dev/nbd* devices
3. Docker container accesses these devices from host kernel

Key point: Kernel modules cannot be loaded from inside Docker. They must be loaded on the host system by a privileged user.

---

Troubleshooting

Problem: "No cached stages found" after build

Diagnosis:

# Check if .qcow2 files exist in Docker volume
docker ps -a | grep pigen_work  # Get container ID
docker inspect <container-id> | grep -A20 "Mounts"  # Find volume ID
docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"

If no .qcow2 files found:

• NBD module not loaded on host
• qemu-utils not in Docker image
• QCOW2 creation failed silently

Problem: NBD module won't load

$ sudo modprobe nbd
modprobe: ERROR: could not insert 'nbd': Operation not permitted

Causes:

• Running in a VM or container without kernel module support
• Kernel doesn't have NBD compiled in
• Secure boot preventing module loading

Solutions:

• Check grep NBD /boot/config-$(uname -r) - should show CONFIG_BLK_DEV_NBD=m or =y
• If VM: Enable NBD in host kernel, not guest
• If container: Use host's Docker, not Docker-in-Docker

Problem: Build fails with "nbd: device nbd0 not found"

Cause: NBD module loaded but Docker container can't access /dev/nbd* devices

Solution:

• Ensure Docker has access to host devices (usually automatic)
• Try --privileged mode (pi-gen already uses this)
• Verify /dev/nbd* exist on host

---

Performance Impact

Without QCOW2 (Before Fix)

Build Type	Time	Downloads	Cache
Full build	3+ hours	1.8GB	None
After failure	3+ hours	1.8GB	None
Fix typo	3+ hours	1.8GB	None

Every change = 3+ hours + 1.8GB downloads

With QCOW2 (After Fix)

Build Type	Time	Downloads	Cache
Full build (first time)	3+ hours	1.8GB	stage0-2 cached
Rebuild from PM3	~70 min	~220MB	Reuses stage0-2
Rebuild from DTPI	~5 min	~140MB	Reuses stage0-PM3

ROI: 70-180 minutes saved per rebuild

---

Quick Setup Script

Save this as scripts/setup-docker-qcow2.sh:

#!/bin/bash
# One-time setup for QCOW2 support in pi-gen Docker builds
set -e

echo "=== Setting up QCOW2 support for pi-gen Docker builds ==="

# Check if running as root
if [ "$EUID" -ne 0 ]; then
    echo "ERROR: This script must be run as root (sudo)"
    echo "Usage: sudo ./scripts/setup-docker-qcow2.sh"
    exit 1
fi

# Load NBD module
echo "Loading NBD kernel module..."
modprobe nbd max_part=8

# Make persistent
echo "Making NBD module persistent..."
echo "nbd" > /etc/modules-load.d/nbd.conf
echo "options nbd max_part=8" > /etc/modprobe.d/nbd.conf

# Verify
if lsmod | grep -q nbd; then
    echo "✓ NBD module loaded successfully"
    echo "  Available devices:"
    ls /dev/nbd* | head -5
else
    echo "✗ ERROR: NBD module failed to load"
    exit 1
fi

echo ""
echo "✓ NBD setup complete!"
echo ""
echo "Next steps (run as regular user):"
echo "  1. Edit /home/work/pi-gen-builder/Dockerfile"
echo "  2. Add 'qemu-utils' to the apt-get install line"
echo "  3. Run: cd /home/work/pi-gen-builder && docker build -t pi-gen ."
echo "  4. Verify: docker run --rm pi-gen which qemu-nbd"
echo ""