Initial commit - Phase 3/4
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
michael
280
DOCKER_QCOW2_SETUP.md
Normal file
280
DOCKER_QCOW2_SETUP.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Docker + QCOW2 Setup Requirements
|
||||
|
||||
**Critical:** This document explains how to enable QCOW2 caching in pi-gen Docker builds. **Skipping this will cost you hours of wasted build time.**
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
Pi-gen's `USE_QCOW2=1` config option **does not work out-of-the-box** with Docker mode. If you don't set this up correctly:
|
||||
|
||||
❌ **No .qcow2 snapshot files are created**
|
||||
❌ **Builds cannot be cached or resumed**
|
||||
❌ **Every build takes full 3+ hours, even after failures**
|
||||
❌ **No incremental `from-pm3` or `from-dtpi` builds**
|
||||
|
||||
**This problem cost us an 82-minute build** when it failed at stagePM3 with zero cache preserved.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
QCOW2 support requires:
|
||||
1. `qemu-utils` package in Docker image (provides `qemu-nbd`)
|
||||
2. NBD kernel module loaded on **host system**
|
||||
3. `/dev/nbd*` devices available to Docker container
|
||||
|
||||
**Pi-gen's default Dockerfile is missing `qemu-utils`**, so QCOW2 silently fails.
|
||||
|
||||
---
|
||||
|
||||
## The Fix (One-Time Setup)
|
||||
|
||||
### Step 1: Load NBD Kernel Module on Host
|
||||
|
||||
**Requires root/sudo access:**
|
||||
|
||||
```bash
|
||||
# Load NBD module
|
||||
sudo modprobe nbd max_part=8
|
||||
|
||||
# Verify it loaded
|
||||
lsmod | grep nbd
|
||||
ls /dev/nbd* # Should show /dev/nbd0, /dev/nbd1, etc.
|
||||
```
|
||||
|
||||
**Make it persistent across reboots:**
|
||||
|
||||
```bash
|
||||
# Auto-load on boot
|
||||
echo "nbd" | sudo tee /etc/modules-load.d/nbd.conf
|
||||
|
||||
# Set max_part option
|
||||
echo "options nbd max_part=8" | sudo tee /etc/modprobe.d/nbd.conf
|
||||
```
|
||||
|
||||
### Step 2: Fix Pi-Gen Dockerfile
|
||||
|
||||
Edit `/home/work/pi-gen-builder/Dockerfile` and add `qemu-utils`:
|
||||
|
||||
```dockerfile
|
||||
RUN apt-get -y update && \
|
||||
apt-get -y install --no-install-recommends \
|
||||
git vim parted \
|
||||
quilt coreutils qemu-user-static qemu-utils debootstrap zerofree zip dosfstools e2fsprogs\
|
||||
libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc \
|
||||
binfmt-support ca-certificates fdisk gpg pigz arch-test \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
```
|
||||
|
||||
**Key change:** Added `qemu-utils` after `qemu-user-static` on line 9.
|
||||
|
||||
### Step 3: Rebuild Pi-Gen Image
|
||||
|
||||
```bash
|
||||
cd /home/work/pi-gen-builder
|
||||
docker build -t pi-gen .
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
Successfully built <image-id>
|
||||
Successfully tagged pi-gen:latest
|
||||
```
|
||||
|
||||
### Step 4: Verify Setup
|
||||
|
||||
```bash
|
||||
# Check NBD on host
|
||||
lsmod | grep nbd
|
||||
|
||||
# Check qemu-nbd in container
|
||||
docker run --rm pi-gen which qemu-nbd
|
||||
# Should output: /usr/bin/qemu-nbd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification: QCOW2 Works
|
||||
|
||||
After setup, run a build and verify .qcow2 files are created:
|
||||
|
||||
```bash
|
||||
# Start a build
|
||||
./build-image.sh full
|
||||
|
||||
# In another terminal, check for .qcow2 files after stage0 completes
|
||||
docker exec pigen_work find /pi-gen/work -name "*.qcow2"
|
||||
|
||||
# Should see files like:
|
||||
# /pi-gen/work/Proxmark3/stage0/EXPORT_IMAGE.qcow2
|
||||
# /pi-gen/work/Proxmark3/stage1/EXPORT_IMAGE.qcow2
|
||||
# etc.
|
||||
```
|
||||
|
||||
If you see .qcow2 files, **caching is working!**
|
||||
|
||||
---
|
||||
|
||||
## What Happens Without This Setup
|
||||
|
||||
### Symptom: Silent Failure
|
||||
|
||||
When QCOW2 is not properly configured:
|
||||
1. Config has `USE_QCOW2=1` ✅
|
||||
2. Build appears to run normally ✅
|
||||
3. **But .qcow2 files are never created** ❌
|
||||
4. Build completes or fails with no cache ❌
|
||||
5. `from-pm3` and `from-dtpi` modes see "No cached stages found" ❌
|
||||
|
||||
### Example: What We Experienced
|
||||
|
||||
```bash
|
||||
$ ./build-image.sh full
|
||||
# ... 82 minutes later ...
|
||||
[21:14:41] Build failed
|
||||
|
||||
$ ls /home/work/pi-gen-builder/work/
|
||||
# Nothing - directory doesn't exist on host (it's in Docker volume)
|
||||
|
||||
$ docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"
|
||||
# No output - no .qcow2 files exist!
|
||||
|
||||
$ ./build-image.sh from-pm3
|
||||
No cached stages found - will build all stages
|
||||
# Another 3 hours wasted!
|
||||
```
|
||||
|
||||
**Total time wasted:** 5+ hours across 2 builds
|
||||
**Total data downloaded:** 3.6GB+ (twice)
|
||||
**Cause:** Missing 10MB `qemu-utils` package and NBD module
|
||||
|
||||
---
|
||||
|
||||
## Why NBD Module Is Required
|
||||
|
||||
QCOW2 (QEMU Copy-On-Write version 2) requires Network Block Device (NBD) to mount images:
|
||||
|
||||
1. **qemu-nbd** creates a virtual block device from .qcow2 file
|
||||
2. **NBD kernel module** provides /dev/nbd* devices
|
||||
3. **Docker container** accesses these devices from host kernel
|
||||
|
||||
**Key point:** Kernel modules **cannot** be loaded from inside Docker. They must be loaded on the host system by a privileged user.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "No cached stages found" after build
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check if .qcow2 files exist in Docker volume
|
||||
docker ps -a | grep pigen_work # Get container ID
|
||||
docker inspect <container-id> | grep -A20 "Mounts" # Find volume ID
|
||||
docker run --rm -v <volume-id>:/work alpine find /work -name "*.qcow2"
|
||||
```
|
||||
|
||||
**If no .qcow2 files found:**
|
||||
- NBD module not loaded on host
|
||||
- `qemu-utils` not in Docker image
|
||||
- QCOW2 creation failed silently
|
||||
|
||||
### Problem: NBD module won't load
|
||||
|
||||
```bash
|
||||
$ sudo modprobe nbd
|
||||
modprobe: ERROR: could not insert 'nbd': Operation not permitted
|
||||
```
|
||||
|
||||
**Causes:**
|
||||
- Running in a VM or container without kernel module support
|
||||
- Kernel doesn't have NBD compiled in
|
||||
- Secure boot preventing module loading
|
||||
|
||||
**Solutions:**
|
||||
- Check `grep NBD /boot/config-$(uname -r)` - should show `CONFIG_BLK_DEV_NBD=m` or `=y`
|
||||
- If VM: Enable NBD in host kernel, not guest
|
||||
- If container: Use host's Docker, not Docker-in-Docker
|
||||
|
||||
### Problem: Build fails with "nbd: device nbd0 not found"
|
||||
|
||||
**Cause:** NBD module loaded but Docker container can't access /dev/nbd* devices
|
||||
|
||||
**Solution:**
|
||||
- Ensure Docker has access to host devices (usually automatic)
|
||||
- Try `--privileged` mode (pi-gen already uses this)
|
||||
- Verify `/dev/nbd*` exist on host
|
||||
|
||||
---
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Without QCOW2 (Before Fix)
|
||||
|
||||
| Build Type | Time | Downloads | Cache |
|
||||
|------------|------|-----------|-------|
|
||||
| **Full build** | 3+ hours | 1.8GB | None |
|
||||
| **After failure** | 3+ hours | 1.8GB | None |
|
||||
| **Fix typo** | 3+ hours | 1.8GB | None |
|
||||
|
||||
**Every change = 3+ hours + 1.8GB downloads**
|
||||
|
||||
### With QCOW2 (After Fix)
|
||||
|
||||
| Build Type | Time | Downloads | Cache |
|
||||
|------------|------|-----------|-------|
|
||||
| **Full build (first time)** | 3+ hours | 1.8GB | stage0-2 cached |
|
||||
| **Rebuild from PM3** | ~70 min | ~220MB | Reuses stage0-2 |
|
||||
| **Rebuild from DTPI** | ~5 min | ~140MB | Reuses stage0-PM3 |
|
||||
|
||||
**ROI: 70-180 minutes saved per rebuild**
|
||||
|
||||
---
|
||||
|
||||
## Quick Setup Script
|
||||
|
||||
Save this as `scripts/setup-docker-qcow2.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# One-time setup for QCOW2 support in pi-gen Docker builds
|
||||
set -e
|
||||
|
||||
echo "=== Setting up QCOW2 support for pi-gen Docker builds ==="
|
||||
|
||||
# Check if running as root
|
||||
if [ "$EUID" -ne 0 ]; then
|
||||
echo "ERROR: This script must be run as root (sudo)"
|
||||
echo "Usage: sudo ./scripts/setup-docker-qcow2.sh"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Load NBD module
|
||||
echo "Loading NBD kernel module..."
|
||||
modprobe nbd max_part=8
|
||||
|
||||
# Make persistent
|
||||
echo "Making NBD module persistent..."
|
||||
echo "nbd" > /etc/modules-load.d/nbd.conf
|
||||
echo "options nbd max_part=8" > /etc/modprobe.d/nbd.conf
|
||||
|
||||
# Verify
|
||||
if lsmod | grep -q nbd; then
|
||||
echo "✓ NBD module loaded successfully"
|
||||
echo " Available devices:"
|
||||
ls /dev/nbd* | head -5
|
||||
else
|
||||
echo "✗ ERROR: NBD module failed to load"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "✓ NBD setup complete!"
|
||||
echo ""
|
||||
echo "Next steps (run as regular user):"
|
||||
echo " 1. Edit /home/work/pi-gen-builder/Dockerfile"
|
||||
echo " 2. Add 'qemu-utils' to the apt-get install line"
|
||||
echo " 3. Run: cd /home/work/pi-gen-builder && docker build -t pi-gen ."
|
||||
echo " 4. Verify: docker run --rm pi-gen which qemu-nbd"
|
||||
echo ""
|
||||
Reference in New Issue
Block a user