otp-import/docs/qr-to-csv-design.md

# QR-to-CSV Utility — Design

Date: 2026-07-08
Status: Draft for review

## Goal and workflow

A companion utility that turns saved OTP QR-code screenshots into rows of the
canonical CSV that the importer consumes. The overall flow is a one-way
pipeline, not a hand-maintained file:

1. **QR -> CSV** (`qr_to_sheet.py`): decode QR images, deduplicate on the
   `otpauth://` data, and append only genuinely-new entries to the CSV.
2. **Optional user edits**: the user may adjust the *label* columns (issuer,
   account). They are instructed never to touch the `otpauth://` data.
3. **CSV -> applet** (`apex_otp_import.py`): push to the Apex, validating for
   human-introduced problems (duplicate names, invalid characters, overlength).

The two stages guard different risks: stage 1 prevents re-importing the same QR;
stage 3 catches bad edits. The `otpauth://` string is the immutable source of
truth throughout.

## Canonical format: CSV

Replaces the `.xlsx` as the canonical format (open, plain-text, git-diffable,
trivial to append and dedup). Minimal schema:

| Column | Editable? | Purpose |
| ------ | --------- | ------- |
| `issuer` | yes (label) | Issuer shown on the applet; seeded from the otpauth. |
| `account` | yes (label) | Account shown on the applet; seeded from the otpauth label. |
| `otpauth_uri` | **no** | The raw decoded `otpauth://` string. Dedup key (stage 1) and crypto source (stage 3). |
| `source_file` | informational | Filename the QR was decoded from. |

The secret remains visible (inside `otpauth_uri`), so nothing is hidden. The
credential name written to the applet is `issuer:account` from the editable
columns; the secret/type/algorithm/digits/period come from parsing
`otpauth_uri`.

A one-time conversion turns the existing `otp-keys-export.xlsx` "OTP Keys" sheet
into `otp-keys.csv` (issuer, account, otpauth_uri, source_file), so no data is
lost and the tools operate on the CSV going forward.

## Shared module: `otpauth.py`

Used by both tools:

- `parse_otpauth(uri) -> OtpAuth`: parse `otpauth://TYPE/LABEL?params` into
  `type` (totp/hotp), `issuer` (from the `issuer` param, else the label prefix
  before `:`), `account` (label after the `issuer:` prefix, else the whole
  label), `secret` (uppercased, spaces removed), `algorithm` (default SHA1),
  `digits` (default 6), `period` (default 30). Percent-decodes label/issuer.
- `canonical_key(otp) -> tuple`: a normalized dedup key
  `(type, issuer, account, secret, algorithm, digits, period)` with defaults
  filled and the secret uppercased. Two scans of the same credential — even with
  the issuer given via the label vs. the `issuer` param — produce the same key;
  a different secret produces a different key.
- `build_otpauth(...) -> str`: reconstruct a URI (used by the xlsx conversion
  for any legacy row lacking a stored URI).

## `qr_to_sheet.py`

CLI: `qr_to_sheet.py [--images DIR] [--csv PATH] [--dry-run]`

1. Enumerate QR images in `--images` (default: the data folder) — `*.png`,
   `*.jpg`.
2. Decode each with OpenCV using the robust multi-pass strategy (direct ->
   grayscale -> upscale 2x/3x/4x/0.5x -> Otsu/adaptive threshold).
3. `parse_otpauth` each decoded `otpauth://totp/...`. An `otpauth-migration://`
   payload is reported as unsupported (out of scope), not parsed.
4. Load the existing CSV (if any); build the set of `canonical_key` values from
   every row's `otpauth_uri`.
5. For each decoded QR: if its `canonical_key` is already present (in the CSV or
   already added this run), skip it as a duplicate; otherwise append a row
   (`issuer`, `account` from the parse; `otpauth_uri` = the raw decoded string;
   `source_file` = image filename).
6. Never rewrite or reorder existing rows — append only.
7. Print a summary: added, skipped-as-duplicate, and **undecodable** images
   (stylized/colored QRs or text-only screenshots) listed by filename for
   manual entry. `--dry-run` reports without writing.

## Importer changes (`apex_otp_import.py`)

- `load_credentials` accepts CSV or XLSX, chosen by file extension.
- Crypto (secret/type/algorithm/digits/period) is derived by
  `parse_otpauth(otpauth_uri)` when an `otpauth_uri` is present; `issuer` and
  `account` come from their columns. When no `otpauth_uri` is present (legacy
  xlsx), it falls back to the individual crypto columns as today.
- `validate_input` is unchanged in intent and still catches the stage-3 risks
  (duplicate `issuer:account`, control/`:`/`/` characters, overlength names,
  bad digits/period, invalid Base32, unsupported algorithm).

## Testing (hardware-free, TDD)

- `parse_otpauth`: issuer from param vs. label prefix; percent-decoding;
  defaults for algorithm/digits/period; account extraction.
- `canonical_key`: equal for label-issuer vs. param-issuer forms of the same
  credential; differs when the secret differs; secret case-insensitive.
- Dedup: given existing CSV rows and a batch of candidates, only the new
  canonical keys are appended; intra-run duplicates collapse.
- CSV read/append round-trip; append preserves existing rows verbatim.
- `load_credentials` from CSV: crypto comes from `otpauth_uri`, labels from the
  editable columns; legacy fallback path still works.

QR image decoding is the untestable I/O edge, validated against the real PNGs.

## Out of scope (YAGNI)

`otpauth-migration://` (Google Authenticator batch export protobuf) — reported
as unsupported; OCR of text-only screenshots; editing or removing existing rows;
backup/recovery `.txt` codes.

## Files (in the `otp-import` repo)

New: `qr_to_sheet.py`, `otpauth.py`, `test_qr.py` (parse/canonical/dedup/CSV
tests), `docs/qr-to-csv-design.md`. Changed: `apex_otp_import.py`
(`load_credentials` CSV + otpauth-authoritative), `test_oath.py` (CSV-load
cases), `README.md`. One-time output: `otp-keys.csv` (git-ignored, lives with
the private data, not in the repo).