otp-import/README.md

# otp-import

Bulk-load your saved authenticator **QR-code screenshots** onto a **VivoKey
Apex** OTP applet (the ykneo-oath applet) — or onto several implants at once — by
way of a plain, human-editable **CSV**.

If you screenshot your two-factor QR codes as you enroll them (see
[Backing up your OTP keys](#backing-up-your-otp-keys) below), these two tools
turn that pile of screenshots into working TOTP credentials on a new or
replacement authenticator, and can load the identical set onto multiple devices.

- `qr_to_sheet.py` — decode QR screenshots into the CSV (no hardware needed).
- `apex_otp_import.py` — write the CSV's credentials onto the applet over PC/SC.

**Duplicates are handled at both stages, so you can re-run freely:**

- **QR → CSV** deduplicates on the `otpauth://` data: re-scanning a folder only
  appends codes that aren't already in the CSV; ones already present are silently
  skipped. Two screenshots of the *same account with different secrets* (e.g. a
  re-enrollment) are both kept — they are genuinely different credentials.
- **CSV → applet** keys credentials by name (`issuer:account`): if a name is
  already on the applet you are asked to overwrite or skip it (or pass
  `--overwrite-all` / `--skip-existing`). Loading the same CSV onto a second
  implant just repeats the write. And before writing, the importer halts if two
  rows would collide on the same name — so a re-enrollment with a new secret
  can't silently clobber the old one; you relabel one of them first.

## Backing up your OTP keys

This utility is only useful if you saved your OTP keys in the first place. When
you turn on TOTP two-factor authentication somewhere, the site shows a QR code
(and usually a "can't scan it? enter this code instead" secret). That QR encodes
an `otpauth://` URI containing the shared secret — capture it **then**, because
you generally cannot retrieve it later without disabling and re-enrolling 2FA.

A good habit, every time you enable 2FA:

1. **Screenshot the QR code** and save the image to secure storage. The tools
   read the `otpauth://` data inside the image, not the filename, so the naming
   scheme is purely for your own organization — but a consistent one keeps the
   folder browsable and sortable. A practical convention is
   `YYYY-MM-DD-<relying-party>-<account>`:

   ```
   2024-05-01-github-alice@example.com.png
   2025-11-02-aws-root@example.com.png
   ```

   Date first so the folder sorts chronologically, then the service (relying
   party), then which account it is.

2. **Save the one-time backup/recovery codes too.** Most sites offer a set of
   single-use recovery codes next to the QR — copy them into a text file beside
   the screenshot (e.g. `2024-05-01-github-alice@example.com-backup-codes.txt`).
   If you ever lose every authenticator, those codes are how you get back in.
   (This tool doesn't process them; they're your independent recovery path.)

3. **Keep it all in secure storage.** These files contain live secrets — treat
   them like passwords. Good homes: a password-manager vault, an encrypted
   volume, or an end-to-end-encrypted cloud folder. Never commit them to a
   repository (this repo's `.gitignore` deliberately excludes `*.png`, `*.csv`,
   and backup-code text files).

With that habit in place, provisioning a new, replacement, or spare Apex is just
pointing these tools at the folder.

---

## The pipeline

```
  QR screenshots  (*.png / *.jpg, each holding an otpauth:// URI)
       |
       |   (1) qr_to_sheet.py
       |       decode each QR; append only NEW otpauth:// data to the CSV
       v
  otp-keys.csv    columns:  issuer | account | otpauth_uri | source_file
       |
       |   (2) optional: edit the issuer / account labels (never otpauth_uri)
       |
       |   (3) apex_otp_import.py
       |       validate the CSV, authenticate if needed, PUT each credential
       v
  Apex OTP applet (ykneo-oath)    one TOTP credential per row
```

1. **QR → CSV** (`qr_to_sheet.py`): decode each screenshot and append any
   **new** credential to the CSV. Deduplication is on the `otpauth://` data, so
   re-running never adds a credential that is already present.
2. **Edit labels (optional)**: adjust the `issuer` and `account` columns to
   taste. **Never edit the `otpauth_uri` column** — it is the immutable source
   of truth (secret and parameters live inside it).
3. **CSV → applet** (`apex_otp_import.py`): validate the CSV and write each
   credential onto the applet, authenticating first if the applet has a
   password.

The two stages guard different things: stage 1 stops you re-importing the same
QR; stage 3 stops you writing something you broke while editing labels.

---

## The CSV

Four columns:

| Column | Edit it? | Meaning |
| ------ | -------- | ------- |
| `issuer` | yes | Issuer shown on the applet (the "Issuer:" part of the name). |
| `account` | yes | Account shown on the applet (the part after "Issuer:"). |
| `otpauth_uri` | **no** | The raw `otpauth://` string from the QR. Dedup key and the source of the secret/algorithm/digits/period. |
| `source_file` | reference | The image filename the row was decoded from. |

The credential **name** written to the applet is `issuer:account` from the
editable columns. The **secret** and other crypto parameters are always taken
from `otpauth_uri`, so editing a label can never corrupt the code that gets
generated. The secret is still visible — it lives inside `otpauth_uri`.

The CSV is disposable: it can be regenerated from the QR screenshots at any time
with `qr_to_sheet.py`.

---

## Requirements

- Python 3.9+
- `pyscard`, `cryptography`, `openpyxl` (for reading legacy `.xlsx`), and
  `opencv-python` (for QR decoding).
- For writing to a card: a PC/SC reader with the Apex (over NFC) or a JavaCard
  carrying the applet presented to it.

```
pip install pyscard cryptography openpyxl opencv-python
```

---

## Tool 1 — `qr_to_sheet.py` (QR → CSV, no card)

Decode a folder of QR screenshots and append new credentials to the CSV. This
tool never touches a reader, so run it any time — the Apex does not need to be
present.

```
python qr_to_sheet.py --images <folder> --csv <path/to/otp-keys.csv>
```

Options:

| Option | Default | Meaning |
| ------ | ------- | ------- |
| `--images DIR` | `.` | Folder of QR images (`*.png`, `*.jpg`, `*.jpeg`). |
| `--csv PATH` | `otp-keys.csv` | The CSV to create or append to. |
| `--dry-run` | off | Report what would be added; write nothing. |

What it does per run:

- Decodes each image (a multi-pass decode copes with scaling/thresholding).
- Parses the `otpauth://totp/...` URI; seeds `issuer`/`account` from it.
- Skips any credential whose `otpauth://` data is already in the CSV
  (deduplicated, existing rows never touched), and any duplicate within the same
  run.
- Appends the genuinely-new rows and prints a summary of added / skipped /
  undecodable.

Example:

```
$ python qr_to_sheet.py --images ./screenshots --csv otp-keys.csv
Scanned 34 image(s): 32 otpauth QR(s) decoded.
  3 new, 29 duplicate(s) skipped.
  could not be decoded (add manually): coinbase-...png, plaid-...png
Appended 3 row(s) to otp-keys.csv.
```

**Undecodable images** — stylized/colored QR codes, or screenshots that only
show the secret as text — are listed by filename. Add those by hand: create a
row with `issuer`, `account`, `source_file`, and an `otpauth_uri` you build from
the printed secret, e.g.
`otpauth://totp/Coinbase:you@example.com?secret=THESECRET&issuer=Coinbase`.

`otpauth-migration://` payloads (Google Authenticator's batch-export QR) are
reported as unsupported, not decoded.

---

## Tool 2 — `apex_otp_import.py` (CSV → applet)

Validate the CSV and write its credentials onto the applet. Needs the Apex on a
reader (except `--dry-run`).

```
python apex_otp_import.py --input otp-keys.csv [options]
```

Options:

| Option | Meaning |
| ------ | ------- |
| `--input PATH` | CSV (or legacy `.xlsx`) to import. Default `otp-keys.csv`. |
| `--reader SUBSTR` | Pick a reader by name substring (e.g. `VivoKey`, `3700 F CL`). Omit to auto-detect the single card holding an OATH applet; you are prompted if more than one is present. |
| `--dry-run` | Validate and print the APDUs that would be sent; no card I/O. |
| `--list` | Read-only: list the credentials already on the applet. |
| `--overwrite-all` | Overwrite any credential that already exists (non-interactive). |
| `--skip-existing` | Skip any credential that already exists (non-interactive). |
| `--password PASS` | Applet password (otherwise prompted if the applet is locked). |
| `--profile-memory` | Record per-credential EEPROM consumption to a CSV log (needs the `javacard-memory` applet). |
| `--set-password PASS` | After a successful import, set an access password — only if the applet has none. |
| `--reset` | Standalone: wipe all credentials and any password, then exit. |
| `--import-qr` | Run the QR → CSV step first (see below), then import. |
| `--qr-images DIR` | QR folder for `--import-qr` (default: the folder of `--input`). |
| `--memlog PATH` | Path for the `--profile-memory` log (default `apex-otp-memlog.csv`). |

By default, when a credential name already exists on the applet you are asked
whether to overwrite or skip it (with all/none shortcuts). Use `--overwrite-all`
or `--skip-existing` for unattended runs.

Recommended sequence for a first import:

```
python apex_otp_import.py --input otp-keys.csv --dry-run           # (1) preview, no card
python apex_otp_import.py --input otp-keys.csv --list              # (2) see what's on the applet
python apex_otp_import.py --input otp-keys.csv                     # (3) import for real
python apex_otp_import.py --input otp-keys.csv --list              # (4) confirm
```

### Passwords

Authenticating to a password-protected applet uses the standard YKOATH scheme:
PBKDF2-HMAC-SHA1 of the password with the applet's device id as salt, then a
mutual challenge-response. `--set-password` sets a password only on an applet
that has none; it does not change or remove an existing one.

Note: unlike a YubiKey, this applet requires authentication before `--reset`
when a password is set. If you set a password and forget it, the OATH applet
cannot be reset over the OATH protocol — it must be deleted/reinstalled with
GlobalPlatform (card-manager keys).

### Memory profiling

`--profile-memory` measures how much persistent memory each credential consumes
by reading the `javacard-memory` applet before and after each `PUT`. Each row of
the log records the timestamp, card UID, memory-applet batch id, credential
name (never the secret), and bytes consumed. It is slower (a memory read per
credential) and off by default.

---

## The three modes

| Command | Does | Card needed |
| ------- | ---- | ----------- |
| `qr_to_sheet.py --images D --csv C` | QR → CSV | no |
| `apex_otp_import.py --input C` | CSV → applet | yes |
| `apex_otp_import.py --input C --import-qr` | QR → CSV → applet (one shot) | yes |

`--import-qr` is the whole pipeline in one command: it decodes new QRs into the
CSV (deduped on `otpauth://`, existing entries silently skipped), then validates
and imports. The QR folder defaults to the CSV's folder; override with
`--qr-images`. If you only want to refresh the CSV, use `qr_to_sheet.py` — the
Apex does not need to be on a reader.

```
# whole pipeline, single command
python apex_otp_import.py --input otp-keys.csv --import-qr --reader VivoKey

# same, with an explicit QR folder and unattended overwrite
python apex_otp_import.py --input otp-keys.csv --import-qr --qr-images ./screenshots --overwrite-all
```

---

## Common workflows

**Add newly-enrolled services.** Save the new QR screenshots into your folder,
then:

```
python qr_to_sheet.py --images ./screenshots --csv otp-keys.csv   # appends only the new ones
# (edit issuer/account labels if you like)
python apex_otp_import.py --input otp-keys.csv --reader VivoKey --overwrite-all
```

**Re-provision a fresh/wiped Apex from scratch.**

```
python apex_otp_import.py --input otp-keys.csv --reader VivoKey
```

**Password-protect after loading.**

```
python apex_otp_import.py --input otp-keys.csv --reader VivoKey --set-password 'mypass'
```

**Wipe an applet (know the password if it has one).**

```
python apex_otp_import.py --reset --reader VivoKey [--password 'mypass']
```

---

## Validation (stage 3)

The CSV is validated before any reader is touched; the run halts with a per-row
report if it finds any of:

- Duplicate credential name (`issuer:account`).
- A secret that is not valid Base32.
- A decoded key longer than 64 bytes, or a composed name longer than 64 bytes.
- An unsupported algorithm (the applet accepts SHA1/SHA256, not SHA512).
- Control characters, or a literal `:` or `/`, inside an issuer or account
  (they collide with the OATH `issuer:account` separator and `period/` prefix).
- A digit count outside 6–8, or a non-positive period.

Fix the offending cells and re-run. The applet itself only enforces the length
limits; the character and digit/period rules are host-side guards for
OATH-convention and authenticator-app compatibility.

---

## Security

This tooling handles live TOTP secrets. **Keep your secrets out of this
repository.** `otp-keys.csv`, any `.xlsx`, the memory log, and QR screenshots
are excluded by `.gitignore`; keep the real data in a private location (e.g. a
synced personal folder) and point the tools at it with `--csv` / `--input`.

---

## Tests

Hardware-free unit tests cover the OATH protocol encoding, password derivation,
input validation, response reassembly, otpauth parsing, dedup, and CSV I/O:

```
python test_oath.py
python test_qr.py
```

---

## How it works

The OATH protocol layer is implemented directly over PC/SC (`pyscard`) with no
Yubico dependency, and is verified against the applet sources
(`com.vivokey.otp.YkneoOath` / `OathObj`) and `doc/Protocol.txt`. See
[docs/design.md](docs/design.md) for the importer/protocol design and
[docs/qr-to-csv-design.md](docs/qr-to-csv-design.md) for the QR-to-CSV design.

- `otpauth.py` — parse/canonicalize `otpauth://` URIs (shared by both tools).
- `qr_to_sheet.py` — QR decode, dedup, CSV append.
- `apex_otp_import.py` — CSV/xlsx load, OATH SELECT/VALIDATE/LIST/PUT, password
  auth, reset, memory profiling, and the CLI.