otp-import/README.md

# otp-import

Move TOTP credentials from saved authenticator **QR-code screenshots** onto a
**VivoKey Apex** OTP applet (the ykneo-oath applet), by way of a plain,
human-editable **CSV**.

Two small command-line tools:

- `qr_to_sheet.py` — decode QR screenshots into the CSV (no hardware needed).
- `apex_otp_import.py` — write the CSV's credentials onto the applet over PC/SC.

---

## The pipeline

```
   QR screenshots            otp-keys.csv                 Apex OTP applet
  ┌───────────────┐        ┌──────────────┐             ┌───────────────┐
  │ *.png / *.jpg │──(1)──▶ │ issuer       │──(3)───────▶ │ TOTP creds    │
  │ otpauth://... │        │ account      │   PUT/auth   │ (ykneo-oath)  │
  └───────────────┘        │ otpauth_uri  │             └───────────────┘
                           │ source_file  │
                    (2) edit labels here
                           └──────────────┘
```

1. **QR → CSV** (`qr_to_sheet.py`): decode each screenshot and append any
   **new** credential to the CSV. Deduplication is on the `otpauth://` data, so
   re-running never adds a credential that is already present.
2. **Edit labels (optional)**: adjust the `issuer` and `account` columns to
   taste. **Never edit the `otpauth_uri` column** — it is the immutable source
   of truth (secret and parameters live inside it).
3. **CSV → applet** (`apex_otp_import.py`): validate the CSV and write each
   credential onto the applet, authenticating first if the applet has a
   password.

The two stages guard different things: stage 1 stops you re-importing the same
QR; stage 3 stops you writing something you broke while editing labels.

---

## The CSV

Four columns:

| Column | Edit it? | Meaning |
| ------ | -------- | ------- |
| `issuer` | yes | Issuer shown on the applet (the "Issuer:" part of the name). |
| `account` | yes | Account shown on the applet (the part after "Issuer:"). |
| `otpauth_uri` | **no** | The raw `otpauth://` string from the QR. Dedup key and the source of the secret/algorithm/digits/period. |
| `source_file` | reference | The image filename the row was decoded from. |

The credential **name** written to the applet is `issuer:account` from the
editable columns. The **secret** and other crypto parameters are always taken
from `otpauth_uri`, so editing a label can never corrupt the code that gets
generated. The secret is still visible — it lives inside `otpauth_uri`.

The CSV is disposable: it can be regenerated from the QR screenshots at any time
with `qr_to_sheet.py`.

---

## Requirements

- Python 3.9+
- `pyscard`, `cryptography`, `openpyxl` (for reading legacy `.xlsx`), and
  `opencv-python` (for QR decoding).
- For writing to a card: a PC/SC reader with the Apex (over NFC) or a JavaCard
  carrying the applet presented to it.

```
pip install pyscard cryptography openpyxl opencv-python
```

---

## Tool 1 — `qr_to_sheet.py` (QR → CSV, no card)

Decode a folder of QR screenshots and append new credentials to the CSV. This
tool never touches a reader, so run it any time — the Apex does not need to be
present.

```
python qr_to_sheet.py --images <folder> --csv <path/to/otp-keys.csv>
```

Options:

| Option | Default | Meaning |
| ------ | ------- | ------- |
| `--images DIR` | `.` | Folder of QR images (`*.png`, `*.jpg`, `*.jpeg`). |
| `--csv PATH` | `otp-keys.csv` | The CSV to create or append to. |
| `--dry-run` | off | Report what would be added; write nothing. |

What it does per run:

- Decodes each image (a multi-pass decode copes with scaling/thresholding).
- Parses the `otpauth://totp/...` URI; seeds `issuer`/`account` from it.
- Skips any credential whose `otpauth://` data is already in the CSV
  (deduplicated, existing rows never touched), and any duplicate within the same
  run.
- Appends the genuinely-new rows and prints a summary of added / skipped /
  undecodable.

Example:

```
$ python qr_to_sheet.py --images ./screenshots --csv otp-keys.csv
Scanned 34 image(s): 32 otpauth QR(s) decoded.
  3 new, 29 duplicate(s) skipped.
  could not be decoded (add manually): coinbase-...png, plaid-...png
Appended 3 row(s) to otp-keys.csv.
```

**Undecodable images** — stylized/colored QR codes, or screenshots that only
show the secret as text — are listed by filename. Add those by hand: create a
row with `issuer`, `account`, `source_file`, and an `otpauth_uri` you build from
the printed secret, e.g.
`otpauth://totp/Coinbase:you@example.com?secret=THESECRET&issuer=Coinbase`.

`otpauth-migration://` payloads (Google Authenticator's batch-export QR) are
reported as unsupported, not decoded.

---

## Tool 2 — `apex_otp_import.py` (CSV → applet)

Validate the CSV and write its credentials onto the applet. Needs the Apex on a
reader (except `--dry-run`).

```
python apex_otp_import.py --input otp-keys.csv [options]
```

Options:

| Option | Meaning |
| ------ | ------- |
| `--input PATH` | CSV (or legacy `.xlsx`) to import. Default `otp-keys.csv`. |
| `--reader SUBSTR` | Pick a reader by name substring (e.g. `VivoKey`, `3700 F CL`). Omit to auto-detect the single card holding an OATH applet; you are prompted if more than one is present. |
| `--dry-run` | Validate and print the APDUs that would be sent; no card I/O. |
| `--list` | Read-only: list the credentials already on the applet. |
| `--overwrite-all` | Overwrite any credential that already exists (non-interactive). |
| `--skip-existing` | Skip any credential that already exists (non-interactive). |
| `--password PASS` | Applet password (otherwise prompted if the applet is locked). |
| `--profile-memory` | Record per-credential EEPROM consumption to a CSV log (needs the `javacard-memory` applet). |
| `--set-password PASS` | After a successful import, set an access password — only if the applet has none. |
| `--reset` | Standalone: wipe all credentials and any password, then exit. |
| `--import-qr` | Run the QR → CSV step first (see below), then import. |
| `--qr-images DIR` | QR folder for `--import-qr` (default: the folder of `--input`). |
| `--memlog PATH` | Path for the `--profile-memory` log (default `apex-otp-memlog.csv`). |

By default, when a credential name already exists on the applet you are asked
whether to overwrite or skip it (with all/none shortcuts). Use `--overwrite-all`
or `--skip-existing` for unattended runs.

Recommended sequence for a first import:

```
python apex_otp_import.py --input otp-keys.csv --dry-run           # (1) preview, no card
python apex_otp_import.py --input otp-keys.csv --list              # (2) see what's on the applet
python apex_otp_import.py --input otp-keys.csv                     # (3) import for real
python apex_otp_import.py --input otp-keys.csv --list              # (4) confirm
```

### Passwords

Authenticating to a password-protected applet uses the standard YKOATH scheme:
PBKDF2-HMAC-SHA1 of the password with the applet's device id as salt, then a
mutual challenge-response. `--set-password` sets a password only on an applet
that has none; it does not change or remove an existing one.

Note: unlike a YubiKey, this applet requires authentication before `--reset`
when a password is set. If you set a password and forget it, the OATH applet
cannot be reset over the OATH protocol — it must be deleted/reinstalled with
GlobalPlatform (card-manager keys).

### Memory profiling

`--profile-memory` measures how much persistent memory each credential consumes
by reading the `javacard-memory` applet before and after each `PUT`. Each row of
the log records the timestamp, card UID, memory-applet batch id, credential
name (never the secret), and bytes consumed. It is slower (a memory read per
credential) and off by default.

---

## The three modes

| Command | Does | Card needed |
| ------- | ---- | ----------- |
| `qr_to_sheet.py --images D --csv C` | QR → CSV | no |
| `apex_otp_import.py --input C` | CSV → applet | yes |
| `apex_otp_import.py --input C --import-qr` | QR → CSV → applet (one shot) | yes |

`--import-qr` is the whole pipeline in one command: it decodes new QRs into the
CSV (deduped on `otpauth://`, existing entries silently skipped), then validates
and imports. The QR folder defaults to the CSV's folder; override with
`--qr-images`. If you only want to refresh the CSV, use `qr_to_sheet.py` — the
Apex does not need to be on a reader.

```
# whole pipeline, single command
python apex_otp_import.py --input otp-keys.csv --import-qr --reader VivoKey

# same, with an explicit QR folder and unattended overwrite
python apex_otp_import.py --input otp-keys.csv --import-qr --qr-images ./screenshots --overwrite-all
```

---

## Common workflows

**Add newly-enrolled services.** Save the new QR screenshots into your folder,
then:

```
python qr_to_sheet.py --images ./screenshots --csv otp-keys.csv   # appends only the new ones
# (edit issuer/account labels if you like)
python apex_otp_import.py --input otp-keys.csv --reader VivoKey --overwrite-all
```

**Re-provision a fresh/wiped Apex from scratch.**

```
python apex_otp_import.py --input otp-keys.csv --reader VivoKey
```

**Password-protect after loading.**

```
python apex_otp_import.py --input otp-keys.csv --reader VivoKey --set-password 'mypass'
```

**Wipe an applet (know the password if it has one).**

```
python apex_otp_import.py --reset --reader VivoKey [--password 'mypass']
```

---

## Validation (stage 3)

The CSV is validated before any reader is touched; the run halts with a per-row
report if it finds any of:

- Duplicate credential name (`issuer:account`).
- A secret that is not valid Base32.
- A decoded key longer than 64 bytes, or a composed name longer than 64 bytes.
- An unsupported algorithm (the applet accepts SHA1/SHA256, not SHA512).
- Control characters, or a literal `:` or `/`, inside an issuer or account
  (they collide with the OATH `issuer:account` separator and `period/` prefix).
- A digit count outside 6–8, or a non-positive period.

Fix the offending cells and re-run. The applet itself only enforces the length
limits; the character and digit/period rules are host-side guards for
OATH-convention and authenticator-app compatibility.

---

## Security

This tooling handles live TOTP secrets. **Keep your secrets out of this
repository.** `otp-keys.csv`, any `.xlsx`, the memory log, and QR screenshots
are excluded by `.gitignore`; keep the real data in a private location (e.g. a
synced personal folder) and point the tools at it with `--csv` / `--input`.

---

## Tests

Hardware-free unit tests cover the OATH protocol encoding, password derivation,
input validation, response reassembly, otpauth parsing, dedup, and CSV I/O:

```
python test_oath.py
python test_qr.py
```

---

## How it works

The OATH protocol layer is implemented directly over PC/SC (`pyscard`) with no
Yubico dependency, and is verified against the applet sources
(`com.vivokey.otp.YkneoOath` / `OathObj`) and `doc/Protocol.txt`. See
[docs/design.md](docs/design.md) for the importer/protocol design and
[docs/qr-to-csv-design.md](docs/qr-to-csv-design.md) for the QR-to-CSV design.

- `otpauth.py` — parse/canonicalize `otpauth://` URIs (shared by both tools).
- `qr_to_sheet.py` — QR decode, dedup, CSV append.
- `apex_otp_import.py` — CSV/xlsx load, OATH SELECT/VALIDATE/LIST/PUT, password
  auth, reset, memory profiling, and the CLI.