otp-import/docs/design.md

# Apex OTP Importer — Design

Date: 2026-07-08
Status: Draft for review

## Goal

A single runnable Python script that reads the curated OTP export
(`otp-keys-export.xlsx`, sheet `OTP Keys`) and writes every credential to a
VivoKey Apex's OTP (OATH) applet over PC/SC, quickly and repeatably. It must
support password-protected OATH applets (authenticate before writing) and,
optionally, profile per-credential on-chip memory consumption.

## Context and constraints

- Host: Windows 11, Python 3.11. Installed and used: `pyscard`, `cryptography`,
  `openpyxl`. No Yubico libraries (`ykman`/`yubikit`) — the OATH protocol is
  implemented directly, so there is no added dependency and no reliance on
  Yubico device enumeration (Apex is not a Yubico device).
- Transport: PC/SC. Multiple contactless readers are typically present
  (ACR1252 CL, OMNIKEY 5022, Identiv uTrust 3700 F CL, VivoKey Smart Reader).
- Input source of truth: the **curated columns** of the spreadsheet
  (Issuer, Account, Secret, Type, Algorithm, Digits, Period) — NOT the
  `otpauth://` URI column. The curated Account values were cleaned during export
  (e.g. Heroku → `amal@vivokey.com`, not the QR label "Amal Graafstra").

## Verified protocol facts

Verified directly against the applet sources in the local repos
`apex-totp` (ykneo-oath, `com.vivokey.otp.YkneoOath` / `OathObj`) and
`javacard-memory` (`de.chrz.jcmemory.JCMemoryApplet`), plus
`apex-totp/doc/Protocol.txt`.

### OATH applet

- Applet AID (from `apex-totp/build.xml`):
  `A0 00 00 05 27 21 01 01 41 50 45 58 01` (`41 50 45 58` = "APEX").
  SELECT strategy: try this full AID first, then fall back to
  `A0 00 00 05 27 21 01 01` (8 B, per repo docs/tests) and
  `A0 00 00 05 27 21 01` (7 B, canonical Yubico). Use the first that returns a
  valid SELECT response (version tag `0x79`). Prefix selection works because
  Yubico Authenticator already talks to Apex with the short AID.
- INS: PUT `0x01`, DELETE `0x02`, SET CODE `0x03`, RESET `0x04` (P1P2 must be
  `0xDEAD`), RENAME `0x05`, LIST `0xA1`, CALCULATE `0xA2`, VALIDATE `0xA3`,
  CALCULATE ALL `0xA4`, SEND REMAINING `0xA5`.
- Tags: NAME `0x71`, NAME_LIST `0x72`, KEY `0x73`, CHALLENGE `0x74`,
  RESPONSE `0x75`, T_RESPONSE `0x76`, NO_RESPONSE `0x77`, PROPERTY `0x78`,
  VERSION `0x79`, IMF `0x7A`, ALGORITHM `0x7B`.
- Type|algo byte: type in high nibble (HOTP `0x10`, TOTP `0x20`), algo in low
  nibble (SHA1 `0x01`, SHA256 `0x02`). **SHA512 is rejected by this applet.**
  All our credentials are TOTP/SHA1 → `0x21`.
- SELECT response TLV: VERSION(`0x79`) + NAME(`0x71`, 8-byte device id/salt) and,
  if a password is set, CHALLENGE(`0x74`, 8 bytes) + ALGORITHM(`0x7B`, 1 byte).
  Presence of the CHALLENGE tag ⇒ applet is locked.
- PUT data: `NAME(0x71,len,name)` + `KEY(0x73, len=2+keylen, [typeAlgo, digits, key…])`.
  PROPERTY/IMF omitted (no touch, TOTP only). Name ≤ 64 B, key ≤ 64 B.
  The applet XORs exactly `keylen` bytes into the 64-byte HMAC ipad/opad, so a
  short secret yields the same HMAC as a zero-padded one; we still right-pad
  short secrets to 14 B to match the Yubico host (HMAC-identical).
- LIST response (repeats): `NAME_LIST(0x72, len=namelen+1, [type, name…])`.
- Long responses (LIST/CALCULATE ALL) chunk with SW `61xx`; fetch the rest with
  SEND REMAINING (`0xA5`) until `9000`.
- Response codes: success `9000`; applet full `6A84`; auth required `6982`;
  wrong syntax / wrong data (incl. wrong password on VALIDATE) `6A80`;
  no such object `6984`.
- Auth state is stored in a CLEAR_ON_DESELECT buffer: any deselect (e.g. a
  memory-applet SELECT) drops authentication and requires a fresh VALIDATE.

### Password authentication (VALIDATE)

Per `doc/Protocol.txt` and `OathObj`:

1. `key = PBKDF2-HMAC-SHA1(password_utf8, salt = 8-byte device id from SELECT,
   iterations = 1000, dklen = 16)`. PBKDF2 always uses HMAC-SHA1 here,
   independent of the credential/challenge algorithm.
2. The challenge-response HMAC uses the algorithm named by the SELECT
   `ALGORITHM` (`0x7B`) tag (low nibble: SHA1 `0x01` / SHA256 `0x02`), defaulting
   to SHA1 — which is what the Yubico host sets in practice (20-byte response).
   `deviceResponse = HMAC(key, deviceChallenge)` where `deviceChallenge` is the
   SELECT `0x74` value.
3. `clientChallenge = os.urandom(8)`.
4. Send VALIDATE (`0xA3`): `RESPONSE(0x75, deviceResponse)` +
   `CHALLENGE(0x74, clientChallenge[8])`. The response length must equal the
   applet's digest length or it is rejected — hence honoring the ALGORITHM tag.
5. Applet returns `RESPONSE(0x75, HMAC(key, clientChallenge))`; the host verifies
   it to confirm mutual auth. Wrong password → `6A80`.

The derived key is stable across re-SELECTs (device id only changes on RESET),
so it is derived once and reused when re-validation is needed during profiling.

### Memory applet (optional)

- Applet AID: `A0 00 00 08 46 6D 65 6D 6F 72 79 01` (`46 6D…79` = "Fmemory").
- SELECT returns 12 bytes: availPersistent(4 B BE), totalBaseline(4 B BE),
  transientReset(2 B), transientDeselect(2 B). Apex Flex baseline = 84336 B.
- `GET_BATCH` = `00 01 00 00 00` → 4-byte batch id.
- Each SELECT runs `requestObjectDeletion()` (GC) then reports free memory.
  Because selecting it deselects OATH, profiling must re-VALIDATE afterward.

## Architecture

One runnable file `apex_otp_import.py`, organized into focused units:

- `Tlv` helpers — encode/parse BER-TLV with 1/2/3-byte lengths (`0x81`/`0x82`).
- `CardChannel` — thin wrapper over a pyscard connection: `transmit(apdu)` plus
  `send(cla, ins, p1, p2, data, le)` that transparently follows `61xx` with
  SEND REMAINING and returns `(data, sw)`.
- `OathSession` — SELECT (with AID fallback), parse device id/challenge/locked,
  `derive_key(password)`, `validate(key)`, `list()`, `put(cred)`,
  and status-word → exception mapping. Pure protocol; unit-testable with a fake
  channel.
- `MemorySession` — SELECT the memory applet, parse availPersistent + baseline,
  `get_batch()`. Absent applet handled gracefully.
- `Credential` — normalized record: issuer, account, secret(bytes), oath_type,
  algorithm, digits, period, and `cred_id` (`issuer:account`, with `period/`
  prefix only if period ≠ 30).
- `load_credentials(xlsx_path)` — read the `OTP Keys` sheet from curated columns.
- `validate_input(creds)` — pre-flight checks; returns a list of problems.
- `pick_reader(...)` — enumerate readers, detect cards, SELECT OATH, choose.
- `main()` — CLI + orchestration in the order below.

Companion `test_oath.py` — hardware-free unit tests.

## Execution flow

1. **Load + validate input first (before any reader access).**
   `load_credentials` → `validate_input`. If any problems, print them all and
   exit non-zero. No interactive fixing — the user edits the spreadsheet.
2. **Reader/card detection.** For each reader with a card inserted, try the OATH
   SELECT. 0 responders → error "Place your Apex on a reader and re-run", exit.
   1 → use it. >1 → list reader names and prompt for the target.
3. **Session.** SELECT OATH. If locked, `getpass` the password and VALIDATE
   (map `6A80` → "incorrect password", exit). LIST existing credential names.
4. **Optional capacity pre-check.** If the memory applet is present, read
   availPersistent and warn (no halt) if `availPersistent < ~250 B × N`.
5. **Import.** For each credential: if its `cred_id` is already on the applet,
   ask per conflict (overwrite / skip; with all/none shortcuts). Otherwise PUT.
   Map `6A84` → "applet full" and stop further PUTs (report remainder as
   skipped). `--profile-memory` wraps each PUT in before/after memory reads
   (re-VALIDATE between, since the memory SELECT deselects OATH) and appends a
   log row.
6. **Report.** Per-credential OK / skipped / failed summary, plus mem-log path
   when profiling.

## Pre-flight validation rules (halt on any)

- Duplicate `cred_id` within the input (e.g. the two
  `SAW:michelle@dangerousthings.com` entries with different secrets).
- Secret missing or not valid Base32 after uppercasing and removing spaces.
- Decoded key length > 64 bytes.
- `cred_id` (UTF-8) length > 64 bytes.
- Algorithm SHA512 (unsupported by the applet), or a non-TOTP/HOTP type.

Each problem is reported with the offending issuer/account and source row.

## Conflict handling (on-device)

Interactive per name already present on the applet: `[o]verwrite` /
`[s]kip` / `[a]` overwrite-all / `[x]` skip-all. Non-interactive overrides:
`--overwrite-all`, `--skip-existing`.

## Memory profiling (`--profile-memory`, opt-in)

Per credential: memory SELECT → availBefore; OATH SELECT + re-VALIDATE (if
locked) → PUT; memory SELECT → availAfter; `consumed = availBefore - availAfter`.
Append to `apex-otp-memlog.csv` with columns:
`timestamp_iso, card_uid, mem_batch, key_name, avail_before, avail_after, consumed_bytes`.
`timestamp_iso` is the host clock; `card_uid` from `FF CA 00 00 00`; the key
secret is never logged. Block-granular allocation means `consumed_bytes` is a
close estimate, not byte-exact.

## CLI

```
apex_otp_import.py [--input PATH] [--reader SUBSTR] [--dry-run] [--list]
                   [--overwrite-all | --skip-existing] [--profile-memory]
                   [--memlog PATH] [--password PASS]
```

- `--dry-run` — load, validate, build every APDU, hex-print them; no card I/O.
- `--list` — read-only: connect, authenticate if needed, print on-applet creds.
- `--password` — supply the applet password non-interactively (else `getpass`).

## Error handling

Central status-word → message map: `6A84` applet full, `6982` auth required,
`6A80` wrong data / incorrect password, `6984` no such object, `61xx` handled by
SEND REMAINING, `6D00`/`6E00` wrong INS/CLA (wrong applet or AID). Card
exceptions (no card, reader gone) surface as clear, actionable messages.

## Testing (`test_oath.py`, no hardware)

- TLV encode/parse round-trips incl. multi-byte lengths.
- `cred_id` formatting (period prefix rule).
- Byte-exact PUT payload for a known credential (type/algo, digits, padded key).
- Byte-exact VALIDATE payload and mutual-response verification against a known
  key/challenge.
- Base32 decode + 14-byte right-pad; HMAC-equivalence of padded vs unpadded.
- PBKDF2/HMAC vectors; RFC 6238 TOTP HMAC sanity check.
- `validate_input` catches each halt condition (dup, bad Base32, oversize name,
  oversize key, SHA512).
- `61xx` reassembly via a fake channel that chunks a long LIST response.

## Password / reset admin (added after initial spec)

The importer also exposes two admin operations on the same protocol layer:

- `--set-password PASS`: after a fully successful import, set an access password
  on the applet **only if it has none** (SET CODE with a SHA1 access key derived
  as above). Runs post-import for efficiency (single session). Does not change or
  remove an existing password.
- `--reset`: standalone; sends RESET (P1P2 = 0xDEAD) to wipe all credentials and
  any password, then exits.

Note: the applet reports the full `type|algo` byte (e.g. `0x21`) in the SELECT
ALGORITHM tag; the low nibble is masked off to select the HMAC algorithm.

## Out of scope (YAGNI)

Changing or removing an existing password; HOTP import; touch-required;
RENAME/DELETE of individual credentials; CSV/text input; on-device code
calculation (beyond ad-hoc post-import verification, not part of the tool).

## Files

Code and docs live in the `otp-import` repo
(`C:\Users\amal\Documents\Repos\otp-import`, remote
`git.dngr.us/VivoKey/otp-import`): `apex_otp_import.py`, `test_oath.py`,
`docs/design.md`, `README.md`, `.gitignore`.

The secret data stays **out of the repo**: the export spreadsheet
(`otp-keys-export.xlsx`), the profiling CSV (`apex-otp-memlog.csv`, created only
when profiling), and QR screenshots remain in the private Nextcloud folder and
are excluded by `.gitignore`. The importer is pointed at the data with
`--input`.

## Risks / notes

- The instance AID on a Fidesmo-provisioned Apex may differ from `build.xml`;
  mitigated by the AID fallback list and validating the SELECT response shape.
- Contactless UID from `FF CA` may be random per tap on some configurations; it
  is logged as-is for correlation, not treated as a stable serial.
- Keep the card still: the whole import runs in one OATH session; a mid-import
  deselect drops auth and (without profiling's re-VALIDATE) would fail PUTs with
  `6982`.