Add Apex OTP importer
Import TOTP credentials from the export spreadsheet into a VivoKey Apex OTP (ykneo-oath) applet over PC/SC. Includes the OATH protocol layer (SELECT/VALIDATE/LIST/PUT with 61xx reassembly), PBKDF2 password authentication, pre-flight input validation, ask-per-conflict handling, optional javacard-memory profiling, and hardware-free unit tests.
This commit is contained in:
231
docs/design.md
Normal file
231
docs/design.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# Apex OTP Importer — Design
|
||||
|
||||
Date: 2026-07-08
|
||||
Status: Draft for review
|
||||
|
||||
## Goal
|
||||
|
||||
A single runnable Python script that reads the curated OTP export
|
||||
(`otp-keys-export.xlsx`, sheet `OTP Keys`) and writes every credential to a
|
||||
VivoKey Apex's OTP (OATH) applet over PC/SC, quickly and repeatably. It must
|
||||
support password-protected OATH applets (authenticate before writing) and,
|
||||
optionally, profile per-credential on-chip memory consumption.
|
||||
|
||||
## Context and constraints
|
||||
|
||||
- Host: Windows 11, Python 3.11. Installed and used: `pyscard`, `cryptography`,
|
||||
`openpyxl`. No Yubico libraries (`ykman`/`yubikit`) — the OATH protocol is
|
||||
implemented directly, so there is no added dependency and no reliance on
|
||||
Yubico device enumeration (Apex is not a Yubico device).
|
||||
- Transport: PC/SC. Multiple contactless readers are typically present
|
||||
(ACR1252 CL, OMNIKEY 5022, Identiv uTrust 3700 F CL, VivoKey Smart Reader).
|
||||
- Input source of truth: the **curated columns** of the spreadsheet
|
||||
(Issuer, Account, Secret, Type, Algorithm, Digits, Period) — NOT the
|
||||
`otpauth://` URI column. The curated Account values were cleaned during export
|
||||
(e.g. Heroku → `amal@vivokey.com`, not the QR label "Amal Graafstra").
|
||||
|
||||
## Verified protocol facts
|
||||
|
||||
Verified directly against the applet sources in the local repos
|
||||
`apex-totp` (ykneo-oath, `com.vivokey.otp.YkneoOath` / `OathObj`) and
|
||||
`javacard-memory` (`de.chrz.jcmemory.JCMemoryApplet`), plus
|
||||
`apex-totp/doc/Protocol.txt`.
|
||||
|
||||
### OATH applet
|
||||
|
||||
- Applet AID (from `apex-totp/build.xml`):
|
||||
`A0 00 00 05 27 21 01 01 41 50 45 58 01` (`41 50 45 58` = "APEX").
|
||||
SELECT strategy: try this full AID first, then fall back to
|
||||
`A0 00 00 05 27 21 01 01` (8 B, per repo docs/tests) and
|
||||
`A0 00 00 05 27 21 01` (7 B, canonical Yubico). Use the first that returns a
|
||||
valid SELECT response (version tag `0x79`). Prefix selection works because
|
||||
Yubico Authenticator already talks to Apex with the short AID.
|
||||
- INS: PUT `0x01`, DELETE `0x02`, SET CODE `0x03`, RESET `0x04` (P1P2 must be
|
||||
`0xDEAD`), RENAME `0x05`, LIST `0xA1`, CALCULATE `0xA2`, VALIDATE `0xA3`,
|
||||
CALCULATE ALL `0xA4`, SEND REMAINING `0xA5`.
|
||||
- Tags: NAME `0x71`, NAME_LIST `0x72`, KEY `0x73`, CHALLENGE `0x74`,
|
||||
RESPONSE `0x75`, T_RESPONSE `0x76`, NO_RESPONSE `0x77`, PROPERTY `0x78`,
|
||||
VERSION `0x79`, IMF `0x7A`, ALGORITHM `0x7B`.
|
||||
- Type|algo byte: type in high nibble (HOTP `0x10`, TOTP `0x20`), algo in low
|
||||
nibble (SHA1 `0x01`, SHA256 `0x02`). **SHA512 is rejected by this applet.**
|
||||
All our credentials are TOTP/SHA1 → `0x21`.
|
||||
- SELECT response TLV: VERSION(`0x79`) + NAME(`0x71`, 8-byte device id/salt) and,
|
||||
if a password is set, CHALLENGE(`0x74`, 8 bytes) + ALGORITHM(`0x7B`, 1 byte).
|
||||
Presence of the CHALLENGE tag ⇒ applet is locked.
|
||||
- PUT data: `NAME(0x71,len,name)` + `KEY(0x73, len=2+keylen, [typeAlgo, digits, key…])`.
|
||||
PROPERTY/IMF omitted (no touch, TOTP only). Name ≤ 64 B, key ≤ 64 B.
|
||||
The applet XORs exactly `keylen` bytes into the 64-byte HMAC ipad/opad, so a
|
||||
short secret yields the same HMAC as a zero-padded one; we still right-pad
|
||||
short secrets to 14 B to match the Yubico host (HMAC-identical).
|
||||
- LIST response (repeats): `NAME_LIST(0x72, len=namelen+1, [type, name…])`.
|
||||
- Long responses (LIST/CALCULATE ALL) chunk with SW `61xx`; fetch the rest with
|
||||
SEND REMAINING (`0xA5`) until `9000`.
|
||||
- Response codes: success `9000`; applet full `6A84`; auth required `6982`;
|
||||
wrong syntax / wrong data (incl. wrong password on VALIDATE) `6A80`;
|
||||
no such object `6984`.
|
||||
- Auth state is stored in a CLEAR_ON_DESELECT buffer: any deselect (e.g. a
|
||||
memory-applet SELECT) drops authentication and requires a fresh VALIDATE.
|
||||
|
||||
### Password authentication (VALIDATE)
|
||||
|
||||
Per `doc/Protocol.txt` and `OathObj`:
|
||||
|
||||
1. `key = PBKDF2-HMAC-SHA1(password_utf8, salt = 8-byte device id from SELECT,
|
||||
iterations = 1000, dklen = 16)`. PBKDF2 always uses HMAC-SHA1 here,
|
||||
independent of the credential/challenge algorithm.
|
||||
2. The challenge-response HMAC uses the algorithm named by the SELECT
|
||||
`ALGORITHM` (`0x7B`) tag (low nibble: SHA1 `0x01` / SHA256 `0x02`), defaulting
|
||||
to SHA1 — which is what the Yubico host sets in practice (20-byte response).
|
||||
`deviceResponse = HMAC(key, deviceChallenge)` where `deviceChallenge` is the
|
||||
SELECT `0x74` value.
|
||||
3. `clientChallenge = os.urandom(8)`.
|
||||
4. Send VALIDATE (`0xA3`): `RESPONSE(0x75, deviceResponse)` +
|
||||
`CHALLENGE(0x74, clientChallenge[8])`. The response length must equal the
|
||||
applet's digest length or it is rejected — hence honoring the ALGORITHM tag.
|
||||
5. Applet returns `RESPONSE(0x75, HMAC(key, clientChallenge))`; the host verifies
|
||||
it to confirm mutual auth. Wrong password → `6A80`.
|
||||
|
||||
The derived key is stable across re-SELECTs (device id only changes on RESET),
|
||||
so it is derived once and reused when re-validation is needed during profiling.
|
||||
|
||||
### Memory applet (optional)
|
||||
|
||||
- Applet AID: `A0 00 00 08 46 6D 65 6D 6F 72 79 01` (`46 6D…79` = "Fmemory").
|
||||
- SELECT returns 12 bytes: availPersistent(4 B BE), totalBaseline(4 B BE),
|
||||
transientReset(2 B), transientDeselect(2 B). Apex Flex baseline = 84336 B.
|
||||
- `GET_BATCH` = `00 01 00 00 00` → 4-byte batch id.
|
||||
- Each SELECT runs `requestObjectDeletion()` (GC) then reports free memory.
|
||||
Because selecting it deselects OATH, profiling must re-VALIDATE afterward.
|
||||
|
||||
## Architecture
|
||||
|
||||
One runnable file `apex_otp_import.py`, organized into focused units:
|
||||
|
||||
- `Tlv` helpers — encode/parse BER-TLV with 1/2/3-byte lengths (`0x81`/`0x82`).
|
||||
- `CardChannel` — thin wrapper over a pyscard connection: `transmit(apdu)` plus
|
||||
`send(cla, ins, p1, p2, data, le)` that transparently follows `61xx` with
|
||||
SEND REMAINING and returns `(data, sw)`.
|
||||
- `OathSession` — SELECT (with AID fallback), parse device id/challenge/locked,
|
||||
`derive_key(password)`, `validate(key)`, `list()`, `put(cred)`,
|
||||
and status-word → exception mapping. Pure protocol; unit-testable with a fake
|
||||
channel.
|
||||
- `MemorySession` — SELECT the memory applet, parse availPersistent + baseline,
|
||||
`get_batch()`. Absent applet handled gracefully.
|
||||
- `Credential` — normalized record: issuer, account, secret(bytes), oath_type,
|
||||
algorithm, digits, period, and `cred_id` (`issuer:account`, with `period/`
|
||||
prefix only if period ≠ 30).
|
||||
- `load_credentials(xlsx_path)` — read the `OTP Keys` sheet from curated columns.
|
||||
- `validate_input(creds)` — pre-flight checks; returns a list of problems.
|
||||
- `pick_reader(...)` — enumerate readers, detect cards, SELECT OATH, choose.
|
||||
- `main()` — CLI + orchestration in the order below.
|
||||
|
||||
Companion `test_oath.py` — hardware-free unit tests.
|
||||
|
||||
## Execution flow
|
||||
|
||||
1. **Load + validate input first (before any reader access).**
|
||||
`load_credentials` → `validate_input`. If any problems, print them all and
|
||||
exit non-zero. No interactive fixing — the user edits the spreadsheet.
|
||||
2. **Reader/card detection.** For each reader with a card inserted, try the OATH
|
||||
SELECT. 0 responders → error "Place your Apex on a reader and re-run", exit.
|
||||
1 → use it. >1 → list reader names and prompt for the target.
|
||||
3. **Session.** SELECT OATH. If locked, `getpass` the password and VALIDATE
|
||||
(map `6A80` → "incorrect password", exit). LIST existing credential names.
|
||||
4. **Optional capacity pre-check.** If the memory applet is present, read
|
||||
availPersistent and warn (no halt) if `availPersistent < ~250 B × N`.
|
||||
5. **Import.** For each credential: if its `cred_id` is already on the applet,
|
||||
ask per conflict (overwrite / skip; with all/none shortcuts). Otherwise PUT.
|
||||
Map `6A84` → "applet full" and stop further PUTs (report remainder as
|
||||
skipped). `--profile-memory` wraps each PUT in before/after memory reads
|
||||
(re-VALIDATE between, since the memory SELECT deselects OATH) and appends a
|
||||
log row.
|
||||
6. **Report.** Per-credential OK / skipped / failed summary, plus mem-log path
|
||||
when profiling.
|
||||
|
||||
## Pre-flight validation rules (halt on any)
|
||||
|
||||
- Duplicate `cred_id` within the input (e.g. the two
|
||||
`SAW:michelle@dangerousthings.com` entries with different secrets).
|
||||
- Secret missing or not valid Base32 after uppercasing and removing spaces.
|
||||
- Decoded key length > 64 bytes.
|
||||
- `cred_id` (UTF-8) length > 64 bytes.
|
||||
- Algorithm SHA512 (unsupported by the applet), or a non-TOTP/HOTP type.
|
||||
|
||||
Each problem is reported with the offending issuer/account and source row.
|
||||
|
||||
## Conflict handling (on-device)
|
||||
|
||||
Interactive per name already present on the applet: `[o]verwrite` /
|
||||
`[s]kip` / `[a]` overwrite-all / `[x]` skip-all. Non-interactive overrides:
|
||||
`--overwrite-all`, `--skip-existing`.
|
||||
|
||||
## Memory profiling (`--profile-memory`, opt-in)
|
||||
|
||||
Per credential: memory SELECT → availBefore; OATH SELECT + re-VALIDATE (if
|
||||
locked) → PUT; memory SELECT → availAfter; `consumed = availBefore - availAfter`.
|
||||
Append to `apex-otp-memlog.csv` with columns:
|
||||
`timestamp_iso, card_uid, mem_batch, key_name, avail_before, avail_after, consumed_bytes`.
|
||||
`timestamp_iso` is the host clock; `card_uid` from `FF CA 00 00 00`; the key
|
||||
secret is never logged. Block-granular allocation means `consumed_bytes` is a
|
||||
close estimate, not byte-exact.
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
apex_otp_import.py [--input PATH] [--reader SUBSTR] [--dry-run] [--list]
|
||||
[--overwrite-all | --skip-existing] [--profile-memory]
|
||||
[--memlog PATH] [--password PASS]
|
||||
```
|
||||
|
||||
- `--dry-run` — load, validate, build every APDU, hex-print them; no card I/O.
|
||||
- `--list` — read-only: connect, authenticate if needed, print on-applet creds.
|
||||
- `--password` — supply the applet password non-interactively (else `getpass`).
|
||||
|
||||
## Error handling
|
||||
|
||||
Central status-word → message map: `6A84` applet full, `6982` auth required,
|
||||
`6A80` wrong data / incorrect password, `6984` no such object, `61xx` handled by
|
||||
SEND REMAINING, `6D00`/`6E00` wrong INS/CLA (wrong applet or AID). Card
|
||||
exceptions (no card, reader gone) surface as clear, actionable messages.
|
||||
|
||||
## Testing (`test_oath.py`, no hardware)
|
||||
|
||||
- TLV encode/parse round-trips incl. multi-byte lengths.
|
||||
- `cred_id` formatting (period prefix rule).
|
||||
- Byte-exact PUT payload for a known credential (type/algo, digits, padded key).
|
||||
- Byte-exact VALIDATE payload and mutual-response verification against a known
|
||||
key/challenge.
|
||||
- Base32 decode + 14-byte right-pad; HMAC-equivalence of padded vs unpadded.
|
||||
- PBKDF2/HMAC vectors; RFC 6238 TOTP HMAC sanity check.
|
||||
- `validate_input` catches each halt condition (dup, bad Base32, oversize name,
|
||||
oversize key, SHA512).
|
||||
- `61xx` reassembly via a fake channel that chunks a long LIST response.
|
||||
|
||||
## Out of scope (YAGNI)
|
||||
|
||||
Setting/changing/removing the applet password (authenticate-only); HOTP import;
|
||||
touch-required; RENAME/DELETE; CSV/text input; on-device code calculation
|
||||
(beyond optional post-import verification, not included).
|
||||
|
||||
## Files
|
||||
|
||||
Code and docs live in the `otp-import` repo
|
||||
(`C:\Users\amal\Documents\Repos\otp-import`, remote
|
||||
`git.dngr.us/VivoKey/otp-import`): `apex_otp_import.py`, `test_oath.py`,
|
||||
`docs/design.md`, `README.md`, `.gitignore`.
|
||||
|
||||
The secret data stays **out of the repo**: the export spreadsheet
|
||||
(`otp-keys-export.xlsx`), the profiling CSV (`apex-otp-memlog.csv`, created only
|
||||
when profiling), and QR screenshots remain in the private Nextcloud folder and
|
||||
are excluded by `.gitignore`. The importer is pointed at the data with
|
||||
`--input`.
|
||||
|
||||
## Risks / notes
|
||||
|
||||
- The instance AID on a Fidesmo-provisioned Apex may differ from `build.xml`;
|
||||
mitigated by the AID fallback list and validating the SELECT response shape.
|
||||
- Contactless UID from `FF CA` may be random per tap on some configurations; it
|
||||
is logged as-is for correlation, not treated as a stable serial.
|
||||
- Keep the card still: the whole import runs in one OATH session; a mid-import
|
||||
deselect drops auth and (without profiling's re-VALIDATE) would fail PUTs with
|
||||
`6982`.
|
||||
Reference in New Issue
Block a user