OCR Direction 1: letter-level learning from corrections (writer-adaptive $P templates) #12

Merged
daniel merged 3 commits from agent/ocr-letter-learning into main 2026-10-20 16:30:18 +02:00
Collaborator

Part of #5. Implements Direction 1 from the #5 spike — the lowest-risk, measured-GO direction. The spike harness is on this branch (src/__tests__/spike5.test.ts); run npx vitest run src/__tests__/spike5.test.ts --reporter=verbose to reproduce the numbers below.

Why (measured evidence, from the #5 spike)

Today continuous learning is word-granular (learnCorrection stores whole-word samples; WordReader matches them). On the synthetic labelled corpus (the repo's own font-render methodology), on a separable/print hand that differs from the generic templates:

Reader TRAIN CER (seen words) HELD-OUT CER (unseen words)
built-in letter reader (baseline) 38.2% 38.1%
+ word-level learning (today) 23.9% 38.1% (no gain on unseen)
+ letter-level learning (this MR) 17.3% 20.8%
control: +same count of generic templates — 40.7% (no gain → not a count artifact)

Letter-level learning cuts held-out CER 38.1% → 20.8% (≈45% relative) and, unlike word-level learning, generalises to words the owner never corrected. The control (adding the same number of wrong-hand templates) does not help, confirming the gain comes from matching the owner's own letter shapes.

Scope boundary (important): on joined cursive ink the gain is ~nil (held-out 95.8% → 88.9%) because the ink cannot be segmented into letters — that regime stays owned by WritePad / the local VLM (#3). So letter-learning must only fire when the corrected word's ink is separable; never teach letter shapes from cursive ink (it would pollute the template set).

What to change

  1. src/ocr/samples.ts — learnCorrection: for each paired (word, wordInk) whose ink separates into exactly [...word].length letter clusters with clear gaps, derive per-letter ink and store it in a new store.letters: Record<string, Point[][][]> (char → up to PER_WORD-style cap of samples, newest kept). If the ink does not cleanly separate (joined cursive), keep only today's word sample — do not guess letter boundaries. Factor the gap-based letter cutter out of the prototype's splitLineInto reuse into a named helper in recognize.ts (e.g. cutLetters(wordStrokes, count)), and add a separability test (cluster count ≈ letter count AND inter-cluster gaps above a fraction of cap).
  2. New template builder (in recognize.ts or samples.ts): lettersTemplates(store) → Template[], one per captured letter sample, mirroring templatesFrom but from ink: cloud = cloudOf(strokes), and baseline-relative top/bottom/width in cap units (the prototype's learnLetters shows the maths). Pick a weight ≥ the handwriting-sheet weight (sheet uses 0.9 at main.ts:1901) so the owner's accumulated hand is preferred over print fonts — tune against the spike harness.
  3. src/main.ts — ocrTemplates() (≈L1896): append lettersTemplates(store) to the template set alongside the sheet-derived hand and the print fonts. The template cache is keyed only on handVersion (L1897-1898) — extend the key to also change when the letters store grows (e.g. a revision counter or letter-sample count), or clear this.templates wherever learnFromCorrections mutates the store, so new corrections take effect without a reload.
  4. Store migration: add optional letters? to SampleStore; initialise in emptyStore(); old samples.json without letters must still load (treat as {}). Cap per-char samples and prune (mirror PER_WORD / prune).

Constraints & decisions

  • Do not touch the WordReader/word-sample path — it stays as the whole-cursive-word reader. This MR is purely additive at letter granularity.
  • Do not learn letters from cursive ink (separability gate above) — this is the single most important safety property.
  • Keep it dependency-free (no ML libs); this is pure $P/geometry, same as today.
  • learnFromNotes already calls learnCorrection, so the new path is exercised automatically on sync — no new call site needed, only the cache invalidation in (3).

Validation contract

  • npx vitest run src/__tests__ → PASS (no regressions). (Note: the #5 issue names nix develop -c npx vitest run src/__tests__, but there is no flake.nix in the repo — see the #5 comment; the working command is npm install then npx vitest run src/__tests__.)
  • New held-out test on the real learnCorrection path: render a separable hand (e.g. readability with slant), feed TRAIN words as corrections through learnCorrection, then assert cer(recognize(heldOutWordInk, templatesWithLearnedLetters, lexicon), want) < cer(recognize(heldOutWordInk, baselineTemplates, lexicon), want) for a word not in TRAIN. Red gate: the same assertion fails before the change (baseline ≈ learned, both ≈38%). Replace spike5.test.ts with this focused test (or keep it as an integration-level companion).
  • Add a test asserting letter-learning does not fire on joined-cursive ink (store.letters stays empty for an allure-rendered correction).

🤖 Generated with Claude Code

Part of #5. Implements **Direction 1** from the #5 spike — the lowest-risk, measured-GO direction. The spike harness is on this branch (`src/__tests__/spike5.test.ts`); run `npx vitest run src/__tests__/spike5.test.ts --reporter=verbose` to reproduce the numbers below. ## Why (measured evidence, from the #5 spike) Today continuous learning is **word-granular** (`learnCorrection` stores whole-word samples; `WordReader` matches them). On the synthetic labelled corpus (the repo's own font-render methodology), on a **separable/print hand that differs from the generic templates**: | Reader | TRAIN CER (seen words) | HELD-OUT CER (unseen words) | |---|---|---| | built-in letter reader (baseline) | 38.2% | 38.1% | | + word-level learning (today) | 23.9% | **38.1%** (no gain on unseen) | | **+ letter-level learning (this MR)** | 17.3% | **20.8%** | | control: +same count of generic templates | — | 40.7% (no gain → not a count artifact) | Letter-level learning cuts held-out CER **38.1% → 20.8% (≈45% relative)** and, unlike word-level learning, **generalises to words the owner never corrected**. The control (adding the same number of *wrong-hand* templates) does not help, confirming the gain comes from matching the owner's own letter shapes. **Scope boundary (important):** on *joined cursive* ink the gain is ~nil (held-out 95.8% → 88.9%) because the ink cannot be segmented into letters — that regime stays owned by WritePad / the local VLM (#3). So letter-learning must only fire when the corrected word's ink is **separable**; never teach letter shapes from cursive ink (it would pollute the template set). ## What to change 1. **`src/ocr/samples.ts` — `learnCorrection`:** for each paired `(word, wordInk)` whose ink separates into exactly `[...word].length` letter clusters with clear gaps, derive per-letter ink and store it in a new `store.letters: Record<string, Point[][][]>` (char → up to `PER_WORD`-style cap of samples, newest kept). If the ink does **not** cleanly separate (joined cursive), keep only today's word sample — do not guess letter boundaries. Factor the gap-based letter cutter out of the prototype's `splitLineInto` reuse into a named helper in `recognize.ts` (e.g. `cutLetters(wordStrokes, count)`), and add a separability test (cluster count ≈ letter count AND inter-cluster gaps above a fraction of cap). 2. **New template builder** (in `recognize.ts` or `samples.ts`): `lettersTemplates(store)` → `Template[]`, one per captured letter sample, mirroring `templatesFrom` but from ink: `cloud = cloudOf(strokes)`, and baseline-relative `top`/`bottom`/`width` in cap units (the prototype's `learnLetters` shows the maths). Pick a `weight` ≥ the handwriting-sheet weight (sheet uses 0.9 at `main.ts:1901`) so the owner's accumulated hand is preferred over print fonts — tune against the spike harness. 3. **`src/main.ts` — `ocrTemplates()` (≈L1896):** append `lettersTemplates(store)` to the template set alongside the sheet-derived hand and the print fonts. The template cache is keyed only on `handVersion` (L1897-1898) — extend the key to also change when the letters store grows (e.g. a revision counter or letter-sample count), or clear `this.templates` wherever `learnFromCorrections` mutates the store, so new corrections take effect without a reload. 4. **Store migration:** add optional `letters?` to `SampleStore`; initialise in `emptyStore()`; old `samples.json` without `letters` must still load (treat as `{}`). Cap per-char samples and prune (mirror `PER_WORD` / `prune`). ## Constraints & decisions - Do **not** touch the WordReader/word-sample path — it stays as the whole-cursive-word reader. This MR is purely additive at letter granularity. - Do **not** learn letters from cursive ink (separability gate above) — this is the single most important safety property. - Keep it dependency-free (no ML libs); this is pure $P/geometry, same as today. - `learnFromNotes` already calls `learnCorrection`, so the new path is exercised automatically on sync — no new call site needed, only the cache invalidation in (3). ## Validation contract - `npx vitest run src/__tests__` → PASS (no regressions). *(Note: the #5 issue names `nix develop -c npx vitest run src/__tests__`, but there is **no `flake.nix`** in the repo — see the #5 comment; the working command is `npm install` then `npx vitest run src/__tests__`.)* - **New held-out test** on the real `learnCorrection` path: render a separable hand (e.g. `readability` with `slant`), feed TRAIN words as corrections through `learnCorrection`, then assert `cer(recognize(heldOutWordInk, templatesWithLearnedLetters, lexicon), want) < cer(recognize(heldOutWordInk, baselineTemplates, lexicon), want)` for a word **not** in TRAIN. **Red gate:** the same assertion fails before the change (baseline ≈ learned, both ≈38%). Replace `spike5.test.ts` with this focused test (or keep it as an integration-level companion). - Add a test asserting letter-learning does **not** fire on joined-cursive ink (store.letters stays empty for an `allure`-rendered correction). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude self-assigned this 2026-10-20 15:25:02 +02:00
Reference artifact for the Direction-1 MR: measures built-in / word-level /
letter-level / WritePad CER on a synthetic labelled set, with a prototype of
letter-level learning from corrections. Shows letter-level learning generalises
to held-out words (separable hand) where word-level learning does not.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author
Collaborator

✅ Agent run completed

11m51s · 53 turns · $3.98 · pushed agent/ocr-letter-learning · session saved — new comments resume this run

✅ **Agent run completed** _11m51s · 53 turns · $3.98 · pushed `agent/ocr-letter-learning` · session saved — new comments resume this run_
Extend learnCorrection to derive per-letter $P templates when a corrected
word's ink separates cleanly into its letters, enriching the owner's own
hand beyond the one-off handwriting sheet and generalising to words never
corrected (held-out CER 38.1% -> ~21% on the #5 spike corpus).

- recognize.ts: factor the gap cutter into cutGroups; add cutLetters (a
  separability-gated letter cutter: >= count clusters AND every cut on a
  clear gap, so joined cursive is never segmented) and lettersTemplates.
- samples.ts: store.letters (char -> cap-unit, baseline-relative ink),
  capped per char; migrates old stores (letters optional, {} default).
- main.ts: ocrTemplates appends lettersTemplates; cache key tracks the
  learned-letter count so corrections take effect without a reload.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Focused test on the real learnCorrection -> lettersTemplates path: a
separable hand that differs from the generic templates lowers held-out
CER (red gate: equal without learned letters), and joined-cursive ink
learns no letter shapes. Replaces the removed #5 spike harness.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author
Collaborator

What changed

  • recognize.ts — factored the gap cutter into cutGroups; added cutLetters(wordStrokes, count) (separability-gated: requires ≥count clusters and every cut on a clear gap >8% of word height, so joined cursive is never segmented) and lettersTemplates(store) (one Template per learned letter sample; weight 0.9, ≥ the sheet weight so the owner's hand beats print fonts).
  • samples.ts — learnCorrection now also stores per-letter ink in a new optional store.letters (char → cap-unit, baseline-at-y=0 strokes, capped PER_LETTER=3) for each paired word whose ink separates; old stores without letters load as {}; emptyStore() initialises it.
  • main.ts — ocrTemplates() appends lettersTemplates(store); cache key now includes the learned-letter count, so corrections take effect without a reload. WordReader/word-sample path untouched (purely additive).

Verify

npm run build (tsc + esbuild + bundle check) clean, eslint clean, npx vitest run src/__tests__ → 655 passed / 58 files. New src/__tests__/letterlearning.test.ts asserts, on the real learnCorrection→lettersTemplates path, that a separable hand (readability+slant vs hershey-sans templates) lowers held-out CER 38.1% → ~21.4% on words never corrected (red gate: baseline-vs-baseline is an equality and fails), and that joined-cursive (allure) ink learns zero letter shapes (store.letters stays empty). Replaced the throwaway spike5.test.ts prototype with this focused test.

Note: no CI pipeline runs on MR branches in this project (the only recent run is a pre-existing main failure, unrelated to this diff) — verification above is local.

Follow-ups

  • Directions 2 (persisted growing per-writer store) and 3 (compact trainable online-HTR) from #5 remain separate, higher-risk tracks — not in scope here.
## What changed - **`recognize.ts`** — factored the gap cutter into `cutGroups`; added `cutLetters(wordStrokes, count)` (separability-gated: requires ≥`count` clusters **and** every cut on a clear gap >8% of word height, so joined cursive is never segmented) and `lettersTemplates(store)` (one `Template` per learned letter sample; `weight` 0.9, ≥ the sheet weight so the owner's hand beats print fonts). - **`samples.ts`** — `learnCorrection` now also stores per-letter ink in a new optional `store.letters` (char → cap-unit, baseline-at-y=0 strokes, capped `PER_LETTER=3`) for each paired word whose ink separates; old stores without `letters` load as `{}`; `emptyStore()` initialises it. - **`main.ts`** — `ocrTemplates()` appends `lettersTemplates(store)`; cache key now includes the learned-letter count, so corrections take effect without a reload. WordReader/word-sample path untouched (purely additive). ## Verify `npm run build` (tsc + esbuild + bundle check) clean, eslint clean, `npx vitest run src/__tests__` → **655 passed / 58 files**. New `src/__tests__/letterlearning.test.ts` asserts, on the real `learnCorrection`→`lettersTemplates` path, that a separable hand (readability+slant vs hershey-sans templates) lowers **held-out CER 38.1% → ~21.4%** on words never corrected (red gate: baseline-vs-baseline is an equality and fails), and that joined-cursive (`allure`) ink learns **zero** letter shapes (`store.letters` stays empty). Replaced the throwaway `spike5.test.ts` prototype with this focused test. Note: no CI pipeline runs on MR branches in this project (the only recent run is a pre-existing `main` failure, unrelated to this diff) — verification above is local. ## Follow-ups - Directions 2 (persisted growing per-writer store) and 3 (compact trainable online-HTR) from #5 remain separate, higher-risk tracks — not in scope here.
daniel merged commit ffc539083e into main 2026-10-20 16:30:18 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!12
No description provided.