OCR Direction 2: richer persisted store — letter + join templates, WordReader synthesis, versioned migration #17

Closed
opened 2026-10-20 15:33:36 +02:00 by claude · 4 comments
Collaborator

Part of #13

Part of #13 (Direction 2). Research + further split, then MRs. This is the hard half of #13 part 1 and the schema half of part 3.

HELD — do not dispatch until !12 (Direction 1, agent/ocr-letter-learning) merges. This issue was created un-assigned on purpose: the letter-template-in-store shape is defined by !12, and as of filing !12's branch has only the spike5.test.ts harness (no learnCorrection→letter-template persistence yet). Dispatch an agent only once !12 has landed and the persisted letter-template shape is readable in src/ocr/samples.ts (SampleStore). The sibling MRs !15 (read-time scaling) and !16 (word-sample diversity) are independent and proceed now.

What this covers (and must split into MRs)

The scaling MR (!15) bounds read-time work and the diversity MR (!16) grows word samples. This issue is the richer schema:

  1. Persist letter templates with retention. !12 adds per-letter $P templates derived from separable corrections. Decide and implement their persisted form in SampleStore, with a diversity-preserving per-character cap (mirror !16's "keep N most diverse, not newest" for word samples — see addSample, src/ocr/samples.ts:119) so the owner's letter set grows but stays bounded. Wire the persisted letter templates into ocrTemplates() (src/main.ts:1896) alongside the hand + fonts.
  2. Join / ligature templates (the creative bet). Capture letter-pair join shapes from corrections whose ink is separable enough to cut (reuse splitLineInto, src/ocr/recognize.ts:482), and extend WordReader (src/ocr/words.ts:183) so it can synthesise the trace of an unseen word from the owner's own letters + joins, instead of only the bundled script fonts (written()/readFonts, words.ts:150/:235). Goal: read a word the owner never wrote, in the owner's own hand. Measure the CER gain on held-out words vs fonts-only using the spike5.test.ts methodology (render a hand = a font; the spike showed joined-word CER 99%→14% once words were seen — the target here is generalisation to unseen words).
  3. Versioned migration. There is no store migration today: loadSamples (src/inkedge.ts:315) casts raw JSON straight to SampleStore and the format: 1 literal (src/ocr/samples.ts:99) is never checked. Adding optional letter/join-template fields is backward-compatible, but any shape change needs a real migrate(raw): SampleStore seam hooked into loadSamples, bumping format. Add the seam here (first format bump), with a format-1 fixture that upgrades losslessly.
  4. Retention policy for the new letter/join fields (part 3): caps + diversity selection, documented like the existing PER_WORD/REMARKS_KEPT/CORRECTIONS_KEPT consts.

Suggested split (for the agent that picks this up)

  • MR: versioned migrate() seam + persisted letter-template field & retention, wired into ocrTemplates().
  • MR: join-template capture from separable corrections + WordReader synthesis of unseen words + CER test. (This is the research-heavy one; may split again.)

Scope boundary

NOT a new engine/ML model (Direction 3, no-go per #5). Pure extension of the existing $P/DTW per-writer store. Read latency must stay within !15's budget — if letter/join templates push a hot path over, shortlist there, don't regress !15.

Validation contract (each eventual MR)

npx vitest run src/__tests__ → PASS + a test showing the richer store improves a held-out read (an unseen word synthesised from owner joins, and/or a letter-template read of a never-seen word) and a latency assertion keeping per-remark read time within budget. (No flake.nix; use npm install + npx vitest.)

Part of #13 Part of #13 (Direction 2). **Research + further split, then MRs.** This is the hard half of #13 part 1 and the schema half of part 3. > **HELD — do not dispatch until !12 (Direction 1, `agent/ocr-letter-learning`) merges.** This issue was created un-assigned on purpose: the letter-template-in-store shape is defined by !12, and as of filing !12's branch has only the `spike5.test.ts` harness (no `learnCorrection`→letter-template persistence yet). Dispatch an agent only once !12 has landed and the persisted letter-template shape is readable in `src/ocr/samples.ts` (`SampleStore`). The sibling MRs !15 (read-time scaling) and !16 (word-sample diversity) are independent and proceed now. ## What this covers (and must split into MRs) The scaling MR (!15) bounds read-time *work* and the diversity MR (!16) grows *word* samples. This issue is the richer **schema**: 1. **Persist letter templates with retention.** !12 adds per-letter $P templates derived from separable corrections. Decide and implement their persisted form in `SampleStore`, with a diversity-preserving per-character cap (mirror !16's "keep N most diverse, not newest" for word samples — see `addSample`, `src/ocr/samples.ts:119`) so the owner's letter set grows but stays bounded. Wire the persisted letter templates into `ocrTemplates()` (`src/main.ts:1896`) alongside the hand + fonts. 2. **Join / ligature templates (the creative bet).** Capture letter-pair join shapes from corrections whose ink is *separable enough* to cut (reuse `splitLineInto`, `src/ocr/recognize.ts:482`), and extend `WordReader` (`src/ocr/words.ts:183`) so it can **synthesise the trace of an unseen word from the owner's own letters + joins**, instead of only the bundled script fonts (`written()`/`readFonts`, `words.ts:150`/`:235`). Goal: read a word the owner never wrote, in the owner's own hand. Measure the CER gain on held-out words vs fonts-only using the `spike5.test.ts` methodology (render a hand = a font; the spike showed joined-word CER 99%→14% once words were seen — the target here is generalisation to *unseen* words). 3. **Versioned migration.** There is **no store migration today**: `loadSamples` (`src/inkedge.ts:315`) casts raw JSON straight to `SampleStore` and the `format: 1` literal (`src/ocr/samples.ts:99`) is never checked. Adding optional letter/join-template fields is backward-compatible, but any shape change needs a real `migrate(raw): SampleStore` seam hooked into `loadSamples`, bumping `format`. Add the seam here (first format bump), with a format-1 fixture that upgrades losslessly. 4. **Retention policy** for the new letter/join fields (part 3): caps + diversity selection, documented like the existing `PER_WORD`/`REMARKS_KEPT`/`CORRECTIONS_KEPT` consts. ## Suggested split (for the agent that picks this up) - MR: versioned `migrate()` seam + persisted **letter**-template field & retention, wired into `ocrTemplates()`. - MR: **join**-template capture from separable corrections + `WordReader` synthesis of unseen words + CER test. (This is the research-heavy one; may split again.) ## Scope boundary NOT a new engine/ML model (Direction 3, no-go per #5). Pure extension of the existing $P/DTW per-writer store. Read latency must stay within !15's budget — if letter/join templates push a hot path over, shortlist there, don't regress !15. ## Validation contract (each eventual MR) `npx vitest run src/__tests__` → PASS + a test showing the richer store improves a **held-out** read (an unseen word synthesised from owner joins, and/or a letter-template read of a never-seen word) and a latency assertion keeping per-remark read time within budget. (No `flake.nix`; use `npm install` + `npx vitest`.)
Author
Collaborator

✅ Agent run completed

6m11s · 27 turns · $2.28 · pushed agent/ocr-letter-diversity, agent/ocr-store-migrate · session saved — new comments resume this run

✅ **Agent run completed** _6m11s · 27 turns · $2.28 · pushed `agent/ocr-letter-diversity`, `agent/ocr-store-migrate` · session saved — new comments resume this run_
Author
Collaborator

What I found

  • Part 1 (persist + wire letter templates) was already delivered by !12 — not just the spike. SampleStore.letters is populated by learnLetters (src/ocr/samples.ts:199) and read via lettersTemplates (recognize.ts:609) wired into ocrTemplates() (main.ts:1918, cache-keyed on learned-letter count). So this issue's remaining work is narrower than the description assumed.
  • Three real gaps remain, cleanly separable: (1) letter retention is newest-wins (.slice(-PER_LETTER=3), samples.ts:215), not diversity-preserving like !16's word retention; (2) there is still no migrate() seam — loadSamples (inkedge.ts:315) casts raw JSON and the format literal is never checked; (3) joins + WordReader synthesis of unseen words is entirely unbuilt (WordReader only synthesises from fonts, words.ts:150/:256; store.letters is unused there).
  • The two schema gaps are fully specified → MRs; the join/synthesis gap is a genuine research bet with an open empirical outcome → child issue for a planning agent to design and split.

Created

  • !18 — feat(ocr): diversity-preserving per-letter retention (part 1 remainder + part 4 for letters; generalise keepDiverse to a cap, use it in learnLetters, red-gate test). Branch agent/ocr-letter-diversity.
  • !19 — refactor(ocr): versioned migrate() seam for the sample store (part 3; migrate(raw) + CURRENT_FORMAT=2, routed through loadSamples, lossless format-1 fixture, red-gate test). Branch agent/ocr-store-migrate.
  • #20 (child, blocks this issue) — join/ligature templates + WordReader synthesis of unseen words + held-out CER measurement (the creative bet; soft-ordered after !18/!19; a negative CER result is a valid "do not ship" outcome).

Follow-ups

  • !18 and !19 are independent and can both proceed now (both branch off main); a trivial samples.ts conflict on merge is expected and fine.
  • #20 reuses !18's generalised keepDiverse(samples, cap) and !19's migrate() seam for its joins? field — its MRs should land after !18/!19 to avoid conflicts. No hard is_blocked_by gate set (soft dep on MRs; joins? is an optional backward-compatible field).
  • This issue stays open until #20 lands (child blocks it); !18/!19 are Part of (not Closes), so they won't auto-close it.
## What I found - **Part 1 (persist + wire letter templates) was already delivered by !12** — not just the spike. `SampleStore.letters` is populated by `learnLetters` (`src/ocr/samples.ts:199`) and read via `lettersTemplates` (`recognize.ts:609`) wired into `ocrTemplates()` (`main.ts:1918`, cache-keyed on learned-letter count). So this issue's remaining work is narrower than the description assumed. - **Three real gaps remain**, cleanly separable: (1) letter retention is *newest-wins* (`.slice(-PER_LETTER=3)`, `samples.ts:215`), not diversity-preserving like !16's word retention; (2) there is still **no `migrate()` seam** — `loadSamples` (`inkedge.ts:315`) casts raw JSON and the `format` literal is never checked; (3) joins + `WordReader` synthesis of unseen words is entirely unbuilt (`WordReader` only synthesises from fonts, `words.ts:150`/`:256`; `store.letters` is unused there). - The two schema gaps are **fully specified → MRs**; the join/synthesis gap is a genuine research bet with an open empirical outcome → **child issue** for a planning agent to design and split. ## Created - **!18** — `feat(ocr): diversity-preserving per-letter retention` (part 1 remainder + part 4 for letters; generalise `keepDiverse` to a cap, use it in `learnLetters`, red-gate test). Branch `agent/ocr-letter-diversity`. - **!19** — `refactor(ocr): versioned migrate() seam for the sample store` (part 3; `migrate(raw)` + `CURRENT_FORMAT=2`, routed through `loadSamples`, lossless format-1 fixture, red-gate test). Branch `agent/ocr-store-migrate`. - **#20** (child, *blocks* this issue) — join/ligature templates + `WordReader` synthesis of unseen words + held-out CER measurement (the creative bet; soft-ordered after !18/!19; a negative CER result is a valid "do not ship" outcome). ## Follow-ups - !18 and !19 are independent and can both proceed now (both branch off `main`); a trivial `samples.ts` conflict on merge is expected and fine. - #20 reuses !18's generalised `keepDiverse(samples, cap)` and !19's `migrate()` seam for its `joins?` field — its MRs should land after !18/!19 to avoid conflicts. No hard `is_blocked_by` gate set (soft dep on MRs; `joins?` is an optional backward-compatible field). - This issue stays open until #20 lands (child `blocks` it); !18/!19 are `Part of` (not `Closes`), so they won't auto-close it.
Author
Collaborator

🤖 Agent is working on this — turn 28

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 18:10:18 UTC

Last message:

Checking status: all of #17's parts have since landed or been resolved, so I'm verifying that and then wrapping up the issue.

🤖 **Agent is working on this** — turn 28 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 18:10:18 UTC_ **Last message:** > Checking status: all of #17's parts have since landed or been resolved, so I'm verifying that and then wrapping up the issue.
Author
Collaborator

Since my last update: every part of this issue is now resolved, so I'm closing it.

  • Part 1 (persisted letter templates, wired into ocrTemplates()): shipped by !12. Its diversity-preserving retention shipped in !18 (merged). On main, keepDiverse(samples, cap) handles both PER_WORD and PER_LETTER = 6 (src/ocr/samples.ts:214, :272).
  • Part 3 (versioned migration): shipped in !19 (merged). migrate(raw) and CURRENT_FORMAT = 2 are in src/ocr/samples.ts:136/:176, loadSamples calls it (src/inkedge.ts:321), and src/__tests__/migrate.test.ts checks that a format-1 store upgrades without loss.
  • Part 4 (retention): done for letters as part of !18. Joins need no retention policy, since no joins field was added.
  • Part 2 (joins and unseen-word synthesis): the answer is no-go, and nothing was shipped. The decisive spike #21 showed there is no reliable way to label bigrams in joined ink: assignment was only 82–96% correct on semi-joined ink, and fully joined ink has nothing to cut. With bigrams that could actually be collected, synthesis performed the same as fonts-only. Even with perfect bigrams, the trust threshold limits it to 5–48% of reads. #20 was closed for this reason. Measurements are in #21; the probe code is on branch spike/21-bigram-probe (evidence only, not for merge).

I'm not filing the two leads from #21 as issues; they are left for #13's owner:

  • A trust rule based on the gap to the runner-up. It only pays off with labelled owner material, which doesn't exist today. Fonts alone rank poorly (top-1 CER 0.63–0.69), so this rule wouldn't save the font pass either.
  • Growing store.words coverage. Words are already learned from every accepted or corrected remark (learnCorrection, src/ocr/samples.ts:421). The remaining gap is lines where the ink and text word counts differ by 2 or more, so nothing is learned. Those cases are already shown to the owner through describeUnlearned. There is no specific change to hand off yet.

Side finding for #13: on any owner hand that isn't literally a bundled font, the font pass of WordReader is never trusted. The correct word costs 0.22–0.40 against FONT_TRUST = 0.08 (src/ocr/words.ts:180). On real joined handwriting, the word reader effectively reads only from the owner's own word samples.

Since my last update: every part of this issue is now resolved, so I'm closing it. - **Part 1 (persisted letter templates, wired into `ocrTemplates()`):** shipped by !12. Its diversity-preserving retention shipped in **!18** (merged). On `main`, `keepDiverse(samples, cap)` handles both `PER_WORD` and `PER_LETTER = 6` (`src/ocr/samples.ts:214`, `:272`). - **Part 3 (versioned migration):** shipped in **!19** (merged). `migrate(raw)` and `CURRENT_FORMAT = 2` are in `src/ocr/samples.ts:136`/`:176`, `loadSamples` calls it (`src/inkedge.ts:321`), and `src/__tests__/migrate.test.ts` checks that a format-1 store upgrades without loss. - **Part 4 (retention):** done for letters as part of !18. Joins need no retention policy, since no joins field was added. - **Part 2 (joins and unseen-word synthesis):** the answer is **no-go**, and nothing was shipped. The decisive spike #21 showed there is no reliable way to label bigrams in joined ink: assignment was only 82–96% correct on semi-joined ink, and fully joined ink has nothing to cut. With bigrams that could actually be collected, synthesis performed the same as fonts-only. Even with perfect bigrams, the trust threshold limits it to 5–48% of reads. #20 was closed for this reason. Measurements are in #21; the probe code is on branch `spike/21-bigram-probe` (evidence only, not for merge). **I'm not filing the two leads from #21 as issues; they are left for #13's owner:** - **A trust rule based on the gap to the runner-up.** It only pays off with labelled owner material, which doesn't exist today. Fonts alone rank poorly (top-1 CER 0.63–0.69), so this rule wouldn't save the font pass either. - **Growing `store.words` coverage.** Words are already learned from every accepted or corrected remark (`learnCorrection`, `src/ocr/samples.ts:421`). The remaining gap is lines where the ink and text word counts differ by 2 or more, so nothing is learned. Those cases are already shown to the owner through `describeUnlearned`. There is no specific change to hand off yet. **Side finding for #13:** on any owner hand that isn't literally a bundled font, the font pass of `WordReader` is never trusted. The correct word costs 0.22–0.40 against `FONT_TRUST = 0.08` (`src/ocr/words.ts:180`). On real joined handwriting, the word reader effectively reads only from the owner's own word samples.
claude 2026-10-20 20:10:18 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
daniel/obsidian-remarkable-round-trip#17
No description provided.