OCR Direction 1: letter-level learning from corrections (writer-adaptive $P templates) #12
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip!12
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/ocr-letter-learning"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #5. Implements Direction 1 from the #5 spike — the lowest-risk, measured-GO direction. The spike harness is on this branch (
src/__tests__/spike5.test.ts); runnpx vitest run src/__tests__/spike5.test.ts --reporter=verboseto reproduce the numbers below.Why (measured evidence, from the #5 spike)
Today continuous learning is word-granular (
learnCorrectionstores whole-word samples;WordReadermatches them). On the synthetic labelled corpus (the repo's own font-render methodology), on a separable/print hand that differs from the generic templates:Letter-level learning cuts held-out CER 38.1% → 20.8% (≈45% relative) and, unlike word-level learning, generalises to words the owner never corrected. The control (adding the same number of wrong-hand templates) does not help, confirming the gain comes from matching the owner's own letter shapes.
Scope boundary (important): on joined cursive ink the gain is ~nil (held-out 95.8% → 88.9%) because the ink cannot be segmented into letters — that regime stays owned by WritePad / the local VLM (#3). So letter-learning must only fire when the corrected word's ink is separable; never teach letter shapes from cursive ink (it would pollute the template set).
What to change
src/ocr/samples.ts—learnCorrection: for each paired(word, wordInk)whose ink separates into exactly[...word].lengthletter clusters with clear gaps, derive per-letter ink and store it in a newstore.letters: Record<string, Point[][][]>(char → up toPER_WORD-style cap of samples, newest kept). If the ink does not cleanly separate (joined cursive), keep only today's word sample — do not guess letter boundaries. Factor the gap-based letter cutter out of the prototype'ssplitLineIntoreuse into a named helper inrecognize.ts(e.g.cutLetters(wordStrokes, count)), and add a separability test (cluster count ≈ letter count AND inter-cluster gaps above a fraction of cap).recognize.tsorsamples.ts):lettersTemplates(store)→Template[], one per captured letter sample, mirroringtemplatesFrombut from ink:cloud = cloudOf(strokes), and baseline-relativetop/bottom/widthin cap units (the prototype'slearnLettersshows the maths). Pick aweight≥ the handwriting-sheet weight (sheet uses 0.9 atmain.ts:1901) so the owner's accumulated hand is preferred over print fonts — tune against the spike harness.src/main.ts—ocrTemplates()(≈L1896): appendlettersTemplates(store)to the template set alongside the sheet-derived hand and the print fonts. The template cache is keyed only onhandVersion(L1897-1898) — extend the key to also change when the letters store grows (e.g. a revision counter or letter-sample count), or clearthis.templateswhereverlearnFromCorrectionsmutates the store, so new corrections take effect without a reload.letters?toSampleStore; initialise inemptyStore(); oldsamples.jsonwithoutlettersmust still load (treat as{}). Cap per-char samples and prune (mirrorPER_WORD/prune).Constraints & decisions
learnFromNotesalready callslearnCorrection, so the new path is exercised automatically on sync — no new call site needed, only the cache invalidation in (3).Validation contract
npx vitest run src/__tests__→ PASS (no regressions). (Note: the #5 issue namesnix develop -c npx vitest run src/__tests__, but there is noflake.nixin the repo — see the #5 comment; the working command isnpm installthennpx vitest run src/__tests__.)learnCorrectionpath: render a separable hand (e.g.readabilitywithslant), feed TRAIN words as corrections throughlearnCorrection, then assertcer(recognize(heldOutWordInk, templatesWithLearnedLetters, lexicon), want) < cer(recognize(heldOutWordInk, baselineTemplates, lexicon), want)for a word not in TRAIN. Red gate: the same assertion fails before the change (baseline ≈ learned, both ≈38%). Replacespike5.test.tswith this focused test (or keep it as an integration-level companion).allure-rendered correction).🤖 Generated with Claude Code
claude referenced this pull request2026-10-20 15:25:05 +02:00
✅ Agent run completed
11m51s · 53 turns · $3.98 · pushed
agent/ocr-letter-learning· session saved — new comments resume this runclaude referenced this pull request2026-10-20 15:25:44 +02:00
claude referenced this pull request2026-10-20 15:26:09 +02:00
claude referenced this pull request2026-10-20 15:26:11 +02:00
claude referenced this pull request2026-10-20 15:26:31 +02:00
claude referenced this pull request2026-10-20 15:26:32 +02:00
claude referenced this pull request2026-10-20 15:29:35 +02:00
claude referenced this pull request2026-10-20 15:32:54 +02:00
claude referenced this pull request2026-10-20 15:34:21 +02:00
claude referenced this pull request2026-10-20 15:34:22 +02:00
What changed
recognize.ts— factored the gap cutter intocutGroups; addedcutLetters(wordStrokes, count)(separability-gated: requires ≥countclusters and every cut on a clear gap >8% of word height, so joined cursive is never segmented) andlettersTemplates(store)(oneTemplateper learned letter sample;weight0.9, ≥ the sheet weight so the owner's hand beats print fonts).samples.ts—learnCorrectionnow also stores per-letter ink in a new optionalstore.letters(char → cap-unit, baseline-at-y=0 strokes, cappedPER_LETTER=3) for each paired word whose ink separates; old stores withoutlettersload as{};emptyStore()initialises it.main.ts—ocrTemplates()appendslettersTemplates(store); cache key now includes the learned-letter count, so corrections take effect without a reload. WordReader/word-sample path untouched (purely additive).Verify
npm run build(tsc + esbuild + bundle check) clean, eslint clean,npx vitest run src/__tests__→ 655 passed / 58 files. Newsrc/__tests__/letterlearning.test.tsasserts, on the reallearnCorrection→lettersTemplatespath, that a separable hand (readability+slant vs hershey-sans templates) lowers held-out CER 38.1% → ~21.4% on words never corrected (red gate: baseline-vs-baseline is an equality and fails), and that joined-cursive (allure) ink learns zero letter shapes (store.lettersstays empty). Replaced the throwawayspike5.test.tsprototype with this focused test.Note: no CI pipeline runs on MR branches in this project (the only recent run is a pre-existing
mainfailure, unrelated to this diff) — verification above is local.Follow-ups
claude referenced this pull request2026-10-20 16:34:09 +02:00
claude referenced this pull request2026-10-20 16:34:09 +02:00
claude referenced this pull request2026-10-20 17:08:52 +02:00
claude referenced this pull request2026-10-20 17:09:13 +02:00
claude referenced this pull request2026-10-20 17:10:46 +02:00
claude referenced this pull request2026-10-20 17:14:13 +02:00
claude referenced this pull request2026-10-20 17:14:36 +02:00
claude referenced this pull request2026-10-20 17:14:50 +02:00
claude referenced this pull request2026-10-20 17:20:18 +02:00
claude referenced this pull request2026-10-20 17:21:40 +02:00
claude referenced this pull request2026-10-20 17:21:41 +02:00
claude referenced this pull request2026-10-20 18:51:57 +02:00
claude referenced this pull request2026-10-20 18:52:05 +02:00
claude referenced this pull request2026-10-20 18:52:06 +02:00