OCR Direction 2: join/ligature templates + WordReader synthesis of unseen words (the creative bet) #20
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Blocks
Depends on
#14 OCR Direction 3: trainable in-plugin online-HTR — PARKED (no-go on current evidence)
daniel/obsidian-remarkable-round-trip
#17 OCR Direction 2: richer persisted store — letter + join templates, WordReader synthesis, versioned migration
daniel/obsidian-remarkable-round-trip
#21 OCR Dir-2: decisive spike — bigram-tiling owner-hand synthesis + a fair held-out measurement (gates the join MRs)
daniel/obsidian-remarkable-round-trip
Reference
daniel/obsidian-remarkable-round-trip#20
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #17
Part of #17 (OCR Direction 2 — richer persisted store), Part of #13. This is issue #17 part 2, the research-heavy creative bet — carved out as its own issue because it has open design decisions and an open empirical outcome, and will likely split again into its own MRs. (The other two parts of #17 are already in flight as MRs: !18 letter-template diversity retention, !19 versioned
migrate()seam.)Goal
Let
WordReader(src/ocr/words.ts:191) read a word the owner has never written, in the owner's own hand — by synthesising that word's expected trace from the owner's own letter templates (store.letters, landed in !12) plus newly-captured join / ligature templates, instead of only from the bundled script fonts.Where we are today
WordReader.read(words.ts:226) scores the ink's trace (resampled, height-normalised main-line polyline + up/down profile) by DTW against: (a) the owner's whole-word samples (this.samples), and (b) candidate words rendered inGlyphSourcefonts viawritten()(words.ts:150) +readFonts()(words.ts:256).store.letters) exist and feed the letter reader (lettersTemplates→recognize), butWordReaderdoes not use them. There is no join capture and no owner-hand word synthesis.Open research questions / design (resolve these, then split into MRs)
cutLetters(recognize.ts:588) returnsnullfor exactly the joined-cursive case where ligatures matter (the letter-separability gate). The issue's lead issplitLineInto(recognize.ts:571) — cuts at widest gaps with no letter-min-gap. Design how to capture letter-pair (bigram) join shapes from corrections whose ink is separable enough to cut into pairs even when not into single letters, and define what the stored "join" actually is (the connecting ink between two letter clusters? the whole bigram shape?). Confirm empirically that enough joins can be harvested from realistic corrections to matter.WordReadercompares traces, not $P clouds.store.lettersholds strokes in cap units (baseline y=0) from which a per-letter trace is derivable (traceOf). To build an unseen word's candidate trace, stitch letter traces + join traces end-to-start in reading order, placed by advance/width, then run throughtraceOf. Decide the fallback for a letter→letter transition with no captured join (straight connector? the font's join? skip the pair?), and how to choose among multiple samples per letter/join.readFonts, scored by the samedtw, with its own trust threshold betweenSAMPLE_TRUST = 0.2(a word the owner verifiably wrote) andFONT_TRUST = 0.08(a foreign font) — a synthesised owner word is neither.joins?: Record<string, Point[][][]>toSampleStore(keyed by letter-pair, e.g."th"), captured in alearnJoinsalongsidelearnCorrection. Give it a diversity-preserving per-pair cap — reuse the generalisedkeepDiverse(samples, cap)from !18 — and document aPER_JOINconst likePER_WORD/PER_LETTER. Route the new field through themigrate()seam from !19.src/__tests__/letterlearning.test.ts's font-as-hand methodology (hand()renders text in a font = a "hand";wordInk(),cer(), the fixedTRAIN/HELD_OUTsplit). FeedTRAINwords as corrections, then measure mean CER onHELD_OUTwords the owner never wrote, read viaWordReaderwith owner-hand synthesis vs. fonts-only. Target: synthesis beats fonts-only on held-out words (the #5 spike showed joined-word CER 99%→14% once words were seen; the bet here is generalisation to unseen words). Red gate: fonts-only on both sides is an equality that fails before the change.Scope boundary
readFonts, so it must go through the same profile+width shortlist (SAMPLE_SHORTLIST/ the 150-candidate cap) and thethis.tracescache — do not add an unbounded per-read cost. (mr-15 codebase note: read cost must stay FLAT in store size.) A latency assertion keeping per-remark read time within budget is part of the validation contract.migrate()seam are already covered by !18 / !19 — do not re-do them here; build on them.A valid outcome is "no"
If, with a faithful implementation, synthesis does not beat fonts-only on held-out words, that is a legitimate negative result — report it on this issue and do not ship a feature that doesn't earn its complexity. The measurement decides whether the join MR lands.
Suggested MR split (for the agent that picks this up)
joins?schema +learnJoinscapture (from pair-separable corrections) +PER_JOINdiversity retention, through the !19 migrate seam. Unit test: joins are captured from separable ink and bounded by the cap.WordReaderowner-hand synthesis (letters + joins) + trust threshold + the held-out CER test and the latency assertion. (May split further if synthesis design proves large.)Ordering
Soft-depends on !18 (reuse
keepDiverse(samples, cap)forPER_JOIN) and !19 (themigrate()seam forjoins?). Neither is a hard blocker — thejoins?field is optional and backward-compatible — but land this work's MRs after !18/!19 merge to avoid conflicts insrc/ocr/samples.ts.Validation contract (each eventual MR)
npm installthennpx vitest run src/__tests__→ PASS, plus (for the synthesis MR) a test showing the richer store improves a held-out read (an unseen word synthesised from owner letters+joins) over fonts-only, and a latency assertion keeping per-remark read time within !15's budget. (Noflake.nix.)claude referenced this issue2026-10-20 17:14:13 +02:00
✅ Agent run completed
7m21s · 28 turns · $2.12 · session saved — new comments resume this run
claude referenced this issue2026-10-20 17:14:50 +02:00
claude referenced this issue2026-10-20 17:14:50 +02:00
What I found
store.letters(the synthesis material) is populated only vialearnLetters→cutLetters, which returnsnullfor joined cursive by design — so letter templates exist for separable hands and are absent for the joined hands whereWordReaderis the operative reader.letterlearning.test.tsfont-as-hand methodology (TRAIN as corrections, held-out viaWordReader): separable hands capture 16–21 letter shapes but fonts-only held-out CER = 1.000 / 0 trusted (a separable trace can't match the joined bundled script fonts — and separable hands are already generalised by the letter reader, !12). Joined hands (allure/hershey-script) capture 0 letters (synthesis can't run) and fonts-only already reads ~0.35–0.45 because the test "hand" is itself a bundled script font.splitLineIntocuts even allure into ⌈n/2⌉ clusters, but at widest gaps with no guarantee a cluster maps to a labelled adjacent letter-pair — assigning cluster→bigram from joined ink is the exact "guess which ink is which → teach wrong shapes" hazardlearnCorrectionalready refuses. The 4 bundled fonts have no semi-joined member, the only regime whereWordReaderis operative and material is capturable.This is a strong structural obstacle, not yet a final "no" (no faithful synthesis implementation was run). It must be resolved before the schema + synthesis MRs are worth building — so I did not hand off the issue's 2-MR split as-is (both MRs rest on the falsified premise). Instead I scoped the decisive spike with a corrected design (bigram-tiling synthesis, honest labelled-bigram harvesting, and a fair fonts-only baseline measured on a joined/semi-joined hand rather than the CER-1.0 straw man).
Created
joins?schema MR and the synthesis MR ever land. Internal 2-MR split + validation contract + the !18/!19 soft-dependency are specified there.Follow-ups
keepDiverse(samples, cap)) and !19 (migrate()seam) merge — both currently open — to avoidsrc/ocr/samples.tsconflicts.claude referenced this issue2026-10-20 18:52:05 +02:00
claude referenced this issue2026-10-20 18:52:06 +02:00
claude referenced this issue2026-10-20 18:59:13 +02:00
claude referenced this issue2026-10-20 18:59:13 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:49:35 +02:00
claude referenced this issue2026-10-20 19:49:35 +02:00
claude referenced this issue2026-10-20 20:06:44 +02:00
claude referenced this issue2026-10-20 20:06:45 +02:00
🤖 Agent is working on this — turn 14
Last activity: tool: mcp__gitlab__forge_set_outcome — {"outcome":"denied"} — updated 18:08:58 UTC
Since my last update: the gating spike, #21, is closed with a no-go, so per this issue's "a valid outcome is 'no'" clause nothing ships. I'm not opening the
joins?schema/capture MR or the synthesis MR. I'm closing #20 as resolved negatively. The full measurement is in #21's report, and the probe code is on branchspike/21-bigram-probe, which is evidence only and not for merge. The deciding findings:learnCorrectiondeliberately refuses.FONT_TRUSTandSAMPLE_TRUSTtrusts only 5–48% of reads.FONT_TRUST = 0.08(src/ocr/words.ts:180). On real joined hands,WordReadereffectively reads only from the owner's word samples.Two leads for whoever owns #17. I haven't filed either as an issue:
store.wordscoverage, which is cheaper and is the only trusted whole-word path today.Latency was never measured, because the result failed the gate before latency mattered.
claude referenced this issue2026-10-20 20:10:24 +02:00
claude referenced this issue2026-10-20 20:11:49 +02:00
claude referenced this issue2026-10-20 20:11:49 +02:00