OCR Direction 2: join/ligature templates + WordReader synthesis of unseen words (the creative bet) #20

Closed
opened 2026-10-20 17:14:08 +02:00 by claude · 4 comments
Collaborator

Part of #17

Part of #17 (OCR Direction 2 — richer persisted store), Part of #13. This is issue #17 part 2, the research-heavy creative bet — carved out as its own issue because it has open design decisions and an open empirical outcome, and will likely split again into its own MRs. (The other two parts of #17 are already in flight as MRs: !18 letter-template diversity retention, !19 versioned migrate() seam.)

Goal

Let WordReader (src/ocr/words.ts:191) read a word the owner has never written, in the owner's own hand — by synthesising that word's expected trace from the owner's own letter templates (store.letters, landed in !12) plus newly-captured join / ligature templates, instead of only from the bundled script fonts.

Where we are today

  • WordReader.read (words.ts:226) scores the ink's trace (resampled, height-normalised main-line polyline + up/down profile) by DTW against: (a) the owner's whole-word samples (this.samples), and (b) candidate words rendered in GlyphSource fonts via written() (words.ts:150) + readFonts() (words.ts:256).
  • The owner's letter templates (store.letters) exist and feed the letter reader (lettersTemplates → recognize), but WordReader does not use them. There is no join capture and no owner-hand word synthesis.
  • A word the owner never wrote is therefore only ever matched against fonts — a different hand — so it reads poorly or falls back to the image.

Open research questions / design (resolve these, then split into MRs)

  1. Harvesting joins from ink that is, by definition, joined. cutLetters (recognize.ts:588) returns null for exactly the joined-cursive case where ligatures matter (the letter-separability gate). The issue's lead is splitLineInto (recognize.ts:571) — cuts at widest gaps with no letter-min-gap. Design how to capture letter-pair (bigram) join shapes from corrections whose ink is separable enough to cut into pairs even when not into single letters, and define what the stored "join" actually is (the connecting ink between two letter clusters? the whole bigram shape?). Confirm empirically that enough joins can be harvested from realistic corrections to matter.
  2. Trace synthesis. WordReader compares traces, not $P clouds. store.letters holds strokes in cap units (baseline y=0) from which a per-letter trace is derivable (traceOf). To build an unseen word's candidate trace, stitch letter traces + join traces end-to-start in reading order, placed by advance/width, then run through traceOf. Decide the fallback for a letter→letter transition with no captured join (straight connector? the font's join? skip the pair?), and how to choose among multiple samples per letter/join.
  3. Wiring + trust. Add synthesised owner-hand candidates as a source alongside readFonts, scored by the same dtw, with its own trust threshold between SAMPLE_TRUST = 0.2 (a word the owner verifiably wrote) and FONT_TRUST = 0.08 (a foreign font) — a synthesised owner word is neither.
  4. Schema + retention (part 3/4). Add optional joins?: Record<string, Point[][][]> to SampleStore (keyed by letter-pair, e.g. "th"), captured in a learnJoins alongside learnCorrection. Give it a diversity-preserving per-pair cap — reuse the generalised keepDiverse(samples, cap) from !18 — and document a PER_JOIN const like PER_WORD/PER_LETTER. Route the new field through the migrate() seam from !19.
  5. Measurement — this is the whole point. Use src/__tests__/letterlearning.test.ts's font-as-hand methodology (hand() renders text in a font = a "hand"; wordInk(), cer(), the fixed TRAIN/HELD_OUT split). Feed TRAIN words as corrections, then measure mean CER on HELD_OUT words the owner never wrote, read via WordReader with owner-hand synthesis vs. fonts-only. Target: synthesis beats fonts-only on held-out words (the #5 spike showed joined-word CER 99%→14% once words were seen; the bet here is generalisation to unseen words). Red gate: fonts-only on both sides is an equality that fails before the change.

Scope boundary

  • NOT a new engine / ML model (Direction 3, no-go per #5). Pure extension of the existing $P/DTW per-writer store.
  • Read latency must stay within !15's budget. Synthesis is per-candidate-word like readFonts, so it must go through the same profile+width shortlist (SAMPLE_SHORTLIST / the 150-candidate cap) and the this.traces cache — do not add an unbounded per-read cost. (mr-15 codebase note: read cost must stay FLAT in store size.) A latency assertion keeping per-remark read time within budget is part of the validation contract.
  • Letter-template retention and the migrate() seam are already covered by !18 / !19 — do not re-do them here; build on them.

A valid outcome is "no"

If, with a faithful implementation, synthesis does not beat fonts-only on held-out words, that is a legitimate negative result — report it on this issue and do not ship a feature that doesn't earn its complexity. The measurement decides whether the join MR lands.

Suggested MR split (for the agent that picks this up)

  • MR: joins? schema + learnJoins capture (from pair-separable corrections) + PER_JOIN diversity retention, through the !19 migrate seam. Unit test: joins are captured from separable ink and bounded by the cap.
  • MR: WordReader owner-hand synthesis (letters + joins) + trust threshold + the held-out CER test and the latency assertion. (May split further if synthesis design proves large.)

Ordering

Soft-depends on !18 (reuse keepDiverse(samples, cap) for PER_JOIN) and !19 (the migrate() seam for joins?). Neither is a hard blocker — the joins? field is optional and backward-compatible — but land this work's MRs after !18/!19 merge to avoid conflicts in src/ocr/samples.ts.

Validation contract (each eventual MR)

npm install then npx vitest run src/__tests__ → PASS, plus (for the synthesis MR) a test showing the richer store improves a held-out read (an unseen word synthesised from owner letters+joins) over fonts-only, and a latency assertion keeping per-remark read time within !15's budget. (No flake.nix.)

Part of #17 Part of #17 (OCR Direction 2 — richer persisted store), Part of #13. This is issue #17 **part 2**, the research-heavy creative bet — carved out as its own issue because it has open design decisions **and an open empirical outcome**, and will likely split again into its own MRs. (The other two parts of #17 are already in flight as MRs: !18 letter-template diversity retention, !19 versioned `migrate()` seam.) ## Goal Let `WordReader` (`src/ocr/words.ts:191`) **read a word the owner has never written, in the owner's own hand** — by synthesising that word's expected trace from the owner's own **letter** templates (`store.letters`, landed in !12) plus newly-captured **join / ligature** templates, instead of only from the bundled script fonts. ## Where we are today - `WordReader.read` (`words.ts:226`) scores the ink's **trace** (resampled, height-normalised main-line polyline + up/down profile) by DTW against: (a) the owner's **whole-word** samples (`this.samples`), and (b) candidate words rendered in `GlyphSource` **fonts** via `written()` (`words.ts:150`) + `readFonts()` (`words.ts:256`). - The owner's **letter** templates (`store.letters`) exist and feed the *letter* reader (`lettersTemplates` → `recognize`), but `WordReader` does **not** use them. There is no join capture and no owner-hand word synthesis. - A word the owner never wrote is therefore only ever matched against *fonts* — a different hand — so it reads poorly or falls back to the image. ## Open research questions / design (resolve these, then split into MRs) 1. **Harvesting joins from ink that is, by definition, joined.** `cutLetters` (`recognize.ts:588`) returns `null` for exactly the joined-cursive case where ligatures matter (the letter-separability gate). The issue's lead is `splitLineInto` (`recognize.ts:571`) — cuts at widest gaps with **no** letter-min-gap. Design how to capture letter-**pair** (bigram) join shapes from corrections whose ink is *separable enough to cut into pairs* even when not into single letters, and define what the stored "join" actually is (the connecting ink between two letter clusters? the whole bigram shape?). Confirm empirically that enough joins can be harvested from realistic corrections to matter. 2. **Trace synthesis.** `WordReader` compares *traces*, not $P clouds. `store.letters` holds strokes in cap units (baseline y=0) from which a per-letter trace is derivable (`traceOf`). To build an unseen word's candidate trace, stitch letter traces + join traces end-to-start in reading order, placed by advance/width, then run through `traceOf`. Decide the **fallback for a letter→letter transition with no captured join** (straight connector? the font's join? skip the pair?), and how to choose among multiple samples per letter/join. 3. **Wiring + trust.** Add synthesised owner-hand candidates as a source alongside `readFonts`, scored by the same `dtw`, with its **own trust threshold** between `SAMPLE_TRUST = 0.2` (a word the owner verifiably wrote) and `FONT_TRUST = 0.08` (a foreign font) — a synthesised owner word is neither. 4. **Schema + retention (part 3/4).** Add optional `joins?: Record<string, Point[][][]>` to `SampleStore` (keyed by letter-pair, e.g. `"th"`), captured in a `learnJoins` alongside `learnCorrection`. Give it a **diversity-preserving per-pair cap** — reuse the generalised `keepDiverse(samples, cap)` from !18 — and document a `PER_JOIN` const like `PER_WORD`/`PER_LETTER`. Route the new field through the `migrate()` seam from !19. 5. **Measurement — this is the whole point.** Use `src/__tests__/letterlearning.test.ts`'s font-as-hand methodology (`hand()` renders text in a font = a "hand"; `wordInk()`, `cer()`, the fixed `TRAIN`/`HELD_OUT` split). Feed `TRAIN` words as corrections, then measure mean CER on `HELD_OUT` words the owner **never wrote**, read via `WordReader` **with owner-hand synthesis vs. fonts-only**. Target: synthesis beats fonts-only on held-out words (the #5 spike showed joined-word CER 99%→14% once words were *seen*; the bet here is generalisation to *unseen* words). **Red gate:** fonts-only on both sides is an equality that fails before the change. ## Scope boundary - **NOT** a new engine / ML model (Direction 3, no-go per #5). Pure extension of the existing $P/DTW per-writer store. - Read latency must stay within !15's budget. Synthesis is per-candidate-word like `readFonts`, so it **must** go through the same profile+width shortlist (`SAMPLE_SHORTLIST` / the 150-candidate cap) and the `this.traces` cache — do not add an unbounded per-read cost. (mr-15 codebase note: read cost must stay FLAT in store size.) A latency assertion keeping per-remark read time within budget is part of the validation contract. - Letter-template retention and the `migrate()` seam are **already covered** by !18 / !19 — do not re-do them here; build on them. ## A valid outcome is "no" If, with a faithful implementation, synthesis does **not** beat fonts-only on held-out words, that is a legitimate negative result — **report it on this issue and do not ship** a feature that doesn't earn its complexity. The measurement decides whether the join MR lands. ## Suggested MR split (for the agent that picks this up) - MR: `joins?` schema + `learnJoins` capture (from pair-separable corrections) + `PER_JOIN` diversity retention, through the !19 migrate seam. Unit test: joins are captured from separable ink and bounded by the cap. - MR: `WordReader` owner-hand synthesis (letters + joins) + trust threshold + the held-out CER test and the latency assertion. (May split further if synthesis design proves large.) ## Ordering Soft-depends on !18 (reuse `keepDiverse(samples, cap)` for `PER_JOIN`) and !19 (the `migrate()` seam for `joins?`). Neither is a hard blocker — the `joins?` field is optional and backward-compatible — but land this work's MRs **after** !18/!19 merge to avoid conflicts in `src/ocr/samples.ts`. ## Validation contract (each eventual MR) `npm install` then `npx vitest run src/__tests__` → PASS, plus (for the synthesis MR) a test showing the richer store improves a **held-out** read (an unseen word synthesised from owner letters+joins) over fonts-only, and a latency assertion keeping per-remark read time within !15's budget. (No `flake.nix`.)
claude self-assigned this 2026-10-20 17:14:08 +02:00
Author
Collaborator

✅ Agent run completed

7m21s · 28 turns · $2.12 · session saved — new comments resume this run

✅ **Agent run completed** _7m21s · 28 turns · $2.12 · session saved — new comments resume this run_
Author
Collaborator

What I found

  • The bet's premise is falsified by the code's harvesting path, so the #20 measurement can't be staged fairly as written. store.letters (the synthesis material) is populated only via learnLetters→cutLetters, which returns null for joined cursive by design — so letter templates exist for separable hands and are absent for the joined hands where WordReader is the operative reader.
  • Probe over the letterlearning.test.ts font-as-hand methodology (TRAIN as corrections, held-out via WordReader): separable hands capture 16–21 letter shapes but fonts-only held-out CER = 1.000 / 0 trusted (a separable trace can't match the joined bundled script fonts — and separable hands are already generalised by the letter reader, !12). Joined hands (allure/hershey-script) capture 0 letters (synthesis can't run) and fonts-only already reads ~0.35–0.45 because the test "hand" is itself a bundled script font.
  • Joins don't rescue it with current tools: splitLineInto cuts even allure into ⌈n/2⌉ clusters, but at widest gaps with no guarantee a cluster maps to a labelled adjacent letter-pair — assigning cluster→bigram from joined ink is the exact "guess which ink is which → teach wrong shapes" hazard learnCorrection already refuses. The 4 bundled fonts have no semi-joined member, the only regime where WordReader is operative and material is capturable.

This is a strong structural obstacle, not yet a final "no" (no faithful synthesis implementation was run). It must be resolved before the schema + synthesis MRs are worth building — so I did not hand off the issue's 2-MR split as-is (both MRs rest on the falsified premise). Instead I scoped the decisive spike with a corrected design (bigram-tiling synthesis, honest labelled-bigram harvesting, and a fair fonts-only baseline measured on a joined/semi-joined hand rather than the CER-1.0 straw man).

Created

  • #21 — decisive spike (blocks #20): bigram-tiling owner-hand synthesis + a fair held-out measurement; gates whether the joins? schema MR and the synthesis MR ever land. Internal 2-MR split + validation contract + the !18/!19 soft-dependency are specified there.

Follow-ups

  • Land #21's eventual MRs after !18 (keepDiverse(samples, cap)) and !19 (migrate() seam) merge — both currently open — to avoid src/ocr/samples.ts conflicts.
  • If the spike comes back "no" on the operative (joined-hand) baseline, report it on #20 and ship nothing — the synthesis does not earn its complexity.
## What I found - **The bet's premise is falsified by the code's harvesting path, so the #20 measurement can't be staged fairly as written.** `store.letters` (the synthesis material) is populated only via `learnLetters`→`cutLetters`, which returns `null` for joined cursive **by design** — so letter templates exist for *separable* hands and are **absent for the joined hands where `WordReader` is the operative reader**. - Probe over the `letterlearning.test.ts` font-as-hand methodology (TRAIN as corrections, held-out via `WordReader`): separable hands capture 16–21 letter shapes but fonts-only held-out CER = **1.000 / 0 trusted** (a separable trace can't match the *joined* bundled script fonts — and separable hands are already generalised by the letter reader, !12). Joined hands (allure/hershey-script) capture **0** letters (synthesis can't run) and fonts-only already reads ~**0.35–0.45** *because the test "hand" is itself a bundled script font*. - **Joins don't rescue it with current tools:** `splitLineInto` cuts even allure into ⌈n/2⌉ clusters, but at widest gaps with no guarantee a cluster maps to a *labelled* adjacent letter-pair — assigning cluster→bigram from joined ink is the exact "guess which ink is which → teach wrong shapes" hazard `learnCorrection` already refuses. The 4 bundled fonts have no *semi-joined* member, the only regime where `WordReader` is operative **and** material is capturable. This is a strong structural obstacle, not yet a final "no" (no faithful synthesis implementation was run). It must be resolved before the schema + synthesis MRs are worth building — so I did **not** hand off the issue's 2-MR split as-is (both MRs rest on the falsified premise). Instead I scoped the decisive spike with a corrected design (bigram-tiling synthesis, honest labelled-bigram harvesting, and a **fair** fonts-only baseline measured on a joined/semi-joined hand rather than the CER-1.0 straw man). ## Created - #21 — decisive spike (blocks #20): bigram-tiling owner-hand synthesis + a fair held-out measurement; gates whether the `joins?` schema MR and the synthesis MR ever land. Internal 2-MR split + validation contract + the !18/!19 soft-dependency are specified there. ## Follow-ups - Land #21's eventual MRs **after** !18 (`keepDiverse(samples, cap)`) and !19 (`migrate()` seam) merge — both currently open — to avoid `src/ocr/samples.ts` conflicts. - If the spike comes back "no" on the operative (joined-hand) baseline, report it on #20 and ship nothing — the synthesis does not earn its complexity.
Author
Collaborator

🤖 Agent is working on this — turn 14

Last activity: tool: mcp__gitlab__forge_set_outcome — {"outcome":"denied"} — updated 18:08:58 UTC

🤖 **Agent is working on this** — turn 14 _Last activity: tool: mcp__gitlab__forge_set_outcome — {"outcome":"denied"} — updated 18:08:58 UTC_
Author
Collaborator

Since my last update: the gating spike, #21, is closed with a no-go, so per this issue's "a valid outcome is 'no'" clause nothing ships. I'm not opening the joins? schema/capture MR or the synthesis MR. I'm closing #20 as resolved negatively. The full measurement is in #21's report, and the probe code is on branch spike/21-bigram-probe, which is evidence only and not for merge. The deciding findings:

  1. There's no provable way to label bigrams.
    • Fully joined ink has no boundary signal at all.
    • On semi-joined ink, assigning gap-cut clusters by expected widths labels bigrams only 82–96% correctly. That is the same kind of "guess which ink is which" that learnCorrection deliberately refuses.
    • With harvested material, synthesis comes out the same as fonts-only on every owner. Effective CER on held-out words is 1.000 → 0.97–1.000.
  2. Even with perfect (oracle) bigrams, the trust threshold is the bottleneck, not ranking. Synthesis ranks the right word first almost every time (top-1 CER about 0.00–0.07). But the correct word's cost is about 0.21, which overlaps the best wrong candidate (0.18–0.30). So a threshold between FONT_TRUST and SAMPLE_TRUST trusts only 5–48% of reads.
  3. Side finding for #17: on any owner hand that isn't literally a bundled font, the font pass is never trusted. The correct word costs 0.22–0.40 against FONT_TRUST = 0.08 (src/ocr/words.ts:180). On real joined hands, WordReader effectively reads only from the owner's word samples.

Two leads for whoever owns #17. I haven't filed either as an issue:

  • A margin-based trust rule: trust a read when the best cost is clearly below the runner-up's. This only pays off once labelled material exists.
  • Growing store.words coverage, which is cheaper and is the only trusted whole-word path today.

Latency was never measured, because the result failed the gate before latency mattered.

Since my last update: the gating spike, #21, is closed with a **no-go**, so per this issue's "a valid outcome is 'no'" clause **nothing ships**. I'm not opening the `joins?` schema/capture MR or the synthesis MR. I'm closing #20 as resolved negatively. The full measurement is in [#21's report](https://forge.n0de.biz/daniel/obsidian-remarkable-round-trip/issues/21), and the probe code is on branch `spike/21-bigram-probe`, which is evidence only and not for merge. The deciding findings: 1. **There's no provable way to label bigrams.** - Fully joined ink has no boundary signal at all. - On semi-joined ink, assigning gap-cut clusters by expected widths labels bigrams only **82–96%** correctly. That is the same kind of "guess which ink is which" that `learnCorrection` deliberately refuses. - With harvested material, synthesis comes out the same as fonts-only on every owner. Effective CER on held-out words is 1.000 → 0.97–1.000. 2. **Even with perfect (oracle) bigrams, the trust threshold is the bottleneck, not ranking.** Synthesis ranks the right word first almost every time (top-1 CER about 0.00–0.07). But the correct word's cost is about 0.21, which overlaps the best wrong candidate (0.18–0.30). So a threshold between `FONT_TRUST` and `SAMPLE_TRUST` trusts only 5–48% of reads. 3. **Side finding for #17:** on any owner hand that isn't literally a bundled font, the font pass is never trusted. The correct word costs 0.22–0.40 against `FONT_TRUST = 0.08` (`src/ocr/words.ts:180`). On real joined hands, `WordReader` effectively reads only from the owner's word samples. Two leads for whoever owns #17. I haven't filed either as an issue: - **A margin-based trust rule:** trust a read when the best cost is clearly below the runner-up's. This only pays off once labelled material exists. - **Growing `store.words` coverage**, which is cheaper and is the only trusted whole-word path today. Latency was never measured, because the result failed the gate before latency mattered.
claude 2026-10-20 20:08:58 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
daniel/obsidian-remarkable-round-trip#20
No description provided.