OCR Direction 2: persisted, growing per-writer model (store schema + scaling) #13

Closed
opened 2026-10-20 15:25:20 +02:00 by claude · 6 comments
Collaborator

Part of #5

Part of #5 (Direction 2). Research + split, then MRs. Builds on the Direction-1 MR (!12); best started after !12 lands so the letter-template shape is settled.

Context (from the #5 spike)

Today the per-writer store (src/ocr/samples.json) keeps ≤3 samples per word (PER_WORD in src/ocr/samples.ts) and, after !12, a capped set of per-letter templates. The spike showed accumulated samples clearly help (train-word CER dropped 38%→24% separable, 99%→14% joined once words were seen), and DTW/$P ran the full labelled set in well under a second — so there is headroom to store and match more.

What to figure out (then split into MRs)

  1. Store schema & caps. Should per-word / per-letter caps grow (e.g. keep the N most diverse samples, not just newest)? Evaluate a richer store: more samples, letter templates (from !12) and join templates (letter-pair ligature shapes captured from separable-enough corrections) so the whole-word DTW reader can synthesise unseen words from the owner's own joins rather than only bundled script fonts.
  2. Scaling study. Measure $P classify and DTW WordReader.read latency as the store grows to realistic sizes (hundreds of letter templates, thousands of word samples). Find where per-remark read time becomes unacceptable for sync, and add shortlisting/caps to stay under budget. Use the spike5.test.ts harness methodology to generate load.
  3. Migration & pruning. Versioned SampleStore migration; bounded growth (the spike store caps are REMARKS_KEPT/CORRECTIONS_KEPT); decide retention policy for letter/join templates.

Scope boundary

NOT a new engine or ML model (that was Direction 3 — see the #5 go/no-go, currently no-go). Pure extension of the existing $P/DTW per-writer store.

Validation contract (for each eventual MR)

npx vitest run src/__tests__ → PASS + a test showing the richer store improves a held-out read and a latency assertion keeping per-remark read time within budget. (The #5 issue names nix develop -c npx vitest …; there is no flake.nix — use npm install + npx vitest.)

Part of #5 Part of #5 (Direction 2). **Research + split**, then MRs. Builds on the Direction-1 MR (!12); best started after !12 lands so the letter-template shape is settled. ## Context (from the #5 spike) Today the per-writer store (`src/ocr/samples.json`) keeps **≤3 samples per word** (`PER_WORD` in `src/ocr/samples.ts`) and, after !12, a capped set of per-letter templates. The spike showed accumulated samples clearly help (train-word CER dropped 38%→24% separable, 99%→14% joined once words were seen), and DTW/$P ran the full labelled set in well under a second — so there is headroom to store and match **more**. ## What to figure out (then split into MRs) 1. **Store schema & caps.** Should per-word / per-letter caps grow (e.g. keep the N most *diverse* samples, not just newest)? Evaluate a richer store: more samples, **letter templates** (from !12) **and join templates** (letter-pair ligature shapes captured from separable-enough corrections) so the whole-word DTW reader can synthesise unseen words from the owner's own joins rather than only bundled script fonts. 2. **Scaling study.** Measure $P `classify` and DTW `WordReader.read` latency as the store grows to realistic sizes (hundreds of letter templates, thousands of word samples). Find where per-remark read time becomes unacceptable for sync, and add shortlisting/caps to stay under budget. Use the `spike5.test.ts` harness methodology to generate load. 3. **Migration & pruning.** Versioned `SampleStore` migration; bounded growth (the spike store caps are `REMARKS_KEPT`/`CORRECTIONS_KEPT`); decide retention policy for letter/join templates. ## Scope boundary NOT a new engine or ML model (that was Direction 3 — see the #5 go/no-go, currently no-go). Pure extension of the existing $P/DTW per-writer store. ## Validation contract (for each eventual MR) `npx vitest run src/__tests__` → PASS + a test showing the richer store improves a held-out read and a latency assertion keeping per-remark read time within budget. (The #5 issue names `nix develop -c npx vitest …`; there is no `flake.nix` — use `npm install` + `npx vitest`.)
claude self-assigned this 2026-10-20 15:25:20 +02:00
Author
Collaborator

❌ Agent run failed

session saved — new comments resume this run

❌ **Agent run failed** _session saved — new comments resume this run_
Author
Collaborator

What I found

Investigated the two read-time hot paths and the store's migration surface, with measured latency (throwaway probe, npm install + vitest, rm2 traces — the repo has no flake.nix).

  • WordReader.read is the whole-word bottleneck (src/ocr/words.ts:218): it runs a full DTW over every owner sample, gated only by a width ratio — unlike readFonts, which profile-shortlists to 150 before DTW. Linear in sample count: 5.7 ms @50 → 28.8 ms @1000 → 80 ms/word @3000. A 10-word remark at 3000 samples ≈ 0.8 s of owner-DTW per remark. → shortlist owner samples before DTW.
  • classify's quick pass is the letter bottleneck (src/ocr/recognize.ts:254): it scores shape over all templates before keeping the top 24, and segment calls it per 1–3-cluster window. recognize(1 word) is linear in template count: 92.7 ms @216 (today's fonts) → 349 ms @1216 → 615 ms @2216 (~0.24 ms/template/word). !12's learned letter templates land exactly here. → bound the quick pass (metric pre-filter and/or per-char cap).
  • No store migration exists: loadSamples (src/inkedge.ts:315) casts raw JSON straight to SampleStore; the format: 1 literal (samples.ts:99) is never checked. Adding optional fields is backward-compatible, but the letter/join-template schema change needs a real migrate() seam.
  • Caps live as consts in samples.ts: PER_WORD=3 (newest-wins slice), REMARKS_KEPT=500, CORRECTIONS_KEPT=2000. Newest-3 wastes the owner's writing-style variation.

Split rationale: the two latency fixes and the word-sample retention change are independent of !12 and ship now; the richer schema (persisted letter + join templates, WordReader synthesis of unseen words, versioned migration) depends on !12's letter-template shape — which, as of now, is not settled (!12's branch has only the spike5.test.ts harness, no learnCorrection→template persistence yet), so it's filed held.

Created

  • !15 — bound read latency as the store grows: shortlist owner samples in WordReader.read; cap classify's template work. Red-gate latency assertions + accuracy guard. (independent of !12)
  • !16 — keep the N most diverse word samples, not the newest; grow PER_WORD. Held-out read improves vs newest-3. (independent of !12; samples.ts only)
  • #17 — richer persisted store: letter + join templates, WordReader synthesis of unseen words, versioned migrate() seam + retention policy. Held (dispatch:false) — dispatch only after !12 merges and the letter-template shape is readable in samples.ts. Needs further research + splitting (suggested sub-MRs in the issue).

!15 and !16 are independent of each other (different files) and can land in parallel.

Follow-ups

  • When !12 merges, dispatch #17 (it is intentionally un-assigned so no agent conflicts with in-flight !12). is_blocked_by can't point at an MR and GitLab link-gating needs Premium, so this hand-off is a manual nudge rather than an auto-gate.
  • #17's join-template / synthesis MR is the one most likely to need a further split once the letter shape is fixed.
## What I found Investigated the two read-time hot paths and the store's migration surface, with **measured** latency (throwaway probe, `npm install` + vitest, rm2 traces — the repo has no `flake.nix`). - **`WordReader.read` is the whole-word bottleneck** (`src/ocr/words.ts:218`): it runs a full DTW over **every** owner sample, gated only by a width ratio — unlike `readFonts`, which profile-shortlists to 150 before DTW. Linear in sample count: **5.7 ms @50 → 28.8 ms @1000 → 80 ms/word @3000**. A 10-word remark at 3000 samples ≈ 0.8 s of owner-DTW per remark. → shortlist owner samples before DTW. - **`classify`'s quick pass is the letter bottleneck** (`src/ocr/recognize.ts:254`): it scores shape over **all** templates before keeping the top 24, and `segment` calls it per 1–3-cluster window. `recognize(1 word)` is linear in template count: **92.7 ms @216 (today's fonts) → 349 ms @1216 → 615 ms @2216** (~0.24 ms/template/word). !12's learned letter templates land exactly here. → bound the quick pass (metric pre-filter and/or per-char cap). - **No store migration exists**: `loadSamples` (`src/inkedge.ts:315`) casts raw JSON straight to `SampleStore`; the `format: 1` literal (`samples.ts:99`) is never checked. Adding optional fields is backward-compatible, but the letter/join-template schema change needs a real `migrate()` seam. - **Caps** live as consts in `samples.ts`: `PER_WORD=3` (newest-wins `slice`), `REMARKS_KEPT=500`, `CORRECTIONS_KEPT=2000`. Newest-3 wastes the owner's writing-style variation. **Split rationale:** the two latency fixes and the word-sample retention change are independent of !12 and ship now; the richer schema (persisted letter + **join** templates, `WordReader` synthesis of unseen words, versioned migration) depends on !12's letter-template shape — which, as of now, is **not settled** (!12's branch has only the `spike5.test.ts` harness, no `learnCorrection`→template persistence yet), so it's filed held. ## Created - **!15** — bound read latency as the store grows: shortlist owner samples in `WordReader.read`; cap `classify`'s template work. Red-gate latency assertions + accuracy guard. *(independent of !12)* - **!16** — keep the N most **diverse** word samples, not the newest; grow `PER_WORD`. Held-out read improves vs newest-3. *(independent of !12; `samples.ts` only)* - **#17** — richer persisted store: letter + join templates, `WordReader` synthesis of unseen words, versioned `migrate()` seam + retention policy. **Held (`dispatch:false`) — dispatch only after !12 merges** and the letter-template shape is readable in `samples.ts`. Needs further research + splitting (suggested sub-MRs in the issue). !15 and !16 are independent of each other (different files) and can land in parallel. ## Follow-ups - When **!12 merges**, dispatch **#17** (it is intentionally un-assigned so no agent conflicts with in-flight !12). `is_blocked_by` can't point at an MR and GitLab link-gating needs Premium, so this hand-off is a manual nudge rather than an auto-gate. - #17's join-template / synthesis MR is the one most likely to need a further split once the letter shape is fixed.
Author
Collaborator

Agent failed ❌

exceeded pending task timeout

**Agent failed** :x: exceeded pending task timeout
Author
Collaborator

🤖 Agent is working on this — turn 14

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 18:11:52 UTC

🤖 **Agent is working on this** — turn 14 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 18:11:52 UTC_
Author
Collaborator

Since my last update: every hand-off from #13 has merged. That covers !15 (read-latency shortlisting: SAMPLE_SHORTLIST=50 in words.ts, with latency asserts in scaling.test.ts), !16 (diversity-based word retention, PER_WORD=10), and via #17, !18 (letter templates kept by diversity through keepDiverse, PER_LETTER=6) and !19 (versioned migrate() seam, CURRENT_FORMAT=2, wired into loadSamples). Only the join/bigram synthesis bet is left. It runs as a chain: #17 waits on #20, which waits on the spike #21 (another agent is on it now). #21 decides whether the joins? schema MR and the synthesis MR ship at all, and "no" is a valid answer. I've linked #13 is_blocked_by #17 so this issue stays parked until that chain closes. Nothing more to do here until then.

Since my last update: every hand-off from #13 has merged. That covers !15 (read-latency shortlisting: `SAMPLE_SHORTLIST=50` in `words.ts`, with latency asserts in `scaling.test.ts`), !16 (diversity-based word retention, `PER_WORD=10`), and via #17, !18 (letter templates kept by diversity through `keepDiverse`, `PER_LETTER=6`) and !19 (versioned `migrate()` seam, `CURRENT_FORMAT=2`, wired into `loadSamples`). Only the join/bigram synthesis bet is left. It runs as a chain: #17 waits on #20, which waits on the spike #21 (another agent is on it now). #21 decides whether the `joins?` schema MR and the synthesis MR ship at all, and "no" is a valid answer. I've linked #13 **is_blocked_by #17** so this issue stays parked until that chain closes. Nothing more to do here until then.
Author
Collaborator

Since my last update: #17 has closed, which unblocks this issue, and all three parts of #13 are now resolved, so I'm closing it.

  • 1. Store schema & caps: words are kept by diversity with PER_WORD = 10 (!16), and letter templates the same way through keepDiverse with PER_LETTER = 6 (!18). Join templates are a no-go: the spike in #21 showed there is no reliable way to label bigrams in joined ink, and synthesis using the bigrams that can be collected did no better than fonts-only. #20 and #17 were closed on that result, so no joins? field was added.
  • 2. Scaling: SAMPLE_SHORTLIST = 50 keeps WordReader.read bounded, and the cost of classify is capped. The latency and accuracy checks are in src/__tests__/scaling.test.ts (!15).
  • 3. Migration & pruning: there is now a migrate(raw) step with CURRENT_FORMAT = 2, called from loadSamples (src/inkedge.ts:321). A format-1 store upgrades without loss, covered by src/__tests__/migrate.test.ts (!19).

I checked all of these on main at eb84e3a.

Note for the owner of #5 (not filed as an issue): #17 found that the WordReader font pass is effectively never trusted on real handwriting. The correct word scores 0.22–0.40 against FONT_TRUST = 0.08 (src/ocr/words.ts:180). In practice, whole-word reads of joined writing come only from the owner's own word samples. Any next OCR direction should start from that fact. The trust rule based on the gap to the runner-up word, described in #21, only pays off once there is labelled material from the owner.

Since my last update: #17 has closed, which unblocks this issue, and all three parts of #13 are now resolved, so I'm closing it. - **1. Store schema & caps:** words are kept by diversity with `PER_WORD = 10` (!16), and letter templates the same way through `keepDiverse` with `PER_LETTER = 6` (!18). **Join templates are a no-go:** the spike in #21 showed there is no reliable way to label bigrams in joined ink, and synthesis using the bigrams that can be collected did no better than fonts-only. #20 and #17 were closed on that result, so no `joins?` field was added. - **2. Scaling:** `SAMPLE_SHORTLIST = 50` keeps `WordReader.read` bounded, and the cost of `classify` is capped. The latency and accuracy checks are in `src/__tests__/scaling.test.ts` (!15). - **3. Migration & pruning:** there is now a `migrate(raw)` step with `CURRENT_FORMAT = 2`, called from `loadSamples` (`src/inkedge.ts:321`). A format-1 store upgrades without loss, covered by `src/__tests__/migrate.test.ts` (!19). I checked all of these on `main` at `eb84e3a`. **Note for the owner of #5 (not filed as an issue):** #17 found that the `WordReader` font pass is effectively never trusted on real handwriting. The correct word scores 0.22–0.40 against `FONT_TRUST = 0.08` (`src/ocr/words.ts:180`). In practice, whole-word reads of joined writing come only from the owner's own word samples. Any next OCR direction should start from that fact. The trust rule based on the gap to the runner-up word, described in #21, only pays off once there is labelled material from the owner.
claude 2026-10-20 20:11:52 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
daniel/obsidian-remarkable-round-trip#13
No description provided.