OCR: deepen continuous learning from corrections + spike an integrated trainable cursive reader #5

Closed
opened 2026-10-20 15:13:41 +02:00 by claude · 7 comments
Collaborator

Part of #2

Part of #2. This is the track that best matches the issue author's stated preference — "a local running and/or integrated js lib solution" and "continuous training on the user specified corrections would be a great opportunity" — using the assets they call out: "the writers per character mappings and some example sentences on hand." The owner explicitly asked for this (note 43818 on #2): "could we train our own handwriting word based ocr … just beefed up specialized for handwriting recognition, incorporating recent researches about continuous training/learning of models?" Higher-effort / higher-risk R&D; needs a research spike before any implementation. Keep it decoupled from the pragmatic local-VLM track (#3) and the calibration track (#4).

What already exists (build on it, don't rebuild)

  • Per-character mappings = the handwriting sheet → the owner's glyphs (src/ink/handsheet.ts SHEET_CHARS, src/ink/glyphs.ts), used as $P templates (templatesFrom in recognize.ts).
  • Example sentences / words = the word sheet (WORD_SHEET_WORDS in handsheet.ts), the vault vocabulary (Lexicon), and corrected remarks.
  • Synthetic-ink generator (the key enabler, already shipped): renderInk() + the sheet glyphs render arbitrary text → device strokes in the owner's own hand (this is exactly what src/__tests__/recognize.test.ts write() does). This removes the usual blocker for training HTR — data volume — because an unlimited labelled corpus in the owner's hand can be synthesised from the per-char sheet (ScribbleAI uses this same isolated-char→synthetic-line trick).
  • Continuous learning today (src/ocr/samples.ts learnCorrection/learnFromNotes): a correction is paired to ink word-by-word and stored as word samples; the whole-word DTW reader (src/ocr/words.ts WordReader) checks the owner's own samples first. Already writer-adaptive — but only at whole-word granularity and only for words seen before.
  • Training export scaffolding: src/ocr/trainingset.ts already emits the corrections as a dataset (ShareGPT/LLaMA-Factory layout); transport/ocr.ts worked-examples is the zero-training analogue.

Why a classifier is the wrong shape (design decision, don't relitigate)

A per-word/per-letter classifier (the "MNIST handbook" shape the owner referenced) fails twice for this task: (1) it only reads words/letters in its training set — most real writing is out-of-vocabulary; (2) cursive can't be pre-segmented into glyphs to classify, and finding the cuts is the hard part. The fix is a segmentation-free sequence model that consumes the raw pen trajectory and emits characters with no pre-cutting and no fixed vocabulary.

Directions to spike (pick/measure, then split into MRs)

  1. Letter-level learning from corrections. Extend learnCorrection to also derive per-letter templates when a corrected word's ink is separable, continuously enriching the owner's $P template set beyond the one-off handwriting sheet. Lowest-risk, incremental, measurable CER gain on the owner's own data. Independent of directions 2–3.
  2. Persisted, growing per-writer model. Today samples live in ocr/samples.json capped at 3 per word (PER_WORD). Evaluate a richer per-writer store (more samples, letter + join templates) and whether DTW/$P scale to it.
  3. Compact trainable online-HTR — the "creative" bet the owner asked for. Concrete target architecture:
    • Model: online (stroke-sequence) BiLSTM + CTC — pen trajectory (dx, dy, pen-up, …) resampled by arc length → ~3× BiLSTM → logits → CTC. No CNN (that is for image OCR); we have strokes. Canonical refs: Google multi-language IAM-OnDB system (https://link.springer.com/article/10.1007/s10032-020-00350-4); small public reference https://github.com/DarthVader69oss/scribbleai .
    • Training data: synthesise an unlimited labelled corpus in the owner's hand via renderInk() + the sheet glyphs (per-char sheet → composed words/lines; word sheet for realistic joins), then mix in the real corrections. Optionally pretrain on a public online-HTR corpus (IAM-OnDB) first.
    • Decoding: CTC beam search biased by the vault Lexicon (already our built-in word LM), so it prefers real vault spellings without being limited to them.
    • Continuous learning: NOT naive per-stroke SGD (catastrophic forgetting). Use replay (buffer of corrections + synthetic, periodic re-fine-tune) and/or a frozen base + writer-specific adapter (LoRA-style; task-free streaming variant Online-LoRA, WACV 2025 https://openaccess.thecvf.com/content/WACV2025/papers/Wei_Online-LoRA_Task-Free_Online_Continual_Learning_via_Low_Rank_Adaptation_WACV_2025_paper.pdf ; EWC, Kirkpatrick 2017). Our corrections pipeline (samples.ts → trainingset.ts) is the data tap.
    • Runtime (decide in the spike): (a) train/fine-tune offline in Python, ship weights, infer in-browser via tfjs-wasm/onnxruntime-web (lower risk; "continuous training" = occasional retrain); or (b) fine-tune in-browser via tfjs (the fully-integrated no-server dream, but slow/memory-hungry — realistically a periodic background adapter fine-tune, not live learning).
    • This is the only route to a genuinely integrated cursive reader (no server, no API). Research (see #2 analysis comment) found no off-the-shelf package that is both ONNX/tfjs-in-browser and on-device-trainable, so this is build-not-adopt. Gate it on a feasibility spike: can it beat WritePad + the local VLM baseline on the owner's corrections, within an Obsidian-acceptable bundle/runtime budget? If not, stop at directions 1–2.

Dependencies

Benefits from #4's diagnostic surface (to measure CER per reader) and from #3's local-VLM baseline (as the accuracy bar to beat). Not hard-blocked, but a fresh agent should read #3/#4 outcomes first.

Scope boundary

NOT the endpoint UX (#3) and NOT threshold calibration (#4). Deliver a spike report first (feasibility + measured CER + bundle/runtime cost + go/no-go per direction), then MRs only for the directions that clear the baseline. Direction 1 (letter-level learning) may proceed as its own MR regardless of the direction-3 outcome, since it's low-risk and independently useful.

Validation contract

Spike deliverable: a comment with measured CER (built-in vs WritePad vs local VLM vs proposed) on a labelled set of the owner's corrections + synthetic hold-out, the bundle/runtime cost of the proposed reader, and a go/no-go per direction. Implementation MRs each: nix develop -c npx vitest run src/__tests__ → PASS + a new test showing the learned-from-correction path improves a held-out read (red gate: the same read is wrong before the learned sample is added).

Part of #2 Part of #2. This is the track that best matches the issue author's stated preference — *"a local running and/or integrated js lib solution"* and *"continuous training on the user specified corrections would be a great opportunity"* — using the assets they call out: *"the writers per character mappings and some example sentences on hand."* **The owner explicitly asked for this** (note 43818 on #2): *"could we train our own handwriting word based ocr … just beefed up specialized for handwriting recognition, incorporating recent researches about continuous training/learning of models?"* Higher-effort / higher-risk R&D; needs a research spike before any implementation. Keep it decoupled from the pragmatic local-VLM track (#3) and the calibration track (#4). ## What already exists (build on it, don't rebuild) - **Per-character mappings** = the handwriting sheet → the owner's glyphs (`src/ink/handsheet.ts` `SHEET_CHARS`, `src/ink/glyphs.ts`), used as $P templates (`templatesFrom` in `recognize.ts`). - **Example sentences / words** = the word sheet (`WORD_SHEET_WORDS` in `handsheet.ts`), the vault vocabulary (`Lexicon`), and corrected remarks. - **Synthetic-ink generator (the key enabler, already shipped):** `renderInk()` + the sheet glyphs render *arbitrary text → device strokes in the owner's own hand* (this is exactly what `src/__tests__/recognize.test.ts` `write()` does). This removes the usual blocker for training HTR — data volume — because an unlimited labelled corpus in the owner's hand can be synthesised from the per-char sheet (ScribbleAI uses this same isolated-char→synthetic-line trick). - **Continuous learning today** (`src/ocr/samples.ts` `learnCorrection`/`learnFromNotes`): a correction is paired to ink **word-by-word** and stored as word samples; the whole-word DTW reader (`src/ocr/words.ts` `WordReader`) checks the owner's own samples first. Already writer-adaptive — but only at whole-word granularity and only for words seen before. - **Training export scaffolding:** `src/ocr/trainingset.ts` already emits the corrections as a dataset (ShareGPT/LLaMA-Factory layout); `transport/ocr.ts` worked-examples is the zero-training analogue. ## Why a *classifier* is the wrong shape (design decision, don't relitigate) A per-word/per-letter classifier (the "MNIST handbook" shape the owner referenced) fails twice for this task: (1) it only reads words/letters in its training set — most real writing is out-of-vocabulary; (2) cursive can't be pre-segmented into glyphs to classify, and finding the cuts is the hard part. The fix is a **segmentation-free sequence model** that consumes the raw pen trajectory and emits characters with no pre-cutting and no fixed vocabulary. ## Directions to spike (pick/measure, then split into MRs) 1. **Letter-level learning from corrections.** Extend `learnCorrection` to also derive per-letter templates when a corrected word's ink is separable, continuously enriching the owner's $P template set beyond the one-off handwriting sheet. Lowest-risk, incremental, measurable CER gain on the owner's own data. Independent of directions 2–3. 2. **Persisted, growing per-writer model.** Today samples live in `ocr/samples.json` capped at 3 per word (`PER_WORD`). Evaluate a richer per-writer store (more samples, letter + join templates) and whether DTW/$P scale to it. 3. **Compact trainable online-HTR — the "creative" bet the owner asked for.** Concrete target architecture: - **Model:** online (stroke-sequence) **BiLSTM + CTC** — pen trajectory `(dx, dy, pen-up, …)` resampled by arc length → ~3× BiLSTM → logits → CTC. No CNN (that is for image OCR); we have strokes. Canonical refs: Google multi-language IAM-OnDB system (https://link.springer.com/article/10.1007/s10032-020-00350-4); small public reference https://github.com/DarthVader69oss/scribbleai . - **Training data:** synthesise an unlimited labelled corpus in the owner's hand via `renderInk()` + the sheet glyphs (per-char sheet → composed words/lines; word sheet for realistic joins), then mix in the real corrections. Optionally pretrain on a public online-HTR corpus (IAM-OnDB) first. - **Decoding:** CTC beam search biased by the vault `Lexicon` (already our built-in word LM), so it prefers real vault spellings without being limited to them. - **Continuous learning:** NOT naive per-stroke SGD (catastrophic forgetting). Use **replay** (buffer of corrections + synthetic, periodic re-fine-tune) and/or a **frozen base + writer-specific adapter** (LoRA-style; task-free streaming variant Online-LoRA, WACV 2025 https://openaccess.thecvf.com/content/WACV2025/papers/Wei_Online-LoRA_Task-Free_Online_Continual_Learning_via_Low_Rank_Adaptation_WACV_2025_paper.pdf ; EWC, Kirkpatrick 2017). Our corrections pipeline (`samples.ts` → `trainingset.ts`) is the data tap. - **Runtime (decide in the spike):** (a) train/fine-tune **offline** in Python, ship weights, infer in-browser via `tfjs-wasm`/`onnxruntime-web` (lower risk; "continuous training" = occasional retrain); or (b) **fine-tune in-browser** via tfjs (the fully-integrated no-server dream, but slow/memory-hungry — realistically a periodic background adapter fine-tune, not live learning). - This is the only route to a genuinely *integrated* cursive reader (no server, no API). Research (see #2 analysis comment) found no off-the-shelf package that is both ONNX/tfjs-in-browser **and** on-device-trainable, so this is build-not-adopt. Gate it on a feasibility spike: can it beat WritePad + the local VLM baseline on the owner's corrections, within an Obsidian-acceptable bundle/runtime budget? If not, stop at directions 1–2. ## Dependencies Benefits from #4's diagnostic surface (to measure CER per reader) and from #3's local-VLM baseline (as the accuracy bar to beat). Not hard-blocked, but a fresh agent should read #3/#4 outcomes first. ## Scope boundary NOT the endpoint UX (#3) and NOT threshold calibration (#4). Deliver a **spike report first** (feasibility + measured CER + bundle/runtime cost + go/no-go per direction), then MRs only for the directions that clear the baseline. Direction 1 (letter-level learning) may proceed as its own MR regardless of the direction-3 outcome, since it's low-risk and independently useful. ## Validation contract Spike deliverable: a comment with measured CER (built-in vs WritePad vs local VLM vs proposed) on a labelled set of the owner's corrections + synthetic hold-out, the bundle/runtime cost of the proposed reader, and a go/no-go per direction. Implementation MRs each: `nix develop -c npx vitest run src/__tests__` → PASS + a new test showing the learned-from-correction path improves a held-out read (red gate: the same read is wrong before the learned sample is added).
claude self-assigned this 2026-10-20 15:13:41 +02:00
Author
Collaborator

✅ Agent run completed

12m38s · 54 turns · $3.80 · pushed agent/ocr-letter-learning · session saved — new comments resume this run

✅ **Agent run completed** _12m38s · 54 turns · $3.80 · pushed `agent/ocr-letter-learning` · session saved — new comments resume this run_
Author
Collaborator

Direction from @daniel (raised on #3) — a custom, word-based, continually-trained HTR model

@daniel proposed, on #3: "train our own [model] focused on word-based OCR, kinda like the handbook LLM OCR example you learn about neural networks with, just beefed up specialized for handwriting recognition, incorporating recent research about continuous training/learning." This is squarely #5's thesis, so recording the concrete blueprint and the honest risks here for whoever picks up the spike.

What "the handbook OCR example, beefed up" actually is

The canonical teaching example is CRNN + CTC: a CNN backbone → 2× BiLSTM → Connectionist Temporal Classification loss, trained on word/line images with no need for character-level segmentation. It's small (single-digit to tens of M params), and it's the right tool here precisely because it's small enough to run — and potentially adapt — locally. The battle-tested in-browser recipe is docTR's MobileNetV2-CRNN (input 32×128×3, 2 BiLSTMs, CTC), deployable via tfjs directly or onnxruntime-web (train in Keras → tf2onnx → WASM/WebGPU). Reported word accuracy with beam-search decoding is ~92% across styles. (TF.js in-browser OCR / docTR, CNN-RNN-CTC approach, SimpleHTR)

Why "word-based" is the strong part — and why this plugin is unusually well-placed

"Word-based" = CTC decoded with Word Beam Search constrained to a dictionary/LM (CTCWordBeamSearch). This plugin already owns the dictionary — the vault vocabulary and the word sheet — which is exactly where a lexicon-constrained small model beats a general VLM on known vocabulary. It also already has the full writer-specific data pipeline the training needs:

  • per-character handwriting-sheet mappings,
  • the joined-up word sheet,
  • a growing corpus of corrected remarks,
  • the existing LLaMA-Factory training export (docs/ocr-corrections.md).

That's a real per-writer dataset most HTR projects don't start with.

Continuous / continual learning — the honest framing

The realistic target is writer adaptation, not training a universal cursive reader from scratch: start from a pretrained base HTR model and adapt it to this hand from the user's corrections. Recent research gives two tractable routes:

  • Few-shot writer adaptation via multimodal in-context learning — personalizes without gradient updates at inference, which is ideal on-device. (Simon et al., 2026)
  • Continual learning for HTR with tiny replay memory — CSSL-MHTR stores only a few image patches for rehearsal, directly targeting catastrophic forgetting under privacy/on-device constraints. (CSSL-MHTR) Parameter-efficient on-device CL (training <2% of params, forgetting 40%→16%) is the transferable technique. (Gated Adaptation)

Decisions the spike must make (real risks)

  1. Universal vs per-writer. General cursive from scratch is data-hungry; the achievable win is "learns your hand." Set expectations accordingly.
  2. On-device training is the hard part. Inference in tfjs/onnxruntime-web is easy; backprop in the renderer is heavy. Three routes: (a) ship a pretrained base + lightweight in-browser fine-tune (tfjs supports training), (b) gradient-free in-context personalization (route above), (c) offline training via the existing export, then import the adapted weights.
  3. Build vs fine-tune an existing base (docTR / SimpleHTR) rather than train from zero.
  4. Relationship to WritePad, which already does lexicon-constrained HTR — replacement, complement, or fallback? (Note #3's scope bars touching WritePad; this issue does not.)

My take: this is the most fit-for-constraints of all the OCR directions (small, local, trainable, and the plugin already has per-writer data) — more promising than an embedded general VLM, which the research shows can't be both small enough to embed and good at cursive (see the #3 analysis). Worth a real spike here.

## Direction from @daniel (raised on #3) — a custom, word-based, continually-trained HTR model @daniel proposed, on #3: *"train our own [model] focused on word-based OCR, kinda like the handbook LLM OCR example you learn about neural networks with, just beefed up specialized for handwriting recognition, incorporating recent research about continuous training/learning."* This is squarely #5's thesis, so recording the concrete blueprint and the honest risks here for whoever picks up the spike. ### What "the handbook OCR example, beefed up" actually is The canonical teaching example is **CRNN + CTC**: a CNN backbone → 2× BiLSTM → Connectionist Temporal Classification loss, trained on word/line images with no need for character-level segmentation. It's small (single-digit to tens of M params), and it's the *right* tool here precisely because it's small enough to run — and potentially adapt — locally. The battle-tested in-browser recipe is docTR's MobileNetV2-CRNN (input `32×128×3`, 2 BiLSTMs, CTC), deployable via **tfjs** directly or **onnxruntime-web** (train in Keras → `tf2onnx` → WASM/WebGPU). Reported word accuracy with beam-search decoding is ~92% across styles. ([TF.js in-browser OCR / docTR](https://blog.tensorflow.org/2022/06/ocr-in-browser-using-tensorflowjs.html), [CNN-RNN-CTC approach](https://link.springer.com/chapter/10.1007/978-3-032-14044-9_4), [SimpleHTR](https://github.com/githubharald/CTCWordBeamSearch)) ### Why "word-based" is the strong part — and why this plugin is unusually well-placed "Word-based" = CTC decoded with **Word Beam Search constrained to a dictionary/LM** ([CTCWordBeamSearch](https://github.com/githubharald/CTCWordBeamSearch)). This plugin *already owns the dictionary* — the vault vocabulary and the word sheet — which is exactly where a lexicon-constrained small model beats a general VLM on *known* vocabulary. It also already has the full writer-specific data pipeline the training needs: - per-character handwriting-sheet mappings, - the joined-up **word sheet**, - a growing corpus of corrected remarks, - the existing **LLaMA-Factory training export** (`docs/ocr-corrections.md`). That's a real per-writer dataset most HTR projects don't start with. ### Continuous / continual learning — the honest framing The realistic target is **writer adaptation**, not training a universal cursive reader from scratch: start from a pretrained base HTR model and adapt it to *this* hand from the user's corrections. Recent research gives two tractable routes: - **Few-shot writer adaptation via multimodal in-context learning** — personalizes *without* gradient updates at inference, which is ideal on-device. ([Simon et al., 2026](https://arxiv.org/abs/2603.29450)) - **Continual learning for HTR with tiny replay memory** — CSSL-MHTR stores only a few image patches for rehearsal, directly targeting catastrophic forgetting under privacy/on-device constraints. ([CSSL-MHTR](https://arxiv.org/pdf/2303.09347)) Parameter-efficient on-device CL (training <2% of params, forgetting 40%→16%) is the transferable technique. ([Gated Adaptation](https://arxiv.org/html/2603.10046v2)) ### Decisions the spike must make (real risks) 1. **Universal vs per-writer.** General cursive from scratch is data-hungry; the achievable win is "learns *your* hand." Set expectations accordingly. 2. **On-device *training* is the hard part.** Inference in tfjs/onnxruntime-web is easy; backprop in the renderer is heavy. Three routes: (a) ship a pretrained base + lightweight in-browser fine-tune (tfjs supports training), (b) gradient-free in-context personalization (route above), (c) offline training via the existing export, then import the adapted weights. 3. **Build vs fine-tune an existing base** (docTR / SimpleHTR) rather than train from zero. 4. **Relationship to WritePad**, which already does lexicon-constrained HTR — replacement, complement, or fallback? (Note #3's scope bars touching WritePad; this issue does not.) My take: this is the most *fit-for-constraints* of all the OCR directions (small, local, trainable, and the plugin already has per-writer data) — more promising than an embedded general VLM, which the research shows can't be both small enough to embed and good at cursive (see the #3 analysis). Worth a real spike here.
Author
Collaborator

What I found

Spike delivered: measured CER + go/no-go per direction. I built a CER harness (src/__tests__/spike5.test.ts, on branch agent/ocr-letter-learning) using the repo's own labelled-data method — render text through a font = a known "hand" — and measured every in-repo reader plus a working prototype of Direction 1.

⚠️ Corpus caveat (honesty): the owner's real corrections live in their vault (ocr/samples.json) and are not available to an agent, so numbers below are on a synthetic labelled corpus (word sheet + remark-like phrases, deterministic seeds). Relative orderings are robust and the method matches the repo's existing tests; absolute CER is synthetic. The local-VLM baseline is not measurable here (no Ollama/network) — it's the bar #3 will establish, expected to lead on cursive.

Measured CER

Separable / print hand (owner hand differs from the generic templates):

Reader TRAIN (seen) HELD-OUT (unseen)
built-in letter reader (baseline) 38.2% 38.1%
+ word-level learning (today) 23.9% 38.1% — no gain on unseen
+ letter-level learning (Direction 1) 17.3% 20.8%
control (+same count of wrong-hand templates) — 40.7% — gain is not a count artifact
WritePad 49.5% 54.6%

Joined cursive hand (allure):

Reader TRAIN (seen) HELD-OUT (unseen)
built-in letter reader 99.4% 95.8%
+ word-level (today) 14.3% 34.2%
+ letter-level (Direction 1) 87.9% 88.9% — ~nil gain
WritePad 14.4% 21.6%
local VLM (#3) — pending #3 (expected best)

Go / no-go

  • Direction 1 — letter-level learning → GO. Cuts held-out CER 38.1% → 20.8% (≈45% rel.) and generalises to words never corrected, where word-level learning gives zero held-out gain. The control rules out a template-count artifact. Lowest risk, no new dependencies. Hard scope limit: ~nil benefit on joined cursive (can't segment letters) → must only fire when the corrected word's ink is separable, never learn letter shapes from cursive ink.
  • Direction 2 — persisted, growing per-writer store → GO (research first). Accumulated samples clearly help (train CER drops sharply); $P/DTW ran the full set in <1s, so there's headroom. Needs design: store schema + migration, caps beyond 3/word, letter+join templates, a scaling/latency study. Builds on Direction 1.
  • Direction 3 — trainable in-plugin online-HTR → NO-GO on current evidence. The cursive bar is already 21.6% CER via WritePad — bundled, free, no training, zero added dependency — and #3's local VLM is expected to beat that. A tfjs-wasm/ONNX-web model (the runtime has zero ML deps today; these add multiple MB + WASM + an in-plugin training loop) to beat both is high cost/risk for low marginal gain. The integrated "trains-on-corrections, no server" win the author wanted is delivered more cheaply by Directions 1–2. Parked with an explicit revisit gate (#14).

(Env note: the validation contract names nix develop -c npx vitest …, but there is no flake.nix in the repo — the working command is npm install then npx vitest run src/__tests__.)

Created

  • !12 — MR: Direction 1 letter-level learning (fully specified; measured GO; red-gate held-out test). Part of #5.
  • #13 — child issue: Direction 2 persisted/growing per-writer store (research → MRs; after !12).
  • #14 — child issue: Direction 3 trainable online-HTR — parked/no-go, held, with a revisit gate.

Follow-ups

  • Depends on #3/#4 (both still in progress): #4's per-reader CER surface lets Direction 1 be measured on the owner's real remarks (the number that actually matters); #3's local-VLM number is the accuracy bar Direction 3's gate checks against. Re-measure on real data once those land.
  • No product code changed on main (spike + hand-offs only); the only commit is the harness on the MR branch.
## What I found **Spike delivered: measured CER + go/no-go per direction.** I built a CER harness (`src/__tests__/spike5.test.ts`, on branch `agent/ocr-letter-learning`) using the repo's own labelled-data method — render text through a font = a known "hand" — and measured every in-repo reader plus a working prototype of Direction 1. **⚠️ Corpus caveat (honesty):** the owner's *real* corrections live in their vault (`ocr/samples.json`) and are **not available to an agent**, so numbers below are on a **synthetic** labelled corpus (word sheet + remark-like phrases, deterministic seeds). Relative orderings are robust and the method matches the repo's existing tests; absolute CER is synthetic. The **local-VLM baseline is not measurable here** (no Ollama/network) — it's the bar #3 will establish, expected to lead on cursive. ### Measured CER **Separable / print hand (owner hand differs from the generic templates):** | Reader | TRAIN (seen) | HELD-OUT (unseen) | |---|---|---| | built-in letter reader (baseline) | 38.2% | 38.1% | | + word-level learning (today) | 23.9% | **38.1%** — no gain on unseen | | **+ letter-level learning (Direction 1)** | 17.3% | **20.8%** | | control (+same count of wrong-hand templates) | — | 40.7% — gain is *not* a count artifact | | WritePad | 49.5% | 54.6% | **Joined cursive hand (allure):** | Reader | TRAIN (seen) | HELD-OUT (unseen) | |---|---|---| | built-in letter reader | 99.4% | 95.8% | | + word-level (today) | 14.3% | 34.2% | | + letter-level (Direction 1) | 87.9% | 88.9% — ~nil gain | | **WritePad** | 14.4% | **21.6%** | | local VLM (#3) | — | *pending #3 (expected best)* | ### Go / no-go - **Direction 1 — letter-level learning → GO.** Cuts held-out CER **38.1% → 20.8% (≈45% rel.)** and **generalises to words never corrected**, where word-level learning gives *zero* held-out gain. The control rules out a template-count artifact. Lowest risk, **no new dependencies**. **Hard scope limit:** ~nil benefit on joined cursive (can't segment letters) → must only fire when the corrected word's ink is *separable*, never learn letter shapes from cursive ink. - **Direction 2 — persisted, growing per-writer store → GO (research first).** Accumulated samples clearly help (train CER drops sharply); $P/DTW ran the full set in <1s, so there's headroom. Needs design: store schema + migration, caps beyond 3/word, letter+join templates, a scaling/latency study. Builds on Direction 1. - **Direction 3 — trainable in-plugin online-HTR → NO-GO on current evidence.** The cursive bar is **already 21.6% CER via WritePad — bundled, free, no training, zero added dependency** — and #3's local VLM is expected to beat that. A tfjs-wasm/ONNX-web model (the runtime has *zero* ML deps today; these add multiple MB + WASM + an in-plugin training loop) to beat *both* is high cost/risk for low marginal gain. The integrated "trains-on-corrections, no server" win the author wanted is delivered more cheaply by Directions 1–2. **Parked with an explicit revisit gate (#14).** (Env note: the validation contract names `nix develop -c npx vitest …`, but there is **no `flake.nix`** in the repo — the working command is `npm install` then `npx vitest run src/__tests__`.) ## Created - **!12** — MR: Direction 1 letter-level learning (fully specified; measured GO; red-gate held-out test). `Part of #5`. - **#13** — child issue: Direction 2 persisted/growing per-writer store (research → MRs; after !12). - **#14** — child issue: Direction 3 trainable online-HTR — **parked/no-go**, held, with a revisit gate. ## Follow-ups - **Depends on #3/#4** (both still in progress): #4's per-reader CER surface lets Direction 1 be measured on the owner's *real* remarks (the number that actually matters); #3's local-VLM number is the accuracy bar Direction 3's gate checks against. Re-measure on real data once those land. - No product code changed on `main` (spike + hand-offs only); the only commit is the harness on the MR branch.
Author
Collaborator

🤖 Agent is working on this — turn 20

Last activity: tool: ToolSearch — {"query":"select:mcp__gitlab__forge_close_issue","max_results":1} — updated 22:36:09 UTC

🤖 **Agent is working on this** — turn 20 _Last activity: tool: ToolSearch — {"query":"select:mcp__gitlab__forge_close_issue","max_results":1} — updated 22:36:09 UTC_
Author
Collaborator

Since my last update: Direction 1 (!12) and the Direction 2 store work have all merged: !15 latency caps, !16 diverse word retention, !18 diverse letter retention and !19 versioned migrate() seam. The last open Direction 2 item is the join/ligature synthesis bet: #20, which is gated by #21, the spike that decides it. #21's last run failed for an infrastructure reason (model unavailable). I've told it on #21 that its prerequisites !18 and !19 have merged, which should resume it. A clean "no" from #21 is a valid outcome. Direction 3 (trainable online-HTR) has already been spiked and is parked as no-go in #14. The #2 roll-up (note 44037) calls it "not started", but that is out of date: the open question is the revisit gate in #14, not a missing spike. This issue stays open until #20 resolves.

Since my last update: **Direction 1** (!12) and the **Direction 2** store work have all merged: !15 latency caps, !16 diverse word retention, !18 diverse letter retention and !19 versioned `migrate()` seam. The last open Direction 2 item is the join/ligature synthesis bet: **#20**, which is gated by **#21**, the spike that decides it. #21's last run failed for an infrastructure reason (model unavailable). I've told it on #21 that its prerequisites !18 and !19 have merged, which should resume it. A clean "no" from #21 is a valid outcome. **Direction 3** (trainable online-HTR) has already been spiked and is **parked as no-go** in #14. The #2 roll-up (note 44037) calls it "not started", but that is out of date: the open question is the revisit gate in #14, not a missing spike. This issue stays open until #20 resolves.
Owner

@claude whats the status here?

@claude whats the status here?
Author
Collaborator

Since my last update: this issue is done. All three directions have a final answer.

  • Direction 1 (learning letter shapes from corrections): shipped in !12. On held-out words with a separable hand, CER dropped from 38% to 21%.
  • Direction 2 (a larger, growing per-writer store): shipped what earned its place.
    • !15 keeps read time flat as the store grows.
    • !16 and !18 keep the most varied word and letter samples instead of just the newest.
    • !19 adds a versioned migration step for the stored samples.
  • The last Direction 2 bet, building unseen words from the owner's letter pairs (#20, decided by the #21 spike): no-go, nothing ships.
    • Letter pairs can't be reliably labelled from joined ink. On semi-joined ink they were labelled correctly only 82–96% of the time.
    • Even with perfectly labelled pairs, the right word scores too close to the wrong ones to be trusted.
    • The full measurements are in #21's report. #13, #17, #20 and #21 are all closed.
  • Direction 3 (training our own handwriting model): parked as no-go in #14. It stays open and assigned to you as a tracked decision, with a gate for when to revisit it.

!33 also shows up as linked here, but it is the Ollama installer from the local-reader track (#3), not part of this issue.

Two leads from #21 that I haven't filed as issues:

  1. Grow the set of confirmed owner words. They are the only whole-word reads we trust on real joined handwriting.
  2. Trust a read when it clearly beats the runner-up, rather than relying on a fixed score cutoff.

Say if you want either one opened as an issue. I'm closing #5.

Since my last update: **this issue is done.** All three directions have a final answer. - **Direction 1 (learning letter shapes from corrections): shipped** in !12. On held-out words with a separable hand, CER dropped from 38% to 21%. - **Direction 2 (a larger, growing per-writer store): shipped what earned its place.** - !15 keeps read time flat as the store grows. - !16 and !18 keep the most varied word and letter samples instead of just the newest. - !19 adds a versioned migration step for the stored samples. - **The last Direction 2 bet, building unseen words from the owner's letter pairs (#20, decided by the #21 spike): no-go, nothing ships.** - Letter pairs can't be reliably labelled from joined ink. On semi-joined ink they were labelled correctly only 82–96% of the time. - Even with perfectly labelled pairs, the right word scores too close to the wrong ones to be trusted. - The full measurements are in #21's report. #13, #17, #20 and #21 are all closed. - **Direction 3 (training our own handwriting model): parked as no-go** in #14. It stays open and assigned to you as a tracked decision, with a gate for when to revisit it. !33 also shows up as linked here, but it is the Ollama installer from the local-reader track (#3), not part of this issue. Two leads from #21 that I haven't filed as issues: 1. Grow the set of confirmed owner words. They are the only whole-word reads we trust on real joined handwriting. 2. Trust a read when it clearly beats the runner-up, rather than relying on a fixed score cutoff. Say if you want either one opened as an issue. I'm closing #5.
claude 2026-10-21 00:36:12 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip#5
No description provided.