improve / simplify stroke to text transcription #2

Closed
opened 2026-10-20 15:06:49 +02:00 by daniel · 11 comments
Owner

the current builtin solution is not very relieable and mostly detects junk, local ollama hasnt been tested yet because of setup difficulties.

reasearch new / creative ways to solve the problem of detecting joined up curly writing (mind we have the writers per character mappings and some example sentences on hand)

a local running and/or integrated js lib solution is preferred over 3rd party toolings and api calls.

if possible continous training on the user specified corrections would be a great opportunity to take.

the current builtin solution is not very relieable and mostly detects junk, local ollama hasnt been tested yet because of setup difficulties. reasearch new / creative ways to solve the problem of detecting joined up curly writing (mind we have the writers per character mappings and some example sentences on hand) a local running and/or integrated js lib solution is preferred over 3rd party toolings and api calls. if possible continous training on the user specified corrections would be a great opportunity to take.
Collaborator

❌ Agent run failed

session saved — new comments resume this run

❌ **Agent run failed** _session saved — new comments resume this run_
Collaborator

What I found

This is a research issue, so here is the full investigation + landscape; the actionable work is split into #3/#4/#5 below.

Root cause of "mostly detects junk." There is no new bug to fix in one place — there is a structural gap. The default reader (ocrMode: "builtin", src/main.ts recognizeLocally) runs a pipeline that has no component capable of arbitrary joined-up cursive, and wrong readings leak past the trust thresholds instead of falling back to the ink image (which is the correct upstream behaviour — show the image unless sure).

flowchart TD
  S[pen strokes] --> B["$P per-letter reader (recognize.ts)<br/>segments by horizontal GAPS"]
  B -->|confidence ≥ 0.65 OCR_TRUST| T[use text]
  B -->|unsure| W["WritePad WASM (writepad.ts)<br/>GPL cursive engine, SILENT fallback"]
  W -->|confidence ≥ 0.70 WRITEPAD_TRUST| T
  W -->|unsure| I[image only]
  B -. junk leaks ≥0.65 .-> T
  W -. junk leaks ≥0.70 .-> T

Each local mechanism has a narrow competence:

  • $P letter reader (recognize.ts) — explicitly "built for separate print letters; joined-up writing comes back with low confidence." Cursive has no inter-letter gaps, so its clusterStrokes/segment step mis-segments → garbage candidates.
  • Whole-word DTW reader (words.ts) — only reads words already in the vault vocabulary, via the owner's samples or script-font traces. Writer-adaptive, but narrow.
  • WritePad WASM (writepad.ts) — genuinely cursive-capable (GPL-3.0, embedded in engine.js; license is clean since the plugin is GPL-3.0), but it is a hidden fallback nobody can see firing, uses fixed English/German dictionaries, and can only be taught vocabulary — not the owner's hand.
  • Vision endpoint (transport/ocr.ts) — opt-in Ollama/LM-Studio/hosted VLM; you noted it is untested due to setup difficulties.

Research landscape (2026) — what can actually read cursive locally. I checked the current options against your "local / integrated JS lib over 3rd-party & API" preference, so no one re-treads the dead ends:

  • ❌ Native navigator.createHandwritingRecognizer (Handwriting Recognition API) — stroke-based, local, and maps exactly onto the data you already have, but it is ChromeOS-only; it rejects in Obsidian's Electron on Linux/macOS/Windows. Not viable. (Chrome docs)
  • ❌ MyScript iinkJS / iink-ts — best-in-class cursive, but cloud/commercial subscription. Violates local-first. (repo)
  • ❌ No off-the-shelf integrated JS lib does robust Latin cursive on-device and trains on corrections. Transformer online-HTR is research-only, not a packaged ONNX/tfjs model. (arXiv 2506.20255, T-OHS) — so an integrated reader is build-not-adopt.
  • ✅ Local vision model via Ollama/LM Studio — in 2026 a local VLM (qwen2.5-VL, MiniCPM-V) is the only thing that reads arbitrary cursive well while keeping ink on your machine. It is "local running" but still an HTTP call to a local server — the honest trade-off against your "integrated JS lib" ideal.

The assets you mentioned are already in the codebase and are exactly what a writer-adapted approach exploits: per-character mappings = the handwriting sheet glyphs (ink/handsheet.ts, used as $P templates); example words/sentences = the word sheet + vault Lexicon + corrected remarks. Continuous learning already exists at word granularity (ocr/samples.ts learnCorrection) — the opportunity is to push it to letter level and/or a trainable model.

Recommendation: two parallel tracks — the pragmatic high-accuracy one (local VLM, #3) and the one matching your stated preference (integrated trainable reader, #5) — plus an immediate reliability fix (#4) so the current reader stops emitting junk regardless of which track wins.

Created

  • #3 — make the local vision-model path (Ollama/LM Studio) reliable & self-serve (pragmatic, highest accuracy now; independent)
  • #4 — stop the built-in reader emitting junk: calibrate confidence + make readings diagnosable (immediate reliability; independent)
  • #5 — deepen continuous learning from corrections + spike an integrated trainable cursive reader (matches your local-JS-lib preference; R&D, read #3/#4 outcomes first)

No MRs opened directly: every track still needs a design/scope decision, so each went to a planning agent rather than a half-specified implementation branch.

Follow-ups

  • Suggested order: #3 (quick self-serve win) and #4 (fail-safe) first, in parallel; #5 as the longer-term R&D once #4 gives a per-reader CER measurement surface. Left unlinked so any can start independently.
  • No change was made to product code in this run (research + triage only).
## What I found **This is a research issue, so here is the full investigation + landscape; the actionable work is split into #3/#4/#5 below.** **Root cause of "mostly detects junk."** There is no new bug to fix in one place — there is a structural gap. The default reader (`ocrMode: "builtin"`, `src/main.ts` `recognizeLocally`) runs a pipeline that has **no component capable of arbitrary joined-up cursive**, and wrong readings leak past the trust thresholds instead of falling back to the ink image (which is the correct upstream behaviour — *show the image unless sure*). ```mermaid flowchart TD S[pen strokes] --> B["$P per-letter reader (recognize.ts)<br/>segments by horizontal GAPS"] B -->|confidence ≥ 0.65 OCR_TRUST| T[use text] B -->|unsure| W["WritePad WASM (writepad.ts)<br/>GPL cursive engine, SILENT fallback"] W -->|confidence ≥ 0.70 WRITEPAD_TRUST| T W -->|unsure| I[image only] B -. junk leaks ≥0.65 .-> T W -. junk leaks ≥0.70 .-> T ``` Each local mechanism has a narrow competence: - **$P letter reader** (`recognize.ts`) — explicitly *"built for separate print letters; joined-up writing comes back with low confidence."* Cursive has no inter-letter gaps, so its `clusterStrokes`/`segment` step mis-segments → garbage candidates. - **Whole-word DTW reader** (`words.ts`) — only reads words already in the vault vocabulary, via the owner's samples or script-font traces. Writer-adaptive, but narrow. - **WritePad WASM** (`writepad.ts`) — genuinely cursive-capable (GPL-3.0, embedded in `engine.js`; license is clean since the plugin is GPL-3.0), but it is a **hidden fallback nobody can see firing**, uses fixed English/German dictionaries, and can only be taught vocabulary — not the owner's hand. - **Vision endpoint** (`transport/ocr.ts`) — opt-in Ollama/LM-Studio/hosted VLM; you noted it is *untested due to setup difficulties*. **Research landscape (2026) — what can actually read cursive locally.** I checked the current options against your "local / integrated JS lib over 3rd-party & API" preference, so no one re-treads the dead ends: - ❌ **Native `navigator.createHandwritingRecognizer` (Handwriting Recognition API)** — stroke-based, local, and maps *exactly* onto the data you already have, but it is **ChromeOS-only**; it rejects in Obsidian's Electron on Linux/macOS/Windows. Not viable. ([Chrome docs](https://developer.chrome.com/docs/web-platform/handwriting-recognition)) - ❌ **MyScript iinkJS / iink-ts** — best-in-class cursive, but cloud/commercial subscription. Violates local-first. ([repo](https://github.com/MyScript/MyScriptJS)) - ❌ **No off-the-shelf integrated JS lib** does robust Latin cursive on-device *and* trains on corrections. Transformer online-HTR is research-only, not a packaged ONNX/tfjs model. ([arXiv 2506.20255](https://arxiv.org/abs/2506.20255), [T-OHS](https://www.sciencedirect.com/science/article/abs/pii/S0031320323008282)) — so an integrated reader is **build-not-adopt**. - ✅ **Local vision model via Ollama/LM Studio** — in 2026 a local VLM (qwen2.5-VL, MiniCPM-V) is the only thing that reads arbitrary cursive *well* while keeping ink on your machine. It is "local running" but still an HTTP call to a local server — the honest trade-off against your "integrated JS lib" ideal. **The assets you mentioned are already in the codebase** and are exactly what a writer-adapted approach exploits: per-character mappings = the handwriting sheet glyphs (`ink/handsheet.ts`, used as $P templates); example words/sentences = the word sheet + vault `Lexicon` + corrected remarks. Continuous learning already exists at **word granularity** (`ocr/samples.ts` `learnCorrection`) — the opportunity is to push it to letter level and/or a trainable model. **Recommendation:** two parallel tracks — the pragmatic high-accuracy one (local VLM, #3) and the one matching your stated preference (integrated trainable reader, #5) — plus an immediate reliability fix (#4) so the current reader stops emitting junk regardless of which track wins. ## Created - **#3** — make the local vision-model path (Ollama/LM Studio) reliable & self-serve *(pragmatic, highest accuracy now; independent)* - **#4** — stop the built-in reader emitting junk: calibrate confidence + make readings diagnosable *(immediate reliability; independent)* - **#5** — deepen continuous learning from corrections + spike an integrated trainable cursive reader *(matches your local-JS-lib preference; R&D, read #3/#4 outcomes first)* No MRs opened directly: every track still needs a design/scope decision, so each went to a planning agent rather than a half-specified implementation branch. ## Follow-ups - Suggested order: **#3** (quick self-serve win) and **#4** (fail-safe) first, in parallel; **#5** as the longer-term R&D once #4 gives a per-reader CER measurement surface. Left unlinked so any can start independently. - No change was made to product code in this run (research + triage only).
Author
Owner

@claude could we train our own handwriting word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?

@claude could we train our own handwriting word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?
Collaborator

@daniel — short answer: yes, and it's the right instinct — but the "MNIST / handbook neural-net OCR" shape needs one important change, and the good news is we already own the one asset that normally makes this too expensive to attempt.

Why not literally "word-based OCR like the MNIST example"

The handbook example classifies one isolated glyph into one of N classes. Two problems for us:

  1. A per-word classifier only reads words it has seen. It can't read a word (or name, or typo) outside its training vocabulary — which is most of what you write. That's the same trap the current whole-word DTW reader (words.ts) is already in.
  2. Cursive has no segmentation. You can't chop joined-up writing into glyphs to feed a classifier — finding the cuts is the hard part.

The architecture that solves both is a segmentation-free sequence model: feed the pen trajectory straight in and let it emit characters, with no pre-cutting and no fixed word list. For online ink (which is what we have — timed strokes, not pixels) the canonical design is stroke-sequence → BiLSTM → CTC (Graves; Google's multi-language IAM-OnDB system Springer). No CNN needed — that's only for image OCR. A tiny public reference doing exactly this on pen strokes: ScribbleAI (BiLSTM+CTC on (x,y,t,p) traces).

The asset that makes this feasible for us specifically

Training an HTR model from scratch normally needs tens of thousands of labelled words — we don't have that. But the plugin can already synthesise unlimited labelled ink in your own hand. renderInk() + the handwriting-sheet glyphs (src/ink/handsheet.ts, src/ink/glyphs.ts) render any text to device strokes in your writing — that's literally what the recognizer test does (write("Bitte noch einmal prüfen", …)). ScribbleAI uses this exact trick (isolated chars → synthetic lines). So:

  • per-character sheet → compose into an endless synthetic corpus of (strokes → text) in your hand;
  • word sheet + example sentences → realistic joins and spacing;
  • your corrections (src/ocr/samples.ts, already exported as a training set by src/ocr/trainingset.ts) → real gold data on top.

That directly uses "the writers per character mappings and some example sentences on hand" — they stop being $P templates and become a data generator.

Continuous training (the part you're most keen on)

The honest version isn't per-stroke SGD in the plugin — that causes catastrophic forgetting (the model gets better at your last correction and worse at everything else). The 2025/2026 answer is one of:

Either way our existing corrections pipeline is the data tap — it already accumulates (ink, corrected-text) pairs word-by-word and survives syncs.

Where it runs (staying local, your stated preference)

Two options, and the spike should pick between them on evidence:

  • Train/fine-tune offline (Python), ship weights, infer in-browser via tfjs-wasm / onnxruntime-web. Lower risk; trainingset.ts already emits the dataset. "Continuous training" becomes an occasional retrain you trigger.
  • Fine-tune in-browser via tfjs. A small BiLSTM-CTC can train in-browser, but it's slow and memory-hungry; the realistic form is a periodic background adapter-fine-tune, not live learning. This is the genuinely integrated, no-server dream — and the riskiest.

Decoding would reuse what we already have: CTC beam search biased by the vault Lexicon (our built-in word LM).

The honest catch

This is a real ML project, not a config change: CTC training + beam decode, CER tuning, bundle size (a few MB of weights + the tfjs/ort runtime), and Obsidian-acceptable latency. So it must earn its place — beat the WritePad + local-VLM baseline on your own corrections, within budget, or we stop at the cheaper wins. That gate is exactly what I've written up as the spike in #5, which I've just sharpened to carry this concrete architecture (online BiLSTM-CTC, synth-from-sheet corpus, replay/adapter continual learning) and marked as your explicitly-requested direction. #3 (local VLM) stays the pragmatic high-accuracy baseline it has to beat.

Net: it's the most on-brand path for this plugin (local, integrated, learns your hand). It's just build-not-adopt, so #5 gates it on a feasibility spike before any implementation.

@daniel — short answer: **yes, and it's the right instinct** — but the "MNIST / handbook neural-net OCR" shape needs one important change, and the good news is we already own the one asset that normally makes this too expensive to attempt. ## Why not literally "word-based OCR like the MNIST example" The handbook example classifies **one isolated glyph** into one of N classes. Two problems for us: 1. **A per-word classifier only reads words it has seen.** It can't read a word (or name, or typo) outside its training vocabulary — which is most of what you write. That's the same trap the current whole-word DTW reader (`words.ts`) is already in. 2. **Cursive has no segmentation.** You can't chop joined-up writing into glyphs to feed a classifier — finding the cuts *is* the hard part. The architecture that solves both is a **segmentation-free sequence model**: feed the pen trajectory straight in and let it emit characters, with no pre-cutting and no fixed word list. For online ink (which is what we have — timed strokes, not pixels) the canonical design is **stroke-sequence → BiLSTM → CTC** (Graves; Google's multi-language IAM-OnDB system [Springer](https://link.springer.com/article/10.1007/s10032-020-00350-4)). No CNN needed — that's only for image OCR. A tiny public reference doing exactly this on pen strokes: [ScribbleAI (BiLSTM+CTC on `(x,y,t,p)` traces)](https://github.com/DarthVader69oss/scribbleai). ## The asset that makes this feasible for *us* specifically Training an HTR model from scratch normally needs tens of thousands of labelled words — we don't have that. **But the plugin can already synthesise unlimited labelled ink in *your own hand*.** `renderInk()` + the handwriting-sheet glyphs (`src/ink/handsheet.ts`, `src/ink/glyphs.ts`) render *any* text to device strokes in your writing — that's literally what the recognizer test does (`write("Bitte noch einmal prüfen", …)`). ScribbleAI uses this exact trick (isolated chars → synthetic lines). So: - **per-character sheet** → compose into an endless synthetic corpus of (strokes → text) in your hand; - **word sheet + example sentences** → realistic joins and spacing; - **your corrections** (`src/ocr/samples.ts`, already exported as a training set by `src/ocr/trainingset.ts`) → real gold data on top. That directly uses "the writers per character mappings and some example sentences on hand" — they stop being $P templates and become a **data generator**. ## Continuous training (the part you're most keen on) The honest version isn't per-stroke SGD in the plugin — that causes **catastrophic forgetting** (the model gets better at your last correction and worse at everything else). The 2025/2026 answer is one of: - **replay** — keep a buffer of corrections + synthetic samples and periodically re-fine-tune ([replay/regularization/architecture taxonomy](https://arxiv.org/pdf/2602.00722)); - **a small frozen base + a writer-specific adapter** you keep adapting — LoRA-style, incl. task-free streaming variants ([Online-LoRA, WACV 2025](https://openaccess.thecvf.com/content/WACV2025/papers/Wei_Online-LoRA_Task-Free_Online_Continual_Learning_via_Low_Rank_Adaptation_WACV_2025_paper.pdf); EWC, Kirkpatrick 2017). Either way our existing corrections pipeline is the data tap — it already accumulates `(ink, corrected-text)` pairs word-by-word and survives syncs. ## Where it runs (staying local, your stated preference) Two options, and the spike should pick between them on evidence: - **Train/fine-tune offline (Python), ship weights, infer in-browser** via `tfjs-wasm` / `onnxruntime-web`. Lower risk; `trainingset.ts` already emits the dataset. "Continuous training" becomes an occasional retrain you trigger. - **Fine-tune in-browser** via tfjs. A small BiLSTM-CTC *can* train in-browser, but it's slow and memory-hungry; the realistic form is a periodic background adapter-fine-tune, not live learning. This is the genuinely *integrated, no-server* dream — and the riskiest. Decoding would reuse what we already have: CTC beam search biased by the vault `Lexicon` (our built-in word LM). ## The honest catch This is a real ML project, not a config change: CTC training + beam decode, CER tuning, bundle size (a few MB of weights + the tfjs/ort runtime), and Obsidian-acceptable latency. So it must earn its place — **beat the WritePad + local-VLM baseline on your own corrections, within budget, or we stop at the cheaper wins.** That gate is exactly what I've written up as the spike in **#5**, which I've just sharpened to carry this concrete architecture (online BiLSTM-CTC, synth-from-sheet corpus, replay/adapter continual learning) and marked as your explicitly-requested direction. #3 (local VLM) stays the pragmatic high-accuracy baseline it has to beat. Net: it's the most on-brand path for this plugin (local, integrated, learns your hand). It's just build-not-adopt, so #5 gates it on a feasibility spike before any implementation.
Author
Owner

@claude whats the status of this issue and its child issues/mrs?

@claude whats the status of this issue and its child issues/mrs?
Collaborator

@daniel — status roll-up. Most of the plan has shipped; 7 MRs merged, one in review, and one proposal waiting on you.

#4 — stop the built-in reader emitting junk ✅ done

Directly addresses the original "mostly detects junk" complaint. All three pieces merged:

  • !9 — per-remark reader + confidence surfaced in the import report (you can now see which reader produced each text and how sure it was)
  • !10 — calibration so the $P letter reader stops passing junk cursive above OCR_TRUST, with labelled regression fixtures
  • !11 — WritePad fallback gated on reading quality + made attributable/toggleable (no more invisible fallback)

#3 — local vision model (Ollama/LM Studio) reliable & self-serve ✅ done

  • !6 — bundled setup guide (pull the model, OLLAMA_ORIGINS=app://obsidian.md, LM Studio equivalent)
  • !7 — Test-connection button + actionable errors (connection refused / 404 route / model not pulled / the CORS origin gotcha)
  • Spawned #8 (see below) from your "can't we just bundle it?" question.

#5 — continuous learning + train-our-own reader 🔄 in progress

This is the track your "train our own handwriting OCR" question feeds into.

  • !12 — Direction 1: letter-level learning from corrections merged — the reader now learns per-letter templates in your hand from your corrections, not just whole words.
  • !15 — Direction 2: bound read latency as the per-writer store grows merged; !16 (keep the N most-diverse samples, not just newest) open, in review.
  • Direction 3 — the trainable online BiLSTM-CTC reader you asked about — not started yet. It's deliberately gated behind a feasibility spike (must beat the WritePad + local-VLM baseline within an Obsidian-acceptable bundle/runtime budget before any implementation). That's the next substantive piece and the one open research question.

⚠️ One thing needs your decision

#8 — one-click "managed local engine" (detect/install Ollama, pull the model, set the origin) — is open and unassigned on purpose, waiting for you to greenlight or close. It came out of your "bundle it so the user doesn't have to set up the LLM stuff" question on #3. The finding: we can't bundle a cursive-capable model (~5–6 GB, must be downloaded; models small enough to embed can't read cursive), but we can collapse the setup into one button that manages a local Ollama for you while keeping the manual endpoint for advanced users. Desktop-only (needs child_process; no mobile). Want this built, or shelved?

TL;DR

Track State
#4 reliability (stop junk) ✅ merged (!9, !10, !11)
#3 local-VLM self-serve ✅ merged (!6, !7)
#5 learn-your-hand (letter-level + store) 🔄 Dir 1–2 merged (!12, !15), !16 in review
#5 train-our-own reader (Dir 3) ⏳ pending feasibility spike
#8 one-click managed engine ❓ awaiting your go/no-go

Child issues #3 and #4 are functionally complete (all their MRs merged) and can be closed; #5 stays open for Direction 3. Happy to close #3/#4 or push the #5 spike next — your call.

@daniel — status roll-up. Most of the plan has shipped; **7 MRs merged**, one in review, and one proposal waiting on you. ## #4 — stop the built-in reader emitting junk ✅ *done* Directly addresses the original "mostly detects junk" complaint. All three pieces merged: - !9 — per-remark reader + confidence surfaced in the import report (you can now see *which* reader produced each text and how sure it was) - !10 — calibration so the $P letter reader stops passing junk cursive above `OCR_TRUST`, with labelled regression fixtures - !11 — WritePad fallback gated on reading quality + made attributable/toggleable (no more invisible fallback) ## #3 — local vision model (Ollama/LM Studio) reliable & self-serve ✅ *done* - !6 — bundled setup guide (pull the model, `OLLAMA_ORIGINS=app://obsidian.md`, LM Studio equivalent) - !7 — **Test-connection** button + actionable errors (connection refused / 404 route / model not pulled / the CORS origin gotcha) - Spawned **#8** (see below) from your "can't we just bundle it?" question. ## #5 — continuous learning + train-our-own reader 🔄 *in progress* This is the track your "train our own handwriting OCR" question feeds into. - !12 — **Direction 1: letter-level learning from corrections** merged — the reader now learns per-letter templates in your hand from your corrections, not just whole words. - !15 — **Direction 2: bound read latency as the per-writer store grows** merged; !16 (*keep the N most-diverse samples, not just newest*) **open, in review**. - **Direction 3 — the trainable online BiLSTM-CTC reader you asked about — not started yet.** It's deliberately gated behind a feasibility spike (must beat the WritePad + local-VLM baseline within an Obsidian-acceptable bundle/runtime budget before any implementation). That's the next substantive piece and the one open research question. ## ⚠️ One thing needs *your* decision **#8 — one-click "managed local engine"** (detect/install Ollama, pull the model, set the origin) — is **open and unassigned on purpose**, waiting for you to greenlight or close. It came out of your "bundle it so the user doesn't have to set up the LLM stuff" question on #3. The finding: we can't *bundle* a cursive-capable model (~5–6 GB, must be downloaded; models small enough to embed can't read cursive), but we **can** collapse the setup into one button that manages a local Ollama for you while keeping the manual endpoint for advanced users. Desktop-only (needs `child_process`; no mobile). **Want this built, or shelved?** ## TL;DR | Track | State | |---|---| | #4 reliability (stop junk) | ✅ merged (!9, !10, !11) | | #3 local-VLM self-serve | ✅ merged (!6, !7) | | #5 learn-your-hand (letter-level + store) | 🔄 Dir 1–2 merged (!12, !15), !16 in review | | #5 train-our-own reader (Dir 3) | ⏳ pending feasibility spike | | #8 one-click managed engine | ❓ awaiting your go/no-go | Child issues #3 and #4 are functionally complete (all their MRs merged) and can be closed; #5 stays open for Direction 3. Happy to close #3/#4 or push the #5 spike next — your call.
Collaborator

Agent failed ❌

exceeded pending task timeout

**Agent failed** :x: exceeded pending task timeout
Collaborator

🤖 Agent is working on this — turn 0

Last activity: session init — updated 16:57:17 UTC

🤖 **Agent is working on this** — turn 0 _Last activity: session init — updated 16:57:17 UTC_
Collaborator

✅ Agent run completed

9s · 3 turns · $0.77 · session saved — new comments resume this run

✅ **Agent run completed** _9s · 3 turns · $0.77 · session saved — new comments resume this run_
Collaborator

@daniel — here's where this ended up. My earlier reply above was written before the feasibility spike. The spike has now run.

Short version: a self-trained model is feasible, but it isn't worth building yet. The spike's measurements say the cheaper "learn your hand" work gets more for the effort right now. The model design is written up and parked in #14, with a clear condition for picking it back up.

What the spike measured (#5)

Character error rate (CER) on held-out words, so lower is better:

Reader Separable hand Cursive hand
Built-in $P letters (baseline) 38.1% 95.8%
+ word-level learning from corrections 38.1% 34.2%
+ letter-level learning (Direction 1, !12) 20.8% 88.9%
WritePad (GPL cursive engine) 54.6% 21.6%
Local VLM (#3) not measured not measured

These numbers come from a synthetic corpus rendered from the handwriting sheets. Agents can't access your real ocr/samples.json, so treat them as directional.

Why the model is parked rather than built

  • The design is unchanged: pen strokes → BiLSTM → CTC, then a beam search biased toward vault words. It would train on synthetic ink in your hand plus your corrections. Continuous learning would use replay or a small per-writer adapter rather than naive per-correction updates, which make the model forget what it already knew.
  • To earn its place, a model of a few MB has to beat 21.6% CER on cursive, which WritePad already reaches for free. It also has to keep bundle size and latency acceptable. The plugin currently has no ML dependencies, so the runtime alone (tfjs or onnxruntime-web) is a real cost.
  • Reusing your sheets to synthesise whole words and letter joins (Direction 2) attacks the same cursive gap much more cheaply. That's what is running now.

Revisit condition for #14: the local VLM is unacceptable to you, and WritePad falls short on your real writing, and Directions 1–2 don't close the gap.

What shipped instead

  • !12: learning letter templates from your corrections.
  • !15 and !16: keeping read latency bounded as the store of samples grows, and keeping the most varied word samples rather than just the newest.
  • !18 and !19: the same varied-sample retention for letters, plus a seam for migrating the stored data format.
  • #21 (a spike to settle the approach, in progress) is a prerequisite for #20 (letter-join templates and synthesising words the reader has never seen; queued). #20 is the direct cheap competitor to the neural model.

If you want the model anyway, say so and I'll take #14 off hold. Its first step would still be a measured spike, a small model that must beat 21.6% cursive CER, before any production code.

@daniel — here's where this ended up. My earlier reply above was written before the feasibility spike. The spike has now run. **Short version:** a self-trained model is feasible, but it isn't worth building yet. The spike's measurements say the cheaper "learn your hand" work gets more for the effort right now. The model design is written up and parked in **#14**, with a clear condition for picking it back up. ### What the spike measured (#5) Character error rate (CER) on held-out words, so lower is better: | Reader | Separable hand | Cursive hand | |---|---|---| | Built-in $P letters (baseline) | 38.1% | 95.8% | | + word-level learning from corrections | 38.1% | 34.2% | | + letter-level learning (Direction 1, !12) | **20.8%** | 88.9% | | WritePad (GPL cursive engine) | 54.6% | **21.6%** | | Local VLM (#3) | not measured | not measured | These numbers come from a **synthetic** corpus rendered from the handwriting sheets. Agents can't access your real `ocr/samples.json`, so treat them as directional. ### Why the model is parked rather than built - The design is unchanged: pen strokes → BiLSTM → CTC, then a beam search biased toward vault words. It would train on synthetic ink in your hand plus your corrections. Continuous learning would use replay or a small per-writer adapter rather than naive per-correction updates, which make the model forget what it already knew. - To earn its place, a model of a few MB has to beat **21.6% CER on cursive**, which WritePad already reaches for free. It also has to keep bundle size and latency acceptable. The plugin currently has no ML dependencies, so the runtime alone (tfjs or onnxruntime-web) is a real cost. - Reusing your sheets to synthesise whole words and letter joins (Direction 2) attacks the same cursive gap much more cheaply. That's what is running now. **Revisit condition for #14:** the local VLM is unacceptable to you, **and** WritePad falls short on your real writing, **and** Directions 1–2 don't close the gap. ### What shipped instead - !12: learning letter templates from your corrections. - !15 and !16: keeping read latency bounded as the store of samples grows, and keeping the most varied word samples rather than just the newest. - !18 and !19: the same varied-sample retention for letters, plus a seam for migrating the stored data format. - **#21** (a spike to settle the approach, in progress) is a prerequisite for **#20** (letter-join templates and synthesising words the reader has never seen; queued). #20 is the direct cheap competitor to the neural model. If you want the model anyway, say so and I'll take #14 off hold. Its first step would still be a measured spike, a small model that must beat 21.6% cursive CER, before any production code.
Collaborator

@daniel — here's an updated status. My earlier roll-up (note above) is out of date on two points: !16 has since merged, and Direction 3 ("train our own reader") has now been spiked and parked, not left "not started".

Track State
#4 stop the built-in reader emitting junk ✅ !9, !10 and !11 merged. Functionally complete; an agent is doing a final check on main.
#3 local VLM (Ollama/LM Studio) self-serve ✅ !6 and !7 merged. Functionally complete; an agent is answering your latest thread there.
#8 one-click managed Ollama engine 🔄 In progress after your go-ahead (calling Ollama is OK, direct download preferred, downloading the model at runtime is fine). An agent is working through the settings and connection code.
#5 learn your hand: Direction 1 ✅ !12 merged (letter-level learning from corrections)
#5 Direction 2: per-writer sample store ✅ !15 and !16 merged (#13); !18 and !19 merged (#17)
#5 Direction 2: letter joins + synthesising unseen words 🔄 #21 spike to settle the approach (agent active), then #20 (queued; starts automatically when #21 closes)
#14 Direction 3: train our own neural reader ⏸️ Parked as no-go for now. See my reply in your "train our own OCR" thread for the numbers and the condition for revisiting it.

Spike headline (synthetic corpus, held-out words, character error rate): on cursive, the built-in reader gets 95.8%, word-level learning 34.2% and WritePad 21.6%. On separable writing, letter-level learning brings the built-in reader from 38.1% down to 20.8%. These are directional only, because agents can't access your real ocr/samples.json.

What you can do now

  • Close #3 and #4 once their current agent runs finish; both have every MR merged.
  • Try the built-in reader plus the corrections loop on real pages. Your real-world error rate is the missing data point for #14's revisit condition.
  • Tell me if you want #14 taken off hold.

Not OCR-related, but in the same repo: #22 (nested callouts) has !24 and !25 merged, and !23 is still open.

@daniel — here's an updated status. My earlier roll-up (note above) is **out of date** on two points: !16 has since merged, and Direction 3 ("train our own reader") has now been spiked and parked, not left "not started". | Track | State | |---|---| | **#4** stop the built-in reader emitting junk | ✅ !9, !10 and !11 merged. Functionally complete; an agent is doing a final check on main. | | **#3** local VLM (Ollama/LM Studio) self-serve | ✅ !6 and !7 merged. Functionally complete; an agent is answering your latest thread there. | | **#8** one-click managed Ollama engine | 🔄 In progress after your go-ahead (calling Ollama is OK, direct download preferred, downloading the model at runtime is fine). An agent is working through the settings and connection code. | | **#5** learn your hand: Direction 1 | ✅ !12 merged (letter-level learning from corrections) | | **#5** Direction 2: per-writer sample store | ✅ !15 and !16 merged (#13); !18 and !19 merged (#17) | | **#5** Direction 2: letter joins + synthesising unseen words | 🔄 **#21** spike to settle the approach (agent active), then **#20** (queued; starts automatically when #21 closes) | | **#14** Direction 3: train our own neural reader | ⏸️ Parked as no-go for now. See my reply in your "train our own OCR" thread for the numbers and the condition for revisiting it. | **Spike headline** (synthetic corpus, held-out words, character error rate): on cursive, the built-in reader gets 95.8%, word-level learning 34.2% and WritePad 21.6%. On separable writing, letter-level learning brings the built-in reader from 38.1% down to 20.8%. These are directional only, because agents can't access your real `ocr/samples.json`. **What you can do now** - Close **#3** and **#4** once their current agent runs finish; both have every MR merged. - Try the built-in reader plus the corrections loop on real pages. Your real-world error rate is the missing data point for #14's revisit condition. - Tell me if you want #14 taken off hold. Not OCR-related, but in the same repo: #22 (nested callouts) has !24 and !25 merged, and !23 is still open.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip#2
No description provided.