improve / simplify stroke to text transcription #2
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip#2
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
the current builtin solution is not very relieable and mostly detects junk, local ollama hasnt been tested yet because of setup difficulties.
reasearch new / creative ways to solve the problem of detecting joined up curly writing (mind we have the writers per character mappings and some example sentences on hand)
a local running and/or integrated js lib solution is preferred over 3rd party toolings and api calls.
if possible continous training on the user specified corrections would be a great opportunity to take.
❌ Agent run failed
session saved — new comments resume this run
claude referenced this issue2026-10-20 15:13:30 +02:00
claude referenced this issue2026-10-20 15:13:32 +02:00
claude referenced this issue2026-10-20 15:14:04 +02:00
What I found
This is a research issue, so here is the full investigation + landscape; the actionable work is split into #3/#4/#5 below.
Root cause of "mostly detects junk." There is no new bug to fix in one place — there is a structural gap. The default reader (
ocrMode: "builtin",src/main.tsrecognizeLocally) runs a pipeline that has no component capable of arbitrary joined-up cursive, and wrong readings leak past the trust thresholds instead of falling back to the ink image (which is the correct upstream behaviour — show the image unless sure).Each local mechanism has a narrow competence:
recognize.ts) — explicitly "built for separate print letters; joined-up writing comes back with low confidence." Cursive has no inter-letter gaps, so itsclusterStrokes/segmentstep mis-segments → garbage candidates.words.ts) — only reads words already in the vault vocabulary, via the owner's samples or script-font traces. Writer-adaptive, but narrow.writepad.ts) — genuinely cursive-capable (GPL-3.0, embedded inengine.js; license is clean since the plugin is GPL-3.0), but it is a hidden fallback nobody can see firing, uses fixed English/German dictionaries, and can only be taught vocabulary — not the owner's hand.transport/ocr.ts) — opt-in Ollama/LM-Studio/hosted VLM; you noted it is untested due to setup difficulties.Research landscape (2026) — what can actually read cursive locally. I checked the current options against your "local / integrated JS lib over 3rd-party & API" preference, so no one re-treads the dead ends:
navigator.createHandwritingRecognizer(Handwriting Recognition API) — stroke-based, local, and maps exactly onto the data you already have, but it is ChromeOS-only; it rejects in Obsidian's Electron on Linux/macOS/Windows. Not viable. (Chrome docs)The assets you mentioned are already in the codebase and are exactly what a writer-adapted approach exploits: per-character mappings = the handwriting sheet glyphs (
ink/handsheet.ts, used as $P templates); example words/sentences = the word sheet + vaultLexicon+ corrected remarks. Continuous learning already exists at word granularity (ocr/samples.tslearnCorrection) — the opportunity is to push it to letter level and/or a trainable model.Recommendation: two parallel tracks — the pragmatic high-accuracy one (local VLM, #3) and the one matching your stated preference (integrated trainable reader, #5) — plus an immediate reliability fix (#4) so the current reader stops emitting junk regardless of which track wins.
Created
No MRs opened directly: every track still needs a design/scope decision, so each went to a planning agent rather than a half-specified implementation branch.
Follow-ups
claude referenced this issue2026-10-20 15:15:59 +02:00
claude referenced this issue2026-10-20 15:19:38 +02:00
claude referenced this issue2026-10-20 15:21:52 +02:00
@claude could we train our own handwriting word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?
@daniel — short answer: yes, and it's the right instinct — but the "MNIST / handbook neural-net OCR" shape needs one important change, and the good news is we already own the one asset that normally makes this too expensive to attempt.
Why not literally "word-based OCR like the MNIST example"
The handbook example classifies one isolated glyph into one of N classes. Two problems for us:
words.ts) is already in.The architecture that solves both is a segmentation-free sequence model: feed the pen trajectory straight in and let it emit characters, with no pre-cutting and no fixed word list. For online ink (which is what we have — timed strokes, not pixels) the canonical design is stroke-sequence → BiLSTM → CTC (Graves; Google's multi-language IAM-OnDB system Springer). No CNN needed — that's only for image OCR. A tiny public reference doing exactly this on pen strokes: ScribbleAI (BiLSTM+CTC on
(x,y,t,p)traces).The asset that makes this feasible for us specifically
Training an HTR model from scratch normally needs tens of thousands of labelled words — we don't have that. But the plugin can already synthesise unlimited labelled ink in your own hand.
renderInk()+ the handwriting-sheet glyphs (src/ink/handsheet.ts,src/ink/glyphs.ts) render any text to device strokes in your writing — that's literally what the recognizer test does (write("Bitte noch einmal prüfen", …)). ScribbleAI uses this exact trick (isolated chars → synthetic lines). So:src/ocr/samples.ts, already exported as a training set bysrc/ocr/trainingset.ts) → real gold data on top.That directly uses "the writers per character mappings and some example sentences on hand" — they stop being $P templates and become a data generator.
Continuous training (the part you're most keen on)
The honest version isn't per-stroke SGD in the plugin — that causes catastrophic forgetting (the model gets better at your last correction and worse at everything else). The 2025/2026 answer is one of:
Either way our existing corrections pipeline is the data tap — it already accumulates
(ink, corrected-text)pairs word-by-word and survives syncs.Where it runs (staying local, your stated preference)
Two options, and the spike should pick between them on evidence:
tfjs-wasm/onnxruntime-web. Lower risk;trainingset.tsalready emits the dataset. "Continuous training" becomes an occasional retrain you trigger.Decoding would reuse what we already have: CTC beam search biased by the vault
Lexicon(our built-in word LM).The honest catch
This is a real ML project, not a config change: CTC training + beam decode, CER tuning, bundle size (a few MB of weights + the tfjs/ort runtime), and Obsidian-acceptable latency. So it must earn its place — beat the WritePad + local-VLM baseline on your own corrections, within budget, or we stop at the cheaper wins. That gate is exactly what I've written up as the spike in #5, which I've just sharpened to carry this concrete architecture (online BiLSTM-CTC, synth-from-sheet corpus, replay/adapter continual learning) and marked as your explicitly-requested direction. #3 (local VLM) stays the pragmatic high-accuracy baseline it has to beat.
Net: it's the most on-brand path for this plugin (local, integrated, learns your hand). It's just build-not-adopt, so #5 gates it on a feasibility spike before any implementation.
@claude whats the status of this issue and its child issues/mrs?
@daniel — status roll-up. Most of the plan has shipped; 7 MRs merged, one in review, and one proposal waiting on you.
#4 — stop the built-in reader emitting junk ✅ done
Directly addresses the original "mostly detects junk" complaint. All three pieces merged:
OCR_TRUST, with labelled regression fixtures#3 — local vision model (Ollama/LM Studio) reliable & self-serve ✅ done
OLLAMA_ORIGINS=app://obsidian.md, LM Studio equivalent)#5 — continuous learning + train-our-own reader 🔄 in progress
This is the track your "train our own handwriting OCR" question feeds into.
⚠️ One thing needs your decision
#8 — one-click "managed local engine" (detect/install Ollama, pull the model, set the origin) — is open and unassigned on purpose, waiting for you to greenlight or close. It came out of your "bundle it so the user doesn't have to set up the LLM stuff" question on #3. The finding: we can't bundle a cursive-capable model (~5–6 GB, must be downloaded; models small enough to embed can't read cursive), but we can collapse the setup into one button that manages a local Ollama for you while keeping the manual endpoint for advanced users. Desktop-only (needs
child_process; no mobile). Want this built, or shelved?TL;DR
Child issues #3 and #4 are functionally complete (all their MRs merged) and can be closed; #5 stays open for Direction 3. Happy to close #3/#4 or push the #5 spike next — your call.
Agent failed ❌
exceeded pending task timeout
claude referenced this issue2026-10-20 18:52:05 +02:00
claude referenced this issue2026-10-20 18:52:06 +02:00
🤖 Agent is working on this — turn 0
Last activity: session init — updated 16:57:17 UTC
✅ Agent run completed
9s · 3 turns · $0.77 · session saved — new comments resume this run
@daniel — here's where this ended up. My earlier reply above was written before the feasibility spike. The spike has now run.
Short version: a self-trained model is feasible, but it isn't worth building yet. The spike's measurements say the cheaper "learn your hand" work gets more for the effort right now. The model design is written up and parked in #14, with a clear condition for picking it back up.
What the spike measured (#5)
Character error rate (CER) on held-out words, so lower is better:
These numbers come from a synthetic corpus rendered from the handwriting sheets. Agents can't access your real
ocr/samples.json, so treat them as directional.Why the model is parked rather than built
Revisit condition for #14: the local VLM is unacceptable to you, and WritePad falls short on your real writing, and Directions 1–2 don't close the gap.
What shipped instead
If you want the model anyway, say so and I'll take #14 off hold. Its first step would still be a measured spike, a small model that must beat 21.6% cursive CER, before any production code.
@daniel — here's an updated status. My earlier roll-up (note above) is out of date on two points: !16 has since merged, and Direction 3 ("train our own reader") has now been spiked and parked, not left "not started".
Spike headline (synthetic corpus, held-out words, character error rate): on cursive, the built-in reader gets 95.8%, word-level learning 34.2% and WritePad 21.6%. On separable writing, letter-level learning brings the built-in reader from 38.1% down to 20.8%. These are directional only, because agents can't access your real
ocr/samples.json.What you can do now
Not OCR-related, but in the same repo: #22 (nested callouts) has !24 and !25 merged, and !23 is still open.