OCR: deepen continuous learning from corrections + spike an integrated trainable cursive reader #5
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip#5
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #2
Part of #2. This is the track that best matches the issue author's stated preference — "a local running and/or integrated js lib solution" and "continuous training on the user specified corrections would be a great opportunity" — using the assets they call out: "the writers per character mappings and some example sentences on hand." The owner explicitly asked for this (note 43818 on #2): "could we train our own handwriting word based ocr … just beefed up specialized for handwriting recognition, incorporating recent researches about continuous training/learning of models?" Higher-effort / higher-risk R&D; needs a research spike before any implementation. Keep it decoupled from the pragmatic local-VLM track (#3) and the calibration track (#4).
What already exists (build on it, don't rebuild)
src/ink/handsheet.tsSHEET_CHARS,src/ink/glyphs.ts), used as $P templates (templatesFrominrecognize.ts).WORD_SHEET_WORDSinhandsheet.ts), the vault vocabulary (Lexicon), and corrected remarks.renderInk()+ the sheet glyphs render arbitrary text → device strokes in the owner's own hand (this is exactly whatsrc/__tests__/recognize.test.tswrite()does). This removes the usual blocker for training HTR — data volume — because an unlimited labelled corpus in the owner's hand can be synthesised from the per-char sheet (ScribbleAI uses this same isolated-char→synthetic-line trick).src/ocr/samples.tslearnCorrection/learnFromNotes): a correction is paired to ink word-by-word and stored as word samples; the whole-word DTW reader (src/ocr/words.tsWordReader) checks the owner's own samples first. Already writer-adaptive — but only at whole-word granularity and only for words seen before.src/ocr/trainingset.tsalready emits the corrections as a dataset (ShareGPT/LLaMA-Factory layout);transport/ocr.tsworked-examples is the zero-training analogue.Why a classifier is the wrong shape (design decision, don't relitigate)
A per-word/per-letter classifier (the "MNIST handbook" shape the owner referenced) fails twice for this task: (1) it only reads words/letters in its training set — most real writing is out-of-vocabulary; (2) cursive can't be pre-segmented into glyphs to classify, and finding the cuts is the hard part. The fix is a segmentation-free sequence model that consumes the raw pen trajectory and emits characters with no pre-cutting and no fixed vocabulary.
Directions to spike (pick/measure, then split into MRs)
learnCorrectionto also derive per-letter templates when a corrected word's ink is separable, continuously enriching the owner's $P template set beyond the one-off handwriting sheet. Lowest-risk, incremental, measurable CER gain on the owner's own data. Independent of directions 2–3.ocr/samples.jsoncapped at 3 per word (PER_WORD). Evaluate a richer per-writer store (more samples, letter + join templates) and whether DTW/$P scale to it.(dx, dy, pen-up, …)resampled by arc length → ~3× BiLSTM → logits → CTC. No CNN (that is for image OCR); we have strokes. Canonical refs: Google multi-language IAM-OnDB system (https://link.springer.com/article/10.1007/s10032-020-00350-4); small public reference https://github.com/DarthVader69oss/scribbleai .renderInk()+ the sheet glyphs (per-char sheet → composed words/lines; word sheet for realistic joins), then mix in the real corrections. Optionally pretrain on a public online-HTR corpus (IAM-OnDB) first.Lexicon(already our built-in word LM), so it prefers real vault spellings without being limited to them.samples.ts→trainingset.ts) is the data tap.tfjs-wasm/onnxruntime-web(lower risk; "continuous training" = occasional retrain); or (b) fine-tune in-browser via tfjs (the fully-integrated no-server dream, but slow/memory-hungry — realistically a periodic background adapter fine-tune, not live learning).Dependencies
Benefits from #4's diagnostic surface (to measure CER per reader) and from #3's local-VLM baseline (as the accuracy bar to beat). Not hard-blocked, but a fresh agent should read #3/#4 outcomes first.
Scope boundary
NOT the endpoint UX (#3) and NOT threshold calibration (#4). Deliver a spike report first (feasibility + measured CER + bundle/runtime cost + go/no-go per direction), then MRs only for the directions that clear the baseline. Direction 1 (letter-level learning) may proceed as its own MR regardless of the direction-3 outcome, since it's low-risk and independently useful.
Validation contract
Spike deliverable: a comment with measured CER (built-in vs WritePad vs local VLM vs proposed) on a labelled set of the owner's corrections + synthetic hold-out, the bundle/runtime cost of the proposed reader, and a go/no-go per direction. Implementation MRs each:
nix develop -c npx vitest run src/__tests__→ PASS + a new test showing the learned-from-correction path improves a held-out read (red gate: the same read is wrong before the learned sample is added).claude referenced this issue2026-10-20 15:13:52 +02:00
✅ Agent run completed
12m38s · 54 turns · $3.80 · pushed
agent/ocr-letter-learning· session saved — new comments resume this runclaude referenced this issue2026-10-20 15:20:08 +02:00
claude referenced this issue2026-10-20 15:24:15 +02:00
claude referenced this issue2026-10-20 15:25:44 +02:00
claude referenced this issue2026-10-20 15:26:01 +02:00
Direction from @daniel (raised on #3) — a custom, word-based, continually-trained HTR model
@daniel proposed, on #3: "train our own [model] focused on word-based OCR, kinda like the handbook LLM OCR example you learn about neural networks with, just beefed up specialized for handwriting recognition, incorporating recent research about continuous training/learning." This is squarely #5's thesis, so recording the concrete blueprint and the honest risks here for whoever picks up the spike.
What "the handbook OCR example, beefed up" actually is
The canonical teaching example is CRNN + CTC: a CNN backbone → 2× BiLSTM → Connectionist Temporal Classification loss, trained on word/line images with no need for character-level segmentation. It's small (single-digit to tens of M params), and it's the right tool here precisely because it's small enough to run — and potentially adapt — locally. The battle-tested in-browser recipe is docTR's MobileNetV2-CRNN (input
32×128×3, 2 BiLSTMs, CTC), deployable via tfjs directly or onnxruntime-web (train in Keras →tf2onnx→ WASM/WebGPU). Reported word accuracy with beam-search decoding is ~92% across styles. (TF.js in-browser OCR / docTR, CNN-RNN-CTC approach, SimpleHTR)Why "word-based" is the strong part — and why this plugin is unusually well-placed
"Word-based" = CTC decoded with Word Beam Search constrained to a dictionary/LM (CTCWordBeamSearch). This plugin already owns the dictionary — the vault vocabulary and the word sheet — which is exactly where a lexicon-constrained small model beats a general VLM on known vocabulary. It also already has the full writer-specific data pipeline the training needs:
docs/ocr-corrections.md).That's a real per-writer dataset most HTR projects don't start with.
Continuous / continual learning — the honest framing
The realistic target is writer adaptation, not training a universal cursive reader from scratch: start from a pretrained base HTR model and adapt it to this hand from the user's corrections. Recent research gives two tractable routes:
Decisions the spike must make (real risks)
My take: this is the most fit-for-constraints of all the OCR directions (small, local, trainable, and the plugin already has per-writer data) — more promising than an embedded general VLM, which the research shows can't be both small enough to embed and good at cursive (see the #3 analysis). Worth a real spike here.
claude referenced this issue2026-10-20 15:26:10 +02:00
What I found
Spike delivered: measured CER + go/no-go per direction. I built a CER harness (
src/__tests__/spike5.test.ts, on branchagent/ocr-letter-learning) using the repo's own labelled-data method — render text through a font = a known "hand" — and measured every in-repo reader plus a working prototype of Direction 1.⚠️ Corpus caveat (honesty): the owner's real corrections live in their vault (
ocr/samples.json) and are not available to an agent, so numbers below are on a synthetic labelled corpus (word sheet + remark-like phrases, deterministic seeds). Relative orderings are robust and the method matches the repo's existing tests; absolute CER is synthetic. The local-VLM baseline is not measurable here (no Ollama/network) — it's the bar #3 will establish, expected to lead on cursive.Measured CER
Separable / print hand (owner hand differs from the generic templates):
Joined cursive hand (allure):
Go / no-go
(Env note: the validation contract names
nix develop -c npx vitest …, but there is noflake.nixin the repo — the working command isnpm installthennpx vitest run src/__tests__.)Created
Part of #5.Follow-ups
main(spike + hand-offs only); the only commit is the harness on the MR branch.claude referenced this issue2026-10-20 15:36:41 +02:00
claude referenced this issue2026-10-20 15:37:08 +02:00
claude referenced this issue2026-10-20 15:37:09 +02:00
claude referenced this issue2026-10-20 16:33:01 +02:00
claude referenced this issue2026-10-20 16:34:09 +02:00
claude referenced this issue2026-10-20 16:34:09 +02:00
🤖 Agent is working on this — turn 20
Last activity: tool: ToolSearch — {"query":"select:mcp__gitlab__forge_close_issue","max_results":1} — updated 22:36:09 UTC
Since my last update: Direction 1 (!12) and the Direction 2 store work have all merged: !15 latency caps, !16 diverse word retention, !18 diverse letter retention and !19 versioned
migrate()seam. The last open Direction 2 item is the join/ligature synthesis bet: #20, which is gated by #21, the spike that decides it. #21's last run failed for an infrastructure reason (model unavailable). I've told it on #21 that its prerequisites !18 and !19 have merged, which should resume it. A clean "no" from #21 is a valid outcome. Direction 3 (trainable online-HTR) has already been spiked and is parked as no-go in #14. The #2 roll-up (note 44037) calls it "not started", but that is out of date: the open question is the revisit gate in #14, not a missing spike. This issue stays open until #20 resolves.claude referenced this issue2026-10-20 18:57:43 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 20:06:07 +02:00
claude referenced this issue2026-10-20 20:06:07 +02:00
claude referenced this issue2026-10-20 20:06:45 +02:00
@claude whats the status here?
Since my last update: this issue is done. All three directions have a final answer.
!33 also shows up as linked here, but it is the Ollama installer from the local-reader track (#3), not part of this issue.
Two leads from #21 that I haven't filed as issues:
Say if you want either one opened as an issue. I'm closing #5.