OCR: make the local vision-model path (Ollama/LM Studio) reliable and self-serve #3

Closed
opened 2026-10-20 15:12:57 +02:00 by claude · 9 comments
Collaborator

Part of #2

Part of #2. Pragmatic path to reading joined-up/cursive writing locally: a local vision model (qwen2.5-VL, MiniCPM-V, etc.) served by Ollama/LM Studio is, in 2026, the only thing that reads arbitrary cursive well while keeping ink on the owner's machine. The engine already exists (src/transport/ocr.ts, ocrMode: "endpoint"), but issue #2 reports it is "untested because of setup difficulties." So the work here is not a new engine — it is removing the setup friction that stops people from ever getting a first successful read.

Why this, and the honest trade-off

The issue author prefers "a local running and/or integrated js lib solution over 3rd party toolings and api calls." Research for #2 (see the analysis comment on #2) found that no integrated JS lib robustly reads cursive today: the native navigator.createHandwritingRecognizer Handwriting Recognition API is ChromeOS-only and rejects in Obsidian's Electron/Linux/macOS; MyScript iinkJS is cloud/commercial; transformer online-HTR exists only as research, not as a packaged on-device model. A local VLM over Ollama is "local running" (ink never leaves the machine) but is still an HTTP call to a local server — it satisfies the local-first intent while being the highest-accuracy cursive option available now. This issue is the pragmatic track; the integrated-JS-lib track is #C (the other child of #2).

What to build

A fresh agent should investigate and split, but the target is:

  1. A "Test connection" action in the OCR settings (src/settingschema.ts, visible when ocrMode === "endpoint"). It should: ping GET {endpoint}/models, confirm the configured ocrModel is present, send a tiny probe image through transcribe() (src/transport/ocr.ts), and show a clear pass/fail with the actual failure (connection refused, 404 route, model not pulled, CORS). The settings UI pattern lives in src/settings*.ts/settingschema.ts — follow the existing control types; a new control.type or a button item may be needed (design decision for the implementer).
  2. Actionable error messages. transcribe() already maps 401/403/404 to messages; extend for the common local failures. The single biggest Ollama gotcha is CORS/host binding — a localhost Ollama rejects requests from Obsidian's app://obsidian.md origin unless OLLAMA_ORIGINS/OLLAMA_HOST is set. Detect and name this.
  3. A bundled setup guide (docs/, linked from the README OCR section and the settings note): ollama pull qwen2.5vl:7b, setting OLLAMA_ORIGINS=app://obsidian.md (or *), LM Studio equivalent, and how to verify. This part alone is a clean low-risk MR and could land first.
  4. Sensible defaults are already present (settingsmodel.ts: endpoint http://localhost:11434/v1, model qwen2.5vl:7b) — verify they are current/good.

Evidence / pointers

  • Reader pipeline + endpoint wiring: src/main.ts (recognizeLocally, the ocrMode === "endpoint" branch ~L800/L1327), src/inkedge.ts ocr(...) (~L240).
  • Request/response + prompt + worked-examples-from-corrections: src/transport/ocr.ts.
  • Settings: src/settingschema.ts L396-458, src/settingsmodel.ts L101-119.

Scope boundary

NOT in scope: changing the built-in local readers (that is #A — calibration/diagnosability) and the integrated trainable reader R&D (#C). Do not touch WritePad here.

Validation contract (for the eventual MR)

  • nix develop -c npx vitest run src/__tests__ → PASS (no regressions; add a unit test for any new error-mapping in transcribe).
  • Manual: with Ollama running locally and qwen2.5vl pulled, the Test-connection action reports success; with Ollama stopped, it reports "connection refused"; with OLLAMA_ORIGINS unset, it reports the CORS/origin hint. Red gate: before the change, a misconfigured endpoint silently yields image-only with no guidance.
Part of #2 Part of #2. Pragmatic path to reading joined-up/cursive writing **locally**: a local vision model (qwen2.5-VL, MiniCPM-V, etc.) served by Ollama/LM Studio is, in 2026, the only thing that reads arbitrary cursive well while keeping ink on the owner's machine. The engine already exists (`src/transport/ocr.ts`, `ocrMode: "endpoint"`), but issue #2 reports it is *"untested because of setup difficulties."* So the work here is **not** a new engine — it is removing the setup friction that stops people from ever getting a first successful read. ## Why this, and the honest trade-off The issue author prefers *"a local running and/or integrated js lib solution over 3rd party toolings and api calls."* Research for #2 (see the analysis comment on #2) found that **no integrated JS lib robustly reads cursive today**: the native `navigator.createHandwritingRecognizer` Handwriting Recognition API is ChromeOS-only and rejects in Obsidian's Electron/Linux/macOS; MyScript iinkJS is cloud/commercial; transformer online-HTR exists only as research, not as a packaged on-device model. A *local* VLM over Ollama is "local running" (ink never leaves the machine) but is still an HTTP call to a local server — it satisfies the local-first intent while being the highest-accuracy cursive option available now. This issue is the pragmatic track; the integrated-JS-lib track is #C (the other child of #2). ## What to build A fresh agent should investigate and split, but the target is: 1. **A "Test connection" action in the OCR settings** (`src/settingschema.ts`, visible when `ocrMode === "endpoint"`). It should: ping `GET {endpoint}/models`, confirm the configured `ocrModel` is present, send a tiny probe image through `transcribe()` (`src/transport/ocr.ts`), and show a clear pass/fail with the actual failure (connection refused, 404 route, model not pulled, CORS). The settings UI pattern lives in `src/settings*.ts`/`settingschema.ts` — follow the existing control types; a new `control.type` or a button item may be needed (design decision for the implementer). 2. **Actionable error messages.** `transcribe()` already maps 401/403/404 to messages; extend for the common local failures. The single biggest Ollama gotcha is CORS/host binding — a localhost Ollama rejects requests from Obsidian's `app://obsidian.md` origin unless `OLLAMA_ORIGINS`/`OLLAMA_HOST` is set. Detect and name this. 3. **A bundled setup guide** (`docs/`, linked from the README OCR section and the settings note): `ollama pull qwen2.5vl:7b`, setting `OLLAMA_ORIGINS=app://obsidian.md` (or `*`), LM Studio equivalent, and how to verify. This part alone is a clean low-risk MR and could land first. 4. **Sensible defaults** are already present (`settingsmodel.ts`: endpoint `http://localhost:11434/v1`, model `qwen2.5vl:7b`) — verify they are current/good. ## Evidence / pointers - Reader pipeline + endpoint wiring: `src/main.ts` (`recognizeLocally`, the `ocrMode === "endpoint"` branch ~L800/L1327), `src/inkedge.ts` `ocr(...)` (~L240). - Request/response + prompt + worked-examples-from-corrections: `src/transport/ocr.ts`. - Settings: `src/settingschema.ts` L396-458, `src/settingsmodel.ts` L101-119. ## Scope boundary NOT in scope: changing the built-in local readers (that is #A — calibration/diagnosability) and the integrated trainable reader R&D (#C). Do not touch WritePad here. ## Validation contract (for the eventual MR) - `nix develop -c npx vitest run src/__tests__` → PASS (no regressions; add a unit test for any new error-mapping in `transcribe`). - Manual: with Ollama running locally and `qwen2.5vl` pulled, the Test-connection action reports success; with Ollama stopped, it reports "connection refused"; with `OLLAMA_ORIGINS` unset, it reports the CORS/origin hint. Red gate: before the change, a misconfigured endpoint silently yields image-only with no guidance.
claude self-assigned this 2026-10-20 15:12:57 +02:00
Author
Collaborator

🤖 Agent is working on this — turn 84

Last activity: assistant message — updated 13:26:33 UTC

Last message:

Done. This wake was a direction question from @daniel about training a custom word-based handwriting model with continuous learning.

What I did

Routed it to the right home (#5) — this is the integrated trainable-reader track, not the endpoint-hardening work of #3. I researched the specifics and posted a concrete, cited blueprint as a comment on #5 so its agent inherits it:

  • The "handbook example, beefed up" = CRNN + CTC (CNN→BiLSTM→CTC), small enough to run and adapt locally, deployable in-browser (tfjs/onnxruntime-web), ~92% word accuracy with beam search.
  • Word-based decoding = CTC + Word-Beam-Search constrained to a dictionary — and the plugin already owns the dictionary (vault vocab + word sheet) and a per-writer dataset (handwriting sheet, corrections corpus, the existing LLaMA-Factory export), which is an unusually strong starting point.
  • Continuous learning framed honestly as writer adaptation (few-shot in-context personalization + continual learning with tiny replay memory to beat catastrophic forgetting), with the real engineering decision flagged: on-device training vs inference, and three concrete routes.

Answered @daniel on #3 with the honest verdict — this is the most fit-for-constraints OCR direction (stronger than the embedded-VLM idea from the prior question, since a model small enough to embed can be adapted locally while a cursive-capable VLM can't be embedded at all) — and pointed to #5 for the full analysis.

#3's scope stays intact: MRs !6 (docs) and !7 (Test-connection + error mapping) remain the deliverables here. No product code changed this turn, nothing new created on #3 (correctly routed to the existing #5) — a clean research/routing completion.

🤖 **Agent is working on this** — turn 84 _Last activity: assistant message — updated 13:26:33 UTC_ **Last message:** > Done. This wake was a direction question from @daniel about training a custom word-based handwriting model with continuous learning. > > ## What I did > > **Routed it to the right home (#5)** — this is the integrated trainable-reader track, not the endpoint-hardening work of #3. I researched the specifics and posted a concrete, cited blueprint as a comment on **#5** so its agent inherits it: > - The "handbook example, beefed up" = **CRNN + CTC** (CNN→BiLSTM→CTC), small enough to run *and* adapt locally, deployable in-browser (tfjs/onnxruntime-web), ~92% word accuracy with beam search. > - **Word-based** decoding = CTC + Word-Beam-Search constrained to a dictionary — and the plugin already owns the dictionary (vault vocab + word sheet) and a per-writer dataset (handwriting sheet, corrections corpus, the existing LLaMA-Factory export), which is an unusually strong starting point. > - **Continuous learning** framed honestly as *writer adaptation* (few-shot in-context personalization + continual learning with tiny replay memory to beat catastrophic forgetting), with the real engineering decision flagged: on-device *training* vs inference, and three concrete routes. > > **Answered @daniel on #3** with the honest verdict — this is the most fit-for-constraints OCR direction (stronger than the embedded-VLM idea from the prior question, since a model small enough to embed *can* be adapted locally while a cursive-capable VLM can't be embedded at all) — and pointed to #5 for the full analysis. > > **#3's scope stays intact**: MRs **!6** (docs) and **!7** (Test-connection + error mapping) remain the deliverables here. No product code changed this turn, nothing new created on #3 (correctly routed to the existing #5) — a clean research/routing completion.
Owner

@claude cant we bundle the inference engine and model into the plugin so the user dosnt have to setup all llm stuff, but still give the endpoint option for advanced users?

@claude cant we bundle the inference engine and model into the plugin so the user dosnt have to setup all llm stuff, but still give the endpoint option for advanced users?
Author
Collaborator

What I found

Investigation of the endpoint path is done and the work is split into two ready MRs. Then @daniel asked the key direction question — can we bundle the engine + model so there's no LLM setup, keeping the endpoint for advanced users? — so I researched it properly before answering.

Answering @daniel's question honestly

Short version: we can bundle the engine, but not a model that reads cursive — and the small models that are embeddable can't read cursive, which is the whole point of #2. So "fully bundled, zero-setup, reads your joined-up writing" isn't shippable today. Here's the breakdown:

  • Engine — yes, bundleable. transformers.js v4 (Sept 2026) runs vision models in-process via WebGPU with an automatic WASM fallback, and works in Obsidian's Electron environment. No existing Obsidian plugin does in-process VLM OCR — it'd be novel. (transformers.js guide, WebGPU in-browser 2026)
  • Model — not bundleable. Qwen2.5-VL-7B is ~5–6 GB even at 4-bit and wants ~32 GB system RAM; the download alone is ~15 GB unquantized. A plugin release can't ship that — it'd have to be downloaded on first use (multi-GB), and a 7B VLM in the renderer is fragile (WASM memory limits, inconsistent WebGPU, no mobile). (Qwen2.5-VL VRAM breakdown, Ollama RAM caveat)
  • The catch that kills the simple version: small models that would embed can't read cursive. On real handwriting, Qwen2.5-VL-3B scores ~24% word accuracy vs ~43% for the 7B (and ~93% for an 11B); sub-2B on-device VLMs like Moondream2 are explicitly "not great with handwriting." Cursive is exactly where the capacity cliff bites hardest — the smaller the model, the worse. (Manchu VLM-OCR study, on-device OCR models compared)

So the trade-off is real: embeddable ⇒ can't do cursive; can do cursive ⇒ not embeddable. That's why #2's research landed on "local VLM over Ollama" rather than an in-plugin model.

Two realistic ways to still honour the "no setup" intent, and both already have a home:

  1. Manage the engine for the user instead of bundling it — a one-click "set up local reading for me" that detects/installs Ollama, pulls the model (with visible progress), and sets OLLAMA_ORIGINS — keeping the manual endpoint for advanced users exactly as you asked. I filed this as #8 (unassigned — your call to greenlight). It's desktop-only and reuses #3's Test-connection plumbing.
  2. In-process reader as R&D — a transformers.js+WebGPU reader is viable as an experimental, desktop, download-on-demand, neat-hand-first reader (not a cursive solution). That ambition belongs to #5 (the integrated trainable-reader track), not here.

My recommendation: land #3 now (it hardens the path you want to keep for advanced users regardless), greenlight #8 for the "feels like zero-setup" win, and let #5 carry the in-process experiment with eyes open about the cursive limitation. Happy to go deeper on any of these.

Created

  • MR !6 — docs(ocr): bundled Ollama/LM Studio setup guide + README/settings links. Docs-only, lands first. (Part of #3)
  • MR !7 — feat(ocr): Test-connection action in settings + actionable error mapping (connection-refused, CORS/OLLAMA_ORIGINS 403, model-not-pulled, raw status passthrough) + unit tests. (Part of #3)
  • #8 — proposal (unassigned): one-click managed-Ollama setup, Part of #2, answering @daniel's question.

Follow-ups

  • @daniel to greenlight or close #8.
  • Red gate confirmed: today main.ts:803 does transcribeStrokes(...).catch(() => null) — every endpoint failure is silently swallowed to image-only with no guidance. MR !7's Test-connection action is the surfacing path; defaults (http://localhost:11434/v1, qwen2.5vl:7b) verified current.
  • Built-in reader (#4) and in-process reader R&D (#5) are out of scope here.
## What I found Investigation of the endpoint path is done and the work is split into two ready MRs. Then @daniel asked the key direction question — **can we bundle the engine + model so there's no LLM setup, keeping the endpoint for advanced users?** — so I researched it properly before answering. ### Answering @daniel's question honestly **Short version: we can bundle the *engine*, but not a model that reads cursive — and the small models that *are* embeddable can't read cursive, which is the whole point of #2.** So "fully bundled, zero-setup, reads your joined-up writing" isn't shippable today. Here's the breakdown: - **Engine — yes, bundleable.** `transformers.js` v4 (Sept 2026) runs vision models in-process via WebGPU with an automatic WASM fallback, and works in Obsidian's Electron environment. No existing Obsidian plugin does in-process VLM OCR — it'd be novel. ([transformers.js guide](https://www.developersdigest.tech/blog/transformers-js-guide), [WebGPU in-browser 2026](https://pinggy.io/blog/run_llm_in_browser_webgpu/)) - **Model — not bundleable.** Qwen2.5-VL-7B is ~5–6 GB even at 4-bit and wants ~32 GB system RAM; the download alone is ~15 GB unquantized. A plugin release can't ship that — it'd have to be *downloaded* on first use (multi-GB), and a 7B VLM in the renderer is fragile (WASM memory limits, inconsistent WebGPU, **no mobile**). ([Qwen2.5-VL VRAM breakdown](https://www.spheron.network/tools/gpu-recommender/Qwen/Qwen2.5-VL-7B-Instruct/), [Ollama RAM caveat](https://github.com/ollama/ollama/issues/14312)) - **The catch that kills the simple version: small models that *would* embed can't read cursive.** On *real handwriting*, Qwen2.5-VL-3B scores ~24% word accuracy vs ~43% for the 7B (and ~93% for an 11B); sub-2B on-device VLMs like Moondream2 are explicitly "not great with handwriting." Cursive is exactly where the capacity cliff bites hardest — the smaller the model, the worse. ([Manchu VLM-OCR study](https://arxiv.org/pdf/2507.06761), [on-device OCR models compared](https://modal.com/blog/8-top-open-source-ocr-models-compared)) So the trade-off is real: *embeddable ⇒ can't do cursive; can do cursive ⇒ not embeddable.* That's why #2's research landed on "local VLM over Ollama" rather than an in-plugin model. **Two realistic ways to still honour the "no setup" intent**, and both already have a home: 1. **Manage the engine for the user instead of bundling it** — a one-click "set up local reading for me" that detects/installs Ollama, pulls the model (with visible progress), and sets `OLLAMA_ORIGINS` — keeping the manual endpoint for advanced users exactly as you asked. I filed this as **#8** (unassigned — your call to greenlight). It's desktop-only and reuses #3's Test-connection plumbing. 2. **In-process reader as R&D** — a `transformers.js`+WebGPU reader is viable as an *experimental, desktop, download-on-demand, neat-hand-first* reader (not a cursive solution). That ambition belongs to **#5** (the integrated trainable-reader track), not here. My recommendation: land #3 now (it hardens the path you want to keep for advanced users regardless), greenlight **#8** for the "feels like zero-setup" win, and let **#5** carry the in-process experiment with eyes open about the cursive limitation. Happy to go deeper on any of these. ## Created - **MR !6** — `docs(ocr)`: bundled Ollama/LM Studio setup guide + README/settings links. Docs-only, lands first. (`Part of #3`) - **MR !7** — `feat(ocr)`: Test-connection action in settings + actionable error mapping (connection-refused, CORS/`OLLAMA_ORIGINS` 403, model-not-pulled, raw status passthrough) + unit tests. (`Part of #3`) - **#8** — proposal (unassigned): one-click managed-Ollama setup, `Part of #2`, answering @daniel's question. ## Follow-ups - @daniel to greenlight or close **#8**. - Red gate confirmed: today `main.ts:803` does `transcribeStrokes(...).catch(() => null)` — every endpoint failure is silently swallowed to image-only with no guidance. MR !7's Test-connection action is the surfacing path; defaults (`http://localhost:11434/v1`, `qwen2.5vl:7b`) verified current. - Built-in reader (#4) and in-process reader R&D (#5) are out of scope here.
Owner

@claude maybe with a custom model we train our own focused on word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?

@claude maybe with a custom model we train our own focused on word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?
Author
Collaborator

Since my last update: @daniel asked about training our own word-based handwriting model ("the handbook OCR example, beefed up," with continuous learning). Short answer — yes, this is genuinely viable and it's the best-fit direction of all the OCR tracks, but it belongs to #5 (the integrated trainable-reader spike), not this endpoint issue. I've posted the full blueprint there; the gist:

  • The "handbook example" is CRNN + CTC (CNN → BiLSTM → CTC loss) — small enough to run and adapt locally, deployable in-browser via tfjs/onnxruntime-web, ~92% word accuracy with beam-search decoding.
  • Word-based = CTC + Word-Beam-Search constrained to a dictionary — and this plugin already owns the dictionary (vault vocab + word sheet) plus a per-writer dataset (handwriting sheet, corrections corpus, the LLaMA-Factory export). That's an unusually strong starting point.
  • Continuous learning is realistic as writer adaptation (learn your hand from corrections), using few-shot in-context personalization and/or continual learning with tiny replay memory to avoid catastrophic forgetting. Honest caveat: the win is per-writer, not a universal cursive reader; and on-device training (vs inference) is the hard engineering call — three routes sketched in #5.

This is a stronger fit than the embedded-VLM idea from your previous question, precisely because a model small enough to embed can be trained/adapted locally, whereas a cursive-capable VLM can't be embedded at all. Full analysis + sources are in #5.

No change to this issue's scope: #3 stays the endpoint-hardening track (MRs !6 and !7 stand). Nothing new created here — the trainable-model direction is routed to #5, which is where that work lives.

Since my last update: @daniel asked about **training our own word-based handwriting model** ("the handbook OCR example, beefed up," with continuous learning). Short answer — **yes, this is genuinely viable and it's the best-fit direction of all the OCR tracks**, but it belongs to **#5** (the integrated trainable-reader spike), not this endpoint issue. I've posted the full blueprint there; the gist: - The "handbook example" is **CRNN + CTC** (CNN → BiLSTM → CTC loss) — small enough to run *and* adapt locally, deployable in-browser via tfjs/onnxruntime-web, ~92% word accuracy with beam-search decoding. - **Word-based** = CTC + **Word-Beam-Search constrained to a dictionary** — and this plugin already owns the dictionary (vault vocab + word sheet) plus a per-writer dataset (handwriting sheet, corrections corpus, the LLaMA-Factory export). That's an unusually strong starting point. - **Continuous learning** is realistic as **writer adaptation** (learn *your* hand from corrections), using few-shot in-context personalization and/or continual learning with tiny replay memory to avoid catastrophic forgetting. Honest caveat: the win is per-writer, not a universal cursive reader; and on-device *training* (vs inference) is the hard engineering call — three routes sketched in #5. This is a stronger fit than the embedded-VLM idea from your previous question, precisely because a model small enough to embed *can* be trained/adapted locally, whereas a cursive-capable VLM can't be embedded at all. Full analysis + sources are in **[#5](https://forge.n0de.biz/daniel/obsidian-remarkable-round-trip/issues/5)**. No change to this issue's scope: **#3** stays the endpoint-hardening track (MRs !6 and !7 stand). Nothing new created here — the trainable-model direction is routed to #5, which is where that work lives.
Author
Collaborator

🤖 Agent is working on this — turn 18

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 16:57:52 UTC

Last message:

Both of @daniel's threads now have in-thread replies. Next I'll post a short status note on #3 and close it, since everything it asked for has merged.

🤖 **Agent is working on this** — turn 18 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 16:57:52 UTC_ **Last message:** > Both of @daniel's threads now have in-thread replies. Next I'll post a short status note on #3 and close it, since everything it asked for has merged.
Author
Collaborator

Answering here in-thread (full analysis with sources is in my earlier comment).

Short answer: we can bundle the engine, but not a model that reads cursive.

  • Engine — yes. transformers.js (WebGPU + WASM fallback) runs vision models in-process in Electron.
  • Model — no. A cursive-capable VLM (Qwen2.5-VL-7B) is ~5–6 GB even at 4-bit and wants lots of RAM, so it can't ship in a plugin release; it would have to be downloaded on first use, and a 7B model inside the renderer is fragile and won't run on mobile.
  • The models small enough to embed can't read cursive. On real handwriting, Qwen2.5-VL-3B gets ~24% word accuracy against ~43% for the 7B. Sub-2B on-device VLMs are explicitly weak at handwriting.

So "bundled, zero-setup, reads joined-up writing" isn't shippable today. Two ways to keep the no-setup intent:

  1. #8: one-click managed local engine. A single button detects or installs Ollama, pulls the model with visible progress, sets OLLAMA_ORIGINS, then verifies the setup with the Test-connection action. The manual endpoint stays as the advanced option, as you asked. #8 is now assigned and in progress.
  2. #5: integrated reader R&D. Your follow-up below about training our own model belongs here.

The endpoint path you want to keep for advanced users is in place: !6 (setup guide) and !7 (Test connection plus clear errors for connection refused, CORS/OLLAMA_ORIGINS, and model not pulled) are both merged.

Answering here in-thread (full analysis with sources is in [my earlier comment](https://forge.n0de.biz/daniel/obsidian-remarkable-round-trip/issues/3#note_43793)). **Short answer: we can bundle the *engine*, but not a model that reads cursive.** - **Engine — yes.** `transformers.js` (WebGPU + WASM fallback) runs vision models in-process in Electron. - **Model — no.** A cursive-capable VLM (Qwen2.5-VL-7B) is ~5–6 GB even at 4-bit and wants lots of RAM, so it can't ship in a plugin release; it would have to be downloaded on first use, and a 7B model inside the renderer is fragile and won't run on mobile. - **The models small enough to embed can't read cursive.** On real handwriting, Qwen2.5-VL-3B gets ~24% word accuracy against ~43% for the 7B. Sub-2B on-device VLMs are explicitly weak at handwriting. So "bundled, zero-setup, reads joined-up writing" isn't shippable today. Two ways to keep the no-setup intent: 1. **#8: one-click managed local engine.** A single button detects or installs Ollama, pulls the model with visible progress, sets `OLLAMA_ORIGINS`, then verifies the setup with the Test-connection action. The manual endpoint stays as the advanced option, as you asked. #8 is now assigned and in progress. 2. **#5: integrated reader R&D.** Your follow-up below about training our own model belongs here. The endpoint path you want to keep for advanced users is in place: **!6** (setup guide) and **!7** (Test connection plus clear errors for connection refused, CORS/`OLLAMA_ORIGINS`, and model not pulled) are both merged.
Author
Collaborator

Answering here in-thread (the full blueprint is on #5, and the summary is in this comment).

Yes. This is the OCR direction that fits your constraints best, and it lives in #5, not here.

  • The "handbook OCR example, beefed up" is a small sequence model trained with CTC loss. Since we have the pen strokes, not just an image, #5 targets an online BiLSTM + CTC model that takes the pen trajectory directly. It doesn't need to cut cursive into letters first, and it isn't limited to a fixed vocabulary. That fixed vocabulary is exactly why a plain per-word classifier, the MNIST shape, would fail here.
  • Word-based: CTC beam search biased toward your vault vocabulary (Lexicon). It prefers your real spellings without being restricted to them.
  • Training data we already have: your handwriting sheet, the word sheet, your corrections, and renderInk(). renderInk() can generate an unlimited labelled corpus in your own hand, which removes the usual data bottleneck.
  • Continuous learning: learning your hand from your corrections. This uses replay plus a frozen base model with a small writer-specific adapter (Online-LoRA / EWC-style). Naive live updates would make the model forget what it already learned.
  • Honest caveats: the gain is per-writer, not a universal cursive reader. Training on-device versus training offline and running in-browser is the real engineering decision, and #5 gates it on a feasibility spike that has to beat the local-VLM baseline.

Progress: Direction 1 of #5 (letter-level learning from corrections) has already landed as !12. The bigger trainable-model spike stays in #5, and nothing new needed to be created for it.

Answering here in-thread (the full blueprint is on **#5**, and the summary is in [this comment](https://forge.n0de.biz/daniel/obsidian-remarkable-round-trip/issues/3#note_43888)). **Yes. This is the OCR direction that fits your constraints best, and it lives in #5, not here.** - **The "handbook OCR example, beefed up"** is a small sequence model trained with CTC loss. Since we have the pen strokes, not just an image, #5 targets an **online BiLSTM + CTC** model that takes the pen trajectory directly. It doesn't need to cut cursive into letters first, and it isn't limited to a fixed vocabulary. That fixed vocabulary is exactly why a plain per-word classifier, the MNIST shape, would fail here. - **Word-based:** CTC beam search biased toward your vault vocabulary (`Lexicon`). It prefers your real spellings without being restricted to them. - **Training data we already have:** your handwriting sheet, the word sheet, your corrections, and `renderInk()`. `renderInk()` can generate an unlimited labelled corpus *in your own hand*, which removes the usual data bottleneck. - **Continuous learning:** learning *your* hand from your corrections. This uses replay plus a frozen base model with a small writer-specific adapter (Online-LoRA / EWC-style). Naive live updates would make the model forget what it already learned. - **Honest caveats:** the gain is per-writer, not a universal cursive reader. Training on-device versus training offline and running in-browser is the real engineering decision, and #5 gates it on a feasibility spike that has to beat the local-VLM baseline. Progress: Direction 1 of #5 (letter-level learning from corrections) has already landed as **!12**. The bigger trainable-model spike stays in #5, and nothing new needed to be created for it.
Author
Collaborator

Since my last update: I've now answered both of @daniel's questions inside their own threads: bundling the engine and model (ic:43750) and training our own word-based model (ic:43841). The substance hasn't changed from my earlier comments. Everything #3 asked for has merged: !6 (Ollama/LM Studio setup guide plus links), !7 (Test-connection action plus actionable errors for connection refused, CORS/OLLAMA_ORIGINS, model not pulled, and passthrough of other HTTP status codes), and the defaults are confirmed still current. The follow-on directions are tracked elsewhere: #8 (one-click managed Ollama, in progress) and #5 (trainable per-writer reader; direction 1 landed as !12). I'm closing #3 as done.

Since my last update: I've now answered both of @daniel's questions inside their own threads: bundling the engine and model (ic:43750) and training our own word-based model (ic:43841). The substance hasn't changed from my earlier comments. Everything #3 asked for has merged: **!6** (Ollama/LM Studio setup guide plus links), **!7** (Test-connection action plus actionable errors for connection refused, CORS/`OLLAMA_ORIGINS`, model not pulled, and passthrough of other HTTP status codes), and the defaults are confirmed still current. The follow-on directions are tracked elsewhere: **#8** (one-click managed Ollama, in progress) and **#5** (trainable per-writer reader; direction 1 landed as !12). I'm closing #3 as done.
claude 2026-10-20 18:57:52 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip#3
No description provided.