OCR: make the local vision-model path (Ollama/LM Studio) reliable and self-serve #3
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip#3
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #2
Part of #2. Pragmatic path to reading joined-up/cursive writing locally: a local vision model (qwen2.5-VL, MiniCPM-V, etc.) served by Ollama/LM Studio is, in 2026, the only thing that reads arbitrary cursive well while keeping ink on the owner's machine. The engine already exists (
src/transport/ocr.ts,ocrMode: "endpoint"), but issue #2 reports it is "untested because of setup difficulties." So the work here is not a new engine — it is removing the setup friction that stops people from ever getting a first successful read.Why this, and the honest trade-off
The issue author prefers "a local running and/or integrated js lib solution over 3rd party toolings and api calls." Research for #2 (see the analysis comment on #2) found that no integrated JS lib robustly reads cursive today: the native
navigator.createHandwritingRecognizerHandwriting Recognition API is ChromeOS-only and rejects in Obsidian's Electron/Linux/macOS; MyScript iinkJS is cloud/commercial; transformer online-HTR exists only as research, not as a packaged on-device model. A local VLM over Ollama is "local running" (ink never leaves the machine) but is still an HTTP call to a local server — it satisfies the local-first intent while being the highest-accuracy cursive option available now. This issue is the pragmatic track; the integrated-JS-lib track is #C (the other child of #2).What to build
A fresh agent should investigate and split, but the target is:
src/settingschema.ts, visible whenocrMode === "endpoint"). It should: pingGET {endpoint}/models, confirm the configuredocrModelis present, send a tiny probe image throughtranscribe()(src/transport/ocr.ts), and show a clear pass/fail with the actual failure (connection refused, 404 route, model not pulled, CORS). The settings UI pattern lives insrc/settings*.ts/settingschema.ts— follow the existing control types; a newcontrol.typeor a button item may be needed (design decision for the implementer).transcribe()already maps 401/403/404 to messages; extend for the common local failures. The single biggest Ollama gotcha is CORS/host binding — a localhost Ollama rejects requests from Obsidian'sapp://obsidian.mdorigin unlessOLLAMA_ORIGINS/OLLAMA_HOSTis set. Detect and name this.docs/, linked from the README OCR section and the settings note):ollama pull qwen2.5vl:7b, settingOLLAMA_ORIGINS=app://obsidian.md(or*), LM Studio equivalent, and how to verify. This part alone is a clean low-risk MR and could land first.settingsmodel.ts: endpointhttp://localhost:11434/v1, modelqwen2.5vl:7b) — verify they are current/good.Evidence / pointers
src/main.ts(recognizeLocally, theocrMode === "endpoint"branch ~L800/L1327),src/inkedge.tsocr(...)(~L240).src/transport/ocr.ts.src/settingschema.tsL396-458,src/settingsmodel.tsL101-119.Scope boundary
NOT in scope: changing the built-in local readers (that is #A — calibration/diagnosability) and the integrated trainable reader R&D (#C). Do not touch WritePad here.
Validation contract (for the eventual MR)
nix develop -c npx vitest run src/__tests__→ PASS (no regressions; add a unit test for any new error-mapping intranscribe).qwen2.5vlpulled, the Test-connection action reports success; with Ollama stopped, it reports "connection refused"; withOLLAMA_ORIGINSunset, it reports the CORS/origin hint. Red gate: before the change, a misconfigured endpoint silently yields image-only with no guidance.🤖 Agent is working on this — turn 84
Last activity: assistant message — updated 13:26:33 UTC
Last message:
claude referenced this issue2026-10-20 15:13:52 +02:00
@claude cant we bundle the inference engine and model into the plugin so the user dosnt have to setup all llm stuff, but still give the endpoint option for advanced users?
What I found
Investigation of the endpoint path is done and the work is split into two ready MRs. Then @daniel asked the key direction question — can we bundle the engine + model so there's no LLM setup, keeping the endpoint for advanced users? — so I researched it properly before answering.
Answering @daniel's question honestly
Short version: we can bundle the engine, but not a model that reads cursive — and the small models that are embeddable can't read cursive, which is the whole point of #2. So "fully bundled, zero-setup, reads your joined-up writing" isn't shippable today. Here's the breakdown:
transformers.jsv4 (Sept 2026) runs vision models in-process via WebGPU with an automatic WASM fallback, and works in Obsidian's Electron environment. No existing Obsidian plugin does in-process VLM OCR — it'd be novel. (transformers.js guide, WebGPU in-browser 2026)So the trade-off is real: embeddable ⇒ can't do cursive; can do cursive ⇒ not embeddable. That's why #2's research landed on "local VLM over Ollama" rather than an in-plugin model.
Two realistic ways to still honour the "no setup" intent, and both already have a home:
OLLAMA_ORIGINS— keeping the manual endpoint for advanced users exactly as you asked. I filed this as #8 (unassigned — your call to greenlight). It's desktop-only and reuses #3's Test-connection plumbing.transformers.js+WebGPU reader is viable as an experimental, desktop, download-on-demand, neat-hand-first reader (not a cursive solution). That ambition belongs to #5 (the integrated trainable-reader track), not here.My recommendation: land #3 now (it hardens the path you want to keep for advanced users regardless), greenlight #8 for the "feels like zero-setup" win, and let #5 carry the in-process experiment with eyes open about the cursive limitation. Happy to go deeper on any of these.
Created
docs(ocr): bundled Ollama/LM Studio setup guide + README/settings links. Docs-only, lands first. (Part of #3)feat(ocr): Test-connection action in settings + actionable error mapping (connection-refused, CORS/OLLAMA_ORIGINS403, model-not-pulled, raw status passthrough) + unit tests. (Part of #3)Part of #2, answering @daniel's question.Follow-ups
main.ts:803doestranscribeStrokes(...).catch(() => null)— every endpoint failure is silently swallowed to image-only with no guidance. MR !7's Test-connection action is the surfacing path; defaults (http://localhost:11434/v1,qwen2.5vl:7b) verified current.@claude maybe with a custom model we train our own focused on word based ocr, kinda like the handbook llm ocr example you learn about neural networks with, just beefed up specialized for handwriting recognition? incorperating recent researches about continous training/learning of models?
Since my last update: @daniel asked about training our own word-based handwriting model ("the handbook OCR example, beefed up," with continuous learning). Short answer — yes, this is genuinely viable and it's the best-fit direction of all the OCR tracks, but it belongs to #5 (the integrated trainable-reader spike), not this endpoint issue. I've posted the full blueprint there; the gist:
This is a stronger fit than the embedded-VLM idea from your previous question, precisely because a model small enough to embed can be trained/adapted locally, whereas a cursive-capable VLM can't be embedded at all. Full analysis + sources are in #5.
No change to this issue's scope: #3 stays the endpoint-hardening track (MRs !6 and !7 stand). Nothing new created here — the trainable-model direction is routed to #5, which is where that work lives.
claude referenced this issue2026-10-20 15:26:31 +02:00
claude referenced this issue2026-10-20 15:26:32 +02:00
claude referenced this issue2026-10-20 16:33:01 +02:00
claude referenced this issue2026-10-20 16:34:09 +02:00
claude referenced this issue2026-10-20 16:34:09 +02:00
🤖 Agent is working on this — turn 18
Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 16:57:52 UTC
Last message:
claude referenced this issue2026-10-20 18:57:21 +02:00
claude referenced this issue2026-10-20 18:57:24 +02:00
Answering here in-thread (full analysis with sources is in my earlier comment).
Short answer: we can bundle the engine, but not a model that reads cursive.
transformers.js(WebGPU + WASM fallback) runs vision models in-process in Electron.So "bundled, zero-setup, reads joined-up writing" isn't shippable today. Two ways to keep the no-setup intent:
OLLAMA_ORIGINS, then verifies the setup with the Test-connection action. The manual endpoint stays as the advanced option, as you asked. #8 is now assigned and in progress.The endpoint path you want to keep for advanced users is in place: !6 (setup guide) and !7 (Test connection plus clear errors for connection refused, CORS/
OLLAMA_ORIGINS, and model not pulled) are both merged.Answering here in-thread (the full blueprint is on #5, and the summary is in this comment).
Yes. This is the OCR direction that fits your constraints best, and it lives in #5, not here.
Lexicon). It prefers your real spellings without being restricted to them.renderInk().renderInk()can generate an unlimited labelled corpus in your own hand, which removes the usual data bottleneck.Progress: Direction 1 of #5 (letter-level learning from corrections) has already landed as !12. The bigger trainable-model spike stays in #5, and nothing new needed to be created for it.
Since my last update: I've now answered both of @daniel's questions inside their own threads: bundling the engine and model (ic:43750) and training our own word-based model (ic:43841). The substance hasn't changed from my earlier comments. Everything #3 asked for has merged: !6 (Ollama/LM Studio setup guide plus links), !7 (Test-connection action plus actionable errors for connection refused, CORS/
OLLAMA_ORIGINS, model not pulled, and passthrough of other HTTP status codes), and the defaults are confirmed still current. The follow-on directions are tracked elsewhere: #8 (one-click managed Ollama, in progress) and #5 (trainable per-writer reader; direction 1 landed as !12). I'm closing #3 as done.claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:02:05 +02:00
claude referenced this issue2026-10-20 19:02:19 +02:00
claude referenced this issue2026-10-20 19:02:19 +02:00
claude referenced this issue2026-10-20 20:06:07 +02:00
claude referenced this issue2026-10-20 20:06:07 +02:00
claude referenced this issue2026-10-21 00:19:36 +02:00