OCR: one-click "managed local engine" — detect/install Ollama, pull the model, set the origin (desktop) #8

Closed
opened 2026-10-20 15:20:24 +02:00 by claude · 8 comments
Collaborator

Part of #2. Proposed by @daniel on #3: "can't we bundle the inference engine and model into the plugin so the user doesn't have to set up all the LLM stuff, but still give the endpoint option for advanced users?"

Unassigned on purpose — this is a proposal for @daniel to greenlight or close. It is the realistic near-term embodiment of that request (see the full analysis on #3): we cannot bundle a cursive-capable model (it is multi-GB and must be downloaded; a model small enough to embed does not read cursive), but we can collapse "set up all the LLM stuff" into one button by managing a local engine for the user while keeping the manual endpoint option for advanced users.

Sketch (desktop only)

A "Set up local reading for me" button in the OCR settings that, on desktop (Electron / Node child_process; not available on mobile):

  1. Detects whether Ollama is installed and running (probe http://localhost:11434). If not installed, link to the installer (or offer to run it).
  2. Runs ollama pull qwen2.5vl:7b with visible progress (it is a ~5 GB download — must be explicit, never silent).
  3. Sets/advises OLLAMA_ORIGINS=app://obsidian.md so Obsidian's origin is accepted, and restarts/needs-restart guidance.
  4. Fills the endpoint + model settings and runs the Test-connection action (built in #3) to confirm.

Why not "bundle" instead

  • Engine: bundleable in principle — transformers.js v4 runs vision models in-process via WebGPU with a WASM fallback and works in Electron. But a 7B VLM in the renderer is fragile (WASM address-space limits, inconsistent WebGPU, no mobile).
  • Model: not bundleable — Qwen2.5-VL-7B is ~5–6 GB even at Q4 and wants ~32 GB system RAM; it must be downloaded, not shipped in a plugin release.
  • Small models don't solve cursive — the whole point of #2. Qwen2.5-VL-3B scores ~24% word accuracy on real handwriting vs ~43% for 7B; sub-2B on-device VLMs (Moondream2 etc.) are explicitly weak on handwriting. So an embeddable model fails the actual task.
  • The genuine in-process / integrated reader R&D (desktop-WebGPU, on-demand download, neat-hand/printed first) already has a home in #5 — this issue is deliberately the lighter "manage Ollama for the user" path, not that R&D.

Open questions for the implementer

  • Is shelling out to a local ollama binary acceptable in a community plugin (desktop-only capability, no mobile)? Obsidian discourages but does not forbid child_process.
  • Install UX if Ollama is absent: link-only, or offer to download/run the installer per-OS?
  • LM Studio has no CLI pull in the same way — likely detect-and-guide only.

Scope boundary

Depends on #3's Test-connection action and error mapping landing first (reuse them to verify). Does not touch #4 (built-in reader) or #5 (in-process reader R&D).

Part of #2. Proposed by @daniel on #3: *"can't we bundle the inference engine and model into the plugin so the user doesn't have to set up all the LLM stuff, but still give the endpoint option for advanced users?"* **Unassigned on purpose — this is a proposal for @daniel to greenlight or close.** It is the realistic near-term embodiment of that request (see the full analysis on #3): we cannot *bundle* a cursive-capable model (it is multi-GB and must be downloaded; a model small enough to embed does not read cursive), but we **can** collapse "set up all the LLM stuff" into one button by *managing* a local engine for the user while keeping the manual endpoint option for advanced users. ## Sketch (desktop only) A "Set up local reading for me" button in the OCR settings that, on desktop (Electron / Node child_process; **not** available on mobile): 1. Detects whether Ollama is installed and running (probe `http://localhost:11434`). If not installed, link to the installer (or offer to run it). 2. Runs `ollama pull qwen2.5vl:7b` with visible progress (it is a ~5 GB download — must be explicit, never silent). 3. Sets/advises `OLLAMA_ORIGINS=app://obsidian.md` so Obsidian's origin is accepted, and restarts/needs-restart guidance. 4. Fills the endpoint + model settings and runs the Test-connection action (built in #3) to confirm. ## Why not "bundle" instead - **Engine:** bundleable in principle — `transformers.js` v4 runs vision models in-process via WebGPU with a WASM fallback and works in Electron. But a 7B VLM in the renderer is fragile (WASM address-space limits, inconsistent WebGPU, no mobile). - **Model:** not bundleable — Qwen2.5-VL-7B is ~5–6 GB even at Q4 and wants ~32 GB system RAM; it must be downloaded, not shipped in a plugin release. - **Small models don't solve cursive** — the whole point of #2. Qwen2.5-VL-3B scores ~24% word accuracy on real handwriting vs ~43% for 7B; sub-2B on-device VLMs (Moondream2 etc.) are explicitly weak on handwriting. So an *embeddable* model fails the actual task. - The genuine in-process / integrated reader R&D (desktop-WebGPU, on-demand download, neat-hand/printed first) already has a home in **#5** — this issue is deliberately the lighter "manage Ollama for the user" path, not that R&D. ## Open questions for the implementer - Is shelling out to a local `ollama` binary acceptable in a community plugin (desktop-only capability, no mobile)? Obsidian discourages but does not forbid `child_process`. - Install UX if Ollama is absent: link-only, or offer to download/run the installer per-OS? - LM Studio has no CLI pull in the same way — likely detect-and-guide only. ## Scope boundary Depends on #3's Test-connection action and error mapping landing first (reuse them to verify). Does not touch #4 (built-in reader) or #5 (in-process reader R&D).
Owner

calling ollama is acceptable, but direct download would be preferred, model download at runtime is also fine

calling ollama is acceptable, but direct download would be preferred, model download at runtime is also fine
Author
Collaborator

❌ Agent run failed

❌ **Agent run failed**
Author
Collaborator

Agent failed ❌

exceeded pending task timeout

**Agent failed** :x: exceeded pending task timeout
Author
Collaborator

🤖 Agent is working on this — turn 13

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 04:38:51 UTC

🤖 **Agent is working on this** — turn 13 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 04:38:51 UTC_
Author
Collaborator

What I found

  • The prerequisites from #3 have landed: !6 (setup docs) and !7 (Test connection plus the 403 → OLLAMA_ORIGINS error mapping in src/transport/ocr.ts). Nothing blocks this work.

  • Following @daniel's note ("direct download preferred, runtime model download fine"), the plugin will not shell out to ollama. Ollama's HTTP API already provides what the CLI would:

    • /api/version detects whether Ollama is running;
    • /api/tags lists the models already downloaded;
    • /api/pull downloads a model and streams NDJSON progress.

    This avoids require("child_process"), which scripts/check-bundle.mjs rejects to keep the mobile build working.

  • Obsidian's requestUrl (obsidianHttp, src/main.ts:2366) buffers the whole response, so it can't show progress. For the download, a desktop-only reader built on Node http and loaded lazily via window.require streams the progress instead. It sends no app://obsidian.md origin header, so the download works even before OLLAMA_ORIGINS is set. Mobile hides the button.

Created

  • !27: "Set up local reading for me" (v1). It:

    • adds a new pure module src/transport/ollama.ts with tests;
    • adds the desktop streaming adapter;
    • shows a ~6 GB download confirmation, then a progress modal with a Cancel button;
    • fills the endpoint and model, switches OCR to the endpoint, and runs Test connection;
    • updates the docs.

    Its validation contract has a failing check today, vitest ollama.test.ts, plus npm test, npm run build (which runs the bundle check) and lint.

  • #26: tracking issue that !27 closes. It is not dispatched separately; it exists so that #28 can wait on it.

  • #28: v2 research and split, covering:

    • downloading the Ollama installer directly and opening it on each OS;
    • checking whether OLLAMA_ORIGINS is actually needed for requestUrl, and if so whether to automate it or guide the user;
    • LM Studio detect-and-guide.

    #28 is linked as blocked by #26.

Follow-ups

  • v1 deliberately limits the missing-Ollama case to a link to ollama.com/download. Downloading and opening the installer directly is the point you said you'd prefer; that is #28.
  • If #28 concludes that child_process is unavoidable, for example to set OLLAMA_ORIGINS automatically, it must get your explicit sign-off before the bundle guard is loosened.
  • If this GitLab instance is on the Free tier, the blocked-by link degrades to a plain related link, and #28 won't start automatically when !27 merges. It would then need a nudge.
## What I found - The prerequisites from #3 have landed: !6 (setup docs) and !7 (Test connection plus the 403 → `OLLAMA_ORIGINS` error mapping in `src/transport/ocr.ts`). Nothing blocks this work. - Following @daniel's note ("direct download preferred, runtime model download fine"), the plugin will **not shell out to `ollama`**. Ollama's HTTP API already provides what the CLI would: - `/api/version` detects whether Ollama is running; - `/api/tags` lists the models already downloaded; - `/api/pull` downloads a model and streams NDJSON progress. This avoids `require("child_process")`, which `scripts/check-bundle.mjs` rejects to keep the mobile build working. - Obsidian's `requestUrl` (`obsidianHttp`, `src/main.ts:2366`) buffers the whole response, so it can't show progress. For the download, a desktop-only reader built on Node `http` and loaded lazily via `window.require` streams the progress instead. It sends no `app://obsidian.md` origin header, so the download works even before `OLLAMA_ORIGINS` is set. Mobile hides the button. ## Created - !27: "Set up local reading for me" (v1). It: - adds a new pure module `src/transport/ollama.ts` with tests; - adds the desktop streaming adapter; - shows a ~6 GB download confirmation, then a progress modal with a Cancel button; - fills the endpoint and model, switches OCR to the endpoint, and runs Test connection; - updates the docs. Its validation contract has a failing check today, `vitest ollama.test.ts`, plus `npm test`, `npm run build` (which runs the bundle check) and lint. - #26: tracking issue that !27 closes. It is not dispatched separately; it exists so that #28 can wait on it. - #28: v2 research and split, covering: - downloading the Ollama installer directly and opening it on each OS; - checking whether `OLLAMA_ORIGINS` is actually needed for `requestUrl`, and if so whether to automate it or guide the user; - LM Studio detect-and-guide. #28 is linked as blocked by #26. ## Follow-ups - v1 deliberately limits the missing-Ollama case to a **link** to ollama.com/download. Downloading and opening the installer directly is the point you said you'd prefer; that is #28. - If #28 concludes that `child_process` is unavoidable, for example to set `OLLAMA_ORIGINS` automatically, it must get your explicit sign-off before the bundle guard is loosened. - If this GitLab instance is on the Free tier, the blocked-by link degrades to a plain related link, and #28 won't start automatically when !27 merges. It would then need a nudge.
Author
Collaborator

Since my last update: !27 has merged and closed #26, so version 1 is now on main (7a021f8). It adds:

  • the "Set up local reading for me" settings button, in src/settings.ts;
  • the setup flow and its progress window, in src/ollamasetup.ts;
  • the Ollama connection code with tests, in src/transport/ollama.ts;
  • updated docs/ocr-endpoint-setup.md and README.

Closing #26 released the blocked-by link, and #28 (version 2) has already been picked up by an agent. It covers downloading the Ollama installer directly, which you said you prefer, plus OLLAMA_ORIGINS and LM Studio. So my Free-tier caveat doesn't apply here and no nudge is needed. I'm leaving #8 open until #28 and the work it splits off are done. Nothing new is needed from you, except sign-off if #28 concludes that child_process is unavoidable.

Since my last update: !27 has merged and closed #26, so version 1 is now on `main` (7a021f8). It adds: - the "Set up local reading for me" settings button, in `src/settings.ts`; - the setup flow and its progress window, in `src/ollamasetup.ts`; - the Ollama connection code with tests, in `src/transport/ollama.ts`; - updated `docs/ocr-endpoint-setup.md` and README. Closing #26 released the blocked-by link, and #28 (version 2) has already been picked up by an agent. It covers downloading the Ollama installer directly, which you said you prefer, plus `OLLAMA_ORIGINS` and LM Studio. So my Free-tier caveat doesn't apply here and no nudge is needed. I'm leaving #8 open until #28 and the work it splits off are done. Nothing new is needed from you, except sign-off if #28 concludes that `child_process` is unavoidable.
Owner

@claude all done here?

@claude all done here?
Author
Collaborator

@daniel Yes, all done here. Since my last update, #28 has closed and its three MRs are merged on main:

Item from the sketch Status
1. Detect Ollama, install it if missing Done. Ollama is detected over HTTP (v1, !27). On macOS and Windows, setup now downloads the official signed installer, checks its SHA-256 and opens it (!33). On Linux it shows a copyable install.sh command.
2. Pull the model with visible progress Done (v1, !27)
3. OLLAMA_ORIGINS Not needed. Tests showed requestUrl sends no Origin header, and Ollama has allowed app://* since v0.1.42. The docs and the 403 hint were reworded accordingly (!31). No child_process anywhere; the bundle guard is unchanged.
4. Fill the settings and run Test connection Done (!27). A bug where Test connection sent a corrupt probe PNG was later fixed in !38.
Bonus: LM Studio When Ollama isn't running, setup detects LM Studio and offers its downloaded vision models (!32).

Two things are left for you; neither needs an agent:

  • Manual check: CI can't run the "download installer, then open it" path in !33. Please try it once on macOS and once on Windows.
  • Plugin review risk: Obsidian's developer policy says plugins shouldn't "install their dependencies". Merging !33 suggests you accept the risk that a reviewer objects to the user-triggered download of the signed installer. If one does, the fallback is v1's plain download link.

I'm closing #8. I haven't filed the optional idea of LM Studio downloading a vision model automatically (POST /api/v1/models/download). Say the word if you want it tracked.

@daniel Yes, all done here. Since my last update, #28 has closed and its three MRs are merged on `main`: | Item from the sketch | Status | |---|---| | 1. Detect Ollama, install it if missing | Done. Ollama is detected over HTTP (v1, !27). On macOS and Windows, setup now downloads the official signed installer, checks its SHA-256 and opens it (!33). On Linux it shows a copyable `install.sh` command. | | 2. Pull the model with visible progress | Done (v1, !27) | | 3. `OLLAMA_ORIGINS` | Not needed. Tests showed `requestUrl` sends no `Origin` header, and Ollama has allowed `app://*` since v0.1.42. The docs and the 403 hint were reworded accordingly (!31). No `child_process` anywhere; the bundle guard is unchanged. | | 4. Fill the settings and run Test connection | Done (!27). A bug where Test connection sent a corrupt probe PNG was later fixed in !38. | | Bonus: LM Studio | When Ollama isn't running, setup detects LM Studio and offers its downloaded vision models (!32). | Two things are left for you; neither needs an agent: - **Manual check:** CI can't run the "download installer, then open it" path in !33. Please try it once on macOS and once on Windows. - **Plugin review risk:** Obsidian's developer policy says plugins shouldn't "install their dependencies". Merging !33 suggests you accept the risk that a reviewer objects to the user-triggered download of the signed installer. If one does, the fallback is v1's plain download link. I'm closing #8. I haven't filed the optional idea of LM Studio downloading a vision model automatically (`POST /api/v1/models/download`). Say the word if you want it tracked.
claude 2026-10-21 06:38:51 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
daniel/obsidian-remarkable-round-trip#8
No description provided.