feat(ocr): LM Studio detect-and-guide in "Set up local reading for me" (desktop) #32

Merged
daniel merged 3 commits from agent/lmstudio-detect-guide into main 2026-10-21 00:38:34 +02:00
Collaborator

Part of #28 (item 3). When Ollama is not running but LM Studio is, offer to point the OCR endpoint at LM Studio, pick a vision model it already has, and run Test connection. This MR adds no download or pull of LM Studio models.

Why / context

v1 (src/ollamasetup.ts, merged in !27) handles Ollama only. When Ollama does not answer, setUpLocalOllama() goes straight to OllamaMissingModal, even if the user already runs LM Studio's server on http://localhost:1234.

LM Studio needs no origin allowlist. requestUrl sends no Origin header (see !31), so pointing the endpoint at it is enough.

LM Studio's APIs (researched on #28, from lmstudio.ai/docs/developer/rest)

Try these in order, falling through on 404 or an unparseable body:

  1. GET /api/v1/models (LM Studio 0.4+) → { models: [{ type: "llm"|"embedding"|"decision", key, display_name, capabilities?: { vision: boolean, … }, loaded_instances: [{ id, … }], … }] }.
    • vision = capabilities?.vision === true, loaded = loaded_instances.length > 0, id = key.
    • Keep only type === "llm".
  2. GET /api/v0/models (0.3.6+) → { data: [{ id, type: "llm"|"vlm"|"embeddings", state: "loaded"|"not-loaded", … }] }.
    • vision = type === "vlm", loaded = state === "loaded". Drop embeddings.
  3. GET /v1/models (OpenAI-compatible, any version) → { data: [{ id }] }.
    • vision = null and loaded = null (unknown).

If the server has Require API token on, it answers 401. Report that as running-but-needs-a-key, not as "not running".

What to change

  1. New src/transport/lmstudio.ts. Pure over an injected HttpClient, in the same style as src/transport/ollama.ts:
    • export const LMSTUDIO_LOCAL = "http://localhost:1234";
    • export interface LmStudioModel { id: string; vision: boolean | null; loaded: boolean | null }
    • export type LmStudioStatus = { running: true; models: LmStudioModel[]; needsApiKey: boolean } | { running: false; reason: string }
    • export async function detectLmStudio(http, base): Promise<LmStudioStatus>. It never throws:
      • a thrown http() on the first call (connection refused) → running: false, with a reason worded like unreachable() in ollama.ts;
      • a 401 → { running: true, models: [], needsApiKey: true };
      • otherwise, the fall-through above.
    • export function pickVisionModel(models): LmStudioModel | null. Prefer a loaded vision model, then any vision model, else null. Never pick a model with vision === false.
  2. New src/lmstudiosetup.ts. The dialogue, following OllamaMissingModal / OllamaPullModal in src/ollamasetup.ts (a Modal with a choice() promise and modal-button-container buttons). Title: "LM Studio is running".
    • Vision models found: a DropdownComponent lists them, with loaded ones first and marked "(loaded)"; the default is pickVisionModel(). CTA "Use LM Studio" resolves { kind: "use", model }.
    • Only unknown-vision models (the /v1/models fallback): list them all, with a note: "pick a vision model (e.g. Qwen2.5-VL); text-only models cannot read handwriting".
    • No usable models: guidance text, no dropdown. "In LM Studio, open Discover, search for Qwen2.5-VL-7B-Instruct (a vision model), download and load it, and make sure the server is running in the Developer tab. Then press Check again."
    • needsApiKey: explain that LM Studio requires an API token, and that the user should paste it into API key and set the endpoint to http://localhost:1234/v1 by hand.
    • Always offer "Check again" → { kind: "retry" }, "Use Ollama instead" → { kind: "ollama" }, and "Close" → { kind: "close" }.
  3. src/ollamasetup.ts:
    • Extract the tail of setUpLocalOllama() (from const settings = host.settings(); through the test notifies) into an exported helper, e.g. applyAndTest(host, endpoint, model, refresh). v1 then calls it with LOCAL_ENDPOINT, model; behaviour and wording stay unchanged.
    • In the if (!found.running) branch, probe detectLmStudio(host.http, LMSTUDIO_LOCAL) before OllamaMissingModal. If it is running, show the LM Studio modal:
      • use → applyAndTest(host, \${LMSTUDIO_LOCAL}/v1`, model, refresh)`;
      • retry → recurse into setUpLocalOllama;
      • ollama → fall through to the existing OllamaMissingModal;
      • close → return.
    • If LM Studio is not running, v1 behaviour is unchanged.
    • When Ollama is running, nothing changes: Ollama stays preferred.
  4. src/settings.ts ocrActions(): in the "Set up local reading for me" description, mention that it also finds LM Studio.
  5. Docs: docs/ocr-endpoint-setup.md → One-click setup (desktop): one bullet saying that if Ollama is not running but LM Studio's server is, it offers LM Studio and the vision models already downloaded there (no download). CHANGELOG-INK.md → Unreleased: one bullet.

Constraints and decisions

  • No model download or load through LM Studio's API. POST /api/v1/models/download and /load exist, but #28 scoped this as detect-and-guide. Leave a follow-up note in the MR if you think it is worth it.
  • LM Studio's /v1/chat/completions accepts the model key (JIT-loads it when JIT loading is on), so store id/key as ocrModel.
  • Everything goes through host.http (requestUrl). No Node and no window.require, so the change stays mobile-safe. The action is still desktop-only through the existing visible: () => Platform.isDesktopApp.
  • Do not change v1's pull/progress flow (OllamaPullModal, pullModel) or transport/ollama.ts.
  • Out of scope: the installer download (sibling MR on agent/ollama-installer-download, which also edits OllamaMissingModal and the not-running branch) and the OLLAMA_ORIGINS docs (!31). Merge origin/main before finishing and resolve any overlap in src/ollamasetup.ts and the docs.

Validation contract

  • Red gate: write src/__tests__/lmstudio.test.ts first. Use a route-keyed fake HttpClient, like fixed() in src/__tests__/ollama.test.ts but answering per URL. Before src/transport/lmstudio.ts exists, npx vitest run src/__tests__/lmstudio.test.ts FAILS.
  • The test file covers:
    • v1 API with a loaded vision model, an unloaded vision model, a text model and an embedding (embedding dropped, vision/loaded flags right);
    • v1 404 → v0 with vlm/llm/embeddings;
    • v1 and v0 404 → /v1/models ids with vision: null;
    • http throws new Error("net::ERR_CONNECTION_REFUSED") → running: false;
    • 401 → needsApiKey: true;
    • pickVisionModel ordering, and null when only vision: false models exist.
  • After: npm ci && npx vitest run src/__tests__/lmstudio.test.ts → PASS. Then npx vitest run → whole suite PASS, npm run build → exit 0 with check-bundle's "bundle clean" line, and npm run lint → exit 0.
  • (No flake.nix in this repo; use plain npm/npx.)

🤖 Generated with Claude Code

Part of #28 (item 3). When Ollama is not running but LM Studio is, offer to point the OCR endpoint at LM Studio, pick a vision model it already has, and run Test connection. This MR adds no download or pull of LM Studio models. ## Why / context v1 (`src/ollamasetup.ts`, merged in !27) handles Ollama only. When Ollama does not answer, `setUpLocalOllama()` goes straight to `OllamaMissingModal`, even if the user already runs LM Studio's server on `http://localhost:1234`. LM Studio needs no origin allowlist. `requestUrl` sends no `Origin` header (see !31), so pointing the endpoint at it is enough. ## LM Studio's APIs (researched on #28, from lmstudio.ai/docs/developer/rest) Try these in order, falling through on 404 or an unparseable body: 1. **`GET /api/v1/models`** (LM Studio 0.4+) → `{ models: [{ type: "llm"|"embedding"|"decision", key, display_name, capabilities?: { vision: boolean, … }, loaded_instances: [{ id, … }], … }] }`. - `vision = capabilities?.vision === true`, `loaded = loaded_instances.length > 0`, `id = key`. - Keep only `type === "llm"`. 2. **`GET /api/v0/models`** (0.3.6+) → `{ data: [{ id, type: "llm"|"vlm"|"embeddings", state: "loaded"|"not-loaded", … }] }`. - `vision = type === "vlm"`, `loaded = state === "loaded"`. Drop `embeddings`. 3. **`GET /v1/models`** (OpenAI-compatible, any version) → `{ data: [{ id }] }`. - `vision = null` and `loaded = null` (unknown). If the server has *Require API token* on, it answers **401**. Report that as running-but-needs-a-key, not as "not running". ## What to change 1. **New `src/transport/lmstudio.ts`.** Pure over an injected `HttpClient`, in the same style as `src/transport/ollama.ts`: - `export const LMSTUDIO_LOCAL = "http://localhost:1234";` - `export interface LmStudioModel { id: string; vision: boolean | null; loaded: boolean | null }` - `export type LmStudioStatus = { running: true; models: LmStudioModel[]; needsApiKey: boolean } | { running: false; reason: string }` - `export async function detectLmStudio(http, base): Promise<LmStudioStatus>`. It **never throws**: - a thrown `http()` on the first call (connection refused) → `running: false`, with a reason worded like `unreachable()` in `ollama.ts`; - a 401 → `{ running: true, models: [], needsApiKey: true }`; - otherwise, the fall-through above. - `export function pickVisionModel(models): LmStudioModel | null`. Prefer a loaded vision model, then any vision model, else null. Never pick a model with `vision === false`. 2. **New `src/lmstudiosetup.ts`.** The dialogue, following `OllamaMissingModal` / `OllamaPullModal` in `src/ollamasetup.ts` (a `Modal` with a `choice()` promise and `modal-button-container` buttons). Title: "LM Studio is running". - **Vision models found:** a `DropdownComponent` lists them, with loaded ones first and marked "(loaded)"; the default is `pickVisionModel()`. CTA **"Use LM Studio"** resolves `{ kind: "use", model }`. - **Only unknown-vision models** (the `/v1/models` fallback): list them all, with a note: "pick a vision model (e.g. Qwen2.5-VL); text-only models cannot read handwriting". - **No usable models:** guidance text, no dropdown. "In LM Studio, open *Discover*, search for **Qwen2.5-VL-7B-Instruct** (a vision model), download and load it, and make sure the server is running in the *Developer* tab. Then press Check again." - **`needsApiKey`:** explain that LM Studio requires an API token, and that the user should paste it into *API key* and set the endpoint to `http://localhost:1234/v1` by hand. - Always offer **"Check again"** → `{ kind: "retry" }`, **"Use Ollama instead"** → `{ kind: "ollama" }`, and **"Close"** → `{ kind: "close" }`. 3. **`src/ollamasetup.ts`**: - Extract the tail of `setUpLocalOllama()` (from `const settings = host.settings();` through the test notifies) into an exported helper, e.g. `applyAndTest(host, endpoint, model, refresh)`. v1 then calls it with `LOCAL_ENDPOINT, model`; behaviour and wording stay unchanged. - In the `if (!found.running)` branch, probe `detectLmStudio(host.http, LMSTUDIO_LOCAL)` **before** `OllamaMissingModal`. If it is running, show the LM Studio modal: - `use` → `applyAndTest(host, \`${LMSTUDIO_LOCAL}/v1\`, model, refresh)`; - `retry` → recurse into `setUpLocalOllama`; - `ollama` → fall through to the existing `OllamaMissingModal`; - `close` → return. - If LM Studio is not running, v1 behaviour is unchanged. - When Ollama **is** running, nothing changes: Ollama stays preferred. 4. **`src/settings.ts`** `ocrActions()`: in the "Set up local reading for me" description, mention that it also finds LM Studio. 5. **Docs:** `docs/ocr-endpoint-setup.md` → *One-click setup (desktop)*: one bullet saying that if Ollama is not running but LM Studio's server is, it offers LM Studio and the vision models already downloaded there (no download). `CHANGELOG-INK.md` → *Unreleased*: one bullet. ## Constraints and decisions - **No model download or load through LM Studio's API.** `POST /api/v1/models/download` and `/load` exist, but #28 scoped this as detect-and-guide. Leave a follow-up note in the MR if you think it is worth it. - LM Studio's `/v1/chat/completions` accepts the model `key` (JIT-loads it when *JIT loading* is on), so store `id`/`key` as `ocrModel`. - Everything goes through `host.http` (`requestUrl`). No Node and no `window.require`, so the change stays mobile-safe. The action is still desktop-only through the existing `visible: () => Platform.isDesktopApp`. - Do not change v1's pull/progress flow (`OllamaPullModal`, `pullModel`) or `transport/ollama.ts`. - Out of scope: the installer download (sibling MR on `agent/ollama-installer-download`, which also edits `OllamaMissingModal` and the not-running branch) and the `OLLAMA_ORIGINS` docs (!31). Merge `origin/main` before finishing and resolve any overlap in `src/ollamasetup.ts` and the docs. ## Validation contract - **Red gate:** write `src/__tests__/lmstudio.test.ts` first. Use a route-keyed fake `HttpClient`, like `fixed()` in `src/__tests__/ollama.test.ts` but answering per URL. Before `src/transport/lmstudio.ts` exists, `npx vitest run src/__tests__/lmstudio.test.ts` FAILS. - The test file covers: - v1 API with a loaded vision model, an unloaded vision model, a text model and an embedding (embedding dropped, vision/loaded flags right); - v1 404 → v0 with `vlm`/`llm`/`embeddings`; - v1 and v0 404 → `/v1/models` ids with `vision: null`; - `http` throws `new Error("net::ERR_CONNECTION_REFUSED")` → `running: false`; - 401 → `needsApiKey: true`; - `pickVisionModel` ordering, and null when only `vision: false` models exist. - After: `npm ci && npx vitest run src/__tests__/lmstudio.test.ts` → PASS. Then `npx vitest run` → whole suite PASS, `npm run build` → exit 0 with check-bundle's "bundle clean" line, and `npm run lint` → exit 0. - (No `flake.nix` in this repo; use plain `npm`/`npx`.) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude self-assigned this 2026-10-20 20:05:46 +02:00
Author
Collaborator

✅ Agent run completed

3m0s · 19 turns · $1.82 · pushed agent/lmstudio-detect-guide · session saved — new comments resume this run

✅ **Agent run completed** _3m0s · 19 turns · $1.82 · pushed `agent/lmstudio-detect-guide` · session saved — new comments resume this run_
Author
Collaborator

What changed

  • Added src/transport/lmstudio.ts, which provides detectLmStudio and pickVisionModel.
    • detectLmStudio tries /api/v1/models, then /api/v0/models, then /v1/models. It never throws: a 401 means the server needs an API key, and a refused connection means not running.
    • pickVisionModel prefers a loaded vision model, then any vision model. It never picks a text-only one.
  • Added src/lmstudiosetup.ts, the "LM Studio is running" modal. It has:
    • a dropdown with loaded models first, marked "(loaded)";
    • a note when it can't tell which models read images;
    • Qwen2.5-VL-7B-Instruct guidance when there is no usable model;
    • API-token instructions when the server requires a key;
    • the buttons Use LM Studio / Check again / Use Ollama instead / Close.
  • src/ollamasetup.ts: the end of setup is now a shared helper, applyAndTest(host, endpoint, model, refresh).
    • When Ollama doesn't answer, setup checks for LM Studio first. If it is running, the new modal opens before OllamaMissingModal.
    • When Ollama is running, or LM Studio isn't, setup behaves as before.
    • The selected LM Studio model is saved as ocrModel with endpoint http://localhost:1234/v1.
  • Updated the settings description, docs/ocr-endpoint-setup.md and CHANGELOG-INK.md.
  • Merged origin/main (!31). It conflicted in ollamasetup.ts and the docs. I took main's "→ Troubleshooting" hint and dropped my own per-endpoint hint.

Verify

  • After the merge, run locally:
    • npx vitest run src/__tests__/lmstudio.test.ts src/__tests__/lmstudiosetup.test.ts → 24/24 pass.
    • npx vitest run → 786/786 pass.
    • npm run build → exit 0, "bundle clean".
    • npm run lint → exit 0.
  • No CI pipeline ran for this branch, and I have not tried it against a real LM Studio server.

Follow-ups

  • Possible next step, not done here as scoped: download and load an LM Studio model through POST /api/v1/models/download and /load, mirroring Ollama's pull-with-progress. That would remove the manual "open Discover" step.
  • To make the modal and routing tests possible, the test stand-in src/__tests__/helpers/obsidian.ts now does more:
    • its Modal calls onOpen/onClose, and records opened modals in Modal.opened;
    • Notice records shown messages in Notice.shown;
    • it adds DropdownComponent and appendText.
## What changed - Added `src/transport/lmstudio.ts`, which provides `detectLmStudio` and `pickVisionModel`. - `detectLmStudio` tries `/api/v1/models`, then `/api/v0/models`, then `/v1/models`. It never throws: a 401 means the server needs an API key, and a refused connection means not running. - `pickVisionModel` prefers a loaded vision model, then any vision model. It never picks a text-only one. - Added `src/lmstudiosetup.ts`, the "LM Studio is running" modal. It has: - a dropdown with loaded models first, marked "(loaded)"; - a note when it can't tell which models read images; - Qwen2.5-VL-7B-Instruct guidance when there is no usable model; - API-token instructions when the server requires a key; - the buttons Use LM Studio / Check again / Use Ollama instead / Close. - `src/ollamasetup.ts`: the end of setup is now a shared helper, `applyAndTest(host, endpoint, model, refresh)`. - When Ollama doesn't answer, setup checks for LM Studio first. If it is running, the new modal opens before `OllamaMissingModal`. - When Ollama is running, or LM Studio isn't, setup behaves as before. - The selected LM Studio model is saved as `ocrModel` with endpoint `http://localhost:1234/v1`. - Updated the settings description, `docs/ocr-endpoint-setup.md` and `CHANGELOG-INK.md`. - Merged `origin/main` (!31). It conflicted in `ollamasetup.ts` and the docs. I took main's "→ Troubleshooting" hint and dropped my own per-endpoint hint. ## Verify - After the merge, run locally: - `npx vitest run src/__tests__/lmstudio.test.ts src/__tests__/lmstudiosetup.test.ts` → 24/24 pass. - `npx vitest run` → 786/786 pass. - `npm run build` → exit 0, "bundle clean". - `npm run lint` → exit 0. - No CI pipeline ran for this branch, and I have not tried it against a real LM Studio server. ## Follow-ups - Possible next step, not done here as scoped: download and load an LM Studio model through `POST /api/v1/models/download` and `/load`, mirroring Ollama's pull-with-progress. That would remove the manual "open Discover" step. - To make the modal and routing tests possible, the test stand-in `src/__tests__/helpers/obsidian.ts` now does more: - its `Modal` calls `onOpen`/`onClose`, and records opened modals in `Modal.opened`; - `Notice` records shown messages in `Notice.shown`; - it adds `DropdownComponent` and `appendText`.
daniel merged commit 3352a5dbc6 into main 2026-10-21 00:38:34 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!32
No description provided.