feat(ocr): handwriting test capture records the endpoint model, per-line read time and failure reason #39

Merged
daniel merged 3 commits from agent/capture-endpoint-model-latency into main 2026-10-21 01:31:47 +02:00
Collaborator

Part of #14

Why

The owner's second capture (handwriting-samples/test-sheets/2026-10-21-004942/) is the first one with vision-endpoint readings. On the same 18 lines (the ink is byte-identical to 2026-10-21-001026), mean CER per reader was:

Reader Mean CER
Endpoint 10.5%
WritePad 30.7%
Built-in 94.8%

The capture can't say which model produced those numbers, or how long each read took. #14's revisit gate, condition 1, is "the local VLM is unacceptable (privacy, latency, setup friction)". Comparing models later (qwen2.5vl:7b vs 3b, or an LM Studio model) needs the model name in the file. On top of that, saveTestSheet swallows endpoint errors (.catch(() => null), src/main.ts ≈ line 2228). So the capture records a failed line as null with no reason, and a NO_TEXT answer also comes out as null, which counts as "failed".

What to change

  1. src/ocr/testcapture.ts
    • CaptureReading gets an optional ms?: number: the wall-clock read time of that reader for that line.
    • HandwritingCapture gets an optional endpoint?: { model: string }. Record the model name only, never the endpoint URL or API key: the capture gets attached to a public-ish forge issue.
    • Add an optional per-item errors?: Partial<Record<ReaderName, string>>, or an equivalent you prefer. A failed read stays null in readings, so summarize()'s "failed" count keeps its meaning, and the error message is kept next to it.
    • Keep version: 1. Every new field is optional, so old captures still parse, and parseCapture must accept both the old and new shapes.
  2. captureMarkdown
    • Add a "Mean time" column to the Mean CER table: the mean ms per reader, or — when absent.
    • Name the endpoint's model under the table, e.g. Endpoint model: qwen2.5vl:7b.
    • List each failed line's error message once, below the readings table.
  3. src/main.ts saveTestSheet
    • Time each reader call: recognize, both writePad.read calls together, and this.ink.ocr. Use performance.now() or Date.now().
    • Set capture.endpoint = { model: this.settings.ocrModel } when ocrMode === "endpoint".
    • Replace .catch(() => null) with a catch that records toTransportError(error).message (or String(error)) in the item's errors.
    • A null from ink.ocr means NO_TEXT, which is a valid answer, so record it as { text: "", accepted: false } and not as a failure.

Constraints and decisions

  • Held out, as before: nothing is learned or remembered (see the doc comment on saveTestSheet).

  • No URL, host or API key in the capture.

  • Don't change the reading logic, the trust thresholds or the prompt. This MR is measurement only.

  • Out of scope:

    • prompt changes for the VLM's Markdown and quote artifacts (*quick* on p1-02, invented quotes on p2-06);
    • an endpoint mode for the offline replay test.

    Both are possible follow-ups once this lands.

Validation contract

  • npx vitest run src/__tests__/testcapture.test.ts → PASS. Add tests that:
    • (a) round-trip a capture through parseCapture with ms, endpoint.model and an error message;
    • (b) check that captureMarkdown shows the mean time column, the model line and the error message;
    • (c) check that an old capture (no new fields) still parses and renders — for time. The committed handwriting-samples/test-sheets/2026-10-21-004942/handwriting-test-2026-10-21-004942.json is a real old-shape fixture.
  • Red gate: test (b) fails on main today, because there's no time column or model line.
  • npx vitest run (full suite) → all pass. npx tsc --noEmit -skipLibCheck and npx eslint src → clean.
  • The repo has no flake.nix, so use plain npm ci && npx … and not nix develop.

🤖 Generated with Claude Code

Part of #14 ## Why The owner's second capture (`handwriting-samples/test-sheets/2026-10-21-004942/`) is the first one with vision-endpoint readings. On the same 18 lines (the ink is byte-identical to `2026-10-21-001026`), mean CER per reader was: | Reader | Mean CER | | --- | --- | | Endpoint | **10.5%** | | WritePad | 30.7% | | Built-in | 94.8% | The capture can't say **which model** produced those numbers, or **how long** each read took. #14's revisit gate, condition 1, is "the local VLM is unacceptable (privacy, **latency**, setup friction)". Comparing models later (qwen2.5vl:7b vs 3b, or an LM Studio model) needs the model name in the file. On top of that, `saveTestSheet` swallows endpoint errors (`.catch(() => null)`, `src/main.ts` ≈ line 2228). So the capture records a failed line as `null` with no reason, and a `NO_TEXT` answer also comes out as `null`, which counts as "failed". ## What to change 1. **`src/ocr/testcapture.ts`** - `CaptureReading` gets an optional `ms?: number`: the wall-clock read time of that reader for that line. - `HandwritingCapture` gets an optional `endpoint?: { model: string }`. **Record the model name only, never the endpoint URL or API key**: the capture gets attached to a public-ish forge issue. - Add an optional per-item `errors?: Partial<Record<ReaderName, string>>`, or an equivalent you prefer. A failed read stays `null` in `readings`, so `summarize()`'s "failed" count keeps its meaning, and the error message is kept next to it. - Keep `version: 1`. Every new field is optional, so old captures still parse, and `parseCapture` must accept both the old and new shapes. 2. **`captureMarkdown`** - Add a "Mean time" column to the Mean CER table: the mean `ms` per reader, or `—` when absent. - Name the endpoint's model under the table, e.g. `Endpoint model: qwen2.5vl:7b`. - List each failed line's error message once, below the readings table. 3. **`src/main.ts` `saveTestSheet`** - Time each reader call: `recognize`, both `writePad.read` calls together, and `this.ink.ocr`. Use `performance.now()` or `Date.now()`. - Set `capture.endpoint = { model: this.settings.ocrModel }` when `ocrMode === "endpoint"`. - Replace `.catch(() => null)` with a catch that records `toTransportError(error).message` (or `String(error)`) in the item's errors. - A `null` from `ink.ocr` means NO_TEXT, which is a valid answer, so record it as `{ text: "", accepted: false }` and not as a failure. ## Constraints and decisions - Held out, as before: nothing is learned or remembered (see the doc comment on `saveTestSheet`). - No URL, host or API key in the capture. - Don't change the reading logic, the trust thresholds or the prompt. This MR is measurement only. - Out of scope: - prompt changes for the VLM's Markdown and quote artifacts (`*quick*` on p1-02, invented quotes on p2-06); - an endpoint mode for the offline replay test. Both are possible follow-ups once this lands. ## Validation contract - `npx vitest run src/__tests__/testcapture.test.ts` → PASS. Add tests that: - (a) round-trip a capture through `parseCapture` with `ms`, `endpoint.model` and an error message; - (b) check that `captureMarkdown` shows the mean time column, the model line and the error message; - (c) check that an old capture (no new fields) still parses and renders `—` for time. The committed `handwriting-samples/test-sheets/2026-10-21-004942/handwriting-test-2026-10-21-004942.json` is a real old-shape fixture. - **Red gate:** test (b) fails on `main` today, because there's no time column or model line. - `npx vitest run` (full suite) → all pass. `npx tsc --noEmit -skipLibCheck` and `npx eslint src` → clean. - The repo has no `flake.nix`, so use plain `npm ci && npx …` and not `nix develop`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude self-assigned this 2026-10-21 01:04:08 +02:00
Author
Collaborator

🤖 Agent is working on this — turn 67

Last activity: assistant message — updated 23:09:06 UTC

Last message:

Test captures now record how long each reader took on each line, which endpoint model did the reading, and why a reading failed. Everything is pushed to the MR branch, and the MR is still open.

Testing: the new tests failed on main before the change and pass now. The full test suite passes (842), and tsc --noEmit -skipLibCheck and eslint src report nothing. npx eslint couldn't find its executable here, so I ran eslint from node_modules directly. No CI pipeline ran on this branch. saveTestSheet itself was only type-checked; I couldn't run it, so the first real capture will be the first live test.

What changed:

  • Capture file (src/ocr/testcapture.ts): each reading can carry its read time. The capture names the endpoint's model (never its URL or key), and each line can carry an error message. A failed reading stays null, so the "Failed" count means the same as before. All new fields are optional and version is still 1. The old real capture (2026-10-21-004942) still loads and shows — for time.
  • Note: the Mean CER table has a Mean time column, followed by an Endpoint model: … line. Each failed line's error appears once in an Errors list below the readings.
  • saveTestSheet (src/main.ts): it times recognize, the two writePad.read calls together, and ink.ocr. It saves the endpoint's error message instead of dropping it. A NO_TEXT answer is now saved as an empty reading, not as a failure.
  • Not in the spec:
    • Error redaction: the endpoint's error messages can contain its address (e.g. Could not reach the OCR endpoint at http://… or ECONNREFUSED 192.168.x.y:11434), which would break the "no URL, host or key" rule. A new captureError() helper replaces the endpoint address, host, API key and any URL with [endpoint] before the message is saved. It has its own test.
    • Unrenderable lines: if a line can't be turned into a picture, the endpoint reading is null with the reason "The line cou
🤖 **Agent is working on this** — turn 67 _Last activity: assistant message — updated 23:09:06 UTC_ **Last message:** > Test captures now record how long each reader took on each line, which endpoint model did the reading, and why a reading failed. Everything is pushed to the MR branch, and the MR is still open. > > **Testing:** the new tests failed on `main` before the change and pass now. The full test suite passes (842), and `tsc --noEmit -skipLibCheck` and `eslint src` report nothing. `npx eslint` couldn't find its executable here, so I ran `eslint` from `node_modules` directly. No CI pipeline ran on this branch. `saveTestSheet` itself was only type-checked; I couldn't run it, so the first real capture will be the first live test. > > **What changed:** > - **Capture file (`src/ocr/testcapture.ts`):** each reading can carry its read time. The capture names the endpoint's model (never its URL or key), and each line can carry an error message. A failed reading stays `null`, so the "Failed" count means the same as before. All new fields are optional and `version` is still 1. The old real capture (`2026-10-21-004942`) still loads and shows `—` for time. > - **Note:** the Mean CER table has a **Mean time** column, followed by an `Endpoint model: …` line. Each failed line's error appears once in an **Errors** list below the readings. > - **`saveTestSheet` (`src/main.ts`):** it times `recognize`, the two `writePad.read` calls together, and `ink.ocr`. It saves the endpoint's error message instead of dropping it. A `NO_TEXT` answer is now saved as an empty reading, not as a failure. > - **Not in the spec:** > - **Error redaction:** the endpoint's error messages can contain its address (e.g. `Could not reach the OCR endpoint at http://…` or `ECONNREFUSED 192.168.x.y:11434`), which would break the "no URL, host or key" rule. A new `captureError()` helper replaces the endpoint address, host, API key and any URL with `[endpoint]` before the message is saved. It has its own test. > - **Unrenderable lines:** if a line can't be turned into a picture, the endpoint reading is `null` with the reason "The line cou
Each reading keeps its wall-clock time (ms), the capture names the
endpoint's model (never its address or key), and an endpoint failure keeps
its message, with the endpoint's URL, host and key redacted. NO_TEXT is now
an empty reading, not a failure. The note gains a Mean time column, the
model line and an Errors list. Old captures still parse and show — for time.

Part of #14

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Collaborator

What changed

  • Merged main into this branch (no conflicts in the files) to clear the conflict flag.
  • src/ocr/testcapture.ts: three new optional fields, so old captures still parse. CaptureReading.ms is the read time. HandwritingCapture.endpoint.model is the model name only. CaptureItem.errors holds the error message, while the reading stays null so the "failed" count means the same. summarize() gains meanMs. The note now has a Mean time column (— when absent), an Endpoint model: … line and an Errors list below the readings.
  • src/main.ts saveTestSheet: times each reader (recognize; both writePad.read calls together; ink.ocr) and records the model when the mode is endpoint. Endpoint errors are now saved instead of dropped. A NO_TEXT answer (null) is now saved as { text: "", accepted: false }, not as a failure.
  • Beyond the spec: toTransportError(...).message can include the endpoint address (Could not reach the OCR endpoint at http://…, ECONNREFUSED 192.168.x.y:11434). A new helper, captureError(), replaces the endpoint setting, its host/hostname, the API key and any http(s):// URL with [endpoint] before the message is saved. It has its own test.
  • If the line's picture can't be drawn, the endpoint reading is null with the error "The line could not be drawn as a picture." (before, it was null with no reason).

Verify

The new tests in testcapture.test.ts failed before the change and pass now. They cover the round-trip with ms, model and error; the note's time column, model line and error; the old-shape fixture 2026-10-21-004942, which renders —; and the redaction. Full suite: 842 pass. tsc and eslint src are clean. No CI pipeline ran on this branch. saveTestSheet itself was only type-checked, not run on a device.

Follow-ups

None. Both out-of-scope items in the description (VLM prompt artifacts, endpoint mode for the offline replay) remain possible follow-ups.

## What changed - Merged `main` into this branch (no conflicts in the files) to clear the conflict flag. - `src/ocr/testcapture.ts`: three new optional fields, so old captures still parse. `CaptureReading.ms` is the read time. `HandwritingCapture.endpoint.model` is the model name only. `CaptureItem.errors` holds the error message, while the reading stays `null` so the "failed" count means the same. `summarize()` gains `meanMs`. The note now has a **Mean time** column (`—` when absent), an `Endpoint model: …` line and an **Errors** list below the readings. - `src/main.ts` `saveTestSheet`: times each reader (`recognize`; both `writePad.read` calls together; `ink.ocr`) and records the model when the mode is endpoint. Endpoint errors are now saved instead of dropped. A `NO_TEXT` answer (`null`) is now saved as `{ text: "", accepted: false }`, not as a failure. - **Beyond the spec:** `toTransportError(...).message` can include the endpoint address (`Could not reach the OCR endpoint at http://…`, `ECONNREFUSED 192.168.x.y:11434`). A new helper, `captureError()`, replaces the endpoint setting, its host/hostname, the API key and any `http(s)://` URL with `[endpoint]` before the message is saved. It has its own test. - If the line's picture can't be drawn, the endpoint reading is `null` with the error "The line could not be drawn as a picture." (before, it was `null` with no reason). ## Verify The new tests in `testcapture.test.ts` failed before the change and pass now. They cover the round-trip with `ms`, model and error; the note's time column, model line and error; the old-shape fixture `2026-10-21-004942`, which renders `—`; and the redaction. Full suite: 842 pass. `tsc` and `eslint src` are clean. No CI pipeline ran on this branch. `saveTestSheet` itself was only type-checked, not run on a device. ## Follow-ups None. Both out-of-scope items in the description (VLM prompt artifacts, endpoint mode for the offline replay) remain possible follow-ups.
daniel merged commit 7015a55dd6 into main 2026-10-21 01:31:47 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!39
No description provided.