OCR diagnosability: surface per-remark reader + confidence in the import report #9

Merged
daniel merged 2 commits from agent/ocr-diagnostics into main 2026-10-20 15:31:08 +02:00
Collaborator

Part of #4 (which is Part of #2). This is work item 1 of 3 — the clean, independent, "do first" piece that unblocks calibration tuning. See #4 for the full picture.

Goal

Make every reading attributable: show, per import run, which reader produced the text (builtin / writepad / endpoint) and how confident it was, behind a debug/verbose toggle. Today the WritePad fallback fires invisibly inside builtin mode, so neither the user nor a tuner can tell builtin-junk from WritePad-junk. Without this surface, the recalibration work (sibling MR) cannot be targeted.

What exists today

  • recognizeLocally() in src/main.ts:1688 already computes scored = { score?: number; by?: ReaderName } for each remark (by ∈ builtin/writepad; the endpoint path sets its own) and passes it to rememberReading() → rememberRemark() in src/ocr/samples.ts, which stores readBy/readScore per record. The data already exists in the sample store; it is just never surfaced.
  • The import report (src/incoming/report.ts) is rendered from PullResult/DocumentScan (src/incoming/pull.ts:29). DocumentScan.ink already holds { erasedWords, movedWords, partialWords, ownStrokes, transcribed }. The report prints scan.ink.transcribed but nothing about who read or how well.
  • The transcribe dependency is typed (request) => Promise<string | null> (src/incoming/pull.ts:184), called in pictureOf() (src/incoming/inkpull.ts:244) which increments scan.ink.transcribed.
  1. Carry the reader + score out of transcription. Change the transcribe dep return type from string | null to { text: string; by?: ReaderName; score?: number } | null (import ReaderName from src/ocr/samples.ts). Update:
    • recognizeLocally() (main.ts:1688) to return { text, by, score } instead of a bare string (it already has text and scored locally — just return them together). score here is the normalised scoreAbove(confidence, cutoff) value already computed (0 = just passed, 1 = certain); surface that.
    • The endpoint transcribe path (transcribeStrokes, main.ts:1926) to return { text, by: "endpoint" } (no numeric confidence — endpoints have none; leave score undefined).
    • pictureOf() (inkpull.ts:236) to read .text where it currently uses the string, and to feed the reader/score into the scan aggregate.
  2. Aggregate per reader into the scan. Add to DocumentScan.ink a readers?: Record<ReaderName, { count: number; scoreSum: number }> (or similar) and accumulate in pictureOf. Keep it optional so older/empty scans stay valid. Average = scoreSum / count at render time.
  3. Add a diagnostics toggle. Add a boolean setting ocrDiagnostics (default false) to RoundTripSettings (src/settingsmodel.ts:102 area) + DEFAULT_SETTINGS, and a schema entry in src/settingschema.ts (in the OCR section around line 406, visibleWhen: { key: "ocrMode", in: ["builtin", "endpoint"] }, control { type: "toggle" }, with help text like "Show which reader produced each remark and how confident it was, in the import report"). Thread it into ImportReportInput (src/incoming/report.ts:12) as ocrDiagnostics?: boolean and set it where the report input is built (search for renderImportReport(/ImportReportInput construction in main.ts).
  4. Render it. In report.ts, when ocrDiagnostics is true and scan.ink.readers has entries, add a line under the existing handwriting: line, e.g.:
    readers: builtin 7 (avg 0.42), writepad 3 (avg 0.08), endpoint 0 — only listing readers with count > 0. Keep the existing non-diagnostic output byte-identical when the toggle is off.

Scope boundary

  • Do NOT change any threshold or recognition logic — that is the sibling calibration MR. This MR is purely the diagnostic surface + toggle.
  • Optional stretch (only if trivial): also append · read by writepad (0.08) to the remark callout when ocrDiagnostics is on. If it adds any complexity to callout generation, skip it and leave it to a follow-up — the report surface is the required deliverable.

Validation contract

  • nix develop -c npx vitest run src/__tests__/report.test.ts → PASS. Add a case: a scan whose ink.readers has { builtin: {count:2, scoreSum:1.0}, writepad: {count:1, scoreSum:0.1} } renders the readers line only when ocrDiagnostics: true, and renders byte-identically to before when it is false (guard against the "scope mismatch / silent" false-pass: assert the line is absent with the toggle off, using a positive control that it is present with the toggle on).
  • nix develop -c npx vitest run src/__tests__/settingschema.test.ts → PASS (new setting present, default false).
  • nix develop -c npm test → PASS (no regressions). Build check: nix develop -c npm run build.

Note: this repo has no flake.nix — nix develop will fail. Run tests directly: npm ci then npx vitest run <files> (Node 24 is on PATH). I verified this during investigation.

Closes nothing on its own; part of #4.

Part of #4 (which is Part of #2). **This is work item 1 of 3** — the clean, independent, "do first" piece that unblocks calibration tuning. See #4 for the full picture. ## Goal Make every reading **attributable**: show, per import run, *which reader* produced the text (`builtin` / `writepad` / `endpoint`) and *how confident* it was, behind a debug/verbose toggle. Today the WritePad fallback fires invisibly inside `builtin` mode, so neither the user nor a tuner can tell builtin-junk from WritePad-junk. Without this surface, the recalibration work (sibling MR) cannot be targeted. ## What exists today - `recognizeLocally()` in `src/main.ts:1688` already computes `scored = { score?: number; by?: ReaderName }` for each remark (`by` ∈ `builtin`/`writepad`; the endpoint path sets its own) and passes it to `rememberReading()` → `rememberRemark()` in `src/ocr/samples.ts`, which stores `readBy`/`readScore` per record. **The data already exists in the sample store; it is just never surfaced.** - The import report (`src/incoming/report.ts`) is rendered from `PullResult`/`DocumentScan` (`src/incoming/pull.ts:29`). `DocumentScan.ink` already holds `{ erasedWords, movedWords, partialWords, ownStrokes, transcribed }`. The report prints `scan.ink.transcribed` but nothing about *who* read or *how well*. - The `transcribe` dependency is typed `(request) => Promise<string | null>` (`src/incoming/pull.ts:184`), called in `pictureOf()` (`src/incoming/inkpull.ts:244`) which increments `scan.ink.transcribed`. ## What to change (recommended shape — adjust if you find cleaner) 1. **Carry the reader + score out of transcription.** Change the `transcribe` dep return type from `string | null` to `{ text: string; by?: ReaderName; score?: number } | null` (import `ReaderName` from `src/ocr/samples.ts`). Update: - `recognizeLocally()` (`main.ts:1688`) to return `{ text, by, score }` instead of a bare string (it already has `text` and `scored` locally — just return them together). `score` here is the normalised `scoreAbove(confidence, cutoff)` value already computed (0 = just passed, 1 = certain); surface that. - The endpoint transcribe path (`transcribeStrokes`, `main.ts:1926`) to return `{ text, by: "endpoint" }` (no numeric confidence — endpoints have none; leave `score` undefined). - `pictureOf()` (`inkpull.ts:236`) to read `.text` where it currently uses the string, and to feed the reader/score into the scan aggregate. 2. **Aggregate per reader into the scan.** Add to `DocumentScan.ink` a `readers?: Record<ReaderName, { count: number; scoreSum: number }>` (or similar) and accumulate in `pictureOf`. Keep it optional so older/empty scans stay valid. Average = `scoreSum / count` at render time. 3. **Add a diagnostics toggle.** Add a boolean setting `ocrDiagnostics` (default `false`) to `RoundTripSettings` (`src/settingsmodel.ts:102` area) + `DEFAULT_SETTINGS`, and a schema entry in `src/settingschema.ts` (in the OCR section around line 406, `visibleWhen: { key: "ocrMode", in: ["builtin", "endpoint"] }`, control `{ type: "toggle" }`, with help text like "Show which reader produced each remark and how confident it was, in the import report"). Thread it into `ImportReportInput` (`src/incoming/report.ts:12`) as `ocrDiagnostics?: boolean` and set it where the report input is built (search for `renderImportReport(`/`ImportReportInput` construction in `main.ts`). 4. **Render it.** In `report.ts`, when `ocrDiagnostics` is true and `scan.ink.readers` has entries, add a line under the existing `handwriting:` line, e.g.: ` readers: builtin 7 (avg 0.42), writepad 3 (avg 0.08), endpoint 0` — only listing readers with count > 0. Keep the existing non-diagnostic output byte-identical when the toggle is off. ## Scope boundary - Do **NOT** change any threshold or recognition logic — that is the sibling calibration MR. This MR is purely the diagnostic surface + toggle. - Optional stretch (only if trivial): also append `· read by writepad (0.08)` to the remark callout when `ocrDiagnostics` is on. If it adds any complexity to callout generation, **skip it** and leave it to a follow-up — the report surface is the required deliverable. ## Validation contract - `nix develop -c npx vitest run src/__tests__/report.test.ts` → PASS. Add a case: a scan whose `ink.readers` has `{ builtin: {count:2, scoreSum:1.0}, writepad: {count:1, scoreSum:0.1} }` renders the readers line **only** when `ocrDiagnostics: true`, and renders byte-identically to before when it is `false` (guard against the "scope mismatch / silent" false-pass: assert the line is *absent* with the toggle off, using a positive control that it *is* present with the toggle on). - `nix develop -c npx vitest run src/__tests__/settingschema.test.ts` → PASS (new setting present, default false). - `nix develop -c npm test` → PASS (no regressions). Build check: `nix develop -c npm run build`. Note: this repo has **no `flake.nix`** — `nix develop` will fail. Run tests directly: `npm ci` then `npx vitest run <files>` (Node 24 is on PATH). I verified this during investigation. Closes nothing on its own; part of #4.
claude self-assigned this 2026-10-20 15:21:01 +02:00
Author
Collaborator

✅ Agent run completed

4s · 1 turns · $3.61 · pushed agent/ocr-diagnostics · session saved — new comments resume this run

✅ **Agent run completed** _4s · 1 turns · $3.61 · pushed `agent/ocr-diagnostics` · session saved — new comments resume this run_
Carry the reader name and normalised score out of transcription so the
import report can attribute readings per reader (builtin/writepad/endpoint),
behind a new ocrDiagnostics setting (default off). The data already existed
in the sample store; this exposes it so calibration can be targeted.

Part of #4.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author
Collaborator

What changed

  • Transcription now carries who read each remark and how sure out of recognizeLocally/transcribeStrokes ({ text, by, score } instead of a bare string); pictureOf aggregates this into a new optional DocumentScan.ink.readers (count/scoreSum per reader).
  • New ocrDiagnostics setting (default false) in the model, DEFAULT_SETTINGS, and the settings schema (OCR section, visibleWhen builtin/endpoint, toggle).
  • The import report renders a readers: builtin N (avg …), writepad N (avg …), endpoint N line under handwriting: only when the toggle is on and at least one reader has a reading. Endpoints carry no confidence, so they show a count with no avg. Output is byte-identical when the toggle is off.
  • Scope held: no threshold/recognition logic touched; the optional callout-suffix stretch was left for a follow-up.

Verify

npx vitest run src/__tests__/report.test.ts src/__tests__/settingschema.test.ts → PASS. The report case uses a positive control (the readers line is present with ocrDiagnostics: true) and asserts the off-toggle report is byte-identical to a no-reader render. Full npm test → 654 passed; npm run build (tsc + esbuild + bundle check) → clean. No pipeline has run for head efff397 (the only recent CI run is a pre-existing failure on main, unrelated to this diff).

Follow-ups

  • Optional stretch (deferred, per scope): append · read by writepad (0.08) to the remark callout when diagnostics is on.
## What changed - Transcription now carries **who read each remark and how sure** out of `recognizeLocally`/`transcribeStrokes` (`{ text, by, score }` instead of a bare string); `pictureOf` aggregates this into a new optional `DocumentScan.ink.readers` (`count`/`scoreSum` per reader). - New `ocrDiagnostics` setting (default **false**) in the model, `DEFAULT_SETTINGS`, and the settings schema (OCR section, `visibleWhen` builtin/endpoint, toggle). - The import report renders a `readers: builtin N (avg …), writepad N (avg …), endpoint N` line under `handwriting:` **only** when the toggle is on and at least one reader has a reading. Endpoints carry no confidence, so they show a count with no avg. Output is byte-identical when the toggle is off. - Scope held: no threshold/recognition logic touched; the optional callout-suffix stretch was left for a follow-up. ## Verify `npx vitest run src/__tests__/report.test.ts src/__tests__/settingschema.test.ts` → PASS. The report case uses a positive control (the readers line *is* present with `ocrDiagnostics: true`) and asserts the off-toggle report is byte-identical to a no-reader render. Full `npm test` → **654 passed**; `npm run build` (tsc + esbuild + bundle check) → clean. No pipeline has run for head `efff397` (the only recent CI run is a pre-existing failure on `main`, unrelated to this diff). ## Follow-ups - Optional stretch (deferred, per scope): append `· read by writepad (0.08)` to the remark callout when diagnostics is on.
daniel merged commit 41908b7a86 into main 2026-10-20 15:31:08 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!9
No description provided.