OCR: stop the built-in reader emitting junk on joined-up writing — calibrate confidence & make readings diagnosable #4

Closed
opened 2026-10-20 15:13:18 +02:00 by claude · 9 comments
Collaborator

Part of #2

Part of #2. Directly addresses the top complaint in #2: the built-in reader "is not very reliable and mostly detects junk." The upstream philosophy is correct — show the ink image unless the reading is trustworthy — so "junk" means wrong readings are leaking past the trust thresholds instead of falling back to the image. The fix is calibration + diagnosability, not a new engine.

Root cause (from #2 investigation)

main.ts recognizeLocally() runs, in ocrMode: "builtin":

  1. recognize() ($P per-letter reader, src/ocr/recognize.ts) → accept if confidence >= OCR_TRUST (0.65).
  2. else WritePad WASM (src/ocr/writepad.ts) → accept if confidence >= WRITEPAD_TRUST (0.7).
  3. else null → image only.

The letter reader segments words into letters by horizontal gaps (clusterStrokes/segment in recognize.ts) — joined-up cursive has no gaps, so it mis-segments and the confidence 1 - perLetter/0.6 can still land above 0.65 on a wrong reading; WritePad can likewise pass 0.7 with a junk reading nudged up by the vault-word bonus (VAULT_BONUS/VAULT_MARGIN in writepad.ts). The existing test src/__tests__/recognize.test.ts:59 asserts cursive should score below OCR_TRUST, so the leak is a calibration gap in the real-stroke regime, plus an invisible WritePad fallback nobody can see firing.

What to do (needs investigation → split into MRs)

  1. Make readings diagnosable first (this unblocks tuning and is a clean MR). Each record already stores readBy/readScore/by/score (src/ocr/samples.ts). Surface, per remark, which reader produced the text and its confidence — in the import report (src/incoming/report.ts) and/or optionally in the callout — behind a debug/verbose toggle. Without this, neither the user nor a tuner can tell builtin-junk from WritePad-junk.
  2. Recalibrate with evidence. Using the owner's real remarks (and the handwriting/word sheets), measure where wrong readings currently pass. Likely outcomes: raise OCR_TRUST/WRITEPAD_TRUST, add a per-letter-cost ceiling that forces image fallback on mis-segmented cursive, and/or gate WritePad behind its own quality check. Do not blind-bump a constant — justify each threshold against measured reads. A small labelled fixture set should be added under src/__tests__/fixtures/ so the calibration is regression-tested.
  3. Consider exposing WritePad. It is currently a hidden fallback inside builtin. Decide (design question for the implementer) whether to make it a visible mode or a toggle, so "junk" can be attributed and turned off.

Scope boundary

NOT the vision-endpoint UX (#3) and NOT the trainable-reader R&D (the other #2 child). This issue is purely: make the existing local readers fail safe to the image and be diagnosable.

Validation contract (for the eventual MRs)

  • nix develop -c npx vitest run src/__tests__/recognize.test.ts src/__tests__/writepad.test.ts → PASS, including a new test proving a known-cursive fixture returns null/below-trust (red gate: that fixture currently returns junk text above threshold).
  • The diagnostic surface shows builtin/writepad/endpoint + confidence per remark in the import report.
Part of #2 Part of #2. Directly addresses the top complaint in #2: the built-in reader *"is not very reliable and mostly detects junk."* The upstream philosophy is correct — **show the ink image unless the reading is trustworthy** — so "junk" means wrong readings are leaking past the trust thresholds instead of falling back to the image. The fix is calibration + diagnosability, **not** a new engine. ## Root cause (from #2 investigation) `main.ts` `recognizeLocally()` runs, in `ocrMode: "builtin"`: 1. `recognize()` ($P per-letter reader, `src/ocr/recognize.ts`) → accept if `confidence >= OCR_TRUST` (**0.65**). 2. else WritePad WASM (`src/ocr/writepad.ts`) → accept if `confidence >= WRITEPAD_TRUST` (**0.7**). 3. else `null` → image only. The letter reader segments words into letters **by horizontal gaps** (`clusterStrokes`/`segment` in `recognize.ts`) — joined-up cursive has no gaps, so it mis-segments and the confidence `1 - perLetter/0.6` can still land above 0.65 on a wrong reading; WritePad can likewise pass 0.7 with a junk reading nudged up by the vault-word bonus (`VAULT_BONUS`/`VAULT_MARGIN` in `writepad.ts`). The existing test `src/__tests__/recognize.test.ts:59` *asserts* cursive should score below `OCR_TRUST`, so the leak is a calibration gap in the real-stroke regime, plus an **invisible WritePad fallback** nobody can see firing. ## What to do (needs investigation → split into MRs) 1. **Make readings diagnosable first** (this unblocks tuning and is a clean MR). Each record already stores `readBy`/`readScore`/`by`/`score` (`src/ocr/samples.ts`). Surface, per remark, *which reader produced the text and its confidence* — in the import report (`src/incoming/report.ts`) and/or optionally in the callout — behind a debug/verbose toggle. Without this, neither the user nor a tuner can tell builtin-junk from WritePad-junk. 2. **Recalibrate with evidence.** Using the owner's real remarks (and the handwriting/word sheets), measure where wrong readings currently pass. Likely outcomes: raise `OCR_TRUST`/`WRITEPAD_TRUST`, add a per-letter-cost ceiling that forces image fallback on mis-segmented cursive, and/or gate WritePad behind its own quality check. **Do not blind-bump a constant** — justify each threshold against measured reads. A small labelled fixture set should be added under `src/__tests__/fixtures/` so the calibration is regression-tested. 3. **Consider exposing WritePad.** It is currently a hidden fallback inside `builtin`. Decide (design question for the implementer) whether to make it a visible mode or a toggle, so "junk" can be attributed and turned off. ## Scope boundary NOT the vision-endpoint UX (#3) and NOT the trainable-reader R&D (the other #2 child). This issue is purely: make the *existing* local readers fail safe to the image and be diagnosable. ## Validation contract (for the eventual MRs) - `nix develop -c npx vitest run src/__tests__/recognize.test.ts src/__tests__/writepad.test.ts` → PASS, including a new test proving a known-cursive fixture returns `null`/below-trust (red gate: that fixture currently returns junk text above threshold). - The diagnostic surface shows `builtin`/`writepad`/`endpoint` + confidence per remark in the import report.
claude self-assigned this 2026-10-20 15:13:18 +02:00
Author
Collaborator

✅ Agent run completed

9m14s · 44 turns · $2.93 · pushed agent/ocr-diagnostics, agent/ocr-letter-calibration, agent/ocr-writepad-gate · session saved — new comments resume this run

✅ **Agent run completed** _9m14s · 44 turns · $2.93 · pushed `agent/ocr-diagnostics`, `agent/ocr-letter-calibration`, `agent/ocr-writepad-gate` · session saved — new comments resume this run_
Author
Collaborator

What I found

  • Both leaks are real and reproducible with bundled-font synthetic strokes (I measured them during investigation). Letter reader recognize(): print scores confidence 0.96–0.97, but hershey-script cursive "Check this again tomorrow" scores 0.669 ≥ OCR_TRUST (0.65) and returns garbage "C%% @w mm %mmm". The wide gap means raising OCR_TRUST (~0.75) plugs it cleanly — print has huge headroom. The existing recognize.test.ts:59 only tests allure cursive (0.527, correctly below), so it never caught the hershey-script leak.
  • WritePad's confidence is a poor discriminator, so a threshold bump can't fix it. A near-perfect read "meeting is an Monday" scores 0.682 (rejected) while partial-junk "the quick brown po,frjunql5" scores 0.720 ≥ WRITEPAD_TRUST (0.7), accepted. The fix must be a quality gate (known-word fraction), exactly as the issue anticipated — not a bumped constant. Note: raising OCR_TRUST pushes more cursive onto WritePad, so these two must fail-safe together.
  • Reader + confidence is already stored per remark (readBy/readScore in src/ocr/samples.ts) — it just isn't surfaced anywhere. Diagnosability is a clean, independent first MR.

Created

Three MR hand-offs, each self-contained with measured evidence, red gates, and a validation contract:

  • !9 — Diagnosability: surface builtin/writepad/endpoint + confidence per remark in the import report behind an ocrDiagnostics toggle. (the "do first" piece — unblocks tuning; threads reader/score out of recognizeLocally)
  • !10 — Letter-reader calibration: labelled fixture set under src/__tests__/fixtures/, red-gate test (hershey-script junk currently passes), raise OCR_TRUST into the measured 0.669–0.961 gap with justification. (touches only recognize.ts + fixtures/tests — no conflicts)
  • !11 — WritePad quality gate + exposure: gate the fallback on known-word fraction (good 0.682 vs junk 0.720 can't be separated by score), add an ocrUseWritePad toggle so it's attributable/switchable, red-gate test.

Recommended merge order: !9 → then !10 / !11. !9 and !11 both touch main.ts:recognizeLocally + settings; whichever lands second rebases (adjacent edits, no logic conflict). !10 is independent. MR-to-MR order can't be gated via is_blocked_by (issue-only), so this is advisory — noted in each MR.

Follow-ups

  • None blocking. Final fail-safe behaviour (raised OCR_TRUST sending more cursive to a gated WritePad) is validated end-to-end once !10 + !11 both land — each carries its own unit red gate meanwhile.
  • A flake.nix does not exist in this repo, so nix develop -c … (the validation-contract command style) fails; tests run via npm ci && npx vitest run … on Node 24. Flagged in every MR.
## What I found - **Both leaks are real and reproducible with bundled-font synthetic strokes** (I measured them during investigation). Letter reader `recognize()`: print scores confidence **0.96–0.97**, but `hershey-script` cursive "Check this again tomorrow" scores **0.669 ≥ OCR_TRUST (0.65)** and returns garbage `"C%% @w mm %mmm"`. The wide gap means raising `OCR_TRUST` (~0.75) plugs it cleanly — print has huge headroom. The existing `recognize.test.ts:59` only tests `allure` cursive (0.527, correctly below), so it never caught the `hershey-script` leak. - **WritePad's confidence is a poor discriminator, so a threshold bump can't fix it.** A near-perfect read `"meeting is an Monday"` scores **0.682 (rejected)** while partial-junk `"the quick brown po,frjunql5"` scores **0.720 ≥ WRITEPAD_TRUST (0.7), accepted**. The fix must be a **quality gate** (known-word fraction), exactly as the issue anticipated — not a bumped constant. Note: raising `OCR_TRUST` pushes *more* cursive onto WritePad, so these two must fail-safe together. - Reader + confidence is **already stored** per remark (`readBy`/`readScore` in `src/ocr/samples.ts`) — it just isn't surfaced anywhere. Diagnosability is a clean, independent first MR. ## Created Three MR hand-offs, each self-contained with measured evidence, red gates, and a validation contract: - **!9** — Diagnosability: surface `builtin`/`writepad`/`endpoint` + confidence per remark in the import report behind an `ocrDiagnostics` toggle. *(the "do first" piece — unblocks tuning; threads reader/score out of `recognizeLocally`)* - **!10** — Letter-reader calibration: labelled fixture set under `src/__tests__/fixtures/`, red-gate test (`hershey-script` junk currently passes), raise `OCR_TRUST` into the measured 0.669–0.961 gap with justification. *(touches only `recognize.ts` + fixtures/tests — no conflicts)* - **!11** — WritePad quality gate + exposure: gate the fallback on known-word fraction (good 0.682 vs junk 0.720 can't be separated by score), add an `ocrUseWritePad` toggle so it's attributable/switchable, red-gate test. **Recommended merge order:** !9 → then !10 / !11. !9 and !11 both touch `main.ts:recognizeLocally` + settings; whichever lands second rebases (adjacent edits, no logic conflict). !10 is independent. MR-to-MR order can't be gated via `is_blocked_by` (issue-only), so this is advisory — noted in each MR. ## Follow-ups - None blocking. Final fail-safe behaviour (raised `OCR_TRUST` sending more cursive to a gated WritePad) is validated end-to-end once !10 + !11 both land — each carries its own unit red gate meanwhile. - A `flake.nix` does not exist in this repo, so `nix develop -c …` (the validation-contract command style) fails; tests run via `npm ci && npx vitest run …` on Node 24. Flagged in every MR.
Author
Collaborator

🤖 Agent is working on this — turn 0

Last activity: session init — updated 16:57:10 UTC

🤖 **Agent is working on this** — turn 0 _Last activity: session init — updated 16:57:10 UTC_
Author
Collaborator

🤖 Agent is working on this — turn 8

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 16:59:23 UTC

Last message:

All three changes are on main. Now I'll run the validation contract against current main.

🤖 **Agent is working on this** — turn 8 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 16:59:23 UTC_ **Last message:** > All three changes are on main. Now I'll run the validation contract against current main.
Author
Collaborator

Since my last update: all three hand-offs (!9 diagnostics, !10 letter-reader calibration and fixtures, !11 WritePad quality gate and toggle) are merged to main (checked at db6c9a7). I ran the validation suite on main: npx vitest run src/__tests__/recognize.test.ts src/__tests__/writepad.test.ts src/__tests__/report.test.ts gave 39 passed / 0 failed. The repo has no flake.nix, so nix develop -c doesn't work and I ran npx vitest directly.

That run includes these tests:

  • Cursive red gate: recognize.test.ts:69 and :80. OCR_TRUST is now 0.78. Measured scores were 0.96–0.97 for print and 0.669 for the leaking cursive.
  • WritePad junk gate: writepad.test.ts:68 checks the known-word-fraction gate, which sits on top of WRITEPAD_TRUST.
  • Report diagnostics: report.test.ts:55 checks that the report gets a readers: builtin N (avg x), writepad N (avg y), endpoint N line when ocrDiagnostics is on and none when it is off.

The report shows a count and average confidence per reader for each imported file, not one line per remark. That is enough to tell which reader produced the junk. WritePad can be turned off with ocrUseWritePad. The validation contract is met, so I'm closing this issue.

Since my last update: all three hand-offs (!9 diagnostics, !10 letter-reader calibration and fixtures, !11 WritePad quality gate and toggle) are merged to main (checked at `db6c9a7`). I ran the validation suite on main: `npx vitest run src/__tests__/recognize.test.ts src/__tests__/writepad.test.ts src/__tests__/report.test.ts` gave **39 passed / 0 failed**. The repo has no `flake.nix`, so `nix develop -c` doesn't work and I ran `npx vitest` directly. That run includes these tests: - **Cursive red gate:** `recognize.test.ts:69` and `:80`. `OCR_TRUST` is now 0.78. Measured scores were 0.96–0.97 for print and 0.669 for the leaking cursive. - **WritePad junk gate:** `writepad.test.ts:68` checks the known-word-fraction gate, which sits on top of `WRITEPAD_TRUST`. - **Report diagnostics:** `report.test.ts:55` checks that the report gets a `readers: builtin N (avg x), writepad N (avg y), endpoint N` line when `ocrDiagnostics` is on and none when it is off. The report shows a count and average confidence per reader for each imported file, not one line per remark. That is enough to tell which reader produced the junk. WritePad can be turned off with `ocrUseWritePad`. The validation contract is met, so I'm closing this issue.
claude 2026-10-20 18:59:23 +02:00
daniel reopened this issue 2026-10-21 00:24:55 +02:00
Author
Collaborator

🤖 Agent is working on this — turn 10

Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 04:46:04 UTC

🤖 **Agent is working on this** — turn 10 _Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 04:46:04 UTC_
Author
Collaborator

Since my last update: this issue was reopened because the owner's first real handwriting capture (handwriting-samples/test-sheets/2026-10-21-001026, joined cursive) still let built-in junk through. Two lines were accepted above OCR_TRUST: p2-05 /IäI→✗rl…,%«%.Jl€ at 0.784 and p2-06 @:@+m«↑@&↑3] at 0.788. !35 fixes this and is the only work left on this issue. It adds builtinTrusted(), which keeps OCR_TRUST and also rejects any reading that is more than 30% symbols. On this capture the junk readings are 59% and 83% symbols, and the most symbol-heavy of the 18 prompts is 21%. A rejected reading then falls through to WritePad and, failing that, to the image. I read the diff. It touches 3 files, the branch merges cleanly into main (git merge-tree), and its agent reports recognize.test.ts 11/11 with a red gate seen failing first. No CI pipeline has run on the branch yet. I also checked the WritePad readings it accepted on the same capture (9 of 18, read from the capture JSON). Their error rate (CER) is 9–31%, and they are legible text with some wrong words, not symbol junk. The ink image stays next to them by default (ocrKeepImage: true). WritePad's confidence barely tracks accuracy here: p2-02 was accepted at 0.773 with 29.7% CER, while p2-01 was rejected at 0.678 with only 14.8%. So raising WRITEPAD_TRUST would not help, and better WritePad accuracy on this hand belongs to #14, not here. Nothing new was created. This issue closes when a human merges !35.

Since my last update: this issue was reopened because the owner's first real handwriting capture (`handwriting-samples/test-sheets/2026-10-21-001026`, joined cursive) still let built-in junk through. Two lines were accepted above `OCR_TRUST`: p2-05 `/IäI→✗rl…,%«%.Jl€` at 0.784 and p2-06 `@:@+m«↑@&↑3]` at 0.788. **!35 fixes this and is the only work left on this issue.** It adds `builtinTrusted()`, which keeps `OCR_TRUST` and also rejects any reading that is more than 30% symbols. On this capture the junk readings are 59% and 83% symbols, and the most symbol-heavy of the 18 prompts is 21%. A rejected reading then falls through to WritePad and, failing that, to the image. I read the diff. It touches 3 files, the branch merges cleanly into main (`git merge-tree`), and its agent reports `recognize.test.ts` 11/11 with a red gate seen failing first. No CI pipeline has run on the branch yet. I also checked the WritePad readings it accepted on the same capture (9 of 18, read from the capture JSON). Their error rate (CER) is 9–31%, and they are legible text with some wrong words, not symbol junk. The ink image stays next to them by default (`ocrKeepImage: true`). WritePad's confidence barely tracks accuracy here: p2-02 was accepted at 0.773 with 29.7% CER, while p2-01 was rejected at 0.678 with only 14.8%. So raising `WRITEPAD_TRUST` would not help, and better WritePad accuracy on this hand belongs to #14, not here. Nothing new was created. This issue closes when a human merges !35.
Owner

@claude all done here?

@claude all done here?
Author
Collaborator

Since my last update: yes, all done. !35 is merged and its symbol-junk gate (builtinTrusted) is in main, used by both recognizeLocally and the test-sheet capture. main also has e43961b ("built-in reader declines joined-up writing instead of guessing"), which adds a regression test running the owner's real capture lines through the reader and expecting it to decline every one (src/__tests__/recognize.test.ts:146).

All six MRs for this issue are merged: !9 (reader and confidence shown in the import report), !10 (calibration with labelled fixtures), !11 (WritePad quality gate and toggle) and !35, plus the two Ollama setup MRs !27 and !33. Nothing is left open here, so I'm closing the issue. Making WritePad read joined-up writing more accurately is tracked in #14.

Since my last update: yes, all done. !35 is merged and its symbol-junk gate (`builtinTrusted`) is in `main`, used by both `recognizeLocally` and the test-sheet capture. `main` also has e43961b ("built-in reader declines joined-up writing instead of guessing"), which adds a regression test running the owner's real capture lines through the reader and expecting it to decline every one (`src/__tests__/recognize.test.ts:146`). All six MRs for this issue are merged: !9 (reader and confidence shown in the import report), !10 (calibration with labelled fixtures), !11 (WritePad quality gate and toggle) and !35, plus the two Ollama setup MRs !27 and !33. Nothing is left open here, so I'm closing the issue. Making WritePad read joined-up writing more accurately is tracked in #14.
claude 2026-10-21 06:46:04 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip#4
No description provided.