OCR calibration: stop the $P letter reader passing junk cursive above OCR_TRUST (+ labelled fixtures) #10

Merged
daniel merged 2 commits from agent/ocr-letter-calibration into main 2026-10-20 15:31:54 +02:00
Collaborator

Part of #4 (which is Part of #2). This is work item 2 of 3. Recalibrate the $P per-letter reader (src/ocr/recognize.ts) so mis-segmented joined-up writing fails safe to the image instead of leaking junk text above the trust threshold. Sibling MR !? handles the WritePad path; this MR is only the letter reader + fixtures.

Measured evidence (I ran this during investigation — reproduce it, don't take it on faith)

Rendering text in bundled fonts as device strokes (the same write(text, font, jitter) helper the existing tests use) and reading via recognize(strokes, templates, lexicon):

input font letter-reader confidence text CER
"Check this again tomorrow" readability (print) 0.971 "Check this again tomorrow" 0.00
"Please review the draft" readability (print) 0.972 correct 0.00
"meeting is on Monday" hershey-sans (print) 0.968 ~correct 0.10
"Bitte noch einmal prüfen" readability (print) 0.961 ~correct 0.04
"Check this again tomorrow" hershey-script (cursive) 0.669 ← PASSES OCR_TRUST=0.65 "C%% @w mm %mmm" 0.80 (junk)
"Please review the draft" hershey-script (cursive) 0.615 (below) junk —
"the quick brown fox jumps" allure (cursive) 0.527 (below) junk —
"meeting is on Monday" allure (cursive) 0.200 (below) junk —

The leak: print scores 0.96–0.97, but one cursive case scores 0.669 ≥ 0.65 with total garbage ("C%% @w mm %mmm", mostly symbol glyphs %/@). There is a wide clean gap between legitimate print (0.96+) and the junk leak (0.669), so raising OCR_TRUST plugs it with large margin and no risk to print.

The existing test recognize.test.ts:59 only exercises allure cursive (which scores 0.527, correctly below) — it never caught the hershey-script leak. That is the calibration gap.

What to do

  1. Add a labelled fixture set under src/__tests__/fixtures/ — a small JSON (or a fixtures/handwriting.ts module) listing labelled samples: { text, font, jitter, kind: "print" | "cursive" }. Seed it with the cases above plus a few more per class so the calibration is regression-tested against a set, not one point. (Strokes are generated deterministically from bundled fonts via the write helper — see recognize.test.ts:7 — so the fixture only needs the parameters, not raw stroke dumps. Keep it this way; it stays diffable.)
  2. Red gate first. Add a test to recognize.test.ts that renders the hershey-script "Check this again tomorrow" fixture and asserts recognize(...).confidence < OCR_TRUST and that it does not return the correct text. Watch it fail on today's code (0.669 ≥ 0.65) before changing anything — that proves the gate bites.
  3. Recalibrate, justified by the table. Raise OCR_TRUST (src/ocr/recognize.ts:431) to a value that sits in the gap (0.669, 0.961) — ~0.75 gives headroom on both sides. Re-run the whole labelled set and confirm: every print fixture stays above the new threshold (the existing "reads print letters" test at recognize.test.ts:39 asserts confidence > OCR_TRUST and must still pass), every cursive fixture lands below. Pick the number from the measured separation and state the justification in a code comment next to the constant (print min vs cursive max observed). Do not blind-bump.
  4. If a single global threshold cannot separate your fixture set cleanly (it should, given the gap), the issue sanctions a per-letter-cost ceiling as a secondary guard — note that the junk here has low per-letter cost (0.669 ⇒ perLetter≈0.20, because symbol glyphs match cheaply), so a cost ceiling alone would NOT catch this case; prefer the threshold. Only add a secondary guard if the threshold move leaves a fixture misclassified, and justify it the same way.

Scope boundary

  • Only src/ocr/recognize.ts + src/__tests__/fixtures/ + recognize.test.ts. Do not touch writepad.ts, main.ts, or settings (sibling MRs).
  • Note the interaction (no code conflict): raising OCR_TRUST sends more cursive to the WritePad fallback, so the WritePad quality gate (sibling MR !?) is what ultimately catches what this MR lets fall through. They are validated together at the end of #4; each has its own unit red gate.

Validation contract

  • nix develop -c npx vitest run src/__tests__/recognize.test.ts → PASS, including the new red-gate test (which fails on main).
  • nix develop -c npm test → PASS (no regressions).

Note: this repo has no flake.nix — nix develop fails. Run directly: npm ci then npx vitest run src/__tests__/recognize.test.ts (Node 24 on PATH). Verified during investigation.

Part of #4.

Part of #4 (which is Part of #2). **This is work item 2 of 3.** Recalibrate the **$P per-letter reader** (`src/ocr/recognize.ts`) so mis-segmented joined-up writing fails safe to the image instead of leaking junk text above the trust threshold. Sibling MR !? handles the WritePad path; this MR is **only** the letter reader + fixtures. ## Measured evidence (I ran this during investigation — reproduce it, don't take it on faith) Rendering text in bundled fonts as device strokes (the same `write(text, font, jitter)` helper the existing tests use) and reading via `recognize(strokes, templates, lexicon)`: | input | font | letter-reader confidence | text | CER | |---|---|---|---|---| | "Check this again tomorrow" | readability (print) | **0.971** | "Check this again tomorrow" | 0.00 | | "Please review the draft" | readability (print) | **0.972** | correct | 0.00 | | "meeting is on Monday" | hershey-sans (print) | **0.968** | ~correct | 0.10 | | "Bitte noch einmal prüfen" | readability (print) | **0.961** | ~correct | 0.04 | | **"Check this again tomorrow"** | **hershey-script (cursive)** | **0.669 ← PASSES OCR_TRUST=0.65** | **"C%% @w mm %mmm"** | **0.80 (junk)** | | "Please review the draft" | hershey-script (cursive) | 0.615 (below) | junk | — | | "the quick brown fox jumps" | allure (cursive) | 0.527 (below) | junk | — | | "meeting is on Monday" | allure (cursive) | 0.200 (below) | junk | — | **The leak:** print scores **0.96–0.97**, but one cursive case scores **0.669 ≥ 0.65** with total garbage (`"C%% @w mm %mmm"`, mostly symbol glyphs `%`/`@`). There is a **wide clean gap** between legitimate print (0.96+) and the junk leak (0.669), so raising `OCR_TRUST` plugs it with large margin and no risk to print. The existing test `recognize.test.ts:59` only exercises `allure` cursive (which scores 0.527, correctly below) — it never caught the `hershey-script` leak. That is the calibration gap. ## What to do 1. **Add a labelled fixture set** under `src/__tests__/fixtures/` — a small JSON (or a `fixtures/handwriting.ts` module) listing labelled samples: `{ text, font, jitter, kind: "print" | "cursive" }`. Seed it with the cases above plus a few more per class so the calibration is regression-tested against a set, not one point. (Strokes are generated deterministically from bundled fonts via the `write` helper — see `recognize.test.ts:7` — so the fixture only needs the parameters, not raw stroke dumps. Keep it this way; it stays diffable.) 2. **Red gate first.** Add a test to `recognize.test.ts` that renders the `hershey-script` "Check this again tomorrow" fixture and asserts `recognize(...).confidence < OCR_TRUST` **and** that it does not return the correct text. Watch it **fail on today's code** (0.669 ≥ 0.65) before changing anything — that proves the gate bites. 3. **Recalibrate, justified by the table.** Raise `OCR_TRUST` (`src/ocr/recognize.ts:431`) to a value that sits in the gap `(0.669, 0.961)` — **~0.75** gives headroom on both sides. Re-run the whole labelled set and confirm: every `print` fixture stays **above** the new threshold (the existing "reads print letters" test at `recognize.test.ts:39` asserts `confidence > OCR_TRUST` and must still pass), every `cursive` fixture lands **below**. Pick the number from the measured separation and **state the justification in a code comment** next to the constant (print min vs cursive max observed). Do **not** blind-bump. 4. If a single global threshold cannot separate your fixture set cleanly (it should, given the gap), the issue sanctions a **per-letter-cost ceiling** as a secondary guard — note that the junk here has *low* per-letter cost (0.669 ⇒ perLetter≈0.20, because symbol glyphs match cheaply), so a cost ceiling alone would NOT catch this case; prefer the threshold. Only add a secondary guard if the threshold move leaves a fixture misclassified, and justify it the same way. ## Scope boundary - **Only** `src/ocr/recognize.ts` + `src/__tests__/fixtures/` + `recognize.test.ts`. Do **not** touch `writepad.ts`, `main.ts`, or settings (sibling MRs). - Note the interaction (no code conflict): raising `OCR_TRUST` sends *more* cursive to the WritePad fallback, so the WritePad quality gate (sibling MR !?) is what ultimately catches what this MR lets fall through. They are validated together at the end of #4; each has its own unit red gate. ## Validation contract - `nix develop -c npx vitest run src/__tests__/recognize.test.ts` → PASS, **including** the new red-gate test (which fails on `main`). - `nix develop -c npm test` → PASS (no regressions). Note: this repo has **no `flake.nix`** — `nix develop` fails. Run directly: `npm ci` then `npx vitest run src/__tests__/recognize.test.ts` (Node 24 on PATH). Verified during investigation. Part of #4.
claude self-assigned this 2026-10-20 15:21:32 +02:00
Author
Collaborator

✅ Agent run completed

8m19s · 39 turns · $2.69 · pushed agent/ocr-letter-calibration · session saved — new comments resume this run

✅ **Agent run completed** _8m19s · 39 turns · $2.69 · pushed `agent/ocr-letter-calibration` · session saved — new comments resume this run_
The $P letter reader leaked junk above the trust threshold on joined-up
writing: hershey-script "Check this again tomorrow" read "C%% @w mm %mmm"
at confidence 0.669, clearing OCR_TRUST=0.65. The existing cursive test
only exercised allure (0.527), so it never caught the hershey-script tail.

Add a labelled print/cursive fixture set and calibrate against the whole
set rather than one point. Measured separation: print reads 0.83-0.98
(min hershey-sans "update the spreadsheet now" 0.831), cursive tops out at
0.739 (hershey-script "we should talk soon"). Raise OCR_TRUST to 0.78,
centred in that gap with ~0.05/0.04 headroom on each side, justified in a
comment by the constant. Includes a red-gate test that fails on 0.65.

Part of #4.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author
Collaborator

What changed

  • Raised OCR_TRUST 0.65 → 0.78 in src/ocr/recognize.ts so mis-segmented cursive fails safe to the image instead of leaking junk text. Justified inline by the measured class separation.
  • Added a labelled print/cursive fixture set at src/__tests__/fixtures/handwriting.ts (params only — strokes rendered deterministically via the write helper, so it stays diffable).
  • Added two tests to recognize.test.ts: a red gate on the documented leak (hershey-script "Check this again tomorrow"), and a whole-set print-vs-cursive separation test.

Calibration note (I calibrated against the set, not one point): the hand-off's single 0.669 leak point was optimistic. Sweeping a wider sample, the real cursive tail reaches 0.739 (hershey-script "we should talk soon") and the print floor is 0.831 (hershey-sans "update the spreadsheet now"). So the clean gap is (0.739, 0.831) and 0.75 would be too tight (0.011 margin). 0.78 sits centred in the real gap (~0.05 below the print floor, ~0.04 above the cursive ceiling). A per-letter-cost ceiling would not catch the leak (its per-letter cost ≈0.20 is low, symbol glyphs match cheaply), so the threshold is the right guard — no secondary guard added.

Verify

  • npx vitest run src/__tests__/recognize.test.ts → 8 pass, incl. the red gate (verified it fails on main at 0.669 ≥ 0.65 before the constant change, passes after).
  • npm test → 57 files / 655 tests pass, no regressions. tsc -noEmit and eslint on the 3 changed files clean.
  • No per-MR test pipeline exists in this repo (only upstream-sync / release workflows); the lone red run is pre-existing on main, unrelated to this diff.

Follow-ups

  • None in scope. Interaction (no code conflict): raising OCR_TRUST routes more cursive to the WritePad fallback, so the WritePad quality gate in the sibling MR is what ultimately catches what this MR lets fall through — validated together at the end of #4.
## What changed - Raised `OCR_TRUST` **0.65 → 0.78** in `src/ocr/recognize.ts` so mis-segmented cursive fails safe to the image instead of leaking junk text. Justified inline by the measured class separation. - Added a labelled print/cursive fixture set at `src/__tests__/fixtures/handwriting.ts` (params only — strokes rendered deterministically via the `write` helper, so it stays diffable). - Added two tests to `recognize.test.ts`: a **red gate** on the documented leak (`hershey-script` "Check this again tomorrow"), and a whole-set print-vs-cursive separation test. **Calibration note (I calibrated against the set, not one point):** the hand-off's single 0.669 leak point was optimistic. Sweeping a wider sample, the real cursive tail reaches **0.739** (`hershey-script` "we should talk soon") and the print floor is **0.831** (`hershey-sans` "update the spreadsheet now"). So the clean gap is **(0.739, 0.831)** and 0.75 would be too tight (0.011 margin). **0.78** sits centred in the real gap (~0.05 below the print floor, ~0.04 above the cursive ceiling). A per-letter-cost ceiling would not catch the leak (its per-letter cost ≈0.20 is low, symbol glyphs match cheaply), so the threshold is the right guard — no secondary guard added. ## Verify - `npx vitest run src/__tests__/recognize.test.ts` → **8 pass**, incl. the red gate (verified it **fails on `main` at 0.669 ≥ 0.65** before the constant change, passes after). - `npm test` → **57 files / 655 tests pass**, no regressions. `tsc -noEmit` and `eslint` on the 3 changed files clean. - No per-MR test pipeline exists in this repo (only `upstream-sync` / `release` workflows); the lone red run is pre-existing on `main`, unrelated to this diff. ## Follow-ups - None in scope. Interaction (no code conflict): raising `OCR_TRUST` routes *more* cursive to the WritePad fallback, so the WritePad quality gate in the sibling MR is what ultimately catches what this MR lets fall through — validated together at the end of #4.
daniel merged commit 7295d52e01 into main 2026-10-20 15:31:54 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!10
No description provided.