fix(ocr): built-in reader must not accept symbol junk on real cursive (OCR_TRUST leaks on owner ink) #35

Merged
daniel merged 2 commits from agent/builtin-symbol-junk-gate into main 2026-10-21 00:25:52 +02:00
Collaborator

Related to #4 and #14. This is a follow-up to #4, which closed after being calibrated on synthetic data only. The first real capture of the owner's hand shows the leak #4 was meant to close.

Why

The owner's first real test sheet is in the repo at handwriting-samples/test-sheets/2026-10-21-001026/handwriting-test-2026-10-21-001026.json. It has 18 lines of fast, joined cursive. The built-in reader accepted 2 of the 18 lines as pure symbol junk:

item prompt built-in text confidence accepted
p2-05 Größe: 42 cm × 18 cm, Preis 12,50 € /IäI→✗rl…,%«%.Jl€ 0.7836 true
p2-06 Frage: ja oder nein? (siehe #14) @:@+m«↑@&↑3] 0.7880 true

Root cause:

  • OCR_TRUST = 0.78 (src/ocr/recognize.ts:497) was calibrated on bundled-font fixtures, where cursive junk topped out at about 0.739. Junk on real ink reaches 0.788.
  • The comment at recognize.ts:483-495 already notes that symbol glyphs match cheaply. The owner's hand store now has symbol templates (@ → ✗ « ↑ % € …, 118 glyphs in capture.hand.glyphs), so joined strokes get tiled with them at high confidence.
  • In recognizeLocally (src/main.ts:1775), an accepted built-in reading means WritePad is never asked. On p2-06, WritePad would have produced Frage'oja oder rams.. (siehe # l4) and passed its own gate.

Raising OCR_TRUST again is ruled out. #4 said "do not blind-bump a constant", and the junk confidences (0.78–0.79) overlap the synthetic print floor (about 0.83) too closely to buy real margin.

What to change

  1. src/ocr/recognize.ts
    • Add export const OCR_MAX_SYMBOLS = 0.3.
    • Add a pure export function builtinTrusted(r: Recognition | null): boolean. It returns r !== null && r.confidence >= OCR_TRUST && symbolFraction(r.text) <= OCR_MAX_SYMBOLS.
    • symbolFraction = (non-whitespace chars that are not \p{L} or \p{N}) ÷ (non-whitespace chars). Use 1 for an empty string.
    • Document the calibration next to the constant, in the style of the existing OCR_TRUST comment:
      • On the real capture, all 18 test-sheet prompts score at most 0.21. The highest are later → done ✓, not now ✗ at about 0.21 and Frage: ja oder nein? (siehe #14) at about 0.19.
      • The two leaked junk readings score 0.59 and 0.83.
  2. src/main.ts — use builtinTrusted(result) in both places that currently compare against OCR_TRUST:
    • recognizeLocally at about line 1775, so a rejected built-in reading falls through to WritePad.
    • The test-sheet save at about line 2209 (accepted:), so future captures record the same decision an import makes.
    • Drop the OCR_TRUST import from main.ts if it becomes unused. scoreAbove(result.confidence, OCR_TRUST) still needs it, so keep it where it is still used.
  3. Tests: add a case to src/__tests__/recognize.test.ts (or a new small test). It covers:
    • builtinTrusted rejects the two literal captured readings above at confidence 0.79.
    • It accepts every prompt in TEST_SHEET_PROMPTS (src/ink/handsheet.ts:71) at confidence 0.9.
    • All existing print fixtures in src/__tests__/fixtures/handwriting.ts must stay trusted.

Constraints and decisions

  • This is a symbol-fraction gate, not a lexicon gate. A known-word gate like WRITEPAD_MIN_KNOWN would also reject legitimate print that is mostly numbers and dates, such as Termin am 14.03. um 9:30 Uhr. The symbol gate needs no vault words, so it also works in the offline replay.
  • Do not change OCR_TRUST, WRITEPAD_TRUST, WRITEPAD_MIN_KNOWN, or the order of readers.
  • Do not change the capture format (CAPTURE_FORMAT / version 1).

Scope boundary

NOT in scope:

  • WritePad's accuracy on this hand (29.6% mean CER) and the neural reader. Both are tracked in #14.
  • A margin-based trust rule (the #13/#17 lead).
  • Any change to how the built-in reader segments joined writing.

Validation contract

  • Red gate: the new test fails today, because builtinTrusted doesn't exist or the junk is accepted.
  • npx vitest run src/__tests__/recognize.test.ts → PASS.
  • npx vitest run → whole suite PASS. In particular, the print fixtures in recognize.test.ts / ocr.test.ts stay trusted.
  • npm run build (tsc + esbuild) → exit 0, and npx eslint src is clean.
  • Optional evidence for the MR comment:
    • Run HANDWRITING_CAPTURE=handwriting-samples/test-sheets/2026-10-21-001026/handwriting-test-2026-10-21-001026.json npx vitest run src/__tests__/testcapture.test.ts.
    • The replay runs without the vault lexicon, so its confidences may differ from the captured ones. Report what it prints. Do not assert on it.

🤖 Generated with Claude Code

Related to #4 and #14. This is a follow-up to #4, which closed after being calibrated on synthetic data only. The first real capture of the owner's hand shows the leak #4 was meant to close. ## Why The owner's first real test sheet is in the repo at `handwriting-samples/test-sheets/2026-10-21-001026/handwriting-test-2026-10-21-001026.json`. It has 18 lines of fast, joined cursive. The built-in reader **accepted 2 of the 18 lines as pure symbol junk**: | item | prompt | built-in text | confidence | accepted | |---|---|---|---|---| | p2-05 | `Größe: 42 cm × 18 cm, Preis 12,50 €` | `/IäI→✗rl…,%«%.Jl€` | 0.7836 | **true** | | p2-06 | `Frage: ja oder nein? (siehe #14)` | `@:@+m«↑@&↑3]` | 0.7880 | **true** | Root cause: - `OCR_TRUST = 0.78` (`src/ocr/recognize.ts:497`) was calibrated on bundled-font fixtures, where cursive junk topped out at about 0.739. Junk on real ink reaches 0.788. - The comment at `recognize.ts:483-495` already notes that symbol glyphs match cheaply. The owner's hand store now has symbol templates (`@ → ✗ « ↑ % €` …, 118 glyphs in `capture.hand.glyphs`), so joined strokes get tiled with them at high confidence. - In `recognizeLocally` (`src/main.ts:1775`), an accepted built-in reading means **WritePad is never asked**. On p2-06, WritePad would have produced `Frage'oja oder rams.. (siehe # l4)` and passed its own gate. Raising `OCR_TRUST` again is ruled out. #4 said "do not blind-bump a constant", and the junk confidences (0.78–0.79) overlap the synthetic print floor (about 0.83) too closely to buy real margin. ## What to change 1. **`src/ocr/recognize.ts`** - Add `export const OCR_MAX_SYMBOLS = 0.3`. - Add a pure `export function builtinTrusted(r: Recognition | null): boolean`. It returns `r !== null && r.confidence >= OCR_TRUST && symbolFraction(r.text) <= OCR_MAX_SYMBOLS`. - `symbolFraction` = (non-whitespace chars that are not `\p{L}` or `\p{N}`) ÷ (non-whitespace chars). Use 1 for an empty string. - Document the calibration next to the constant, in the style of the existing `OCR_TRUST` comment: - On the real capture, all 18 test-sheet **prompts** score at most 0.21. The highest are `later → done ✓, not now ✗` at about 0.21 and `Frage: ja oder nein? (siehe #14)` at about 0.19. - The two leaked junk readings score 0.59 and 0.83. 2. **`src/main.ts`** — use `builtinTrusted(result)` in both places that currently compare against `OCR_TRUST`: - `recognizeLocally` at about line 1775, so a rejected built-in reading falls through to WritePad. - The test-sheet save at about line 2209 (`accepted:`), so future captures record the same decision an import makes. - Drop the `OCR_TRUST` import from main.ts if it becomes unused. `scoreAbove(result.confidence, OCR_TRUST)` still needs it, so keep it where it is still used. 3. **Tests:** add a case to `src/__tests__/recognize.test.ts` (or a new small test). It covers: - `builtinTrusted` rejects the two literal captured readings above at confidence 0.79. - It accepts every prompt in `TEST_SHEET_PROMPTS` (`src/ink/handsheet.ts:71`) at confidence 0.9. - All existing print fixtures in `src/__tests__/fixtures/handwriting.ts` must stay trusted. ## Constraints and decisions - This is a symbol-fraction gate, not a lexicon gate. A known-word gate like `WRITEPAD_MIN_KNOWN` would also reject legitimate print that is mostly numbers and dates, such as `Termin am 14.03. um 9:30 Uhr`. The symbol gate needs no vault words, so it also works in the offline replay. - Do not change `OCR_TRUST`, `WRITEPAD_TRUST`, `WRITEPAD_MIN_KNOWN`, or the order of readers. - Do not change the capture format (`CAPTURE_FORMAT` / version 1). ## Scope boundary NOT in scope: - WritePad's accuracy on this hand (29.6% mean CER) and the neural reader. Both are tracked in #14. - A margin-based trust rule (the #13/#17 lead). - Any change to how the built-in reader segments joined writing. ## Validation contract - **Red gate:** the new test fails today, because `builtinTrusted` doesn't exist or the junk is accepted. - `npx vitest run src/__tests__/recognize.test.ts` → PASS. - `npx vitest run` → whole suite PASS. In particular, the print fixtures in `recognize.test.ts` / `ocr.test.ts` stay trusted. - `npm run build` (tsc + esbuild) → exit 0, and `npx eslint src` is clean. - Optional evidence for the MR comment: - Run `HANDWRITING_CAPTURE=handwriting-samples/test-sheets/2026-10-21-001026/handwriting-test-2026-10-21-001026.json npx vitest run src/__tests__/testcapture.test.ts`. - The replay runs without the vault lexicon, so its confidences may differ from the captured ones. Report what it prints. Do not assert on it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude self-assigned this 2026-10-21 00:18:49 +02:00
Author
Collaborator

✅ Agent run completed

3m43s · 24 turns · $0.60 · pushed agent/builtin-symbol-junk-gate · session saved — new comments resume this run

✅ **Agent run completed** _3m43s · 24 turns · $0.60 · pushed `agent/builtin-symbol-junk-gate` · session saved — new comments resume this run_
The first real capture of the owner's hand leaked two lines of symbol junk
above OCR_TRUST, so WritePad was never asked. Gate the built-in reader on
its share of symbols too (OCR_MAX_SYMBOLS = 0.3), in recognizeLocally and
in the test-sheet save.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Collaborator

What changed

  • src/ocr/recognize.ts: added OCR_MAX_SYMBOLS = 0.3 and a pure builtinTrusted(r), which requires confidence ≥ OCR_TRUST and a symbol fraction ≤ 0.3. The symbol fraction is the share of non-space characters that are neither \p{L} nor \p{N}, and it is 1 for empty text. The calibration is documented next to the constant. OCR_TRUST, WRITEPAD_*, the reader order and the capture format are unchanged.
  • src/main.ts: recognizeLocally and the test-sheet accepted: field now use builtinTrusted, so a rejected built-in reading falls through to WritePad. The OCR_TRUST import stays because scoreAbove still uses it.
  • recognize.test.ts: three new cases.
    • The two literal junk readings are rejected at 0.79.
    • Every TEST_SHEET_PROMPTS entry is trusted at 0.9, and still rejected just below OCR_TRUST.
    • All PRINT_SAMPLES stay trusted.

Verify

  • Red gate: the new tests failed before the change (builtinTrusted is not a function).
  • Now: recognize.test.ts 11/11 pass. The full vitest run gives 778 pass / 0 fail / 1 skipped. npm run build exits 0 and eslint src/ is clean. No CI pipeline has run on this branch yet.
  • Symbol fractions (measured): the junk readings score 0.588 and 0.833. The highest prompts score 0.211 (later → done ✓, not now ✗), then 0.185 (Frage: …, Größe: …). Termin am 14.03. um 9:30 Uhr scores 0.130.
  • Gate applied to the captured built-in readings of 2026-10-21-001026: p2-05 (0.7836) and p2-06 (0.7880) change from accepted=true to false. None of the other 16 items is accepted.
  • Optional HANDWRITING_CAPTURE replay (no vault words, reported but not asserted): built-in mean CER is 94.8% captured vs 94.5% replayed, WritePad 29.6% vs 30.7%. On p2-06, WritePad now replays "Fragi 'oju odor namr..c side # 14)" (50.0% CER). The replay prints CER only, not confidences.

Follow-ups

None. WritePad's accuracy on this hand and the neural reader stay in #14.

## What changed - `src/ocr/recognize.ts`: added `OCR_MAX_SYMBOLS = 0.3` and a pure `builtinTrusted(r)`, which requires confidence ≥ `OCR_TRUST` **and** a symbol fraction ≤ 0.3. The symbol fraction is the share of non-space characters that are neither `\p{L}` nor `\p{N}`, and it is 1 for empty text. The calibration is documented next to the constant. `OCR_TRUST`, `WRITEPAD_*`, the reader order and the capture format are unchanged. - `src/main.ts`: `recognizeLocally` and the test-sheet `accepted:` field now use `builtinTrusted`, so a rejected built-in reading falls through to WritePad. The `OCR_TRUST` import stays because `scoreAbove` still uses it. - `recognize.test.ts`: three new cases. - The two literal junk readings are rejected at 0.79. - Every `TEST_SHEET_PROMPTS` entry is trusted at 0.9, and still rejected just below `OCR_TRUST`. - All `PRINT_SAMPLES` stay trusted. ## Verify - **Red gate:** the new tests failed before the change (`builtinTrusted is not a function`). - **Now:** `recognize.test.ts` 11/11 pass. The full `vitest run` gives 778 pass / 0 fail / 1 skipped. `npm run build` exits 0 and `eslint src/` is clean. No CI pipeline has run on this branch yet. - **Symbol fractions (measured):** the junk readings score 0.588 and 0.833. The highest prompts score 0.211 (`later → done ✓, not now ✗`), then 0.185 (`Frage: …`, `Größe: …`). `Termin am 14.03. um 9:30 Uhr` scores 0.130. - **Gate applied to the captured built-in readings of `2026-10-21-001026`:** p2-05 (0.7836) and p2-06 (0.7880) change from `accepted=true` to `false`. None of the other 16 items is accepted. - **Optional `HANDWRITING_CAPTURE` replay** (no vault words, reported but not asserted): built-in mean CER is 94.8% captured vs 94.5% replayed, WritePad 29.6% vs 30.7%. On p2-06, WritePad now replays `"Fragi 'oju odor namr..c side # 14)"` (50.0% CER). The replay prints CER only, not confidences. ## Follow-ups None. WritePad's accuracy on this hand and the neural reader stay in #14.
daniel merged commit 67092a62f9 into main 2026-10-21 00:25:52 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
daniel/obsidian-remarkable-round-trip!35
No description provided.