OCR Dir-2: decisive spike — bigram-tiling owner-hand synthesis + a fair held-out measurement (gates the join MRs) #21
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Blocks
#20 OCR Direction 2: join/ligature templates + WordReader synthesis of unseen words (the creative bet)
daniel/obsidian-remarkable-round-trip
Reference
daniel/obsidian-remarkable-round-trip#21
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #20
Part of #20 (and #17 part 2). This is the decider for the join/ligature synthesis bet. Build it and measure; ship the schema + synthesis MRs only if synthesis beats fonts-only on a hand where
WordReaderis the operative reader. A clean "no" is a valid, shippable outcome (report it, ship nothing).Why this spike exists (investigation on #20)
The issue's suggested path — "stitch owner letter traces + join traces to read an unseen joined word" — rests on a premise the code falsifies. A throwaway probe over
letterlearning.test.ts's font-as-hand methodology (TRAIN fed as corrections, held-out read viaWordReader) found:Three structural facts fall out:
store.lettersis populated only vialearnLetters→cutLetters(src/ocr/samples.ts:199,src/ocr/recognize.ts:588), which returnsnullfor joined cursive by design. So letter templates exist for separable hands and are absent for the joined hands whereWordReaderis the reader in play.WordReaderfonts-only scores CER 1.000 / 0 trusted (a separable trace can't match the joined bundled script fonts). But separable hands are already generalised by the letter reader (recognize+lettersTemplates, shipped in !12 — see the existingletterlearning.test.ts). So "synthesis beats fonts-only=1.0" there beats a path that isn't the operative reader and is redundant with !12.store.lettersis empty (so letter-based synthesis can't run), and fonts-only already reads ~0.35–0.45 because the test "hand" is itself a bundled script font — an unrealistically easy baseline.Net: letter material exists only where
WordReaderisn't operative, and is absent where it is. The measurement #20 specifies can't stage the bet fairly as written.What this spike must do
1. Redesign synthesis as bigram-tiling (not letters + connector)
Because single-letter material is unavailable for joined hands, define the stored join = the whole labelled bigram shape in cap units (both letters + their ligature), keyed by pair (
"th"). Synthesise an unseen word's candidate trace by tiling overlapping/abutting bigrams in reading order (unigram letter templates only fill word edges / unpaired gaps), placed by width/advance, then run throughtraceOf(src/ocr/words.ts:84). Decide the fallback for a pair with no captured bigram (the font's join for that pair? a straight connector? skip the candidate) and how to pick among multiple samples per bigram.2. Solve labelled-bigram harvesting honestly — or conclude it can't be
splitLineInto(src/ocr/recognize.ts:571) cuts even allure into ⌈n/2⌉ clusters (probe: 65/65), but at widest gaps with no guarantee a cluster covers exactly two adjacent, known letters. Blindly assigning clusters→bigrams is the exact "guess which ink is which word → teach the wrong shapes" hazardlearnCorrectionalready refuses. The spike must only accept a bigram when the cut provably covers exactly two adjacent letters (needs a boundary signalcutLettersdoesn't give today), e.g. by restricting capture to the semi-joined regime where some single cuts are clear and the unclear ones collapse exactly two letters. If no reliable labelling exists, that is the "no" — report it.3. Build a fair measurement (the whole point)
WordReaderis the reader in play, NOT a separable hand (CER 1.0 straw man). The existing 4 fonts have no semi-joined member, so either (a) construct a semi-joined hand (e.g. compose separable glyphs with partial joins / intermittent slant), or (b) seed a controlled synthetic store with known owner letters+bigrams and read held-out words rendered in a matching joined hand.4. Latency
Synthesis is per-candidate like
readFonts, so it MUST reuse the profile+width shortlist (SAMPLE_SHORTLIST, the 150-candidate cap inreadFonts,src/ocr/words.ts:256) and thethis.tracescache — read cost stays FLAT in store size (mr-15). A latency assertion within !15's budget (seesrc/__tests__/scaling.test.ts) is part of the contract.Trust threshold
A synthesised owner word is neither a verified owner word nor a foreign font: give it its own threshold between
SAMPLE_TRUST = 0.2andFONT_TRUST = 0.08(src/ocr/words.ts:179).If the spike is positive — suggested MR split
joins?: Record<string, Point[][][]>onSampleStore, alearnJoins(labelled-bigram capture from pair-separable ink) called alongsidelearnLettersinlearnCorrection, aPER_JOINconst, diversity-preserving per-pair cap via the generalisedkeepDiverse(samples, cap)(from !18), routed through themigrate()seam (from !19). Unit test: joins captured from separable ink, bounded by the cap.WordReaderbigram-tiling synthesis + own trust threshold + the fair held-out CER test + the latency assertion. Wirestore.letters/store.joinsinto theWordReaderconstructor (today it only takesstore.words— seesrc/main.ts:1773).Dependencies / ordering
Soft-depends on !18 (generalised
keepDiverse(samples, cap)) and !19 (migrate()seam) — both currently open.joins?is optional/backward-compatible so not a hard blocker, but land the capture MR after !18/!19 merge to avoid conflicts insrc/ocr/samples.ts.Validation contract
npm installthennpx vitest run src/__tests__→ PASS. The synthesis MR additionally ships: a held-out CER test where synthesis beats fonts-only on a joined/semi-joined hand (operative baseline, not CER 1.0), and a latency assertion within !15's budget. (Noflake.nix.)claude referenced this issue2026-10-20 17:21:15 +02:00
❌ Agent run failed
1m23s · session saved — new comments resume this run
Agent failed ❌
agent SDK query failed: Error: Claude Code returned an error result: There's an issue with the selected model (claude-opus-4-8). It may not exist or you may not have access to it.
at om.readMessages (file:///usr/local/lib/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:88:20439)
at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
[runner] spawn env: cwd=/workspace (exists+executable) | executable=/usr/local/lib/node_modules/@anthropic-ai/claude-code/bin/claude.exe (exists+executable) | HOME=/tmp/agent-home
Status note from the #5 planning agent: this spike's last run failed for an infrastructure reason (
claude-opus-4-8model unavailable), not because of anything in the spike itself. Both soft prerequisites named above have now merged: !18 (generalisedkeepDiverse(samples, cap)+ diversity-preserving letter retention) and !19 (versionedmigrate()seam,CURRENT_FORMAT=2). Nothing blocks this spike any more. Please branch off currentmain(71d0788or later) so the capture MR builds on both.claude referenced this issue2026-10-20 18:52:05 +02:00
claude referenced this issue2026-10-20 18:52:06 +02:00
🤖 Agent is working on this — turn 6
Last activity: tool: mcp__gitlab__forge_close_issue — {} — updated 18:08:11 UTC
claude referenced this issue2026-10-20 18:59:13 +02:00
claude referenced this issue2026-10-20 18:59:13 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
What I found
Result: no-go. Nothing ships. The capture and synthesis MRs don't earn their place, because no reliable way to harvest labelled bigrams exists. With perfect labels synthesis would help modestly, but without them it does nothing.
How it was measured. The spike is a throwaway probe on branch
spike/21-bigram-probe, with run logs in.agent-tmp/spike-21/. Not for merge.learnCorrection. 37 held-out words. A lexicon of about 300 words including look-alike distractors.WordReader([allure, hershey-script], readability, store.words). "Synth" means owner samples first, then the cheaper trusted match out of the fonts and the synthesis.Results (mean over 4 owners). Each cell is effective CER / trusted reads out of 148 / wrongly trusted. "Effective" means untrusted reads count as unread.
recognizeCER: oracle synthesis gives a small gain over fonts-only (S1 0.77 → 0.70; J1 0.98 → 0.81). Harvested synthesis is identical to fonts-only on every owner.Why it's a no:
LETTER_GAP. Each cluster can be assigned 1–3 letters by a unique best fit to the expected widths. That labels bigrams only 82–96% correctly (91/100, 82/90, 106/110, 101/105, 96/118, 79/85, 108/116, 78/84). Checking single-letter clusters with the letter reader made it worse (16/24 … 47/53).learnCorrectionrefuses. The only provably labelled bigrams are 2-letter words written as one cluster, and they cover far too few pairs to matter.FONT_TRUSTandSAMPLE_TRUSTonly trusts 5–48% of words.FONT_TRUST = 0.08. So the operativeWordReaderon real joined hands is effectively owner word samples only (an owner rewrite costs about 0.13 and is trusted 63–98% of the time). The #20 table's fonts-only CER of 0.35–0.45 comes from the test hand being a bundled font.Not done: I didn't measure latency, because the gate failed before it mattered. A design would reuse the 150-candidate shortlist and a trace cache, as the probe does.
Created
None. The probe lives only on branch
spike/21-bigram-probe(evidence, no MR).Follow-ups
store.wordscoverage (more confirmed words per remark) is a cheaper win than synthesis.claude referenced this issue2026-10-20 20:06:44 +02:00
claude referenced this issue2026-10-20 20:06:45 +02:00
@claude all done here?
Since my last update: yes, done here. The spike's answer is no-go, and nothing ships. No MRs were opened, and none are pending. The only artefact is the throwaway probe on branch
spike/21-bigram-probe. It is evidence only and is not meant to be merged; you can delete it whenever you like. I'm closing this issue now. The two follow-ups in my report above (a trust rule based on the margin over the runner-up, and growingstore.wordscoverage) are recommendations for #20. I haven't filed them as issues.claude referenced this issue2026-10-20 20:08:53 +02:00