OCR Direction 2: keep the N most diverse word samples, not just the newest #16
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip!16
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/ocr-sample-diversity"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #13 (Direction 2, store schema & caps — issue #13 part 1 slice + part 3 word-sample retention). Independent of !12; touches only
src/ocr/samples.ts, so it can land in parallel with the scaling MR (!15) and with !12.Why
addSample(src/ocr/samples.ts:119) keeps only the newestPER_WORD = 3samples per word:The #5 spike showed accumulated samples clearly help, and the scaling headroom is there (DTW ran the full labelled set in well under a second). Keeping the newest 3 is wasteful when the owner writes a word many times: three near-duplicate recent samples cover one way of writing it, while an earlier, differently-slanted/joined instance that would match a future remark is discarded. The whole-word reader (
WordReader,src/ocr/words.ts) checks owner samples first, so sample diversity directly drives recall on the owner's own hand.What to change (
src/ocr/samples.tsonly)PER_WORD3 → 8–12; justify against the latency budget the scaling MR !15 establishes — note the two MRs are complementary: !15 bounds read-time work so a larger store stays in budget, this MR grows the store without keeping junk)..slice(-PER_WORD)inaddSamplewith diversity-preserving retention: when a word is over cap, drop the sample that is most redundant (closest to another kept sample) rather than the oldest, so the kept set spans the owner's variation. Use a cheap shape distance for "closest" —traceOf+dtwfromwords.ts, or acloudOf/matchDistanceproxy fromrecognize.ts. Keep newest-last ordering for callers that rely on recency, and keep the existing early-return for empty word/strokes.Keep
addSample's signature and theSampleStore.words: Record<string, Point[][][]>shape unchanged — this is retention policy only, no schema/format change (so no migration needed; a store with ≤3 samples stays valid).Scope boundary
WordReaderinternals — richer-store child issue, after !12.REMARKS_KEPT/CORRECTIONS_KEPT) — leaveprunealone.Validation contract
npx vitest run src/__tests__→ PASS, plus a new test insrc/__tests__/samples.test.ts(or a new file) that:PER_WORDare near-duplicates of one style and an earlier sample is the only cover for a second style; a held-out read of the second style viaWordReaderseeded fromstore.wordsis wrong under newest-3 retention. With diversity retention it reads correctly → the test passes only after the change.(No
flake.nix; usenpm install+npx vitest.)claude referenced this pull request2026-10-20 15:32:54 +02:00
🤖 Agent is working on this — turn 35
Last activity: assistant message — updated 14:34:41 UTC
Last message:
claude referenced this pull request2026-10-20 15:34:21 +02:00
claude referenced this pull request2026-10-20 15:34:22 +02:00
What changed
src/ocr/samples.ts: raisedPER_WORD3 → 10 and replaced newest-wins.slice(-PER_WORD)inaddSamplewith diversity-preserving retention (keepDiverse) — when a word is over cap, the most redundant sample (nearest in$Pcloud distance, viacloudOf/matchDistancefromrecognize.ts) is dropped, not the oldest. Survivors keep newest-last order; ties drop the older sample so recency still breaks even cases.addSample's signature andSampleStore.wordsshape are unchanged, so an old ≤3-sample store stays valid — no migration.prune/remark caps untouched;WordReaderinternals untouched.src/__tests__/samples.test.ts: newdiversity-preserving sample retentionblock + rewrote the old "keeps three samples" test for the new cap.Verify
Red gate confirmed: with
PER_WORDbumped but retention left as newest-wins, the new diversity tests fail at thetrustedassertion (the earlier distinct-style sample is evicted); after switching tokeepDiversethey pass. The test seeds aWordReaderoverstore.wordsand reads a held-out remark in a second style whose only cover is an earlier sample — read correctly (trusted:true) only under diversity retention. Also asserts kept count respects the cap and that a near-duplicate never evicts a distinct sample. Full suitenpx vitest run src/__tests__→ 671 pass;tsc -noEmitandeslintclean. (No MR test pipeline is configured for this repo —.forgejo/.githubworkflows are scheduled/tag-triggered only; validation is the local vitest contract.)Follow-ups
None. Scope boundaries respected — read-time shortlisting/latency caps remain !15's; richer letter/join templates remain the post-!12 child issue.
@claude resolve merge conflicts
claude referenced this pull request2026-10-20 16:34:09 +02:00
claude referenced this pull request2026-10-20 16:34:09 +02:00
Since my last update: resolved the merge conflict with
main.mainhad advanced with !15 (read-time scaling) and the Direction-1 letter-learning work, both of which also touchsrc/ocr/samples.ts— the only conflicting file. I mergedorigin/maininto the branch and unioned the overlap: the import now pulls bothcloudOf/matchDistance(diversity) andcutLetters(letter learning);PER_WORD = 10andkeepDiverseare kept alongside main'sPER_LETTER/LETTER_PLACESandlearnLetters.addSamplestays word-only andlearnCorrectioncalls both, matching main's split — no logic lost on either side. No schema/format change.Verify:
tsc -noEmitclean,eslintclean on the changed files,npx vitest run src/__tests__→ 677 passed (60 files), including the diversity red-gate test. Merge commite128695pushed; MR is mergeable again.claude referenced this pull request2026-10-20 17:08:52 +02:00
claude referenced this pull request2026-10-20 17:10:50 +02:00
claude referenced this pull request2026-10-20 17:14:50 +02:00
claude referenced this pull request2026-10-20 17:14:50 +02:00
claude referenced this pull request2026-10-20 18:51:57 +02:00
claude referenced this pull request2026-10-20 18:52:05 +02:00
claude referenced this pull request2026-10-20 18:52:06 +02:00
claude referenced this pull request2026-10-20 18:59:13 +02:00
claude referenced this pull request2026-10-20 18:59:13 +02:00
claude referenced this pull request2026-10-20 18:59:28 +02:00
claude referenced this pull request2026-10-20 19:00:17 +02:00
claude referenced this pull request2026-10-20 19:00:28 +02:00
claude referenced this pull request2026-10-20 19:00:28 +02:00