feat(ocr): handwriting test sheet — copy prepared text on the tablet, save the ink to the vault for evaluation #34
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
daniel/obsidian-remarkable-round-trip!34
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/handwriting-test-sheet"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Related to #14. Do not close #14: it stays parked. This MR builds the measurement tool its revisit gate condition 2 asks for, "per-reader CER on the owner's real cursive".
Why
On #14 the owner says the plugin's readings of their handwriting are "mostly garbage". They asked for training/verification templates: prepared text they copy by hand on the reMarkable, with the result stored in the vault in a format that keeps the ink strokes, so they can attach it to a forge issue. Today an agent can't measure anything on the real hand. The only CER we have (21.6%, WritePad) comes from synthetic bundled-font strokes (
src/__tests__/writepad.test.ts). The owner's real ink only exists inreMarkable/.sync/samples.jsonand per-remarkink/*.json, and neither has ground-truth text.The repo already has nearly all the pieces. This MR adds a third sheet kind alongside the letter sheet and the word sheet.
What to build
1. A test sheet layout:
src/ink/handsheet.tsSheetKind = "letters" | "words" | "test". InbuildSheet, add atestbranch:writeandwraphelpers, and wrap long prompts;≈ 230 * unit. The implementer picks the value; Paper Pro Move must still get ≥ 2 rows per page (rowsPerPagealready guarantees ≥ 2).Handwriting test ${p}/${n}. Instructions: "Copy each line once, below it, the way you normally write — joined-up if that is how you write. Don't correct the text to make it neater. Skip any you like."export const TEST_SHEET_PROMPTS: Record<number, string[]>withTEST_SHEET_VERSION = 1. Keep the old versions buildable, under the same rule asSHEET_VERSION: a sheet waiting on the tablet is read back with the layout it was sent with. Starting set for v1. The implementer may tweak it, but should keep the mix of DE/EN, digits, punctuation and jargon, and keep it ≤ ~3 pages on rm2:learnWordsandlearnHandboth inline. Turn it into an exportedboxInk(pending, pages, kind): Map<SheetBox, Stroke[]>, whereStrokeisincoming/rmlines.ts, so the full stroke is kept: tool, color, thicknessScale, and per-point x/y/width/pressure.learnWordskeeps its current output by mapping down to{x,y}.learnHandmay use the helper, or be left alone if the refactor gets awkward. This is the only behaviour-preserving refactor allowed; the existingink.test.tsandsamples.test.tssheet tests must stay green.PendingSheet.kindbecomes"words" | "test" | undefined.2. The capture format: new pure module
src/ocr/testcapture.tsThis module has no Obsidian imports, so it can be unit-tested and replayed headless.
points: number[], 4 numbers per point), likeInkStrokes.pointsinink/inkimage.tsstores its flat per-point values. A full sheet then stays at a few hundred KB, which is attachable.samples.jsonword samples, note paths or remark text. The file holds only the prepared prompts, the owner's ink of those prompts, the hand library (their letter shapes, which are already shared in.sync) and readings. Note this in the docs.3. Commands + wiring:
src/inkedge.ts,src/main.ts, settingsMirror the word sheet everywhere it appears.
grep -n pendingWordSheet srclists every touch point:settingsmodel.ts:87/159: addpendingTestSheet: PendingSheet | null(defaultnull).sync/syncstate.ts:59-63SharedHandwritingandmain.tshandwritingOf(≈2446) /adoptHandwriting(≈538-550): share it across vault copies. Read it tolerantly: ahandwriting.jsonwritten before this change has no field, so mapundefinedtonull.main.ts:809: add it toknown, so the sheet is never imported as a new notebook.inkedge.ts:405detectPageSize: add it toours.inkedge.ts: addsendTestSheet(api, parentId), a copy ofsendWordSheetwith title"Handwriting test".main.tscommands, next to the word-sheet ones at ≈308-315:send-handwriting-test-sheet: Send handwriting test sheet to reMarkable, a copy ofsendWordSheet()(≈2100);save-handwriting-test-sheet: Save handwriting test sheet to the vault. This command:readNotebook) and callsboxInk. If no box has ink, show a notice like the word sheet's: "Nothing written on the test sheet yet…";recognize(strokes, await this.ocrTemplates(), lexicon, await this.wordReader())and, ifocrUseWritePad,(await this.writePad())?.read(strokes, lexicon)directly. Do NOT callrecognizeLocally/rememberReading/addSample. The sheet must stay held-out, or it contaminates the sample store and every later evaluation. RecordacceptedagainstOCR_TRUST/WRITEPAD_TRUST, asrecognizeLocally(≈1737) decides it. Also record the raw text when it was not accepted: that is the whole point. IfocrMode === "endpoint", also callthis.ink.ocr(http, png, undefined, false, [])with no worked examples and record that reading. An endpoint failure recordsnulland must not abort the save;strokesToPng(≈1715);<handwritingFolder>/test-sheets/<YYYY-MM-DD-HHmmss>/:handwriting-test-<stamp>.json: theHandwritingCapture. This is the one file to attach to an issue;p1-03.png…;Handwriting test <stamp>.md: a table of prompt | built-in | WritePad | (endpoint) with per-row CER, the mean CER per reader, the embedded PNGs, and a line saying "Attachhandwriting-test-<stamp>.jsonto the forge issue.";settings.ts≈165-180: next to the word sheet, add a "Handwriting test" row with Send and Save buttons and a description, and "A test sheet is waiting on the tablet." when one is pending.4. Offline replay: what an agent runs on an attached file
src/__tests__/testcapture.test.ts:cer,summarizeandcaptureMarkdownon a small synthetic capture. Build the capture frombuildSheet(DEVICE_PROFILES.rm2, "test")boxes, with prompt strokes written in theallurefont via thewrite()helper pattern fromwritepad.test.ts;.rm. Take the sheet's pages plus user strokes inside boxes, the same way the existing sheet tests inink.test.ts/samples.test.tsfake a filled sheet. Then check thatboxInkputs each box's strokes under the right prompt and that the JSON survivesJSON.parse(JSON.stringify());process.env.HANDWRITING_CAPTURE(describe.skipIf(!process.env.HANDWRITING_CAPTURE)). It loads that file, re-reads every item with today'srecognize(templates fromcapture.handif present, else readability, asocrTemplatesdoes) andWritePad, without a lexicon, and prints a per-reader table of captured CER vs replayed CER plus the mean. It asserts only that the file parses andformat/versionmatch. It must not assert accuracy.5. Docs
.json.docs/ocr-corrections.md: document the capture format, the "held-out, never learned from" rule, and the replay command.CHANGELOG-INK.mdentry.Constraints and decisions
.rmpages. The.rmalso contains the sheet's own strokes, needs the diff to separate them, and isn't human-viewable. The JSON carries the full stroke data (x, y, width, pressure) and the box geometry, which is everything a reader consumes.learnWordsoutput shape,PendingSheetfor existing kinds, theSHEET_VERSIONletter and word layouts (a pending sheet on someone's tablet must still read back).Out of scope
Validation contract
npm ci && npx vitest run src/__tests__/testcapture.test.ts src/__tests__/ink.test.ts src/__tests__/samples.test.ts src/__tests__/writepad.test.ts→ all PASS. The repo has no flake.nix, so use plainnpx, notnix develop.npx vitest run(full suite) → PASS;npm run build(includesscripts/check-bundle.mjs) → exit 0.buildSheet(profile, "test"),boxInkandsrc/ocr/testcapture.tsdon't exist, so the new test file fails to compile and import. After it, the round-trip test must find every prompt's strokes under its own prompt. Assert the per-box stroke counts equal what the fixture wrote, not just "> 0", so the test is not vacuous.HANDWRITING_CAPTURE=<tmp> npx vitest run src/__tests__/testcapture.test.ts. It must print a non-empty table with one row per reader. Paste that output into the MR.🤖 Generated with Claude Code
✅ Agent run completed
15m56s · 76 turns · $3.47 · pushed
agent/handwriting-test-sheet· session saved — new comments resume this run⚠️ Verification gate not satisfied: awaiting intent review. Outcome parked — not marked done.
claude referenced this pull request2026-10-20 23:27:44 +02:00
What changed
src/ink/handsheet.ts): newSheetKind"test"withTEST_SHEET_PROMPTS/TEST_SHEET_VERSION = 1, using the 18 prompts as given. Each row prints the prompt in black (wrapped when it's long) above one writing line with the v2 guides. Boxes are 230 units high: 3 pages on rm2 and Paper Pro, 4 on Move, 5 rows per page on Move. New exportedboxInk(pending, pages, kind)keeps the fullStroke.learnWordsandlearnHandboth use it now and keep the same output and key order. The existing sheet tests pass unchanged.src/ocr/testcapture.ts, no Obsidian imports): theHandwritingCapture/CaptureItemtypes with flat x,y,width,pressure points, pluscer(moved here;writepad.test.tsimports it),summarize,captureMarkdown,captureItems,replayTemplatesandparseCapture.pendingTestSheetadded to settings,SharedHandwriting(an oldhandwriting.jsonwithout the field reads asnull, with a test),adoptHandwriting,handwritingOf,knownanddetectPageSize. AlsosendTestSheet/readTestSheetininkedge.ts, two commands, and a settings row with Send / Save buttons. Save never callsrecognizeLocally,rememberReadingoraddSample, and the endpoint gets[]worked examples. Docs added to the README,docs/ocr-corrections.mdandCHANGELOG-INK.md.Decisions you may want to redirect:
read()returnnull. When that happens, the save stores the reading made without the lexicon, markedaccepted: false. Otherwise the text that was rejected would be lost.itemCer). A sentence that runs onto a second row comes back with a\n, which shouldn't count as a misreading. Long prompts like the 63-character "Zwölf Boxkämpfer…" may not fit one line at normal writing size, so the box leaves room below the line.ceritself is unchanged.summarizehas an extrafailedcount for endpointnullreadings, which are left out ofmeanCer.Verify
npx vitest runpasses: 775 tests, 1 skipped.npm run buildexits 0, includingcheck-bundle, andeslint src/is clean.origin/main, the new test file fails withCannot find module '../ocr/testcapture'. The round-trip test checks that every prompt's stroke count equals what the fixture wrote. As a control, I narrowed the box hit-test inboxInk, and it failed (expected 25 to be 26).HANDWRITING_CAPTURE_OUT=/tmp/hw-capture.json npx vitest run src/__tests__/testcapture.test.ts(488 KB for 17 lines), then ranHANDWRITING_CAPTURE=/tmp/hw-capture.json npx vitest run src/__tests__/testcapture.test.ts. The output, cut after the first 6 of 34 item rows:Captured and replayed match because the synthetic file was read without a lexicon too. These numbers come from bundled-font strokes, not a measurement of real handwriting.
Follow-ups
✅ Agent run completed
11m7s · 19 turns · $1.00
Acceptance-intent assessment
This MR closes no issue: it is "Related to #14", which stays parked. So I checked the diff against two things: the owner's request on #14 (note 45305: prepared text to copy by hand, stored in the vault in a format that keeps the ink strokes, attachable to a forge issue) and every item of this MR's own spec. The diff delivers all five sections of the spec. I re-ran the four contract test files plus
syncstate.test.tslocally: 84 passed, 0 failed, 1 skipped (the replay block, which is gated onHANDWRITING_CAPTURE).npm run buildexits 0, includingcheck-bundle.Criteria
Owner request (#14)
handwriting-test-<stamp>.jsonto the forge issue."§1 Sheet layout (
handsheet.ts)SheetKindnow includes"test". The test branch has one full-width column, the prompt in black wrapped viawrite/wrapabove one writing line, and the v2 guides (dashed cap line, dotted x-height, solid baseline, dotted descender). The box is 230·unit high. Header and instructions match the spec.TEST_SHEET_PROMPTS/TEST_SHEET_VERSION = 1use the spec's list verbatim. Older versions stay buildable throughpending.version. The test asserts ≥ 2 rows on every device and ≤ 3 pages on rm2.boxInk(pending, pages, kind)is exported and keeps the fullStroke.learnWordsmaps it down to{x,y}with the same output shape.learnHanduses it too. The existingink/samplestests stay green.PendingSheet.kindis now"words" | "test" | undefined. Letter and word layouts are untouched; the test assertsbuildSheet(profile)equals the v2 letters.§2 Capture format (
ocr/testcapture.ts)CAPTURE_FORMAT,HandwritingCaptureandCaptureItemmatch the spec's shape. Points are stored flat, 4 per point.cermoved here, andwritepad.test.tsnow imports it.summarizeandcaptureMarkdownare present; extras such as thefailedcount and an optionaljsonNameparameter are additive.hand(aHandLibrary) and the readings: no lexicon, samples, paths or remark text. The README,docs/ocr-corrections.mdand the CHANGELOG all say so.§3 Commands + wiring
pendingTestSheetis added to the settings model and defaults,SharedHandwriting,handwritingOfandadoptHandwriting. It is read tolerantly (undefined → null), and a test covers that.knowninmain.tsand tooursindetectPageSize.sendTestSheetininkedge.tsuses the title "Handwriting test".send-handwriting-test-sheetandsave-handwriting-test-sheetcommands exist. Save does the following:recognize(strokes, ocrTemplates(), lexicon, wordReader())andwritePad.read(strokes, lexicon)directly. It never callsrecognizeLocally,rememberReadingoraddSample.acceptedagainstOCR_TRUST/WRITEPAD_TRUST, and keeps the raw text even when the reading was not accepted.ink.ocr(http, png, undefined, false, []). A failure is caught and stored asnull.strokesToPng, plus the JSON and the note under<handwritingFolder>/test-sheets/<stamp>/.§4 Offline replay tests
cer,summarizeandcaptureMarkdownon a synthetic capture built frombuildSheet(rm2, "test")with prompts written in theallurefont..rm, then asserts the per-box stroke count equals what the fixture wrote and that the prompt order is exact, with a skipped box included. It also checks that the JSON survivesJSON.parse(JSON.stringify()). This is not vacuous: the MR note reports that narrowing the box hit-test inboxInkmakes it fail.describe.skipIf(!HANDWRITING_CAPTURE). It reads with no lexicon and builds templates fromcapture.hand, falling back to readability/hershey (the same base set asocrTemplates). It prints captured vs replayed CER per reader and per item, and asserts only the format and version. The positive-control output is pasted in MR note 45325.§5 Docs
.json) and a row in the command table.docs/ocr-corrections.mdcovers the capture format, the "held out, never learned from" rule, the privacy rule and the replay command. There is aCHANGELOG-INK.mdentry.Constraints / out of scope
Notes for the owner, none of which fails a requirement:
null, the stored reading is the one made without the lexicon, marked not accepted.itemCercollapses whitespace before comparing.acceptedwhenever its text is non-empty.Verdict
Approved