OCR Direction 3: trainable in-plugin online-HTR — PARKED (no-go on current evidence) #14
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Depends on
#20 OCR Direction 2: join/ligature templates + WordReader synthesis of unseen words (the creative bet)
daniel/obsidian-remarkable-round-trip
Reference
daniel/obsidian-remarkable-round-trip#14
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #5
Part of #5 (Direction 3). PARKED / no-go per the #5 spike. Held (un-dispatched) as a tracked decision with an explicit revisit gate — do not start an agent on this unless the gate below is met.
Why no-go (measured, from the #5 spike)
Direction 3 is "a small stroke-sequence model trained in the plugin (tfjs-wasm / ONNX-web) that beats WritePad + the local VLM on the owner's corrections, within an Obsidian bundle/runtime budget." On the evidence:
rmapi-jsonly). tfjs-wasm or onnxruntime-web add multiple MB + a WASM runtime to an Obsidian plugin, plus an in-plugin training loop. Research for #2 already found no off-the-shelf package that is both browser-ONNX/tfjs and on-device-trainable — this is build-not-adopt.Expected marginal value low, cost/risk high → stop at Directions 1–2, as the #5 scope boundary explicitly permits.
Revisit gate (dispatch only if ALL hold)
If revisited, the first step is still a feasibility spike (can a tiny CTC-over-stroke-features model trained on sheets + a few hundred corrections beat 21.6% CER within a ≤~a-few-MB bundle budget), not implementation.
claude referenced this issue2026-10-20 15:25:42 +02:00
claude referenced this issue2026-10-20 15:26:31 +02:00
claude referenced this issue2026-10-20 15:26:32 +02:00
claude referenced this issue2026-10-20 18:52:05 +02:00
claude referenced this issue2026-10-20 18:52:06 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
claude referenced this issue2026-10-20 19:00:28 +02:00
✅ Agent run completed
7s · 2 turns · $0.94 · session saved — new comments resume this run
What I found
An agent was dispatched on this parked issue, but the revisit gate is not met. None of the three conditions holds today:
maintoday (7a021f8), and #28 (v2: installer download and LM Studio) is in progress. @daniel's only feedback on that path was "calling ollama is acceptable" (note on #8). Nobody has said the VLM is rejected for privacy, latency or setup friction.ocr/samples.jsonand #4's per-reader surface (!9) on real remarks. Agents can't access that data, and no owner report has come in.Created
None. No code, MR or child issue: the gate is not met, so per this issue's own terms there is nothing to build or spike.
Follow-ups
Since my last update: #20 has closed. It came back no-go: owner-hand synthesis from letters and joins did not beat fonts-only on held-out words. The measurement is in #21. This releases the blocked-by link, but the revisit gate is still not met, so I'm not starting a spike or opening any MR or child issue. Gate status now:
main, and #28 (v2) is in progress.WordReader's font pass is never trusted (correct-word cost 0.22–0.40 againstFONT_TRUST = 0.08), so it reads only from your stored word samples. It remains unmeasured on your real data.So the only open gate conditions (1 and 2) depend on your judgment or your data, which no agent can supply. Since there's no automated blocker left, I'm handing this to you so the issue stays parked instead of waking agents.
To unpark it, comment here with either of these, and re-assign the agent:
Or simply tell me to explore the neural reader regardless. Even then, the first step stays a feasibility spike: a tiny CTC model over stroke features, a bundle of a few MB at most, that has to beat 21.6% CER.
Agent completed ✅
There's nothing new on #14 since my handover, so I posted nothing and changed nothing.
The newest note on the issue is still my update after #20 closed. Nobody has replied, and none of the three conditions for restarting the work has changed:
The issue is assigned to you, not the agent, so it should stay parked. To restart it, comment on #14 with one of the three unpark options from my last note and reassign the agent.
@claude yeah i feel what the plugin currently suggests for my handwriting is mostly garbage. but to better work and train/verify that can we maybe add some training templates where i can copy text you prepared? it shall be stored in the notes vault ina format that retains the ink strokes, so i can easily attach it in the forge issue
✅ Agent run completed
3m19s · 28 turns · $1.08 · pushed
agent/handwriting-test-sheetWhat I found
src/ink/handsheet.ts). A test sheet of prepared sentences fits that same mechanism as a third kind.Created
<handwriting folder>/test-sheets/<stamp>/containing:handwriting-test-<stamp>.json: your full strokes per line (x, y, width, pressure), the prompt text, the page size, your learned letters, and what each reader (built-in, WritePad, and the vision endpoint if you use it) read, even when the reading was too unsure to accept. Attach this one file to the issue.HANDWRITING_CAPTURE=<file> npx vitest run src/__tests__/testcapture.test.ts.Follow-ups
.jsonhere. The next run on #14 can then measure real error rates per reader and check gate condition 2 properly.claude referenced this issue2026-10-20 23:43:24 +02:00
claude referenced this issue2026-10-20 23:43:24 +02:00
claude referenced this issue2026-10-20 23:56:02 +02:00
@claude i pushed the handwriting samples to the repo, any other closed issuesworth repoeningnow?
✅ Agent run completed
4m44s · 26 turns · $1.47 · pushed
agent/builtin-symbol-junk-gateWhat I found
handwriting-samples/test-sheets/2026-10-21-001026/, 18 lines of joined cursive) confirms "mostly garbage":/IäI→✗rl…,%«%.Jl€at 0.784 confidence and p2-06@:@+m«↑@&↑3]at 0.788. Both clearedOCR_TRUST = 0.78, which #4 calibrated on synthetic fonts only. An accepted built-in reading also means WritePad is never asked.Created
Follow-ups
handwriting-samples/(e.g. "But Rendering Bad / Remark it") have no text labels, so nothing can be measured from them. If you add a<same-name>.txtnext to some of them with the intended text, they become a second test set of real remarks, which is useful for the vision-model comparison too.claude referenced this issue2026-10-21 00:26:07 +02:00
claude referenced this issue2026-10-21 00:26:07 +02:00
@claude tried the setup ollama for me, it downloaded, then i clicked test endpoint and it showed an error 400 from the api, something about invalid args or something
❌ Agent run failed
6m35s
Agent failed ❌
agent job failed: container agent OOMKilled; container agent OOMKilled
@claude updated the samples in the repo with ollama results
✅ Agent run completed
7m45s · 65 turns · $1.97 · pushed
agent/fix-ocr-probe-png,agent/capture-endpoint-model-latencyWhat I found
Ollama reads your hand about 3× better than WritePad. Same 18 lines; the ink in
2026-10-21-004942is byte-identical to the first capture:Its weak spots:
*quick*) and quotes you didn't write (p2-06).The 2 junk lines the built-in reader "used" (p2-05, p2-06) come from your installed build, which predates !35. On current
mainboth are rejected (symbol share 0.59 and 0.83, limit 0.30). Rebuild and the leak is gone.Your "Test connection → 400" is a real bug, now fixed. The tiny probe image the button sends was a corrupt PNG: one wrong base64 character, so the image data failed its CRC check, and Ollama rejected it. Imports send real images, which is why reading works.
#14 gate: conditions 2 and 3 are met. Condition 1 is not: the local VLM is clearly the accuracy leader. An in-plugin model trained on a few hundred samples would have to beat 10.5% CER (not 21.6% or 29.6%), which is very unlikely. #14 stays parked.
Created
Follow-ups
claude referenced this issue2026-10-21 01:05:54 +02:00