A handwritten letter can be described two completely different ways. As a picture, dark pixels on a light page. Or as a recipe, this stroke, then that one, in this order, at this speed. Both describe the same "a". Neither, alone, is the whole story.
Modern AI is very good at the first description. Vision-language models, the same family powering today's image chatbots, read photographs fluently. So it's tempting to point one at a page of handwriting, render the ink as an image, and ask it to do optical character recognition. As recent work shows, this naive approach reads handwriting surprisingly poorly. A model trained mostly on printed text and natural scenes stumbles on the loops and ligatures of a real hand.
Give the model the recipe, not just the photo
A 2024 study from Google researchers, led by Anastasiia Fadeeva, proposes a fix that is elegant precisely because it changes the representation rather than the model. They encode digital ink, "online" handwriting captured as it's written, as a tokenized sequence that carries both views at once: the strokes laid out as a time-ordered sequence the model can read like text, and the same strokes rendered as an image. The model sees what the writing looks like and the order and motion that produced it.
The payoff is twofold. First, this combined representation matches or beats specialised online handwriting recognisers built for exactly this task, while running on off-the-shelf vision-language models with no architectural surgery. Second, it generalises: the authors show it working across two different VLM families and several public datasets. A careful ablation pins down which parts of the representation do the heavy lifting, evidence, not just a result.
Why two views help
The image view captures shape and proportion, the things a reader sees. The stroke view captures dynamics, the order, direction and rhythm a reader never sees but that disambiguate look-alike letters and messy joins. Together they resolve cases each would get wrong alone. It's the same reason a teacher who watches a child write learns more than one who only grades the finished page.
Key takeaways
- Rendering handwriting as an image and running OCR with a general model works poorly.
- Encoding ink as both a stroke sequence and an image lets off-the-shelf models read it well.
- The approach matches specialised recognisers across multiple models and datasets.
- Shape and dynamics are complementary; combining them resolves ambiguous letters.
Vahini lives on exactly this complementarity. The analyser reads the image today, across five Indic scripts. The sensor pen adds the stroke-order view. The research is a clear signal that the two views together, not the photo alone, are where accurate, script-flexible reading is headed.
Representing Online Handwriting for Recognition in Large Vision-Language Models
Anastasiia Fadeeva, Philippe Schlattner, Andrii Maksai, Mark Collier, Efi Kokiopoulou, Jesse Berent, Claudiu Musat. arXiv:2402.15307 (2024).
Read the paperSummary and interpretation are our own. We link the original work so you can read it in full; we don't reproduce it.