All posts Research

The hardest part: reading a hand it has never seen

VKVishnu KosuriFeb 20265 min read
In short

Handwriting varies so much between people and devices that a model trained on one set of writers often fails on the next, a gap known as domain shift, and a leading reason lab results don't survive real users.

Domain adaptation, including domain-adversarial training that erases writer-specific quirks, generalises a model to new hands with little or no new labelled data, the difference between a demo and a product.

The short version. The full post has the detail and the why.

In this post

  1. Closing the gap without new labels
  2. Key takeaways

A handwriting model can look brilliant in the lab and fall apart in the real world, for a simple reason. The hands it learned from are not the hand in front of it now.

Everyone's writing is a fingerprint. The angle you hold the pen, your speed, the pressure, the idiosyncratic way you loop an "l" or cross a "t", all of it varies enormously from person to person, and even device to device. A model trained to reconstruct strokes from the motion of some writers, on some pens, quietly assumes the next writer will look statistically similar. When they don't, accuracy drops. Researchers call this gap covariate shift, or more broadly, a change of domain.

Closing the gap without new labels

You could, in theory, fix this by collecting labelled data from every new writer, but that defeats the purpose of a tool meant to just work. The more elegant answer is domain adaptation: techniques that let a model trained on one distribution perform well on a new one, using little or no labelled data from the new domain.

A 2024 ICDAR workshop paper by Florent Imbert and colleagues applies this idea directly to handwriting trajectory reconstruction from inertial pens. A common ingredient in this family of methods is domain-adversarial training: alongside its main task, the network is pushed to learn features that a second "discriminator" cannot use to tell which domain a sample came from. If the discriminator can't distinguish writer A from writer B in the model's internal representation, then whatever the model learned is general, not specific to the writers it trained on.

BEFORE: two writers, two clusters AFTER: aligned, one space
Domain adaptation pulls the internal representations of different writers into a shared space, so a model trained on one hand keeps working on another it never saw.

It's a humble-sounding goal with outsized importance. Generalisation is the difference between a demo and a product, between a system that flatters its own training set and one a stranger can pick up and trust on the first try.

Key takeaways

  • Handwriting varies so much between people and devices that models often fail to transfer.
  • This "domain shift" is a leading reason lab results don't survive contact with real users.
  • Domain adaptation generalises a model to new writers with little or no new labelled data.
  • Domain-adversarial training learns representations that erase writer-specific quirks.

For Vahini, every new user and every new pen is a fresh domain. Treating generalisation as a first-class problem, not an afterthought, is what lets the analyser stay honest across thousands of different hands and five different scripts, instead of only the ones it happened to learn from.

The paper

Domain Adaptation for Handwriting Trajectory Reconstruction from IMU Sensors

Florent Imbert, Romain Tavenard, Yann Soullard, Eric Anquetil. ICDAR 2024 Workshops (ADAPDA); Springer LNCS vol. 14935.

Read the paper

Summary and interpretation are our own. We link the original work so you can read it in full; we don't reproduce it.

Read next