Preliminary Validation of a Cursive Handwriting Reconstruction Algorithm from a Sensorized Ink Pen

Lack of handwriting automatization in childhood can cause difficulties within and outside the school context. Therefore, the objective quantification of the handwriting performance is key. A Smart Ink Pen (SIP) used on paper demonstrated its validity in characterizing primary school children's handwriting process, although missing information on the handwriting product. To overcome this limitation, a trace reconstruction algorithm was developed, based on the force and IMU signals measured by the SIP. A total of 353 words "uno", written in cursive from the BVSCO-3 battery by 50 Italian students of the 5th grade of primary school, was reconstructed. The quality of the reconstructions was validated through an Optical Character Recognition (OCR) algorithm (Google Vision), using the scans of the actual traces as a reference. The character recognition rates were 81.02% and 59.49%, the character error rates 21.48% and 47.65%, for scans and reconstructions, respectively. A deeper analysis revealed that 15% of the reconstructions were read in the opposite direction by the OCR algorithm, likely due to a non sufficient sampling rate for the last portion of the words. A characterization of the differences between the good, bad and opposite reconstructions allowed to identify some directions of improvement. An increase of the SIP sampling rate, a better modeling of the thickness of the trace, a finer estimation of the relative distance between the IMU sensor and the tip and the reconstruction of the tip trajectory during in-air movements could improve the trace reconstruction algorithm. In addition, the possibility to leverage transfer learning approaches on the OCR algorithm using the reconstructed traces for additional training could further improve the performances in terms of character recognition rate. However, the preliminary results obtained are promising and highlight the possibility of reconstructing handwriting traces while maintaining the naturalness of the gesture.Clinical relevance- This contribution establishes the possibility of reconstructing handwritten traces from the kinematic and force signals recorded by an ecological sensorized ink pen.