Instant messaging with emotion-embedded vectorized handwritings on mobile devices

Abstract

Generally, handwriting can reflect writers’ personality, thoughts, and emotions, i.e., handwriting can deliver emotion- and sincereness-embedded messages. However, texting messages and notes such as emails and instant messages replace handwriting letters and notes in communication due to the popularity and availability of mobile devices and personal computers. Furthermore, the commonly used input methods and devices also limit handwriting messaging. For example, limited mobile screen sizes make writing multiple Chinese characters difficult. As a result, this work aims at designing and developing a handwriting messaging system based on our handwriting characteristic exploration and texting-handwriting difference discovery. We first discover Chinese texting issues for mobile devices and emotion-delivered effectiveness of handwriting with pilot studies. Then, our emotion-embedded handwriting messaging framework is implemented to record writing strokes on the entire touch screen, vectorize them as Bézier curves, and send them as instant messages. Vectorizing strokes can preserve personal and emotional handwriting features with compact networking traffic without deteriorating the convenience of instant messaging. Finally, we conduct user studies to verify that our handwriting messaging system is preferred while intending to deliver contents with sincerity and emotion.

1 Introduction

With the advance of technology, computer-based personal messages such as emails and instant messages become popular and replace handwriting letters and notes in communication. Mobile devices make them even more universally popular. Users generally choose a preferred systematic style and font for texting, and this makes messages lack of personality, emotion, and sincerity. Because different users may have different handwriting, and even the same person may have different handwriting under different emotions, it is easier to deliver emotion, feeling, and sincerity. Therefore, this work aims at adding handwriting characteristics into instant messaging for more personality and emotion.

Before further describing and discussing the focus and technical details of this article, we define the terms used in this article for clarity. Traditionally, handwriting is used to describe the process of writing with a pen or pencil in the hand, and this work uses it to denote the act of composing digital messages with an electronic device while maintaining writer’s personal writing characteristics including stroke shapes and orders. Handwriting messages and notes are the products of handwriting. Similarly, we use texting to describe the process of composing digital messages encoding with the format for personal computers and mobile devices, typically consisting of alphabetic and numeric characters. Currently, there are three popular input methods including typing, speech recognition [1–4], and stroke recognition [5–9]. We give the details of each input method in Section 3. Texting messages and notes are the products of texting. This work uses messaging/instant messaging to describe the act of sending and receiving electronic messages between two or more users with stationary and mobile devices. Therefore, we use handwriting texting messaging to denote the act of messaging under the contents composed by handwriting/texting respectively.

There are a variety of issues while texting with those commonly used input methods including typing with a keyboard, stroke recognition, and speech recognition. Although typing with a keyboard is still the prevalent method, it requires practicing to get familiar with the setting for gliding typing. For example, there are multiple homographs for a Chinese sound; multiple selections may pop up when using phonetic input. Typing is hard for entering special symbols. Stroke recognition [1–4] based on stroke orders and character structures is an alternative, but recognition rate is an issue. It is even worse on mobile phones due to the fat finger issues. Another popular method is speech recognition [5–9], but recognition rate is problematic, and some environments are not proper for certain environments and situations such as a noisy station or a meeting occasion. We conduct a set of pilot studies to verify and assert these commonly observed issues. Furthermore, the results of these studies also let us design to overcome these limitations by directly collecting the hand-written characters/symbols based on written tracks on the touch screen for messaging due to the following reasons:

• 1. Handwriting can more easily deliver emotion and personality.

• 2. Handwriting can directly express their emotion with drawings with less constraints in remembering the vocabularies and symbols existing in other input methods.

• 3. Handwriting can also achieve availability, portability, convenience, and instantaneity which are important requirement for instant messaging.

After implementation, we conduct a set of user studies to verify that our system can let users prefer handwriting over other texting input methods in some scenarios due to easiness in expressing personality and emotions and arbitrary symbol drawing with few constraints. Moreover, our proposed input can free users from frustration of writing recognition to get better experiences and make instant messaging more intuitive and friendly.

This paper makes the following contributions: First, we identify an interesting and important input method for instant messaging on mobile devices: an intuitive and emotion-embedded handwriting input framework. Second, the core of our method is to record and send user’s handwriting messages with vectorized strokes using Bézier curves for preservation of personal and emotional handwriting features with compact networking traffic without deteriorating the convenience of instant messaging. Furthermore, we also simulate a scroll paper through a moving writing window and use the entire touch screen to ease the handwriting process and overcome fat finger issues. Finally, we embed the input framework onto a popular instant messaging system to show its easiness of implementation. As demonstrated in the results, our system is preferred while intending to deliver contents with sincerity and emotion. The rest of the paper is organized as follows: Section 2 describes the method used in this study. Section 3 reviews those previous research done related to this work. Section 4 describes our pre-studies to understand issues existing in current texting. Section 5 gives implementation details of our designed handwriting texting mechanic using the concepts collected in Section 4. Section 6 discusses evaluations of our designed handwriting texting mechanic. Section 7 concludes with a discussion of limitations and future works.

2 Methods

This work aims at exploring handwriting characteristics and discovering differences between texting and handwriting in order to design a handwriting-based instant messaging system. In order to better understand the progress of different input methods, this article first reviews directly related works including the personality connection with handwriting, stroke recognition, handwriting delivery, speech recognition, and stroke vectorization. In order to achieve this goal, we take the following steps: First, we conduct a pilot study to understand the effectiveness in emotional delivery with handwriting and discover texting issues in commonly used Chinese input methods for mobile devices. Second, according to the criteria concluded in the pilot study, we design and implement emotion-embedded handwriting messaging by recording writing strokes on the entire touch screen, vectorizing them as Bézier curves, and sending them as instant messages. Third, we verify our designed system by conducting user studies to show that users prefer our handwriting messaging system while they intend to deliver information with sincerity and emotion. All participants of both initial pilot studies and final user studies are selected randomly in front of Gongguan Station of Metro Transport in Taipei. They have varied sexes, ages, and experience in mobile and computer usage, but all participants have experience in using instant messaging on mobile devices and computers. We interview the participants and ask them to answer our designed questionnaires. All questions are in 5-point Likert scales where 1 to 5 ranks the degree of satisfaction and agreement to the criteria as very disagree, disagree, neutral, agree, and very agree. After collecting all valid questionnaires, we use the mean score to indicate the degree of satisfaction and agreement of all questionnaires.

3 Related works

Texting input is the research topics for decades because of the need to control the computers, to write documents, and to write emails and instant messages, and there are a large number of results on this field. However, since this work aims at developing a handwriting messaging framework, the following only discusses those works directly related to this research.

Handwriting vs. personality: Generally, we believe that handwriting generally reflects writer’s emotion and personality. Therefore, there are research [10–13] aiming at discovering these relationships. Castelnuovo et al. [10] analyze the connection of “imitation” and “natural writing” with intelligence, originality, anxiety, compulsiveness, sexuality, and temperament and conclude that handwriting highly reflects writer’s personality and “natural writing” does more profoundly than “imitation.” Lemke et al. [11] use handwriting to predict personality and intelligence. Williams et al. [12] reveal that there is obvious connection between personal handwriting and extraversion and behavioral model. Peeples et al. [13] also analyze the affection of handwriting on different genders and personality traits. All research reveal that handwriting has obvious connection with several personality traits, i.e., handwriting generally reveals personal styles and emotions.

Stroke recognition: Stroke recognition can be viewed as the problem of transforming text in the 2D spatial cursive forms into symbolic representation. There are two categories: online and offline based on the way of data collection. Online indicates that the writing order is available while offline only has the finished writing results. Order information are collected using an electronic pad or touch screen by recording the ordered two-dimensional coordinates of successive points pressing on the device. Plamondon et al. [2] provide a comprehensive survey on stroke recognition in both categories, and Tappert et al. [1] review those online recognition state-of-the-art algorithms. However, all research reviewed in these two papers mainly focus on the recognition of alphabetic languages, but Chinese Mandarin is a pictographic language by connecting pictographic characters to form sentences instead of spelling the vocabularies. A Chinese character is generally composed of mostly straight lines or “poly-line” strokes. Many characters can be decomposed into relatively independent substructures, called radicals, and different characters may share some common radicals. Generally, recognition can utilize this property to simplify their recognition. Liu et al. [3] survey the advances in online Chinese character recognition. Although there are more feature extraction algorithms [14, 15] invented for image content understanding, developing proper features for Chinese characters is still problematic because of different personal writing habits. Furthermore, the concept can be extended furthermore to recognize the traffic sign while driving [16]. With the advance of mobile devices, social messaging becomes very important [17]. As a result, instant messaging replaces letter writing for personal communication, and typing is the most common input manner for message creation. Since mobile devices have a small screen for typing, they generally cause the following typing difficulties: fast input and fat finger. Therefore, with the entrance of scanners, graphics tablets, and touch screens, Plamondon et al. [2] propose to take handwriting as input and use stroke recognition to replace inconvenient typing. Prochasson et al. [4] collect handwriting data including skeletons, rebuses, and phonetic writing to construct a recognition system for texting input. Although stroke recognition can make texting more intuitive, the robustness of recognition is still problematic due to different writing habits among different users. Therefore, we intend to directly deliver the writing or drawing in a vectorized format without recognition. Furthermore, computer characters and words cannot deliver the emotion and feeling of the user while he/she creates the message. Therefore, this work aims at adding emotional aspects which are missing from texting into instant messaging through handwriting.

Handwriting delivery: Norihisa et al. [18] introduce an on-door communication system by designing a communication interface to let users read/write information on the same virtual whiteboard. However, it aims at providing a multi-person simultaneous working area, and thus, it directly records the writing and drawing as a bitmap which is not efficient for memory usage and network transmission. Furthermore, it is also intended to leave short messages, and thus, it has a limited writing area. Although this approach can indeed increase the joy and practicability through instant messaging, it is not suitable for high-frequency communication usage. Therefore, our system overcomes these limitations by encoding the writing with our designed vectorized format to write an infinite long sheet in a similar manner of texting.

Speech recognition: Speech is one of the most natural and user-friendly mechanisms for information access and spoken language processing technologies. Therefore, speech recognition is the research focus for decades. Kong et al. [8] and Wang et al. [9] focus on extracting spectro-temporal modulation information for enhancing recognition accuracy. There are research [5, 6] focusing on Mandarin Chinese texting input. Furthermore, Jin et al. [7] propose a syllable-lattice-based speaker-independent large-vocabulary continuous speech recognition system. McLoughlin et al. [19] present a subjective intelligibility testing method for Mandarin Chinese input. The improvement in recognition accuracy also pushes the possibility of using speech recognition to become a real-world input method. After decades of diligent research efforts, speech recognition reaches a point where many useful and commercially beneficial applications have recently become feasible. However, these algorithms generally require high computation cost. With advance in hardware, they become feasible and have been adopted as an input method for all mobile devices in current days. For example, Janet et al. [20] propose iSay-SMS to input texts with speech recognition, and IOS provides Siri to act as a voice secretary. Lee et al. [21] provide an overview on the field and its application possibility. Furthermore, some instant messaging softwares such as iMessage [22] and WhatsApp [23] also allow users to directly record and deliver personal voice and speech. However, sending speeches requires larger Internet bandwidth valuated in mobile service, i.e., sending speeches is more expensive and costly. Additionally, both speech recognition and speech messaging require to talk to devices which may not be suitable in some situations and locations such as private contents and being in public libraries. Our system can overcome these limitations by vectorizing handwriting for messaging.

Stroke vectorization: There are research [24–26] focusing on vectorizing the black region of a scanned image for infinite resolution. However, scanned input images lose online construction information and require a complex algorithm to detect the connection and cross regions. Since our algorithm directly record the handwriting process, we can directly vectorizes the handwriting motion gesture of the finger for network traffic efficiency. The process is easier and efficient for easy embedment into all mobile devices.

4 Pilot study

Texting messaging is publicly accepted because of its availability, formality, and easy message composition. However, it still has certain limitations such as typing special symbol and delivering emotion and sincereness. Therefore, we conduct a pilot study to understand its effectiveness and usage limitations. Our study consists of two phases: The first focuses on understanding handwriting characteristics and differences between handwriting and texting through a set of questionnaires. The second devotes to discovering the issues of texting methods for mobile devices through a set of interviews and another set of questionnaires. Finally, we summarize the surveyed results as design criteria for our handwriting-based instant messaging system.

4.1 Difference between handwriting and texting

With the advance of computer technologies, texting replaces handwriting. We would like to understand the differences between handwriting and texting on the aspect of characteristics and usages through questionnaires. We conduct the study by randomly selecting 168 subjects in front of Gongguan Station of Metro Transport in Taipei. Their ages range from 15 to 65 years old with a mean of 35.6, and the participants comprise 50 females and 118 males. All are volunteers and have experience in using instant messaging on mobile devices and computers.

4.1.1 Emotion delivery

Although previous research [10, 11] show the positive connection between handwriting and personality, we would like to confirm that handwriting can deliver more emotional aspects than texting during instant messaging. We first select five aspects including personality, emotion, sincerity, intimacy, and personal style based on emotional and personal states discussed in the previous research. Then, we form a set of questionnaires to let participants to indicate whether handwriting, texting, both, or neighbor can deliver these aspects. Finally, Fig. 1 shows the statistics of our questionnaires, and handwriting has more capabilities on expressing five emotional aspects while comparing to texting. We examine whether these five emotional characteristics are more related to handwriting or texting respectively and understand whether there exists obvious differences between handwriting and texting based on the expressing percentage. Handwriting scores higher than texting in all five emotional aspects; therefore, we can conclude that handwriting contains more emotional expressions than texting in public understanding.

Fig. 1

Statistics of pre-study emotion delivery. This shows the statistics of whether handwriting and texting have the ability to deliver emotional characteristics

4.1.2 Texting and handwriting usage circumstance preference

Although more and more people prefer texting to handwriting while communication because of its convenience and availability, different circumstances may affect our preference on the format of messages and contents. For example, while chatting with friends and giving greeting to friends, a writer may prefer emotion-embedded handwriting over formalized standard texting. We wonder whether texting is preferred for all different circumstances or whether users may prefer handwriting over texting under certain circumstances. While observing the occasions for messaging in our daily life, generally, there are several different occasions such as discussing official and business affairs with colleagues and superiors, occasional conservation with families and friends, and giving respects to a family, a friend, and a lover. We can simply categorize them into three circumstances including delivering messages under relaxing circumstances, embedding sincerity into created messages, and composing formal information in official and business occasions. Therefore, this study would like to investigate whether a user wants to send/receive a message to/from others composed with handwriting or texting under three different circumstances. Figure 2 shows the statistics of our questionnaires. While “sending messages,” senders generally prefer texting for relaxing and formal circumstances, but senders prefer handwriting for sincere circumstances. While “receiving messages,” receivers strongly prefer texting for relax and formal circumstances, but receivers prefer handwriting for sincere circumstances. We notice that participants who prefer handwriting in the occasion of “receiving messages” are generally higher than in the occasion of “sending messages.” This is probably because texting is preferred due to its convenience and efficiency, but handwriting can contain more personal feeling and deliver more emotional aspects. Texting and handwriting preferences indeed depend on the circumstances especially while senders would like to deliver their sincerity. Furthermore, handwriting can express more emotion and sincerity.

Fig. 2

Statistics of pre-study usage preference. First, there are two states: sending messages and receiving messages. Second, there are three circumstances: relaxing, sincerity, and formality for each state

4.1.3 Emotion preference

The previous section shows that users prefer texting for more circumstances. It is a general belief that the preference is due to its convenience, popularity, input easiness, and input speed. However, we would like to conclude whether these are dominant factors for the preference. Therefore, this study aims at evaluating whether users change their usage preference while handwriting is assumed to construct messages as fast and convenient as texting. Figure 3 shows the statistics of this study. Although handwriting can get higher preference in the formal circumstances, there are still more users preferring texting instead of handwriting in this occasion because correct delivering the content and meaning is much more important than the expression of personal emotion. However, handwriting obviously overwhelms texting in the sincere circumstance. Moreover, the ratio on handwriting is growing even over 50% in the relaxing circumstance. In other words, while handwriting is convenient and fast, users prefer handwriting instead of texting while they want to message with emotion and personal feeling in occasional conditions.

Fig. 3

Statistics of pre-study usage preference with the speed and convenient assumption. This shows the statistics of usage preference while assuming that handwriting can be as fast and convenient as texting

4.2 Usage of different input methods

As shown in the previous study, texting is preferred for more circumstances because of its convenience. Therefore, it is important to understand the strength and limitations of different input methods. We start our analysis by randomly interviewing 20 heavy mobile messaging users about the problems and preferences of all texting methods they have ever used. Their ages range from 18 to 40 years old with a mean of 25.6, and the participants comprise 5 females and 15 males. They are students and workers from National Taiwan University of Science and Technology. All are volunteers. According to our interview, there are three main input methods including keyboard typing, stroke recognition, and speech recognition. The interview also reveals the important aspects which affect the preference of users to choose the input method while texting messaging. The following sections list the issues we discover for corresponding input methods. Then, we use these found aspects to form questionnaires and ask participants to answer them with 164 valid ones. Their ages range from 15 to 65 years old with a mean of 34.6, and the participants comprise 49 females and 115 males. They are randomly selected in front of Gongguan Station of Metro Transport in Taipei. All are volunteers and have experience in using instant messaging on mobile devices and computers. Furthermore, participants only fill in those questions related to the input methods which they have experienced before.

4.2.1 Keyboard typing

Typing with a keyboard is the most common and basic input method, and all 164 subjects have the experience of typing with the virtual keyboard on the screen. Therefore, the following summarizes the issues about its operability and its usage for Chinese input. First, familiarity affects the speed of the input as the issue (a) shown in Fig. 4, i.e., the operability problem is negligible for those who are really familiar with the setting. Second, the small screen of the smart devices generally induces the fat finger issue for failing to select the desired key (the issue (b) in Fig. 4). Third, the placement of each key in the virtual keyboard is different from the computer keyboard as the issue (c) shown in Fig. 4 with nearly half agreeing and 26% disagreeing. Finally, while inputting Chinese with spelling, there generally exist multiple homographs for a single sound, and users have to select the correct one from the list which is generally time-consuming unless they memorize the listing as the issue (c) shown in Fig. 4. Although machine learning predicts the possible candidates, it may result in the need of extra time for searching while the prediction fails due to the small usability of the desired word. The issue (d) in Fig. 4 shows the statistics; the ratios of approvals and disapprovals are very close.

Fig. 4

Statistics about potential keyboard problems. This shows the statistical result of the potential keyboard issues. Issue (a): The slow input speed generally reflects the unfamiliarity of the keyboard configuration. Issue (b): It is hard to select the desired key because of the small size of the virtual buttons and the large size of our fingers. Issue (c): The configuration of the virtual keyboard is different from the real one to induce typing confusion. Issue (d): Selection of homographs for Chinese induces another familiarity issue

It is similar to the familiarity problem: for those who are familiar with the candidate selection system, it is not confusing and inconvenient. We can conclude that the practicability of keyboard typing highly depends on its familiarity. In other words, users can type text fast and smoothly with a keyboard once they get used to it. On the contrary, users do not choose it if they have trouble in learning it.

4.2.2 Stroke recognition

Education teaches handwriting, and thus, handwriting is the most natural input method with a proper stroke recognition algorithm. By writing on the touch screen, the recognition system recognizes the possibly corresponding character according to the shape, structure, and the order of the strokes. Ideally, while the recognition is perfect, those issues listed in keyboard typing can be solved, but stroke recognition may fail occasionally. The following lists the possible issues found by our investigation. First, failure in recognition happens more frequently than expected as the issue (a), the issue (b), and the issue (c) shown in Fig. 5 especially when his/her handwriting is very scratchy. Since recognition is based on the comparison of the orders, directions, structures, and the shapes of strokes to those in the database, users would better memorize the order and write neatly to mimic the one in the database to have a high recognition rate. This is sometimes strenuous and makes input slow. Furthermore, it is difficult to write complicated characters on a limited area as the issue (d) shown in Fig. 5. In other words, the smaller devices, the more easily they have to face this issue.

Fig. 5

Statistics of the potential stroke recognition issues. This shows the statistical result of the potential stroke recognition issues. Issue (a): This shows that recognition failure may be due to scratchy handwriting. Issue (b): This shows that recognition failure may happen despite of neat handwriting. Issue (c): This shows that users need to write carefully for a high recognition rate which leads a slow input speed. Issue (d): This shows that the limited writing area makes it difficult to write complicated characters

4.2.3 Speech recognition

Speech recognition recognizes users’ talking for inputting sentences of words without the need of extra editing to get the desired results ideally. However, they are not well accepted due to the following issues. First, accents, verbiage, and slurred speaking may results in a high failure rate as the issue (a) shown in Fig. 6. The issue (b) in Fig. 6 also shows that enunciated speaking can still make recognition fail with an approval rate of nearly 50%. Second, slurred speaking may also result in a high failure rate as the issue (c) shown in Fig. 6. Finally, speech recognition is not suitable for some occasions and environments such as office rooms and public museums generally required to be quiet as the issue (c) and the issue (d) shown in Fig. 6. Especially, those, who feel weird to speak to a device in public and want to keep personal conversations secret, prefer not to use speech recognition as their input method. To our surprise, most subjects prefer not to use speech recognition in public while some subjects pointed out that it is suitable for speech recognition in some scenarios where users cannot touch their devices, such as driving a car.

Fig. 6

Statistics of the potential speech recognition problems. This shows the statistical result of the potential speech recognition issues. Issue (a): This shows a possible high failure rate due to slurred speaking. Issue (b): This shows recognition failure despite of enunciated speaking. Issue (c): Speech recognition is not suitable for the places required to be quiet. Issue (d): Speech recognition is not preferred for those who value privacy

4.2.4 Other input types

In addition to these commonly used input methods, we also investigate whether users have problems and issues when inputting texts and contents into the devices such as having no idea to input the desired contents and feeling confused about the usage of the input method as shown in Fig. 7. First, inputting numbers and symbols usually depends on typing with a keyboard which is bad news for who are not familiar with the keyboard input. Additionally, some systems provide a list or table of symbols for selection, but finding the desired one is a time-consuming task. Furthermore, users generally need to switch from the character panel to the symbol panel to induce non-smooth texting experience. Second, there are also participants reflecting that texting cannot express emotion and personal feeling easily. Although there are emoticons designed to add emotion and feeling into cold texting, each emoticon is usually composed of selection of several special symbols for non-smooth texting experiences. Fortunately, several systems nowadays have built-in emoticon panel typing with the character input system to relieve the severeness of this issue. Therefore, there are only 34% of participants seeming to be bothered. Third, when a sentence contains more than two languages, switching between two input methods annoys users. Multilingual inputs happen frequently, and this would require switch of input panels and manners. Our investigation shows that 56% means that this problem would be a pain in the neck. Fourth, mathematical and chemical equations are needed in technical discussion and their input methods generally upset users. Since an equation contains a bunch of special symbols, line-by-line entering is hard to express such complicated representations. When discussing questions on mathematics through instant messaging, they use a picture of their handwritten equations as a substitute. Our result also shows the approval of this issue. Finally, many participants also point out the difficulties of having corresponding texts for dialects. Since dialects do not have their own text, this cannot be considered as a text-entering problem strictly.

Fig. 7

Other potential input problems. This shows the statistics of other potential issues for input. From top to bottom are “Unfamiliarity with numerical keyboard configuration,” “Issue in the emoticons input manner,” “Issue with multilingual panel switching,” “Issue in the mathematical and chemical equation input manner,” and “Issue in the dialect input manner”

4.3 Summary

Based on the results of our pilot study, we summarize our findings as the following key points:

• 1. Handwriting generally can deliver and encode more personal feeling, sincerity, emotion, and characteristics than texting. Therefore, this work aims at messaging with handwriting instead of texting.

• 2. Texting is the preferred message construction method for instant messaging on mobile devices and computers because of its convenience, portability, and availability. Therefore, any input method must be able to provide comparable functionalities along with other characteristics for the replacement possibility.

• 3. Texting is sometimes not plausible because it requires a strenuous learning curve for new users especially for those elders and children. Therefore, our system must be more intuitive and easy to use without a strenuous learning curve.

• 4. Due to the limitation of encoding space, texting cannot deliver some contents especially those containing complex symbols and expressions. Therefore, our system must allow users to deliver more complicated contents.

• 5. Stroke recognition still has robustness issues and does not aim at composing easiness. Furthermore, speech recognition is intuitive, but it also has robustness issues and is not suitable in many occasions. Therefore, our system should provide more intuitive writing input and allows the user to use in as many occasions as possible.

5 System design

Texting has difficulties in delivering emotion and personal feelings inside messages. On the contrary, handwriting can easily embed emotion with fewer constraints when entering characters, numbers, and symbols. During our investigation, some participants complain about the radiculitis of composing Chinese characters by spelling phonetically and selecting from the candidate set even when they can write the characters. However, since the spirit of communication is to understand each other, it is not so necessary to use the formal and default computer character set, and it is possible to compose message directly using handwriting for easiness and intuitiveness. Consequently, we aim at developing a handwriting-simulated communication system to relieve the texting issues and meet the demands of handwriting. We develop our system for smart mobile devices with a handwriting-simulated input interface to record those strokes written on the touch screen directly, vectorize the stroke for compactness, and deliver them instantly to render on others’ screens. Generally, users would like to review their previously composed messages for effective communication and examination, and thus, we design a paper-scroll-simulated interface for browsing the sent and received handwriting messages. This section lists out the potential issues of our system and our corresponding solutions along with our implementation.

5.1 User interface

Based on summarized key points in our pilot study, we first determine the following design criteria for solving the following potential texting problems.

• 1. It is difficult to write some complex characters in a small limited area for stroke recognition; similarly, our stroke recording area should be as large as possible.

• 2. Since our system aims at rebuilding the letter-writing experience, our system should have a manner to accommodate all words in a scroll of letter. Furthermore, our system must also take largely and sparsely written characters into consideration because users must write them with their fat fingers on small devices.

• 3. Our system must provide intuitive revisions and modification functions to correct the input errors.

• 4. Handwriting is the main input for our system, but users may still want to use texting for messaging. Therefore, our system also provides intuitive ways to incorporate texting with our handwriting. Furthermore, we also provide an intuitive way to switch to other input methods.

Our user interface as shown in Fig. 8 a ¹ follows the similar concept of instant messaging applications. Our system puts received messages on the left while setting delivered ones on the right. Unlike other applications, our user interface can hybridly display both texting and handwriting messages. Furthermore, our system allows four input methods including “handwriting,” “texting,” “stickers,” and “other,” and we equip our system with an input switching button on the right-bottom corner for switching easiness as shown in Fig. 8 b. Handwriting is our major difference from other texting messaging systems, and thus, the following sections focus on discussing its design.

Fig. 8

Chatting interface and handwriting interface. a This shows a snapshot of our messaging system including our messaging window of browsing the received/sent messages and functional buttons placed on the right-bottom corner. b This illustrates our handwriting interface consisting of the full screen area of a mobile device for writing except for essential editing tool buttons placed on the right-top corner and a paper-scroll-simulated previewing window located at the right-bottom corner

5.2 Maximize effective handwriting region

Generally, the allowable writing area of stroke recognition for both the IOS and Android systems is too small to write complex characters. This situation is even worse since users generally use their fingers instead of pens for writing. Enough writing spaces should improve the handwriting experiences on mobile devices, and thus, our first goal is to enlarge the limited writing region. Our solution to this issue is to use all available screen space for writing. Furthermore, we also want to let users examine those previously sent and received messages while writing new messages. Therefore, we make the writing region as a transparent cover on the top of the chatting screen as shown in Fig. 8 b to allow seeing the chatting screen and writing on the full screen simultaneously. However, our system still keeps editing buttons for operation easiness and convenience along with a paper-scroll-simulated previewing window to let writers examine their writing characters and drawing symbols. Therefore, our actual effective writing region is the full screen area except for these buttons and the previewing window.

5.3 Simulate letter writing on mobile devices

Since Chineses generally write characters along a line from left to right and from top to bottom, our system let users to write and draw characters and symbols along a line by aligning them with their bounding boxes for message composition. We construct the background of our system as a scroll of paper with horizontal bars as shown in Fig. 9. Since finger-written characters on the touch screen cannot be as small and dense as pen-written ones on the paper, the touch screen generally does not have enough space for a letter. The screen only displays the focused part of the paper, and the horizontal blue lines on the screen indicate those writing lines on the paper. Users can write whatever they want in the active writing region and pan to other parts to continue their writing just as how we write on a letter. We design two gestures to simulate letter writing on mobile devices: the first is one-finger dragging for character writing, and the second is two-finger sliding for panning the contents. Furthermore, we add horizontal and vertical sliding bars to move along a writing line and switch among different writing lines.

Fig. 9

Paper-scroll-simulated preview window. This shows the paper-scroll-simulated display concept by having the focus frame moving on the top of a paper and only displaying a part of the paper to simulate hand movement while writing characters on a scroll of paper

Furthermore, the displayed region should be automatically positioned, i.e., automatically keep the active writing line in the center of the screen. Figure 10 shows an example: while a user first writes in the first line, he/she moves the screen to the position between the first and second lines but closer to the second line, he/she moves his/her finger away from the touch screen, and our system automatically places the focusing frame to the second line. This way can ensure that while users move to other lines, the writing base line is always at the same screen position. We also add two speed buttons onto the surface for faster and more convenient writing. The first button named “move-to-end” has the functionality to move the focusing frame to the end of the current active line. Figure 11 shows an example: when a user finishes a character and pushes this button, the focusing frame moves to the right side of the latest written character. The second button named “move-to-next-line” has the functionality to move the focusing frame to the beginning of the line next to the current one. When a user finishes the current line, he/she can push this button to continue writing in the next line. We also give users a previewing area in the right-bottom of the screen to examine the current handwriting results with a square frame to indicate the active writing location for more information about what to write.

Fig. 10

Automatically positioning onto a writing line. This illustrates our writing-line-positioning mechanism. a Our focus frame is first located at the first line. b A user uses his/her two fingers to slide the focus frame to the border between the first and second lines. c After the finger leaves the screen, our focus frame automatically move to the second line

Fig. 11

Automatically moving to the end of writing in a line. This shows our “move-to-end” function. a A user just finishes writing a character. b After clicking the move-to-end button, the focus frame moves to the right to let all finished contents on the left respectively. c The frame finally moves to the right side of the latest character on the line to keep writing

5.4 Handwriting revision

Modification is generally necessary for human writing. Therefore, we also provide similar editing tools as those provided by most texting systems. First, we provide a “undo” function to unwind back to the latest saved point. In order to provide smooth writing experiences to users, our system records the focusing frame position of each written character along with all strokes written in the frame position to ensure that the “undo” function can bring back to the same previous writing position. For example, after writing a character and sliding to write another, he/she suddenly finds the incorrectness of the previous character and wants to remove the character with the “undo” function. Our system first slides the focusing frame back and removes all the stroke information recorded along with the same frame position to let him/her continue their writing seamlessly. Second, we also provide a “redo” function to recover the steps saved in the “undo” stack. Third, we also provide an “eraser” function to let users use one-finger dragging gesture to remove the undesired strokes because users sometimes want to erase those strokes and characters written long time ago, and the “undo” function no longer works. The “eraser” function is also redoable and undoable. For instance, if users find that they mistakenly remove desired strokes, they can also use the “undo” function to restore those removed strokes. As a result, users can edit the writing message everywhere in our system.

5.5 Hybrid input with text and vectorized handwriting

Although we aim at designing a handwriting-based messaging system, there still exist certain texting-desired circumstances such as copying texting information elsewhere after the previously written characters and symbols. Hence, we allow to switch among other input methods, such as texting and speech recognition, and our handwriting method so that users easily achieve the desired copy-and-paste job. Moreover, this is also helpful while the users forget how to write the desired character. Our system hides the writing region when switching to other input methods to let users set the cursor position in the previewing window as shown in Fig. 12. Furthermore, in the texting mode, the “confirm” button has the same functionality as the “next line” button in the writing mode. One-finger dragging in the previewing block can move the position of message, and one-finger tapping on the entered text can edit them again.

Fig. 12

Hybrid handwriting and texting interface. This shows the hybrid handwriting and texting interface to add texting into the writing in the previewing window

5.6 Implementation

To verify the feasibility of our designed communication system, we implement our designed handwriting-simulated system on the IOS platform. However, there are some implementation issues:

• 1. Since our target devices are those mobile devices, the size of recorded and encoded stroke data should be small enough for efficient transportation. In addition, our system should be able to render the recorded strokes properly and automatically without having aliasing artifacts onto the screen with a different screen size of different mobile devices. Therefore, we record the strokes as vectorized Bézier curves.

• 2. Because our handwriting system is an auxiliary tool for instant messaging through network communication, our system should also provide a client-to-client communication framework.

5.6.1 Stroke information

While storing the written characters as a set of bitmaps, we have to face the following problem:

• 1. Our system must be able to resize the bitmaps for different resolutions.

• 2. The resolution of bitmaps must be high enough for the highest possible devices in order to reconstruct the results without aliasing artifacts.

Both require a really high resolution of bitmaps which are not efficient for instant messaging. As a result, although there are research focusing on enhancing bitmap resolution [27, 28] and developing efficient encoding algorithms for stroke-based handwriting and drawing bitmaps, our system chooses not to use them as our encoding mechanism. Since strokes and drawings can be decomposed into a set of strokes, we can store these strokes as a set of vectorized Bézier curves. Therefore, we regard each stroke as a vector graphics by encoding a stroke as a set of the control points rather than a map of pixel values. After recording strokes as a set of control points, we can directly render the vectorized Bézier curves on other screens based on the screen size using the Bézier curve rendering algorithm while only using relatively smaller memory spaces. We go one step further to reduce the size of the handwriting information as follows. Instead of representing a control point in the floating point format as most of vector graphics based applications, we adopt unsigned short precision since floating point is not necessary to display a nice stroke on the screen of mobile devices in our case. By using the unsigned short precision, we can reduce the size of a control point to be just 4 bytes, and each axis stores integer data from 0 to 65,535 which is large enough to write a message. Later, our system compresses the stroke information with GZip and transforms the zipped data to the Base64 string format for efficient Internet transportation.

5.6.2 Server

Although our system should be a simple a client-to-client communication framework, we cannot guarantee that all users be online while one of them attempts to handwrite a message to send. It seems not appropriate to use a peer-to-peer protocol. Therefore, we choose a client-server model. When a user wants to deliver messages to others, the message is sent to our server first for temporary storage, and while other users are online to check their messages, our server then forwards the stored messages to them. While all communication parties are online, our system passes the delivered messages to the others as soon as possible by using MQTT (formerly MQ Telemetry Transport) as our communication interface along with node.js and mongodb to build the server and database, respectively.

6 Results and discussion

After designing our system, we design several user studies to verify whether it achieves our design goals and satisfies user requirements: a friendly and intuitive mechanism can let users prefer handwriting in message creation and our system is simple and intuitive and has a short learning curve and usage comparison with other mechanisms. Totally, there are 42 participants attending our user studies, and all with normal or corrected-to-normal vision. Their ages range from 15 to 55 years old and 15 are females and 27 are males. Furthermore, there are 13 participants aging from 15 to 24, 16 participants aging from 25 to 34, 9 participants aging from 35 to 44, and 4 participants aging greater than 45. All are volunteers and do not have any experience with our system. The study is conducted under a iPhone 4s with 3.5-in. (diagonal) widescreen multi-touch display of a 960 × 640 resolution. Before the studies, we first briefly introduce the characteristics, functions, and usage of our system and let users use the system for 15 min. During the try-out period, participants can freely discuss with the conductors and other participants about our system. Then, participants are asked to run several designed tests. After testing, participants are asked to fill out our designed questionnaires containing the following parts. First, whether does he/she change the preference from handwriting to texting in some circumstances? Second, why does he/she change the usage preference in the specific circumstances? Third, what are their opinions about the design criteria of our system? Finally, what are their satisfaction rating of our system when comparing to other input methods?

6.1 User preferences

We used the same set of circumstances while conducting the pilot study, and we ask participants whether they prefer our handwriting system over other input methods in the selected circumstance. Figure 13 shows the statistical results. While numerically comparing to the statistical results of the pilot study, while using our proposed system also, participants are more willing to change their preference to handwriting in all circumstances. Furthermore, this tendency is more obvious while participants want to deliver contents with emotion and personal feeling including relaxing and sincerity circumstances.

Fig. 13

Usage preference of handwriting and texting while using our system. This shows the statistics of usage preference of handwriting and texting over delivering messages after while using our system

Preference to handwriting grows while comparing to the scenario without our system. Later, Fig. 14 in Section 6.2 also shows that our system can achieve our assumed goal, which is “handwriting letters is as convenient and fast as texting messages.”

Fig. 14

Statics about why switching to handwriting. This shows the statistical results of why users change their preference to handwriting, “handwriting can embed emotion and sincerity,” “handwriting can input all possible contents,” and “handwriting allows drawing arbitrarily”

6.2 Preference analysis

Handwriting contains more emotions and sincerity than texting, and it also provides more freedom to write and draw contents. In addition to understand that our system is preferred for certain circumstances, we also would like to investigate the reasons why users change their preference from texting to handwriting while using our system. Based on the main opinions expressed by 42 participants in the interviewing section, we list our reasons as “handwriting can embed emotion and sincerity,” “handwriting can input all possible contents,” and “handwriting allows drawing arbitrarily.” Figure 14 shows the statistics. “handwriting can embed emotion and sincerity” accounts for 81%. This also reflects the same trend revealed in the previous section: 89% participants prefer handwriting in the sincerity circumstance. Furthermore, “handwriting can input all possible contents” accounts for 85.7%, and it is also a significant factor. Finally, “handwriting allows drawing arbitrarily” accounts for 64.3%, and it is also an important factor, but it is less important than the other two. This reflects that not all users want to draw figures in their messages. However, drawing is indeed more funny than just typing plain text to make more than half subjects approve on this reason.

6.3 Satisfaction

When designing and implementing our system, we target several important properties including “full-screen writing region,” “simulating letter-writing,” and “hybrid input.” After designing our interfaces, we would like to understand their effectiveness including the degree of practicability on “full-screen writing region,” the degree of accomplishment on “simulating letter-writing,” and the degree of practicability on “hybrid input.” Figure 15 shows our investigation results. First, the degree of practicability on “full-screen writing region” shows that there are 76% participants considering it is helpful. The study result concludes that a larger writing region can truly improve handwriting experiences. Second, the degree of accomplishment on “simulating letter-writing,” i.e., the difference between our system and the real letter-writing, shows that there are totally 42% participants considering they are similar and 15% considering they are dissimilar. Forty-three percent have no idea about their similarity, and they generally reflect that the writing medium, one use fingers and the other use pens, is different and they cannot do any judgment on this investigation. Finally, the degree of practicability on “hybrid input” shows that half of participants consider that it is practical while only 12% consider that it is impractical. We can conclude that since mobile devices stores most data in text format, hybrid input can be helpful in such applications.

Fig. 15

Statistics on satisfaction of our designed handwriting interfaces. This shows the statistics of satisfaction of our designed handwriting interfaces. Issue (a): This shows the degree of practicability on our designed user interface of “full-screen writing region.” Issue (b): This shows the degree of accomplishment in our designed user interface of “simulating letter-writing.” Issue (c): This shows the degree of practicability in our designed of “hybrid input”

6.4 Comparison with other methods

During our pilot study, participants generally express their selection of their primary input method based on the following three factors including “operability,” “intuition,” and “speed.” Therefore, we also want to find out the difference between our system and other input methods on the aspect of “operability,” “intuition,” and “speed.” Figure 16 shows the investigated statistics. We find that the ratios on “operability” and “speed” are not over 50%. This shows a similar phenomenon as our pilot study. Those who excel in texting would not agree that our handwriting-simulated system has better operability and higher input speed. However, most participants agree that handwriting is indeed more intuitive than texting since handwriting is the most basic skill in their life.

Fig. 16

Comparison between handwriting and texting. This shows the comparison between handwriting and texting on the aspect of “operability,” “intuition,” and “speed.” Yes denotes that participants prefer using our system over typing, No D. denotes that participants like both methods, and No denotes participants prefer using typing over our system

Next, Fig. 17 shows the comparison to stroke recognition. For all three factors, more participants agree that handwriting outperform stroke recognition. Furthermore, although input methods are based on handwriting, our recognition-free system avoids those recognition problems, which can improve the input speed while comparing to stroke recognition.

Fig. 17

Comparison between handwriting and stroke recognition. This shows the comparison between handwriting and stroke recognition on the aspect of “operability,” “intuition,” and “speed.” Yes denotes that participants prefer using our system over stroke recognition, No D. denotes that participants like both methods, and No denotes participants prefer using stroke recognition over our system

Finally, Fig. 18 shows the comparison to speech recognition. Since speech recognition rate is problematic, and since our system is recognition-free without recognition problems, more than 50% participants agree that our system has better operability than speech recognition, and the p value of t test is less than 0.001 to show significant difference. However, there are 45% participants considering that two methods have comparable performance. This is because most participants think that our input speed is slower than speech recognition. As a result, although the speech recognition rate is problematic, inputting with saying a sentence directly is much faster than handwriting.

Fig. 18

Comparison between handwriting and speech recognition. This shows the comparison between handwriting and speech recognition on the aspect of “operability”, “intuition”, and “speed”. Yes denotes that participants prefer using our system over speech recognition, No D. denotes that participants like both methods, and No denotes participants prefer using speech recognition over our system

Our user studies show that no matter users are familiar with texting or not, users may change from texting to handwriting because of its intuitiveness, no need of recognition, the requirement of no extra learning, and its capability of expressing emotions and sincerity. Moreover, handwriting allows to write/draw contents that are hard to express via other input methods.

7 Conclusions

This work explores the difference between handwriting and texting in emotion and sincerity delivery and focuses on solving texting input issues on mobile phones. After conducting pre-studies, we identify issues in texting for instant messaging and design a handwriting-based mechanism for quick, simple, and intuitive message construction. Finally, we conduct usability tests to learn that our mechanism is preferred under certain circumstances because of its easiness to create messages and abilities to deliver emotion and sincerity. Furthermore, our handwriting interface can also help intuitively, easily, and quickly create texting messages when incorporating with stroke recognition. However, our system is not without limitations. Since we treat input as pictorial data, it is hard for character-based processing such as copying and replacing. Although communication generally does not need these functions, they are helpful for speed enhancement. Therefore, we would like to incorporate our handwriting with stroke recognition for possible character-based manipulation. Additionally, our input mechanism still cannot totally replace pen writing because of tactics. We would like to incorporate a stylus for enhancing pen writing tactics when inputting. Finally, our writing user interface is for small screen sizes such as iPhones and not optimized for plate computers. Since plates have a larger screen size, we would like to design a more suitable and intuitive interface for smoothly and intuitively texting.

8 Endnote

¹ Because Chinese characters are much more complicated than English characters, we use Chinese characters for evaluation to show that our interface is convenient for handwriting even in complicated Chinese characters.