I was going to ask if anyone could pinpoint when apple started publishing, but I found their first post - July 2017. They even have an email inviting students, researchers, and devs to ask questions.
The input is a 48x48 image. For handwriting recognition, many systems use stroke information (online) to augment the purely image based (offline) data. This significantly reduces error rates.
For the smaller ICDAR 2013 Chinese Handwriting Recognition Competition (~8,000 characters), CR(1) accuracy went from 94.77% (offline) to 97.39% (online). Human recognition rates were 95.19% (offline) and 96.13% (online).
Surely you could still use it for hints? (ie. "stroke X followed by stroke Y has a Z% higher chance of being this character rather than that one", much like some phone keyboards use the prior/next words entered to correct words so they make sense in context).
Somewhat unrelated but it's funny how we talk about human recognition rates being below 100%. Obviously, a human somewhere had to decide what all the correct answers were. I understand that it has a specific meaning in the context of machine learning though.
Over twenty years ago, I was in a conversation with a developer who was working on Japanese handwriting recognition for the Go PenPoint OS. He told me that stroke order, which is supposed to be consistent in Japanese writing, was a big help. It's interesting that this article says that there was a need to "achieve stroke-order independence". I'm wondering what the issues are here and why stroke order was useful in the 90s but not considered so now. I'm also not sure if the developer I spoke to was concerned with Chinese input as well, or just Japanese, and how that might affect the methods used.
I don't know much about Japanese, but in written Chinese, the stroke order is less likely to be consistent. One of the challenges for early leaners of writing Chinese characters is to "get the stroke order right", and many, even sufficiently educated adults could struggle getting all orders right. Also, writing Chinese characters in cursive often requires different stroke orders for the same character due to aesthetic reasons, and the boundary between print and cursive style in hand writings could be very blurry. Therefore, I could see stroke-order becomes a very noisy, inconsistent and less reliable input.
Standard stroke order for any given character is occasionally different between Chinese and Japanese [1].
Characters also have different variants, e.g. the radical on top of 'grass' 草 has variants with 3, 4, or even 6 strokes [1]. There are two 4-stroke variants that look different and have different stroke order. Plus cursive handwriting will often elide and merge strokes to make things flow better.
Full stroke order independence is probably unrealistic (some parts of a character will always be written last), but allowing for some local inversions is probably necessary.
With handwriting recognition you differ between online and offline recognition. With online you have the stroke order, which helps a lot for character segmentation. Character segmentation is the unsolved problem for handwriting recognition, at least for Latin scripts. With offline you only have the image of a word. If you depend on online information you can't recognize text from images taken with a camera.
Segmentation is not needed for a modern deep learning system, since this is learned by the neural network. It’s a solved problem for many handwritting recognition problems.
Before you ask someone to correct you, you needed to raise examples which Chinese education isn't rigorous. AFAIK, East-Asian countries education are all consistent and rigorous and focus very much on STEM. And they all focus on good hand writing on paper. Good handwriting can probably get a few extra marks on writing exams. But after their students finished high-school and attended college entrance exam, fewer people keep up with good hand writing since everyone types right now.
I'm trying to learn Chinese, and sometimes get the stroke order wrong. All the other keyboards are phonetic systems, not graphical systems to type the characters. I appreciate the stroke-order independence so I don't have to try so many times to find the character I need.
> I'm wondering what the issues are here and why stroke order was useful in the 90s but not considered so now
People expect more from technology.
So while in the 90's it was amazing that technology could get the correct character if you clearly wrote each character with the correct stroke order, today, people expect to be able to hurriedly scrawl (possibly in cursive) and have the system get the right character. Those are two very different problems.
See for example the images of cursive characters in the linked article. They are very common variations, containing simplifications, but they don't follow anything like the correct stroke order for the character except maybe in broad terms such as writing from left-to-right, or top-to-bottom
From context I assume it's referring to offline recognition where there's no stroke order to work with.
I doubt it's anything to do with the fact that people are inconsistent about stroke order - that's true, but they're not that inconsistent and there's no reason a recognition system couldn't account for variations in order just as it does for other factors.
I don't think I knew that there even was a cursive form of Chinese. That does complicate things, especially the "off-line" case where all one has is a static image.
What would be really cool is to have an augmented VR setup where the user is writing in midair and the VR shows light trace of it, while the OCR recognizing the writing.
slightly off-topic, but the images in the article illustrate it beautifully: I think many people don't realise (I didn't) what an amazing skill it is to be able to read handwriting (cursive). I took it for granted to be able to read hand-written German, English, Spanish, and totally forgot that it must have taken me some time as a kid to acquire that skill (beyond reading and writing generally).
However, when I studied Chinese for a bit, the (printed) characters are tricky to start with, but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible.
I think stroke order is hugely important, in that it forms the conventions for fast abbreviated writing. For example, the "mouth" radical, a square, "口", when written quickly basically looks like a lower-case "b", because the left lower stroke comes first, then top and right in one stroke, then the bottom stroke.
More complicated characters basically become a squiggle, and it's really magical, if you think about it, that you can just read it. I mean, look at figures 5, 6, 7 in that article.
For me, a nice illustration of Moravec's paradox of AI [1]: what we think is difficult (chess, differentiation, ...) is easy, and what we think is easy (reading handwriting, putting an egg in the fridge, walking (!) and opening a door [2]) is hard...
> but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible.
It is when you are starting out, but it's not so bad once you get used to it. The best way to learn to read Chinese handwriting is to learn to write Chinese handwriting, and I don't just mean practising writing by hand, but to work through a textbook aimed at teaching cursive Chinese.
This book is a good introduction in English [0], and if you are comfortable with Chinese only, this book is excellent and also far more comprehensive [1].
Once you've gone through those books, you'll be able to read most handwritten stuff without too much struggle.
In the examples in the article, do you find the second example for 的 legible (figure 5)? Or either of the characters in figure 7 - without being primed, would you know what characters they were meant to be?
I once worked with a paediatric neurologist whose handwriting was so bad, even he couldn't read it. We had a patient with a referral from him which we couldn't read, took it down the hall to him... he scrunched up his face... and gave up reading it: "Well, I know the patient, and they need...". Absolutely lovely bloke, but I'd hate to get a referral from him without being able to talk to him :)
> do you find the second example for 的 legible (figure 5)?
Yes. All of the 的 examples are clearly 的 that I'd get without being primed.
Figure 7 is more tricky but I'd know 五 without being primed, and 王 definitely after being primed or if I was reading handwriting that had more context.
Thanks for the info. I've been learning now for about a year, and text is definitely my weak spot. I see handwritten chinese and it frequently looks nothing like the nice, crisp computer fonts (which are hard enough just by themselves) - not so much that it's messy, but that I often just simply can't see how you make out the elements. I find it amazing that not only do native chinese speakers recognise thousands of characters, but also wild handwritten variants, and also the same again in simplified/traditional.
I once lightheartedly asked my teacher if exercise books in china had bigger gaps between the lines, as I couldn't see how to fit a vertically-complex hanzi in. She wandered over and scrawled something that neatly fit in the line. "How the hell did you do that... ?" :)
> Thanks for the info. I've been learning now for about a year,
A year is probably still too early to be able to read handwriting well. You probably need to have about 2,000 known characters under your belt and also have a good feel for the language such that you are able to fill in gaps based on context - that is be able to complete parts of the sentence based on knowledge of what words/characters are likely to fit.
There are many cursive characters that reduce to something quite similar so having knowledge of what fits in the sentence makes things easier.
As an example, with 王 above, by itself it might not look like anything you can recognise, but if it was written like that on a note starting out like 尊敬的王老师 then from context it narrows down the possibilities significantly.
> She wandered over and scrawled something that neatly fit in the line. "How the hell did you do that... ?" :)
Practice! It's one thing to know how to write a character, it's another thing to have the fine motor skills required to write it proportionally within a confined space, and you can only develop that skill through practice. If you're interested, one of my side projects [0] is a site that lets you generate custom Chinese worksheets of varying grid sizes and fonts (including some handwriting fonts) and other things, which can be useful for such practice.
A year is definitely too early - I'm a middling student at it (and weak at text), and would be lucky to get 100 characters by sight. As you say though, it's all about practice, and I need to apply some "bum glue" (to quote Stephen King) and just sit down and do it. A Chinese colleague of mine said that throughout high school, he had nightly homework of 6 new characters, pretty much every night. Ouch!
Thanks for the link - it will help and I've been meaning to look for one (phone apps with your finger aren't quite the same).
> A Chinese colleague of mine said that throughout high school, he had nightly homework of 6 new characters, pretty much every night. Ouch!
'Pretty much every night' is the most important part.
Learning Chinese characters is a grind and you just need to keep chipping away at it.
Pick a nice sustainable number of characters (maybe 6, maybe 3, maybe 10 but something that takes maybe 30 minutes to an hour to do including revision) and then do it every day without fail.
Number of days in a row spent learning is far more important than number of new words learnt per day, so always try to optimise for the former rather than the latter.
At some point, almost every Chinese learner who has spent more than 3 years learning the language has looked back and thought 'why oh why didn't I just learn 3 characters a day'.
HN is so very educational. Where it impresses me most, where it matters to me most, is when it is a subject I not only know nothing about, but is also a subject I'd have never been interested in seeking out.
In this case, I had no idea there was cursive Chinese. I've since gone on a search and learned more. I'd have never done that, had I not seen your comment.
Speaking of stroke order, it's hard to imagine that the image based approach should be more efficient, seeing "the need to achieve stroke-order independence".
It seems so much more difficult if you can't use the stroke order information, or the stroke direction. Even the speed over the stroke should be able to help you along, especially as you say for cursive characters. Without stroke order it seems almost impossible to decode those in a systematic way.
I suppose this is a general feature of deep learning, it's still not clear how to make good use of all the structural information we possess about a problem, instead a very brute force approach is chosen.
I'm very interested in this field! But I didn't stop at 30,000 characters - I'm making a dataset of 75,000 characters from 1500 different fonts. Does anyone want me to publish it?
I use Apple's Chinese handwriting keyboard almost every day, living in Taiwan. Unfortunately it only shows 4 possible choices, and often the character I need is not one of those. So I use my own Pingtype keyboard.
Enter a character into the bottom text field, click Search, and see the character decomposition above, and the characters & words containing that character below. Then when you're ready, click Insert to add it to the text field at the top for translation.
I have no idea how to do marketing for this tool, and commenting here is the only way I've had ~1300 views in the last 6 months since I published it.
Chinese language student, native English speaker. I haven't found any handwriting recognition commercially available for the English language that compares to what is available for Chinese Traditional / Simplified.
The best English handwriting recognition software I've found is the old Palm Pilot Graffiti writing system. It's not really developed since the late 90's / 00s. Back in the day, it was blazing fast (and pretty accurate after learning the method) compared to shitty small mechanical keyboards of the era.
After diving into Chinese I realized how deficient the English HR landscape was and how badly under-researched the area is. I think the standard QWERTY keyboard is just too efficient for the Euro zone languages to make the research investment worthwhile. My 2 cents / guess.
There are Graffiti like apps available on IOS. What I've found is that with english keyboard auto-correct, my typing is so much faster without Graffiti using the standard Apple IOS English keyboard. Kind of a disappointment...
Another conjecture, I suspect much of the morbidity in English HR research has something to do with Graffiti patent lock up between what ever remains of Palm, Xerox, and some heretofore unheard of company called Communication Intelligence Corporation.
Unfortunately they don't provide any details about the model or the training. Would be nice to have an example as a baseline for real-time performance on iOS devices.
43 comments
[ 4.6 ms ] story [ 86.5 ms ] threadhttps://machinelearning.apple.com/2017/07/01/Welcome.html
Thanks HN!
For the smaller ICDAR 2013 Chinese Handwriting Recognition Competition (~8,000 characters), CR(1) accuracy went from 94.77% (offline) to 97.39% (online). Human recognition rates were 95.19% (offline) and 96.13% (online).
One of the examples there was a Chinese character being drawn as something like a 2. The character looks nothing like a 2
Characters also have different variants, e.g. the radical on top of 'grass' 草 has variants with 3, 4, or even 6 strokes [1]. There are two 4-stroke variants that look different and have different stroke order. Plus cursive handwriting will often elide and merge strokes to make things flow better.
Full stroke order independence is probably unrealistic (some parts of a character will always be written last), but allowing for some local inversions is probably necessary.
[1] https://en.wikipedia.org/wiki/Stroke_order#Stroke_order_per_...
[2] https://en.wiktionary.org/wiki/%E8%89%B9
People expect more from technology.
So while in the 90's it was amazing that technology could get the correct character if you clearly wrote each character with the correct stroke order, today, people expect to be able to hurriedly scrawl (possibly in cursive) and have the system get the right character. Those are two very different problems.
See for example the images of cursive characters in the linked article. They are very common variations, containing simplifications, but they don't follow anything like the correct stroke order for the character except maybe in broad terms such as writing from left-to-right, or top-to-bottom
I doubt it's anything to do with the fact that people are inconsistent about stroke order - that's true, but they're not that inconsistent and there's no reason a recognition system couldn't account for variations in order just as it does for other factors.
However, when I studied Chinese for a bit, the (printed) characters are tricky to start with, but reading a fluent writer's handwriting ("unconstrained", as they call it in the article) is nigh impossible.
I think stroke order is hugely important, in that it forms the conventions for fast abbreviated writing. For example, the "mouth" radical, a square, "口", when written quickly basically looks like a lower-case "b", because the left lower stroke comes first, then top and right in one stroke, then the bottom stroke.
More complicated characters basically become a squiggle, and it's really magical, if you think about it, that you can just read it. I mean, look at figures 5, 6, 7 in that article.
For me, a nice illustration of Moravec's paradox of AI [1]: what we think is difficult (chess, differentiation, ...) is easy, and what we think is easy (reading handwriting, putting an egg in the fridge, walking (!) and opening a door [2]) is hard...
[1] https://en.wikipedia.org/wiki/Moravec%27s_paradox
[2] https://www.youtube.com/watch?v=g0TaYhjpOfo ("A Compilation of Robots Falling Down at the [2015] DARPA Robotics Challenge")
It is when you are starting out, but it's not so bad once you get used to it. The best way to learn to read Chinese handwriting is to learn to write Chinese handwriting, and I don't just mean practising writing by hand, but to work through a textbook aimed at teaching cursive Chinese.
This book is a good introduction in English [0], and if you are comfortable with Chinese only, this book is excellent and also far more comprehensive [1].
Once you've gone through those books, you'll be able to read most handwritten stuff without too much struggle.
0: https://www.amazon.com/Chinese-Cursive-Script-Introduction-P...
1: https://www.amazon.cn/图书/dp/B00C907XM2
I once worked with a paediatric neurologist whose handwriting was so bad, even he couldn't read it. We had a patient with a referral from him which we couldn't read, took it down the hall to him... he scrunched up his face... and gave up reading it: "Well, I know the patient, and they need...". Absolutely lovely bloke, but I'd hate to get a referral from him without being able to talk to him :)
Yes. All of the 的 examples are clearly 的 that I'd get without being primed.
Figure 7 is more tricky but I'd know 五 without being primed, and 王 definitely after being primed or if I was reading handwriting that had more context.
I once lightheartedly asked my teacher if exercise books in china had bigger gaps between the lines, as I couldn't see how to fit a vertically-complex hanzi in. She wandered over and scrawled something that neatly fit in the line. "How the hell did you do that... ?" :)
A year is probably still too early to be able to read handwriting well. You probably need to have about 2,000 known characters under your belt and also have a good feel for the language such that you are able to fill in gaps based on context - that is be able to complete parts of the sentence based on knowledge of what words/characters are likely to fit.
There are many cursive characters that reduce to something quite similar so having knowledge of what fits in the sentence makes things easier.
As an example, with 王 above, by itself it might not look like anything you can recognise, but if it was written like that on a note starting out like 尊敬的王老师 then from context it narrows down the possibilities significantly.
> She wandered over and scrawled something that neatly fit in the line. "How the hell did you do that... ?" :)
Practice! It's one thing to know how to write a character, it's another thing to have the fine motor skills required to write it proportionally within a confined space, and you can only develop that skill through practice. If you're interested, one of my side projects [0] is a site that lets you generate custom Chinese worksheets of varying grid sizes and fonts (including some handwriting fonts) and other things, which can be useful for such practice.
0: https://www.hanzigrids.com/
Thanks for the link - it will help and I've been meaning to look for one (phone apps with your finger aren't quite the same).
'Pretty much every night' is the most important part.
Learning Chinese characters is a grind and you just need to keep chipping away at it.
Pick a nice sustainable number of characters (maybe 6, maybe 3, maybe 10 but something that takes maybe 30 minutes to an hour to do including revision) and then do it every day without fail.
Number of days in a row spent learning is far more important than number of new words learnt per day, so always try to optimise for the former rather than the latter.
At some point, almost every Chinese learner who has spent more than 3 years learning the language has looked back and thought 'why oh why didn't I just learn 3 characters a day'.
In this case, I had no idea there was cursive Chinese. I've since gone on a search and learned more. I'd have never done that, had I not seen your comment.
Thanks.
It seems so much more difficult if you can't use the stroke order information, or the stroke direction. Even the speed over the stroke should be able to help you along, especially as you say for cursive characters. Without stroke order it seems almost impossible to decode those in a systematic way.
I suppose this is a general feature of deep learning, it's still not clear how to make good use of all the structural information we possess about a problem, instead a very brute force approach is chosen.
I use Apple's Chinese handwriting keyboard almost every day, living in Taiwan. Unfortunately it only shows 4 possible choices, and often the character I need is not one of those. So I use my own Pingtype keyboard.
https://pingtype.github.io
Enter a character into the bottom text field, click Search, and see the character decomposition above, and the characters & words containing that character below. Then when you're ready, click Insert to add it to the text field at the top for translation.
I have no idea how to do marketing for this tool, and commenting here is the only way I've had ~1300 views in the last 6 months since I published it.
The best English handwriting recognition software I've found is the old Palm Pilot Graffiti writing system. It's not really developed since the late 90's / 00s. Back in the day, it was blazing fast (and pretty accurate after learning the method) compared to shitty small mechanical keyboards of the era.
After diving into Chinese I realized how deficient the English HR landscape was and how badly under-researched the area is. I think the standard QWERTY keyboard is just too efficient for the Euro zone languages to make the research investment worthwhile. My 2 cents / guess.
There are Graffiti like apps available on IOS. What I've found is that with english keyboard auto-correct, my typing is so much faster without Graffiti using the standard Apple IOS English keyboard. Kind of a disappointment...
Another conjecture, I suspect much of the morbidity in English HR research has something to do with Graffiti patent lock up between what ever remains of Palm, Xerox, and some heretofore unheard of company called Communication Intelligence Corporation.
I stand corrected on everything stated above!
I'll have to borrow a friend's phone to play with this and see how it compares to the old Graffiti.