This looks very cool and could come in quite handy.
In case anyone from the project is monitoring - text selection did seem to work fine for me in FireFox (ESR 24.3) despite the "Not Supported" text being displayed.
Yeah, I just haven't gotten around packaging the whole thing as a Firefox Addon. It's actually technically possible to run the whole thing on a normal unprivileged webpage (in fact, that's my development environment).
Doesn't work great. Went to reddit's advice animal page to try it out and it doesn't seem to work with livememe (I think they have an invisible layer over their images to try and block hot linking).
Bottom: TN[ FACTTNATl'M MAWING TNISM[M[ g
INST[AD of DRIVING D[TERMIN[D TN#rWASA ll[
Maybe it needs to be a certain font for better results. Still pretty cool. Hopefully all the kinks get worked out. I would definitely find this useful.
EDIT: need to make sure the language is set to "internet meme" and it works much better.
By default it uses Ocrad.js, a pure javascript OCR engine (ported via emscripten, see http://antimatter15.github.io/ocrad.js/demo.html). But if you right click on the selection and change the language to "Internet Meme", it should transcribe it correctly (note that this sends the selection off to a server for remote processing- It's not the default for privacy and scalability considerations at the moment).
In any case, pretty cool project, I'm a bit amazed how far we've come since I've last played with OCRs (and defeated one bad CAPTCHA implementation, still in use at pastebin.com it seems).
The biggest thing I'd like to see is enabling in-page (control/command-f) search. In my quick scan through the page it looks like it doesn't do that… is that right? Are there plans to add invisible text to the DOM that control-f can find?
One problem with that is that it processes images lazily. It continually extrapolates cursor moments ~1 second into the future and processes those relevant parts of relevant images. But it should be possible that after an image is processed (or even eagerly by looking up previously recognized regions from the cached OCR server), the page could be made Ctrl+F-able.
Randall Munroe's handwriting is a bit difficult to OCR because a lot of the letters are smushed together close enough that the it's not possible to unambiguously segment the text into distinct letters (which is a necessary first step in any OCR engine that I'm aware of). Maybe Google's (or Vicarious's) magical convolutional neural net that can solve CAPTCHAs would fare better.
> it's not possible to unambiguously segment the text into distinct letters (which is a necessary first step in any OCR engine that I'm aware of)
This made me realize I never saw such a thing as OWR, i.e. a software that would first try to recognized whole words, then go down to character level if no satisfying match found.
> it's not possible to unambiguously segment the text into distinct letters (which is a necessary first step in any OCR engine that I'm aware of)
In my experience, the ability to handle overlapping letters (which is very common on type-written text and professionally typeset material) is one of the key things that separate the relatively lightweight OCRs (like Ocrad and GOCR) from the big complicated ones (Tesseract, Cuneiform, Abbyy etc). Whitespace character segmentation cannot be taken for granted if you want to do any useful OCR of "historical" material.
Cool, I implemented the stroke width transform for text detection about a year ago. Nice to see someone else using implementing it, but I'm pretty sure convolutional neural nets do a better job at text localization.
One of the rules for the heuristic for what images to ignore is that it needs to have over 19,000 square px, and that first image was a bit under that.
Cool idea, definitely worth exploring the possibilities. A quick run showed me that it often interprets the "i" as "l" whenever the the gap between the line and dot is not apparant
1. The implementation of Stroke Width Transform is not super good. So far, http://libccv.org/ has the best implementation of SWT. But again, you can neither make the head nor the tail of that implementation.
2. There are just too many false text regions and the text detection accuracy is no where near what you can call good. A mixed use of multiple OCR engines might give better results.
All that said, you can't take away the cleverness of the application of detecting text. Mind == Blown, on that area.
I actually modeled my implementation after libccv's implementation. Part of what libccv seems to do is to run it multiple times at different scales, which isn't something that's very computationally feasable for a pure javascript implementation. My implementation has a second stage color filter which refines the SWT (this is something of a tradeoff that improves accuracy for machine-generated text and reduces accuracy for natural scenes, and I'm under the impression that the corpus used by SWT focuses on the latter).
Ocrad is being used as the default because it runs locally and it's small enough that it's easy to ship with. The remote OCR engine uses Tesseract which gets much closer to acceptable in a lot of circumstances.
But there is a lot of work which can be done to improve it. I have a friend who constantly nags me for not having a solid test corpus to run regression analysis/parameter tuning/science. Certainly it lacks the rigor of an academic and scientific endeavor, but I've always imagined this as a sort of advanced proof of concept. I think the application of transparent and automatic computer vision, deserves to be part of the interaction paradigm for the next generation of operating systems and browsers.
Curious, what makes you find those to be better than Chrome? I recently switched to FF for a variety of random work reasons, and found it so much worse than chrome (basic UI, dev tools, speed) that I switched back asap. Maybe I'm missing something awesome about them.
I use Safari, and I find it to be better than Chrome because it's easier to sync with my iPhone and iPad, and with iCloud keychain even my passwords are synced.
Right this very moment (well, a few moments ago when I wasn't procrastinating on HN) I was in the midst of extracting data from a client's old website in preparation of creating a new website.
A lot of that data is contained within images.
From a few preliminary tests, I'm hugely impressed. This seems on-par with any other OCR software I've used, and the fact that it happens in realtime in the browser is amazing.
I tried it on a piece of content I'd just had to type out, that was originally in an image. Typing out the content took about 10 minutes. Copying and pasting with Naptha, and then making some minor edits/corrections, did the same thing in about 2 minutes.
There's actually been a bit of research on the error rates you need to beat for OCR to be cost-effective vs. having people re-type. I don't have the references handy, but I believe it's generally cost effective to OCR with error rates up to nearly 2%, and most current "consumer grade" OCR is well below 1% error rates for scans that aren't absolutely atrociously poor quality.
My Msc thesis was on reducing OCR error rates by pre-processing of various forms, and while I managed to get some reduction in error rates, one of the things I found was actually that given how low the error rates generally was to begin with, you have a very tiny budget in terms of extra processing time before further error reduction just isn't worth it - if a human needs to check the document for errors anyway, a "quick and dirty" scan+OCR is often far better than even spending the time to get "as good as possible" results. Spending even a few extra seconds per page to place the page perfectly in a scanner, or waiting a few extra seconds for more complicated processing, can be a net loss.
It's a perfect example of "worse is better": OCR, at least for typed text, is good enough today that the best available solutions aren't really worthwhile to spend resources on (for users) unless/until they give results so perfect it doesn't need to be checked by a person afterwards.
It was suggested to me by a friend that to get good OCR results, run it through the scanner/OCR twice, then diff the results. Usually one or the other will get it right, and if you run the two results through a difference editor like 'meld', it's quick to fix.
That may work for some cases, and especially with horrible OCR engines and low quality scanners, but frankly when I did my research into this, the results varied extremely little from run to run, and you could usually easily identify specific artefacts in the source that tripped the engine up (rather than problems with the quality of the scan). E.g. letters that were damaged, or had run together, creases in the paper etc.
With really low res scanners I can image it could make a big difference.
Back in the late 90's I worked for a company that did a lot of OCRing and they ran the same image through multiple engines and then manually corrected the results. I think they had 3 engines, all from different companies, which processed all images and put the results into a custom format. Human beings were then employed to manually merge and correct the final text. It worked fairly well, especially considering the hardware/software available at the time.
The biggest problem was stuffing too many files into an NTFS directory. Apparently, NTFS didn't like tens of thousands of files in one directory. :)
To a certain extent, of course. The 2% was based on the assumption that if you are benchmarking against re-typing, you expect the same kind of quality you'd get from having a good typist re-typing the documents.
From my own experiments, I tend to find that you can read through and correct errors only relatively marginally faster than you can type because you either follow along with the cursor or need to be able to position the cursor very quickly when you find an error, and as the error rate increases, trying to position the cursor to each error very quickly gets too slow.
Dropping accuracy in your effort to correct the text doesn't really seem to speed things up much. You likely speed it up if you're willing to assume that anything that passes the spellchecker is ok (but it won't be, especially as modern OCR's often try to rely on data about sequences of letters, or dictionaries, when they're uncertain about characters)
If you're ok with lower accuracy, e.g. for search, and the alternative is not processing the document at all, then it'd be drastically different.
I remember seeing that from the project list and really wishing I could download it right away.
Just another example that the "idea are worthless!" saying is bullshit. This was a great idea, anyone implementing it first decently would get success with it.
Now that is pretty damn cool. Will help at work when marketing people do not copy paste email/article and just put screenshot of it and if you want to quite something from that picture...
@antimatter15, i have a project that does client-side image analysis and decompses document structures. it looks like your OCR code would be a great replacement for the server-side Tesseract ocr i currently use :)
here's what the project does now with js + web workers:
processing time is < 1500ms in Chrome and < 2000ms in FF
the code is open source, though using it isnt yet polished. i'm working slowly on a blog post series to detail how to use the lib(s). https://github.com/leeoniya/pXY.js
140 comments
[ 2.4 ms ] story [ 236 ms ] threadIn case anyone from the project is monitoring - text selection did seem to work fine for me in FireFox (ESR 24.3) despite the "Not Supported" text being displayed.
FF 28 seems to be working fine with the "Weenie Hut Jr." version...is it just the add-on that isn't supported?
awesome tech, btw
http://en.wikipedia.org/wiki/Bookmarklet
Here is a copy/paste example from imgur:
http://i.imgur.com/sKQXx8v.jpg
Top: vou SAID w[ W[R[ |[AVINĞ`ON TIM[TOAV
Bottom: TN[ FACTTNATl'M MAWING TNISM[M[ g INST[AD of DRIVING D[TERMIN[D TN#rWASA ll[
Maybe it needs to be a certain font for better results. Still pretty cool. Hopefully all the kinks get worked out. I would definitely find this useful.
EDIT: need to make sure the language is set to "internet meme" and it works much better.
Top: YOU SAID WE WERE LEAVING'ON TIME:TODAY
Bottom: THE FACT THAT I'M MAKING THIS MEME INSTEAD OF DRIVING DETERMINED THAT WAS A LIE
Next time I'll RTFM.
YOU SAID WE WERE LEAVING'ON TIME:TODAY
TN[ FACTTNAT |'M MAWING TNIS M[M[ INST[AD of DRIVING D[TERMIN[D TN#rWASA ll[
I imagine that the thick outline of the font makes it hard to detect the edge of the letters, especially since it obscures the true "background".
e: using the Internet Meme language worked much better!
YOU SAID WE WERE LEAVING'ON TIME:TODAY
(:/J
THE FACT THAT I'M MAKING THIS MEME INSTEAD OF DRIVING DETERMINED THAT WAS A LIE
In any case, pretty cool project, I'm a bit amazed how far we've come since I've last played with OCRs (and defeated one bad CAPTCHA implementation, still in use at pastebin.com it seems).
http://www.xkcd.com/ bottom line here is recognized as: "T1EN°5'lI'ONAl.1?E£ONNH\56PNCE(YHCEPlP6fiN(N)SURLH’PR3AO-i‘lDlsIr'£7E‘5IJ%z"
This made me realize I never saw such a thing as OWR, i.e. a software that would first try to recognized whole words, then go down to character level if no satisfying match found.
Found out this exists already: https://en.wikipedia.org/wiki/Intelligent_word_recognition
In my experience, the ability to handle overlapping letters (which is very common on type-written text and professionally typeset material) is one of the key things that separate the relatively lightweight OCRs (like Ocrad and GOCR) from the big complicated ones (Tesseract, Cuneiform, Abbyy etc). Whitespace character segmentation cannot be taken for granted if you want to do any useful OCR of "historical" material.
Maybe soon I won't feel guilty for leaving my alt attributes empty.
Also for: http://www.wsoddata.com/clients/8bec9b10/ads/300x250_static/... It can't get the top-right text correctly
Awesome tech though
Sarcasm is hard to read on the internet. I'm usually pretty good at it, but this one flew right past me.
1. The implementation of Stroke Width Transform is not super good. So far, http://libccv.org/ has the best implementation of SWT. But again, you can neither make the head nor the tail of that implementation.
2. There are just too many false text regions and the text detection accuracy is no where near what you can call good. A mixed use of multiple OCR engines might give better results.
All that said, you can't take away the cleverness of the application of detecting text. Mind == Blown, on that area.
Ocrad is being used as the default because it runs locally and it's small enough that it's easy to ship with. The remote OCR engine uses Tesseract which gets much closer to acceptable in a lot of circumstances.
But there is a lot of work which can be done to improve it. I have a friend who constantly nags me for not having a solid test corpus to run regression analysis/parameter tuning/science. Certainly it lacks the rigor of an academic and scientific endeavor, but I've always imagined this as a sort of advanced proof of concept. I think the application of transparent and automatic computer vision, deserves to be part of the interaction paradigm for the next generation of operating systems and browsers.
Right this very moment (well, a few moments ago when I wasn't procrastinating on HN) I was in the midst of extracting data from a client's old website in preparation of creating a new website.
A lot of that data is contained within images.
From a few preliminary tests, I'm hugely impressed. This seems on-par with any other OCR software I've used, and the fact that it happens in realtime in the browser is amazing.
I tried it on a piece of content I'd just had to type out, that was originally in an image. Typing out the content took about 10 minutes. Copying and pasting with Naptha, and then making some minor edits/corrections, did the same thing in about 2 minutes.
My Msc thesis was on reducing OCR error rates by pre-processing of various forms, and while I managed to get some reduction in error rates, one of the things I found was actually that given how low the error rates generally was to begin with, you have a very tiny budget in terms of extra processing time before further error reduction just isn't worth it - if a human needs to check the document for errors anyway, a "quick and dirty" scan+OCR is often far better than even spending the time to get "as good as possible" results. Spending even a few extra seconds per page to place the page perfectly in a scanner, or waiting a few extra seconds for more complicated processing, can be a net loss.
It's a perfect example of "worse is better": OCR, at least for typed text, is good enough today that the best available solutions aren't really worthwhile to spend resources on (for users) unless/until they give results so perfect it doesn't need to be checked by a person afterwards.
With really low res scanners I can image it could make a big difference.
The biggest problem was stuffing too many files into an NTFS directory. Apparently, NTFS didn't like tens of thousands of files in one directory. :)
Doesn't that depend entirely on what you're using the text for and how accurate it needs to be?
From my own experiments, I tend to find that you can read through and correct errors only relatively marginally faster than you can type because you either follow along with the cursor or need to be able to position the cursor very quickly when you find an error, and as the error rate increases, trying to position the cursor to each error very quickly gets too slow.
Dropping accuracy in your effort to correct the text doesn't really seem to speed things up much. You likely speed it up if you're willing to assume that anything that passes the spellchecker is ok (but it won't be, especially as modern OCR's often try to rely on data about sequences of letters, or dictionaries, when they're uncertain about characters)
If you're ok with lower accuracy, e.g. for search, and the alternative is not processing the document at all, then it'd be drastically different.
I remember seeing that from the project list and really wishing I could download it right away.
Just another example that the "idea are worthless!" saying is bullshit. This was a great idea, anyone implementing it first decently would get success with it.
http://bostinno.streetwise.co/all-series/photos-recap-and-wi...
https://github.com/vincentsiao/Lightboard
here's what the project does now with js + web workers:
http://i.imgur.com/QvXSkY2.png
processing time is < 1500ms in Chrome and < 2000ms in FF
the code is open source, though using it isnt yet polished. i'm working slowly on a blog post series to detail how to use the lib(s). https://github.com/leeoniya/pXY.js
a walkthrough of the base lib is here: http://o-0.me/pXY/
But in my experience, the recognition quality isn't good enough to replace Tesseract if you have that capability.