I’m frustrated that there is no easy way to capture text from pictures. Thank you for building this.
This type of functionality should be integrated or readily available in MacOS.
I would love to have a way to do basic math operations, unit conversions, etc. without resorting to write them in spotlight. For example hovering over a price, it should convert it to a different currency. Or compare similar types of informations from different sources.
The problem with Naptha is that the OCR quality of Tesseract (used by Project Naptha) is not very good for screenshots. Historically Tesseract has been optimized for text documents. So the OCR results of this new macOCR app are significantly better.
Another alternative is Copyfish. It is cross-platform and uses cloud ocr:
Sure but it's fairly assured that 80% at least are from the USA, possibly higher. Also it's fairly easy to extrapolate the premise of "the 10000" to include the rest of the world
So there actually is a very quiet functionality hiding in MacOS Notes.app I noticed recently.
If you take any screenshot to clipboard that includes some text, then paste it into Notes, it will silently name the resulting image file using the text pictured.
Not terribly useful but I did find it helpful once while taking screenshots for documentation.
Not a Mac user or developer, poking at the source code because I might be interested in building a Linux equivalent. Why is code so damn complicated these days? What does all this crap do? Why isn't the source code of things these days 100% human-readable?
Why are you complaining about code that is (hopefully) obviously auto-generated and not "written" by the author, but still necessary nonetheless? It's because you are "Not a Mac user or developer", so we should be a bit more considerate on you
Maybe the issue your complaint unexpectedly tries to surface is that many awesome, highly useful projects like this one depend on code that isn't human-readable. I think this is a noteworthy point and should be discussed more often.
But then again, having the code in some form of source control /at all/ is far, far better than the alternative, which is depending on some instructions in a README.md or just hoping the user will know how to use XCode properly such that the real contribution of the project is used
Maybe your post also somewhat points out the fact that to newcomers or people looking at XCode code (auto-generated or otherwise) for the first time, it's /not/ obvious which files you should be looking at, and so we should give the parent poster some slack. Is this a problem that projects should worry about or take into consideration when auto-generated code starts to mix with non-generated code in source control?
This looks like a .xcodeproj file. This isn't source code, it's auto generated metadata for the Xcode IDE. You have to check it into source control because Xcode is dumb and sometimes breaks your project if it can't find its own metadata.
That is not the source code, it is XCode IDE project file, you don't need it.
This application calls the native macOS libraries to do the OCR so I don't think you'd find anything useful here to do a Linux port - you can certainly use the idea and combine it with a linux compatible OCR library though.
I’ve tried with Tesseract, the Vision API, Google Vision and Azure’s equivalent (also kicked off using an Alfred workflow).
By far the best for text was Google Vision and then Azure. Whilst Google Cloud and Azure both also do handwriting recognition, Azure did better at this.
The cloud platforms performed better than pure on device with Apple’s vision API outperforming Tesseract.
Good to know! I thought most cloud OCR services were paid only but it turns out google vision has 1000 free invocations a month with Azure at 5000. That should be plenty for most people (certainly for me).
Do you have the source for those workflows/would you be willing to share them?
Very nice. Does this work with screen fonts / pixel perfect fonts too? That'd be my biggest gripe with all the OCR tools: what looks like the simplest fonts of them all to OCR are usually the one the less well detected.
For example using tesseract on Linux and trying to OCR the "terminus" font I get better result by first resizing the screenshot to something bigger (and blurry) and even then it's far from perfect OCR'ing. When in the first place it's a pixel perfect font...
(and, yes, there are cases where OCR'ing screen fonts make sense)
I actually use another version with Google Vision (it is in a whole other class to on device OCR), and it is quite challenging to make Google Vision not work. For example it even works for hand written text.
I’ve always wondered whether you couldn’t build a not-fully-ML app for screen-font recognition, that just takes a bunch of fonts, renders out every individual glyph at every size, trims them, converts them to an alpha mask, and then generates an indexible image fingerprint of said alpha mask.
The OCR software would then just need to be smart enough to recognize “things that look like glyphs”, and put bounding boxes around them; and everything from there could be implemented in logic, rather than a model. (Just apply the same transforms to the thing in the bounding box, and then search the fingerprint DB.)
Outstanding. I have a question, and it's not a complaint or a request for anything except a bit of data: What kinds of text does it support, as in, would it work for handwritten English, or German rotated 45 degrees, or Japanese...?
I dug into the code, looks like it use VNRecognizeTextRequest[1] from Vision framework by Apple here [2].
And the docs says
>By default, a text recognition request first locates all possible glyphs or characters in the input image, then analyzes each string.
Since the code doesn't specify any preferred languages, I think it would try to detect any languages supported by the framework.
From short googling, I found this thread [3]. Looks like the supported languages depends on the MacOS version, and it only support en, fr, it, de, es, pt, zh on Big Sur.
Reminds me of PowerSnap on the Amiga, that did a very simple version of this (it did not run a proper OCR engine, but relied on a very close match with one of the installed fonts, so worked great for UI of apps that didn't support cut and paste, not so great for images unless they were created using the same font and not scaled, but still very useful).
I didn't try the app yet but I use an app called "Yomiwa" to OCR Japanese. The biggest issue I find is that it seems to be trained on black on white images so if I try to OCR a sign with different colors, or some text on a product or menu, it often fails. For example I just tried it on the ingredients of the label of the soda I'm drinking which is white on black and it's completely failing.
OCRmyPDF does that. Given the name I assume it's only for PDFs given the name, but theoretically it should work for any image without the need to extract it from the PDF.
I've been successfully using Mathpix Snip [1] to do general OCR for quite some time.
It's not as well communicated as its initial purpose of applying OCR to LaTeX equations, but it currently supports much more than that, such as mixed text/math and tables.
On a personal note, I'm actually surprised it wasn't mentioned thus far in this thread.
Great project! This is one of those tools that solves a problem that's right in front of you. I can see this being built into screenshot software, just like mark-up tools are these days.
Also, does anyone know of a similar project for Linux?
No project I'm aware of, but I did cobble together a script to do something similar. In my I case also wanted it to work with copyq so there's some noise related to it.
tesseract insists on adding on txt extension and what I assume is some intermediary file txtr, making it awkward to use with mktemp. Probably explained in the manual which I skipped.
But like others have said tesseract is not very reliable, at least with default settings -- it's common for it to add extra spaces or various single quotes, or omit spaces.
I wasn't familiar with that "-r", but FWIW the gsed man page says "(for portability use POSIX -E)"
Abutting that "r" option against the "i" option is likely why you ended up with a file named .txtr and therefore implies that it did not actually hear the "-r" you intended
I've had the best luck picking an actual backup suffix such as "-i.bak" or "-i~" to keep BSD sed and GNU sed on the same page, although I've also seen scripts that go as far as "--version" sniffing and changing the actual invocation as "${SED_I} -E" type stuff
You are absolutely correct! I wasn't aware that "-i" takes an optional suffix, so it didn't even occur to me to look at the sed line as possible cause for this extension weirdness.
Yeah, but "optional" in the _worst possible way_ since, due to the getopt library difference, the GNU version wants any empty suffix value abutted, and the BSD version wants it separated away from the "-i", burning thousands of hours of humanity over the years :-(
Woe unto those who write scripts as "sed -i -e /whatever/" since for half(?) of their users they'll end up with "somefile-e"
It requires self-written macro, however it can do much more than that, including parsing & formatting OCRed text. For one job I went directly Image->OCR->File so I could copy OCRs into text for some non-elegant hardcodes ;-)
Is the OCR part readily available using this `VNRecognizedTextObservation`? So it suggests possible option and the best one needs to be chosen? Would love to see how it's implemented.
Wow this is very nice, if this can be made really solid, then you are very close to a kyc as a service product.
Meaning, compare issuing date with the official formats for the time frames, then compare if the fonts and checksums are correct and an API to send the results to the customers. Most platforms dealing with fiat are waiting for something like that, they are suffering from doing this manually or semi manually with prohibitive employee costs.
If this can be done platform agnostic and fraud proof, the companies will throw themselves at the product. Are you planning anything towards that direction?
Personally, I use a combination of tesseract and MacOS' screen capture to achieve the same thing. Very handy to have if you use an app like BetterTouchTool to run it with a quick hotkey.
nice, thanks for sharing! for anyone having trouble configuring the shell execution: use /bin/sh/ for launch path and "-c" (without quotes) for parameter. Otherwise BTT will autofill "(null)" and then it won't work and it won't tell you why
83 comments
[ 4.4 ms ] story [ 170 ms ] threadThis type of functionality should be integrated or readily available in MacOS.
I would love to have a way to do basic math operations, unit conversions, etc. without resorting to write them in spotlight. For example hovering over a price, it should convert it to a different currency. Or compare similar types of informations from different sources.
Your solution goes in this direction, thanks!
It should be pretty trivial for someone to hook this into https://insect.sh/.
https://projectnaptha.com/
We had this fully client-side in the browser all the way back in 2013.
Another alternative is Copyfish. It is cross-platform and uses cloud ocr:
https://github.com/A9T9/Copyfish
The cloud == someone else's computer. Never forget that.
> The cloud == someone else's computer.
I think everyone around here knows this and can make their own decisions, if and for what data they use "the cloud".
https://xkcd.com/1053/
If you take any screenshot to clipboard that includes some text, then paste it into Notes, it will silently name the resulting image file using the text pictured.
Not terribly useful but I did find it helpful once while taking screenshots for documentation.
/* Begin PBXBuildFile section / 0425D1C16E9B7E34F8EBCCFB229F6BCF / Pods-ocr-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = E52F12A9CD9DA185DB6C7CFAF9971233 / Pods-ocr-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; 69F017594F16B64B4E70E96B863F38D1 / Pods-ocr-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 812D67335813B22DFC54237ACEB07CC8 / Pods-ocr-dummy.m /; }; 9D8F5FD727B32865EE80BA6ACDA12AF4 / ScreenCapture.swift in Sources / = {isa = PBXBuildFile; fileRef = CAE82544998B753F1708876308FF330D / ScreenCapture.swift /; }; AC8C4224C366FAD03EFFDC427D793373 / ScreenCapture-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 2BFFD24873C787E751AFC41D8C497ECB / ScreenCapture-dummy.m /; }; BE8E791706F107976678CAA1DE681FA6 / ScreenCapture-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = B9D6CB7E3F7CD4599F66F1F010D4CADD / ScreenCapture-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; CA9117D8B1C22828347BFE8326E2F7D2 / ScreenRecorder.swift in Sources / = {isa = PBXBuildFile; fileRef = FC34AC3B539E1EFA3B0D1E086E1BA1D9 / ScreenRecorder.swift /; }; / End PBXBuildFile section */
You can achieve the same thing using Tesseract on Linux, or even better quality using Google Vision.
[1] https://files.littlebird.com.au/Shared-Image-2021-05-22-17-1...
Maybe the issue your complaint unexpectedly tries to surface is that many awesome, highly useful projects like this one depend on code that isn't human-readable. I think this is a noteworthy point and should be discussed more often.
But then again, having the code in some form of source control /at all/ is far, far better than the alternative, which is depending on some instructions in a README.md or just hoping the user will know how to use XCode properly such that the real contribution of the project is used
Maybe your post also somewhat points out the fact that to newcomers or people looking at XCode code (auto-generated or otherwise) for the first time, it's /not/ obvious which files you should be looking at, and so we should give the parent poster some slack. Is this a problem that projects should worry about or take into consideration when auto-generated code starts to mix with non-generated code in source control?
N.B.: the parent post was talking about ./Pods/Pods.xcodeproj/project.pbxproj https://github.com/schappim/macOCR/blob/ca9a6379e07a8e1a5eaa...
The actual code here that isn't just Xcode boilerplate is in this very simple to read file: https://github.com/schappim/macOCR/blob/master/ocr/main.swif...
This application calls the native macOS libraries to do the OCR so I don't think you'd find anything useful here to do a Linux port - you can certainly use the idea and combine it with a linux compatible OCR library though.
/usr/local/bin/ocr | say
Any idea on how this compares to tessaract (or other local OCR).
I currently have an Alfred workflow that invokes tessaract and it works decently well but the accuracy could be better.
By far the best for text was Google Vision and then Azure. Whilst Google Cloud and Azure both also do handwriting recognition, Azure did better at this.
The cloud platforms performed better than pure on device with Apple’s vision API outperforming Tesseract.
Do you have the source for those workflows/would you be willing to share them?
For example using tesseract on Linux and trying to OCR the "terminus" font I get better result by first resizing the screenshot to something bigger (and blurry) and even then it's far from perfect OCR'ing. When in the first place it's a pixel perfect font...
(and, yes, there are cases where OCR'ing screen fonts make sense)
The OCR software would then just need to be smart enough to recognize “things that look like glyphs”, and put bounding boxes around them; and everything from there could be implemented in logic, rather than a model. (Just apply the same transforms to the thing in the bounding box, and then search the fingerprint DB.)
I tweeted a GIF on how it looks like https://twitter.com/cheeaun/status/1395973544983425025.
And the docs says >By default, a text recognition request first locates all possible glyphs or characters in the input image, then analyzes each string.
Since the code doesn't specify any preferred languages, I think it would try to detect any languages supported by the framework.
From short googling, I found this thread [3]. Looks like the supported languages depends on the MacOS version, and it only support en, fr, it, de, es, pt, zh on Big Sur.
Not sure about the rotation though.
[1] https://developer.apple.com/documentation/vision/vnrecognize...
[2] https://github.com/schappim/macOCR/blob/master/ocr/main.swif...
[3] https://developer.apple.com/forums/thread/121048
https://github.com/jbarlow83/OCRmyPDF
I've been successfully using Mathpix Snip [1] to do general OCR for quite some time.
It's not as well communicated as its initial purpose of applying OCR to LaTeX equations, but it currently supports much more than that, such as mixed text/math and tables.
On a personal note, I'm actually surprised it wasn't mentioned thus far in this thread.
See more discussion here on HN in [2], [3].
[1] https://mathpix.com
[2] https://news.ycombinator.com/item?id=16535358
[3] https://news.ycombinator.com/item?id=21871780
Also, does anyone know of a similar project for Linux?
But like others have said tesseract is not very reliable, at least with default settings -- it's common for it to add extra spaces or various single quotes, or omit spaces.
Abutting that "r" option against the "i" option is likely why you ended up with a file named .txtr and therefore implies that it did not actually hear the "-r" you intended
I've had the best luck picking an actual backup suffix such as "-i.bak" or "-i~" to keep BSD sed and GNU sed on the same page, although I've also seen scripts that go as far as "--version" sniffing and changing the actual invocation as "${SED_I} -E" type stuff
Woe unto those who write scripts as "sed -i -e /whatever/" since for half(?) of their users they'll end up with "somefile-e"
I’m using (and I’d like to recommend) https://www.keyboardmaestro.com/ for this.
It requires self-written macro, however it can do much more than that, including parsing & formatting OCRed text. For one job I went directly Image->OCR->File so I could copy OCRs into text for some non-elegant hardcodes ;-)
https://owlocr.com/
I'll keep an eye on this one too though!
https://gist.github.com/jerieljan/f86843388c4ee9ecbc44d687a3...