This is essentially a solved problem. Whenever someone sends me a screenshot that contains any text information (tables, etc), I pass it to an LLM and it correctly interprets the content of it. On modern versions of macOS you can just select text in images relatively painlessly, too.
Agreed! The one that I really don't like is that social platforms promote / prefer screenshots of text. Search engines promote sites that link to themselves. All the good parts of URLs are missing. How often I see something interesting, just to realize it's a screenshot and I have to go dig around myself figuring out where it came from.
Preview on Mac does automatic OCR. I'm sure other tools exist that are similarly friction-free on other platforms, but it took me under 5 seconds (drag the image from the webpage into my downloads folder, click on it, and then select the relevant snippet and CMD+C to copy it).
I imagine I'd have similar frustrations if I couldn't copy-paste the text easily though!
I agree that screenshots of text that are cut off from essential context are enough to make me pull my hair out, it creates so much extra work— but the modern feature of automatic text recognition in screenshots and images that allows for copy and paste has been incredible. Along with indexing that allows it to be searched, regular screenshots have become one of the most robust and future proof ways for me to preserve context from my workspace. When I look back into archived screenshots it helps me to recapture all kinds of things that I wouldn’t have thought to explicitly record.
The ability to highlight/copy/etc text on Macs/iOS these days is such a killer feature. I use it almost every day, both for copying/translating text in screenshots or taking photos of text to then copy it into my notes later (eg school notice boards or event posters etc).
It gave me a "living in the future" feeling the day someone sent me a picture of a phone number through imessage. Barely thinking, I pressed on the phone number in the image and I was prompted to call it. It was like technology and primitive intuition teamed up to create that moment.
This article is a specific case of a more general piece of advice: ask questions well (provide context like clickable links, trim down your query to the minimal reproducible case, pose high-precision questions, etc.).
I disagree. I use screenshots all the time, because it:
- Preserves the full 80 character width without line-wrapping, which destroys readability
- Guarantees monospace, so tabular data doesn't get all misaligned
- Preserves a good coding font, so it doesn't come out as some hairline-width Courier on the other end
- Preserves syntax highlighting, very helpful
Obviously if somebody needs a whole file or whole log, then send the whole thing as an attachment. But very often I'll still include a screenshot of the relevant part. With line numbers, it's not difficult to jump to the right part of the attached file.
Screenshots are incredibly useful for keeping code and terminal output looking like code and terminal output, and not getting completely mangled in an e-mail or chat message being read on a mobile device or in a narrow column.
I personally hate screenshots of kernel panics. Or anything else where you might be dealing with 64-bit hex addresses like "0xffffffff81b7ed80"
Typing that from a picture is infinitely more error prone than just cut/paste.
Screenshots of text! Luxury! In my day, the screenshots were embedded in a Word document too.
But I can't be the only one appalled at the suggestion to use an LLM to parse the text. The sheer, prodigious waste of computing power, just to round-trip text to an image and back to text, when what's really missing is a computer user interface that makes it as simple to send text or other snippets as it is to send screenshots.
When I get these, there's usually enough context that I can find the actual text.
That being said, I've had to twist some arms in a previous job for new employees attaching screenshots of a log viewer instead of the whole logs. The big problem was training: Once I made it very clear to the entire team that unedited logs were critical to solving problems, management made sure that all newcomers knew how to attach unedited logs.
Like other commenters point out, automatic OCR on Apple platforms is a godsend, and it's such a great use of our modern AI capabilities that it should be a standard feature in every document viewer on every platform.
Another thing I wish was more common is metadata in screenshots, especially on phones. Eg if I take a screenshot of a picture in Instagram, I wish a URL of the picture was embedded (eg instagram.com/p/ABCD1234/). If I take a screenshot in the browser, include the URL that's being viewed (+ path to the DOM element in the viewport). If I take a screenshot in a maps app, include the bounding coordinates. If I take a screenshot in a PDF viewer, include a SHA1 hash of the document being viewed + offset in the document so that if I send the screenshot to someone else with the same document, it can seamlessly link to it. Etc etc.
There are probably privacy concerns to solve here, but no idea is new in computer science and I'm pretty sure some grad student somewhere has already explored the topic in depth (it just never made it to mainstream computing platforms).
It feels like screenshots have become the de facto common denominator in our mobile computing era, since platforms have abstracted files away from us. Lots of people who have only ever used phones as their main computing devices are confused when it comes to files, but everyone seems to understand screenshots.
My preference -- Link or attachment to the full document or code in context (if needed) ... along with screenshot of a relevant portion. (Many times the former is optional because there is enough context already.)
It is extra work to do both but I like to be through even when asking for help. Even if the other side doesn't need it -- because I myself might not remember all the nuances when I refer to that conversation later.
Also screenshot preserves (before any fixes) the exact way things looked when I confronted a certain situation. The visual of the screenshot serves as a much stronger reminder of that situation and my thinking ...way better than mere copy pasted text.
> I have to either very carefully type some of the code into a search box or (these days) get my coding agent to find the relevant module for me.
Your coding agent is not very smart if it can't deal with something as simple as OCR'ing an image and processing all the references in it, or letting you just select text from an image and searching or copying to the clipboard.
76 comments
[ 3.0 ms ] story [ 62.7 ms ] threadLinux desktop users will get there one day.
I imagine I'd have similar frustrations if I couldn't copy-paste the text easily though!
What about just asking them what file that is?
- Preserves the full 80 character width without line-wrapping, which destroys readability
- Guarantees monospace, so tabular data doesn't get all misaligned
- Preserves a good coding font, so it doesn't come out as some hairline-width Courier on the other end
- Preserves syntax highlighting, very helpful
Obviously if somebody needs a whole file or whole log, then send the whole thing as an attachment. But very often I'll still include a screenshot of the relevant part. With line numbers, it's not difficult to jump to the right part of the attached file.
Screenshots are incredibly useful for keeping code and terminal output looking like code and terminal output, and not getting completely mangled in an e-mail or chat message being read on a mobile device or in a narrow column.
Nothing against screenshots unless they are lacking context
But I can't be the only one appalled at the suggestion to use an LLM to parse the text. The sheer, prodigious waste of computing power, just to round-trip text to an image and back to text, when what's really missing is a computer user interface that makes it as simple to send text or other snippets as it is to send screenshots.
That being said, I've had to twist some arms in a previous job for new employees attaching screenshots of a log viewer instead of the whole logs. The big problem was training: Once I made it very clear to the entire team that unedited logs were critical to solving problems, management made sure that all newcomers knew how to attach unedited logs.
Another thing I wish was more common is metadata in screenshots, especially on phones. Eg if I take a screenshot of a picture in Instagram, I wish a URL of the picture was embedded (eg instagram.com/p/ABCD1234/). If I take a screenshot in the browser, include the URL that's being viewed (+ path to the DOM element in the viewport). If I take a screenshot in a maps app, include the bounding coordinates. If I take a screenshot in a PDF viewer, include a SHA1 hash of the document being viewed + offset in the document so that if I send the screenshot to someone else with the same document, it can seamlessly link to it. Etc etc.
There are probably privacy concerns to solve here, but no idea is new in computer science and I'm pretty sure some grad student somewhere has already explored the topic in depth (it just never made it to mainstream computing platforms).
It feels like screenshots have become the de facto common denominator in our mobile computing era, since platforms have abstracted files away from us. Lots of people who have only ever used phones as their main computing devices are confused when it comes to files, but everyone seems to understand screenshots.
Also, necessary shout out to Screenshot Conf! https://screenshot.arquipelago.org
SVG maybe?
It is extra work to do both but I like to be through even when asking for help. Even if the other side doesn't need it -- because I myself might not remember all the nuances when I refer to that conversation later.
Also screenshot preserves (before any fixes) the exact way things looked when I confronted a certain situation. The visual of the screenshot serves as a much stronger reminder of that situation and my thinking ...way better than mere copy pasted text.
Your coding agent is not very smart if it can't deal with something as simple as OCR'ing an image and processing all the references in it, or letting you just select text from an image and searching or copying to the clipboard.