Author here: the gist is that it's practically impossible to scroll to the beginning of an iMessage chat, and that's the only way to get to the beginning of a conversation. These old conversations take up several gigabytes of space, yet there's no good way to search them (because search for old messages is unreliable).
This aspect of the iMessage user experience has remained untouched for a decade, despite everyone's chats growing longer. Searching for particular iMessages sucks because the results miss the broader context of the conversation (a single message is not useful when you want to reread a conversation).
(Also just noting, as the post author I created a title better-suited to HN's 80 character limit.)
Related, I emailed Tim Cook about iPhone sensor future plans, and a high-level exec of like 25+ years wrote back to me within a week. Apple is remarkably good at getting back to cold emails to top execs.
Interesting, perhaps I’ll cold email them about the pinned contact feature in iMessage being totally worthless.
If it just pinned the message thread to the top and didn’t change the icon to a giant circle that sometimes doesn’t show notifications, it’d be fine. I pinned my GF’s message thread and I ended up missing lots of messages and notifications from her and ended up unpinning her so the notifications would show up.
Not to mention the comically huge circle icon that is centered when everything else is left aligned.
I had a problem that appeared to be a corner case with Apple Pay. I wrote a paper letter to Tim Cook about it, and within a week got contacted by a staff person in his office. He worked exclusively with me to identify issues I had not considered, and followed up on all of the direct effects and side effects until they were resolved.
I emailed Craig about several controversies and he was very quick to reply to me. We last talked about the rumor that Apple was giving iPhones trust scores.
I actually did try emailing Tim Cook a few months ago, but I got bounce backs. I tried tim@apple.com and timcook@apple.com but the mailer daemons said the addresses weren't registered. It sounds like a crazy thing to do, but I remembered how Steve Jobs would reply to users' emails and read an article where Tim said he set aside time to read users' emails.[0]
It would probably work better for the two Steves than it would for many others.
According to the below random site, there 218 Jobs and 10,602 Wozniaks. It claims 410,801 Steves.
I’m somewhat surprised that acknowledging outside emails is a thing. I know unsolicited ideas/content/etc. is commonly rejected without review in other fields. Is this successful beyond identifying technical bugs?
- I’m never filing a bug report ever again on a bug filing system where I don’t get to see the progress in a transparent manner or that I might have to pay for a developer account to see that progress (assuming that’s an option)
- I’m not wasting my time emailing a top level company executive to inform of decades old most basic of bugs because if they don’t know of it then that’s even a bigger problem.
If you can remember a term, any term, that was written in the message, you can search & jump to it directly. For my girlfriend and I, it was the bar we went to on our first date. This feature only gives you small subsets (a couple dozen, maybe) messages near the query result, but then you can just search for the last message again and get the next “page,” so to speak. Of course, remembering such a term is hard, which is what I assume you mean by “search for old messages is unreliable,” but it’s at least a possible workaround.
Hm, yet I have no problem searching phrases that go back at least five years.. I would say the burden of proof is on the author here to extract the database from the phone and prove that the messages didn’t get deleted, lost in an upgrade, or otherwise cleaned up.
Can’t you use something like iMazing (I’m not a paid shill just a satisfied user and there are many similar other products from competitors) and do an offline dump from the backup?
Article mentions this: “...there’s a whole slew of third party tools that slurp up your iMessages, and Apple should care about this, because having a strong demand for apps that process users’ messages should be seen as a privacy failure by the company.”
I save them on the expectation that Apple will someday release a better tool for browsing them. Especially the texts between me and my wife, they are a story of our life from the very first one.
I was in a similar situation with email. In 2005 or so I was migrating to new computer when I found a directory of photos I forgot to back up. Rather than burn them to DVD or upload them to my domain, I decided to just email them to myself. I sent a handful of images, then replied to that same email with the next 10, and so on, until the five dozen or so images were uploaded in a single email thread.
When I went to open the email on my new machine, the loading indicator just spun and spun until finally my browser crashed. When viewing an email conversation, Gmail will display previews of all the emails in the chain, including their attachments, which was causing my browser to crash. Next I tried connecting my Google account to Apple Mail, but Gmail's SMTP server would time out before the attachments were able to download.
After a while I forgot about the photos all together, until I randomly stumbled across them while searching my Gmail account for some other files I had sent myself around that time. At this point it was 2014 or so, and my browser was very easily able to open the messages so I could download the photos, which of course at that point I had zero interest in.
I used gmail to subscribe to lkml at some point. Then switched back to my own email server. Forgot to unsubscribe on gmail. A few years later I deleted that folder. And couldn't open Gmail anymore for weeks, everything would just time out.
With photos, I guess this could end up being around 1GB. Maybe this will push you off the free plan to the $1/mo plan, but I don't see it being that big of a needle-mover for services revenue.
I made the mistake of turning on automatic photo back up (I have no idea how to bulk copy photos off the iphone without a mac, I've given them up as lost honestly, most of them are random crap anyway) and now my icloud account is completely full and I keep getting told I need to send Apple money.
I definitely prefer my current phone that lets me just rsync ~/Pictures.
Not that I'm suggesting you should send Apple money: but it's like £0.79 per month for 50gb - less than £10/year. For me at least: that's pretty cheap compared to other services.
I paid $0.99 for photosync. It syncs to many cloud providers, and (most importantly to me) to ssh/sftp.
You can set it to sync whenever your phone is inside a particular location boundary (e.g. at home, where there is wifi), so it works transparently with zero ports open to the internet, and no firewall-bypassing cloud services.
My sister was having a problem where she was taking all the iCloud storage for our family because she keeps sending videos to people but couldn't figure out a way to efficiently delete them. I had to write up a Python script (https://gist.github.com/saagarjha/615dee3c04e226b44828e910a5...) to scrape the iMessage database for large attachments, and then print out the text to messages around it so that you could search for it and jump to around that place in the chat history and then delete the files from there. It's kind of insane how hard it is to manage old messages :/ I feel like Apple didn't really prepare for Messages coming in from iCloud meaning that every device had every single message you've ever sent, and now it's really hard to get through them without reading the database manually…
Pulled it up on my phone just to see what they considered "large", looks like 4MB is the cutoff (nothing smaller than that shows up for me). But nearly everything I'm seeing is a video or a photo taken with my real (not phone) camera. And a random PDF I apparently sent over iMessages.
However, deleting through this interface would be a pain. You select "Edit" and then each attachment is selected individually.
It seems to just show you the largest set (50? 100?). If you delete them, then you will eventually show smaller ones once you get through enough of them.
Really wish there was a "delete all" or "select all".
Thanks. That makes sense. But I didn’t feel like deleting anything. Just my wife and I on the iCloud storage and there’s plenty of it at present. My photo collection that I’ve been cleaning up is a bigger offender right now than messages. Damn cats doing cute things.
I am convinced iMessages right now is a tool to keep your storage full, push people to buy higher storage iPhone, and pay iCloud Storage backup to increase their Services Revenue.
None of the problem listed in the blog post or what you point out are new. They have been there since day one. And it is not that they neglect it either like iTunes or Aperture. They added Memoji!
Which is why I am surprised a the popularity of iMessages in US ( and France ). I think it is mostly due to SMS usage before iMessages was a thing.
When I am on the iMessage dialogue screen with a contact, I can usually click on their name, then Info, and it displays all the video/photo attachments sent in that chat. I believe I can then delete them from there. Is that useful? I guess if there's a number of photos you'd like to keep, but not the large videos, it's a more complicated issue.
We need a better communication standard. Apple made a killing selling their proprietary iMessage layer on top of SMS, but we could be actually making changes in this sector if we all used a standardized protocol. At least then we have a documented interface that we can choose to store however we want, or (perhaps wishful thinking) in a modular style. RCS is a great start, but I've already started to put my money on Matrix. While the latter won't ever see adoption by Google or Apple, it's a pretty convincing attempt at rich, decentralized communication.
Is it really "on top of SMS" though? The way I see, it's just a messaging app like WhatsApp, Telegram etc but they just combined it with the SMS app on their platform.
I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
There were a billion people you could send messages to, and receive messages from. That’s what got them of the ground.
A system where you can’t message half your friends because they bought their phone from a different manufacturer would have had more trouble getting traction.
iMessage is a vestigial product from the days when people had unlimited data but also only 400 text messages a month. It made sense then. It doesn't make much sense now.
It still makes sense over SMS/MMS which are terrible protocols in comparison. Even if you get unlimited messaging now, group and media messages are still remarkably unreliable.
iMessage is not a protocol, it's an app. Nobody uses SMS because everybody uses a messaging app. It comes down to whether you choose to use a cross-platform one or an iOS-exclusive one.
I think it is less that we have more SMS allotment and more that nobody wants to actually use them anymore, because they're insecure / don't work on wifi / are slow / etc.
> I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
Well, it's not useful now, but imagine in the 00s when a minority of users had iPhones. Having two clients, one for SMS and one for iMessages, would have gone against the simplicity Apple was aiming for with respect to user experience. Combining it meant that users could get a better experience (than SMS/MMS at least) when communicating with other iMessage peoples, but not have to swap out clients to chat with others.
The value to Apple was that if you tried to SMS someone, iPhone would check if the destination phone number was registered with Apple, and silently switch to iMessage via Apple instead.
Cynically, they intercepted millions of people's comms with hardly a murmer. Charitably they switched millions of people from country-dependent, pricey and limited SMS to internet messaging with read receipts, pictures, video, emoji, audio recordings without any hassle of logging in, changing apps, choosing clients, etc.
One underrated feature of iMessage is graceful degradation. If I'm unable to send/receive data for whatever reason, my message (and their replies) automatically fall back to SMS. There's been a few occasions where this was my only way of communicating important information, so I'm grateful for it.
Yes. If the message is unable to be delivered to the recipient via iMessage, then it is sent as SMS instead. So for example if you try to send a message to someone and their phone is turned off or doesn’t have signal, it will eventually fail and then send as SMS.
This is all controlled via a setting in iOS iMessage settings “Send as SMS”, which I believe is turned on by default but I’m not sure. So your mileage may vary.
Personally as a college student, all my friends use iPhones still. I have always wanted to switch to andriod, but one of the things that really holds me back is the fact iMessage still feels leagues above how SMS and texting works on andriod. Many subtle features of Imessage really make a difference with the overall texting experience on the iPhone over the long term.
Most of my friends do not have whatsapp, and we are constantly sharing photos and video. Regular SMS can barely send pictures let alone videos, and I don't want to go through other means to simply shoot my friend a quick video of something.
For me it's the fact that you can generally see the other person typing in an iMessage, while you can't in normal SMS.
That tells me whether the person is replying right now, so it's worth keeping my phone out, or busy and therefore won't be getting back to me for a while.
No. iMessage has the low entry barrier of SMS, but has features like a real chat app. SMS is not an iMessage replacement. Instead you have to get everyone you message with to change to something like WhatsApp.
This blue coloured bubble thing I've never heard anyone outside of Silicon Valley engineers/media mention, but I've heard it extrapolated as a deciding factor to the rest of the world by those same people many times.
I don't think most people actually give a shit about the bubble colour.
> iMessage still feels leagues above how SMS and texting works on andriod
iMessage is a messaging app that uses internet. It should be compared to WhatsApp, Telegram, Signal, WeChat, FB Messenger.
It doesn't make sense to compare iMessage to SMS. Because nobody uses SMS as their main communication method. The only people who ever use SMS are iPhone users when they text Android users.
All this confusion comes from the fact that Apple merged their messaging app with the SMS app. This creates the illusion that iMessaging isn't just a messaging app but it is actually something built on top of SMS. It's just a messaging app that is not cross-platform.
> I don't want to go through other means to simply shoot my friend a quick video of something.
By "other means", do you mean installing a new messaging app? I'm guessing this isn't about the effort of downloading an app but rather it's about getting used to a new app and convincing your friends to switch to it.
“convincing your friends to switch to it” is the real issue. Or remembering which platform various friends use and keeping that straight. And when you have multiple friends who prefer different platforms, figuring out the intersection of messaging platforms that they all use. Or when you have a friend on Facebook messenger but you don’t know their phone number, or a friend who can’t have WeChat installed on their corporate phone, or any number of other situations.
I think that in America most people still use SMS, but in Europe barely anyone I know does, so RCS is basically a non-starter. It's all WhatsApp and Telegram (and some Signal). Like my kids sports teams coordinate everything through WhatsApp so I'm basically forced to use it. Would love to see an alternative, but getting everyone to switch would require a lot.
It's a question of who comes first- Linux was first to market as a cloud OS, so it dominated. Email came first for long-form asynchronous messages, so it dominates.
Searching Messages on my Mac is a joke. If type the name of a friend in the search field, it returns (after unreasonable delay) various results, mostly of texts about the person I'm searching for.
How does it not realize that it should first search recipient names and present these results first? A search for "Delilah" should not show a bunch of texts about Delilah, obscuring the one thread with my friend Delilah.
I think there’s a separate idiomatic autocomplete flow for what you’re trying to do. Try hitting “New Message”; typing your friend’s name; and selecting the result. The New Message window will transform into the existing-conversation window for that contact.
Not saying having it also work from search wouldn’t be good — but having this flow available is probably why the other issue hasn’t been fixed sooner.
> ry hitting “New Message”; typing your friend’s name; and selecting the result. The New Message window will transform into the existing-conversation window for that contact.
I think this is how MS teams chat works too. It kind of makes sense, because it has a workflow for solving the "do you want your existing chat with Alice, or are you trying to start a new group chat with Alice & Bob" .
It does make sense if you're looking to send a new message. In my case, I'm typically doing a search so I can find information/photos from that thread. In this context, it's not so intuitive, and it never occurred to me to even try.
So, I just tried this in Messages on Big Sur. If I search for a name, I get a list of existing-conversations that the person with that name is in at the top of the results, then the list of text results within conversations.
Messages on Big Sur is (basically) the iOS Messages app ported over with Catalyst, so it has a lot of improvements compared to the highly-stale app it replaced.
I'm not sure if it even works anymore or if there's a better option, but I've used PhoneView <https://www.ecamm.com/mac/phoneview/> in the past to make a PDF archive of my iMessages.
The cynic in me says this is all part of the plan. Sure, we'll give you seemingly impossibly large amounts of cloud storage for free, learn to never delete anything that you ever talk about in your entire life.
Ten years down the road, "oh, looks like you need to start paying us monthly for more cloud storage!"
I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive... And how many will in another ten years.
> I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive...
Gmail’s storage space has consistently grown over the years to keep up with the average user’s demand.
Google don’t want to do anything — like charging for email storage — that would disincentivize their mass of sensors^H users from feeding them ever more data.
(If you’re an outlier with way more storage than usual, they’ll happily charge you, knowing that the data point you represent is mostly going to get pruned out of their models anyway.)
This was previously true. Free storage at Google hasn't increased from 15 GB since 2013. (I have a few promotional bonuses that bring me up to about 19 GB, but it's been that way for... over half a decade without increase.) Personally, my Google storage is mostly full, even though I haven't used Google Drive or Gmail since 2016, and have removed much of my personal content since then.
Also, now they're actually effectively decreasing everyone's storage. Previously, Google Drive document types and Google Photos didn't count towards that 15 GB limit. They will as of June 1st, 2021: https://blog.google/products/photos/storage-policy-update/
So, while cloud storage usage has exploded over the past eight years, Google's storage limits have not only not expanded to meet it, they've actually begun to tighten the belt.
They don’t need that, they can just start saving pics in HEIC at 4MB per pic, and backups start exploding.
I use Gimp and generally obtain the same quality with a 450KB JPEG, sometimes 50KB depending on the contents. Images can be remarkably small. They should try increasing the compression ratio until a change is noticeable, but do they care?
I've used a mess of find and rm to clear up space on my local drive, but afaik they'll still take up space in iCloud storage -- I agree that there should be a way to more efficiently discard them, and that automatically deleting old ones is not the right thing to do
If you have Messages set up on a Mac you can access your chat archives under ~/Library/Messages/Archive. Each conversation is stored by date in an .ichat file which can be re-opened in the Messages app.
1) I do clean installs when I update macOS, so the Library folder is created new each time. Which means that archived messages only go back to that point. (I browsed via Terminal and used the command line 'open' command to view them.)
I do retain old ~/Library folders and it's possible to use the same technique in finding messages there.
2) But, and here's the big limitation: at a quick glance, they're simply individual messages, by date. There's no way to follow entire threads. I suppose it would be possible to write your own program to organize these. But the point of the article is that Apple should make it easy. Your method isn't easy.
Edit: response to sibling comment. I'm still on Catalina and the above works. I have no idea about Big Sur.
On a similar note, deleting old photos to save space on iCloud is very hard. I could mass-delete photos older than a certain date in the Mac Photos app (or whatever its called these days) using a smart album, but then that mass deletion would not sync to iCloud. Or, I could log into iCloud via a web page and select a load of photos (but not more than 1000 at a time) and delete them - but then it takes ages - like about 20 minutes. Why does it take so long to delete 1000 photos from the cloud? Its unfathomable. And then after deleting them you have to go and find them in the iCloud bin and delete them again - which also takes ages.
I can only conclude that they have made it deliberately very slow and painful to clear space on iCloud. Because they'd rather you paid for more space ...
> I could mass-delete photos older than a certain date in the Mac Photos app (or whatever its called these days) using a smart album, but then that mass deletion would not sync to iCloud.
Why wouldn’t it sync? If I did this right now, my expectation is that it would sync. Is there a limitation in how this works?
Not sure if that's available outside the EU as well, but over here you can export an archive with all data they have on you including messages and attached media. I'm sure they leave out things but I use it to backup my chats.
FB Messenger does have data takeout, it comes in some kind of JSON format. Be prepared for some surprisingly massive file downloads though -- my chat history alone was over 50gb!
Flexibits developed an app to do this search called Chatology (https://flexibits.com/chatology) but they've discontinued it since Big Sur came out. Works pretty well on my Mojave machine.
It unfortunately doesn't work well if you've enabled Messages in iCloud [1]. You can only search the messages that're in your device's local cache. Which isn't actively cleared much, that I've seen, but it does mean that it's not very helpful for any messages that predate the current install of your OS.
I'd assume this is why they discontinued it, rather than adapt it to the new from-iOS Messages in Big Sur.
You'd be surprised how few engineers work on apple projects relative to their size, and how apple underpays compared to other big tech companies like FB or Google unless your working on the current hot project.
I want a chat app that is: E2EE and allows me to change phone numbers.
I started migrating people to Signal when that whole WhatsApp ToS hullaballoo happened, but now I need to change my phone number and apparently this is not supported in Signal, which is just crazy to me.
Can confirm. Shameless plug: I wrote a script to extract the pertinent information from the SQLite database into an HTML page a while back: https://github.com/kccqzy/imessage-db-extract/blob/master/im... It's open source abandonware though. Lost interest in keeping up with the schema updates because I no longer use iMessage.
That's an undocumented implementation detail. It might change without warning at any time. Are you seriously recommending this as a solution for users who want to be able to read past messages?
I’d imagine it’d be difficult for Apple to migrate to anything substantially different as it’d require rewriting a potentially multi GB file, the space for which many users would not have. It’s effectively a massive, distributed, database migration.
Apple's chat.db has an esoteric schema owing to the fact they never designed it from the ground up and instead kept adding new columns and tables with each macOS release. This makes their queries super complicated with multiple joins.
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.
Yes, with a catch: it requires an always on Mac (which can be in the cloud) or a jailbroken iOS device. We plan to make it work without the catch.
I've worked with 10+ messaging platforms so far and Apple's protocol is the most obfuscated and complicated to reverse engineer. Apple has invested millions of dollars to make the iMessage protocol super hard to reverse engineer. It's how they sell tons of iPhones after all:
> However, Craig Federighi, Apple’s Senior Vice President of Software Engineering and the executive in charge of iOS, feared that “iMessage on Android would simply serve to remove [an] obstacle to iPhone families giving their kids Android phones”. (https://www.androidpolice.com/2021/04/28/apple-admits-that-i...)
Likely because each iMessage receiving device has a private key to decrypt the messages, and senders encrypt the iMessage against all the keys in your key bag.
To access iCloud and or services you also need to have a device security key that is tied to genuine hardware.
Looks like a super cool project. I'm always happy to see new client-side software for communication protocols written that aims to improve upon built-in! And iMessages in particular could certainly use it. With the demise of iTunes, it now feels like iMessages is probably one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
This bit did make me wonder though:
>We plan to make it work without the catch.
Out of curiosity, do really think that's realistic, or even desirable long term? iMessage is ultimately an Apple service that runs heavily on Apple's infrastructure, and is directly subsidized by sales of their highly vertically integrated hardware platforms. If it turns into a cat-and-mouse fight it seems like they're always going to have the eternal upper hand, which in turn seems like it'd make for a subpar user experience (ie., breaks randomly which for an instant messaging service would be pretty bad). Also seems like they might actually be motivated to respond rather than ignore it since it'd actually be directly leeching their infra if it will work on PCs/Android without a Mac/iDevice purchase in the equation (unlike hackintoshes for example, where whatever debate there is to be had about probably very low "lost sales" it doesn't actually cost them anything).
Obviously you've probably thought this all through, but seems like just requiring an old cheap Mac or old jb'd idevice and thus avoiding Apple might be an easier path. Or alternately just stick to offering a flat out better client with the iMessage bit being Mac-only. Will be interested to see how it goes though!
> one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
The Messages app on Mac was rewritten for Big Sur (last year). It gained some features but also some bugs in the process.
A little feedback on the landing page: Almost turned away because I thought „Sign up with Google“ was the only choice and I missed the little „Prefer entering email instead?“ below. I just prefer not having a Google account ;)
Maybe the styling of that link could be adjusted so that it isn’t glanced over that easily.
Source: I built a third-party desktop client for iMessage at https://texts.com
Does the texts.com client solve the original problem? Does it make it easy to go back and view all those old text messages? I know the website says "from forever ago easily". But you didn't say that explicitly in your post.
I assume you're being modest and not "pushing" your solution. But if it simply solves the problem, maybe you should be shouting this out loud!
Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
> Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
We recommend using a simple MITM proxy or a firewall app like Little Snitch / LuLu to inspect the network traffic.
> You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
Yes – there's a peer-to-peer connection between the Windows and Mac devices.
Just took a look at your product, I'm very impressed and have added myself to the waitlist. I'm almost even more impressed by the domain name itself though - how on earth did you manage to bag that domain?!
I had done something too, but I also needed a Manifest.mbdb parser to reconstruct backup to disk, then process the messages and copy all the images around to make it work. I did it mostly to make a simple-to-read archive of chats before deleting them.
(You need an unencrypted backup directory to run it over)
Note: It looks like the flat directory structure of the backup is now 256 dirs prefixed by first two characters of file name. I'm not sure, but a small change to the name "sha_file" may be needed to just path.join that with the first two chars of the sha.
Python 2 but can be ported forward without much effort. Manifest.mbdb is now Manifest.plist (there's a sqlite thing now too). timestamp sql might not work so great, would need to check query.
I don't mean to be pessimistic , but I wonder how you're going to be anything but reactionary to apples changes to the database. I've written a similar (much simpler) version of what you're proposing to sell- the database queries are hysterical.
I've been thinking about using the accessibility I framework to ETL the messages into Kinesis to facilitate event driven architecture. 'archived' Messages and attachments are stored as bsob.
Hell, I've even experimented with some goofy stuff, like FaceTime to VOIP- asterisk is just so flexible
A piece of feedback: it would be nice to know what the price is on your site. You say "We make money by charging you a monthly subscription," but there's no pricing.
The fact so many people on this site can throw something functional together without much effort is even more disappointing apple doesn't do anything about it.
I've built an iMessage chat.db reader as well. To me the most astounding thing is that chat.db is unencrypted sqlite. Anyone who can see your home folder can read all your private chats or upload the chat.db to be read somewhere else.
Why would a lack of encryption make it not reverse engineering? There's a database schema, it was engineered* and used to build a database. They went from the live data back to the schema.
*For some definitions of "engineered", apparently.
Which you end up applying to random things like ruby because Emacs.app uses a ruby-based launcher and anything less than Full Disk Access breaks dired.
iPhone is the only apple device I own but it crashes it can't go too far back into my messaging history for any one individual too far.
Usually once I scroll to ~5 months back with people I text regularly the iMessage app just straight crashes ... it's a little absurd. To top it off, every couple pages it gives a loading icon and introduces severe latency for whatever reason. There don't seem to be any apps in their App Store that let you search and lookup old messages either.
This is actively biting me right now. Are there any solutions for accessing the iMessage/chat.db database on my phone or via iCloud from a Linux system (Ubuntu in my case)?
This is basically one taken out of microsoft's .doc format playbook.
At first I thought Apple was slow and just didn't have time to get to some sort of export for iMessage chats. Mail had imap, calendar had caldav, address book had carddav. The original mac stuff did other standards like AIM and jabber.
But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
What's annoying is that I couldn't save my messages, I could only upgrade to a bigger iphone.
What's downright evil is all the people - I know several - who have lost messages + embedded photos/videos from loved ones who have died because there was no export.
You said it! Apple has been unethical from the get go and yet somehow the cult doesn’t care. I’m unhappy apple user but still on 6S and will be switching over to Android phones until pine phone (Linux OS) becomes better.
>But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
In an SQLiteDB any programmer can learn to navigate, and any third party can build a tool to export them?
Sorry but the "internal emails" prove nothing about Messages. It's just a quick and dirty implementation that "works good enough" for reading the latest, but not as an archive.
Yeah I agree. The lack of a good schema/export function sounds to me like negligence not maliciousness. It’s a different issue from only allowing Apple-tech-users to use iMessage
> But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
Any source for that? I couldn't find any direct evidence for your claim.
Thanks, that's interesting, I didn't know that. On the one hand, I shouldn't be surprised since vendor lock-in is the standard way tech companies try to gain corporate advantage. On the other hand, it's kind of depressing to learn once again that customers count very little and no matter what these companies say, they will always try to abuse us in every way the law allows.
Looking over your site, I'm curious about a few things:
- The FAQ says you'll eventually open source your SDK. I admire that as a software engineer, but from a business perspective, isn't this risky? It seems like--if you don't pass messages through an intermediary service--your value-add is mostly in this SDK + your design. All of these elements are copy-able.
- The jobs page includes a backend engineer, so what will go through your backend, if not any message data?
- Are there legal concerns about reverse-engineering these APIs and using them? Some of your supported chat services have open APIs, but others seem intentionally closed. (And a couple parent companies seem downright hostile to 3rd party use.)
my biggest problem with iOS storage is that if you pay to store a photo in iCloud and then use Messages to send it to 5 friends, you are now paying to store it in iCloud 6x. And of course it’s totally unfair that there is no priced option between 200GB and 1TB. Same with Google Photos.
Kinda dying to know if they dedup on the server? Cause technically speaking if everyone was on Apple then that 1 photo becomes 12 not to mention it getting send to another layer of people.
The comparison to Photos in TFA is apt. Apple could copy the most relevant subset of features, in particular the Years/Months/Days/All navigation views, and it would dramatically improve the utility of iMessage for old messages.
Improving the abysmal search result quality is probably more important, though.
Many comments here are suggesting you can look at the DB on a Mac. What about people who have Windows computers? Is there a way to extract that information?
I also have a similar gripe with the browser. How can I sync my Safari tabs that are open on my iPhone with my PC?
275 comments
[ 2.6 ms ] story [ 310 ms ] threadThis aspect of the iMessage user experience has remained untouched for a decade, despite everyone's chats growing longer. Searching for particular iMessages sucks because the results miss the broader context of the conversation (a single message is not useful when you want to reread a conversation).
(Also just noting, as the post author I created a title better-suited to HN's 80 character limit.)
If you do, tell us about it here on HN.
If it just pinned the message thread to the top and didn’t change the icon to a giant circle that sometimes doesn’t show notifications, it’d be fine. I pinned my GF’s message thread and I ended up missing lots of messages and notifications from her and ended up unpinning her so the notifications would show up.
Not to mention the comically huge circle icon that is centered when everything else is left aligned.
It only made things worse :(
It defeats the entire purpose of pinning.
I had a problem that appeared to be a corner case with Apple Pay. I wrote a paper letter to Tim Cook about it, and within a week got contacted by a staff person in his office. He worked exclusively with me to identify issues I had not considered, and followed up on all of the direct effects and side effects until they were resolved.
[0] - https://www.inc.com/business-insider/tim-cook-wakes-up-at-4-...
http://howmanyofme.com/search/
Also, be sure to file a bug report / enhancement at https://www.apple.com/feedback/iphone.html
- I’m not wasting my time emailing a top level company executive to inform of decades old most basic of bugs because if they don’t know of it then that’s even a bigger problem.
The scroll-load-scroll-load cycle does suck, though.
When I went to open the email on my new machine, the loading indicator just spun and spun until finally my browser crashed. When viewing an email conversation, Gmail will display previews of all the emails in the chain, including their attachments, which was causing my browser to crash. Next I tried connecting my Google account to Apple Mail, but Gmail's SMTP server would time out before the attachments were able to download.
After a while I forgot about the photos all together, until I randomly stumbled across them while searching my Gmail account for some other files I had sent myself around that time. At this point it was 2014 or so, and my browser was very easily able to open the messages so I could download the photos, which of course at that point I had zero interest in.
With photos, I guess this could end up being around 1GB. Maybe this will push you off the free plan to the $1/mo plan, but I don't see it being that big of a needle-mover for services revenue.
That's a nice bonus, no matter how much the needle moves.
https://appleinsider.com/articles/21/01/27/there-are-1b-ipho...
I definitely prefer my current phone that lets me just rsync ~/Pictures.
You can set it to sync whenever your phone is inside a particular location boundary (e.g. at home, where there is wifi), so it works transparently with zero ports open to the internet, and no firewall-bypassing cloud services.
However, deleting through this interface would be a pain. You select "Edit" and then each attachment is selected individually.
Really wish there was a "delete all" or "select all".
None of the problem listed in the blog post or what you point out are new. They have been there since day one. And it is not that they neglect it either like iTunes or Aperture. They added Memoji!
Which is why I am surprised a the popularity of iMessages in US ( and France ). I think it is mostly due to SMS usage before iMessages was a thing.
I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
A system where you can’t message half your friends because they bought their phone from a different manufacturer would have had more trouble getting traction.
Well, it's not useful now, but imagine in the 00s when a minority of users had iPhones. Having two clients, one for SMS and one for iMessages, would have gone against the simplicity Apple was aiming for with respect to user experience. Combining it meant that users could get a better experience (than SMS/MMS at least) when communicating with other iMessage peoples, but not have to swap out clients to chat with others.
Cynically, they intercepted millions of people's comms with hardly a murmer. Charitably they switched millions of people from country-dependent, pricey and limited SMS to internet messaging with read receipts, pictures, video, emoji, audio recordings without any hassle of logging in, changing apps, choosing clients, etc.
This is all controlled via a setting in iOS iMessage settings “Send as SMS”, which I believe is turned on by default but I’m not sure. So your mileage may vary.
Most of my friends do not have whatsapp, and we are constantly sharing photos and video. Regular SMS can barely send pictures let alone videos, and I don't want to go through other means to simply shoot my friend a quick video of something.
That tells me whether the person is replying right now, so it's worth keeping my phone out, or busy and therefore won't be getting back to me for a while.
The time savings add up.
I don't think most people actually give a shit about the bubble colour.
iMessage is a messaging app that uses internet. It should be compared to WhatsApp, Telegram, Signal, WeChat, FB Messenger.
It doesn't make sense to compare iMessage to SMS. Because nobody uses SMS as their main communication method. The only people who ever use SMS are iPhone users when they text Android users.
All this confusion comes from the fact that Apple merged their messaging app with the SMS app. This creates the illusion that iMessaging isn't just a messaging app but it is actually something built on top of SMS. It's just a messaging app that is not cross-platform.
> I don't want to go through other means to simply shoot my friend a quick video of something.
By "other means", do you mean installing a new messaging app? I'm guessing this isn't about the effort of downloading an app but rather it's about getting used to a new app and convincing your friends to switch to it.
How does it not realize that it should first search recipient names and present these results first? A search for "Delilah" should not show a bunch of texts about Delilah, obscuring the one thread with my friend Delilah.
Not saying having it also work from search wouldn’t be good — but having this flow available is probably why the other issue hasn’t been fixed sooner.
I think this is how MS teams chat works too. It kind of makes sense, because it has a workflow for solving the "do you want your existing chat with Alice, or are you trying to start a new group chat with Alice & Bob" .
Messages on Big Sur is (basically) the iOS Messages app ported over with Catalyst, so it has a lot of improvements compared to the highly-stale app it replaced.
It may be a problem with PhoneView, but I'm not convinced that it is. Also, due to the size of the backup, I do have to do it to an external drive.
Ten years down the road, "oh, looks like you need to start paying us monthly for more cloud storage!"
I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive... And how many will in another ten years.
Gmail’s storage space has consistently grown over the years to keep up with the average user’s demand.
Google don’t want to do anything — like charging for email storage — that would disincentivize their mass of sensors^H users from feeding them ever more data.
(If you’re an outlier with way more storage than usual, they’ll happily charge you, knowing that the data point you represent is mostly going to get pruned out of their models anyway.)
Also, now they're actually effectively decreasing everyone's storage. Previously, Google Drive document types and Google Photos didn't count towards that 15 GB limit. They will as of June 1st, 2021: https://blog.google/products/photos/storage-policy-update/
So, while cloud storage usage has exploded over the past eight years, Google's storage limits have not only not expanded to meet it, they've actually begun to tighten the belt.
I use Gimp and generally obtain the same quality with a 450KB JPEG, sometimes 50KB depending on the contents. Images can be remarkably small. They should try increasing the compression ratio until a change is noticeable, but do they care?
That fine for final/production images but it's a terrible idea for your originals, or really anything you might want to edit beyond cropping).
https://appletoolbox.com/qa-where-are-imessage-files-stored/
It does have some important limitations.
1) I do clean installs when I update macOS, so the Library folder is created new each time. Which means that archived messages only go back to that point. (I browsed via Terminal and used the command line 'open' command to view them.)
I do retain old ~/Library folders and it's possible to use the same technique in finding messages there.
2) But, and here's the big limitation: at a quick glance, they're simply individual messages, by date. There's no way to follow entire threads. I suppose it would be possible to write your own program to organize these. But the point of the article is that Apple should make it easy. Your method isn't easy.
Edit: response to sibling comment. I'm still on Catalina and the above works. I have no idea about Big Sur.
I can only conclude that they have made it deliberately very slow and painful to clear space on iCloud. Because they'd rather you paid for more space ...
Why wouldn’t it sync? If I did this right now, my expectation is that it would sync. Is there a limitation in how this works?
Take a photo on you iPhone and it take days for i to show up in the photos app, it instantly shows up in iCloud accessed through the browser.
Same thing in various other directions, delete on web, and it takes days to disappear from other places. Makes zero sense.
Often if i just took some pictures i have to send them via airdrop instead of just opening Photos. Come on Apple!
https://www.facebook.com/help/212802592074644
https://www.facebook.com/help/930396167085762/
I'd assume this is why they discontinued it, rather than adapt it to the new from-iOS Messages in Big Sur.
[1] https://support.apple.com/en-us/HT208532
I started migrating people to Signal when that whole WhatsApp ToS hullaballoo happened, but now I need to change my phone number and apparently this is not supported in Signal, which is just crazy to me.
https://spin.atomicobject.com/2020/05/22/search-imessage-sql...
Why would you not recommend it when there's no alternative?
Lighten up a little, GP.
And, yeah, it’s a total mess.
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.
I've worked with 10+ messaging platforms so far and Apple's protocol is the most obfuscated and complicated to reverse engineer. Apple has invested millions of dollars to make the iMessage protocol super hard to reverse engineer. It's how they sell tons of iPhones after all:
> However, Craig Federighi, Apple’s Senior Vice President of Software Engineering and the executive in charge of iOS, feared that “iMessage on Android would simply serve to remove [an] obstacle to iPhone families giving their kids Android phones”. (https://www.androidpolice.com/2021/04/28/apple-admits-that-i...)
That’s surprising. How come?
To access iCloud and or services you also need to have a device security key that is tied to genuine hardware.
This bit did make me wonder though:
>We plan to make it work without the catch.
Out of curiosity, do really think that's realistic, or even desirable long term? iMessage is ultimately an Apple service that runs heavily on Apple's infrastructure, and is directly subsidized by sales of their highly vertically integrated hardware platforms. If it turns into a cat-and-mouse fight it seems like they're always going to have the eternal upper hand, which in turn seems like it'd make for a subpar user experience (ie., breaks randomly which for an instant messaging service would be pretty bad). Also seems like they might actually be motivated to respond rather than ignore it since it'd actually be directly leeching their infra if it will work on PCs/Android without a Mac/iDevice purchase in the equation (unlike hackintoshes for example, where whatever debate there is to be had about probably very low "lost sales" it doesn't actually cost them anything).
Obviously you've probably thought this all through, but seems like just requiring an old cheap Mac or old jb'd idevice and thus avoiding Apple might be an easier path. Or alternately just stick to offering a flat out better client with the iMessage bit being Mac-only. Will be interested to see how it goes though!
The Messages app on Mac was rewritten for Big Sur (last year). It gained some features but also some bugs in the process.
Interesting, how do you plan to do that?
A little feedback on the landing page: Almost turned away because I thought „Sign up with Google“ was the only choice and I missed the little „Prefer entering email instead?“ below. I just prefer not having a Google account ;)
Maybe the styling of that link could be adjusted so that it isn’t glanced over that easily.
Does the texts.com client solve the original problem? Does it make it easy to go back and view all those old text messages? I know the website says "from forever ago easily". But you didn't say that explicitly in your post.
I assume you're being modest and not "pushing" your solution. But if it simply solves the problem, maybe you should be shouting this out loud!
Also, a "jump to date" feature (like Telegram) is planned (and super easy to build.)
You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
We recommend using a simple MITM proxy or a firewall app like Little Snitch / LuLu to inspect the network traffic.
> You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
Yes – there's a peer-to-peer connection between the Windows and Mac devices.
https://github.com/CamHenlin/iMessageWebClient
I also have some other iMessage related git repos on my GitHub. Interesting stuff!
EDIT: No, apparently not. Weird; we all thought that was going to make him a multi-millionaire.
That said, this is it: https://gist.github.com/brianv0/35f36a32366a2c34be8d
(You need an unencrypted backup directory to run it over)
Note: It looks like the flat directory structure of the backup is now 256 dirs prefixed by first two characters of file name. I'm not sure, but a small change to the name "sha_file" may be needed to just path.join that with the first two chars of the sha.
Python 2 but can be ported forward without much effort. Manifest.mbdb is now Manifest.plist (there's a sqlite thing now too). timestamp sql might not work so great, would need to check query.
I've been thinking about using the accessibility I framework to ETL the messages into Kinesis to facilitate event driven architecture. 'archived' Messages and attachments are stored as bsob.
Hell, I've even experimented with some goofy stuff, like FaceTime to VOIP- asterisk is just so flexible
https://github.com/michael-danello/iMessageWrapped
EDIT: It is absurd Apple doesn't provide better message organization and UX/UI.
It's impressively insecure.
*For some definitions of "engineered", apparently.
Usually once I scroll to ~5 months back with people I text regularly the iMessage app just straight crashes ... it's a little absurd. To top it off, every couple pages it gives a loading icon and introduces severe latency for whatever reason. There don't seem to be any apps in their App Store that let you search and lookup old messages either.
This is actively biting me right now. Are there any solutions for accessing the iMessage/chat.db database on my phone or via iCloud from a Linux system (Ubuntu in my case)?
This is basically one taken out of microsoft's .doc format playbook.
At first I thought Apple was slow and just didn't have time to get to some sort of export for iMessage chats. Mail had imap, calendar had caldav, address book had carddav. The original mac stuff did other standards like AIM and jabber.
But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
What's annoying is that I couldn't save my messages, I could only upgrade to a bigger iphone.
What's downright evil is all the people - I know several - who have lost messages + embedded photos/videos from loved ones who have died because there was no export.
In an SQLiteDB any programmer can learn to navigate, and any third party can build a tool to export them?
Sorry but the "internal emails" prove nothing about Messages. It's just a quick and dirty implementation that "works good enough" for reading the latest, but not as an archive.
Any source for that? I couldn't find any direct evidence for your claim.
https://texasnewsexpress.com/index.php/04162021-1249/apple-r...
- The FAQ says you'll eventually open source your SDK. I admire that as a software engineer, but from a business perspective, isn't this risky? It seems like--if you don't pass messages through an intermediary service--your value-add is mostly in this SDK + your design. All of these elements are copy-able. - The jobs page includes a backend engineer, so what will go through your backend, if not any message data?
- Are there legal concerns about reverse-engineering these APIs and using them? Some of your supported chat services have open APIs, but others seem intentionally closed. (And a couple parent companies seem downright hostile to 3rd party use.)
I also have a similar gripe with the browser. How can I sync my Safari tabs that are open on my iPhone with my PC?
I would recommend switching iOS browser to be able to sync your browsing across vendors
i encourage othes to consider doing the same. its been great for me.
just in case.
i hope we all learn what good digital hygiene is.