Ask HN: Microsoft crawls private links – how can this be legal?
I've built a tiny private website to share wedding photos. In order to protect the privacy of my guests, I've implemented authentication via supabase's magic link – guests need to enter their mail address and get an access link with a short-lived token sent to their mailbox. Unfortunately, this didn't work for users of mailboxes hosted with Microsoft, because Microsoft clicks all links in emails for "security" purposes. I've run some experiments and they seem to download all assets from the website linked to, including images, and try to crawl all other pages that can be reached through the first link in the mail, rendering not only the magiclink useless, but also downloading massive amounts of password protected images - a clear violation of GDPR. Why hasn't this been stopped?
15 comments
[ 1.7 ms ] story [ 40.1 ms ] threadSOP. many email providers do this. the magic link shouldn't directly take you to the thing, but rather be an interstitial, so that security vetting prefetch can be managed.
Further crawling seems quite bad. Are the pages actually public, protected only by URL obscurity? Not justifying MSFT behavior here, but you say "password protected" but the scheme you've described doesn't seem to be that. Can you set a session cookie after the magic link, and assuming MSFT crawler doesn't save cookies, "defeat" it that way? Or, identify it by UA and defeat it that way?
I think you need to provide more info about how they are crawling if you want a solution.
I don't see how this is a GDPR violation. Crawling the data and evaluting security or malware issues doesn't fall under GDPR. Saving the content would, and probably is additionally a violation on your part, but you haven't indicated they are doing that.
Also not clear why the magiclink doesn't work. Just because MSFT used it to crawl, why can't the user also use it later?
> SOP. many email providers do this
Heh, are they inadvertently unsubscribing folks from newsletters by making requests to the "Unsubscribe" links too?
Most Unsubscribe pages I've seen recently appear to be a two or more step process but I'm sure there's some folks out there making unsubscribe pages that do the thing on the initial GET request alone... (not justifying making a GET perform a side effect, just saying it's interesting.)
How I've seen developers approach this, is the initial GET request loads a form, and requires that the user clicks something once more, to then submit a POST request and confirm the unsubscribe action.
[0] https://developer.mozilla.org/en-US/docs/Glossary/Safe/HTTP
You're right, this is interesting, something I have wondered about before.
The magic link is taking you to a website that requires you to click to activate the link's token. MSFT is not only crawling all links on the page, but also clicking all elements with a click handler. It's necessary to add a recaptcha, but the problem stays the same: taking a secret that was not addressed to you doesn't allow you to access, process, and store the PII that you can find with that secret.
> I don't see how this is a GDPR violation. Crawling the data and evaluting security or malware issues doesn't fall under GDPR. Saving the content would...
‘processing’ means any operation (...) whether or not by automated means, such as (...) retrieval, storage, (...), use, (...) or destruction [https://gdpr-info.eu/art-4-gdpr/]
Microsoft retrieves the data, stores it (temporarily), uses it and deletes it (from memory). So all of these actions are subject to GDPR. The data cannot be deemed "publicly available", according to https://iapp.org/news/a/publicly-available-data-under-gdpr-m...
> Also not clear why the magiclink doesn't work. Just because MSFT used it to crawl, why can't the user also use it later?
The magic link contains a ~40 byte long token that is valid for a couple of minutes, just like an OTP.
> massive amounts of password protected images
Not password protected if the password is part of the URL.
> a clear violation of GDPR
Unclear to me which PII is being stored or used.
It's not self-implemented, you can check it out here: https://github.com/supabase/gotrue
> Not password protected if the password is part of the URL.
It's a token that's valid for a couple of minutes – just like a password reset token. Indeed, in the given implementation, it's the very same as the password reset token. If you consider this implementation as "not password protected", any website with a password reset functionality is "not password protected".
https://www.microsoft.com/en-us/research/publication/metadat...
edit: In which case its part of the TOS and has government support. I would look for the press release statement for the change of terms of services but i dont want to search for "Child Sexual Abuse Material". Its apparently called photodna at microsoft.
If i recall apple changed theirs a few months ago.
edit: If you think i am being overly paranoid, last time i looked for the topic google put some pedophile self help sites into my results.
I'm pretty sure that a robots.txt is only a suggestion, not legally binding in any way.
If enough people put honeypot links in their emails .....