Whoever thought it was acceptable to have a string of text that renders something unreadable or not immediately obvious to the human eye, was a complete moron.
I realize ASCII was limited, but one thing I like about it is I can understand every single character code, and program to handle all the edge cases with certainty.
The tags were needed for language indication to control CJK glyph variants. Flag emoji were grafted onto this scheme. The key is that tag sequences have to start with a valid introductory codepoint. Simple enough to strip out anything that isn't a flag.
> Simple enough to strip out anything that isn't a flag.
But it's not simple enough for every company/app to independently research the entire Unicode code point space (which is MASSIVE) to find out what kinds of "fix ups" that app needs to do to clean the data it consumes.
It's complicated or at least has difficult tradeoffs. Maybe you invest a lot of time carefully surveying all of the Unicode planes and decide which ones you care to keep unchanged and which ones you filter/strip. For every code point you reject or change, there is going to be some user who is confused or dissatisfied with the limitations of your app.
Maybe you intended to reply to my comment's parent? I don't necessarily disagree.
But also, according to the Ars article comment describing the Unicode character range, it has been deprecated, so maybe someone involved in the Unicode standard saw problems with it.
Sure, but there's a big difference between a handful (or a few dozen), compared to thousands of obscure code points. One I can learn, the other I need to dedicate an unreasonably large chunk of my life to fully understand.
> I realize ASCII was limited, but one thing I like about it is I can understand every single character code
It's great for teaching and other things, but everyday life is filled with many characters, is the suggestion we'd have one ASCII per language where there is more distinct characters, or what would we do? I don't see what else we could have done, that would have worked for the world, but I'm curious to hear ideas.
My suspicion is that the spam filter programmers didn't do a comprehensive evaluation of every code point on every plane of Unicode because... well that's a massive job. So your "sanitize and denormalize" tasks are actually massive mappings which were likely imperfectly created.
I'm basing my comment on the article. Since they are susceptible to ASCII smuggling, they can't have been doing unicode normalization.
The task isn't that massive. Python's unicodedata (for example) contains all the info already. Including the normalization (not denormalization as I wrote before) function.
MS and the other big buggers are pretty incapable of running an email system that you would really want to use. They also have to support those that you might consider spammers too.
However, I'm sure that you are also familiar with the HN standard issue meme that it is impossible to run your own email system.
So, your email provider is crap and you can't do it yourself!
Bollocks! I use Exim (1) and rspamd (2) and run quite a few bijou email systems. It does require some effort.
I still run my own DNS and HTTP/HTTPS server. It's basically free of effort compared to what setting up and maintaining SMTP was (I used to do that on my colo because of the aforementioned, now I just point my MX records and escape the hassle). Hell, I even run my own ASN+BGP+IP space & it's easier than email was. Anything is doable though if you have a high enough tolerance of course.
This is what I always tell people. If something is obviously sketchy, don't bother trying to make it safe. Just discard it. No reason to play the attacker's game. (Obviously you need to know what inputs are sketchy vs just uncommon, like people with a single quote in their last name)
I estimate that I can identify 99.5% of spam by the subject line alone. I don't know why its such a hard problem, and why invoices for Norton 360 Premium sent from a random gmail address keep passing as legitimate email.
Hell, just block all emails mentioning Norton 360. Wouldn't be any great loss to the world.
But what I don't understand is MS letting these through the spam filter but randomly, in a thread between myself and another person both using MS account emails, sending a single email in that thread to spam.
One thing to be careful of is email hosts that dump your account after x months of inactivity. Logging in once in a while is good, got burned by this once.
You should check out spamgourmet; they came up with roughly the same thing over 25 years ago, and implemented it, and handed it out to all for free. ;)
Getting bombarded by "Paypal" <somethingsomething@something.onmicrosoft.com> spam for months and Microsoft (outlook) can't handle that. This is beyond their level.
Yeah; imagine getting not junk...actual PayPal phishing emails, daily, multiple, from onmicrosoft.com (owned by Microsoft), in Outlook (owned by Microsoft), and report phishing does nothing.
> The root domain onmicrosoft.com is owned and managed by Microsoft Corporation, which uses it as the default domain for Microsoft 365 and Azure cloud environments
imagine that these people have most money in the world. Bill Gates (owns MS). Elon Musk (owns / owned PayPal). Now stop wondering why and start using other services.
Parent-poster wasn't lamenting monopolies, they were just incredulous that some supposedly well-resourced and integrated company still can't get its act together to stop a well-understood problem.
As a common-name@gmail.com holder, unfiltered spam is less commonplace than
a) websites adding me to distribution/mailing lists without verifying I was the person to type it in,
b) people who either accidentally transform their address into mine, or forget part of their address.
These are from all parts of the globe. I have access to bank accounts in South America, Disney employee music royalty earnings tax disclosures and forms, European subscribers online platforms, AWS account recovery options, veterinarian records in Studio City, private school/PTA leadership website access in Mountain View.
I stopped trying to return unopened mail, nobody cared.
I don't do anything with any of this because I'm not a giant fool, but people, have your users verify their email addresses before you trust them.
Ha! Me too. I have first.last@gmail.com, which means firstlast@gmail.com (without the dot) also maps to my email account. (first and last are placeholders for my actual name, of course).
I get lots of stuff intended for other people (including a mildly famous person with my name whose actual email address is last.first@gmail.com).
A few years ago, someone set up a shopify account with my email address. Overnight while I slept, the shopify account was created, did some bad stuff, and got suspended for fraudulent activity. The whole story was told through the series of emails coming in over a couple of hours. Shopify did not required the fraudster to confirm the email address to activate the account.
When I asked shopify to unlink my email address from this fraudster account, they instructed me to click on the "forgot my password" link on their login page, click on the "change password" link in the resulting email, and then login and remove my email from the account. This was my only option, they claimed. Obviously, I was never going to connect my IP address with some fraudster's Shopify account, so I just left it as-is.
I guess services feel it adds too much "friction" to force a user to verify the specified email account before allowing them to use the service.
Currently, my experience is that spam filters often block non-spam messages and allow many spam messages through. Platform notifications are spam! Yes, even if I signed up for it, I agreed to be contacted by the company, I never agreed to be spammed by them 100 messages a day.
Platforms should aggregate all their notifications into a single daily or weekly email unless the thing is marked urgent! It's not difficult to do!
38 comments
[ 0.20 ms ] story [ 5.5 ms ] threadI realize ASCII was limited, but one thing I like about it is I can understand every single character code, and program to handle all the edge cases with certainty.
But it's not simple enough for every company/app to independently research the entire Unicode code point space (which is MASSIVE) to find out what kinds of "fix ups" that app needs to do to clean the data it consumes.
It's complicated or at least has difficult tradeoffs. Maybe you invest a lot of time carefully surveying all of the Unicode planes and decide which ones you care to keep unchanged and which ones you filter/strip. For every code point you reject or change, there is going to be some user who is confused or dissatisfied with the limitations of your app.
I don’t think it’s reasonable to blame the Unicode authors for not anticipating this turn of events.
But also, according to the Ars article comment describing the Unicode character range, it has been deprecated, so maybe someone involved in the Unicode standard saw problems with it.
It's great for teaching and other things, but everyday life is filled with many characters, is the suggestion we'd have one ASCII per language where there is more distinct characters, or what would we do? I don't see what else we could have done, that would have worked for the world, but I'm curious to hear ideas.
There was a rationale (ISO country codes to modify a flag to display that national flag).
Maybe the problem wasn't the proposal, but the lack of the ability for others to reject it for being insecure.
If you want an image, embed an image.
My suspicion is that the spam filter programmers didn't do a comprehensive evaluation of every code point on every plane of Unicode because... well that's a massive job. So your "sanitize and denormalize" tasks are actually massive mappings which were likely imperfectly created.
The task isn't that massive. Python's unicodedata (for example) contains all the info already. Including the normalization (not denormalization as I wrote before) function.
Is there something I'm missing?
However, I'm sure that you are also familiar with the HN standard issue meme that it is impossible to run your own email system.
So, your email provider is crap and you can't do it yourself!
Bollocks! I use Exim (1) and rspamd (2) and run quite a few bijou email systems. It does require some effort.
You pays your money and you makes your choice ...
1. https://www.exim.org/exim-html-current/doc/html/spec_html/in... 2. https://docs.rspamd.com/
You have to do email due-dil and off you trot. Gmail, MS and co really do respect the rules in my estimation and don't simply dump email out of spite.
If you go in with at least everything up to SPF working, you are normally good to go. DKIM and DMARC and all the rest are even better.
I run my own DNS too and am on good terms with upstream so DNSSEC is available. Like I said, it requires some effort but so do most things in life.
But what I don't understand is MS letting these through the spam filter but randomly, in a thread between myself and another person both using MS account emails, sending a single email in that thread to spam.
I realized the other day I haven't even checked my main email in months and I missed absolutely noting important.
> The root domain onmicrosoft.com is owned and managed by Microsoft Corporation, which uses it as the default domain for Microsoft 365 and Azure cloud environments
Parent-poster wasn't lamenting monopolies, they were just incredulous that some supposedly well-resourced and integrated company still can't get its act together to stop a well-understood problem.
These are from all parts of the globe. I have access to bank accounts in South America, Disney employee music royalty earnings tax disclosures and forms, European subscribers online platforms, AWS account recovery options, veterinarian records in Studio City, private school/PTA leadership website access in Mountain View.
I stopped trying to return unopened mail, nobody cared.
I don't do anything with any of this because I'm not a giant fool, but people, have your users verify their email addresses before you trust them.
I get lots of stuff intended for other people (including a mildly famous person with my name whose actual email address is last.first@gmail.com).
A few years ago, someone set up a shopify account with my email address. Overnight while I slept, the shopify account was created, did some bad stuff, and got suspended for fraudulent activity. The whole story was told through the series of emails coming in over a couple of hours. Shopify did not required the fraudster to confirm the email address to activate the account.
When I asked shopify to unlink my email address from this fraudster account, they instructed me to click on the "forgot my password" link on their login page, click on the "change password" link in the resulting email, and then login and remove my email from the account. This was my only option, they claimed. Obviously, I was never going to connect my IP address with some fraudster's Shopify account, so I just left it as-is.
I guess services feel it adds too much "friction" to force a user to verify the specified email account before allowing them to use the service.
Platforms should aggregate all their notifications into a single daily or weekly email unless the thing is marked urgent! It's not difficult to do!