Very good problem description and nice list of countermeasures!
However, the following countermeasure made be wonder:
> Manually retype excerpts to avoid invisible characters and homoglyphs.
Isn't this something you can automate? We should create linters for plain text (rather than code)? For example, depending on the language, reduce the text to a certain set of characters. Every character not in this whitelist is either replaced, or causes an error message the user (journalist) needs to deal with (i.e. remove it, or replace it with an innocent alternative, perhaps even proposing this replacement back to the linter project).
Of course, there are multiple ways for linting, which might become a fingerprint on it own. But then, if there are only 3 or 4 of such linter styles actually used (ideally, standardize on exactly one linting style), you can only tell which linter was used by the journalist, without any information about their source.
Using RegEx is generally frowned upon; RegEx is a bad language to write anything in apart from prototypes. Furthermore, this will not work for most text as even US English text contains special characters. Think paraphrasing other languages, names and imported words.
As the article you link argues, use of regular expressions when they are inappropriate is bad. This particular case - finding and replacing certain characters with other characters - is pretty well-suited to the problem, and is probably more readable than a bunch of open code to do the same thing.
(I'm not sure what you mean by DoS attacks - are you referring to the exponential case of backtracking? If so, don't use a regex engine with that problem, and don't use lookbehind/lookahead assertions, which aren't needed to solve this problem.)
> This particular case - finding and replacing certain characters with other characters - is pretty well-suited to the problem, and is probably more readable than a bunch of open code to do the same thing.
No, it is not a good solution to the problem; you ignore my earlier comments. English or latin text is not comprised of the sole ASCII characterset; it contains characters outside this set (quoting other languages, names, imported words for example).
Honestly, I do get your point about inappropriate use of regex, but this kind of simple text manipulation is well suited for regex. The biggest argument against using regex for this kind of problem is performance verses writing the same code programmatically in the host language (assuming you're using a fast AOT compiled language). However even that is a non-issue given the small quantities of text you're decoding.
Also I'd bet the regex in this instance would actually work out more readable because the transformations are basic so you're localising the text manipulation to simple rules rather than multiple lines of byte array reading and thus also potentially having to manually build in your own rudimentary unicode support too.
I think people dislike [Regex] for the some of the same reasons The Principle or Least Power makes sense.
If you can do the same manipulation in three ones of code it’s more likely to be correct and stay correct. And when you look at it again in six months you won’t have to stare at it. All those little time sucks add up as the code grows.
Firsly, for relatively simple regex expressions, any competent developer should be able to grok them very quickly - at least as quick as the equivelant C#/Java/whatever code.
Secondly, it may be that regex is the most performant solution, and sometimes that matters quite a bit.
Honestly, I just don't get why some people are intimidated by regex.
Regex is seldom the most performant code. I mean the engine themselves are fantastic pieces of engineering but in tight loops I've found I can get - sometimes significant - performance improvements by replacing regex matches or substitutions with purpose written string manipulation. Obviously the results depend massively on several big variables:
1/ the regex engine
2/ host language
3/ problem you're trying to solve
But I've found generally I was better off not using regex for performance critical code.
HOWEVER (!!!) where regex consistently wins is development time. Not just writing the code, but testing (it's trivially easy to test regex) and updating the pattern matching (Vs updating the equipment character matching in an imperative language).
Yeah regex can get ugly quickly, but then so can any language if misused.
I think the general problem with any automated solution is that there is so much room to game them. For instance, I could selectively replace a few visible ASCII characters with non-ASCII look a likes. Then, the investigators just need to see which characters are missing. Even with the OCR option you could selectively add typos.
I was kinda on the fence, but I was considering my target: Journalists. A journalist may not notice simple differences like an extra space here or there when reading, but probably wouldn't retype a double space. I agree that this should be automated in some way, but it's a bit of an arms race.
What kind of arms do you envision for normalizing text down to Latin-1, or even ASCII, while normalizing any whitespace to single space characters?
There is a known variant when every subscriber of a confidential text receives a slightly different copy with the same meaning. But it's much harder to implement, and does not scale.
Not that much harder to implement. You could automate that too with some basic text substitution. Eg replacing various instances of "and" with ampasand or plus sign. You could also vary joined words like "without" / "with out", and alternative between
the types of quotation marks used (as there are several in unicode).
And this is without breaking into more intelligent heuristics where you swap out synonyms ("more intelligent" because you'd need to be careful not to alter passages that need to be kept verbatim, like quoted text or where a synonym might alter the context of the sentence.but with a little care I think that is achievable as well)
I'm curious as to why you think a linter tool for sanitizing the text is less effective than the other measures you describe. As long as the target problem is zero-width characters, this seems like it should be more effective than all but #1. I would agree if you include the synonym or other lexical fingerprints.
Humans make fixes that are hard to codify in programming. I know it isn't perfect, but with my audience in mind (journalists) I thought it was probably safer than something automatic.
That’s a good start, but unless I’m misreading it[1] the range of homoglyphs it checks for is rather small. You might be better off importing the Unicode Consortium’s list of ‘confusables’[2] if you’re planned automated linting.
Using the Option-Right Arrow to move through the text a word at a time has a problem though. The cursor appears to gets stuck at that point, and MS Word shows the font changing.
The two different sentences do not look different for me in Lynx. I guess Lynx just passes the raw characters on and any Unicode handling happens in the terminal.
It's about the Kangxi Radicals Unicode block, compared the CJK characters block. If you want me to write a blog post about it, please comment and I'll get around to it.
It's interesting to me, because I've seen this effect (while copy pasting) and wondered why... if I hadn't been translating I would never have noticed.
Most journalists don't have a technical background and don't know how Unicode works. Plus round-tripping through 7bit ASCII is lossy to characters you may want to keep (accented loanwords / names, non-english text) and doesnt prevent all the attacks in the article (providing text with slightly different spellings / word orders / etc). 7bit ASCII also has invisible control characters of its own in the 0x0 - 0xF1 range...
Or maybe black listing is not the best approach, maybe a mix of multiple approaches. First strip out stuff and then view the text in a program that displays "unconventional" characters?
As a test I pasted the post's test sentences in Vim and the invisible characters are replaced by blocks of <XXXX> that are very hard not to notice. The more you think about it the more tricky corner cases you find :O
Since I'm looking at what I assume are ASCII characters, I just typed this into the console:
"We're not the same text, even though we look the same.".split("").forEach(function(c, i){ if(c.charCodeAt(0) > 127) {console.log("Danger: weird character detected: code " + c.charCodeAt(0) + " at index " + i);} });
For languages other than English (or even for English with some accented characters), you'll need to do a bit better than just spotting non-ASCII characters. Finding zero-width/not fully normalized characters would be a bit more robust, but still not perfect. IIRC, the zero-width joiner/non-joiner are important for rendering various Indian languages correctly. But I guess it depends on your use case, and how many false positives you can tolerate.
What I'd want for most cases is to "paste as printable ASCII." That covers a large number of use cases (the article's example is printable ASCII ostensibly to reach the largest audience).
I'm not sure what to do with more complicated language mixing, but I suppose a script to tell you what languages are being used and where there might be "out-of-place" character codes would help.
They are obvious if you have two copies and are actively looking for differences, otherwise not. If someone sent me a document with square brackets somewhere i wouldn't question their use of square brackets over parenthesis, unless they don't match.
There are many more options: Insert one of the many non-printing ASCII codes. Add an extra space, especially before a newline. Add or remove an extra newline at the bottom. Change punctuation where the choice is believable either way: Single quotes for double, commas for semicolons, periods for colons.
This reminds me of a time not too long ago when I was teaching programming courses; at least one of the students would somehow manage to get one of these or other weird characters into source code, resulting in much confusion. On the bright side, I take advantage of the opportunity and an impromptu lesson in data representation and character encoding soon follows.
It's also one of the things where a hex editor is extremely useful --- even if you're not working on low-level, seeing the bits directly can be a great confirmation of correctness.
I had this happen to me once (and only once, I learned my lesson). Our professor gave us a PDF with the problem description and it had a bit of code in it we were to put into our final program. Well, when I copy and pasted it some of the spaces copied as non-ASCII space.
Oh god pasting code from a PDF, that brings me back.
Doesn't your editor handle that? I don't exactly remember but I kinda remember having a button to convert pasted characters to ascii (mostly used for those annoying stylized unicode quotes)
Apparently Atom didn't/doesn't have that feature, at least to automatically display when problematic characters are present. I believe I ended up taking a hex editor to my code to see what was wrong, since it was only a single loop that appeared to stop the compilation.
I use LibreOffice and these appear as grey characters (at least some of the characters do). I've wondered what the grey characters were in the past and now I know.
Copy and paste the examples into LibreOffice and you will immediately see them.
A fun fact is that the thumbprint text box in the certificate viewer in Windows starts with an invisible character.
So if you have cert and just want to copy paste the thunbprint to some file or application which needs to load it, then copying the full thumbprint probably won't work.
At one job I had to declare a moratorium on sending certain bits of information through Outlook because the number of people who needed help cutting and pasting it correctly was becoming a problem. Everything went into the wiki or config files that were checked in or used as attachments.
And don’t get me started on Microsoft and their fucking smart quotes...
This. I used to maintain a software project that consisted of a few inter-communicating services on clients' windows machines (not just servers, but that would have been optimal). The most difficult part of making a sale was getting the implementation guys to correctly install these components, issue a self-signed key from the machine's local CA, and bind it to the local dns/ssl port. Not to mention most people don't even really understand how/why certificates work, so if they ran into the tiniest snag it was going to completely block progress until a developer could take a look. Barf.
Working with certificates on Windows in general is error-prone and difficult to automate (this coming from someone who spent more than a decade developing in .NET).
I've dealt with zero width characters on Windows causing problems in a variety of situations; all appear to have originated by copying something from SharePoint or Lync / Skype4business. It was included in a SQL query, causing it to fail to parse. Another time, one ended up somehow getting into a database field which then created filesystem paths containing the character, which was a much trickier one to figure out.
At one of my jobs we used SharePoint for documentation and I would constantly have issues with it inserting zero width characters, particularly when pasting into a terminal. I finally got tired of it and wrote a small Chrome plugin that copies text without those characters.
Back in the Usenet days, around the late '80s, I had read somewhere/somehow a few years before about how classified information would be printed with tiny differences in spacing to track leaks to foreign adversaries. In some newsgroup, I happened to mention I had read about this technique, which while obvious once you think of it was apparently not well known. Certainly by that time, it was very well known to the KGB.
Months after my public comment, I got a phone call from an AT&T inventor who was prosecuting a patent on the same technology. They were very interested where I had read about the technique in the public literature. Alas, I could not remember where I had read that little factoid so I wasn't much use to them. It was disturbing to them that their patent claim was out in the open literature somewhere, but they could not find it.
The general idea is called a Canary Trap, and IIRC it was popularized by Tom Clancy. The most common form is to have some subtle variations in the content or wording, which would give away the origin.
I spent far too long looking at your comment history trying to figure out if you really were the author (since others have shown up here) until I stumbled on something semi-verifiable and visited Wikipedia (which should have been my first stop), only to learn the author Tom Clancy passed away a few years ago. :)
One of the alliances in Eve online used to do this type of counter intelligence to assist in identifying spies releasing internal communications. They’d also introduce invisible watermarks in any images they posted.
My understanding was their leadership had a tool to help them select a few synonyms in anything they wrote, and a few synonyms was all it took to identify the account releasing the communication.
More than one. PL, Goons, Test, various Russian alliances, and some smaller players all would. I really loved EvEs (counter)intelligence scene, it felt like a great place to safely dabble in real tools and techniques like this.
I think they mention it as being ineffective. The method mentioned in the book involves inserting inflammatory quotes into different versions of a report, in the hopes that the quote will show up somewhere.
For what it's worth, there are genuine use-cases for these. My friends and I use them for our IRC bots so they can mention people's names without notifying them.
Elon Musk did this nearly a decade ago at Tesla to try to identify a source of leaks. He sent many copies of a memo with slightly different versions to key team members.
It backfired when one of the top executives forwarded his own copy to the rest of the team.
Recently I played around with zero-width chars to test some inputs. It's awesome because big payloads can be just one innocuous line. The downside is that navigating through text with arrows will also navigate through invisible chars appearing stuck for a couple clicks in one place.
> it appears both homoglyph substitution and zero-width fingerprinting have been discovered by others
They were discovered a long time ago, and there are many other ways to hide data in documents.
Remember that you only need to encode enough bits for a relatively unique ID (and not unique for all files in the universe but only for files with the same content - for a low-distribution file, even 2-5 bits might be enough). On the application level, the most common applications and formats have a very large number of features you can utilize to encode data or simply insert it (e.g., Word, Excel, the PDF standard). On the bit level, unless the application vendor has invested in writing exceptionally tight, secure code, you probably can find someplace to hide/encode a few bits in a file.
But I think the author is on the right track with the solutions ...
> Use a tool that strips non-whitelisted characters from text before sharing it with others.
A more general solution is needed: Something that normalizes data in many formats, from text to Word to PDF to JPG to WAV to markup languages.
Personally, for non-security reasons, I'd love a utility that normalizes text to 7-bit ASCII (e.g., from UTF-8 characters higher than 7 bits) and that fits very efficiently into workflow (e.g., something that normalizes text in the clipboard if I press a hotkey combination). Anybody know of one?
The difference being that “fingerprint” means deriving a signature from some existing data, and “watermark” means altering data to make it identifiable?
The existence of homoglyphs in Unicode is a failure of Unicode's mission. Two sequences of characters that render the same should be the same. Encoding invisible semantic information into Unicode is a huge mistake.
I disagree, if anything they didn't go far enough. Unifying Chinese and Japanese characters making them locale dependent was, IMO, a mistake. The kind of problem I have with this can be seen with the so called "Turkish I" problem. The Turkish language has 4 'i's: i, I, ı and İ. Unicode decided to encode the first two using the points for latin lower case i and upper case I. In Turkish, the capitalization rules say that i and ı are lower case, and their upper case counterparts are İ and I. You can see that if you have a byte stream for which you don't know the locale, you cannot correctly apply capitalization to it. That way all 'I's would be unambiguous in all contexts. This is not as trivial a problem as it sounds[1]. This could have been avoided if Unicode had a Turkish i and I. You can extrapolate this issue to entire languages.
This is beyond the scope of Unicode. Unicode should be how text is displayed. If there is meaning that is not in the display of the character, then it is outside the scope of Unicode.
After all, we have many uses for the letter 'a', such as a) a*b=c and b) a as in apple. Should those 'a's have different Unicode code points?
Semantic meaning comes from context, and there is no context for a Unicode code point. Trying to insert semantic meaning is both a mistake and a patently impossible task. The article points out some of the wreckage attempting to do this causes.
> Should those 'a's have different Unicode code points?
Unicode does have separate code points for mathematical symbols. See for instance U+1D44E MATHEMATICAL ITALIC SMALL A. They see little use in practice though, apart from people using them for funky fonts on Reddit and such.
They are used by OpenType math fonts and systems that use them (e.g. the new equation editor in Word from 2007 on wards, and Unicode-capable TeX systems with the `unicode-math` package).
That's where defaults become important. When switching to a Turkish locale the system would try to use the Turkish version for the characters, but of course this would make it harder for Turkish speakers to switch back and forth with other languages...
I guess we've always been in the days of "worse is better", like using CSV even though ASCII encodes characters specific to record separation[1].
So thanks to Unicode, things that look the same can have different representations, and things that look different can have the same one. (Limited) text used to be easy, now it's even worse than recording pen-strokes in a vector graphics format. How is this still useful?
How can it be "a failure of Unicode's mission" when the mission does not really include that? The foremost reason that Unicode exists is the unification of tons of existing legacy character sets and encodings and not the technologically perfect system. There are enough duplicates and invisible characters in a single character set (yes some have duplicates by accident or design), across multiple character sets (so you like to map Latin A and Cyrillic А to the same character while retaining the collation order for each case...), or simply necessary to multilingual systems that jeopardize any attempt to "perfect" it.
Granted, there are several failed experiments (e.g. interlinear annotations which are completely obsoleted by markup languages) and several pain points (e.g. shrug Emoji) inside Unicode. I don't really like them, you may not like them, but how would you make such a system without all these interim works?
Then use an image, not a text. (Or use a font-mapped text which is not externally exposed by itself. Seemingly similar to PDF, but alas, PDF supports textual selection ;-) If you cannot afford that you'd better acknowledge the complexity of multilingual systems.
This isn't as new as the author thinks. Doesn't have to be new to be interesting, of course :)
I know this was done at one large tech company around 2010 for an internal announcement email. Different people got copies with slightly different Unicode whitespace, despite the email having ostensibly gone directly to an "all employees" alias.
The fingerprinting was noticed within half an hour or so. (Somebody pasted a sentence to an internal IRC for discussion, and as is the case with Unicode and IRC, it inevitably showed up as being garbled in exciting ways for some people).
149 comments
[ 4.7 ms ] story [ 220 ms ] threadHowever, the following countermeasure made be wonder:
> Manually retype excerpts to avoid invisible characters and homoglyphs.
Isn't this something you can automate? We should create linters for plain text (rather than code)? For example, depending on the language, reduce the text to a certain set of characters. Every character not in this whitelist is either replaced, or causes an error message the user (journalist) needs to deal with (i.e. remove it, or replace it with an innocent alternative, perhaps even proposing this replacement back to the linter project).
Of course, there are multiple ways for linting, which might become a fingerprint on it own. But then, if there are only 3 or 4 of such linter styles actually used (ideally, standardize on exactly one linting style), you can only tell which linter was used by the journalist, without any information about their source.
Regular expressions are powerful, and are used in plenty of production software.
* RegEx is not very readable
* RegEx can be (very) slow
* It's not trivial to write RegEx code that achieves your goal in a high-quality way. Often quircks and edge-cases are missed.
I'm not saying that you should never use them, but oftentimes a (much) better alternative of achieving your goals is present.
See https://blog.codinghorror.com/regular-expressions-now-you-ha...
(I'm not sure what you mean by DoS attacks - are you referring to the exponential case of backtracking? If so, don't use a regex engine with that problem, and don't use lookbehind/lookahead assertions, which aren't needed to solve this problem.)
No, it is not a good solution to the problem; you ignore my earlier comments. English or latin text is not comprised of the sole ASCII characterset; it contains characters outside this set (quoting other languages, names, imported words for example).
Honestly, I do get your point about inappropriate use of regex, but this kind of simple text manipulation is well suited for regex. The biggest argument against using regex for this kind of problem is performance verses writing the same code programmatically in the host language (assuming you're using a fast AOT compiled language). However even that is a non-issue given the small quantities of text you're decoding.
Also I'd bet the regex in this instance would actually work out more readable because the transformations are basic so you're localising the text manipulation to simple rules rather than multiple lines of byte array reading and thus also potentially having to manually build in your own rudimentary unicode support too.
If you can do the same manipulation in three ones of code it’s more likely to be correct and stay correct. And when you look at it again in six months you won’t have to stare at it. All those little time sucks add up as the code grows.
Edit: autocorrect got me twice.
Firsly, for relatively simple regex expressions, any competent developer should be able to grok them very quickly - at least as quick as the equivelant C#/Java/whatever code.
Secondly, it may be that regex is the most performant solution, and sometimes that matters quite a bit.
Honestly, I just don't get why some people are intimidated by regex.
1/ the regex engine
2/ host language
3/ problem you're trying to solve
But I've found generally I was better off not using regex for performance critical code.
HOWEVER (!!!) where regex consistently wins is development time. Not just writing the code, but testing (it's trivially easy to test regex) and updating the pattern matching (Vs updating the equipment character matching in an imperative language).
Yeah regex can get ugly quickly, but then so can any language if misused.
I suspect he put it low on the list because it could be a cat and mouse game of trying to anticipate all the potential information leaks.
I was kinda on the fence, but I was considering my target: Journalists. A journalist may not notice simple differences like an extra space here or there when reading, but probably wouldn't retype a double space. I agree that this should be automated in some way, but it's a bit of an arms race.
There is a known variant when every subscriber of a confidential text receives a slightly different copy with the same meaning. But it's much harder to implement, and does not scale.
And this is without breaking into more intelligent heuristics where you swap out synonyms ("more intelligent" because you'd need to be careful not to alter passages that need to be kept verbatim, like quoted text or where a synonym might alter the context of the sentence.but with a little care I think that is achievable as well)
https://www.zachaysan.com/writing/2018-01-01-fingerprinting-...
Humans make fixes that are hard to codify in programming. I know it isn't perfect, but with my audience in mind (journalists) I thought it was probably safer than something automatic.
UTF-8 source code is nice for i18n, but it also opens the door to these kinds of attacks.
[1] https://github.com/NebulousLabs/glyphcheck/blob/f6483dd9e97a...
[2] http://www.unicode.org/Public/security/latest/confusables.tx...
Damian Conway is a wonderful mad genius.
It might affect string handling where you split a string by space characters, and then compare words with a dictionary.
For example, Pingtype English which translates word-by-word to Chinese. (note the words "the" and "same" not getting translated in the examples).
https://pingtype.github.io/english.html
Google Translate handles it fine.
Using the Option-Right Arrow to move through the text a word at a time has a problem though. The cursor appears to gets stuck at that point, and MS Word shows the font changing.
I dare you to inspect the html in Safari, Chrome, Firefox, and even curl, to see if you can view the raw character encoding.
Some you can, some you can't.
I've also found a lot of identical characters when handling Chinese text. Note that Google Translate does not handle these correctly.
https://github.com/pingtype/pingtype.github.io/blob/master/r...
It's about the Kangxi Radicals Unicode block, compared the CJK characters block. If you want me to write a blog post about it, please comment and I'll get around to it.
Even though some are noticeably different:
⿌ 黾
黾 is the simplified version of ⿌.
(In http://pingtype.github.io click Advanced > Regional, paste into the Simplified text box, then click "Simplified to Traditional")
http://kb.mozillazine.org/Network.IDN.blacklist_chars
Or maybe black listing is not the best approach, maybe a mix of multiple approaches. First strip out stuff and then view the text in a program that displays "unconventional" characters? As a test I pasted the post's test sentences in Vim and the invisible characters are replaced by blocks of <XXXX> that are very hard not to notice. The more you think about it the more tricky corner cases you find :O
"We're not the same text, even though we look the same.".split("").forEach(function(c, i){ if(c.charCodeAt(0) > 127) {console.log("Danger: weird character detected: code " + c.charCodeAt(0) + " at index " + i);} });
I'm not sure what to do with more complicated language mixing, but I suppose a script to tell you what languages are being used and where there might be "out-of-place" character codes would help.
https://en.wikipedia.org/wiki/IDN_homograph_attack
- replace quotation marks with two apostrophes
- replace “I” with “l”
- replace parentheses with slashes or square brackets
- replace commas with semicolons
Each of these substitutions can communicate a single bit while being ASCII-safe and likely won’t change word wrapping opposed to synonyms.
Speaking of quotation marks, the non-ASCII quotes in your second item and apostrophe in the last sentence stand out too.
It's also one of the things where a hex editor is extremely useful --- even if you're not working on low-level, seeing the bits directly can be a great confirmation of correctness.
Doesn't your editor handle that? I don't exactly remember but I kinda remember having a button to convert pasted characters to ascii (mostly used for those annoying stylized unicode quotes)
Copy and paste the examples into LibreOffice and you will immediately see them.
So if you have cert and just want to copy paste the thunbprint to some file or application which needs to load it, then copying the full thumbprint probably won't work.
When I said fun I meant frustrating.
And don’t get me started on Microsoft and their fucking smart quotes...
Working with certificates on Windows in general is error-prone and difficult to automate (this coming from someone who spent more than a decade developing in .NET).
Months after my public comment, I got a phone call from an AT&T inventor who was prosecuting a patent on the same technology. They were very interested where I had read about the technique in the public literature. Alas, I could not remember where I had read that little factoid so I wasn't much use to them. It was disturbing to them that their patent claim was out in the open literature somewhere, but they could not find it.
See https://qz.com/1002927/computer-printers-have-been-quietly-e...
Edit here we go https://en.m.wikipedia.org/wiki/Canary_trap
The Canary Trap, aka The Barium Meal.
My understanding was their leadership had a tool to help them select a few synonyms in anything they wrote, and a few synonyms was all it took to identify the account releasing the communication.
https://books.google.com/books?id=Vj-NDQAAQBAJ&pg=PA346&dq=c...
Red Sparrow also has a canary trap, and they refer to it thusly.
https://books.google.com/books?id=l-lnAwAAQBAJ&pg=PA428&dq=c...
It backfired when one of the top executives forwarded his own copy to the rest of the team.
https://www.cbsnews.com/news/should-management-spy-on-employ...
They were discovered a long time ago, and there are many other ways to hide data in documents.
Remember that you only need to encode enough bits for a relatively unique ID (and not unique for all files in the universe but only for files with the same content - for a low-distribution file, even 2-5 bits might be enough). On the application level, the most common applications and formats have a very large number of features you can utilize to encode data or simply insert it (e.g., Word, Excel, the PDF standard). On the bit level, unless the application vendor has invested in writing exceptionally tight, secure code, you probably can find someplace to hide/encode a few bits in a file.
But I think the author is on the right track with the solutions ...
> Use a tool that strips non-whitelisted characters from text before sharing it with others.
A more general solution is needed: Something that normalizes data in many formats, from text to Word to PDF to JPG to WAV to markup languages.
Personally, for non-security reasons, I'd love a utility that normalizes text to 7-bit ASCII (e.g., from UTF-8 characters higher than 7 bits) and that fits very efficiently into workflow (e.g., something that normalizes text in the clipboard if I press a hotkey combination). Anybody know of one?
https://crashcourse.housegordon.org/coreutils-multibyte-supp...
[1] https://gizmodo.com/382026/a-cellphones-missing-dot-kills-tw...
After all, we have many uses for the letter 'a', such as a) a*b=c and b) a as in apple. Should those 'a's have different Unicode code points?
Semantic meaning comes from context, and there is no context for a Unicode code point. Trying to insert semantic meaning is both a mistake and a patently impossible task. The article points out some of the wreckage attempting to do this causes.
Unicode does have separate code points for mathematical symbols. See for instance U+1D44E MATHEMATICAL ITALIC SMALL A. They see little use in practice though, apart from people using them for funky fonts on Reddit and such.
You mean like ⒜ or ⓐ or ᵃ? Not to be confused with any of ªａ𝐚𝑎𝒂𝒶𝓪𝔞𝕒𝖆𝖺𝗮𝘢𝙖𝚊ₐ. Unicode has a whole lot of stuff that nobody uses…
There's also a separate Cyrillic а.
I guess we've always been in the days of "worse is better", like using CSV even though ASCII encodes characters specific to record separation[1].
[1]: https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text
Should the Greek alphabet be removed? Why have delta when d is just as good?
What I'm talking about is when the glyphs are identical, i.e. homoglyphs. Homoglyphs should be removed.
Granted, there are several failed experiments (e.g. interlinear annotations which are completely obsoleted by markup languages) and several pain points (e.g. shrug Emoji) inside Unicode. I don't really like them, you may not like them, but how would you make such a system without all these interim works?
I know this was done at one large tech company around 2010 for an internal announcement email. Different people got copies with slightly different Unicode whitespace, despite the email having ostensibly gone directly to an "all employees" alias.
The fingerprinting was noticed within half an hour or so. (Somebody pasted a sentence to an internal IRC for discussion, and as is the case with Unicode and IRC, it inevitably showed up as being garbled in exciting ways for some people).