RFC 9839 and Bad Unicode (tbray.org)

[dead]

Seems like libraries that serialize to JSON should have an option to filter out these bad characters.

I’m frustrated by things like Unicode where it’s “good” except… you need to know to exclude some of them. Unicode feels like a wild jungle of complexity. An understandable consequence of trying to formalize so many ways to write language. But it really sucks to have to reason about some characters being special compared to others.

The only sanity I’ve found is to treat Unicode strings as if they’re some proprietary data unit format. You can accept them, store them, render them, and compare them with each other for (data, not semantic) equality. But you just don’t ever try to reason about their content. Heck I’m not even comfortable trying to concatenate them or anything like that.

I have had real-world programs broken by blind assumption of "does not deliberately contain controls" (form feed is particularly common for things intended to be paginated, escape is common for things designed for a terminal, etc.) and even "is fully UTF-8" (there are lots of old data files and logs that are never going away).

If you aren't doing something useful with the text, you're best off passing a byte-sequence through unchanged. Unfortunately, Microsoft Windows exists, so sometimes you have to pass `char16_t` sequences through instead.

The worst part about UTF-16 is that invalid UTF-16 is fundamentally different than invalid UTF-8. When translating between them (really: when transforming external data into an internal form for processing), the former can use WTF-8 whereas the latter can use Python-style surrogateescape, but you can't mix these.

I'm not certain... On one hand I agree that some characters are problematic (or invalid) - like unpaired surrogates. But the worst case scenario is imo when people designing data structures and protocols start to feel the need to disallow arbitrary classes of characters, even properly escaped.

In the example, username validation is a job of another layer. For example I want to make sure username is shorter than 60 characters, has no emojis or zalgo text, and yes, no null bytes, and return a proper error from the API. I don't want my JSON parsing to fail on completely different layer pre-validation.

And for username some classes are obviously bad - like explained. But what if I send text files that actually use those weird tabs. I expect things that work in my language utf8 "string" type to be encodable. Even more importantly, I see plenty of use cases for null byte, and it is in fact often seen in JSON in the wild.

On the other hand, if we have to use a restricted set of "normal" Unicode characters, having a standard feels useful - better than everyone creating their own mini standard. So I think I like the idea, just don't buy the argumentation or examples in the blog post.

[dead]

> PRECISion · You may find yourself wondering why the IETF waited until 2025 to provide help with Bad Unicode. It didn’t; here’s RFC 8264: PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols; the first PRECIS predecessor was published in 2002. 8264 is 43 pages long, containing a very thorough discussion of many more potential Bad Unicode issues than 9839 does.

I’d also suggest people check out the accompanying RFCs 8265 and 8266:

PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols:

— https://www.rfc-editor.org/rfc/rfc8264

Preparation, Enforcement, and Comparison of Internationalized Strings: Representing Usernames and Passwords

— https://www.rfc-editor.org/rfc/rfc8265

Preparation, Enforcement, and Comparison of Internationalized Strings Representing Nicknames:

— https://www.rfc-editor.org/rfc/rfc8266

Generally speaking, you don’t want usernames being displayed that can change the text direction, or passwords that have different byte representations depending on the device that was used to type it in. These RFCs have specific profiles to avoid that.

I think for these kinds of purposes, failing closed is more secure than failing open. I’d rather disallow whatever the latest emoji to hit the streets is from usernames than potentially allow it to screw up every page that displays usernames.

I think there should be a restriction in the standard on how many Unicode scalar values a graphical unit can have.

Last time I checked (a couple of years ago admittedly) there was no such restriction in the standard. There was however a recommendation to restrict a graphical unit to 128 bytes for "streaming applications".

Bringing this or at least a limit on the scalar units into the standard would make implementation and processing so much easier without restricting sensible applications.

It's worth noting that Unicode already defines a "General Category" for all code points that categorizes some of these types of "weird" characters.

https://en.wikipedia.org/wiki/Unicode_character_property#Gen...

e.g. in Python,

   import unicodedata
   print(unicodedata.category(chr(0)))
   print(unicodedata.category(chr(0xdead)))

Shows "Cc" (control) and "Cs" (surrogate).

I was not able to understand why these code points are bad. The post states that they are bad, but why? Any examples? Any actual situations and PoC that might help me understand how will that break "my code"?

Excluding all of "legacy controls" not just as literals but also escaped strings (e.g. "\u0027") seems too much. C1 is essentially unused AFAIK and that's okay, but a number of C0 characters do see real-world use (escape, EOF, NUL). IMHO there are valid and reasonable use cases to use some of them.

how does this compare to Go `unicode.IsPrint(r rune)`? https://pkg.go.dev/unicode#IsPrint

what does bad/dangerous this code catch that `unicode.IsPrint` is not catching?

or other way, what good/useful does `unicode.IsPrint`removing, that this code keeps?

It seems like most of these are handled by just rejecting invalid UTF-8 byte sequences (ideally, erroring out altogether) when interpreting a string as UTF-8. I mean, unpaired surrogates, or any surrogate for that matter, is already illegal as a UTF-8 byte sequence. Any competent language that uses UTF-8 for strings should already be returning errors when given such sequences.

The list of code points which are problematic (non-printing, etc) are IMO much more useful and nontrivial. But it’d be useful to treat those as a separate concept from plain-old illegal UTF-8 byte sequences.

I am torn on one decision: Whether to control inputs, or to wrap untrusted input in a datatype that displays it safely (web+log+debug).

I don't understand how this helps.

Defining a subset of unicode to accept does not obviate the need to check that values conform to type definitions.

Imagine the possibilities:

  PS C:\> mkdir x
  
      Directory: C:\
  
  Mode                 LastWriteTime         Length Name
  ----                 -------------         ------ ----
  d----          23.08.2025    21:43                x
  
  PS C:\> cd x
  PS C:\x> new-item -ItemType File 'C꞉⧵Windows⧵System32⧵rundll32.exe'
  
      Directory: C:\x
  
  Mode                 LastWriteTime         Length Name
  ----                 -------------         ------ ----
  -a---          23.08.2025    21:43              0 C꞉⧵Windows⧵System32⧵rundll32.exe
  
  PS C:\x> $f=gi .\C꞉⧵Windows⧵System32⧵rundll32.exe
  PS C:\x> $f | fl -Property * -Force
  
  PSPath              : Microsoft.PowerShell.Core\FileSystem::C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe
  PSParentPath        : Microsoft.PowerShell.Core\FileSystem::C:\x
  PSChildName         : C꞉⧵Windows⧵System32⧵rundll32.exe
  PSDrive             : C
  PSProvider          : Microsoft.PowerShell.Core\FileSystem
  PSIsContainer       : False
  Mode                : -a---
  ModeWithoutHardLink : -a---
  VersionInfo         : File:             C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe
                        InternalName:
                        OriginalFilename:
                        FileVersion:
                        FileDescription:
                        Product:
                        ProductVersion:
                        Debug:            False
                        Patched:          False
                        PreRelease:       False
                        PrivateBuild:     False
                        SpecialBuild:     False
                        Language:
  
  BaseName            : C꞉⧵Windows⧵System32⧵rundll32
  ResolvedTarget      : C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe
  Target              :
  LinkType            :
  Name                : C꞉⧵Windows⧵System32⧵rundll32.exe
  Length              : 0
  DirectoryName       : C:\x
  Directory           : C:\x
  IsReadOnly          : False
  Exists              : True
  FullName            : C:\x\C꞉⧵Windows⧵System32⧵rundll32.exe
  Extension           : .exe
  CreationTime        : 23.08.2025 21:43:35
  CreationTimeUtc     : 23.08.2025 19:43:35
  LastAccessTime      : 23.08.2025 21:43:35
  LastAccessTimeUtc   : 23.08.2025 19:43:35
  LastWriteTime       : 23.08.2025 21:43:35
  LastWriteTimeUtc    : 23.08.2025 19:43:35
  LinkTarget          :
  UnixFileMode        : -1
  Attributes          : Archive
  
  PS C:\x>

it's almost like every language need another standards: safeunicode.

which does away with control flow and specially positioning garbage. and doesn't consider as valid unknown (or missing) surrogates.

why do i want very specialized text tabulation and positioning chars in my string?

it's as if they tried to solve text encoding AND solve CSV and pagination and desktop publishing all in one go.

[dead]

Did anyone else find the use if ABNF annoying?

  unicode-assignable =
   %x9 / %xA / %xD /               ; useful controls
   %x20-7E /                       ; exclude C1 controls and DEL
   %xA0-D7FF /                     ; exclude surrogates
   %xE000-FDCF /                   ; exclude FDD0 nonchars
   %xFDF0-FFFD /                   ; exclude FFFE and FFFF nonchars
   %x10000-1FFFD / %x20000-2FFFD / ; (repeat per plane)
   %x30000-3FFFD / %x40000-4FFFD /
   %x50000-5FFFD / %x60000-6FFFD /
   %x70000-7FFFD / %x80000-8FFFD /
   %x90000-9FFFD / %xA0000-AFFFD /
   %xB0000-BFFFD / %xC0000-CFFFD /
   %xD0000-DFFFD / %xE0000-EFFFD /
   %xF0000-FFFFD / %x100000-10FFFD

I mean, just define ranges.

Also, where are the test vectors? Because when I implement this, that's the first thing I have to write, and you could save me a lot of work here. Bonus points if it's in JSON and UTF-8 already, though the invalid UTF-8 in an RFC might really gum things up: hex encode maybe?

I do not agree that Unicode is good. I think that Unicode is not good.

I also think that, regardless of the character set, what to include (control characters, graphic characters, maximum length, etc) will have to depend on the specific application anyways, so trying to include/exclude in JSON doesn't work as well.

Giving a name to a specific subset (or sometimes a superset, but usually subset) of Unicode (or any other character sets, such as ASCII or ISO 2022 or TRON code) can be useful, but don't assume it is good for everyone or even is good for most things, because it isn't.

RFC 9839 does give names to some subsets of Unicode, which may sometimes be useful, but should not automatically assume that is right for what you will be making. My opinion is to consider to not use or require Unicode.

The form feed character (U+000C) should probably have been included in useful controls. That's used in a fair amount of program source code.

> The first code point is zero, in Unicode jargon U+0000. In human-readable text it has no meaning, but it will interfere with the operation of certain programming languages.

This part encourages more active usage of U+0000, so that programmers of certain programming languages get a message that they are not welcome

No, please do in fact allow unpaired surrogates and other weird characters so people can round trip "bad" data like file names through your protocol.