It was a common technique back in the day with both dot matrix and "letter quality" printers to print a line and then go back and print it again to either get a bold effect by printing the same characters twice or to overlay one character on top of another. If the spacing was right you could have drawn accented characters that way.
Those kinds of printers were delightfully mechanical. Writing a little loop in GW-BASIC to grab key-at-a-time input (INKEY$) and writing it out to the parallel port was a ton of fun. You could create overstrike-type output like that easily. It also almost felt like controlling a robot. (I mean, you really were...)
It's a "feature" typewriters had. On mechanical ones you had to manually move the carriage to the left after the bell rang indicating you are out of space in the line, thus "Carriage Return" and also move paper up to move to the next line.
The purpose is more obvious if you understand what makes \r and \n different, and why both existed. But I guess that's becoming lost knowledge now.
Ultimately, many "character sets" combined printable characters, cursor/head control, and record boundaries into one serialized byte stream that could be used by terminals, printers, programs for all sorts of purposes.
It’s a shame that ASCII unit/record/group boundary characters were practically never used, at least since microcomputers took over. Instead we’re stuck with broken replacements like CSV.
Right? And they're a really obvious solution to a problem that we somehow still have even though they've been around forever. The only problem I've ever seen is that there's no good way to type or display them
It is a bit of a chicken and egg problem here. If those characters were used then at least for CSV support better text editors would find a way to handle them. Since they aren't nobody wants to try to use them in a CSV like file.
If there was a good way to type of display them, then normal people would be typing them every day (say, pasting them into comment forms or whatever), and then we would need to have some standard for quoting them.
In that alternate universe nobody would have put them into some varchar column? The edge case still exists, which means that the standard would need to support it somehow, and we're back to exactly square one.
But if I was writing a similar standard today and someone was pressuring me that I really had to make some kind of mechanism for nesting, I'd use something that transforms the field to remove the need for escaping, in a big and obvious way. Probably base64. That prevents most of the implementation issues you see with CSV.
Neither does a trivial variation of CSV where quotes are not special and commas must be escaped in whatever way. The separator is completely immaterial, the resulting format is identical.
Formally, yes, but in practice there's an enormous difference in the number and kind of records that might contain a comma and the number and kind that might contain a non-printable record seperator.
This difference carries over to C, where NULL got the job done for string termination pretty darn well, even if there were strong critiques to be weighed against it.
I think the C example is meaningfully different. \0 in C is a string terminator. There is no more string after it, so you can just read the string "while (c != 0)", and there is never any need to escape it, because by definition strings cannot contain that character.
From a purely linguistic standpoint, the choice is sound. Obviously it's objectively a disaster but for unrelated reasons.
sizeof s (fun fact: you can omit the parentheses because it's not a function, it's an operator) reports the size of the array s, which is a variable of automatic storage duration.
Consider the following program:
#include <stdio.h>
#include <string.h>
int main()
{
const char * const s = "gogo\x00gogo";
const char t[] = "gogo\x00gogo";
printf("%d,%d,%d,%d\n",
(int)sizeof s,
(int)strlen(s), // DO NOT EVER USE THIS FUNCTION
// YOU WILL BE FIRED IF YOU DO
(int)sizeof t,
(int)strlen(t) // DO NOT EVER USE THIS FUNCTION
// YOU WILL BE FIRED IF YOU DO
);
}
> 8,4,10,4
I think a big reason they never took off is also that there's no visual representation of them, or input method for them.
I'd be happy to use them instead of CSV if I could edit records in Notepad or TextEdit the way I can with commas and quoted strings. See the separators, type them.
But of course once you do that, somebody now wants to insert a list of five values separated by unit or record separators into a field. They'd just be common ASCII characters along with tab, CR, LF, etc, and need to be escaped the same.
Those characters were "control characters" and used for streaming data over serial, parallel and other devices. Control character could cause some real surprises when you used them.
XML has gaps in its escaping. (Oops I wrote JSON at first, got the technicalities mixed up.) Null-terminated strings don't have escaping. Fixed width formats don't have escaping. Type-length-value formats don't have escaping.
And in practice I do not expect CSV with embedded nulls to work properly, so there's already precedent to reject certain characters entirely in a CSV-like format.
You always strip them out from input because a record value can’t contain more records.
CSV is more broken than you imagine. Try opening an American CSV file with Excel in Germany or France. You’ll discover that it doesn’t parse, and the reason is that the ‘C’ in CSV apparently stands for “could be a semicolon too” because comma is the decimal separator in these countries and therefore must be available for use inside values.
I work in data engineering, I have seen more ways to fuck up data than any human has any business seeing, that's not new to me. Here's some more ideas for maximum fun with CSVs:
- Make sure to write your own parser! The format is so simple!
- Is the first line a header or not? The RFC says you should look at the MIME type to tell. You can't make this up.
- Is the line terminator "\r\n", "\n" or "\r"?
- What happens if the number of records for a line is incorrect? Well, that SHOULD'nt happen...
- Make sure to involve Excel, nothing ever went wrong feeding data to Excel.
- Are end-of-line spaces relevant or not? Should we trim them?
There is no defending of CSV, it's a broken format and it's broken beyond repair. The point is that the separator is immaterial. Replacing commas with something else gains us exactly nothing, and results in an identically broken format.
I love your post, and agree to everything, except the last 2 sentences. Yes, CSV is broken beyond repair. But this doesn't make the choice of separator immaterial. It's broken for other reasons AND for the separator. A character that's already profusely found in texts and numbers... I would even argue that it would be difficult to find a WORSE separator. Probably only space and hyphen qualify.
Full stop would be even worse. Apostrophe would be runner up...
Always the seemingly simple solutions have most issues in real world... At some level still trying to edit stuff by hand might be where we have gone wrong for long time. Why don't we have any sensible agreed structured format for which tools would be on all and every device...
>> Instead we’re stuck with broken replacements like CSV.
> ASCII separators are equally as broken. How do you represent a string containing the record separator in a record-separated file?
ASCII separators may not be perfect, but they're certainly less broken than CSV. With CSV, you have the pervasive problem of how to represent a common, everyday character in the file. With ASCII separators, you only have the problem of representing an unusual, untypeable
character, which is much less troublesome.
I'm happy with CSV values containing a comma being quoted and a quoted quote character being represented by doubling it (similar to SQL doubling apostrophe). I don't see what pervasive problem remains.
So many people look at a simple spec where correct implementation is a handful of lines, and many libraries are available, and decide they'll do an incorrect one-line implementation instead.
It's the temptation to just concatenate or split based on commas, or to hack a broken implementation of quotes on top, that causes so many issues.
So the problem isn't CSV the spec, it's the use of CSV without using correct library implementations. I can hardly blame that on the format itself and shouldn't be a reason to reach for something else that could also be poorly parsed with self-written code. The other extreme is YAML where the complex spec makes using any implementation is concerning.
So many people keep messing up that at a certain point you can't just say it's their fault, there's a responsibility to guide them into better behavior.
There are two main options for that. Make it so that poorly parsing with self-written code is still very likely to do the right thing, or make the parsing hard enough that people go get a library.
This is not actually the case. The problem was you were transporting CSV often using a mechanism that used control characters at the hardware level. Using those characters often would have unpredictable results when storing, retrieving, and transmitting data.
Interestingly, if you ever scan a FedEx barcode, they use tons of ASCII separator characters. About the only time I've seen them significantly used though.
The better question is what percentage of _printers_ could handle arbitrary overlap, and I think the answer is, all of them. It's just a head striking paper, backspace and strike twice is always an option.
Printers were fare more common than hardware terminals for most of the initial history of computing.
The hardware is not capable of conjuring up new characters, so this is limited to detecting things like bold and underline… and maybe mapping the overprinting to characters that already exist in the font.
If you focus on printers I think you're missing the point of the question. It's about why it's in this character set made for interactive use on a screen.
Printers are an obvious answer for a different question.
Edit: Its 1983 predecessor that added accented characters was made for the VT220, and it used a buffer of 8 bit character codes, no way to combine. (Some of those terminals could be sent a limited number of custom characters to store in RAM, but that's a very different mechanism as far as I can tell.)
Edit 2: I made an error before, the naked accents were not in the predecessor, they were even newer.
Why? Back in the day you could output a text file directly onto the printer and it would print, possibly interpreting control characters like backspace or carriage return to overwrite existing characters.
For that, the printer would also need to interpret the characters in the correct character set.
But who would design a character set that only works properly on printers in 1985? Hence the question of whether and which operating systems properly supported that kind of use.
It's still a pretty crazy idea to add something that only works on printers at that time. It's very much worth asking about OS support, and "it's there because printers" leaves a lot of story untold, it's not a sufficient answer.
We did printing with heads on specialized form paper. If you wanted to change the font you opened up the printer, took out the head, and replaced it with a new head. The character set standard defined how the head was physically manufactured. You want all the printers in your fleet to produce identical output.
In 1985 "OS Support" meant picking up the phone and talking to the vendor directly.
> they're all pretty much useless on their own for anything besides ASCII art.
The asker completely ignores that asking questions about accent marks, like they themselves are doing in that very post, would be a lot more annoying without being able to write said accent marks.
For character sets, typing is use, so the asker is directly using them in order to ask what their use is. It's sufficient for their use to be talking about them.
That's cute, except you don't need to invent anything new here. There are already accent marks used in the character set. Note that they did not ask why there are symbols ÀÁÂ etc, so the presence of accents is already understood and accepted. And given that the character set is constructed to include accented letters, it is extremely reasonable to include the ability to talk about the accent marks themselves.
Otherwise you would have to resort to the quite clumsy "the thing on top of àìòèù" to introduce any discussion of the uses of grave accents to people who don't yet know what the phrase "grave accent" means instead of just writing "`", which would be akin to saying that we don't need to write the letter "A" by itself to talk about the letter "A" and could instead say things like "the letter at the beginning of the words Apple and Among". Like...yes...you can do that. But it's less good than the alternative of not doing that.
My younger self would have used the umlaut character for 'ditto'.
To some extent the character set was still evolving, for example the Euro sign was not around until decades later and that would need to be bolted together with backspace characters or escape codes, maybe even downloaded characters, with the printer specific manual (Epson) studied at great length.
In the DOS era (and before with home micros that were programmed in BASIC) it was quite normal to compose things for the printer that you had no expectation of seeing on screen, not that anyone read much on screen (as everyone had vast piles of paper on their desk).
Until quite recently some POS systems were very much tied in to a very specific printer, at least these character sets were a step forward from hard coding a BASIC program to an exact make and model of printer.
Creative use of early Epson printers was something I wasted a lot of time with, never with a word processor, because they did not really exist unless you paid hundreds of pounds to get one.
I was coding forms for tabular data, doing silly things because the printed manual made that too tempting, so any excuse to use a character or escape code sequence was fair game.
This was in an era when handwriting was commonly used so the convention for 'ditto' could be freestyled by me to be middle dots, umlauts or anything CHR(). The convenience of the keyboard, in an era before the PC was not really there.
> Having an OS drawing characters on a bitmap display, a prerequisite to composing, is a very new development, way more recent than the character definitions leading to above encoding.
New? Some computers of the 1980s could already do this. At least the 16 bit home computers had bitmap drawn characters on the screen.
Edit: Looks like somebody doesn't believe that computers in the 80s had such a thing.
> On the Amiga, rendering text is similar to rendering lines and shapes. The Amiga graphics library provides text functions based around the RastPort structure, which makes it easy to intermix graphics and text.
> In order to render text, the Amiga needs to have a graphical representation for each symbol or text character. These individual images are known as glyphs. The Amiga gets each glyph from a font in the system font list. At present, the fonts in the system list contain a bitmap of a specific point size for all the characters and symbols of the font.
I didn't comment on the character sets but on the claim that text being drawn as bitmaps instead with a fixed character mode was "a very new development".
They exist so that typists can insert random characters that look similar to what they actually meant to type. This has a nice income-generating effect for developers who know how to handle incorrectly encoded data. I make good money fixing data processing pipelines written to expect utf-8 only to be given something else.
Yup, ASCII was a multi-byte character set, using overstrike with BS (backspace). Little-known fact, that. There's still a holdover of this in terminal apps, which use this for underscoring and bolding.
I'm surprised it wasn't mentioned, but they were also used for text entry in some text editing applications.
For example, one could type ë by entering ¨ then following with e. The ¨ would be displayed at the position where the combined character would be, while waiting for the second character to be entered. Once the second character is entered, the display would be updated with the correct combined character.
I can't claim I know how they relate exactly to the accented characters being
encoded in character sets but they seem to be at least historically an influence. Pressing a dead key which doesn't advance the cursor and then overwrite the basic character over it is certainly faster than using backspace (also cheaper if you think about character pricing).
That the ECMA specs only talk about using BACKSPACE is surprising. At least those OS I used only supported the dead key approach but of course that was decades after the specs were written.
77 comments
[ 2.4 ms ] story [ 53.0 ms ] threadUltimately, many "character sets" combined printable characters, cursor/head control, and record boundaries into one serialized byte stream that could be used by terminals, printers, programs for all sorts of purposes.
Any such solution will always be broken, proof is left as an exercise for the reader. Hint: use Ogden's lemma.
So if you decide not to support "strings", just "text", then those separators Just Work.
But if I was writing a similar standard today and someone was pressuring me that I really had to make some kind of mechanism for nesting, I'd use something that transforms the field to remove the need for escaping, in a big and obvious way. Probably base64. That prevents most of the implementation issues you see with CSV.
CSV needs escaping, so don't do CSV.
This difference carries over to C, where NULL got the job done for string termination pretty darn well, even if there were strong critiques to be weighed against it.
From a purely linguistic standpoint, the choice is sound. Obviously it's objectively a disaster but for unrelated reasons.
They can, only some functions stop at the first zero:
Consider the following program:
> They can, only some functions stop at the first zero:
> char s[] = "gogo\x00gogo";
And in what human language is that `\x00` a valid character? What rune/glyph/symbol represents it?
Nothing, you say? Well, looks like a good choice to me.
I think a big reason they never took off is also that there's no visual representation of them, or input method for them.
I'd be happy to use them instead of CSV if I could edit records in Notepad or TextEdit the way I can with commas and quoted strings. See the separators, type them.
But of course once you do that, somebody now wants to insert a list of five values separated by unit or record separators into a field. They'd just be common ASCII characters along with tab, CR, LF, etc, and need to be escaped the same.
There's no escaping escaping...
> There's no escaping escaping...
This is the truth.
And in practice I do not expect CSV with embedded nulls to work properly, so there's already precedent to reject certain characters entirely in a CSV-like format.
CSV is more broken than you imagine. Try opening an American CSV file with Excel in Germany or France. You’ll discover that it doesn’t parse, and the reason is that the ‘C’ in CSV apparently stands for “could be a semicolon too” because comma is the decimal separator in these countries and therefore must be available for use inside values.
- Make sure to write your own parser! The format is so simple!
- Is the first line a header or not? The RFC says you should look at the MIME type to tell. You can't make this up.
- Is the line terminator "\r\n", "\n" or "\r"?
- What happens if the number of records for a line is incorrect? Well, that SHOULD'nt happen...
- Make sure to involve Excel, nothing ever went wrong feeding data to Excel.
- Are end-of-line spaces relevant or not? Should we trim them?
There is no defending of CSV, it's a broken format and it's broken beyond repair. The point is that the separator is immaterial. Replacing commas with something else gains us exactly nothing, and results in an identically broken format.
For one, I think you'd almost certainly use Record Separator, and \r and \n would be normal text characters inside of a field. And no trimming spaces.
And the behavior upon encountering corruption isn't really up to the spec, is it?
Headers might be more trouble than they're worth, but if they're in I'd say they need to be mandatory. One code path.
You'll find yourself with a much less broken format.
Always the seemingly simple solutions have most issues in real world... At some level still trying to edit stuff by hand might be where we have gone wrong for long time. Why don't we have any sensible agreed structured format for which tools would be on all and every device...
> ASCII separators are equally as broken. How do you represent a string containing the record separator in a record-separated file?
ASCII separators may not be perfect, but they're certainly less broken than CSV. With CSV, you have the pervasive problem of how to represent a common, everyday character in the file. With ASCII separators, you only have the problem of representing an unusual, untypeable character, which is much less troublesome.
It's the temptation to just concatenate or split based on commas, or to hack a broken implementation of quotes on top, that causes so many issues.
There are two main options for that. Make it so that poorly parsing with self-written code is still very likely to do the right thing, or make the parsing hard enough that people go get a library.
Printers were fare more common than hardware terminals for most of the initial history of computing.
Printers are an obvious answer for a different question.
Edit: Its 1983 predecessor that added accented characters was made for the VT220, and it used a buffer of 8 bit character codes, no way to combine. (Some of those terminals could be sent a limited number of custom characters to store in RAM, but that's a very different mechanism as far as I can tell.)
Edit 2: I made an error before, the naked accents were not in the predecessor, they were even newer.
For that, the printer would also need to interpret the characters in the correct character set.
In 1985 "OS Support" meant picking up the phone and talking to the vendor directly.
The answer is that the character sets were designed with specific hardware in mind and often with backwards compatibility.
The accented characters were in use for typewriter and printer with a typewriter like interface or even hardware.
Someone in 1985 who has mostly printers. ASCII dates from the 70’s.
Yes, and ASCII did not have these characters.
If the expanded count doesn't match, a diacritic might be present.
[0] https://learn.microsoft.com/en-us/dotnet/api/system.text.nor...
The asker completely ignores that asking questions about accent marks, like they themselves are doing in that very post, would be a lot more annoying without being able to write said accent marks.
You can use it for referring to the new Glőrbl character that I invented.
Otherwise you would have to resort to the quite clumsy "the thing on top of àìòèù" to introduce any discussion of the uses of grave accents to people who don't yet know what the phrase "grave accent" means instead of just writing "`", which would be akin to saying that we don't need to write the letter "A" by itself to talk about the letter "A" and could instead say things like "the letter at the beginning of the words Apple and Among". Like...yes...you can do that. But it's less good than the alternative of not doing that.
To some extent the character set was still evolving, for example the Euro sign was not around until decades later and that would need to be bolted together with backspace characters or escape codes, maybe even downloaded characters, with the printer specific manual (Epson) studied at great length.
In the DOS era (and before with home micros that were programmed in BASIC) it was quite normal to compose things for the printer that you had no expectation of seeing on screen, not that anyone read much on screen (as everyone had vast piles of paper on their desk).
Until quite recently some POS systems were very much tied in to a very specific printer, at least these character sets were a step forward from hard coding a BASIC program to an exact make and model of printer.
https://en.wikipedia.org/wiki/Ditto_mark
Guessing maybe there’s some keyboards which were missing quote/prime characters but had umlauts?
I was coding forms for tabular data, doing silly things because the printed manual made that too tempting, so any excuse to use a character or escape code sequence was fair game.
This was in an era when handwriting was commonly used so the convention for 'ditto' could be freestyled by me to be middle dots, umlauts or anything CHR(). The convenience of the keyboard, in an era before the PC was not really there.
New? Some computers of the 1980s could already do this. At least the 16 bit home computers had bitmap drawn characters on the screen.
Edit: Looks like somebody doesn't believe that computers in the 80s had such a thing.
> On the Amiga, rendering text is similar to rendering lines and shapes. The Amiga graphics library provides text functions based around the RastPort structure, which makes it easy to intermix graphics and text.
> In order to render text, the Amiga needs to have a graphical representation for each symbol or text character. These individual images are known as glyphs. The Amiga gets each glyph from a font in the system font list. At present, the fonts in the system list contain a bitmap of a specific point size for all the characters and symbols of the font.
https://wiki.amigaos.net/wiki/Graphics_Library_and_Text
At least 40 years is not very new.
For example, one could type ë by entering ¨ then following with e. The ¨ would be displayed at the position where the combined character would be, while waiting for the second character to be entered. Once the second character is entered, the display would be updated with the correct combined character.
Otherwise HYVÄÄ YÖTÄ was HYV{{ Y|T{, which was only little miserable.
But if you changed the ROM into Swedish ROM, {a|b} become äaöbå, which was basically unreadable.
https://en.wikipedia.org/wiki/Dead_key
I can't claim I know how they relate exactly to the accented characters being encoded in character sets but they seem to be at least historically an influence. Pressing a dead key which doesn't advance the cursor and then overwrite the basic character over it is certainly faster than using backspace (also cheaper if you think about character pricing).
That the ECMA specs only talk about using BACKSPACE is surprising. At least those OS I used only supported the dead key approach but of course that was decades after the specs were written.