This is a good PEP. As f-strings were introduced I switched everything except the simplest things to it.
Especially allowing backslash and arbitrary quotes inside the expressions is a welcome and useful change, as this makes f-strings work more as expected without having to remember a lot of special edge cases and work arounds.
Python f-strings are modelled after `string.format`, which is inspired by C# and introduced relatively later. Such interpolated strings would have been much more inconvenient for the older printf-like syntax.
> Never really understood why not all strings were f-strings in python.
Sometimes you don't want interpolation, and it seems more reasonable to default to a version without such a feature than to default to a version with it and have an alternate syntax for non-interpolated strings. Or, at least, that makes sense to me, but I have no knowledge of their motivations.
Be sude that would make it hell to write strings with literal {}.
I like having different types of strings with different sets of magic characters. Even having identical `'` and `"` strings just to avoid backslashes in a few cases is great.
Yea, Swift really shows how it should be done. One string literal type that can do everything f-strings, r-strings, and """-strings can do and much more.
Logging modules mostly, especially if you have objects that can take a while to serialize to a string, can benefit from the "old" format string approaches. (Percent formatting is for some reason favored, although there's no real good reason to not just use the .format() approach outside of the standard logging library being a dinosaur.)
For conditional formatting (where you don't know if the formatting is needed or the contents of what you're formatting into the string but you know what you want to format), f-strings also lose their use pretty much immediately.
They're great for a quick inline format but less useful for things like string templates you want to use over and over.
Agreed about conditional formatting in current practice, though I wonder how much of it is historical momentum and how Python could have developed if it started with f-strings only. One could use simple functions for replacing string templates if f-strings were the only option.
f-strings are also the fastest way to format a string in Python from what I've seen.
$ python3.12 -m timeit 'f"Welcome to {2 ** 3}."'
5000000 loops, best of 5: 69.8 nsec per loop
$ python3.12 -m timeit '"Welcome to {}.".format(2 ** 3)'
2000000 loops, best of 5: 112 nsec per loop
$ python3.12 -m timeit '"Welcome to %d." % 2 ** 3'
5000000 loops, best of 5: 82.2 nsec per loop
$ python3.12 -m timeit '"Welcome to " + str(2 ** 3) + "."'
5000000 loops, best of 5: 77.5 nsec per loop
TIL. str.format and % parse the format string at runtime, so it's expected that they're slower. But I thought the f-string desugars to something similar to the last one. Turns out it's not, it compiles to specialized opcodes and avoids function calls:
I used to be skeptical, but the great David Beazley convinced me that f-strings rocked because they were faster and better, in his talk "The Fun of Reinvention (Screencast)".
The point he made was to embrace the new features of Python, and put old Python out of its misery, instead of wasting your time with backwards compatibility.
Just call whatever you're doing in the latest version of Python a "prototype", then ship it.
At 10:20 he talks about things you can do to put old Python out of its misery (dead parrot), including f-strings and ordered dictionaries.
For your information, this has been already added to Python 3.12 released last year. (PEP should really highlight a `Python-Version` field for accepted proposals if the target version has been already released.)
"Highlight" in the sense of making it more pronounced in some way (e.g. background color, a link to the release notes etc). Also it takes time from the acceptance to the general availability of given PEP, and I think it shouldn't be highlighted for that period.
Really there needs another state beyond “Accepted”. “Implemented in Python 3.x” would be a lot more clear that something is not only accepted but actually available.
Also one better: how to actually use it. There can be differences between the PEP and its exact implementation.
I wonder sometimes if the very notion of a string concatenation operator was a detour in the wrong direction of PL design back in the days, just because they didn't come up with the idea of strings with embedded expressions first. If you have those, then concat(a,b) is just "{a}{b}", making it very clear that operands and result are all strings, and dodging the whole issue of overloading + in a non-numeric and non-commutative context (or else coming up with a different operator).
Good point, and I think a syntax played a great role here. Before C#-style formatting syntax, many if not most formatting syntaxes were signaled by a single character (`%` in C printf, `$` in sh, `~` in Common Lisp and so on), probably because it was easy to implement with a simple scanner. Those were obviously difficult to extend, because the last character is implicit and not only embedded expressions but pretty much every additional syntax had to be sandwiched between the signaling character and other existing syntaxes [1]. Grouping characters are much more flexible in comparison and it should have been a natural extension to allow any expression inside.
[1] Some did recognize this difficulty and introduced an additional pair of grouping characters (e.g. `%|asdf|` or `${asdf}`). These were however rare, and those that ultimately allowed an arbitrary expression were even rarer.
Being semantically equivalent is not sufficient, because performance matters - concat(a,b) is just "{a}{b}" if and only if a compiler is sufficiently smart to reduce it to the minimum of operations needed, avoiding having to parse the format string every time that line is invoked, avoiding having apply runtime introspection to find out mapping between variable name "a" and the location of that string, etc, etc - without that, it makes all sense for there to be an explicit concatenation and programmers choosing to use that instead of a more general format string.
Semantically, these say the same thing, so all of those concerns exist for “concat(a, b)” too. Avoiding having to parse the concat every time the line is invoked, avoiding having to apply runtime introspection to find out mapping between the variables and their locations, etc.
> because performance matters - concat(a,b) is just "{a}{b}" if and only if a compiler is sufficiently smart to reduce it to the minimum of operations needed
If I remember correctly these kinds of format strings have to be resolved by the compiler since they refer to variables in the source by name, which is information most languages do not keep around at runtime and a security nightmare if done dynamically. Also performance sensitive language like C aren't exactly known for a fast strcat, it is usually left as an exercise to the user to avoid catastrophic runtime behavior caused by the eternal search for nul.
I suppose format strings can be transformed into a series of concatenation function calls. Or a single n-arity call.
(format "{a} {b} cde {f}")
Could be a macro that returns:
(concat a " " b " cde " f)
The non-trivial part seems to be replacing the original format call site with the compiled version in the code. When I try to think it through, I conclude that everything should be a macro that outputs a compiled program. Just seems wrong in a dynamic language but it seems people do it all the time:
It is really not a problem for even a very simple compiler to rewrite "{a}{b}" into a simple function call.
I'm not sure what you mean by "runtime introspection" in this context, because literals with embedded expressions don't require any runtime parsing or lookup - such a literal is parsed completely at compile-time, and in languages with lexical scoping, all variables etc are resolved then. It really is just syntactic sugar for concatenating; it's not at all like runtime templates.
String interpolation is not the same as C-style format strings.
Format strings live in userland as part of the standard library and have the many drawbacks you note.
Proper string interpolation is part of the language grammar itself, and the bits between the braces are usually full-fledged expressions that resolve variable names the same way that every other expression does. Since it's a language construct, it's trivial for it to just compile down to a series of concat(a, b) operations.
It sounds like Python had a half-complete string interpolation mechanism until this PEP, so I can understand why there'd be confusion on this point, but that is a problem with Python's implementation, not something that is an inherent risk with string interpolation. In a new language it is easier to implement string interpolation properly than to introduce a half measure like Python's.
From a practical standpoint you have a point. To me string interpolation always seems like the ugly hack compared to building the result with language operators [1].
In the end interpolation is either dumb and insufficient for most advanced use cases or it is a complex mini language within the real language with tons of corner cases. Doing everything with language facilities looks much more elegant to me.
[1] The various forms of the concatenation operator (+, ., ||, &, :) are most common but not unique. Perl has a sort of scalar string multiplication where `"*" x 80` turns into 80 asterisks.
But string interpolation like that is, effectively, a language operator. Which even in the simplest case (no mini language to format values) is exactly as expressive as concat operator.
Python just lets you multiply strings, so you can do ("foo" * 80) etc. Which of course has all the same problems as overloading + for string ops. Perl had a better idea there, but that in turn is kinda forced by the design that implicitly converts strings to numbers and vice versa - you really, really don't want to be in a situation where 1+2, "1"+2, 1+"2", and "1"+"2" are all valid but inconsistent (looking at you, JS...).
I think you're confusing string interpolation with C-style format strings.
Format strings are what you describe: either too weak or a completely insane sub-language.
The kind of string interpolation that OP is talking about is different in that what is in between the braces is just an expression and can usually be an arbitrary expression. It doesn't need to be a complex mini language because it just uses the same expression language that you're using everywhere else, with the only additional rule being that your expression must return a string (or in some languages something that can be coerced to a string).
From reading this PEP, it sounds like Python's original implementation was a bizarre half measure—it was almost like proper string interpolation but had a bunch of restrictions that showed that they stopped short of changing the parser to just go into expression mode in between the braces.
It's definitely more visually intuitive for someone new to the language. But Python has used asterisk for unpacking lists in other places, so the first time I saw it done for array spreading it made perfect sense.
Agreed. This is one of those cases where "there should be one -- and preferably only one -- obvious way to do it" ought to apply. Of course, there are practical backwards compatibility reasons to keep + working as it does.
But overloading + for lists in general is a footgun anyway, especially in a dynamically typed language where the actual behavior is determined at runtime. And Python went one step further and also defined += for lists, and did it in a way that is even worse, because this:
xs += [1]
does not behave the same as this:
xs = xs + [1]
(the latter creates a new list and assigns the reference to it to xs; the former mutates the list in-place without changing the value of the variable)
Then on top of that you have scoping issues, too, because += is considered an assignment operator, and thus variable on the left side of it is considered a local unless you have a "global" declaration. So if you do this:
xs = []
def foo(): xs += [1]
it doesn't work as expected, because xs inside foo is a new local that shadows the global xs - and it fails at runtime because local xs is not initialized. You have to either do "global xs", or else write xs.append(1) in this context. At which point you might as well just use .append() everywhere, since at least it works consistently.
All in all, it's a good illustration of what happens when people get too "clever" with syntactic sugar when designing a PL.
i think this was probably doable with some form of tuple magic in one form or another, but i agree. now if only they would add the spread operators to the ‘operator’ package… (don’t ask why— the heart wants what it wants…)
I don't think any sane person would use mutable strings considering the benefits you get by having an immutable string type by default. I have always said that if I were to make my own scheme (my language of choice) I the mutability of strings (and pairs) is one of the first things I would change.
Or were you thinking of anything else? I am not saying that a mutable string type should not be provided, just that it should not be the default one.
> Arbitrary nesting of expressions without expansion of escape sequences is available in many other languages that employ a string interpolation method that uses expressions instead of just variable names. Some examples:
basically Swift but { instead of ( as { is 'more decorative' so a better indicator that something special is going on. \ is already the escape character for such thing as \n and friends, so no extra escaping needed for e.g. ${
I just don't understand why everyone loves f-string so much. I do understand they might be better than + op? Perhaps. I've always liked .format method. You separate your template from your input data, and you can choose when the interpolation or execution takes place. You can re-use your template. It's a few more letters to type, but does not require complicated grammar extensions.
I would continue to use .format, but all my teammates are converting everything to fstrings, and I think my cognitive load is much higher because of it.
Most of my use cases for string interpolation doesn't involve the concept of templates, so for me embedding the "data" inside the "template" is easier to read most of the time. Perhaps it's because the data and the "slots inside the template" are colocated? I had prefered keywords when `str.format` expression gets long anyway, so I guess I might not be the best person to judge.
I understand this. Some part of me also resisted them until I worked out that, if used consistently, they _entirely_ eliminate format string bugs (of the "a user-supplied string contains string formatting characters" variety). And then I was a convert!
48 comments
[ 8.6 ms ] story [ 302 ms ] threadEspecially allowing backslash and arbitrary quotes inside the expressions is a welcome and useful change, as this makes f-strings work more as expected without having to remember a lot of special edge cases and work arounds.
Never really understood why not all strings were f-strings in python.
Now? Maybe because another breaking change in strings is the worst nightmare of the current generation of Python developers.
Sometimes you don't want interpolation, and it seems more reasonable to default to a version without such a feature than to default to a version with it and have an alternate syntax for non-interpolated strings. Or, at least, that makes sense to me, but I have no knowledge of their motivations.
If Python were designed from scratch today, I suspect it'd do something similar to JS.
I like having different types of strings with different sets of magic characters. Even having identical `'` and `"` strings just to avoid backslashes in a few cases is great.
For conditional formatting (where you don't know if the formatting is needed or the contents of what you're formatting into the string but you know what you want to format), f-strings also lose their use pretty much immediately.
They're great for a quick inline format but less useful for things like string templates you want to use over and over.
https://bugs.python.org/issue25483
https://bugs.python.org/issue27078
The point he made was to embrace the new features of Python, and put old Python out of its misery, instead of wasting your time with backwards compatibility.
Just call whatever you're doing in the latest version of Python a "prototype", then ship it.
At 10:20 he talks about things you can do to put old Python out of its misery (dead parrot), including f-strings and ordered dictionaries.
The Fun of Reinvention (Screencast):
https://www.youtube.com/watch?v=js_0wjzuMfc
Invited Keynote Talk from PyCon Israel, June 12, 2017. I cause trouble and build a framework using all sorts of new Python 3.6+ features.
I've got some great news: https://cdn.zappy.app/339d2e16228a195e5149cdcf05edd572.png
Also one better: how to actually use it. There can be differences between the PEP and its exact implementation.
[1] Some did recognize this difficulty and introduced an additional pair of grouping characters (e.g. `%|asdf|` or `${asdf}`). These were however rare, and those that ultimately allowed an arbitrary expression were even rarer.
If I remember correctly these kinds of format strings have to be resolved by the compiler since they refer to variables in the source by name, which is information most languages do not keep around at runtime and a security nightmare if done dynamically. Also performance sensitive language like C aren't exactly known for a fast strcat, it is usually left as an exercise to the user to avoid catastrophic runtime behavior caused by the eternal search for nul.
https://news.ycombinator.com/item?id=39240528
Partially evaluated programs.
I'm not sure what you mean by "runtime introspection" in this context, because literals with embedded expressions don't require any runtime parsing or lookup - such a literal is parsed completely at compile-time, and in languages with lexical scoping, all variables etc are resolved then. It really is just syntactic sugar for concatenating; it's not at all like runtime templates.
Format strings live in userland as part of the standard library and have the many drawbacks you note.
Proper string interpolation is part of the language grammar itself, and the bits between the braces are usually full-fledged expressions that resolve variable names the same way that every other expression does. Since it's a language construct, it's trivial for it to just compile down to a series of concat(a, b) operations.
It sounds like Python had a half-complete string interpolation mechanism until this PEP, so I can understand why there'd be confusion on this point, but that is a problem with Python's implementation, not something that is an inherent risk with string interpolation. In a new language it is easier to implement string interpolation properly than to introduce a half measure like Python's.
In the end interpolation is either dumb and insufficient for most advanced use cases or it is a complex mini language within the real language with tons of corner cases. Doing everything with language facilities looks much more elegant to me.
[1] The various forms of the concatenation operator (+, ., ||, &, :) are most common but not unique. Perl has a sort of scalar string multiplication where `"*" x 80` turns into 80 asterisks.
Python just lets you multiply strings, so you can do ("foo" * 80) etc. Which of course has all the same problems as overloading + for string ops. Perl had a better idea there, but that in turn is kinda forced by the design that implicitly converts strings to numbers and vice versa - you really, really don't want to be in a situation where 1+2, "1"+2, 1+"2", and "1"+"2" are all valid but inconsistent (looking at you, JS...).
Format strings are what you describe: either too weak or a completely insane sub-language.
The kind of string interpolation that OP is talking about is different in that what is in between the braces is just an expression and can usually be an arbitrary expression. It doesn't need to be a complex mini language because it just uses the same expression language that you're using everywhere else, with the only additional rule being that your expression must return a string (or in some languages something that can be coerced to a string).
From reading this PEP, it sounds like Python's original implementation was a bizarre half measure—it was almost like proper string interpolation but had a bunch of restrictions that showed that they stopped short of changing the parser to just go into expression mode in between the braces.
But overloading + for lists in general is a footgun anyway, especially in a dynamically typed language where the actual behavior is determined at runtime. And Python went one step further and also defined += for lists, and did it in a way that is even worse, because this:
does not behave the same as this: (the latter creates a new list and assigns the reference to it to xs; the former mutates the list in-place without changing the value of the variable)Then on top of that you have scoping issues, too, because += is considered an assignment operator, and thus variable on the left side of it is considered a local unless you have a "global" declaration. So if you do this:
it doesn't work as expected, because xs inside foo is a new local that shadows the global xs - and it fails at runtime because local xs is not initialized. You have to either do "global xs", or else write xs.append(1) in this context. At which point you might as well just use .append() everywhere, since at least it works consistently.All in all, it's a good illustration of what happens when people get too "clever" with syntactic sugar when designing a PL.
Or were you thinking of anything else? I am not saying that a mutable string type should not be provided, just that it should not be the default one.
I would continue to use .format, but all my teammates are converting everything to fstrings, and I think my cognitive load is much higher because of it.