I think this and pep 498 are a pretty horrible proposals and does not fit well with the rest of the language and it's spirit.
1. Decrements readability. When variables are hidden inside strings, you need to check for presence of a variable in two contexts. In the code and inside the strings. With format and % functions, you will be able to spot the variable in the argument list, which is part of the code itself.
2. Security concerns regarding automatic evaluation of expressions inside the strings.
Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
I agree. For me the automatic "pulling" of local variables from the context feels like "magic", and I think that's an undesirable property for a programming language. Of course, PEP498 has the same issue.
> For me the automatic "pulling" of local variables from the context feels like "magic", and I think that's an undesirable property for a programming language.
Sounds like "Explicit is better than implicit" from the Tao of Python.
That string interpolation was one of my least favorite things about Perl. We tended to have "$"s in strings just because of the usecase, and I was so afraid of that accidentally being interpolated that I never used double quotes, and instead used single quotes. Single quotes in Perl don't interpolate \n, so this "feature" made me write more code rather than less.
For 1, it might be a matter of taste/familiarity. For me, interpolation in Julia makes string formatting easier to understand, at least for short strings. The issue is Julia's interpolation (I hope to god to be wrong about this, if it's wrong, someone correct me) doesn't have format specifiers, which put me in a pickle a few days ago.
For 2, can you expand upon this? Not a challenge, but I'm not able to imagine how this would be a security concern, and this might be a failure of my understanding here.
It is more of a matter of breaking a general principle, which is don't mix executable stuff (expressions) with non-executable stuff (strings) implicitly. This has been a frequent cause of various security issues in the past eg: Sql injection, xss, various remote code execution vulnerabilities.
Strings should be dumb and without any ambiguity. On top of breaking this general principle of not mixing data and code, this pep also want to introduce numerous rules regarding various edge cases, which further makes it harder to reason about it, which increases the chances of slipping a vulnerability. For example, right now, you see a long string, you only need to look at the argument list to see what possible things it can do. Because the string itself is dumb, it cannot lie, and whatever execution that is required to produce that string, happens in the context of code itself. With this pep, this changes. Now a string is not dumb. It is smart. It can do 'stuff' on its own. It can 'hide' things, it can lie and masquerade as something innocent. I think this is bad.
Take an example of PHP. You can embedd php code in what ever content. In fact you can embedd php code in a valid image file in the metadata fields, and pass it as an Image. if you can call the image file as a script, then the embedded php code will execute to do your bidding. This has to be one of the most commonly used technique to hack sites running php. Here is one from last day [1].
Those are some of my reasons for the concerns..I know I am not still giving anything specific. But if we could easily think specific cases about how something could be exploited, we wouldn't be implementing that in the first place. So sometimes we must rely on general principles learned from the past while assessing the issues associated with something. Hence my concerns.
So I wondered about this, but from what I understand (correct me if I'm wrong), the string must explicitly be "f", and thus, a generic string, specifically one created by taking data from input, isn't vulnerable in this way. You could, of course, act on user input inside an f string, but the fault of that lies with the developer not sanitizing inputs, which is an issue with anything.
You do have a good argument concerning not mixing of data and execution as a general principle, though, and I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP implemented, if so, I apologize for misunderstanding). In my mind, this helps with short strings, with simple expressions and makes them more readable--in my opinion, as I said...it's just a convenience. Still, allowing a simple expression allows anything really to be executed, so I guess you have an argument there.
> You could, of course, act on user input inside an f string, but the fault of that lies with the developer not sanitizing inputs, which is an issue with anything....
A good language protects the experienced developers from making a stupid mistake, and forbids the novices from them to trigger a learning response and guide them to better ways.
Example: In python if you use a normal map to implement an algorithm that depends on the elements being ordered in some way, then you will get different results for each run. If I remember correctly, I think python will randomize the order of the map every time it is iterated over. You see, developers could have got away with blaming the user for using an ordinary map when they should have used ordered map. But instead, they implementation map in such a way that it would be impossible to use it incorrectly.
> I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP..
Yes. correct.
>it's just a convenience..
I come from a land filled with such 'conveniences'. it is called PHP. You can compare
var_dump("228" == "0xe4"); -> True
See? a convenience. You can compare hex and decimal strings with out explicit conversion. And what is the result? Countless exploits and security holes that leverage things like these, all that can be possible blamed on the programmer. So was this worth it? Where would you draw the line? How much correctness are you willing to trade off for a minor convenience?
I may be over-reacting. But I think this pep is a step in the wrong direction and should be aggressively corrected.
> If I remember correctly, I think python will randomize the order of the map every time it is iterated over.
No, a normal non-ordered dict in python is guaranteed to have the same iteration order as long as no keys are removed or inserted (i.e., if the only changes are changing the values associated with keys.)
However, the Zen of Python (https://www.python.org/dev/peps/pep-0020/ or run `import this` in a Python interpreter) says "There should be one-- and preferably only one --obvious way to do it.".
> 2. Security concerns regarding automatic evaluation of expressions inside the strings.
How are there security concerns? Its a kind of string literal that is expanded into a calls to builtin formatting functions at compile time, so there is no more security concern with evaluating expressions in it then there is with evaluating expressions in the non-string parts of your code.
> Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
You can use those without an explicit list of variables, its optional to use explicit variables for clarity (though in many cases the natural naming will also make it redundant in simple cases.)
>its optional to use explicit variables for clarity
I don't think it is optional. If it was optional, then the behavior would be to interpolate from local variables by default, and do them selectively if an argument list is provided.
So someone, in the past, thought about this and decided that it is best to always require an explicit list of variables for interpolation.
This pep is trying to undo that decision. Why? What was the reason to always require an explicit list of variables? Didn't they think that it would be convenient to automatically interpolate from locals? What was it that made them decide against it? Isn't that reason still relevant?
Without answering these questions, I think it is not wise to continue with this pep.
I like the general thrust of PEP 498 (`f""` is explicit, unlike Ruby or Bash's magic `"` vs. `'`).
That said, I think it would be much improved if it went the way of Scala and JavaScript (2015+) and made `r`, `u`, `b`, and `f` all interpolation providers. That way libraries could provide interpolators without having to extend the language to create a new one.
For example, in Scala or JavaScript, you can create a string context / template tag for composing JSON strings, resulting in a nice syntax like this (scala-like):
json"""{
"my": $data,
"anArray": $someArray
}"""
The `data` and `someArray` values serialized to JSON by the `json` interpolation provider.
At that point, internationalization is just creating your own interpolation provider and importing it as `i`, which is far less machinery than this PEP requires.
The idea that a translation string can execute Python code via ${expression} is really quite scary. Imagine a user downloading a third-party localization file that ends up being malicious.
Overall, I see there being two major styles for building strings. Since I use C++, I can't do literal interpolation, but I can get close with variadic templates. I hope you guys won't me talking about a statically typed language here, but it should be on topic to string interpolation at least.
//interpolation
print("My name is ", name, " and next year, I will be ", age + 1, " years old.\n");
//arguments
print("My name is {0}, and next year, I will be {1} years old.\n", name, age + 1);
//arguments with hinting (:label is ignored)
print("My name is {0:name}, and next year, I will be {1:age} years old."\n", fetch(), *values + 1);
print(hex(size, 4), " KiB"); //interpolation formatting
print("{0,4x} KiB", size); //argument formatting (more covenient, but more cryptic and limited)
//without the print component:
string s{"My name is: ", name, "\n"}; //interpolation
string s{"My name is: {0}\n", name}; //arguments
The latter is certainly vastly superior for localization. Because in another language, it may be more grammatically correct to reverse the parameters, eg: "来年の歳は{1}と名前は{0}"; which is something C printf doesn't support, shy of unofficial extensions for argument reordering.
But I find that as the string gets longer and longer, the former is much more readable. (You'll have to imagine the syntax highlighting, which helps a lot.) Eg when you have 17 arguments, it can be annoying to count to find the 11th parameter, whereas with interpolation, it's right there where it's used.
Ultimately, I decided on the former being the default, and added a format class for the latter, which is also really handy for building your argument lists somewhere else. Plus the default won't clobber your escape characters (in my case, braces.)
//inline format
print("I am {0}, and I'm {1} years old.\n", format{name, age});
//external format
format f;
f.append(fetch()); //person's age
f.append(*values + 1); //person's name
... //you could fetch the string from a translation table below, too
print("I am {0}, and I'm almost {1} years old.\n", f);
Whenever string::append(format) is invoked, it will use all of the values inside the format array to parse {#} entries in the string. Of course, it's also safe and will simply skip over a reference to {2} in a format array with only one value.
Curious to hear others' opinions on the above, and any suggestions for improvement.
I'm currently working on a heavily internationalized website written in Python 3[1] and as such I've had opportunity to work with a few ways of internationalizing content. My thoughts:
1. Internationalization is, as far as I know, an unsolved problem. This solution doesn't stand out as obviously better than the alternatives. Unsolved problems don't belong at the language level. I'd rather wait until a real solution to this problem is discovered.
2. What happened to explicit is better than implicit? Isn't that a core value of Python? Overriding implementations of builtin functions is all too magical, allowing spooky action at a distance. I'm extremely skeptical of solutions that don't allow me to grep my code for where a function is implemented. `gettext`, at least is clearly calling a function.
19 comments
[ 4.5 ms ] story [ 58.9 ms ] threadhttps://www.reddit.com/r/Python/comments/3gfaji/pep_501_tran...
I think this and pep 498 are a pretty horrible proposals and does not fit well with the rest of the language and it's spirit.
1. Decrements readability. When variables are hidden inside strings, you need to check for presence of a variable in two contexts. In the code and inside the strings. With format and % functions, you will be able to spot the variable in the argument list, which is part of the code itself.
2. Security concerns regarding automatic evaluation of expressions inside the strings.
Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
Sounds like "Explicit is better than implicit" from the Tao of Python.
That is not always true, e.g.:
For 2, can you expand upon this? Not a challenge, but I'm not able to imagine how this would be a security concern, and this might be a failure of my understanding here.
It is more of a matter of breaking a general principle, which is don't mix executable stuff (expressions) with non-executable stuff (strings) implicitly. This has been a frequent cause of various security issues in the past eg: Sql injection, xss, various remote code execution vulnerabilities.
Strings should be dumb and without any ambiguity. On top of breaking this general principle of not mixing data and code, this pep also want to introduce numerous rules regarding various edge cases, which further makes it harder to reason about it, which increases the chances of slipping a vulnerability. For example, right now, you see a long string, you only need to look at the argument list to see what possible things it can do. Because the string itself is dumb, it cannot lie, and whatever execution that is required to produce that string, happens in the context of code itself. With this pep, this changes. Now a string is not dumb. It is smart. It can do 'stuff' on its own. It can 'hide' things, it can lie and masquerade as something innocent. I think this is bad.
Take an example of PHP. You can embedd php code in what ever content. In fact you can embedd php code in a valid image file in the metadata fields, and pass it as an Image. if you can call the image file as a script, then the embedded php code will execute to do your bidding. This has to be one of the most commonly used technique to hack sites running php. Here is one from last day [1].
Those are some of my reasons for the concerns..I know I am not still giving anything specific. But if we could easily think specific cases about how something could be exploited, we wouldn't be implementing that in the first place. So sometimes we must rely on general principles learned from the past while assessing the issues associated with something. Hence my concerns.
[1] https://www.reddit.com/r/PHP/comments/3gq3mh/my_site_was_hac...
You do have a good argument concerning not mixing of data and execution as a general principle, though, and I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP implemented, if so, I apologize for misunderstanding). In my mind, this helps with short strings, with simple expressions and makes them more readable--in my opinion, as I said...it's just a convenience. Still, allowing a simple expression allows anything really to be executed, so I guess you have an argument there.
A good language protects the experienced developers from making a stupid mistake, and forbids the novices from them to trigger a learning response and guide them to better ways.
Example: In python if you use a normal map to implement an algorithm that depends on the elements being ordered in some way, then you will get different results for each run. If I remember correctly, I think python will randomize the order of the map every time it is iterated over. You see, developers could have got away with blaming the user for using an ordinary map when they should have used ordered map. But instead, they implementation map in such a way that it would be impossible to use it incorrectly.
> I think may be you gave the PHP example as how this mixing can lead to certain unintended issues (not necessarily in python with this PEP..
Yes. correct.
>it's just a convenience..
I come from a land filled with such 'conveniences'. it is called PHP. You can compare
See? a convenience. You can compare hex and decimal strings with out explicit conversion. And what is the result? Countless exploits and security holes that leverage things like these, all that can be possible blamed on the programmer. So was this worth it? Where would you draw the line? How much correctness are you willing to trade off for a minor convenience?I may be over-reacting. But I think this pep is a step in the wrong direction and should be aggressively corrected.
No, a normal non-ordered dict in python is guaranteed to have the same iteration order as long as no keys are removed or inserted (i.e., if the only changes are changing the values associated with keys.)
1. `"Value: %i" % value`
2. `"Value: {}".format(value)`
3. `string.Template('Value: $value').substitute(value=value)`
However, the Zen of Python (https://www.python.org/dev/peps/pep-0020/ or run `import this` in a Python interpreter) says "There should be one-- and preferably only one --obvious way to do it.".
How are there security concerns? Its a kind of string literal that is expanded into a calls to builtin formatting functions at compile time, so there is no more security concern with evaluating expressions in it then there is with evaluating expressions in the non-string parts of your code.
> Also, why was format and % made in such a way as to require an explicit list of variables? What was the reasoning behind it? Is that reason no longer relevant?
You can use those without an explicit list of variables, its optional to use explicit variables for clarity (though in many cases the natural naming will also make it redundant in simple cases.)
It is more about breaking a general principle of not mixing executable stuff and non-executable stuff. I have written more about it here.
https://news.ycombinator.com/item?id=10063637
>its optional to use explicit variables for clarity
I don't think it is optional. If it was optional, then the behavior would be to interpolate from local variables by default, and do them selectively if an argument list is provided.
So someone, in the past, thought about this and decided that it is best to always require an explicit list of variables for interpolation.
This pep is trying to undo that decision. Why? What was the reason to always require an explicit list of variables? Didn't they think that it would be convenient to automatically interpolate from locals? What was it that made them decide against it? Isn't that reason still relevant?
Without answering these questions, I think it is not wise to continue with this pep.
That said, I think it would be much improved if it went the way of Scala and JavaScript (2015+) and made `r`, `u`, `b`, and `f` all interpolation providers. That way libraries could provide interpolators without having to extend the language to create a new one.
For example, in Scala or JavaScript, you can create a string context / template tag for composing JSON strings, resulting in a nice syntax like this (scala-like):
The `data` and `someArray` values serialized to JSON by the `json` interpolation provider.At that point, internationalization is just creating your own interpolation provider and importing it as `i`, which is far less machinery than this PEP requires.
Both are equally explicit, though f"" is perhaps marginally more clear because it is somewhat mnemonic.
Overall, I see there being two major styles for building strings. Since I use C++, I can't do literal interpolation, but I can get close with variadic templates. I hope you guys won't me talking about a statically typed language here, but it should be on topic to string interpolation at least.
The latter is certainly vastly superior for localization. Because in another language, it may be more grammatically correct to reverse the parameters, eg: "来年の歳は{1}と名前は{0}"; which is something C printf doesn't support, shy of unofficial extensions for argument reordering.But I find that as the string gets longer and longer, the former is much more readable. (You'll have to imagine the syntax highlighting, which helps a lot.) Eg when you have 17 arguments, it can be annoying to count to find the 11th parameter, whereas with interpolation, it's right there where it's used.
Ultimately, I decided on the former being the default, and added a format class for the latter, which is also really handy for building your argument lists somewhere else. Plus the default won't clobber your escape characters (in my case, braces.)
Whenever string::append(format) is invoked, it will use all of the values inside the format array to parse {#} entries in the string. Of course, it's also safe and will simply skip over a reference to {2} in a format array with only one value.Curious to hear others' opinions on the above, and any suggestions for improvement.
1. Internationalization is, as far as I know, an unsolved problem. This solution doesn't stand out as obviously better than the alternatives. Unsolved problems don't belong at the language level. I'd rather wait until a real solution to this problem is discovered.
2. What happened to explicit is better than implicit? Isn't that a core value of Python? Overriding implementations of builtin functions is all too magical, allowing spooky action at a distance. I'm extremely skeptical of solutions that don't allow me to grep my code for where a function is implemented. `gettext`, at least is clearly calling a function.
[1] http://globalcitizen.org