Ha, that's funny. Just ran into this issue a few days ago. Spent an hour or two trying to figure what was going on, thought I was going crazy. Finally caved and asked an experienced python programmer, and they immediately pointed out that lists are mutable. Goes to show the value of working with experienced programmers.
Ouch, that is one extremely non-user friendly feature going on there. The only reason it works at all is because most people use default for strings/numbers only.
That functionality really should be changed for the next major python release, and have default evaluate each time the function is called - which is how 99.9% of people using it expect it to work.
You don't change behaviour like that once it's been introduced. That will just break existing code in very subtle ways.
There's plenty of code out there that rely on default values being assigned a mutable value and changing it -- even in a major point release -- would be far worse off. Python's been around for 20 years; it's too late to make a change like that.
It IS a "wart" in that you need to be aware of it to know that it's an issue, but once you are, it is an obvious thing to spot.
They made other compatibility-breaking changes in Python 3, though. It'd be interesting to know why this particular feature was kept. I guess from another point of view it's simpler to have everything in the `def` line executed when that line is reached (which is when the function is defined).
> That functionality really should be changed for the next major python release
Make lists immutable you mean? Like tuple()? Edit: Or force re-evaluation on every function call (could be pretty slow)?
The best practice is to just use a tuple (immutable) instead of a list to begin with. You can always do mylist = list(defaulttuple) inside your function first thing if you really want to work with it as a list.
Also, both pylint and pep8, two python static analyzers, will all warn you about any mutable default arguments you provide. Many Python developers have these hooked into their editor or precommit hooks to catch things like this (among others).
Tuples shouldn't be differentiated from lists primarily in terms of mutable vs immutable. They represent different usages semantically, even though their primary functional difference is in their mutability.
Lists are largely for homogeneous collections, whereas tuples are heterogeneous. When you mix them, you abuse what they signal to the reader.
Python doesn't do much for type safety. Either accept the pitfalls of mutability as a tradeoff, or subclass list to make a safer type.
That's actually super insightful (homogeneous vs heterogeneous). I never made that connection but looking back I see it everywhere in Python code, and even did it unknowingly myself.
Why didn't you use an array (from array module in stdlib) for homogeneous collection then? Python is dynamically typed, which means that, by default, we assume different types everywhere (heterogeneity). If you want to express homogeneity just do so explicitly.
No, spamizbad is correct that wahnfrieden's comment was insightful. List and tuple have totally different semantics; they also happen to differ in mutability which just confuses the issue unfortunately.
I'm not that sure about "totally different" semantics; in Python lists and tuples happen to share quite a lot of public interface. As they both are heterogeneous by default (in Python), I disagree with this being their biggest difference. It's not a case in OCaml or Erlang, where lists and tuples differ significantly in both interface and complexity of basic actions. For example, what is a computational complexity of getting nth element of tuple and list in Python, and why it's exactly the same for both of them? In Erlang, where lists are real lists, getting nth element is O(n) in case of lists and O(1) in case of tuples.
In Python, lists and tuples mean almost exactly the same thing. That's how Python is implemented. Those two things have "totally different" semantics in many other languages, not so in Python. As so, using the one over the other in many cases is purely stylistic matter, not the matter of semantics... unless it's about mutability, of course.
If they're as different as you and grandparent suggest, then please list those differences, and mean in Python. As I said, in OCaml Lists and tuples are completely different types, starting with the fact that tuples are not iterables (!). The same is true in Erlang, where you need to explicitly convert a tuple to list to get all the elements of a tuple. Both list and tuple are iterable in Python, however.
I just see much more similarities between lists and tuples in Python than differences, to the point that I wouldn't mind calling tuples an immutable_list or frozenlist, or something like that.
It's a perfectly accurate statement. If a tuple is homogeneous, it's only because all of its slots happen to be for objects of the same type (e.g. a row of a database table all of whose columns are integers, or the arguments of a function which takes 3 integers as input). In contrast, homogeneous lists are the norm (other than mixing None with non-None values). When was the last time you needed a heterogeneous list?
The quote comes from Guido van Rossum almost verbatim so that may be a good place to start if you want to dig deeper.
As I've also seen, a good rephrasing is that lists have order whereas tuples have structure. Yet another mnemonic I've heard that seems apt is that tuples are appropriate where a named-tuple could also conceivably be used.
That's crazy-interesting and seems like a very good rule of thumb. I've been using Python for a very long time and had never heard this. Why is it not in the PEP?!
The quote is "Tuples are for heterogeneous data, list are for homogeneous data. Tuples are not read-only lists." Guido van Rossum March 12, 2003 in python-dev.
However, don’t treat Guido’s statement as a mandate!
"Yes, taking Guido too seriously can have that effect on people <1.1 wink> The trick is in knowing when Guido is proscribing dogmatic law, and when he is describing his intent when he designed feature(tte)s of Python, or his point of view at this moment, or how he wished people would see it, or reacting to what he had for lunch. He's just human, you know, and no more infallible than the best of us." Thomas Wouters March 12, 2003 in python-dev
For example, consider object.__class__.__bases__. That's a tuple of homogenous data. It's not of a fixed length, and the base class at a given position has no intrinsic name, other than "first", "second", etc.
Do you think the use of tuple here is a bug? Should it be a list instead?
Example of some typical tuples:
(1,'A')
(2,'B')
(3,'C')
In Python, of course lists can be heterogenous, so you could have a list like [1, 'A', 2, 'B', 3, 'C'] but it would be an atypical/un-idiomatic use case. A list of tuples or possibly a dict would make more sense for that.
In a dynamically typed language with a type heirarchy with a common root, it is impossible to distinguish between a heterogenous list whose entities are instances of a variety of specific types and a homogenous list whose members are all members of one of the common supertypes of the specific types of the objects.
With a language that uses duck typing (so that the meaningful "types" of objects are often not their explicit classes but the implicit -- both in definition and in association with particular objects -- interfaces expected by the code which accesses them), this ambiguity is even more pronounced.
This is true, and I'm sure Guido is fully aware of it, but it's also not entirely relevant to what he was saying. Regardless of inheritance, interfaces, and duck typing, the explicit classes of the objects still do matter.
Yes, the list of [1,'A',2,'B',3,'C'] is really after all just a list of [Object, Object, Object, Object, Object, Object]. And the list itself is also just an Object.
But the explicit classes of the elements of the list are different ([int,str,int,str,int,str]), and often that actually does matter for the use case, because int and str have different interfaces, and "adding" ints means something different than "adding" strs for example. It really just depends on what you're going to do with that data, and I think Guido was also talking in terms of the idiomatic use cases for the different data structures.
> This is true, and I'm sure Guido is fully aware of it
Well, yeah. I'd be kind of stunned if he wasn't. I was making a note qualifying the examples that were given in isolation, not questioning the basic premise.
> Regardless of inheritance, interfaces, and duck typing, the explicit classes of the objects still do matter.
Whether or not that's true depends on the use case; the really relevant sense of homogenous or heterogenous that distinguishes the idiomatic use of tuples vs. lists is contextual, and is more about the interfaces that the consuming code expects the members to provide than the explicit classes per se.
There's nothing unidiomatic in Python about using a list for a series of items of different explicit classes that are going to have the same operations (supported by a common interface they all share) performed on them.
> Tuples shouldn't be differentiated from lists primarily in terms of mutable vs immutable.
This. In particular, tuples are for collections whose length is fixed and meaningful, and for which you could give names to each field. Lists are for collections whose length doesn't correspond to anything in particular and which you could imagine growing and shrinking.
As an aside, django makes this mistake and confuses people: using tuples where semantically they should be lists, presumably out of a desire for immutability.
A class's __bases__ is a tuple of base classes. The length is not fixed, and there's no special name for [i] other than (i+1)-th; only its ordering is important.
Do believe that the use of a tuple here is a bug, like you do for Django?
The __bases__ can be re-assigned. Assignment to an instance is guaranteed atomic in Python. Some of the list operations are not. Thread-safe guarantees might be another reason for restricting list's mutability.
You're right, at a certain point functionality and other characteristics make all the difference. It's just a generality for higher-level code I suppose.
No, the list issue is the smaller problem here, imo. The big problem is the example given at the bottom of the page.
def print_now(now=time.time()):
print now
The expectation here is that it will give the current time as the default, but the expectation is broken by the very unexpected behavior of the language. That's the kind of thing you want to avoid if at all possible. I see no reason why anybody would not want the default to evaluate every time - tiny bit of speed seems like a very premature optimization here.
If you're worried about speed, you could make a class/instance variable, and then set the default to that variable. That's what 'default' is actually doing in the current implementation - and it sounds unexpected in the extreme.
It's only unexpected if you didn't learn when Python instantiates default variables and how it passes them. Or put another way, if you treat Python like some other language you learned, apparently not realizing it is a different language.
Now personally I hardly every use default values, but to me, when I see this -
> I see no reason why anybody would not want the default to evaluate every time
> Or put another way, if you treat Python like some other language you learned, apparently not realizing it is a different language.
No, it's not "some other language", it's every other language. Why? Because the behavior being described isn't really a default. It's more like a static variable.
I want to create a function that tells me the temperature - but also lets me know the temperature yesterday. Clearly, the best way for this function to work is to have a default value of the current time. Python default values simply won't work here, as the time will be when I first start the program up and not when I call the function tomorrow for the current temperature.
Why would I ever want to have the value from when I first launched the program?! Your argument that "this is how python does it" is not a very good one. Why should I have put it in the method? I want a default value! The reason for having the default value on a function in the first place is to allow me to make functions work with a default value. If this doesn't work, the feature is useless. As I said in the first post, the only reason the feature is usable at all is because, luckily, people nearly always use only numbers or strings for default values.
EDIT: Plus your observation that you rarely use default values is probably for a good reason: they don't really work. Maybe if they worked, you'd use them?
> EDIT: Plus your observation that you rarely use default values is probably for a good reason: they don't really work. Maybe if they worked, you'd use them?
Honestly, no, it's probably because I don't write methods to be generic enough.
Ignoring that,
>argument that "this is how python does it" is not a very good one
Except that if you're writing Python you have to work by Pythons rules, so yes it is.
> The reason for having the default value on a function in the first place is to allow me to make functions work with a default value
Then in python, make sure you are using datatypes that are passed by value, not by reference. That is the core of the problem and it bites everyone at some point in every language because the rules of passing by reference or value are different in a great number of languages.
def foo(name=[]):
In python, is not saying if I didn't pass something make name an empty list, it says 'if name is not defined in this scope, create an empty list.' But with Pythons rules a name[] was created in that scope when the function was defined. It was the programmers mistake for assuming otherwise.
Once again, we are writing Python here, Pythons rules are the only ones that matter. If you are getting unexpected results because you're not following the rules, the issue is yours. Python is not called C#, or Java, or C or Ruby or whatever because it is not C# or Java or whatever.
time.time() is a specific case, but let's say I had very_expensive_function_to_call() as a default value. I would want that function to be called once at parse time, and the value to be kept going forward, instead of calling the function each time.
Both ways have their advantages, python (and the python implementations) just fell one way instead of another.
The accepted explanation seems wrong to me. Whether or not functions are first class objects or not, the behavior of def is a matter of design choice and in this case the wrong choice has been made. All the rationales are broken -- they're uncommon use cases and perfectly easy to implement without the behavior, i like Python but this is a wart.
Wow I have been using Python for years and didn't realize default values were only created when defined. I don't think I have ever been bitten by it, but it would have been one helluva bug to track down.
I've been bitten by that exactly once. In order to track it down, I had to tear out pieces of my program to issolate where it is, to the point where I was working with a single function that did nothing other than mutate a dict that was a default value and print said value; and code that called it twice.
When I saw the title for this post, my first thought was "hmm, mutable default arguments are pretty tricky, hopefully that's somewhere on the list". I wasn't disappointed.
Also worth mentioning is the closure gotcha: when you create functions in a loop, the functions' context points to the context of the latest iteration of the loop. Other languages might have this issue as well and provide alternatives based on map-style constructs. In Python, you can take advantage of the behaviour described by the OP to make your inner functions have their own context, which is initialized when the function gets created.
The most frustrating thing that bit me when I was first learning python was the error when you pass the wrong number of arguments to a method:
>>> class A(object):
... def method(self, a): pass
...
>>> A().method()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() takes exactly 2 arguments (1 given)
Once you understand more about how method dispatch works it makes sense, but it's really confusing to be told you were doing something you don't think you are.
I've been coding in Python for 8 years now and only this year came up against this behavior. I had a good long laugh after realizing what had just eaten up my previous couple hours.
I actually figured it out by printing out the id of the object being modified in my code, and when seeing it was the same had to really scratch my head. Default args are initialized when the function (or similarly, class) is defined, not when executed.
As a newbie to Python, my biggest issue when coming up to speed on a new project is chasing down the types of arguments to and return values from functions. Previously I was a Java programmer and while I don't miss its verbosity, I do miss how I always knew what types I was working with. Does anybody have any tips that make this easier?
The biggest tip: The Python interactive shell is your friend. Use it to play with things you're not sure of. IPython[1] is a more advanced Python shell you may like. Lots of people also swear by pdb, the Python debugger. Personally, I get less use out of it than others, but YMMV.
But overall, this is a problem involving documentation, design and naming, and your own mindset.
Ideally you would never really be put in the position of not knowing, because it would be clearly documented. Of course, we all know how well that tends to work.
Second, well-designed systems with good naming practices will make it fairly obvious most of the time what sort of arguments are expected, and what you can expect to get back. (This is related to the next point.)
Finally, coming from a "static" language, your brain just isn't trained to perform its own type inference and deal with duck typing. This is mostly a matter of practice, you get better at it. But if you aren't already familiar with various kinds of typing, be sure to do some reading. A better understanding of the differences will help make sense of what you're seeing.
Newbie here with limited Python experience. What would you suggest? I have no projects in mind, just want to learn and have been mashing away at Python for months. What ever it is I make needs to run on a Mac or iDevice - via browser is fine.
It was suggested to me that I get a grasp of basic concepts before going there. I found it fairly easy to make a very very basic application with series of buttons that worked etc via objective c and the interface builder, but had no idea what I was doing when I hit code (which was the whole point of playing around). I have a better understanding of some coding basics now that I have done a small amount of python (Zed Shaw). Maybe it's time I went back to objective c - you've got me thinking... Thanks.
You're going to miss the interpreter. I started programming with python too, and now I'm moving onto Java (for Android), and while static typing is nice, I really miss being able to just try things out in a few seconds.
Working with a GUI framework is very different than web development and CLI.
Yeah, hacking around in ipython is nice, but you don't have to sacrifice all of it for static typing. play.golang.org is fairly quick if you just want to try something.
I would put the unfortunate scoping design above mutable defaults on the list of things that confused me.
Also, I really don't like the decision to make Python 2 disobey the system locale and use ascii as the default encoding. Lots of Python programs out there have broken i18n for no good reason because of that (nobody starts teaching python by telling you to use .decode('utf-8')/.encode('utf-8') when you pipe data, and by the time you realize what's wrong, you often already have code out there.)
That's not a bug, it's just the result of Python variables being references to values, instead of actual values. When you write "[[]]", what you're doing is creating a list that holds a reference to another list.
If you use a variable it's easier to understand:
a = []
arrays = [a] * 5
What you have now is [a, a, a, a, a], so if you modify one of the lists, it's no wonder they all get modified: they're references to the same list.
That's because of the implicit copy() only copying by value 1 level deep?
On the topic of the bug in OP, would a Python newbie expect the similarly confusing behavior caused by interactions between (im)mutability and function arguments being passed as a "reference"?
Using OP's foo():
>>> a = [1,2,3]
>>> foo(a)
[1, 2, 3, 9]
>>> a
[1, 2, 3, 9]
if 'a' were an immutable object like an int, it wouldn't be modified.
>>> def bar(n):
n += 1
print(n)
>>> b = 3
>>> bar(b)
4
>>> b
3
86 comments
[ 3.4 ms ] story [ 212 ms ] threadThat functionality really should be changed for the next major python release, and have default evaluate each time the function is called - which is how 99.9% of people using it expect it to work.
There's plenty of code out there that rely on default values being assigned a mutable value and changing it -- even in a major point release -- would be far worse off. Python's been around for 20 years; it's too late to make a change like that.
It IS a "wart" in that you need to be aware of it to know that it's an issue, but once you are, it is an obvious thing to spot.
Make lists immutable you mean? Like tuple()? Edit: Or force re-evaluation on every function call (could be pretty slow)?
The best practice is to just use a tuple (immutable) instead of a list to begin with. You can always do mylist = list(defaulttuple) inside your function first thing if you really want to work with it as a list.
Also, both pylint and pep8, two python static analyzers, will all warn you about any mutable default arguments you provide. Many Python developers have these hooked into their editor or precommit hooks to catch things like this (among others).
Lists are largely for homogeneous collections, whereas tuples are heterogeneous. When you mix them, you abuse what they signal to the reader.
Python doesn't do much for type safety. Either accept the pitfalls of mutability as a tradeoff, or subclass list to make a safer type.
In Python, lists and tuples mean almost exactly the same thing. That's how Python is implemented. Those two things have "totally different" semantics in many other languages, not so in Python. As so, using the one over the other in many cases is purely stylistic matter, not the matter of semantics... unless it's about mutability, of course.
If they're as different as you and grandparent suggest, then please list those differences, and mean in Python. As I said, in OCaml Lists and tuples are completely different types, starting with the fact that tuples are not iterables (!). The same is true in Erlang, where you need to explicitly convert a tuple to list to get all the elements of a tuple. Both list and tuple are iterable in Python, however.
I just see much more similarities between lists and tuples in Python than differences, to the point that I wouldn't mind calling tuples an immutable_list or frozenlist, or something like that.
What? What makes you think that?
As I've also seen, a good rephrasing is that lists have order whereas tuples have structure. Yet another mnemonic I've heard that seems apt is that tuples are appropriate where a named-tuple could also conceivably be used.
However, don’t treat Guido’s statement as a mandate!
"Yes, taking Guido too seriously can have that effect on people <1.1 wink> The trick is in knowing when Guido is proscribing dogmatic law, and when he is describing his intent when he designed feature(tte)s of Python, or his point of view at this moment, or how he wished people would see it, or reacting to what he had for lunch. He's just human, you know, and no more infallible than the best of us." Thomas Wouters March 12, 2003 in python-dev
For example, consider object.__class__.__bases__. That's a tuple of homogenous data. It's not of a fixed length, and the base class at a given position has no intrinsic name, other than "first", "second", etc.
Do you think the use of tuple here is a bug? Should it be a list instead?
GvR aside, even the documentation on python.org mentions this: http://docs.python.org/2/tutorial/datastructures.html#tuples...
Example of some typical tuples: (1,'A') (2,'B') (3,'C')
In Python, of course lists can be heterogenous, so you could have a list like [1, 'A', 2, 'B', 3, 'C'] but it would be an atypical/un-idiomatic use case. A list of tuples or possibly a dict would make more sense for that.
With a language that uses duck typing (so that the meaningful "types" of objects are often not their explicit classes but the implicit -- both in definition and in association with particular objects -- interfaces expected by the code which accesses them), this ambiguity is even more pronounced.
Yes, the list of [1,'A',2,'B',3,'C'] is really after all just a list of [Object, Object, Object, Object, Object, Object]. And the list itself is also just an Object.
But the explicit classes of the elements of the list are different ([int,str,int,str,int,str]), and often that actually does matter for the use case, because int and str have different interfaces, and "adding" ints means something different than "adding" strs for example. It really just depends on what you're going to do with that data, and I think Guido was also talking in terms of the idiomatic use cases for the different data structures.
Well, yeah. I'd be kind of stunned if he wasn't. I was making a note qualifying the examples that were given in isolation, not questioning the basic premise.
> Regardless of inheritance, interfaces, and duck typing, the explicit classes of the objects still do matter.
Whether or not that's true depends on the use case; the really relevant sense of homogenous or heterogenous that distinguishes the idiomatic use of tuples vs. lists is contextual, and is more about the interfaces that the consuming code expects the members to provide than the explicit classes per se.
There's nothing unidiomatic in Python about using a list for a series of items of different explicit classes that are going to have the same operations (supported by a common interface they all share) performed on them.
This. In particular, tuples are for collections whose length is fixed and meaningful, and for which you could give names to each field. Lists are for collections whose length doesn't correspond to anything in particular and which you could imagine growing and shrinking.
As an aside, django makes this mistake and confuses people: using tuples where semantically they should be lists, presumably out of a desire for immutability.
Do believe that the use of a tuple here is a bug, like you do for Django?
The __bases__ can be re-assigned. Assignment to an instance is guaranteed atomic in Python. Some of the list operations are not. Thread-safe guarantees might be another reason for restricting list's mutability.
It is probably also true (generally speaking) that people tend to use lists when the cardinality is not fixed and tuples otherwise.
No, that's not good advice. Tuples are not just immutable lists. See below.
If you're worried about speed, you could make a class/instance variable, and then set the default to that variable. That's what 'default' is actually doing in the current implementation - and it sounds unexpected in the extreme.
Now personally I hardly every use default values, but to me, when I see this - > I see no reason why anybody would not want the default to evaluate every time
It means you should have put it in the method.
No, it's not "some other language", it's every other language. Why? Because the behavior being described isn't really a default. It's more like a static variable.
Why would I ever want to have the value from when I first launched the program?! Your argument that "this is how python does it" is not a very good one. Why should I have put it in the method? I want a default value! The reason for having the default value on a function in the first place is to allow me to make functions work with a default value. If this doesn't work, the feature is useless. As I said in the first post, the only reason the feature is usable at all is because, luckily, people nearly always use only numbers or strings for default values.
EDIT: Plus your observation that you rarely use default values is probably for a good reason: they don't really work. Maybe if they worked, you'd use them?
Honestly, no, it's probably because I don't write methods to be generic enough.
Ignoring that,
>argument that "this is how python does it" is not a very good one
Except that if you're writing Python you have to work by Pythons rules, so yes it is.
> The reason for having the default value on a function in the first place is to allow me to make functions work with a default value
Then in python, make sure you are using datatypes that are passed by value, not by reference. That is the core of the problem and it bites everyone at some point in every language because the rules of passing by reference or value are different in a great number of languages.
def foo(name=[]):
In python, is not saying if I didn't pass something make name an empty list, it says 'if name is not defined in this scope, create an empty list.' But with Pythons rules a name[] was created in that scope when the function was defined. It was the programmers mistake for assuming otherwise.
Once again, we are writing Python here, Pythons rules are the only ones that matter. If you are getting unexpected results because you're not following the rules, the issue is yours. Python is not called C#, or Java, or C or Ruby or whatever because it is not C# or Java or whatever.
Both ways have their advantages, python (and the python implementations) just fell one way instead of another.
http://stackoverflow.com/questions/1132941/least-astonishmen...
current behavior is totally fine. it is well defined that somelist.append(x) appends x to somelist inplace.
https://news.ycombinator.com/item?id=5998675
I was wrong, and now I know better. Thank you.
Edit:
import mock
def print_now(now=mock.sentinel.now): now = now if now != mock.sentinel.now else time.time()
I actually figured it out by printing out the id of the object being modified in my code, and when seeing it was the same had to really scratch my head. Default args are initialized when the function (or similarly, class) is defined, not when executed.
But overall, this is a problem involving documentation, design and naming, and your own mindset.
Ideally you would never really be put in the position of not knowing, because it would be clearly documented. Of course, we all know how well that tends to work.
Second, well-designed systems with good naming practices will make it fairly obvious most of the time what sort of arguments are expected, and what you can expect to get back. (This is related to the next point.)
Finally, coming from a "static" language, your brain just isn't trained to perform its own type inference and deal with duck typing. This is mostly a matter of practice, you get better at it. But if you aren't already familiar with various kinds of typing, be sure to do some reading. A better understanding of the differences will help make sense of what you're seeing.
[1] http://ipython.org/
Also, I really don't like the decision to make Python 2 disobey the system locale and use ascii as the default encoding. Lots of Python programs out there have broken i18n for no good reason because of that (nobody starts teaching python by telling you to use .decode('utf-8')/.encode('utf-8') when you pipe data, and by the time you realize what's wrong, you often already have code out there.)
If you use a variable it's easier to understand:
What you have now is [a, a, a, a, a], so if you modify one of the lists, it's no wonder they all get modified: they're references to the same list.On the topic of the bug in OP, would a Python newbie expect the similarly confusing behavior caused by interactions between (im)mutability and function arguments being passed as a "reference"?
Using OP's foo():
if 'a' were an immutable object like an int, it wouldn't be modified.def f(x=()): x = list(x) ...
def g(y=(())) y = dict(y) ...
f() is the pattern for lists, g() is for dicts. Note that you can pass in regular lists to f() and regular dicts to g().