No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. And it gets called differently when a list comprehension calls it each step of the loop or when the list function loops completely through and then the list comprehension calls the instances.
Nothing to see here.
> No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created.
Isn't that exactly what GP said? And gave an example how to create that effect explicitly in functional languages, using refs?
It's consistent, but unexpected. Especially if you don't know python well enough. And it leads to ugly effect sometimes, even for experienced pythonistas.
Yes, this bit me last week. I stumble over it about once a year. Recently encountered it in an open source project and filed an issue. I don't know where I fall on the "experienced" spectrum, but I've written a lot of python in the last 10 years.
Naively I would expect that each iteration of a for loop creates a new loop variable (all with the same name, but effectively each in their own scope) and so each closure holds a reference to a variable named number whose value never changes, and there is a distinct one of these number variables per closure .
That is not how variables in Python work. Variable assignment is just name binding. This is clearly mentioned in every Python beginner tutorial I know of.
Python code is a script. Follow the script and you know what Python does (with only a few exceptions, see e.g. below).
About the scope: Python knows in principal only 2 scopes: local and global. If you define a function (or a class) Python creates a new emtpy/filled with the function arguments local scope (system table). That gets filled successively by local variable assignments etc. There happens a little bit of magic: if you use a variable of the next higher scope, it gets copied (!) to the local scope. You can follow that easily by print(locals()).
(You can also use the global scope, but that is another topic).
Functions.partial can also supply captures in that way, and is slightly more obvious (and less likely to generate linter complaints if the kwarg is mutable).
all one needs to understand is that there are names that refer to objects in Python. Nothing is passed by value.
The closure returns whatever object the name refers at the moment. That is all.
def loop():
for i in range(10):
yield lambda j=i: j
`j` refers to the same object (int) `i` refers to at the time of the lambda creation (the desired result). `j` is local to the lambda (each lambda has its own local namespace).
Just `lambda: i` code would result in an object that `i` refers to at the time of calling the lambda (`i` name is not local to the lambda).
eager:
for item in loop():
item()
lazy:
items = []
for item in loop():
items.append(item)
for item in items:
item()
Each next() into loop increments a shared counter so when you print it out as you go it 1, 2, 3 but when you print it out once you've called it a bunch is 9, 9, 9.
It’s a closure oddity. The closure captures a reference to the variable, not the value of the variable at the time of its creation. Given that, the results are entirely predictable.
If we're really going to argue, I'd argue for a 'for' oddity. The for statement could've been defined with a fresh binding per iteration instead of a mutation.
That ship already sailed when Python 3 made comprehension expressions (like list comprehensions) introduce a nested scope. They're even a kind of 'for' too. (Previously, [x for x in y] would clobber any x in the same function. Fixing this meant implicit nested scopes.)
It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.
I get why it behaves like this, but I find it odd that Python claims to have "lexical scoping". This is not lexical (as you read it). This is based on somewhat hidden reference lookups.
That's a design decision, not a law of nature. JS 'for let' lacks this problem; probably the committee would have named it just 'for' if not for legacy.
Not really -- in practice that would imply that ``for`` needs to freeze the value of ``locals`` during every iteration, which doesn't make sense to me.
This is definitely a python gotcha. In fact, it's literally one of the canonical ones [1]. But it's purely the interaction between the way that names/variables work in python, and late-binding closures.
More explicitly: it's best to think of "variables" in python as "names", because in code, you refer to names, but the underlying variable might change. This is also why python is best described neither as call-by-reference nor call-by-value, but rather "call-by-assignment" or "call-by-object": you're passing a reference to a variable bound to a particular name. Unless you're writing C extensions, you never actually have a reference to the variable itself -- instead, to the name.
Meanwhile, closures in python are always late-binding: they are themselves "live", mutable objects, in the sense that they're just a reference to the namespace they're closing, and *not* a copy. So if you mutate that namespace between establishing the closure and making use of it, you'll see the mutated version.
Yes, the best way to understand python closures is that scopes are first class objects in python (accessible via locals()) and closures close over the containing scope itself, not the single variables.
Also worth keeping in mind that, AFAIK, there is one scope per function, which differs from other languages that can have nested scopes inside a function. I think the list comprehension is the only exception in python3, which was changed so that names no longer leak out of it, which was frankly insane in py2.
How does that work in practice? Python stores local variables in the stack frame with a compiler-calculated index. Do you have a growable stack frame? Add stack frames for every loop iteration?
I don't think the Python compiler is to know if a loop variable is closed over and then adjust the code generation. So this change would likely introduce a lot of complexity, or be a pessimisation in the normal case.
I think the simple, although less helpful, semantics are clearly better here. FWIW I don't think this comes up in practice very much. Maybe because callbacks in Python APIs aren't common (and its not due to this).
it comes up often enough for me to be annoying. The defaulted argument works, but it is an ugly hack. It is not obvious to me that the surprising semantics are clearly better, but python has a long history to preferring implementation simplicity to user ergonomics.
See my other reply about how [x for x in y] now works. It'd be a variation on that.
I agree the current semantics are better by the measure of performance of the simplest possible implementation. Though that's not Python's main design goal.
'var' is JavaScript's older variable declaration construct. Variables created this way are live from the beginning of the function that contains them (or globally if there isn't one). So a block with braces (such as you'd use for a for or while loop body) doesn't actually restrict the scope of var `v` below:
console.log(v); // <-- v is a variable here, we can access its value even though it is only declared below
// prints 'undefined'
{
var v = 1;
console.log(v); // prints 1
}
console.log(v); // prints 1
You used to (and might still) see a workaround to recover more restrictive scoping, known as the "IIFE" (Immediately Evaluated Function Expression): (function () { var v; ... code using v here ... })() creates a function (and thus a more restrictive scope for v to live in) and evaluates it once; this is a sort of poor man's block scoping.
`let` and `const` were created to fill this gap. They have block scope and are special-cased in for loops (well, `let` is; you can't reassign to a `const` variable so nontrivial for loops won't work):
console.log(l); // <-- throws ReferenceError: l is not defined
{
// pretend the console.log line above is commented out, so we can reach this line
let l = 1;
console.log(l); // prints 1, as expected
}
console.log(l); // throws ReferenceError: l is not defined
// ^^ l was only in scope for the block above
Fun(?) fact: it's not technically true that const can't be used in for loops:
for (const i = 0; i > 0; ) {
console.log('this is stupid');
}
let ran = false;
for (const i = 0; !ran; ran = true) {
console.log('this is also stupid');
}
Yes. The usual way to do this is to bind them as defaults to an argument, for example:
def loop():
for number in range(10):
def func_w_closure(_num=number):
return _num
yield func_w_closure
This works because default arguments in python are evaluated exactly once, at function definition time. So it's a way of effectively copying the ``number`` out of the closure[1] and into the function definition.
[1] side note, closures in python are always late-binding, which is what causes the behavior in OP
I’ve never thought that leaking this type of implementation detail into the return value (and return type!) was a nice solution. I like the double closure better, and one can shorten it a bit with a lambda.
For those who prefer a functional style, functools.partial can also solve this problem.
(I use Python, and I like a lot of things about Python, but I don’t like its scoping rules at all, nor do I like the way that Python’s closures work. I would use a double lambda.)
It doesn't work because of the way Python's variables are scoped. Your fixed_number variable is still shared across all instances of inner. Python doesn't have any sort of block scoping like you seem to think it has.
The answer is simple. f strings are evaluated at run time. The behaviour is correct. Up until you call the lambda, they are just a function definition.
What's so interesting about the Elephant case? You store the value of i at run time, so it can be accessed later. lambda doesn't have that capability, as it's inheriting i from the block.
> The answer is simple. f strings are evaluated at run time.
You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching.
> The behaviour is correct.
Python doesn't have a standard. Nobody knows what behavior is correct. That's the only behavior Python has, so, again, calling it correct is just not saying anything.
> What's so interesting about the Elephant case?
This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.
You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
That is not correct. Python generally has two "runtimes": one is on import, when all of the module level code is evaluated; and the runtime, when the bodies of various functions and methods are evaluated. The two can be combined, e.g. by importing a module from within a function, but generally they are separate.
> Bytecode is just a form of caching
Not quite. It is perfectly possible to deploy just the bytecode and disable recompiling on import.
> Python doesn't have a standard
Of course it does -- CPython is the reference implementation, so its behaviour is "correct".
> You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching.
You don't understand what runtime in python is. Compilation into bytecode is an implementation detail and not part of runtime. Even when you start a python scripts from a bytecode state, there is a runtime. f-strings are evaluated at runtime. When you declare them within a function, yes they will be parsed, but they will not be run.
> This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.
In your case, you put them in lambdas, which is equivalent to putting them into a function context. The f-string will only be evaluated when lambda/function is called. In your Elephant case the runtime happens when i=n, whereas in lambda case, the runtime is when i=9 (because there is no local copy of the variable/context). Plain and simple. If you don't understand such a simple distinction, I recommend you to start reading the bytecode as it's much clearer at that level.
> You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
It's not an inconsistency, if you understand the principle.
50 comments
[ 7.0 ms ] story [ 348 ms ] threadIn safe languages like ML you would have to create an explicit reference to get that effect, and then it is obvious to the reader.
Python is not built for complex abstractions, but unfortunately it is (ab)used for that, particularly in machine learning.
Isn't that exactly what GP said? And gave an example how to create that effect explicitly in functional languages, using refs?
About the scope: Python knows in principal only 2 scopes: local and global. If you define a function (or a class) Python creates a new emtpy/filled with the function arguments local scope (system table). That gets filled successively by local variable assignments etc. There happens a little bit of magic: if you use a variable of the next higher scope, it gets copied (!) to the local scope. You can follow that easily by print(locals()). (You can also use the global scope, but that is another topic).
Edit: had to clarify some points
Pass a keyword argument with the default value equal to the value you want to close over.
Voila!
The closure returns whatever object the name refers at the moment. That is all.
`j` refers to the same object (int) `i` refers to at the time of the lambda creation (the desired result). `j` is local to the lambda (each lambda has its own local namespace).Just `lambda: i` code would result in an object that `i` refers to at the time of calling the lambda (`i` name is not local to the lambda).
https://docs.python.org/3/faq/programming.html#why-do-lambda...
It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.
That's a design decision, not a law of nature. JS 'for let' lacks this problem; probably the committee would have named it just 'for' if not for legacy.
This is definitely a python gotcha. In fact, it's literally one of the canonical ones [1]. But it's purely the interaction between the way that names/variables work in python, and late-binding closures.
More explicitly: it's best to think of "variables" in python as "names", because in code, you refer to names, but the underlying variable might change. This is also why python is best described neither as call-by-reference nor call-by-value, but rather "call-by-assignment" or "call-by-object": you're passing a reference to a variable bound to a particular name. Unless you're writing C extensions, you never actually have a reference to the variable itself -- instead, to the name.
Meanwhile, closures in python are always late-binding: they are themselves "live", mutable objects, in the sense that they're just a reference to the namespace they're closing, and *not* a copy. So if you mutate that namespace between establishing the closure and making use of it, you'll see the mutated version.
[1] https://docs.python-guide.org/writing/gotchas/#late-binding-...
Also worth keeping in mind that, AFAIK, there is one scope per function, which differs from other languages that can have nested scopes inside a function. I think the list comprehension is the only exception in python3, which was changed so that names no longer leak out of it, which was frankly insane in py2.
I don't think the Python compiler is to know if a loop variable is closed over and then adjust the code generation. So this change would likely introduce a lot of complexity, or be a pessimisation in the normal case.
I think the simple, although less helpful, semantics are clearly better here. FWIW I don't think this comes up in practice very much. Maybe because callbacks in Python APIs aren't common (and its not due to this).
I agree the current semantics are better by the measure of performance of the simplest possible implementation. Though that's not Python's main design goal.
This logs from 0 to 9:
for(let i = 0; i < 10; ++i;) setTimeout(() => console.log(i));
This logs from 10, 10 times: for(var i = 0; i < 10; ++i;) setTimeout(() => console.log(i));
`let` and `const` were created to fill this gap. They have block scope and are special-cased in for loops (well, `let` is; you can't reassign to a `const` variable so nontrivial for loops won't work):
The interaction with `for` is explained well on MDN, including the special casing: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe..."More precisely, let declarations are special-cased by for loops..." (followed by a more detailed explanation of why)
See also https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
I was spoiled by this and driven nuts by function level scoping like python's and javascript's 'var', and the adoption of 'let' made me very happy.
My current annoyance with JS is its lack of -
but there's a proposal for that, and the 'match' proposal (which would be glorious) depends on it on the basis it's a shoo-in, so I am very hopeful.Note that I know:
already works but it's Not The Same (although significantly less upsetting than (() => { ... })() is ;).[1] side note, closures in python are always late-binding, which is what causes the behavior in OP
For those who prefer a functional style, functools.partial can also solve this problem.
(I use Python, and I like a lot of things about Python, but I don’t like its scoping rules at all, nor do I like the way that Python’s closures work. I would use a double lambda.)
``` def loop(): for number in range(10): fixed_number = number def inner(): return fixed_number yield inner ```
And some more stuff:
What's so interesting about the Elephant case? You store the value of i at run time, so it can be accessed later. lambda doesn't have that capability, as it's inheriting i from the block.
You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching.
> The behaviour is correct.
Python doesn't have a standard. Nobody knows what behavior is correct. That's the only behavior Python has, so, again, calling it correct is just not saying anything.
> What's so interesting about the Elephant case?
This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.
You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
That is not correct. Python generally has two "runtimes": one is on import, when all of the module level code is evaluated; and the runtime, when the bodies of various functions and methods are evaluated. The two can be combined, e.g. by importing a module from within a function, but generally they are separate.
> Bytecode is just a form of caching
Not quite. It is perfectly possible to deploy just the bytecode and disable recompiling on import.
> Python doesn't have a standard
Of course it does -- CPython is the reference implementation, so its behaviour is "correct".
You don't understand what runtime in python is. Compilation into bytecode is an implementation detail and not part of runtime. Even when you start a python scripts from a bytecode state, there is a runtime. f-strings are evaluated at runtime. When you declare them within a function, yes they will be parsed, but they will not be run.
> This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.
In your case, you put them in lambdas, which is equivalent to putting them into a function context. The f-string will only be evaluated when lambda/function is called. In your Elephant case the runtime happens when i=n, whereas in lambda case, the runtime is when i=9 (because there is no local copy of the variable/context). Plain and simple. If you don't understand such a simple distinction, I recommend you to start reading the bytecode as it's much clearer at that level.
> You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
It's not an inconsistency, if you understand the principle.