I really enjoy the comprehension syntax too, which is a bit odd at first. But it lets you think about the typical map/filter approach you'd take in JS, PHP, etc. to build a list OR an iterator in almost the same way. I started to love lazy evaluation once I got big into rust, but writing all that stuff out in vanilla typescript is hard. It's wild how much JS is in the world that builds and allocates giant datasets for what all could just be a little iterator.
Generators are great, and their comprehensions included are some of the best things about python. It's wild how many people jump straight to building a huge dataset (straight to numpy) when all they really need is a few iterators to lazy evaluate. I'd held out on properly learning python for a decade, but it turns out I actually love python I just don't love the weird subset of it most devs practice, that evaluates and allocates for EVERYTHING.
The only difference between your two solutions is the first one returns a list and the second, a tuple. Neither one is a generator.
Edit: That was wrong. TIL if you write what looks like a tuple comprehension, it's actually a generator.
To answer the original question though, if you write it as a generator with the yield keyword
def square(nums: list):
for num in nums:
yield num * num
will consume less memory because it won't have the return value in memory (the list). You can verify this by putting a print statement after the yield - it won't be executed until you call next on the generator.
It's basically the same. A lot of people feel it's more pythonic to spread this out across multiple lines with understandable indentation, while comprehensions can smush it all together. But the list comprehension was already there, of course.
If you are going to iterate through some of the resulting thing but not all of it then the generator means you aren't throwing away a bunch of the work that you've done.
It can also be more cache friendly. It doesn't need to allocate a whole new lists worth of memory.
One of the downsides of it being lazy is that if list x is mutated between when you create the generator and when you consume it then those changes are reflected in the generator.
I've done some micro benchmarks and it really depends on what you are doing. Profiling it with cProfile, Pyspy or using %timeit in an Ipython shell will tell you if it makes a difference.
Running a benchmark on my system comparing your generator expression version versus their generator function shows that they perform within a small margin of each other. Yours was marginally (< 1%) faster. (not a real proper benchmark, there's quite a bit running on this computer right now but it still has a lot of free cores so should be close to the same result from a proper benchmark)
I'd bet they have very similar performance-metrics, but the yield syntax is more extensible (i.e. you're not limited to one expression) and debug-able (you can put breakpoints within the function).
Also the name and the generator is nicer (for some definition of nice):
>>> def square_vals(x : list):
... return (v * v for v in x)
...
>>> square_vals([1,2,3])
<generator object square_vals.<locals>.<genexpr> at 0x786b8511f5e0>
>>> def square_vals_yields(x: list):
... for v in x:
... yield v * v
...
>>> square_vals_yields([1,2,3])
<generator object square_vals_yields at 0x786b851f5ff0>
I think it's more idiomatic to pass generator-comprehensions into functions rather than return them from functions
Don't forget you can chain multiple generators together too. This can have huge memory and compute time benefits with lazy evaluation depending how you structure your code. A very crude example below...
# Really big generator that would probably crash as a list
divmods = (divmod(x,5) for x in range(100000000000000000000000000000000))
i like them... a little too much. i keep abusing them more and more until every part of my program is a some sort of coroutine involved in an extremely elaborate dance of having to remember to randomly next() to keep the return values from all the yield expressions in lockstep. you're not having fun until you're next()ing and .send()ing the same generator. or trying to remember how a trampoline works. and then you realize that you have a problem. one that even generators can't solve. xD
but like seriously sometimes i feel like they are just strictly better than functions. ok time to take my medication.
a lot of coffee, wishing i had something even stronger xD.
honestly, it is genuinely a situation where print debugging is your friend… i get every bit of code that yields in as close physical proximity as possible and trace it with my fingers like i’m a child counting on my fingers. because you really need to have an unassailable mental model in order for you to get it right at all most of the time (more so than “regular” programming).
on the flip side, once you figure it out you never really need to touch it again, and you can usually “scale” the solution fairly easily as long as you don’t change the dynamics of the system (e.g. you can bring in N faux-coroutines in as long as only one pair is operating at once, but can’t as easily turn it from a two-person dance to a three person… i don’t know any three person dances lol).
23 comments
[ 0.23 ms ] story [ 71.2 ms ] threadEdit: That was wrong. TIL if you write what looks like a tuple comprehension, it's actually a generator.
To answer the original question though, if you write it as a generator with the yield keyword
will consume less memory because it won't have the return value in memory (the list). You can verify this by putting a print statement after the yield - it won't be executed until you call next on the generator.If you are going to iterate through some of the resulting thing but not all of it then the generator means you aren't throwing away a bunch of the work that you've done.
It can also be more cache friendly. It doesn't need to allocate a whole new lists worth of memory.
One of the downsides of it being lazy is that if list x is mutated between when you create the generator and when you consume it then those changes are reflected in the generator.
I've done some micro benchmarks and it really depends on what you are doing. Profiling it with cProfile, Pyspy or using %timeit in an Ipython shell will tell you if it makes a difference.
yield from
Instead of
return
Also the name and the generator is nicer (for some definition of nice):
I think it's more idiomatic to pass generator-comprehensions into functions rather than return them from functions6.2.8. Generator expressions ... The parentheses can be omitted on calls with only one argument.
[1] https://docs.python.org/3/reference/expressions.html#generat...
The first case creates an anonymous generator. When in doubt you can always use the dis module: https://docs.python.org/3/library/dis.html
# Really big generator that would probably crash as a list
divmods = (divmod(x,5) for x in range(100000000000000000000000000000000))
# Create a generator from a generator
remains = (remainder for n, remainder in divmods)
# Runs instantly
next(remains)
but like seriously sometimes i feel like they are just strictly better than functions. ok time to take my medication.
honestly, it is genuinely a situation where print debugging is your friend… i get every bit of code that yields in as close physical proximity as possible and trace it with my fingers like i’m a child counting on my fingers. because you really need to have an unassailable mental model in order for you to get it right at all most of the time (more so than “regular” programming).
on the flip side, once you figure it out you never really need to touch it again, and you can usually “scale” the solution fairly easily as long as you don’t change the dynamics of the system (e.g. you can bring in N faux-coroutines in as long as only one pair is operating at once, but can’t as easily turn it from a two-person dance to a three person… i don’t know any three person dances lol).