It's a pain in the ass to write functional code in python because the syntax is not optimized for it. A functional programming language makes lambda and (partial) application as cheap as possible, which Python does not.
I think you can get 95% of the benefit of FP just by favoring abstractions which present a functional interface. Python hasn't really gotten in my way, in that respect, though it would be nice to have first-class persistent data structures with structural sharing.
> What can you do that cannot be as easily done with generator expressions?
Technically nothing, but at the same time if you already have a function (especially a builtin) map/filter is more convenient and can be faster. And they're first-class objects which comprehensions are not, which can be convenient.
The goal is not necessarily to prune all redundancies: reduce already existed when sum/min/max/any/all were added, despite the latter being fairly simple special cases of the former (not quite entirely for min/max, but not far).
> Also, once map(), filter() and reduce() are gone, there aren't a whole lot of places where you really need [lambda].
By far the most common use of lambda for me is the "key" argument for sort() or sorted(). As in something like:
foo.sort(key=lambda x:x[2])
Yes, there are itemgetter and attrgetter in operator module, but I always have to look up their names and writing the lambda expression is simply faster.
Yeah, I don't get the "hate" for the functional operations, though comprehensions have better syntax
And no, I don't want to 'def' every small function I might need, pep-8 recommendations aside. Key as mentioned is one use, but there are other cases where it's especially useful
I’ve really come to miss reduce in Python’s builtins since std::accumulate has become a staple in my C++ development. It seems surprising that with Python 3’s increased support for functional concepts, it would relegate reduce to the standard library.
Not really, increasing support for functional concept was never a goal of Python 3 (or at all), it was mostly incidental and it did regress in other ways than just shoving the (relatively little used due to the existence of min/max/any/all) out of the builtins e.g. while extended unpacking was added, unpacking iterable function parameters was removed.
This is broadly true across language. In Java, it was Collections, List, Elixir it's Enum, Python is your mentioned stdlib namespaces. Knowing your collection-ey stdlib functions will only help your code be more idiomatic and likely performant.
I spent some time comparing Python's [0] map/filter and comprehensions against Snigl [1].
Comprehensions are just plain faster, especially if you perform say filter and map at the same time.
That and a lambda implementation that only a mother could love, lack of tail call optimization etc.
What bothers me about comprehensions is they come with a separate set of rules, like LOOP in Common Lisp or templates in C++; but I'm pretty sure the restrictions are part of what makes them fast.
It's still an assignment operation, and you can still get weird effects in the enclosing scope. I did something like this by accident:
Python 3.7.1 (default, Nov 28 2018, 21:47:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: lookup = {}
In [2]: [x for (x, lookup[x]) in ('ab', 'cd', 'ef') if x in lookup]
Out[2]: ['a', 'c', 'e']
In [3]: lookup
Out[3]: {'a': 'b', 'c': 'd', 'e': 'f'}
The whacky thing there isn't the list comprehension though, it's the implicit assignment of `lookup[x]`. If you desugar it some (which isn't really possible, but
[(x, lookup.setdefault(x, y)) for (x, y) in ('ab', 'cd', 'ef')]
comes close), what's happening makes a lot more sense. It works exactly the same way as any other scope: first checks locally, then nonlocally or globally. You're modifying a nonlocal. The syntax you're using to modify the nonlocal is surprising, but that's the surprising part, not the ability to modify a nonlocal.
I think I was personally surprised by two assumptions that I implicitly held, both of which disappear when you write them down:
1. The syntax of comprehensions led me to think that they introduce some kind of a variable binding, such as `let` in many Lisps, and that binding `lookup[x]` would have been a syntax error. Of course Python didn't introduce anything like that for comprehensions, and it's just a normal assignment operation, with an extra scope inserted in Python 3 to stop `x` leaking into the enclosing scope.
2. Because of the swap idiom `x, y = y, x` I thought tuple assignment happens somehow in parallel, so my code should have caused an undefined variable error, even if it is not a syntax error. But of course it just evaluates the right hand side first and then assigns to the left hand side in left-to-right order, which the language reference https://docs.python.org/3/reference/simple_stmts.html#assign... notes "sometimes result[s] in confusion" (near the end of the section).
It appears to work without list comprehensions as well:
Python 3.7.1 (default, Nov 6 2018, 18:49:54)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: lookup = {}
In [2]: a, lookup[a] = 'xy'
In [3]: a
Out[3]: 'x'
In [4]: lookup
Out[4]: {'x': 'y'}
I wasn't previously aware that this was possible, but I guess it's no different from
> I think dropping filter() and map() is pretty uncontroversial; filter(P, S) is almost always written clearer as [x for x in S if P(x)]
Performance reasons aside, I strongly disagree with this statement. Anyone can see that `filter` is going to apply the predicate to the iteratable in the first statement. The second statement is super awkward, declares a temp var `x` which to my limited Pythonic eyes looks like it's just calling the identity function on `x` that comes back from the "filter".
Again, I have no doubt Python is optimized to perform better with the second form, but that seems a limitation in the interpreter. The syntax to non-Python devs is pretty raunchy imho.
There is going to be a pretty sharp break between people who accept that working with a collection deserves special syntax support and those who don't. The readability issue for me isn't that the syntax doesn't look nice, it is that I've invested years into making a very efficient little mental space that deals with "data2 = fn(data1)" style constructs and I want to use it here like I do everywhere else.
Some silly-high percentage of for loops can be cast into this data2 = fn(data1). Why am I being told that it isn't readable? If this isn't easy to read, then something is wrong with the function syntax.
A chaining operator, to recast fn3(fn2(fn1(data))) => data > fn1 > fn2 > fn3 with reduce is much more readable, debuggable and easy to reason about than having to remember what syntax todays language uses to talk about collections and how that works in to the flow of data in my program.
I agree with you for the example with a named function P, but as Guido points out, filter predicates are typically simple expression which you don't want to create a named function for. So it would be say `filter(lambda x: x < 17, S)` versus `[x for x in S if x < 17]`. I think the second one is nicer even if the repeated x is ugly. Neither is great though. The comprehension syntax assumes filters are accompanied by maps. A filter without a map is a somewhat ugly special-case.
The real issue is the lack of a lambda syntax with a pythonic feel.
The list comprehension syntax is just set notation with Python keywords instead of symbols. Haskell, JavaScript, and Julia have similar syntax. It's also similar to the equivalent loop.
Which of these is clearer?
[x * y for x in S if P(x) > 0]
map(lambda x: x * y, filter(lambda x: P(x) > 0, S))
Does anyone know the origin story of reduce/filter in Python? It seemed interesting. I remember watching a talk by Guido. He mentioned that someone submitted a PR with it, and that it was very complete, so he merged it. But I'm curious what the decision making process was like for Python back then, or if there even was a formal process.
Programming language archaeology is a field that ought to get more attention.
39 comments
[ 3.3 ms ] story [ 90.1 ms ] threadStrangely FP is not popular among python developers.
There's an analogous case with filter and `if not cond: continue`.
map and filter calls are trivially rewritten to generator expressions, but it's not necessarily an improvement.
Technically nothing, but at the same time if you already have a function (especially a builtin) map/filter is more convenient and can be faster. And they're first-class objects which comprehensions are not, which can be convenient.
The goal is not necessarily to prune all redundancies: reduce already existed when sum/min/max/any/all were added, despite the latter being fairly simple special cases of the former (not quite entirely for min/max, but not far).
By far the most common use of lambda for me is the "key" argument for sort() or sorted(). As in something like:
foo.sort(key=lambda x:x[2])
Yes, there are itemgetter and attrgetter in operator module, but I always have to look up their names and writing the lambda expression is simply faster.
And no, I don't want to 'def' every small function I might need, pep-8 recommendations aside. Key as mentioned is one use, but there are other cases where it's especially useful
I highly recommend to become familiar with the functools, itertools and collections modules to anybody using Python and not using them regularly.
Comprehensions are just plain faster, especially if you perform say filter and map at the same time.
That and a lambda implementation that only a mother could love, lack of tail call optimization etc.
What bothers me about comprehensions is they come with a separate set of rules, like LOOP in Common Lisp or templates in C++; but I'm pretty sure the restrictions are part of what makes them fast.
[0] https://gitlab.com/sifoo/snigl/blob/master/bench/seq.py
[1] https://gitlab.com/sifoo/snigl/blob/master/bench/seq.sl
> filter(P, S) is almost always written clearer as [x for x in S if P(x)]
sure, Guido, if you think it's clear that the variable x is in the enclosing scope...
1. The syntax of comprehensions led me to think that they introduce some kind of a variable binding, such as `let` in many Lisps, and that binding `lookup[x]` would have been a syntax error. Of course Python didn't introduce anything like that for comprehensions, and it's just a normal assignment operation, with an extra scope inserted in Python 3 to stop `x` leaking into the enclosing scope.
2. Because of the swap idiom `x, y = y, x` I thought tuple assignment happens somehow in parallel, so my code should have caused an undefined variable error, even if it is not a syntax error. But of course it just evaluates the right hand side first and then assigns to the left hand side in left-to-right order, which the language reference https://docs.python.org/3/reference/simple_stmts.html#assign... notes "sometimes result[s] in confusion" (near the end of the section).
(this was a great example btw -- I write python3 a lot, and I was staring at it for a long time before Icould figure it out)
Comprehensions are often slower if you're applying a function regardless, definitely so if that function is a builtin.
Performance reasons aside, I strongly disagree with this statement. Anyone can see that `filter` is going to apply the predicate to the iteratable in the first statement. The second statement is super awkward, declares a temp var `x` which to my limited Pythonic eyes looks like it's just calling the identity function on `x` that comes back from the "filter".
Again, I have no doubt Python is optimized to perform better with the second form, but that seems a limitation in the interpreter. The syntax to non-Python devs is pretty raunchy imho.
There is going to be a pretty sharp break between people who accept that working with a collection deserves special syntax support and those who don't. The readability issue for me isn't that the syntax doesn't look nice, it is that I've invested years into making a very efficient little mental space that deals with "data2 = fn(data1)" style constructs and I want to use it here like I do everywhere else.
Some silly-high percentage of for loops can be cast into this data2 = fn(data1). Why am I being told that it isn't readable? If this isn't easy to read, then something is wrong with the function syntax.
A chaining operator, to recast fn3(fn2(fn1(data))) => data > fn1 > fn2 > fn3 with reduce is much more readable, debuggable and easy to reason about than having to remember what syntax todays language uses to talk about collections and how that works in to the flow of data in my program.
The real issue is the lack of a lambda syntax with a pythonic feel.
Which of these is clearer?
Programming language archaeology is a field that ought to get more attention.