Well, maybe fileinput is meant to be used for making shell one-liners using the -m switch or so, like in the link you posted, or like perl/awk. Does that play nice with Python's block syntax, though?
The functionality is almost less than trivial. It's like, trading 3 lines of super straightforward code for one line of import statement and a library dependency.
Pretty sure it tries to emulate awk's programming model, but even if I had a use for that I'd rather write these 3 lines of code myself - so the code is actually clear without looking at documentation, and so that it can be modified easily.
Idk, it never occurred to me the need for a library that does more than "for line in sys.stdin:" but less than what you can easily write in under a minute.
It handles files being passed on the command line too. Your particular usage might only take a minute, but to make it play nice with other expected CLI usages, takes more thought (and you will probably forget something).
> Idk, it never occurred to me the need for a library that does more than "for line in sys.stdin:"
Of course if you'd only used fileinput for that it would be worthless.
However what it does is way more useful, namely iterate on the lines of all files provided as parameter (or sys.argv[:1]), fallback to sys.stdin if no files were provided, and swap the special sentinel `-` by sys.stdin.
I actually really like fileinput - to me it's a great example of the good kind of batteries included. I appreciate easy built-in utilities for super common programming tasks! Those are great to have.
Not just awk but cat, grep, etc.---even python itself. It's nice to have something that "recommends" such a common Unix convention, a lot like getopt but moreso. In Perl this would be `while (<>) { ... }` and in Ruby `ARGF.each_line do |line| ... end`. I'm glad they are keeping it!
There’s a big risk with using <> in perl, if any of the command line arguments might come from somewhere untrusted - if you use perl it’s worth reviewing the discussion under https://perldoc.perl.org/perlop.html#I%2fO-Operators
Why reinvent the wheel, when Python ships with an already written and tested solution? The 3 lines of code you would write are probably going to miss some corner case.
If I'm reading some unfamiliar code I would rather see one use of a standard library function call than several lines that I have to read and understand. And if it was really the first time I came across the `fileinput` module, I only have to pay the cost of reading the documentation once, and then forever benefit from having to read and understand less code whenever it is used.
It was on the PEP's original list of things to remove, but they decided to keep it because it's handy for quick scripts. I use it all the time for scripts that I delete shortly after use.
There are much more of it. I think around 1/3 of standard library is legacy and burden for python. I recall there was a PyCon talk about why you may not want to be a part of standard library. Release process and support of old versions is very strict and heavy, it could delay releases and API change dramatically. You can look at situation with urllib for example, there are 3 versions of them, but most popular is requests 3rd party module.
AIFF is used widely in audio production software; for exporting uncompressed audio many application default to AIFF. It’s the most universally useful lossless format; WAV support is ubiquitous too but AIFF has better metadata. FLAC and ALAC are good alternatives today, but not on the same level of support yet. For example, for purposes of price differentiation, Pioneer reserves FLAC and ALAC support for their most expensive digital turntables.
If you’re writing a Python script that outputs an audio file, outputting to AIFF seems like a save bet. Then again I’m not sure how many people are actually using the module from the stdlib--but neither does the PEP’s author, it seems. His level of research was asking his friends on Twitter:
This seems like a good idea to me, but what are the chances that something like this actually lands? I know Guido did not seem to like this approach (to put it mildly), although I'm not sure he still has veto power having stepped down as BDFL.
Also, the changes sound like a major version bump to me, but I doubt people want that to happen after Python 2 to 3...
I'm not sure Guido's argument was with the argument indicating some stdlib parts are outdated as the argument that new library features are not worth using because twisted wants to support python 2
This seems like a vast swath of things to deprecate with one PEP, but it looks like the author of the PEP talked with a large number of core developers about issues that they had with the stdlib and decided deprecated packages that way, which I think should be good for approval of the PEP.
I have a couple of questions:
- I realize there may not be a fork with this PEP implemented, but how might this impact Python's local relative build time, and how might that convert over to the build pipelines? Drastically faster build times would be really nice.
- Are Python 2 -> 3 migrations for stdlib packages mostly rewrites or mostly tacking on compatibility layers like `from __future__ import unicode_literals`? If stdlib packages were updated with Python 3 syntax, it might indicate sustained demand for said package going forward. I'm not sure.
> Are Python 2 -> 3 migrations for stdlib packages mostly rewrites or mostly tacking on compatibility layers like `from __future__ import unicode_literals`?
Packages in the stdlib don't need compatibility layer since they are always used with the Python version they were written for, it's motly incremental rewrites.
Regarding the build time, I don't think most of the time is spent testing those parts of the stdlib.
I'm not the OP, but I suspect that what was means was backporting features to py2.7, which presumably use some new Python 3 which haven't been built in the old language version.
I don't think that back porting features to 2.7 it's in anyone's radar anymore given that it goes end of life in just over 7 months. Anything that isn't already there is likely not going to benefit many people
You'd be surprised I think. In my job, EOL noticed are all it takes, often, to push people onto the next version, even reluctantly. I imagine many situations where Python 2 is still used are not developer driven but, rather, business value propositions. Many of my clients wouldn't understand a discussion about upgrading because of Unicode strings, but if I were to mention to my clients that Python 2 is no longer receiving bugfixes and officially has reached EOL status you can bet they would all throw money at it.
How many people actually used Python's MSI library?
It was Windows-only (presumably a wrapper around the native tooling), and was primarily for creating Python's own installer, which apparently doesn't get built as an MSI anymore.
I use it for packaging my Python application. But it seems pretty low-level, so I suppose I could call the win32 API directly through ctypes when they remove it.
You've been able to do Win32 stuff in an AppX since early in Windows 10 history, it's mostly "unsandboxed". MSIX, the "new" AppX, has Windows 7/8 support for Win32-focused installs in preview.
Semi-relatedly, Microsoft has already been exploring with the Python team MSIX deployments for the Python interpreter itself (you can even find it in the Microsoft Store now on Windows 10, and typing `python` on a command line in recent builds of Windows 10 will auto jump you to the Store if you don't have a python.exe in your PATH/installed).
(Also, getting in the weeds, the "mostly unsandboxed" refers to that MSIX installs will do a few things to insure at all costs clean uninstall, including system folder virtualization/redirection, registry virtualization/redirection, and similar techniques. Windows and MSI has been doing a lot of this sort of thing under-the-hood silently since XP as a part of compatibility work, MSIX just does it more obviously/automatically/always.)
Serious question - Why put off DeprecationWarnings to 3.9? If these modules are as dead as this PEP claims, then I'd wonder if Python scripts using such modules would be actively looking to transition to 3.x in the first place.
Hmm, should perhaps this be worthy of a major version number bump? Yes, it's not as large or as breaking of a change as the 2.7->3.0 transition, but by removing modules from the standard library, backwards compatibility is being broken. Now, perhaps most people don't use those features anymore, but for those people who do use and rely on these modules, a major version number change would be a welcome signal that their code will no longer work with the new version.
But, after the initial pain of the 2.7->3.0 transition, I doubt we'll ever see another major version number jump, even if a logical use of version numbers would merit it.
Does this mean we're going to have Python 2 installed on things forever? (Particularly anything that touches NNTP servers, as there's no replacement for that lib)
If it's simple, then what's the point in removing it? Dealing with standard application level protocols like SMTP, NNTP, FTP, IMAP, POP, and HTTP shouldn't require installing third party packages.
So that they can be improved / maintained. "stdlib is where packages go to die" is kind of true. You can use contact http from stdlib for example, but that's how we get urllib, urllib2, 3rd party urllib3 and requests.
For the point of removing the modules, check the "rationale" part of the PEP.
The original post you responded to mentioned that NNTP did not have a replacement, so I don't see how it could be improved/maintained if it's just not there anymore.
> but by removing modules from the standard library, backwards compatibility is being broken.
API 101 - there is very little cost to leaving them in, but a hidden major cost to having them disappear, usually for non-developers.
--
By the way, "batteries included" is one of the BEST features of python.
Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools?
It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.
More languages need "batteries included", maybe like Perl.
I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.
> By the way, "batteries included" is one of the BEST features of python.
None of these libraries need to be nuked from existence as a result of this change. I'd wager they'll move into PyPI modules so that teams relying on them could safely continue to do so.
> Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools? It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.
That's one extreme, but I don't think that's what's being proposed. The proposed model is closer to what Rust does today, where the core is slim, opening up new potential use cases, and the more complex functionality built on top of it is left to the community to maintain.
Take a look over some of the modules they're deprecating, like smtpd. What kind of standard library requires an SMTP daemon built in? That's akin to a homeowners toolkit including a planishing hammer [2] for some reason.
> I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.
Depends on how it's done, honestly. Check out this Rust "scripting" system [1]. It has full support for third-party crates.
Sorry, my "batteries included" comment was independent of the changes proposed. I just love the rich library python provides (compared to other scripting languages).
I hope you didn't take my reply to be aggressive in any way! Standard library is always a point of contention in any environment and it's always good to explore all angles. It's kind of the ultimate bikeshed in a lot of ways. Python indeed is a very batteries-included language and there's a lot to like about that. I wonder if there's a way of preserving that by differentiating between a 'lite' and 'core' distribution so we can still meet the goals of the embedded / small system community?
> Take a look over some of the modules they're deprecating, like smtpd. What kind of standard library requires an SMTP daemon built in?
Just because it's not widely used doesn't mean it cannot be included in the stdlib. For instance, where I work, we have a MTA test framework that makes extensive use of smtpd to receive messages processed by our configured MTA which we then write to disk and subsequently make assertions on the message (e.g., contains certain headers, has certain recipients, etc.).
Having various protocols (client and server) in the stdlib is not a bad thing and, personally, I think it's very useful for testing purposes.
> API 101 - there is very little cost to leaving them in
That's not true as newcomers will typically bias towards bundled modules instead of 3rd party ones. If it's well accepted that the bundled modules are bad in some way then you are encouraging further use of them by leaving them in. That's real cost, and not just on the Python maintainers side of thing.
Just how is a beginner, or in fact anyone, supposed to go and choose the "right" 3rd party module, when it seems half the stuff in pip is either half implemented, or even just a hello world example.
By simply asking that question you're already more likely to get a better result than if there was a known-wrong choice bundled into the standard library.
But really that's just a tutorial/discovery issue, which has many solutions. And not even a skill isolated to programming - how do you pick the "right" item on Amazon to buy?
> By the way, "batteries included" is one of the BEST features of python.
> More languages need "batteries included", maybe like Perl.
Agreed with both points, the lack of batteries makes Lua quite annoying for me personally. Python and Perl are much more comfortable because you don't have to hunt down the most common operations.
Of course, Perl 6 has decided not ship batteries included, replacing batteries with a fusion reactor instead.
> Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools?
Sure. But did you read the list of proposed deprecations?
It's not like that homeowners toolkit becomes any more useful when it's also got a buggy whip tool, a set of special allen-like keys that only open the case of a TRS80 or a Commodore64, and one of those picks for getting stones out of horses feet.
Most of the proposed things to take out seem obviously "right" to me. AIFF sound file support? MacOS9 binhex? CGI? _Maybe_people are still using those things, but it seems to me like they're edge-case enough to let them know now that they'll need to do "pip install aifc" or "pip install cgi" sometime after late 2021 if they update their python installation (which they can choose to put off until at least 2024 or so, just like all of us who're still sometimes using Python 2.7...)
The approval process for getting libraries okayed for production is a massive pain in the ass, and for good reason. Without a batteries included language, it'd be easier to write all those tools in-house than hope that maybe in two months I can get a library out in production, even though it's just JSON parsing or something. The "this is an official part of the language and held to high standards, guaranteed to be suitable for poduction and needing no approval beyond already being approved to use the language" makes Python way more preferable IRL than languages I love, but don't even include a random number generator in the standard library.
Python minor versions allow things to be removed and thus 'break backwards compatibility'. There is a process for warning of upcoming deprecation then removing it after that. For modules this is documented in PEP 4 (from 2000). Similar deprecation schedules are used by other important ecosystems such as Django.
We'd be on way more than Python 3 if the major version was bumped for each one.
It's also a bad idea. Semver doesn't distinguish between removing an API that has been deprecated for a decade and Python 2->3. Or, for that matter, between removing one function and removing large parts of the standard library.
You don't see a difference between removing a rarely-used module that has been deprecated for a decade, giving developers 10 years to replace it on the one hand and overhauling the entire language, breaking not just some but most code, on the other hand?
I'm sure Python 2.7 broke compatibility, but you don't see people refusing to upgrade from 2.6 ten years after its release.
Of course there's a difference, but it's irrelevant here, because no matter how long breaking changes have been announced, they break existing code. I prefer a versioning scheme that signals this.
I was once a fervent devotee of semver. It's so predictable! It's structured! There's a system! Everyone loves a system, especially engineers.
I've since become disillusioned.
The problem with semver, in my experience, is that it's impossible to predict whether a change will actually break someone else's code. Of course there are certain classes of change that are more likely to cause problems for other people. Changing a function signature, or deleting a function outright, is obviously a breaking change.
But the line between breaking and non-breaking isn't a bright one. Move away from the obvious examples and things start to get murky. Even the humble bugfix can be problematic. What if a client application unwittingly relies on the buggy behavior? Now that fix is breaking for them. Is that a contrived concern? Maybe—though anyone who has written an emulator can attest that this a real problem.
What about a non-breaking feature addition? Let's say the new feature requires some extra branches in a function, but doesn't change the function's interface or behavior for people who don't use the feature. Fine, non-breaking. Now say these branches alter the function's performance, and a client application's batch job that used to run in under an hour now takes four hours. It does run, so it's not "broken," but four hours is an unacceptable runtime to the users. Is that still a non-breaking change?
What these examples demonstrate to me is that semver's breaking/non-breaking change concept is incoherent. It conceives of changes as universally breaking or non-breaking, out of context, but a change can only be breaking or non-breaking in the context of a specific client application. Even the seemingly obvious example of deleting a function is non-breaking for applications that don't use the function!
I think the way we release software reflects that we know this deep down, even if we don't admit it. Imagine how you might handle upgrading a library in an application you've written. The new library is a bugfix release. Do you upgrade the library, push it to production at 100%, and gallop off to lunch without so much as a glance over your shoulder? My guess is no. Personally I'd be running my test suite, reading the library's change log, making a gradual release, and keeping a close watch on instrumentation during and after.
The interactions between software systems are simply too complex, too nuanced, too specific to the particular applications. We put all these safeguards in place and do things cautiously because we've been burnt too many times. And we've learned that in actual fact, the line between breaking and non-breaking doesn't exist.
Red Hat 6 is supported for another 18 months and still ships with Py 2.6.
Python's slow-moving gently gently approach to breaking changes has not been good for the ecosystem. I'll be glad when 2.x is dead. 7 months 8 days 14 hours... https://pythonclock.org/
"just"? From where I'm sitting that's a pretty large chunk of IT.
This has knock-on effects: authors that want to deploy scripts/apps with the minimum fuss will avoid adding deps to whatever /opt-based repo RH ships Python 2.recent (and the hoops you have to jump through to install and activate that). So they remain compatible with 2.6.
All of the other applications and 3rd-party modules shipping with RH 6 are also chained to Py2.6.
Many conservative shops (industry verticals) will refuse to upgrade _anything_ until they absolutely have to. I suspect we live in slightly different IT worlds (lucky you!). This is a problem I see frequently and that's why I'm suggesting Python needs more strict impetus for timely upgrades, not more decade-long opportunities for balkanization and incompatibility.
What I mean is not that Red Hat is insignificant, but that it's not special that it does not upgrade Python. Even in non-conservative shops that generally would just use the latest version, using Python 3 instead of 2 was not a no-brainer for a long time.
> But, after the initial pain of the 2.7->3.0 transition, I doubt we'll ever see another major version number jump, even if a logical use of version numbers would merit it.
Do you think the transition would have been less painful if 3.0 had been called 2.8?
If Python 2.8 had come out and failed to run almost all code written for Python 2.7 then that probably would have been truly disastrous for the language.
My one objection to this is the `imp` module. As far as I can tell it is the only way (short of `sys.path` modification) to specifically load a module found at a specific path. For testing systems I use this quite extensively... Looks like I need to make some comments...
I use imp.reload(my_module) all the time when %run-ing things from ipython that import my_module after editing it. What's the alternative for that use?
Just one data point here, but with the exception of the cgi module a long time ago, I haven’t used any of these modules in the past 15 years of web programming, ETL scripts, or data processing. In the keep list, I’ve used fileinput once that I can remember.
This has been a long time coming and I'm surprised it's taken until 2019. A better time to have done this would have been back at the 2-3 migration; I'm not sure the community will be able to survive another transition like that, but hopefully lessons have been learned from the previous schism. A step in the right direction for sure!
For example-- why does "add" add two fragments, but "mul" multiplies one fragment by a scalar value? Was the assumption that "mul" will probably just be used for attenuation?
It would be fun to build a tk-based audio app around audioop for the sole purpose of complaining about the deprecation of this module. :)
The only thing I worry about is the uu module. Even though 99.99% of people will never use it, there are also probably literally millions if not billions of pieces of content that are encoded using that standard. I realize it's only a few lines of code, but from an archival perspective it seems like there's something distasteful about making it harder for future generations to access content that may have historical value. At least send a quick note to the library of congress or the archivist community first to get an idea of to what extent this stuff is getting used.
This module is pure python. It looks like if people rely on it, they could easily put the code in a pypi module, and maintain it there. It doesn't really make sense to keep it in the core if it has no dedicated maintainers.
That's a bit dramatic. Our knowledge of the encoding doesn't go away because Python removed support. Old versions will continue to work. The code is preserved in VCS. And finally, you/someone can release a 3rd party module to PyPI to fill the gap.
The useful functionality (encoding/decoding) is covered by binascii, which uu in fact uses internally.
What the uu module proper provides is a pair of awful file-to-file interfaces. Default to spitting the decoded file directly to disk at with the name in its header is, uh, questionable.
Isn't it just being removed from the standard library? It can still exist on pypi especially if it is as important as you suggest, someone will definitely host it on pypi.
Cute dog going for school like human baby see his cute video I am sure you can't stop laugh
http://bit.ly/2Wq6Ynr
United nation recently showed the real face of india
See what they said in report
After knowing this I am sure you will hate them if you have soft heart
https://youtu.be/oUskNtdWeIY
Considering how many different types of data have and can be stored in IFF chunks (and their order-swapped RIFF counterparts) I'm almost insulted by the PEP author considering IFF to just be "an old audio file format".
I think my Commodore/Amiga persecution complex is acting up again.
If we're going to lean more on PyPI, it would be nice to clean it up some. There's a lot of cruft that makes it very hard to find the right module to use to write a new project with. And it would be nice if package naming convention were more standard, and suggested extending existing modules rather than writing entirely new ones. If you want HTTP functionality, you use the "requests" package, which uses urllib3. Why not io::net::http::client, extending io::net::http, and so on?
Since the parser module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the parser module is scheduled for removal in 3.9.
Hm interesting comment. Does anybody know what the new approach to parsing Python in 3.9 is ? I searched python-dev@ but couldn't find any references to it.
I found an interesting tidbit about Rust "switching" from LL to LR here:
> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
Python, lean and mean? Seems like an incredibly niche use case to restrict the python community to.
My only concern with this is that they won't be made available on Pypi for those that still need it. They will pull out the code but they expect the community to package it back up and make it available. Their hope is that the community will improve it before making it available, but I suspect the reality will be that the moment it is deprecated, someone will just go through and make all of them available in Pypi exactly as is.
Someone linked to Amber Brown's talk and I thought this point would be important for discussion. I think there is something that isn't being considered here when it comes to relying on PyPi.
3PP(Third Party Packages) issues are responsible for a lot of application security vulnerabilities today. Every large enterprise organization has no idea what packages a developer is pulling onto their laptop and into their codebase.
As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. You have no idea who is on the other end of that push too. It's too hard to keep track of changes and causes us to have to pin packages and versions that have been ok'd for use instead of a new release of Python. You have no guarantee the security engineer who reviewed the code didn't miss anything either.
There’s really two separate concepts here: review and the standard library. Why not separate concerns and have a review process that isn’t coupled to the standard library? There’s no reason why any PyPI package can’t have meaningful reviews published.
The question that you’re ultimately seeking the answer to is “what code has been reviewed and which reviewers do I trust?” - lots of ways to solve that.
Either the reviewer is trusted and might as well review the libraries as part of the standard library or they are not trusted and reviews by them are useless.
But keeping them together does not need a standard library, each python distribution could have a set of blessed packages, those packages could be "unblessed" without breaking applications using them
If the two tasks require different skillsets (and they do) then separating them could easily mean more people are able and willing to help with one or the other.
> There’s no reason why any PyPI package can’t have meaningful reviews published
You are completely correct. There's no reason at all why any random PyPI package can't have meaningful reviews published. I would go so far as to say that this is true without any changes to the standard library or current processes at all.
With that in mind, I'm thinking about all the various packages my colleages use. I don't think I've seen published reviews for an appreciable fraction of them in any language. This suggests that perhaps enabling reviews might not be the hard problem to solve here.
This thought process also hilights to me that the major advantage of a stdlib is that you have a higher degree of assurance that its contents have seen meaningful review by multiple sets of eyes. It's not just the potential for review that matters, it's the degree of assurance.
> “what code has been reviewed and which reviewers do I trust?”
Thank you for putting this so succinctly. There are other supply-chain problems in decentralizing control of the modules though:
Modules might only have a few people or one person responsible for them. Consequently suspect commits by a malfeasant or compromised (hacked) team member might not be noticed. Maybe this is a variant of the old saw "with enough eyes all bugs are shallow" but my gut feeling is that commits to Python core will get more inspection than those to a small 3rd party module. Today, modules get some review for free, just by being part of stdlib. Even if I know and trust @jdoe I have less assurances that they didn't get phished and their repo tweaked.
Also trusted team members/organizations may change over time. The browser extensions world is the poster child for this, where we've seen not only similarly-named malicious extensions posted to stores but also once-legitimate extensions quietly purchased and subsequently subverted by bad actors.
I like the idea of a review process but I have a hard time imagining a crowd-sourced system that wouldn't get gamed. We have a "dissolution of responsibility" problem: millions of companies rely on these components but have no explicit responsibility of care. Perhaps that needs to change, somehow.
Companies (and anyone interested) could simply pool funds together and use that to ask already trusted and respected auditors to do reviews.
And anyone who pays into the fund should be able to vote on what package to review. (And there should be a weighted lottery, so eventually small contributors' wishes have a chance to get fulfilled.)
For good measure anyone can put this on the blockchain, make a flattr/patreon thing out of this. (Somehow use github sponsoring...) Who knows.
I suspect you're gravely over-estimating how much value there is in the dustier "batteries included" modules over an out of tree component for security.
The problem is that security assumptions change. If you're looking at some code in 1995 that accesses an API, it seems reasonable that it disables validation for the SSL library it brings in to access an HTTPS URL. Handling that properly would have been a huge pain, faster is better.
But in 2019 you'd be appalled, how is this thing not depending on certifi and requests? As it is it's wide open to a MitM attack and you'd get blind sided.
A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() and in 1970s Unix I'm sure this was invaluable. But obviously today "copying strings into fixed width buffers with no terminator" is essentially never what you wanted, but people assume based on the name that strncpy() will do some other thing they do want, and insert a security defect into their code.
> A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy()
Arguably, what makes C the perfect example is not necessarily its standard library functions per se but the language's official string data type.
> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers,
Yes, but do you like funding them? This is all freely downloadable open source software.
> As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night.
This is an appealing sentiment, but you will get hacked if you rely on it.
Python 2's standard library didn't do SSL certificate verification by default until the fairly controversial PEP 476 from late 2014, which changed the default in a point release of Python 2.7. Through 2014 (and for some time later, given how slow it is to get new upstream releases through distros onto someone's computer), the standard thing to do to use HTTPS securely was to use requests, a module developed by someone who got commit privileges taken away on another of his projects for making multiple releases a day. You were more secure with his code than with the standard library.
I think you have no more idea who's on the other end of pushes to the standard library than who's on the other end of pushes to PyPI. (Which is to say, in part, that you can equally well have an idea of both of these if you put effort into it. I've met the maintainers of several of my third-party dependencies at PyCon.) Something being in the standard library doesn't mean it has a higher class of developer behind it. There's more overhead, but it doesn't mean there's more maturity; many third-party module developers
are more experienced (either in their field, if it's something like crypto, or just in general as responsible developers) than standard library developers. And if anything it means that security updates are slower and rarer because the process is more painful.
Consider the argument of the 2009 paper "Security Impact Ratings Considered Harmful" https://arxiv.org/abs/0904.4058 (disclosure: I'm a coauthor). Whether people find a vulnerability important enough to patch has little correlation in practice with how exploitable it is. And the only thing that gets regular attention is the latest development version of the code. If some module was significantly refactored or reimplemented, no upstream developer is looking for bugs in the old version. So, if you want to be safe and you haven't personally both audited and fuzzed the code you're running, you actually want to be running the latest released version of that code, regardless of whether someone stuck a CVSS on the old version yet. I've met very few companies who can upgrade to the latest Python 3.x minor release promptly when it comes out (and several who are still on Python 2!). I've met many companies who can pip install the latest versions of their dependencies without too much trouble, though.
> Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of xml. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used.
Developers don't have a compelling reason to use 3rd party modules instead of the standard library. Therefore they don't. You consider that a problem and want to encourage them to use 3rd party modules more.
In short, you want developers to use 3rd party libraries because there is no compelling reason to do so?
> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
That's a silly reason. Just make a separate distribution with a stripped down standard library.
>> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
>That's a silly reason. Just make a separate distribution with a stripped down standard library.
True, you could have a std and core library. I think Rust does this for embedded. IMHO if you take away the large standard lib, except for pandas I have no more reasons to use Python. I don't like the language that much, but the std library is pretty good for scripting when you need something on a target system that only has standard Python.
To give a specific example, there's a lot of headache that new Python users could be spared if only they used requests instead of urllib2. But they don't know that it's compelling if they never go look, and just stick to stdlib.
Unfortunately, I think people usually search StackOverflow, or just type into Google and get pointed to SO. If they type in "how do I download a web page in python" they get:
1. urllib [1]
2. BeautifulSoup and a comment mentioning requests [2]
3. requests [3]
4. urllib, httplib [4]
Which is looking better than I expected... but no information on 2 or 3 relating to how you install those libraries. So 1 and 4 will Just Work.
I'd love if something like requests was included along with Python. But requests uses urllib3, so it's not like you could get rid of that. The main page for requests says they're working on a v3...you wouldn't want that kind of churn in the stdlib. Personally, I kind of like the structure of low-level libraries and fancier high-level libraries I use most often.
So I'm not quite sure what is the best thing to do. It sounds like requests III is a Python3 forward version. So it might make sense to ship that?
Python has decided to include packages developed outside of the stdlib before; I think json, pathlib, and maybe optparse.
230 comments
[ 3.5 ms ] story [ 236 ms ] threadI've never had a strong opinion about "batteries included", but boy there are some weird ones in there...
https://github.com/python/cpython/blob/master/Lib/imghdr.py
Pretty sure it tries to emulate awk's programming model, but even if I had a use for that I'd rather write these 3 lines of code myself - so the code is actually clear without looking at documentation, and so that it can be modified easily.
Of course if you'd only used fileinput for that it would be worthless.
However what it does is way more useful, namely iterate on the lines of all files provided as parameter (or sys.argv[:1]), fallback to sys.stdin if no files were provided, and swap the special sentinel `-` by sys.stdin.
Different strokes I suppose?
Why reinvent the wheel, when Python ships with an already written and tested solution? The 3 lines of code you would write are probably going to miss some corner case.
If I'm reading some unfamiliar code I would rather see one use of a standard library function call than several lines that I have to read and understand. And if it was really the first time I came across the `fileinput` module, I only have to pay the cost of reading the documentation once, and then forever benefit from having to read and understand less code whenever it is used.
Here's a better search, still 100k+: https://github.com/search?l=Python&q=%22import+fileinput%22&...
If you’re writing a Python script that outputs an audio file, outputting to AIFF seems like a save bet. Then again I’m not sure how many people are actually using the module from the stdlib--but neither does the PEP’s author, it seems. His level of research was asking his friends on Twitter:
https://twitter.com/ChristianHeimes/status/11302577994753351...
Also, the changes sound like a major version bump to me, but I doubt people want that to happen after Python 2 to 3...
[0]https://www.python.org/dev/peps/pep-8100/#results
I have a couple of questions:
- I realize there may not be a fork with this PEP implemented, but how might this impact Python's local relative build time, and how might that convert over to the build pipelines? Drastically faster build times would be really nice.
- Are Python 2 -> 3 migrations for stdlib packages mostly rewrites or mostly tacking on compatibility layers like `from __future__ import unicode_literals`? If stdlib packages were updated with Python 3 syntax, it might indicate sustained demand for said package going forward. I'm not sure.
Packages in the stdlib don't need compatibility layer since they are always used with the Python version they were written for, it's motly incremental rewrites.
Regarding the build time, I don't think most of the time is spent testing those parts of the stdlib.
stdlib usually written in Python, not in C. So I guess it won't affect the build time much. It will reduce test time but I'm not sure.
It was Windows-only (presumably a wrapper around the native tooling), and was primarily for creating Python's own installer, which apparently doesn't get built as an MSI anymore.
Semi-relatedly, Microsoft has already been exploring with the Python team MSIX deployments for the Python interpreter itself (you can even find it in the Microsoft Store now on Windows 10, and typing `python` on a command line in recent builds of Windows 10 will auto jump you to the Store if you don't have a python.exe in your PATH/installed).
But, after the initial pain of the 2.7->3.0 transition, I doubt we'll ever see another major version number jump, even if a logical use of version numbers would merit it.
For the point of removing the modules, check the "rationale" part of the PEP.
API 101 - there is very little cost to leaving them in, but a hidden major cost to having them disappear, usually for non-developers.
--
By the way, "batteries included" is one of the BEST features of python.
Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools?
It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.
More languages need "batteries included", maybe like Perl.
I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.
None of these libraries need to be nuked from existence as a result of this change. I'd wager they'll move into PyPI modules so that teams relying on them could safely continue to do so.
> Have you tried to fix something in your house with a "homeowner's toolkit" which is usually something like a hammer, pliers, 2 screwdrivers, a putty knife and a few more basic tools? It is REALLY tedious, like writing a C program with a few basic tools like stdio and ctypes.
That's one extreme, but I don't think that's what's being proposed. The proposed model is closer to what Rust does today, where the core is slim, opening up new potential use cases, and the more complex functionality built on top of it is left to the community to maintain.
Take a look over some of the modules they're deprecating, like smtpd. What kind of standard library requires an SMTP daemon built in? That's akin to a homeowners toolkit including a planishing hammer [2] for some reason.
> I think if the cost of deploying a script is 1, deploying it + a dependency is literally something like 100x. You have to make assumptions about all the environments the script will run in, and they are usually wrong.
Depends on how it's done, honestly. Check out this Rust "scripting" system [1]. It has full support for third-party crates.
[1] https://github.com/DanielKeep/cargo-script
[2] https://en.wikipedia.org/wiki/Planishing
Just because it's not widely used doesn't mean it cannot be included in the stdlib. For instance, where I work, we have a MTA test framework that makes extensive use of smtpd to receive messages processed by our configured MTA which we then write to disk and subsequently make assertions on the message (e.g., contains certain headers, has certain recipients, etc.).
Having various protocols (client and server) in the stdlib is not a bad thing and, personally, I think it's very useful for testing purposes.
That's not true as newcomers will typically bias towards bundled modules instead of 3rd party ones. If it's well accepted that the bundled modules are bad in some way then you are encouraging further use of them by leaving them in. That's real cost, and not just on the Python maintainers side of thing.
maybe you could make it explicit for new scripts?
But really that's just a tutorial/discovery issue, which has many solutions. And not even a skill isolated to programming - how do you pick the "right" item on Amazon to buy?
Built-in functionality is less important if you can easily use packages.
Agreed with both points, the lack of batteries makes Lua quite annoying for me personally. Python and Perl are much more comfortable because you don't have to hunt down the most common operations.
Of course, Perl 6 has decided not ship batteries included, replacing batteries with a fusion reactor instead.
Sure. But did you read the list of proposed deprecations?
It's not like that homeowners toolkit becomes any more useful when it's also got a buggy whip tool, a set of special allen-like keys that only open the case of a TRS80 or a Commodore64, and one of those picks for getting stones out of horses feet.
Most of the proposed things to take out seem obviously "right" to me. AIFF sound file support? MacOS9 binhex? CGI? _Maybe_people are still using those things, but it seems to me like they're edge-case enough to let them know now that they'll need to do "pip install aifc" or "pip install cgi" sometime after late 2021 if they update their python installation (which they can choose to put off until at least 2024 or so, just like all of us who're still sometimes using Python 2.7...)
We'd be on way more than Python 3 if the major version was bumped for each one.
https://semver.org/spec/v2.0.0.html
I'm sure Python 2.7 broke compatibility, but you don't see people refusing to upgrade from 2.6 ten years after its release.
How about <Very breaking changes>.<Breaking changes>.<Bugfixes>? That's what Python already does.
Semver is garbage.
I've since become disillusioned.
The problem with semver, in my experience, is that it's impossible to predict whether a change will actually break someone else's code. Of course there are certain classes of change that are more likely to cause problems for other people. Changing a function signature, or deleting a function outright, is obviously a breaking change.
But the line between breaking and non-breaking isn't a bright one. Move away from the obvious examples and things start to get murky. Even the humble bugfix can be problematic. What if a client application unwittingly relies on the buggy behavior? Now that fix is breaking for them. Is that a contrived concern? Maybe—though anyone who has written an emulator can attest that this a real problem.
What about a non-breaking feature addition? Let's say the new feature requires some extra branches in a function, but doesn't change the function's interface or behavior for people who don't use the feature. Fine, non-breaking. Now say these branches alter the function's performance, and a client application's batch job that used to run in under an hour now takes four hours. It does run, so it's not "broken," but four hours is an unacceptable runtime to the users. Is that still a non-breaking change?
What these examples demonstrate to me is that semver's breaking/non-breaking change concept is incoherent. It conceives of changes as universally breaking or non-breaking, out of context, but a change can only be breaking or non-breaking in the context of a specific client application. Even the seemingly obvious example of deleting a function is non-breaking for applications that don't use the function!
I think the way we release software reflects that we know this deep down, even if we don't admit it. Imagine how you might handle upgrading a library in an application you've written. The new library is a bugfix release. Do you upgrade the library, push it to production at 100%, and gallop off to lunch without so much as a glance over your shoulder? My guess is no. Personally I'd be running my test suite, reading the library's change log, making a gradual release, and keeping a close watch on instrumentation during and after.
The interactions between software systems are simply too complex, too nuanced, too specific to the particular applications. We put all these safeguards in place and do things cautiously because we've been burnt too many times. And we've learned that in actual fact, the line between breaking and non-breaking doesn't exist.
Python's slow-moving gently gently approach to breaking changes has not been good for the ecosystem. I'll be glad when 2.x is dead. 7 months 8 days 14 hours... https://pythonclock.org/
This has knock-on effects: authors that want to deploy scripts/apps with the minimum fuss will avoid adding deps to whatever /opt-based repo RH ships Python 2.recent (and the hoops you have to jump through to install and activate that). So they remain compatible with 2.6.
All of the other applications and 3rd-party modules shipping with RH 6 are also chained to Py2.6.
Many conservative shops (industry verticals) will refuse to upgrade _anything_ until they absolutely have to. I suspect we live in slightly different IT worlds (lucky you!). This is a problem I see frequently and that's why I'm suggesting Python needs more strict impetus for timely upgrades, not more decade-long opportunities for balkanization and incompatibility.
Do you think the transition would have been less painful if 3.0 had been called 2.8?
?
For example-- why does "add" add two fragments, but "mul" multiplies one fragment by a scalar value? Was the assumption that "mul" will probably just be used for attenuation?
It would be fun to build a tk-based audio app around audioop for the sole purpose of complaining about the deprecation of this module. :)
What the uu module proper provides is a pair of awful file-to-file interfaces. Default to spitting the decoded file directly to disk at with the name in its header is, uh, questionable.
Funny dog driving own car See his video on link below http://bit.ly/2WdSKpA
Cute dog going for school like human baby see his cute video I am sure you can't stop laugh http://bit.ly/2Wq6Ynr
United nation recently showed the real face of india See what they said in report After knowing this I am sure you will hate them if you have soft heart https://youtu.be/oUskNtdWeIY
Best way to get rid from face moles with the help of home remedies its easy and simple way You don't need to waste money for this https://zoomtips.blogspot.com/2019/05/easily-get-rid-from-fa...
I think my Commodore/Amiga persecution complex is acting up again.
Hm interesting comment. Does anybody know what the new approach to parsing Python in 3.9 is ? I searched python-dev@ but couldn't find any references to it.
I found an interesting tidbit about Rust "switching" from LL to LR here:
https://www.reddit.com/r/ProgrammingLanguages/comments/brhdt...
And I noticed some rules in Python's grammar that are awkward in LL parsing (set and dict literals, and comprehensions).
I wonder if those things motivated the switch? They certainly work though.
https://discuss.python.org/t/switch-pythons-parsing-tech-to-...
And I posted here about it:
https://www.reddit.com/r/ProgrammingLanguages/comments/brz2y...
It looks like the set and dict literals I noticed weren't so much the motivating use cases, but even more fundamentally assignments and keyword args!
Python, lean and mean? Seems like an incredibly niche use case to restrict the python community to.
3PP(Third Party Packages) issues are responsible for a lot of application security vulnerabilities today. Every large enterprise organization has no idea what packages a developer is pulling onto their laptop and into their codebase.
As a security engineer, I like having a core team and a standard library in place that has gone through a long mature process with experienced developers, instead of someone who just git pushes code every night. You have no idea who is on the other end of that push too. It's too hard to keep track of changes and causes us to have to pin packages and versions that have been ok'd for use instead of a new release of Python. You have no guarantee the security engineer who reviewed the code didn't miss anything either.
The question that you’re ultimately seeking the answer to is “what code has been reviewed and which reviewers do I trust?” - lots of ways to solve that.
You are completely correct. There's no reason at all why any random PyPI package can't have meaningful reviews published. I would go so far as to say that this is true without any changes to the standard library or current processes at all.
With that in mind, I'm thinking about all the various packages my colleages use. I don't think I've seen published reviews for an appreciable fraction of them in any language. This suggests that perhaps enabling reviews might not be the hard problem to solve here.
This thought process also hilights to me that the major advantage of a stdlib is that you have a higher degree of assurance that its contents have seen meaningful review by multiple sets of eyes. It's not just the potential for review that matters, it's the degree of assurance.
Thank you for putting this so succinctly. There are other supply-chain problems in decentralizing control of the modules though:
Modules might only have a few people or one person responsible for them. Consequently suspect commits by a malfeasant or compromised (hacked) team member might not be noticed. Maybe this is a variant of the old saw "with enough eyes all bugs are shallow" but my gut feeling is that commits to Python core will get more inspection than those to a small 3rd party module. Today, modules get some review for free, just by being part of stdlib. Even if I know and trust @jdoe I have less assurances that they didn't get phished and their repo tweaked.
Also trusted team members/organizations may change over time. The browser extensions world is the poster child for this, where we've seen not only similarly-named malicious extensions posted to stores but also once-legitimate extensions quietly purchased and subsequently subverted by bad actors.
This is one example (adware: could've been much worse): https://www.bleepingcomputer.com/news/security/-particle-chr...
I like the idea of a review process but I have a hard time imagining a crowd-sourced system that wouldn't get gamed. We have a "dissolution of responsibility" problem: millions of companies rely on these components but have no explicit responsibility of care. Perhaps that needs to change, somehow.
And anyone who pays into the fund should be able to vote on what package to review. (And there should be a weighted lottery, so eventually small contributors' wishes have a chance to get fulfilled.)
For good measure anyone can put this on the blockchain, make a flattr/patreon thing out of this. (Somehow use github sponsoring...) Who knows.
The problem is that security assumptions change. If you're looking at some code in 1995 that accesses an API, it seems reasonable that it disables validation for the SSL library it brings in to access an HTTPS URL. Handling that properly would have been a huge pain, faster is better.
But in 2019 you'd be appalled, how is this thing not depending on certifi and requests? As it is it's wide open to a MitM attack and you'd get blind sided.
A much older example: The C standard library includes a function that's very narrowly useful for copying strings into fixed width buffers with no terminator, called strncpy() and in 1970s Unix I'm sure this was invaluable. But obviously today "copying strings into fixed width buffers with no terminator" is essentially never what you wanted, but people assume based on the name that strncpy() will do some other thing they do want, and insert a security defect into their code.
Arguably, what makes C the perfect example is not necessarily its standard library functions per se but the language's official string data type.
Yes, but do you like funding them? This is all freely downloadable open source software.
This is an appealing sentiment, but you will get hacked if you rely on it.
Python 2's standard library didn't do SSL certificate verification by default until the fairly controversial PEP 476 from late 2014, which changed the default in a point release of Python 2.7. Through 2014 (and for some time later, given how slow it is to get new upstream releases through distros onto someone's computer), the standard thing to do to use HTTPS securely was to use requests, a module developed by someone who got commit privileges taken away on another of his projects for making multiple releases a day. You were more secure with his code than with the standard library.
I think you have no more idea who's on the other end of pushes to the standard library than who's on the other end of pushes to PyPI. (Which is to say, in part, that you can equally well have an idea of both of these if you put effort into it. I've met the maintainers of several of my third-party dependencies at PyCon.) Something being in the standard library doesn't mean it has a higher class of developer behind it. There's more overhead, but it doesn't mean there's more maturity; many third-party module developers are more experienced (either in their field, if it's something like crypto, or just in general as responsible developers) than standard library developers. And if anything it means that security updates are slower and rarer because the process is more painful.
Consider the argument of the 2009 paper "Security Impact Ratings Considered Harmful" https://arxiv.org/abs/0904.4058 (disclosure: I'm a coauthor). Whether people find a vulnerability important enough to patch has little correlation in practice with how exploitable it is. And the only thing that gets regular attention is the latest development version of the code. If some module was significantly refactored or reimplemented, no upstream developer is looking for bugs in the old version. So, if you want to be safe and you haven't personally both audited and fuzzed the code you're running, you actually want to be running the latest released version of that code, regardless of whether someone stuck a CVSS on the old version yet. I've met very few companies who can upgrade to the latest Python 3.x minor release promptly when it comes out (and several who are still on Python 2!). I've met many companies who can pip install the latest versions of their dependencies without too much trouble, though.
Developers don't have a compelling reason to use 3rd party modules instead of the standard library. Therefore they don't. You consider that a problem and want to encourage them to use 3rd party modules more.
In short, you want developers to use 3rd party libraries because there is no compelling reason to do so?
> A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
That's a silly reason. Just make a separate distribution with a stripped down standard library.
>That's a silly reason. Just make a separate distribution with a stripped down standard library.
True, you could have a std and core library. I think Rust does this for embedded. IMHO if you take away the large standard lib, except for pandas I have no more reasons to use Python. I don't like the language that much, but the std library is pretty good for scripting when you need something on a target system that only has standard Python.
https://docs.python.org/3.8/library/urllib.request.html#modu...
1. urllib [1] 2. BeautifulSoup and a comment mentioning requests [2] 3. requests [3] 4. urllib, httplib [4]
Which is looking better than I expected... but no information on 2 or 3 relating to how you install those libraries. So 1 and 4 will Just Work.
[1]: https://stackoverflow.com/questions/45717889/read-the-text-o... [2]: https://stackoverflow.com/questions/26050064/automating-down... [3]: https://stackoverflow.com/questions/44553348/how-to-download... [4]: https://stackoverflow.com/questions/2646288/retrieve-some-in...
So I'm not quite sure what is the best thing to do. It sounds like requests III is a Python3 forward version. So it might make sense to ship that?
Python has decided to include packages developed outside of the stdlib before; I think json, pathlib, and maybe optparse.