If you want a lazy iterator, then you will need to keep the file handle open. I don’t see how this one use case invalidates the general guideline of not sitting on file descriptors that you are finished with.
How you feel about this is key to the article's argument. It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail.
That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.
It sounds desirable, but isn't feasible unless the language is stateless. If you iterate over a thing that is stateful, you need to know whether the iterator is lazy or not to decide if you can modify the underlying thing.
I don't think that's right. You can always turn it into a greedy iterable (e.g. convert it to a list) if you need the certainty that you can modify the underlying thing, while still reaping some performance benefits from the maybe-lazy iterator in the rest of your use cases. Assuming the iterator wasn't infinite, of course, but in that case you already knew it was lazy.
Is it really that arduous to have a high-level understanding of what the method you're calling does, at least at the level of knowing whether or not it returns the type of object you want and whether it takes ownership of its arguments?
There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
I'm not sure what you're arguing for. My point is that you cannot, in general, abstract over both lazy and eager iterators. There are irreconcilable differences in how they work (and how you have to treat them).
> It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail.
That's both right and wrong. That it's a lazy iterator itself may be an implementation detail that doesn't effect the interface (if the interface doesn't assure reusability; a concrete collection and a lazy iterator do have different behavior in important ways) but leaving a dangling external resource is not. For certain resources, when you are given a reference rather than the opened resource, you can't build a lazy iterator without also leaving dangling external resource that the caller doesn't have the information to close. That is a significant difference.
The fact that python has a lazy and non-lazy range is pretty much the point. Yes there are (common) trivial cases where you don't care if the range is lazy or not, but in general it is something you need to be aware of.
The assumption behind the paragraph following the example does not imply lazy evaluation, but rather the opposite - the broken one is obvious if you have lazy evaluation in mind, whereas the docstrings imply the caller would be using these the same and not be expecting lazy evaluation.
Many good points are made here, but I would point out that there is a significant difference between working with JSON and CSV: data that can be operated on can typically be extracted from a CSV by working iteratively over each line. The same can not be said about JSON files.
It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory, while the same is often impractical and unnecessary for a CSV (when it is millions of lines). The standard methods and idioms of doing these things in Python reflects this.
It's not uncommon to see files or streams in "JSON lines" format. For files where each line is a record and the records are sorted or indexed, it's possible to seek to a record of interest just as in CSV.
Those files are individual JSON entries though, as per the spec. They're not read as a JSON (since they aren't) they are read per line, and each line is parsed as a JSON.
I find it interesting that you are seeing one of the prime defects of python's json parsing implementation (no streaming support) as an inherent property of json parsing. Do you use any other languages that share this defect, or is this take based on your familiarity with python?
The point of a file context manager is to close the file when you're done with it. The CSV reader isn't done with it.
Given that the CSV reader does not control ownership of the file handle, it shouldn't return a lazy iterator when given a file handle. It should accept a path, then it could own the file handle until the iterator is complete or dropped.
So, yea, the CSV reader does a bad. But so what? That is a deficiency of the CSV library, not context managers.
The CSV library is part of the standard library, though. Its way of doing things has been blessed by the Python maintainers.
OK, it might still be correct to say the problem is with the CSV library. But if the context manager "best practices" lead to popular parts of the standard library being bad and unidiomatic, that's at best unpythonic. After all, "There should be one -- and preferably only one -- obvious way to do it".
Libraries that accept filepaths (except perhaps as a convenience) are bad. The CSV library is well-designed. It isn’t responsible for closing files because it doesn’t take files, it takes TextIO. None of this is to say context managers are bad; I think they’re great but they only work as intended.
That's being a little pedantic IMO. From a practical, user-facing perspective, the CSV library Reader takes in files. And they know this, which is why the documentation specifically calls out gotchas related to passing in a file object and the example code shows how it should work with a file object [1].
It’s not pedantic. It’s a very important distinction for exactly this use case. The CSV library doesn’t and shouldn’t close the source object because it may not have a close method and even if it does, someone else might be planning on closing it later (the scope that owns it) which might be an error depending on the implementation of the source’s close() method. Only the resource owner should close it, and the CSV library shouldn’t be the owner.
> It isn’t responsible for closing files because it doesn’t take files, it takes TextIO.
TextIO has a close() method, so this is irrelevant.
Rather, the author of the CSV library made a design decision about whether a CSV reader should take ownership of the underlying IO or not. If it takes ownership, it has the responsibility of closing the file when it's finished. If it doesn't, it doesn't. The decision was made for it not to take ownership.
Was this a good decision? Personally, i don't think so. There is nothing useful you can do with an IO object once a CSV reader has used it, so there is no point retaining ownership separately. It just makes more work for the programmer. This is compounded by the fact that there is no way to obtain the underlying IO object from a CSV reader, so you have to hold on to the files separately.
In the normal use case where you open a single file and then read from it in a single block of code, this is not a big deal, because you can wrap that in a with block for the file. But as the author points out, it's a serious impediment to using lazy readers. I have also run into this problem when managing multiple open files; i have a tool which opens a bunch of CSV files and merges them in some strange way, and it is sensible to close each file when it is finished, but to do that, i need to hang on to the files, and build some machinery to close the right one at the right time.
On the other hand, i can't think of any downsides to deciding, way back when, that CSV readers should close their underlying files. But it's not safe to change that decision now, so we're stuck with it.
> Was this a good decision? Personally, i don't think so.
I think the opposite. I've had issues with TextIOWrapper because it does (unexpectedly) take ownership of the IO object you give it, and closes it. Learned that the hard way, and had to replace TextIOWrapper by `codecs` reader and writer.
> There is nothing useful you can do with an IO object once a CSV reader has used it
You can seek the data back to the start to re-read it differently. Though I'll acknowledge that it's significantly more useful for a csv writer (and then if reader and writer diverge you've created an inconvenient asymmetry).
> This is compounded by the fact that there is no way to obtain the underlying IO object from a CSV reader, so you have to hold on to the files separately.
The problem here isn't the CSV library, it's that the Load function divorces the context manager from the actual processing context. It's a bad abstraction.
def load(filepath):
return csv.reader(filepath)
def main:
...
with load(myFilepath) as reader:
for item in reader:
...
where the CSV reader both opens and closes the file, or
def load(file):
return csv.reader(file)
def main:
...
with open(myFilepath) as file
for item in load(file):
...
where the CSV reader is ignorant about opening or closing anything. If desired, add a method to the first API that takes an open file, but don’t close it if callers do
> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource.
I don't think it's premature, but properly closing files is definitely not about optimization. Maybe "a notably higher standard of code quality"?
By automatic closing do you mean not at all and expecting the GC to do it? Or with a context manager.
I think trusting an implementation detail of the underlying interpreter is lazy at best and an anti-pattern at worst. We should aim to show our intent as much as possible, and closing a file or using a context manager certainly expresses our intent.
I mean imagine a hypothetical language where it was idiomatic to let the GC close files for you and the language was specified in such a way that it was guaranteed to work efficiently and wasn't just an implementation detail.
Would code in that language have lower code quality than code in a language where files have to be closed manually? That's what the other poster seemed to be implying, but I think the opposite is true
I don’t quite see what you mean by ”efficiently”. A language where garbage collection of file objects is guaranteed to happen within N milliseconds? The issue is holding on to an external resource for longer than needed. and by definition, garbage collection is deferred cleanup.
I mean "efficiently" in the sense that the consequences of having extraneous open file objects are managed appropriately so they don't blow up the system. I don't think that necessarily means collecting them all right away, but that's one option
Well, for many resources I would agree with you, but for files specifically, closing a file usually does a lot of things, which the article completely forgets about.
In particular, write buffers are almost never flushed unless you do it explicitly or on file handle close.
IMO part of what’s happening here is that Python’s GC model works well for memory and leaks for other resources.
Rust’s ownership model works very well for many non-memory resources IMO, and even for memory it is very safe and efficient, with the only cost being mental overhead at times. Something like Rust’s ownership system but with an easy escape hatch for “loose” resource management (ie GC) could solve a lot of the cognitive load with resources, I think. Rc is helpful, but not nearly as ergonomic as a GC language.
I think withoutboats had a similar blog post a year or two ago.
I'm not sure it is fair to attribute this to Python's GC model, in that what is commonly thought of as garbage collection is not involved with the issue being discussed.
However, in principle, this issue is very simmilar to that which garbage collectors set out to solve.
In effect, python's context managers combined with the "with" syntax recreate the sementics of C's local variables in that:
1) They can be passed outside the lexical scope in which they were defined but
2) They're usage is illegal after execution has exited the "block" in which they are defined.
3) There is nothing which prevents a programmer from keeping and attempt to use a reference to them after execution has left the code block.
The approach I take in python is the same I take in C. Most of the time, I only need to file for a short period, so I can "stack" allocate it with a context manager and let the runtime handle closing it for me. In cases where stack-like sementic do not fit, you need to use heap-like semantics and manually open/close it.
If you’re not familiar with Rust, one of the core features is that you can “pass” or “borrow” that stack-like scoping in a safe way. IE, if you create a file handle, it gets automatically cleaned up when the current scope ends… unless you transfer ownership to other code, and then it is cleaned when THAT scope ends. It also prevents references from dangling after the resource has been cleaned, even in other classes or functions.
This lets you model ownership rather than memory, and the compiler will translate that (for the most part) to memory management for you. What I was trying to say is that I think managing ownership as a first-class language feature is more generally useful than trying to abstract away the concern with something like GC.
It's not about the memory model. GC will take care of all kinds of resources.
The problem is that different kinds of resources are, well, different. Part of the confusion is when documentation talks about "scarce" resources, without explaining what this really means (or whether it's really the best word to describe the difference), so then people make assumptions, often incorrect ones.
With memory, if you leave it around for GC to eventually get to it, the only side effect is an increase in memory pressure.
But with open files, there are a lot of other observable side effects of keeping the handle open, aside from running out of file descriptors. For example, it would place significant limitations on the ability to open the same file by a different process, or by unrelated code in the same process. And on Windows, you wouldn't be able to delete such a file, or any directory that contains it directly or indirectly.
There are lots of reasons to close a file when you are done with it, especially if you are writing to that file. These considerations are not about “pythonicity”, but rather about best practices for unix.
> Any resource may be limited or require exclusivity; that's why they're called resources. Similarly one should always explicitly call dict.clear when finished with a dict.
I think there is an important distinction between resources for which there may be contention within the process, vs. external resources. It's tractable to provide sufficient certainty that in an internal resource isn't subject to problematic contention either analytically or by testing or a combination, but that's less tractable with external resources. So, use guaranteed freeing of resources (context managers, if possible, in Python) for external resources or for internal resources that you can't otherwise be reasonably sure are free of contention issues.
I'm lost between the no-true-scotsmen and strawmen the post tries to construct.
The advice about closing the descriptors is for practical reasons. There's no need for "true believers" taking that advice to extreme. Just like clearing the dict if you're dropping the reference anyway is useless.
If you can consume the file immediately, you can close it. If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.
“If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.”
It isn’t an issue if you process a few files that way (unless you’re running your system very close to the limit), but it is an issue if you have a long running process that lazily iterates over files often. Examples are iterating over a file system (say a kind of grep) or a web server where an often called entry point does this.
Sure. But I was getting at something different. You can choose your open files limit on one side and the approach (lazy or not) on the other. If you choose lazy there's no way around having to leave the file open.
I'd like to emphasize the point even further for CPython users. Here's the author's closing sentence:
>Your setup script is not going to run out of file descriptors because you wrote open("README.md").read().
In CPython, files are closed (if necessary) in __del__, so the descriptor is released immediately after that line executes, thanks to CPython's reference-counting mechanism. That idiom wouldn't cause you to run out of descriptors even in a long program.
(The author knows this. Their point is that even if you're NOT using CPython, you STILL shouldn't worry about it too much, and I think I agree.)
I know, I know, CPython's reference-counting behavior is considered an "implementation detail", but for RAM-intensive workloads which are common in Scientific Computing (especially biological imaging), it's often a necessity. So if I were to force myself to use context managers in cases like this, I'd be losing the benefits of terse code without actually preserving portability. (I didn't have portability in the first place.)
In other words, if you're dependent on ref-counting anyway, you may as well admit it and enjoy the benefits that come with it!
The Python standard makes no guarantee, but in CPython, the example above is deterministic. The cycle-detecting garbage collector is not involved in this case. Just refcounting.
If open() throws an exception, perhaps after the FD is already open (Python does non-trivial IO to a file immediately after open), or read() throws an exception, the file object is at the mercy of any try/except higher in the stack, any bugs/"features" contained in those, and any sys.excepthook installed like Sentry
The example above is deterministic only on current versions of CPython and only if it is guaranteed to succeed and if it is the only line of code in the program now and in perpetuity.
> The author knows this. Their point is that even if you're NOT using CPython, you STILL shouldn't worry about it too much, and I think I agree.
The author is completely wrong and so are you.
Used to be you could not install some libraries in pypy because setuptools leaked fds handles, and if your installation was not simple enough it’d run out.
You could not install libraries because some folks couldn’t be arsed to properly close their fds.
Hmm. I just ran out of file descriptors on a little personal project of mine just a few days ago.
I kept calling pty.openpty() and I forgot to close them. My program was calling it three times (stdout, stderr, stdin) every 10 seconds :(
I ended up writing a context manager for it and now it works well. Unfortunately contextlib.closing() wouldn't work as pty.openpty() returns two objects.
- reference counting is an implementation detail, it is not part of the spec
- that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)
- even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles
- the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all..
- the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production
The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown.
Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code"
Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..
By default Windows locks a file while it is open and does not let you delete it. Other platforms do not, so code which naively reads and then deletes a file without explicitly closing it will work fine until you try to run it on Windows.
Unclosed file handles are incredibly painful on Windows. The user can't edit or delete the file while it's in use and gets cryptic errors about it. There is no UI showing which process is holding the file handle.
The standard practice on Windows when faced with a file that's got a stuck handle is to reboot.
Wow, thank you for putting into writing one of my greatest pet peeves about Windows. Never thought about how often I encounter this issue until now, and how bad the UX is.
Honestly, everything about the Explorer feels 2 decades behind he rest of Windows 10. Like, the rest of MS is has moved on, but their file browser is like an IE6-esque albatross. Everything makes it lock up.
Wacko conspiracy theory: Microsoft really wants the end user completely separated and unaware of the file system so the user can't tell the difference between cloud and local files. This allows Microsoft to create dependency on its own cloud services, the government to have access to everyone's files, and laptop makers to cut down local storage significantly.
Therefore Explorer is no longer going to get any love from Microsoft.
That does depend how the file is opened. If it's opened with full file share rights (read/write/delete) then it won't block access.
Delete is different in the sense that deleting a file is just adding a flag which tells the OS to delete the file once all file handles are closed. However newer versions of Windows 10 use POSIX delete semantics by default on NTFS. This means the file will be deleted immediately.
> However newer versions of Windows 10 use POSIX delete semantics by default on NTFS. This means the file will be deleted immediately.
Is that true? I thought POSIX deletion semantics only happened if FILE_DISPOSITION_POSIX_SEMANTICS is passed to SetFileInformationByHandle function (at FileDispositionInfoEx level), otherwise traditional Windows deletion semantics applied. (If POSIX semantics was the default, that might break legacy apps which were not expecting it – the kind of breaking change that Microsoft is usually careful to avoid.)
If you use the DeleteFileW function then it handles all that for you, including setting FILE_DISPOSITION_POSIX_SEMANTICS. At least it did the last time I tested it. But sure, don't take my word for it. Test it.
Edit: Just tested again. I'm not sure how you create files with FILE_SHARE_DELETE in Python so I used another program to create a file and hold the handle open. I then used `os.remove` to successfully delete the file while the handle was open. I could also delete the file from Windows Explorer. This was on Windows 10 1909 on the machine I use for development, thus it is possible that I've set some experimental option somewhere that affects Windows' behaviour. So further tests are welcome.
I am just surprised that Microsoft would change the default behaviour of DeleteFileW in a Windows 10 build. It seems like the kind of thing that would cause backward compatibility issues. Indeed, I've even found people reporting things breaking due to this [1]. Maybe Microsoft's culture doesn't put the same emphasis on backward compatibility that it used to?
For a time Adobe Acrobat would automatically open all PDF files on all USB sticks on my computer. Since at the time moving PDFs around was my main reason to use USB sticks it didn't took long until I simply decided to remove Adobe Acrobat from my computer.
What are you all doing where deleting a file that’s in use is such a primary concern, or even at all desirable?
I’ve had as much annoyance with files that have been “deleted” but are still using disk space (and still being used!) on Linux, as files that can’t be deleted because they’re in use on Windows.
At least “can’t delete it because it’s in use” gives you a clue what’s going on, rather than a mystery amount of missing “free” disk space.
> - reference counting is an implementation detail, it is not part of the spec
> - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)
Exactly. That was a major if not the major drive for context managers & properly closing files: the PFS and the language team were getting much more supportive of third-party implementations, and much more aware of issues in "normal" Python code e.g. though it didn't work out well in the end, ctypes also got merged into the stdlib in 2.5 with the express purpose of limiting the need for writing cpython extensions to bind to native libraries.
> - the random system resource will eventually be garbage collected
Not necessarily! In fact this is one of the major problems for non-memory resources in GC'd languages: the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, if fd consumption is not matched by enough memory consumption (or a properly matching pattern thereof) it's rather possible to keep leaking fds without the GC ever running (or ever running a collection which reclaims your file objects).
For instance, back then trying to install some packages would crash under pypy due to file exhaustion, because setuptools leaked fds all over the place: https://bitbucket.org/pypy/pypy/issues/878
>the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure,
So why not teach it about fd pressure?
There's no reason not to teach a GC about fd pressure - the reason that's rarely done is that language implementers generally don't care that much about the system interface - they just link against libc and call it done.
All objects subject to GC have a memory impact, very few of them have an fd impact. You either have to separately track those that impact fds, or do a lot (up to 100%) unnecessary work in response to fd pressure. Even if user code can disable the fd pressure response, you've got to carry the baggage of a GC with extra code to handle it around. It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language. If you had a facility for swapping in special purpose GCs for particular applications, building a specialty GC that added fd awareness might be a net win for some apps, but...
>It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language.
Yes, that's what I said:
>language implementers generally don't care that much about the system interface - they just link against libc and call it done.
There's no research done in this area (at least I don't know about any). I'm sure if people cared about it, they would find good solutions. But they mostly don't care about the system interface - which leads to the problems that this article is talking about.
(Not that Python is on the fore-front of CS research - it's downstream of more innovative languages - but if those languages cared and had done research about the system interface, they would have come up with good ideas that might flow down to Python)
_The csv version returns an iterator over a closed file_
That's why method/function naming is critical and underestimated. Without having worked with python too much:
_json.load()_ is a load: implies the action is done on return. I understand there's nothing to do after it, other than use the return value.
_csv.reader()_ clearly returns an object that it's going to be used to traverse the underlying, so the underlying must remain open. Implies a stateful operation to me.
IMO, the author is looking at the central problem with GC systems: non-deterministic destruction, and unclear ownership.
> This trivial example has deeper implications. If one accepts this practice, then one must also accept that storing a file handle anywhere, such as on an instance, is also disallowed. Unless of course that object then virally implements it owns context manager, ad infinitum.
Yes, and I just accept that. This happens in C#, too[1]:
> If your class owns a field or property, and its type implements IDisposable, the containing class itself should also implement IDisposable. A class that instantiates an IDisposable implementation and storing it as an instance member, is also responsible for its cleanup.
And for good reasons; in this regard, the languages are basically identical. I'd say any language that uses GC and instead has some sort of language-based guard system (Python's `with`, C#'s `using`) or even just through convention (e.g., Java, back before try-with-resources) will end up having that happen. Depending on the GC to collect these resources isn't wise, since no behavior on the GC is actually guaranteed.
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
I'd call that fixed; the file is properly and deterministically closed, when the generator is exhausted or manually closed. (By calling the generator's `close()` member function.) Now, it is now up to the calling code to remember at every use site to do that, and it should be made an explicit part of the documentation for the interface. But, the same for context managers, or things having a close: it's part of the contract that you will call that .close(), or wrap it in a with block, or the resource leaks until the GC gets to it. (I do wish generators were context managers, to make this easier.)
IME, the bigger problem here is that GC forces, for resources that have a destructor beyond "free the memory" (that is, what the GC itself handles), for each call-site to remember to clean up the resource. Files are one thing, and generally, yes, you'll probably get lucky and not run into issues. Locks are more fun, and often leaving it to the GC to release a lock is a lot more painful. (Then there are threads, child processes, sockets, …) RAII fixes this for all types of resources by bounding the lifetime of the resource to the scope(s) that it lives in.
(And, because it inevitably comes up, IME, people worry too much about cycles; cycles do happen, but are incredibly rare compared to non-memory resources (those that are more than just memory, e.g., files, locks, etc.), and vanishingly rare compared to all resources (those that are more than just memory and those that are just memory, like vectors or strings). Yes, they are an issue of sorts in RAII, but there are ways to handle them. I do not think the bad with cycles outweighs the good of RAII.)
It's not a problem with gc in general. Erlang for example you typically don't close files if you know the lifetime of your process is transient, you do, if it's permanent. End of rule.
Is that really idiomatic? I don't write much erlang but it seems like that wouldn't work for any code shared between two processes, one transient and one permanent.
Why would you share file opening code between two such processes. That sounds like premature abstraction. If you're really doing something in a library like the gp csv example then yeah, close it.
Well, RAII 'fixes' the problem by making it uniformly complicated, instead of uniformly simple.
For memory resources, there simply is no ownership in GC-land. This frees you to use patterns that you just can't implement in RAII-only. Famously, atomic compare-and-swap for heap-allocated objects is much harder to implement with manual memory management (who owns the old pointer after a successful CAS?)[0].
And sure, situations where ownership is not clear are rare [1], but so are file handles that aren't immediately closed, so in practice GC + local RAII seems to cover a huge amount of cases.
> The csv version returns an iterator over a closed file. It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface. Moreover, according to this best practice, it's impossible to write the csv version correctly. As absurd as it sounds, it's just an abstraction that can't exist.
This is simply not true. A load being lazy or not will often be a part of API documentation. In some cases, documentation will include that a .close() needs to be manually called. It's also trivial to read an entire CSV file, close it, and return an array of lines, or a dataframe, or whatever. Of course, you probably don't need to do it with a generator, and it won't be a two-liner (although @Izkata gives a two-liner example). In that implementation, you'd achieve behavior parity with json.loads(...).
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
I'm no fan of Python (even though I use it every day), but this is extremely well documented in PEP 342[1]. I'm honestly not sure what the complaint is here. I guess the standard Python CSV library is a bit wonky, but that's about it.
This is like a worse version of javascript semicolons.
Sure you dont HAVE to... but there's gotchas and corner cases to not doing it that you don't have to think about if you close your file desciptors when you're done with them
This is to some extent a Python-specific problem. In most languages with destructors, you're guaranteed a destructor call when a local variable's scope is exited. But not Python.
Python isn't RAII, it's an explicit close language. That's what "with" is for. Python's "with" clauses, unusually, play well with exceptions at scope exit. Nested "with" clauses can fault out with a close exception and still have everything unwind properly. Very few languages get that right. Python probably has the most useful exception system around.
Calling destructors from the garbage collector has a whole other set of problems. Suppose you're writing to a network and the network connection fails during close. Now you can be stuck in the garbage collector calling the destructor waiting for the network to time out. I'm not sure how garbage-collected Python implementations handle this. C# can potentially get into that state, too. Anybody know?
Raising an exception in a C++ destructor is usually not happy-making. Microsoft tried to make that work in Managed C++, which has "re-animation". A destructor can pass a reference to its own object outside of the destructor, which "re-animates" the object. Turned out not to be a good idea.
I would have liked to see exceptions in Rust, which has enough static ownership analysis to determine whether a destructor is trying to make something outlive its lifetime. Most of the problems with destructors vs exceptions involve scoping issues. If such errors are caught at compile time, exceptions are better behaved.
In most managed-memory languages, try...finally (and using/try-with-resources which are based on it) handle exceptions just fine. The only problem that sometimes occurs is that exceptions thrown by a finally handler override the exception being bubbled, which can be bad if you wanted to handle that exception later on, but otherwise the resources get cleared properly. D has better behavior here, where throwing an exception on this path will automatically keep track of many in-flight exceptions.
Also, the problem of finalizers blocking is partly handled by the fact that finalizers are not run by the GC threads, but by a separate finalizer thread. Still, if a finalizer blocks, eventually you will run out memory, as all objects which require finalization will be pooling up.
Also, the re-animation behavior you describe is common to finalizers in all GC languages which expose them, it's not in any way specific to Managed C++ (C++/CLR). It is simply a consequence of how finalization works: when an object which requires finalization is eligible for GC, it is not collected, but moved to the finalization queue. Objects in that queue are still live objects. When the finalizer thread successfully runs the finalizer of such an object, it is simply removed from the finalization queue. If it's finalizer has added back a reference from some other GC root to the object, then it will still stay live. The only special property is that an object can't be finalized twice, so even if its finalizer attaches it to the live object graph once, when it becomes unreachable again it will be collected, it will not be put back in the finalization queue, it will be directly collected.
Basically, I don't think there is too much chance to find a valid use for this behavior, but it is very hard to prevent it without complicating the language significantly.
> Basically, I don't think there is too much chance to find a valid use for this behavior, but it is very hard to prevent it without complicating the language significantly.
One approach is to provide a mechanism for cleanup that doesn't require the object to still be alive. Java has phantom references for this:
Sure, there are other possible cleanup mechanisms, and I believe Java is depreciating finalizers in favor of those other mechanisms. My point was only about finalizers themselves: a language with the finalizer mechanism I was describing above would need to add a lot of features to prevent finalized objects from becoming reachable again.
> One approach is to provide a mechanism for cleanup that doesn't require the object to still be alive. Java has phantom references for this:
Java has pretty much deprecated finalizers due to how non-deterministic their run is. Modern java uses try-with-resources blocks which… are the exact same thing as Python's with.
> That's what "with" is for. Python's "with" clauses, unusually, play well with exceptions at scope exit. [...] Very few languages get this right.
I wonder if I'm missing something subtle here. What languages with garbage collection, exceptions and higher order functions get this wrong? I'd expect this to work fine in common lisp, smalltalk, ruby, or pretty much anything else that offers a similar idiom -- either by macros or by passing thunks.
> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource.
Maybe the same standard should be applied to other kinds of resources. Maybe garbage collection was a mistake.
> It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface.
Lazily-loaded things are different from eagerly loaded things in important ways. If it's not possible to make that distinction visible in your language, get a better language.
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
Which is exactly what we wanted. It is fixed.
> Furthermore it demonstrates that often the context is not being managed locally. If a file object is passed another function, then it's being used outside of the context. Let's revisit the json version, which works because the file is fully read. Doesn't a json parser have some expensive parsing to do after it's read the file? It might even throw an error. And isn't it be desirable, trivial, and likely that the implementation releases interest in the file as soon as possible?
Look at iteratees for the full solution to this problem. Managing context can be hard, but it doesn't have to be; you just have to use the right abstractions.
> Those threads are essentially debating whether a resource pool is "leaking" resources.
If you open a pool and never close it, of course you're leaking everything in that pool.
> if all this orphaned file descriptor paranoia was well-founded, it would have been a problem back then too.
It was. Long-running python programs broke because they ran out of file descriptors.
I wasn’t aware of these functions. At first glance, their addition to Python seems to be a significant violation of “one-- and preferably only one --obvious way to do it”.
What are the advantages of `read_text` over the traditional `file` read methods?
GC advocates say that managing memory lifetimes is an excessive burden to programmers.
Anti-GC advocates say that memory is just one particular instance of management of life-timed resources, and that special casing it is an ugly hack and you should be using RAII. Rust probably is the leading exponent (C++'s way is more of a mess).
The RAII critics point out that RAII also glosses over part of the problem because often de-allocating resources can itself fail (e.g. closing a file) and RAII (in Rust or C++) has no proper way to deal with that.
Then, there is the Reference counting camp, the only people who can say that they solve the something close to the general automatic resource de-allocation problem -- because lifetimes are precisely tracked. But reference counting can't cope with cycles and generally has poor performance. Also, it does not directly help with de-allocation failures. One interesting recent variation are attempts at static reference counting: http://strlen.com/lobster/.
Last there are the old and the new muddle through camp. The old muddle through camp (e.g. java, python) adds finalizers with crap semantics (call-me-maybe). The new muddle-through camp uses GC for memory but adds some extra mechanism for other resources (defer in go, <close> in lua 5.4). Defer in go seems like an inferior version of rust-style RAII. Lua 5.4's <close> also has/had problems (see http://lua-users.org/lists/lua-l/2019-06/msg00179.html).
The author of the article advocates old-style muddling through, but this feels unsatisfactory because it's not robust in general and likely worse than following current "best practice". But it still seems like there is still no really good general solution available. Rust-style ownership tracking would have solved the lifetime ambiguity of the file for the csv reader, but would not be able to deal with file close errors properly. These can be somewhat handled with exception based mechanisms, but not without problems either -- because multiple things can go out of scope at once, requiring cleanup and one or more of these cleanups may fail themselves but try catch statements (with stack unwinding) offer no natural way to deal with multiple failures being propagated up at once.
114 comments
[ 3.7 ms ] story [ 179 ms ] threadUm, just consume the iterator immediately, then it's the same as the json one:
The equivalent for the latter json example that separates the read and parse would be:That's fine for some use cases, but TFA wanted a lazy iterator.
That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.
There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
That's both right and wrong. That it's a lazy iterator itself may be an implementation detail that doesn't effect the interface (if the interface doesn't assure reusability; a concrete collection and a lazy iterator do have different behavior in important ways) but leaving a dangling external resource is not. For certain resources, when you are given a reference rather than the opened resource, you can't build a lazy iterator without also leaving dangling external resource that the caller doesn't have the information to close. That is a significant difference.
It certainly isn't an implementation detail. How can you write a correct program without knowing this behavior?
> """with a different file format"""
The assumption behind the paragraph following the example does not imply lazy evaluation, but rather the opposite - the broken one is obvious if you have lazy evaluation in mind, whereas the docstrings imply the caller would be using these the same and not be expecting lazy evaluation.
Pass the responsibility as far up as required. Make the `load` function return a context manager. So any use of load would be like:
`load` could be written with `contextlib.contextmanager` as:It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory, while the same is often impractical and unnecessary for a CSV (when it is millions of lines). The standard methods and idioms of doing these things in Python reflects this.
It may be normal, but there's no reason it is required. Streaming JSON processing makes sense just like streaming XML processing.
http://jsonlines.org/
In other words, collection of JSON strings in a file != a JSON file.
Given that the CSV reader does not control ownership of the file handle, it shouldn't return a lazy iterator when given a file handle. It should accept a path, then it could own the file handle until the iterator is complete or dropped.
So, yea, the CSV reader does a bad. But so what? That is a deficiency of the CSV library, not context managers.
OK, it might still be correct to say the problem is with the CSV library. But if the context manager "best practices" lead to popular parts of the standard library being bad and unidiomatic, that's at best unpythonic. After all, "There should be one -- and preferably only one -- obvious way to do it".
[1] https://docs.python.org/3/library/csv.html#csv.reader
I’ve used the CSV library many times with an in memory TextIO object (like an http response). It’s an important distinction.
TextIO has a close() method, so this is irrelevant.
Rather, the author of the CSV library made a design decision about whether a CSV reader should take ownership of the underlying IO or not. If it takes ownership, it has the responsibility of closing the file when it's finished. If it doesn't, it doesn't. The decision was made for it not to take ownership.
Was this a good decision? Personally, i don't think so. There is nothing useful you can do with an IO object once a CSV reader has used it, so there is no point retaining ownership separately. It just makes more work for the programmer. This is compounded by the fact that there is no way to obtain the underlying IO object from a CSV reader, so you have to hold on to the files separately.
In the normal use case where you open a single file and then read from it in a single block of code, this is not a big deal, because you can wrap that in a with block for the file. But as the author points out, it's a serious impediment to using lazy readers. I have also run into this problem when managing multiple open files; i have a tool which opens a bunch of CSV files and merges them in some strange way, and it is sensible to close each file when it is finished, but to do that, i need to hang on to the files, and build some machinery to close the right one at the right time.
On the other hand, i can't think of any downsides to deciding, way back when, that CSV readers should close their underlying files. But it's not safe to change that decision now, so we're stuck with it.
I think the opposite. I've had issues with TextIOWrapper because it does (unexpectedly) take ownership of the IO object you give it, and closes it. Learned that the hard way, and had to replace TextIOWrapper by `codecs` reader and writer.
> There is nothing useful you can do with an IO object once a CSV reader has used it
You can seek the data back to the start to re-read it differently. Though I'll acknowledge that it's significantly more useful for a csv writer (and then if reader and writer diverge you've created an inconvenient asymmetry).
> This is compounded by the fact that there is no way to obtain the underlying IO object from a CSV reader, so you have to hold on to the files separately.
Not exactly a difficult feat.
No, it's not. It's a deficiency of the proposed use of the CSV library, which is just dumb.
Leaving a dangling external resource open isn't an irrelevant implementation detail.
I don't think it's premature, but properly closing files is definitely not about optimization. Maybe "a notably higher standard of code quality"?
It seems to me like it would be the opposite, automatic closing means less boilerplate and more expressive code
I think trusting an implementation detail of the underlying interpreter is lazy at best and an anti-pattern at worst. We should aim to show our intent as much as possible, and closing a file or using a context manager certainly expresses our intent.
Would code in that language have lower code quality than code in a language where files have to be closed manually? That's what the other poster seemed to be implying, but I think the opposite is true
In particular, write buffers are almost never flushed unless you do it explicitly or on file handle close.
Rust’s ownership model works very well for many non-memory resources IMO, and even for memory it is very safe and efficient, with the only cost being mental overhead at times. Something like Rust’s ownership system but with an easy escape hatch for “loose” resource management (ie GC) could solve a lot of the cognitive load with resources, I think. Rc is helpful, but not nearly as ergonomic as a GC language.
I think withoutboats had a similar blog post a year or two ago.
However, in principle, this issue is very simmilar to that which garbage collectors set out to solve.
In effect, python's context managers combined with the "with" syntax recreate the sementics of C's local variables in that:
1) They can be passed outside the lexical scope in which they were defined but
2) They're usage is illegal after execution has exited the "block" in which they are defined.
3) There is nothing which prevents a programmer from keeping and attempt to use a reference to them after execution has left the code block.
The approach I take in python is the same I take in C. Most of the time, I only need to file for a short period, so I can "stack" allocate it with a context manager and let the runtime handle closing it for me. In cases where stack-like sementic do not fit, you need to use heap-like semantics and manually open/close it.
This lets you model ownership rather than memory, and the compiler will translate that (for the most part) to memory management for you. What I was trying to say is that I think managing ownership as a first-class language feature is more generally useful than trying to abstract away the concern with something like GC.
The problem is that different kinds of resources are, well, different. Part of the confusion is when documentation talks about "scarce" resources, without explaining what this really means (or whether it's really the best word to describe the difference), so then people make assumptions, often incorrect ones.
With memory, if you leave it around for GC to eventually get to it, the only side effect is an increase in memory pressure.
But with open files, there are a lot of other observable side effects of keeping the handle open, aside from running out of file descriptors. For example, it would place significant limitations on the ability to open the same file by a different process, or by unrelated code in the same process. And on Windows, you wouldn't be able to delete such a file, or any directory that contains it directly or indirectly.
I think there is an important distinction between resources for which there may be contention within the process, vs. external resources. It's tractable to provide sufficient certainty that in an internal resource isn't subject to problematic contention either analytically or by testing or a combination, but that's less tractable with external resources. So, use guaranteed freeing of resources (context managers, if possible, in Python) for external resources or for internal resources that you can't otherwise be reasonably sure are free of contention issues.
The advice about closing the descriptors is for practical reasons. There's no need for "true believers" taking that advice to extreme. Just like clearing the dict if you're dropping the reference anyway is useless.
If you can consume the file immediately, you can close it. If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.
It isn’t an issue if you process a few files that way (unless you’re running your system very close to the limit), but it is an issue if you have a long running process that lazily iterates over files often. Examples are iterating over a file system (say a kind of grep) or a web server where an often called entry point does this.
>Your setup script is not going to run out of file descriptors because you wrote open("README.md").read().
In CPython, files are closed (if necessary) in __del__, so the descriptor is released immediately after that line executes, thanks to CPython's reference-counting mechanism. That idiom wouldn't cause you to run out of descriptors even in a long program.
(The author knows this. Their point is that even if you're NOT using CPython, you STILL shouldn't worry about it too much, and I think I agree.)
I know, I know, CPython's reference-counting behavior is considered an "implementation detail", but for RAM-intensive workloads which are common in Scientific Computing (especially biological imaging), it's often a necessity. So if I were to force myself to use context managers in cases like this, I'd be losing the benefits of terse code without actually preserving portability. (I didn't have portability in the first place.)
In other words, if you're dependent on ref-counting anyway, you may as well admit it and enjoy the benefits that come with it!
A “with” statement ensures the release happens deterministically, which it turns out is a desirable trait to have.
The example above is deterministic only on current versions of CPython and only if it is guaranteed to succeed and if it is the only line of code in the program now and in perpetuity.
The author is completely wrong and so are you.
Used to be you could not install some libraries in pypy because setuptools leaked fds handles, and if your installation was not simple enough it’d run out.
You could not install libraries because some folks couldn’t be arsed to properly close their fds.
I kept calling pty.openpty() and I forgot to close them. My program was calling it three times (stdout, stderr, stdin) every 10 seconds :(
I ended up writing a context manager for it and now it works well. Unfortunately contextlib.closing() wouldn't work as pty.openpty() returns two objects.
- reference counting is an implementation detail, it is not part of the spec
- that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)
- even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles
- the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all..
- the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production
The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown.
Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code"
Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..
The standard practice on Windows when faced with a file that's got a stuck handle is to reboot.
Connecting to a slow network share? Rclicked and one context extension is acting up? Just feeling cranky today? Well (not responding).
And if you want to close the (not responding) window, I hope you didn't want any of your other windows, including you in-progress file copies.
Therefore Explorer is no longer going to get any love from Microsoft.
You can’t do this in Windows and that’s also why you can’t delete in-use files or directories with files inside
Delete is different in the sense that deleting a file is just adding a flag which tells the OS to delete the file once all file handles are closed. However newer versions of Windows 10 use POSIX delete semantics by default on NTFS. This means the file will be deleted immediately.
Is that true? I thought POSIX deletion semantics only happened if FILE_DISPOSITION_POSIX_SEMANTICS is passed to SetFileInformationByHandle function (at FileDispositionInfoEx level), otherwise traditional Windows deletion semantics applied. (If POSIX semantics was the default, that might break legacy apps which were not expecting it – the kind of breaking change that Microsoft is usually careful to avoid.)
Edit: Just tested again. I'm not sure how you create files with FILE_SHARE_DELETE in Python so I used another program to create a file and hold the handle open. I then used `os.remove` to successfully delete the file while the handle was open. I could also delete the file from Windows Explorer. This was on Windows 10 1909 on the machine I use for development, thus it is possible that I've set some experimental option somewhere that affects Windows' behaviour. So further tests are welcome.
I am just surprised that Microsoft would change the default behaviour of DeleteFileW in a Windows 10 build. It seems like the kind of thing that would cause backward compatibility issues. Indeed, I've even found people reporting things breaking due to this [1]. Maybe Microsoft's culture doesn't put the same emphasis on backward compatibility that it used to?
[1] https://stackoverflow.com/questions/60424732/did-the-behavio...
What about process explorer?
https://docs.microsoft.com/en-us/sysinternals/downloads/proc...
I’ve had as much annoyance with files that have been “deleted” but are still using disk space (and still being used!) on Linux, as files that can’t be deleted because they’re in use on Windows.
At least “can’t delete it because it’s in use” gives you a clue what’s going on, rather than a mystery amount of missing “free” disk space.
> - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython)
Exactly. That was a major if not the major drive for context managers & properly closing files: the PFS and the language team were getting much more supportive of third-party implementations, and much more aware of issues in "normal" Python code e.g. though it didn't work out well in the end, ctypes also got merged into the stdlib in 2.5 with the express purpose of limiting the need for writing cpython extensions to bind to native libraries.
> - the random system resource will eventually be garbage collected
Not necessarily! In fact this is one of the major problems for non-memory resources in GC'd languages: the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, if fd consumption is not matched by enough memory consumption (or a properly matching pattern thereof) it's rather possible to keep leaking fds without the GC ever running (or ever running a collection which reclaims your file objects).
For instance, back then trying to install some packages would crash under pypy due to file exhaustion, because setuptools leaked fds all over the place: https://bitbucket.org/pypy/pypy/issues/878
The stdlib also did: https://bugs.python.org/issue13133
So why not teach it about fd pressure?
There's no reason not to teach a GC about fd pressure - the reason that's rarely done is that language implementers generally don't care that much about the system interface - they just link against libc and call it done.
All objects subject to GC have a memory impact, very few of them have an fd impact. You either have to separately track those that impact fds, or do a lot (up to 100%) unnecessary work in response to fd pressure. Even if user code can disable the fd pressure response, you've got to carry the baggage of a GC with extra code to handle it around. It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language. If you had a facility for swapping in special purpose GCs for particular applications, building a specialty GC that added fd awareness might be a net win for some apps, but...
Yes, that's what I said:
>language implementers generally don't care that much about the system interface - they just link against libc and call it done.
There's no research done in this area (at least I don't know about any). I'm sure if people cared about it, they would find good solutions. But they mostly don't care about the system interface - which leads to the problems that this article is talking about.
(Not that Python is on the fore-front of CS research - it's downstream of more innovative languages - but if those languages cared and had done research about the system interface, they would have come up with good ideas that might flow down to Python)
That's why method/function naming is critical and underestimated. Without having worked with python too much:
_json.load()_ is a load: implies the action is done on return. I understand there's nothing to do after it, other than use the return value.
_csv.reader()_ clearly returns an object that it's going to be used to traverse the underlying, so the underlying must remain open. Implies a stateful operation to me.
Besides, examples are clear:
https://docs.python.org/3.8/library/csv.html#examples
Classical problem of wanting to use The Perfect Abstraction.
No, it's not:
This both uses csv.reader and fulfills the required contract correctly.Yes, it's tricky to both follow the “assure a resource is closed” pattern and return a lazy iterator over the external resource which you open.
If you are going to lazily iterate over a resource and do the context manager pattern, you have to do the work within the context manager.
> This trivial example has deeper implications. If one accepts this practice, then one must also accept that storing a file handle anywhere, such as on an instance, is also disallowed. Unless of course that object then virally implements it owns context manager, ad infinitum.
Yes, and I just accept that. This happens in C#, too[1]:
> If your class owns a field or property, and its type implements IDisposable, the containing class itself should also implement IDisposable. A class that instantiates an IDisposable implementation and storing it as an instance member, is also responsible for its cleanup.
And for good reasons; in this regard, the languages are basically identical. I'd say any language that uses GC and instead has some sort of language-based guard system (Python's `with`, C#'s `using`) or even just through convention (e.g., Java, back before try-with-resources) will end up having that happen. Depending on the GC to collect these resources isn't wise, since no behavior on the GC is actually guaranteed.
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
I'd call that fixed; the file is properly and deterministically closed, when the generator is exhausted or manually closed. (By calling the generator's `close()` member function.) Now, it is now up to the calling code to remember at every use site to do that, and it should be made an explicit part of the documentation for the interface. But, the same for context managers, or things having a close: it's part of the contract that you will call that .close(), or wrap it in a with block, or the resource leaks until the GC gets to it. (I do wish generators were context managers, to make this easier.)
IME, the bigger problem here is that GC forces, for resources that have a destructor beyond "free the memory" (that is, what the GC itself handles), for each call-site to remember to clean up the resource. Files are one thing, and generally, yes, you'll probably get lucky and not run into issues. Locks are more fun, and often leaving it to the GC to release a lock is a lot more painful. (Then there are threads, child processes, sockets, …) RAII fixes this for all types of resources by bounding the lifetime of the resource to the scope(s) that it lives in.
(And, because it inevitably comes up, IME, people worry too much about cycles; cycles do happen, but are incredibly rare compared to non-memory resources (those that are more than just memory, e.g., files, locks, etc.), and vanishingly rare compared to all resources (those that are more than just memory and those that are just memory, like vectors or strings). Yes, they are an issue of sorts in RAII, but there are ways to handle them. I do not think the bad with cycles outweighs the good of RAII.)
[1]: https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...
For memory resources, there simply is no ownership in GC-land. This frees you to use patterns that you just can't implement in RAII-only. Famously, atomic compare-and-swap for heap-allocated objects is much harder to implement with manual memory management (who owns the old pointer after a successful CAS?)[0].
And sure, situations where ownership is not clear are rare [1], but so are file handles that aren't immediately closed, so in practice GC + local RAII seems to cover a huge amount of cases.
[0] https://www.drdobbs.com/lock-free-data-structures/184401865
[1] Rust could work as a case study here: how many places require `unsafe`?
This is simply not true. A load being lazy or not will often be a part of API documentation. In some cases, documentation will include that a .close() needs to be manually called. It's also trivial to read an entire CSV file, close it, and return an array of lines, or a dataframe, or whatever. Of course, you probably don't need to do it with a generator, and it won't be a two-liner (although @Izkata gives a two-liner example). In that implementation, you'd achieve behavior parity with json.loads(...).
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
I'm no fan of Python (even though I use it every day), but this is extremely well documented in PEP 342[1]. I'm honestly not sure what the complaint is here. I guess the standard Python CSV library is a bit wonky, but that's about it.
[1] https://legacy.python.org/dev/peps/pep-0342/
Sure you dont HAVE to... but there's gotchas and corner cases to not doing it that you don't have to think about if you close your file desciptors when you're done with them
Python isn't RAII, it's an explicit close language. That's what "with" is for. Python's "with" clauses, unusually, play well with exceptions at scope exit. Nested "with" clauses can fault out with a close exception and still have everything unwind properly. Very few languages get that right. Python probably has the most useful exception system around.
Calling destructors from the garbage collector has a whole other set of problems. Suppose you're writing to a network and the network connection fails during close. Now you can be stuck in the garbage collector calling the destructor waiting for the network to time out. I'm not sure how garbage-collected Python implementations handle this. C# can potentially get into that state, too. Anybody know?
Raising an exception in a C++ destructor is usually not happy-making. Microsoft tried to make that work in Managed C++, which has "re-animation". A destructor can pass a reference to its own object outside of the destructor, which "re-animates" the object. Turned out not to be a good idea.
I would have liked to see exceptions in Rust, which has enough static ownership analysis to determine whether a destructor is trying to make something outlive its lifetime. Most of the problems with destructors vs exceptions involve scoping issues. If such errors are caught at compile time, exceptions are better behaved.
Also, the problem of finalizers blocking is partly handled by the fact that finalizers are not run by the GC threads, but by a separate finalizer thread. Still, if a finalizer blocks, eventually you will run out memory, as all objects which require finalization will be pooling up.
Also, the re-animation behavior you describe is common to finalizers in all GC languages which expose them, it's not in any way specific to Managed C++ (C++/CLR). It is simply a consequence of how finalization works: when an object which requires finalization is eligible for GC, it is not collected, but moved to the finalization queue. Objects in that queue are still live objects. When the finalizer thread successfully runs the finalizer of such an object, it is simply removed from the finalization queue. If it's finalizer has added back a reference from some other GC root to the object, then it will still stay live. The only special property is that an object can't be finalized twice, so even if its finalizer attaches it to the live object graph once, when it becomes unreachable again it will be collected, it will not be put back in the finalization queue, it will be directly collected.
Basically, I don't think there is too much chance to find a valid use for this behavior, but it is very hard to prevent it without complicating the language significantly.
One approach is to provide a mechanism for cleanup that doesn't require the object to still be alive. Java has phantom references for this:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base...
https://stackoverflow.com/a/43659951/116639
And i learn that Smalltalk introduced the "ephemeron", which is now present in some other languages:
https://en.wikipedia.org/wiki/Ephemeron
Java has pretty much deprecated finalizers due to how non-deterministic their run is. Modern java uses try-with-resources blocks which… are the exact same thing as Python's with.
I wonder if I'm missing something subtle here. What languages with garbage collection, exceptions and higher order functions get this wrong? I'd expect this to work fine in common lisp, smalltalk, ruby, or pretty much anything else that offers a similar idiom -- either by macros or by passing thunks.
Maybe the same standard should be applied to other kinds of resources. Maybe garbage collection was a mistake.
> It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface.
Lazily-loaded things are different from eagerly loaded things in important ways. If it's not possible to make that distinction visible in your language, get a better language.
> No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected.
Which is exactly what we wanted. It is fixed.
> Furthermore it demonstrates that often the context is not being managed locally. If a file object is passed another function, then it's being used outside of the context. Let's revisit the json version, which works because the file is fully read. Doesn't a json parser have some expensive parsing to do after it's read the file? It might even throw an error. And isn't it be desirable, trivial, and likely that the implementation releases interest in the file as soon as possible?
Look at iteratees for the full solution to this problem. Managing context can be hard, but it doesn't have to be; you just have to use the right abstractions.
> Those threads are essentially debating whether a resource pool is "leaking" resources.
If you open a pool and never close it, of course you're leaking everything in that pool.
> if all this orphaned file descriptor paranoia was well-founded, it would have been a problem back then too.
It was. Long-running python programs broke because they ran out of file descriptors.
How about even simpler (python3)
Ta-da. I don't even have to worry about opening files, let alone closing.https://docs.python.org/3/library/pathlib.html#pathlib.Path....
What are the advantages of `read_text` over the traditional `file` read methods?
Anti-GC advocates say that memory is just one particular instance of management of life-timed resources, and that special casing it is an ugly hack and you should be using RAII. Rust probably is the leading exponent (C++'s way is more of a mess).
The RAII critics point out that RAII also glosses over part of the problem because often de-allocating resources can itself fail (e.g. closing a file) and RAII (in Rust or C++) has no proper way to deal with that.
Then, there is the Reference counting camp, the only people who can say that they solve the something close to the general automatic resource de-allocation problem -- because lifetimes are precisely tracked. But reference counting can't cope with cycles and generally has poor performance. Also, it does not directly help with de-allocation failures. One interesting recent variation are attempts at static reference counting: http://strlen.com/lobster/.
Last there are the old and the new muddle through camp. The old muddle through camp (e.g. java, python) adds finalizers with crap semantics (call-me-maybe). The new muddle-through camp uses GC for memory but adds some extra mechanism for other resources (defer in go, <close> in lua 5.4). Defer in go seems like an inferior version of rust-style RAII. Lua 5.4's <close> also has/had problems (see http://lua-users.org/lists/lua-l/2019-06/msg00179.html).
The author of the article advocates old-style muddling through, but this feels unsatisfactory because it's not robust in general and likely worse than following current "best practice". But it still seems like there is still no really good general solution available. Rust-style ownership tracking would have solved the lifetime ambiguity of the file for the csv reader, but would not be able to deal with file close errors properly. These can be somewhat handled with exception based mechanisms, but not without problems either -- because multiple things can go out of scope at once, requiring cleanup and one or more of these cleanups may fail themselves but try catch statements (with stack unwinding) offer no natural way to deal with multiple failures being propagated up at once.