I like the simplicity. I definitely get the payoff for standalone Python scripts, where once the script errors out the memory is cleared.
But do you see a similar payoff for Jupyter notebooks (or similar)?
This is a pretty good implementation. I like the simplicity of it, reminds me of SQLite backed storage decorators we used to have, where the data was persisted to a DB instead of the file system (altho thats just a different storage engine)
Does this also take care of the thundering heard problem? That was one of the cases where lru_cache really blows
What are you using it for? A disk based cache seems almost contradictory for my use cases, I would love to hear yours. Anything that I would store on disk, even as a cache, I can generally put in SQLite.
The degree of reduction is nice considering the countless times in the past where I wrote my own file cache logic using if/else statements, temporary files, pickle, and bespoke sqlite databases.
Why give myself the headache of maintaining so much extra code when someone already wrote it.
I write some toy Python scripts/apps for a couple APIs (the Pokeapi, Spacetraders, etc). I use the HTTPX library as a request client, and wasn't aware of the Hishel library for caching requests.
I used DiskCache to cache responses for ~15 minutes so I wasn't sending live requests every time I tested the app.
I'm not building anything "cloud scale," most things I run are off my local machine. Having a convenient, fast local cache that's simple to use (DiskCache) has so many uses, it's hard to think of them all! I might use a cache with no expiration to store some configs, or to store a serialized object for later retrieval. I might use it as an in-memory object cache while the program loads, so I don't have to spin up a Redis server.
Thread safety is a big issue with ours, we'll run into issues when two different processes attempt to write to the same location, or we'll get a bad read. This is a better solution for large scale workloads.
Ours is more meant for single-process scripts like an LLM workflow.
I was curious to see an alternative to this, but how is this an alternative? You're saying I can implement my own caching of function calls that invalidates when the arguments or source code change...? These feel like entirely separate layers. Did I miss where diskcache does this stuff?
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
It does feel like diskcache is for performance gains in production rather than dev time gains. It seems like it would be bothersome to quickly invalidate the cache in case of a bad write.
Diskcache works well, we just wanted a dependency free version that we had more control over (easier cache key deletion). I think you'd have to write a custom hashing function for diskcache to use the function source code as a key.
I'm also unsure if Diskcache supports ignoring certain fields in the function call.
Good questions! You made me check the docs because those seem like very legitimate issues. So firstly, DiskCache by default just checks the function name, not the source code, but you could hack it to include the source code. I personally usually just deleted the cache if I knew the function meaningfully changed.
If you aren’t caching LLM functions during development, then you’re an even greater glutton for punishment than the normal engineer.
My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value.
What’s cool about caching results locally to files during development is the ease of invalidating caches — just delete the file named after the function and key you want.
Weird comment... yes, they are, but what is faster for me to type `rm .cache_dir/function_name*.pickle` or just to delete the one sqlite file in my file manager / vscode file tree.
Regardless, there are other reasons why sqlite is nice for this, you gain control over locking and thread safety without having to implement it all from scratch
I feel like adding an argument to the decorator that labels the "version" of the function would make deliberate cache invalidation more straightforward for cache users.
The version input makes sense, I could also see some developers disliking that ux because of it's verbosity. But to deliberately invalidate you have to make a manual effort in either case.
To me, it's more that invalidating the cache without it requires knowledge of implementation details (i.e. where the cache is stored, and how the cache files are named) that ideally shouldn't have to "leak" for the cache to be generally useful.
(Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)
I recently wrote a version of this that I use in my projects, some things I do differently that you may or may not care about:
- from your code it seems you're not sorting kwargs, I would strongly recommend sorting them so that whether you call f(a=1, b=2) or f(b=2, a=1) the cache key is the same
- I use inspect.signature to convert all args to kwargs, this way it doesn't matter how a function gets called, the cache logic is always consistent. I know this is relatively slow but it only gets called once per function (I call it outside the wrapper) and the DX benefits are nice (in this same note, you could probably move the inspect.getsource call outside your wrapper fn for a speed boost)
I also took the opposite approach to ignore_params, and made the __dict__ params that get hashed opt-in, which works well when caching instance methods
Making the __dict__ opt-in makes it a lot more user-friendly at the expense of a little verbosity. That makes sense.
These tips make sense, we often use named args in our function calls (not using them has caused so many bugs), but we don't really enforce the order. Copilot doesn't always get it right either.
By moving inspect.getsource out of the wrapper, do you mean initializing it when the module is imported? I'm curious how that improves performance.
Yeah I too try to avoid positional args as much as possible, huge source of bugs and time wasting especially when refactoring code
Re inspect.getsource, I'm not sure if it'd be a huge performance impact, but if it's in the wrapper fn it will get called every time the function gets called, while if it's outside it will be called only when the decorator runs (eg when the module containing the function being decorated is imported).
EDIT: on a quick test, over 100k function calls, with inspect.getsource inside the wrapper it runs in 2.7s on my Apple M2, and that's not even including the md5 hash, so I suspect this should dramatically improve performance for you
Be warned. The above function is used as part of the hash. The ostensible purpose is to prevent using cached values of functions who's code has changed, but it does not handle dependencies of that function.
How do you suggest one might fix that issue? Also pin the cache to a hash of all dependency versions? And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache?
Your example isn't really a problem with OPs utility, but a specific example of a broader dependency management problem that affects just about everything. The answers usually boil down to 1) invest heavily in a kick ass test suite, 2) never upgrade or 3) upgrade and pray nothing breaks.
+1, we considered traversing the function's dependencies to key the cache on (not just the initial function source code), but decided to leave this in a as a constraint. Otherwise we also blowing up the cache when we didn't want it to happen.
> How do you suggest one might fix that issue? Also pin the cache to a hash of all dependency versions?
Pretty much. Recursively collect dependencies by analyzing the AST of the code.
> And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache?
You're saying that like it's some kind of ridiculous ask, but yes. The current implementation is already "Blow[ing] up the whole cache" whenever the code for the decorated function is changed anyways. I'd guess that additionally handling dependencies recursively would only modestly increase the rate of "Blow[ing] up the whole cache".
> Your example isn't really a problem with OPs utility...
Whether or not this is a problem in practice obviously depends on your use case. Maybe you don't generally care if functions return the correct result, but many do.
> [This is] a specific example of a broader dependency management problem that affects just about everything.
Dependency resolution is not trivial per se, but it's a pretty common problem. Every single package manager, build system (make), etc. have all solved this.
Sha1 is a better choice even for non-cryptographic use cases, it's quite a bit faster than md5. Even better would be something like xxhash!
According to a quick bash script I wrote to benchmark the popular hash functions, md5 comes out last compared to sha1, sha256, sha512, and blake2, and by a decent margin!
A good rule of thumb is to never use md5 at all. Not even for non-cryptographic use cases. It's not only broken, but also very slow!
I think python objects have a __hash__ method available on them as well that can be used for hashing. That should be even much faster than sha1, but for this use case I'm not sure how much it really matters. Would be interesting to benchmark!
joblib.Memory provides caching functions and works by explicitly saving the inputs and outputs to files. It is designed to work with non-hashable and potentially large input and output data types such as numpy arrays.
Recently, I experimented with various techniques to cache some JSON responses from FastAPI, using Python decorators for both in-memory and disk caching on a single machine. After benchmarking the performance, I found the results somewhat disappointing (500 req/s vs 5k req/s). While caching did lead to a tenfold improvement in speed compared to no caching, I believe the primary bottleneck was Python's inherent performance limitations, which made it X times slower than a comparable program written in C. Consequently, I decided to remove the cache decorator and instead put a simple nginx caching reverse proxy in front of FastAPI. This resulted in performance gains that were an order of magnitude better (60k req/s) than those achieved with Python based caching.
We also found a lot of cases where caching ends up being actually slower than doing the operation. The 100% solution would probably be to use a SQL db the way diskcache does it, but this is easier to use for us.
Reminds me of a little prototype I wrote a while ago that tried to do something similar with Javascript's Proxy class. https://github.com/emileindik/cashola
The main difference is that it stores the state of an object, not a function.
If your data is JSON serializable then it could be a cool way to save and resume application state.
59 comments
[ 0.19 ms ] story [ 126 ms ] threadI think it could help if you forget to save the output of a function within a single cell like this:
1. print(f(x)) # -> check what happened 2. out = f(x) # -> turns out we want to save this, so we have to wait again
https://bookdown.org/yihui/rmarkdown-cookbook/cache.html
Does this also take care of the thundering heard problem? That was one of the cases where lru_cache really blows
Sometimes caching can actually be slower for certain functions, because just performing that operation is faster than pickle.load/pickle.dump.
Why give myself the headache of maintaining so much extra code when someone already wrote it.
I used DiskCache to cache responses for ~15 minutes so I wasn't sending live requests every time I tested the app.
I'm not building anything "cloud scale," most things I run are off my local machine. Having a convenient, fast local cache that's simple to use (DiskCache) has so many uses, it's hard to think of them all! I might use a cache with no expiration to store some configs, or to store a serialized object for later retrieval. I might use it as an in-memory object cache while the program loads, so I don't have to spin up a Redis server.
It's a very mature library, too, nice and polished, I've never once experienced a bug with it.
Ours is more meant for single-process scripts like an LLM workflow.
mature and works well
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
I'm also unsure if Diskcache supports ignoring certain fields in the function call.
And it does support ignoring certain args yes.
My local file cache Python decorator also allows the decorator to define the hash manually, either by the decorator’s parameter function call that plucks a value from the cached function params, or by calling a global function from anywhere with any arbitrary value.
What’s cool about caching results locally to files during development is the ease of invalidating caches — just delete the file named after the function and key you want.
Regardless, there are other reasons why sqlite is nice for this, you gain control over locking and thread safety without having to implement it all from scratch
(Buy hey, we are talking about what is famously one of the hard problems in comp sci, so (respectful) disagreements on how best to do it should be expected :-)
- from your code it seems you're not sorting kwargs, I would strongly recommend sorting them so that whether you call f(a=1, b=2) or f(b=2, a=1) the cache key is the same
- I use inspect.signature to convert all args to kwargs, this way it doesn't matter how a function gets called, the cache logic is always consistent. I know this is relatively slow but it only gets called once per function (I call it outside the wrapper) and the DX benefits are nice (in this same note, you could probably move the inspect.getsource call outside your wrapper fn for a speed boost)
I also took the opposite approach to ignore_params, and made the __dict__ params that get hashed opt-in, which works well when caching instance methods
These tips make sense, we often use named args in our function calls (not using them has caused so many bugs), but we don't really enforce the order. Copilot doesn't always get it right either.
By moving inspect.getsource out of the wrapper, do you mean initializing it when the module is imported? I'm curious how that improves performance.
Re inspect.getsource, I'm not sure if it'd be a huge performance impact, but if it's in the wrapper fn it will get called every time the function gets called, while if it's outside it will be called only when the decorator runs (eg when the module containing the function being decorated is imported).
eg: https://gist.github.com/mpeg/ff1d99fde06f39916b5aaadd76b534f...
EDIT: on a quick test, over 100k function calls, with inspect.getsource inside the wrapper it runs in 2.7s on my Apple M2, and that's not even including the md5 hash, so I suspect this should dramatically improve performance for you
Your example isn't really a problem with OPs utility, but a specific example of a broader dependency management problem that affects just about everything. The answers usually boil down to 1) invest heavily in a kick ass test suite, 2) never upgrade or 3) upgrade and pray nothing breaks.
Pretty much. Recursively collect dependencies by analyzing the AST of the code.
> And then if one minor update And let's say the dependency did change, but it's generally inert (more error handling around edge cases, for example), how do you factor that in? Blow up the whole cache?
You're saying that like it's some kind of ridiculous ask, but yes. The current implementation is already "Blow[ing] up the whole cache" whenever the code for the decorated function is changed anyways. I'd guess that additionally handling dependencies recursively would only modestly increase the rate of "Blow[ing] up the whole cache".
> Your example isn't really a problem with OPs utility...
Whether or not this is a problem in practice obviously depends on your use case. Maybe you don't generally care if functions return the correct result, but many do.
> [This is] a specific example of a broader dependency management problem that affects just about everything.
Dependency resolution is not trivial per se, but it's a pretty common problem. Every single package manager, build system (make), etc. have all solved this.
Sha1 is a better choice even for non-cryptographic use cases, it's quite a bit faster than md5. Even better would be something like xxhash!
According to a quick bash script I wrote to benchmark the popular hash functions, md5 comes out last compared to sha1, sha256, sha512, and blake2, and by a decent margin!
A good rule of thumb is to never use md5 at all. Not even for non-cryptographic use cases. It's not only broken, but also very slow!
https://github.com/stanfordnlp/dspy/blob/main/dsp/modules/ca...
""" Caching Libraries
""" From https://pypi.org/project/diskcache/The main difference is that it stores the state of an object, not a function.
If your data is JSON serializable then it could be a cool way to save and resume application state.
https://github.com/andrewgazelka/smart-cache