"I think now caching is probably best understood as a tool for making software simpler" - that's cute. Caching might be beneficial for many cases, but if it doesn't do one thing then this is simplifying software. There's that famous quote "There are only two hard things in Computer Science: cache invalidation and naming things.", and, sure, it's a bit ironical, but there's some truth in there.
There is also an important (but often overlooked) detail that you/your application may not be the only user of the cache. At which point caching, indeed, is an optimization via abstraction: when you fetch an X, you are in no position to predict that the next fifty completely unrelated to you requests would also want to fetch the same X, so it should probably be cached to be readily served.
Which is why solving the "I want my data in fast storage as often as possible" problem may be counter-productive on the whole: you ain't the only client of the system; let it breath and server requests from others.
Even more obvious if you think about the case of hardware-managed caches! The ISA typically exposes some simple cache control instructions (and I guess non-temporal loads/stores?), but apart from that, the actual choice of storage location is abstracted away from you (and your compiler).
On top of the other things mentioned (caching always introduces complexity with lifetime tracking, and thus can't make things simple), the article's got it the wrong way around.
When code has abstract interfaces for data access, introducing caching can be simpler (but not simple) by localizing it in the abstraction implementation which has or doesn't have caching.
But it is not an abstraction (you can perfectly well do caching without any abstractions, and it's frequently done exactly that way).
"fast storage" is about performance, your abstraction includes performance elements. If you go that down, then you are optimizing on your abstraction designs. What doesn't have to be wrong, but then don't say that is not optimization.
Caching is an optimisation. Sometimes caching can be abstracted away, eg CPU cache or build cache are pretty much abstracted away for a usual web developer. But web page caching is very hard to abstract without any abstraction leaks and weird bugs. And even CPU cache is no longer an abstraction if you deal with very high performance code.
I was discussing this with someone recently, caching is one of those things that people might do behind the scenes, thinking that it doesn't affect the API but in fact it can create all sorts of issues/complexity.
This article is talking about single-writer, single-reader storage. I think it's correct in that context. Most of the hairy problems with caches don't come up until you're multi-writer, multi-reader.
Note: the author means that caching can be used as an implementation detail in an (abstracted) storage access system, as opposed to a baseline of having multiple storage systems (fast, medium, slow) and managing them directly.
This was confusing to me – the most obvious way to judge the purpose of a system is to compare with the baseline of not having that system at all, especially in the case of caching where the program is functionally complete and correct without a cache. Anyway, there may not be a right or wrong here. Just tripped me up.
Yes "good" caching - a consistent storage interface - is an abstraction over "bad" caching - multiple different storage interfaces with different speeds. But caching overall is not an abstraction over not having caching.
Most optimizations require you to think about how your code is structured, so as a side-effect you make the code more understandable.
In this article, it's cache levels forcing you to separate different types of data because they're accessed at different frequencies. Another example is Rust's borrow checker, whose main purpose is arguably to facilitate both safe and efficient memory management, but which can also be used to enforce invariants that aren't clearly memory-related (e.g. builder pattern, temp files that auto-delete after they're dropped).
These aren't abstractions though. An abstraction is the opposite, hiding structure when it's noisy and making it easier to change. For example, if you already have an architecture in mind and don't want to manually determine how frequently each type of data is accessed, it's better to use a compiler or library that automatically determines what to cache with little to no code or thought on your end; that's abstraction. Similarly, the abstract analogue to Rust's borrow checker is garbage collection, which allows programmers to not think about their data-structures' lifetimes at all. The cost is usually performance and you understand your application less in some ways (although you understand it more in other ways; abstraction hides details but too many details make it hard to see the big picture. Ideally, with abstractions in the right places, you hide only the "unimportant" details in ways that insignificantly affect performance).
As can be seen from other comments, people tend to focus on the consistency implications, but something not discussed often in the context of distributed systems is that caches tend to introduce bimodality and metastability [0] [1].
See e.g. DynamoDB for an example of design taking it into account [2].
What? No, caching means a specific thing: keeping a copy of data away from the source of truth, closer to where you want to read it. Caching always makes systems more complex, it never makes things simpler, and it damn sure doesn't serve as any kind of abstraction unless you're redefining what words mean to indulge your technical philosophizing.
Sometimes posts are so difficult to read they're hard to respond to. I think I get what they're saying. I think they're saying that they think caching should be simple, or at least, that it should be obvious how you should cache in your particular situation such that you don't need things like algorithms. But that argument is kind of nonsense, because really everything in software is an algorithm.
Caching is storing a copy of data in a place or way that it is faster to retrieve than it would be otherwise. Caching is not an abstraction; it is a computer science technique to achieve improved performance.
Caching does not make software simpler. In fact, it always, by necessity, makes software more complex. For example, there are:
- Routines to look up data in a fast storage medium
- Routines to retrieve data from a slow storage medium and store them in a fast storage medium
- Routines to remove the cache if an expiration is reached
- Routines to remove cache entries if we run out of cache storage
- Routines to remove the oldest unused cache entry
- Routines to remove the newest cache entry
- Routines to store the age of each cache entry access
- Routines to remove cache entries which have been used the least
- Routines to remove specific cache entries regardless of age
- Routines to store data in the cache at the same time as slow storage
- Routines to store data in cache and only write to slow storage occasionally
- Routines to clear out the data and get it again on-demand/as necessary
- Routines to inform other systems about the state of your cache
- ...and many, many more
Each routine involves a calculation that determines whether the cache will be beneficial. A hit or miss can lead to operations which may add or remove latency, may or may not run into consistency problems, may or may not require remediation. The cache may need to be warmed up, or it may be fine starting cold. Clearing the cache (ex. restarts) may cause such a drastic cascading failure that the system cannot be started again. And there is often a large amount of statistics and analysis needed to optimize a caching strategy.
These are just a few of the considerations of caching. Caching is famously one of the hardest problems in computer science. How caching is implemented, and what it affects, can be very complex, and needs to be considered carefully. If you try to abstract it away, it usually leads to problems. Though if you don't try to abstract it away, it also leads to problems. Because of all of that, abstracting caching away into "general storage engine" is simply impossible in many cases.
Caching also isn't just having data in fast storage. Caching is cheating. You want to provide your data faster than actually works with your normal data storage (or transfer mechanism, etc). So you cheat, by copying it somewhere faster. And you cheat again, by trying to figure out how to look it up fast. And cheat again, by trying to figure out how to deal with its state being ultimately separate from the state of the "real" data in storage.
Basically caching is us trying to be really clever and work around our inherent limitations. But often we're not as smart as we think we are, and our clever cheat can bite us. So my advice is to design your system to work well without caching. You will thank yourself later, when you finally are dealing with the bug bites, and realize you dodged a bullet before.
Ousterhout's grad students did work on ramcloud with some research at facebook and Amazon on cache use at scale in complex organizations.
One bit of interesting trivia say for facebook (from memory): if you add all the RAM caches in redis/memcached/disk + db caches to make the thing work at scale, then for about 20-30% more memory you could've had the whole thing in memory 100% of the time.
Optimization isn't separable from abstraction. Abstraction is something that can be implemented in more than one way, while meeting the terms of its contract. That flexibility allows for optimization.
30 comments
[ 2.8 ms ] story [ 29.2 ms ] threadWhich is why solving the "I want my data in fast storage as often as possible" problem may be counter-productive on the whole: you ain't the only client of the system; let it breath and server requests from others.
When code has abstract interfaces for data access, introducing caching can be simpler (but not simple) by localizing it in the abstraction implementation which has or doesn't have caching.
But it is not an abstraction (you can perfectly well do caching without any abstractions, and it's frequently done exactly that way).
If everything is caching, why even introduce the term: language should help us describe ideas, it should not be superfluous.
Who told you that?
> you don't have to go all the way back to some backend database or API server or SSD [...] Caching is thus a tool to improve performance.
That's called "latency." This is not at all the same as "performance."
> My feelings now are that that perspective on caching is wrong
I agree.
This was confusing to me – the most obvious way to judge the purpose of a system is to compare with the baseline of not having that system at all, especially in the case of caching where the program is functionally complete and correct without a cache. Anyway, there may not be a right or wrong here. Just tripped me up.
In this article, it's cache levels forcing you to separate different types of data because they're accessed at different frequencies. Another example is Rust's borrow checker, whose main purpose is arguably to facilitate both safe and efficient memory management, but which can also be used to enforce invariants that aren't clearly memory-related (e.g. builder pattern, temp files that auto-delete after they're dropped).
These aren't abstractions though. An abstraction is the opposite, hiding structure when it's noisy and making it easier to change. For example, if you already have an architecture in mind and don't want to manually determine how frequently each type of data is accessed, it's better to use a compiler or library that automatically determines what to cache with little to no code or thought on your end; that's abstraction. Similarly, the abstract analogue to Rust's borrow checker is garbage collection, which allows programmers to not think about their data-structures' lifetimes at all. The cost is usually performance and you understand your application less in some ways (although you understand it more in other ways; abstraction hides details but too many details make it hard to see the big picture. Ideally, with abstractions in the right places, you hide only the "unimportant" details in ways that insignificantly affect performance).
[0] https://brooker.co.za/blog/2021/08/27/caches.html
[1] https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
[2] https://brooker.co.za/blog/2022/07/12/dynamodb.html
Caching is storing a copy of data in a place or way that it is faster to retrieve than it would be otherwise. Caching is not an abstraction; it is a computer science technique to achieve improved performance.
Caching does not make software simpler. In fact, it always, by necessity, makes software more complex. For example, there are:
Each routine involves a calculation that determines whether the cache will be beneficial. A hit or miss can lead to operations which may add or remove latency, may or may not run into consistency problems, may or may not require remediation. The cache may need to be warmed up, or it may be fine starting cold. Clearing the cache (ex. restarts) may cause such a drastic cascading failure that the system cannot be started again. And there is often a large amount of statistics and analysis needed to optimize a caching strategy.These are just a few of the considerations of caching. Caching is famously one of the hardest problems in computer science. How caching is implemented, and what it affects, can be very complex, and needs to be considered carefully. If you try to abstract it away, it usually leads to problems. Though if you don't try to abstract it away, it also leads to problems. Because of all of that, abstracting caching away into "general storage engine" is simply impossible in many cases.
Caching also isn't just having data in fast storage. Caching is cheating. You want to provide your data faster than actually works with your normal data storage (or transfer mechanism, etc). So you cheat, by copying it somewhere faster. And you cheat again, by trying to figure out how to look it up fast. And cheat again, by trying to figure out how to deal with its state being ultimately separate from the state of the "real" data in storage.
Basically caching is us trying to be really clever and work around our inherent limitations. But often we're not as smart as we think we are, and our clever cheat can bite us. So my advice is to design your system to work well without caching. You will thank yourself later, when you finally are dealing with the bug bites, and realize you dodged a bullet before.
One bit of interesting trivia say for facebook (from memory): if you add all the RAM caches in redis/memcached/disk + db caches to make the thing work at scale, then for about 20-30% more memory you could've had the whole thing in memory 100% of the time.