Kind of reminds me the first time I started freaking out over the "Memory Bug" on my Solaris 2.6 install in 1998. So many people got worried by that data, that they modified several of the system binaries to report "filesystem cache" as "free" in later versions of Solaris.
What's old is always new, again.
Linux makes it (reasonably) clear:
shephard@u804:~$ free
total used free shared buffers cached
Mem: 515440 39976 475464 0 2684 16452
-/+ buffers/cache: 20840 494600
Swap: 192740 0 192740
shephard@u804:~$ dd if=/dev/zero of=/tmp/475MB bs=1048576 count=475
475+0 records in
475+0 records out
498073600 bytes (498 MB) copied, 73.7244 s, 6.8 MB/s
shephard@u804:~$ free
total used free shared buffers cached
Mem: 515440 508312 7128 0 1760 475756
-/+ buffers/cache: 30796 484644
Swap: 192740 0 192740
shephard@u804:~$
Many people don't know how kernel caches and buffers work. Even many developers get confused by this and think that a system is almost out of memory. Luckily someone created http://www.linuxatemyram.com/ - that saves me time to explain it over and over to people who panic about the issue.
I think several people nailed this when the story broke, it really was a pretty superficial observation for such an apparently broad analysis.
Slightly off topic, but it just struck me that I'd like to see app icons in the task manager and windowed programs highlighted in some way. The unabridged list of processes is a little too lengthy if you're looking to quickly nip something in the bud. I know there are task manager replacements, but to me they're not as intuitive.
I'm by no means fan of windows, but the article was an obvious troll from the beginning, as it was quite evident that they were mixing up cache with real memory usage.
What really surprised me was that such a pointless 'story' generated so much fuss. Waste of time.
In some cases, the sysadmin is right. Say you are concerned with performance, not energy-efficiency. If you have an I/O-bound workload that is only using 50% of your CPU cycles, wouldn't it be nice to automatically use the other 50% (otherwise wasted, mind you) to figure out some other things you might need in the future and prefetch them? On the memory side, what good is that 1GB or unused RAM doing you? If you need more for your application, the OS can drop its cache on the floor so quickly you'll hardly notice the latency.
Granted, this prefetching is not without downsides. DRAM banks have low-power states you can put them into if they're not being used, so if you can shove all of your used memory into a contiguous region of your physical address space and free up a DIMM's worth, you might be able to save power. This is unlikely, however, because the virtual-to-physical address mapping function is usually designed to spread the load very evenly across all available DRAM banks. One more likely source of power savings is from avoiding the CPU and disk activity of prefetching, especially if prefetch accuracy is low. Most people tend not to care about this though, and want the OS to make their app run faster by using otherwise-idle resources to do prefetching.
It was an 80's mainframe. I doubt power saving was one concern. We had folks typing forms in data-entry terminals 24x7 generating the tapes that were then fed to the bigger machine.
But you have an interesting idea. I wonder if there is any OS that changes memory distribution and allocation strategy according to desired power consumption on, say, portable computers.
The Linux kernel supports, IIRC, plug-and-play memory and processors and it could, conceivably (it it doesn't already) power up and down unused parts of the machine in order to reduce power consumption and heat dissipation. Even reads and writes to and from memory could be grouped in bursts if that could save power.
For instance, right now my running applications are within half of the memory, both cores are running slowly and the ethernet interface is disconnected. One memory module, one core and a network interface could be powered off without perceivable performance degradation. Also, as much data could be cached in the remaining memory so the disk could be powered down too or, and that would be interesting, spun at a lower speed.
Lots of interesting ideas in one lazy saturday afternoon. That's what I like in HN. :-)
In the mainframe world you're often paying by the processing unit consumed -- if you're not using the cycles you're not paying for them, and if you're using cycles you don't actually need to use you're just burning money.
He could read the available documentation and literature. This is not some grand secret. This feature has been around in Vista for years, there's no excuse for this level of ignorance. It's just laziness.
Any operating system developer who knows anything about paging would know exactly what was going on. Especially since you can immediately tell using the Windows task manager that it was doing caching.
I wonder if Windows 7 behaves like Linux in that by default, when faced with memory pressure, it will much prefer to swap out application pages rather than abandon buffer cache.
This is really annoying when this memory pressure is due to trying to fit more (and potentially useless) things into the buffer cache. Let's say you're performing a backup of a 100+ GB directory to another machine using rsync. Linux will want to jam that entire directory into buffer cache, which of course won't fit. Instead of immediately pitching out the existing disk cache, it will aggressively (depending on the value of swappiness) swap out application pages, (potentially owned by Oracle or MySQL) rather than lose existing buffer cache.
This is a real problem, and one that's a pain to fix.
No, Windows 7 and Vista always abandon the Superfetch cache before paging out anything else -- this is verified in the comments section. It's the sensible thing to do, since
a. Superfetch data is heuristic and application pages are mostly not.
b. Superfetch data always has a backing store (the actual file on disk), so it's really cheap to forget about it. This may or may not be true for application pages.
21 comments
[ 2.9 ms ] story [ 19.6 ms ] threadWhat's old is always new, again.
Linux makes it (reasonably) clear:
Slightly off topic, but it just struck me that I'd like to see app icons in the task manager and windowed programs highlighted in some way. The unabridged list of processes is a little too lengthy if you're looking to quickly nip something in the bud. I know there are task manager replacements, but to me they're not as intuitive.
One for the Win8 suggestion box maybe.
What really surprised me was that such a pointless 'story' generated so much fuss. Waste of time.
And how real is the use of memory for cache?
All this reminds me of a discussion I had with a mainframe admin in the mid-to-late 80's:
Me: Look at this numbers. We are using more than 90% of the processor time and about 100% of the memory. Is it normal?
Sysadmin: Good. We paid them in full. It's better if we use the whole computer, not just part of it.
Granted, this prefetching is not without downsides. DRAM banks have low-power states you can put them into if they're not being used, so if you can shove all of your used memory into a contiguous region of your physical address space and free up a DIMM's worth, you might be able to save power. This is unlikely, however, because the virtual-to-physical address mapping function is usually designed to spread the load very evenly across all available DRAM banks. One more likely source of power savings is from avoiding the CPU and disk activity of prefetching, especially if prefetch accuracy is low. Most people tend not to care about this though, and want the OS to make their app run faster by using otherwise-idle resources to do prefetching.
But you have an interesting idea. I wonder if there is any OS that changes memory distribution and allocation strategy according to desired power consumption on, say, portable computers.
The Linux kernel supports, IIRC, plug-and-play memory and processors and it could, conceivably (it it doesn't already) power up and down unused parts of the machine in order to reduce power consumption and heat dissipation. Even reads and writes to and from memory could be grouped in bursts if that could save power.
For instance, right now my running applications are within half of the memory, both cores are running slowly and the ethernet interface is disconnected. One memory module, one core and a network interface could be powered off without perceivable performance degradation. Also, as much data could be cached in the remaining memory so the disk could be powered down too or, and that would be interesting, spun at a lower speed.
Lots of interesting ideas in one lazy saturday afternoon. That's what I like in HN. :-)
http://en.wikipedia.org/wiki/Technical_features_new_to_Windo...
Better yet, the same article circa Feb. 2007: http://en.wikipedia.org/w/index.php?title=Technical_features...
This is really annoying when this memory pressure is due to trying to fit more (and potentially useless) things into the buffer cache. Let's say you're performing a backup of a 100+ GB directory to another machine using rsync. Linux will want to jam that entire directory into buffer cache, which of course won't fit. Instead of immediately pitching out the existing disk cache, it will aggressively (depending on the value of swappiness) swap out application pages, (potentially owned by Oracle or MySQL) rather than lose existing buffer cache.
This is a real problem, and one that's a pain to fix.
a. Superfetch data is heuristic and application pages are mostly not.
b. Superfetch data always has a backing store (the actual file on disk), so it's really cheap to forget about it. This may or may not be true for application pages.