Ok, I made the comment in http://news.ycombinator.com/item?id=1138840, that "it's surprising that the CTO of a company specialized on system performance doesn't know how OS cache memory works." It turns out that he really has no clue.
To the people who said the Windows source code is needed to understand how it works, hogwash. The file caching in Linux has been doing it this way for years. Windows does it for years as well. It's fairly standard OS caching technique.
The file system cache doesn't contribute to committed bytes, the metric that the Exo CTO is appealing to - in this respect the Ars Technica article is mistaken. (Actually, the fact that the Ars Technica article used Task Manager made me inclined to dismiss it as clueless.)
Estimating real memory pressure is tough. There's the working set; but the working set may expand when you have sufficient physical memory, even though the marginal performance gains may be small. Similarly, the file system cache may expand when you have sufficient physical memory, but the marginal gains become small there too.
Probably the best unified metric would be the amount of time applications spend blocked on I/O, where that I/O is either paging committed bytes in from disk, or data that would be cacheable - and would have been cached - on a system with more memory. But that's extremely hard to predict.
If you've spent any amount of time at the Ars forums, you wouldn't refer to Peter as "clueless".
Committed bytes is completely the wrong metric to appeal to, since it includes pages that have been committed but have never been touched. Windows reserves space for such pages in the page file (it never overcommits), but saying that it leads to "memory pressure" is ridiculous.
I wrote off the article, rather than the author. If I indeed read more of the author, perhaps my opinion would be updated, but the article on its own doesn't add much weight of authority. The trouble with Task Manager is that it doesn't have sufficient per-process memory data to be useful. Process Explorer is better, because you can view Working Set (Private|Shareable|Shared) and Private Bytes. I'd view both private working set and private bytes as the most important metrics for a Windows app, but depending on the circumstances (such as how likely multiple instances are) other things may be considered.
Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
As to the Ars forums, with a brief perusal it seems similar to the programming Reddit - i.e. a lot of opinionated and frequently misinformed noise, interleaved with a small amount of signal. It looks like a time sink, to be frank.
But to give Peter his due, I registered with the forums and scanned through his past posts (DrPizza), and I can't find much that I disagree with; I particularly agree with him on the consolification of PC games, and the lack of MW2's appeal to a mature gamer, etc.
Peter did admit his mistake about superfetch (he thought the xpnet thing was counting that too as used memory), but he's still right about the overall metric being wrong -- see the comments to http://exo-blog.blogspot.com/2010/02/editorial-what-took-you... .
edit: and do you think he doesn't know about procexp/perfmon? :)
Why do you insist on making this personal? I discounted an article, not a person.
By bringing the person into this, you turn it into an irrational, emotional thing, where facts don't matter. That's not where you want to be.
And at the risk of extending this thing far beyond its worthy lifetime, that comment you link to proves very little. As I already said:
> Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
Allocating a huge bunch of memory and not touching it is one statistical pattern, but it's not particularly common, which is key. It's an outlier, so it isn't strong evidence. On average, committed bytes can be interpreted as a proxy for memory pressure, but like I said, there's likely to be a fair amount of variance there, so it's a poor proxy. There certainly wouldn't be a 1:1 relationship between committed memory and physical memory pressure; if I had to guess, I would guess somewhere like 60% of committed memory should ideally be in physical ram.
For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB working set total. But that working set includes both private and shared working set pages - calculating the active working set properly would require dividing shared page sizes by the number of copies in different processes.
But just for fun, I plotted the log of working set values in MB along the X axis, and the log of private bytes in MB along the Y axis, for each process, and this is what I got:
Looks like a pretty close correlation to me - yes, there are some spikes in private bytes, but on average, private bytes is a proxy for committed bytes (I excluded shared pages for confounding reasons, as they are usually things like EXEs and DLLs - not counted against commit charge), committed bytes is a proxy for physical memory usage, which in turn is a proxy for physical memory pressure.
No, a large part of file caching in Vista and 7 is preemptive -- Linux doesn't do it by default (I think there's a utility called preload that does it).
Actually, there was one nugget of truth in the article, and that was the _best_ way to determine if your operating system is low on memory, is to watch it's swapping activity. This is probably true across all operating systems, Linux, Solaris, etc..
An operating system will typically not swap out (or in) cache memory to disk, so, as long as you are exhibiting no swapping behavior, (And you have swap enabled) then you have enough memory. I think it was Adrian Cockroft, in fact, in "Sun Performance and Tuning: Sparc & Solaris" who in fact said that was _the_ way to determine if you have memory pressure on your operating system - run vmstat and watch the si and so column for activity.
An OS can game that kind of statistic, though, by spending more memory on what Windows would call working set (the "hot" set of pages for a process which are preferentially kept in physical memory) at the expense of the file system cache.
Also, predictive prefetching may increase paging in, but in an async fashion, so the application is never actually blocked. This may cause spurious increases in the stats. You need to examine the amount of time threads are actually blocked by the paging behaviour.
LoseThos doesn't use paging. (Identity maps virtual to physical.) It's disk cache is dedicated and cannot be reconfigured. LoseThos does not swap memory to disk as with virtual memory. Moron! What part of "no paging" do you not understand? It does "swap" out one task and swap in another. This is just storing and loading CPU registers. It can swap a task in half a microsecond.
It polls the disk -- a task doing disk access reads a status I/O port and requests itself to be swapped-out. Other tasks run and the disk task is rescheduled where it polls again. This is cooperative multitasking. It expands to fill all CPU. Yet, other tasks are not degraded.
The memory guage is free memory, not used memory. A block pool is reserved for each cpu core of around 64 meg. On an 8-core machine this is 7*64Meg. The default disk cache is ... I forget...Oops memrep is broken. The accounting for the 64Meg core block pools is hard to explain--they reduce the total block pool, but they are free memory.
Okay, fixed memrep. Disk Cache default size is 128Meg.
BBC compares me to an African doing land reform. I have made an operating system with full privilege and do the trivial solution which they see as African.
I'm not sure, but can you malloc an 8 Gig chunk under windows or linux?
If I do any good research, they call me Iran doing nukes.
Ba ha! I saw a video of English engineers doing a rocket driven pinwheel antimine device in WWII.
In heaven the Germans are engineers, French chefs and Brits police; in hell Germans police, French engineers and Brits cooks.
God is a God of truth. He's not about phoniness. He said Shakespeare had a vile heart. You can't argue with that. "desdemona" in othello is "there's the money."
Lets expose the Brits as racist. I'm all about truth. Come-on, admit it publically and don't hide.
God pointed to sports. Can't argue with that.
In the body of Christ, the hand can't say to the foot, I don't need you. We even need assholes.
You don't get it, Brits -- I'm not reinventing Linux. What's the point of that. This is a programmer's paradise where tinkering is the goal. It's simple and low line count and easy for amateurs to understand. It's not a mainframe operating system. Back then, you had a thousand users sharing a Gig. I have one user with 12 Gig. My disk cache is 128Meg but my entire operating system disk is 5Meg! I don't need to swap to disk and polling is fine for one home user who only cares about the focus task.
I'm not even sure the facts in this case matter if the CTO is willing to make such a fool of himself in public. Even if I agreed with his point of view, it's impossible for me to take him seriously when he writes such needlessly incendiary rants on his company's official blog. It reminds me of conspiracy theorists.
Heh. I've been reading the various discussions about all of this, and having considerable experience digging into internals to find performance problems, it still seems to come down to one thing.
Regardless of whether the CTO is correct or not, the tone and poor attitude of his posts have convinced me that he is not someone I would want to do business with.
15 comments
[ 2.9 ms ] story [ 33.1 ms ] threadTo the people who said the Windows source code is needed to understand how it works, hogwash. The file caching in Linux has been doing it this way for years. Windows does it for years as well. It's fairly standard OS caching technique.
Estimating real memory pressure is tough. There's the working set; but the working set may expand when you have sufficient physical memory, even though the marginal performance gains may be small. Similarly, the file system cache may expand when you have sufficient physical memory, but the marginal gains become small there too.
Probably the best unified metric would be the amount of time applications spend blocked on I/O, where that I/O is either paging committed bytes in from disk, or data that would be cacheable - and would have been cached - on a system with more memory. But that's extremely hard to predict.
Committed bytes is completely the wrong metric to appeal to, since it includes pages that have been committed but have never been touched. Windows reserves space for such pages in the page file (it never overcommits), but saying that it leads to "memory pressure" is ridiculous.
Please explain, for the (majority?) of us who don't spend time at the Ars Forums
Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
As to the Ars forums, with a brief perusal it seems similar to the programming Reddit - i.e. a lot of opinionated and frequently misinformed noise, interleaved with a small amount of signal. It looks like a time sink, to be frank.
But to give Peter his due, I registered with the forums and scanned through his past posts (DrPizza), and I can't find much that I disagree with; I particularly agree with him on the consolification of PC games, and the lack of MW2's appeal to a mature gamer, etc.
edit: and do you think he doesn't know about procexp/perfmon? :)
By bringing the person into this, you turn it into an irrational, emotional thing, where facts don't matter. That's not where you want to be.
And at the risk of extending this thing far beyond its worthy lifetime, that comment you link to proves very little. As I already said:
> Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns.
Allocating a huge bunch of memory and not touching it is one statistical pattern, but it's not particularly common, which is key. It's an outlier, so it isn't strong evidence. On average, committed bytes can be interpreted as a proxy for memory pressure, but like I said, there's likely to be a fair amount of variance there, so it's a poor proxy. There certainly wouldn't be a 1:1 relationship between committed memory and physical memory pressure; if I had to guess, I would guess somewhere like 60% of committed memory should ideally be in physical ram.
For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB working set total. But that working set includes both private and shared working set pages - calculating the active working set properly would require dividing shared page sizes by the number of copies in different processes.
But just for fun, I plotted the log of working set values in MB along the X axis, and the log of private bytes in MB along the Y axis, for each process, and this is what I got:
http://imgurl.filetac.com/img/68343980.png
Looks like a pretty close correlation to me - yes, there are some spikes in private bytes, but on average, private bytes is a proxy for committed bytes (I excluded shared pages for confounding reasons, as they are usually things like EXEs and DLLs - not counted against commit charge), committed bytes is a proxy for physical memory usage, which in turn is a proxy for physical memory pressure.
http://blogs.zdnet.com/BTL/?p=31024&page=1
FWIW, I never downvoted you - I actually upvoted you when I saw you dropped to 0.
An operating system will typically not swap out (or in) cache memory to disk, so, as long as you are exhibiting no swapping behavior, (And you have swap enabled) then you have enough memory. I think it was Adrian Cockroft, in fact, in "Sun Performance and Tuning: Sparc & Solaris" who in fact said that was _the_ way to determine if you have memory pressure on your operating system - run vmstat and watch the si and so column for activity.
Also, predictive prefetching may increase paging in, but in an async fashion, so the application is never actually blocked. This may cause spurious increases in the stats. You need to examine the amount of time threads are actually blocked by the paging behaviour.
It polls the disk -- a task doing disk access reads a status I/O port and requests itself to be swapped-out. Other tasks run and the disk task is rescheduled where it polls again. This is cooperative multitasking. It expands to fill all CPU. Yet, other tasks are not degraded.
The memory guage is free memory, not used memory. A block pool is reserved for each cpu core of around 64 meg. On an 8-core machine this is 7*64Meg. The default disk cache is ... I forget...Oops memrep is broken. The accounting for the 64Meg core block pools is hard to explain--they reduce the total block pool, but they are free memory.
Okay, fixed memrep. Disk Cache default size is 128Meg.
BBC compares me to an African doing land reform. I have made an operating system with full privilege and do the trivial solution which they see as African.
I'm not sure, but can you malloc an 8 Gig chunk under windows or linux?
If I do any good research, they call me Iran doing nukes.
Ba ha! I saw a video of English engineers doing a rocket driven pinwheel antimine device in WWII.
In heaven the Germans are engineers, French chefs and Brits police; in hell Germans police, French engineers and Brits cooks.
God is a God of truth. He's not about phoniness. He said Shakespeare had a vile heart. You can't argue with that. "desdemona" in othello is "there's the money."
Lets expose the Brits as racist. I'm all about truth. Come-on, admit it publically and don't hide.
God pointed to sports. Can't argue with that.
In the body of Christ, the hand can't say to the foot, I don't need you. We even need assholes.
You don't get it, Brits -- I'm not reinventing Linux. What's the point of that. This is a programmer's paradise where tinkering is the goal. It's simple and low line count and easy for amateurs to understand. It's not a mainframe operating system. Back then, you had a thousand users sharing a Gig. I have one user with 12 Gig. My disk cache is 128Meg but my entire operating system disk is 5Meg! I don't need to swap to disk and polling is fine for one home user who only cares about the focus task.
Does it _feel_ fast?