Tell HN: RAM Requirements are going through the roof
Then there's the desktop. My desktop PC, for instance. It's a great 6 months old working computer, with 8 cores and 16GB of RAM. "This ought to be enough for all my needs", I thought. The Bill Gates in me was right so far, but unfortunately this won't be enough for long.
Yesterday I took a look at my memory usage, out of curiosity: 11.4GB. Woah. I had a look at the breakdown.
- Chrome : ~3GB
- Firefox : ~1.5GB
- Java (eclipse) : ~1.2GB
- Rest in tons of various work-related apps.
What worries me now is how hard we are getting struck with RAM-hogging web pages. Since I began writing this post, my freshly restarted Chrome browser with 9 open tabs (Hacker News (x2), Coding Horror, Google Search for "Mac osX Lion review 6 months", MoPub Ad Service monitoring, Google Analytics Visitors Overview, Android Developer Console, Twitter, Gmail) is taking up roughly 500MB of RAM. That's insane. On my 2010 Macbook Pro with 4GB of RAM, this means 20% of my overall caching capacity is taken up by my core web needs. Needless to say, I can't use my macbook anymore.
Last summer I was working on a little java experiment - a cross-platform 3D labyrinth. I wanted the overall memory and data footprint to be as low as possible so it could be played on a low-end android phone with really fast loading, and yet to keep the game space as big as I could, so I designed my own dedicated data structures overnight. I made the following calculation: over 500MB of raw, uncompressed data, I could store the description of an area roughly equivalent to a map of Europe with a resolution of 2.5 meters per pixel.
I'm not claiming any feat here, just stating the obvious: web programmers are doing something that's definitely not cool for our current RAM budget. We forgot any sense of measure. It's unnecessary for a twitter feed following 175 persons for an hour or so to claim a 70MB RAM footprint. You're not the only nor the worst offender, Twitter. Except that I had 26 new tweets to display, I clicked, and the footprint suddenly grew to 76MB. 26 tweets = 6MB. We're talking about 140 characters tweets, let's be generous and multiply that amount by 100 to take into account the tweeter's profile (which are all 10,000 characters essays as we all know), and we get a total of 364 000 new characters, which end up claiming more than 6 million bytes in RAM. Which means the RAM impact of adding 26 new tweets to a webpage is at very very least 10 times higher than it could be, and probably more like a thousand times too high.
Like I said, I'm only using Twitter to state a point. I mean nothing wrong with Twitter's web devs - actually I'm myself a very poor web developer - and I'm pretty sure the blame could also be put on Google Chrome instead, but I remember a day in 2002 where my Internet Explorer was trying to load a 4MB webpage from the hard drive, causing a RAM footprint above 40MB and a failure after 20 minutes of waiting in front of a white screen. Back then I was merely a junior consultant working on QA, and I was the one to tell the devs that they were definitely doing something not cool at all for the user's computer. True, we were showing rather complex and impressive amounts of "data" on our webapp with much simpler but oh so wrongly implemented "UI" (in that case, hundreds of unnecessary nested table anchors), but I can't help thinking back at those times where it was simply impossible to make a product that was not cool for the user's comp...
90 comments
[ 4.3 ms ] story [ 171 ms ] threadWhat's the point of having a tonne of ram only to let it sit idle?
I "only" have 4 gb of ram and chrome is taking up 1.2 gb of that, even though i have 12 windows open with ~5 tabs in each.
The problem is that an extension like "1-Up for Google+" doesn't need 20 MiB of RAM just to play a sound and display a green mushroom, for a total of about 150 KiB of assets and maybe a few KiB of code.
You'd be surprised how big uncompressed data can get, compare a .BMP with a .PNG of the same image for example.
2 seconds of uncompressed sound, at 44.1 kHz, in stereo and 8 bits per sample is 170 KiB. 17 images of on average 24 by 24 pixels at 32 bits per pixel is 68 KiB. Three other, bigger images give another 40 KiB of uncompressed data. A total ~280 KiB. There's a few KiB of HTML but I don't know whether and how I should count that. Even if we take into account headers, padding, objects and higher quality data (e.g. 16 bits per sample), I assume all assets will still account for less than half a megabyte.
Now that I look at the code, I see it's including a bundled version of jQuery. I don't know how big the representation of Javascript code is in memory, but 29 MiB seems a bit much (correction, while I was typing this, it went to 31 MiB).
It might seems disingenuous to single out this extensions, but truth to be told all my extensions need a similar, way too high amount of private memory.
If it causes the load of another library which nothing else was using that could count especially if it's a different version of jquery etc.
If 2 processes are both using the same library and it is linked to their addresses spaces, who's memory usage does it count against?
Apps don't generally change their behavior based on the size of physical RAM, but maybe Chrome does this as a very advanced optimization.
Let's say you load up you video editing program, maybe it only needs 512MB of memory to fit the whole program into memory, now what happens when you load in LOTR directors cut?
Especially if you aren't running any other large programs, would it still make sense to restrict it to 300MB (or 1GB , or 2GB etc)?
I think it's safer to assume that OS developers know what they are doing and can be trusted to handle memory contention issues better than most of us can.
As an aside some programs (like redis , memcached etc) do allow you to restrict the amount of RAM which they use but this is because they are explicitly greedy with memory and are probably used by people with enough skills to measure actual usage and tune accordingly.
I don't necessarily. I just know how much RAM I want to be able to give it.
I'd love to be able to test apps in lower-RAM scenarios. Just because I've got 8g or 16g doesn't mean everyone does, but I have no way to test what that experience will be like.
Additionally, I may have two processes that are both rather RAM hungry that I need to run. Give me the option of constraining one of them so that I can give RAM priority to one or the other rather than "oh, well the OS people are smarter than me, so I guess I have to trust that app 1 will get RAM priority over app 2, or they'll just both have to be equal".
Although you should bear in mind that computers with less memory probably have less everything else (including slower memory busses) so your best bet is to actually find a slower computer and test with that.
Most OSes will let you prioritize other processes over others (e.g Linux has "nice"), whether this is just for CPU preemption priority or whether it does other resources as well I'm not sure.
However I think most swapping systems will try and keep the most frequently used pages in RAM , so if one process has preemption priority over another this problem would solve itself as it's memory area would just get accessed more often.
Again this is kind of finger in the air stuff so anyone more familiar with OS design should feel free to correct me.
https://wiki.archlinux.org/index.php/Chromium_Tips_and_Tweak...
If you ran the same set of apps on a 2GB machine you'd get different numbers.
Besides, I'll have to take your word on that, since my 4GB macbook pro has become completely unuseable due constant swapping, but it could be completely unrelated.
Which is annoying, because it leads to double-buffering.
As for developers, matters are worse. For one, I have the pleasure of working with (debugging, deploying, writing) JS and Flash code. I would love to use the extra 2 GBs of RAM for running 3 more virtual machines to do more testing, but I cannot.
The solution is not to bury our heads in the sand, but to fix the memory leaks and/or memory requirements. As the OP points out, there is no reason that a web page that can be encapsulated in a half a meg of HTML needs to take up hundreds of megs of RAM once parsed and running.
That said, there might be the occasional rogue app or web site, which should be uninstalled or avoided.
How would I know if a web site is "rogue"?
This was absolutely unintentional and involves at least two intermediates over which the original blogger had no control whatsoever.
Link to the page in question: http://fh2012.tumblr.com/page/6 It so happens it's the tumblr blog of the current favorite to the French Presidential election, and it was the first post so I noticed it :P
I would have preferred a web in which scrolling (and certain other operations, like the Back button) was always instant regardless of how the page is coded, and consequently I did not have to watch out for rogue pages or sites.
ADDED. Or at least a web in which there were fewer ways for a page to go unresponsive, which would have made it easier for readers and writers to anticipate responsiveness problems.
I wouldn't say your macbook is useless though. I have only one computer and it's a 2008 laptop with 2 GB RAM. And I cope very well, although I experience the occasional lockup (I don't use swap).
I might try it out sometime though.
Are the underlying systems running web applications (aka browsers) doing it wrong? Or are the webapp devs not doing what needs to be done in general? Or am I raising a false alarm and is everything alright?
As for my macbook, it has an unrelated problem, that's for sure, but all it does is magnify the memory issues for me.
8GB of DDR2/3 from a good vendor has been less than $50 for a while now.
But really, buying more RAM is only a temporary solution. I sometimes feel like the memory demands of web browsers are out of line and I wonder what causes that.
Unfortunately, the flipside is that if you're using a computer more than a few years old, then anything more than your set of 'core' sites tends to slow your computer to a halt as your browser causes everything to be constantly swapping in and out of memory.
On the laptop I'm writing this from, I've got a Chrome process with 827 MB of VM size -- but only 62 MB of pages actually in use. The rest is large memory-mapped regions consisting mostly of untouched pages -- which makes sense, because on a 64-bit machine, virtual address space is practically free.
Chrome says: "This is the best indicator of browser memory resource usage"
That aside, i recently hired the cheapest VPS i could find; 128MB RAM. I could not run apt-get; i ad to upgrade to 256MB.
To be honest, I don't really address where your 3GB for Chrome is going. All those memory figures you see should be taken with a pinch of salt. Reserving address space, using it and such are not so straightforward to untangle.
Your OS doesn't offer anything like http://williamedwardscoder.tumblr.com/post/13363076806/buffc... sadly.
I like file schmantics because it gives us access control and an api that is straightforward to use
Having said that many of the languages we are using such as Javascript are not exactly storage efficient combined with combined with the fact that many people who would probably not have considered themselves "programmer" enough to write native apps in C++ back in the day are writing things in Javascript and putting them online.
I often see people "benchmark" things like operating systems or browsers based on RAM usage (people coming to the conclusion that 32bit Win XP is more "efficient" than Win7 for example) but I think it is far more important to measure swap file usage as this is the thing that actually hurts performance. Afaik there is no performance hit in overwriting a flip-flop in memory that already has a value stored but there is a huge penalty for hitting a disk. So I'd be quite happy to have all my RAM utilized all the time if I have 0 swapping.
First of all if you are using a 64bit platform your pointers are going to be twice as big as they used to be but you get to access so much more memory that this issue solves itself.
Also IIRC Windows 7 does things like arrange your frequently used data into a contiguous area of the disk and then reads all that stuff into VM at bootup. This will result in higher RAM usage (due to many pages being loaded into physical RAM) but better overall performance since if other stuff needs the RAM more urgently it can either swap the cache back to disk or overwrite the cache entirely.
Let's say you have 8GB of RAM and are watching a HD video which is 7GB uncompressed and the rest of your system only really needs 1GB of your RAM, surely it makes sense what whilst you are watching the video you can use spare CPU cores to uncompress the rest of the video and put it into RAM.
I think it's more the case that computing will always expand to fill the amount of memory available. Bearing in mind that even the cost of swapping is dramatically reducing with SSDs etc.
That's not clear to me. Why do something ahead of time that can be done faster than needed on the fly anyway? You're basically assuming that when system resources are needed for something else halfway through your video, processing power will be more useful than space.
I'm sure there are lots of examples where doing complex calculations whenever you have CPU spare and swapping them out to disk as needed would be preferable to having to contend with whatever else needs to use it at the time.
I think the key is how to OS deals with memory contention issues rather than just minimizing memory usage.
Someone with a better understanding of OSes than me could probably clarify.
I guess the performance hit is when you write a page of RAM to the disk, perhaps there are some cases where you don't bother to do this if memory contention is high?
One thing people often don't realise is there isn't really 'wasted' RAM, your OS usually uses the rest to cache things, especially files. File caching can greatly speed up disc reads and writes. (The following only applies to linux, I'm not sure about other OS's; all three have very different memory models). When memory is getting high, a few things can happen (depending on whether you have swap). The file cache is preferentially cleared first, but swapping pages to disc is also considered readily. An interesting thing is that swapped pages aren't deleted from swap straight away, only if they are modified (so swap usually doesn't decrease in size). There is also the Out Of Memory killer, which can kill applications using too much memory. It really is an interesting area. You can read more from this page http://tldp.org/LDP/tlk/mm/memory.html. Just remember all OS's are different.
It makes a ton of sense for all those people (and other species, from different time/space geometries) who can comprehend a movie by experiencing its time dimension as a single event. Ordinary humans may find it less useful, albeit having no delay when skipping back and forth may have some utility for those of us who experience time in a more traditional way ;-)
The only time this would make sense is in a content creation environment where quick access to uncompressed frames makes a difference. However, those environments are already CPU starved. Typical consumers would not see a benefit from this scenario. More benefit will be had with the transition to SSDs, they will be eventually be much much faster & require less power than current hard drives.
I think wiskers is right though: It depends on the box. On my PC (2gigs) the browsers rarely make it to 1gig.
Actually, I don't even get the sense that chrome has an adequate view of its own resource needs. My wife often finds that her browser and computer is underresponsive because chrome is using so much memory for background tabs.
There's the whole javascript framework with it's own allocator, memory pools which will stay around for a long time. Some of the browser elements use the same kind of vm to actually display the UI to you, adding some more mem usage. Then there's the whole page which had to be read, parsed into a tree, processed to create appropriate model for display (all stages are preserved in an editable way). This has to be a live model for the whole time, since javascript might need to interact with it - you can't just flatten it into a screenshot and display that. Now the images - they also need to be loaded and decompressed for display. Also the text isn't that simple - there's the whole engine behind loading fonts, analysing each letter / pair / triple / ... for special spacing rules to be applied. There are multiple glyphs, font variants, sizes, etc. to rasterise. That text needs to be distributed properly inside of the DOM model generated above. That text flow actually affects the model again - they're linked, so they get processed again (text metrics need to be recomputed on each page resize, which may make some elements longer/shorter). Then the theme goes on top of this - since the application displays itself, the needed libraries / theme elements will be loaded too. But of course not all browsers use the theme from OS for displaying the form elements, so another layer needs to be loaded here. .... I could go on forever about elements which are needed for rendering the page.
So yes - lots of that could be much lighter. Lots of that could be made into modules or pulled some levels higher to make them more general. Lots of that might be just a result of sloppy coding noone ever got to fix.
But just remember that what you're actually using is a system on top of a system on top of a system on top of... If you used gopher on a real 80x25 term you'd use kilobytes. If you used first web browser with no image support and no scripting, you'd be under megabytes. But you don't - you're looking at a realtime rendering of 10 different models coming together and interacting with each other with next to no lag. And it all probably renders with hardware support to give you a very smooth scrolling experience. So no, web developers themselves have almost nothing to do with this whole mess. Unless they made a silly mistake and are leaking memory like crazy... they're not the ones to look at first.
(I simplified in many places, not all facts are relevant to every OS/browser combination, etc. just trying to make a point here)
You and others just filled that gap, thank you, I feel less stupid now.
The operating system plays a huge role in memory usage, many modern ones cache indiscriminately, preload most used applications, etc etc
Comparing RAM usage is almost apples and oranges these days, unless we're talking similar setups.
Edit: Debian minimal install using the Awesome window manager.
Chrome leaks memory. Check memory usage again after force-quitting Chrome and relaunching it. I have to restart Chrome daily to regain 3-5 gigs of swap.
Browsers have sub second back buttons, how much pre-rendered ram space do you think that takes up?
I'm sure the os & browser devs are all good and great. I'm sure the web devs are all fine and accurate; that the subject is a complex one and we're already all doing the best we can.
But that's not my point: I'm worried that one day we'll run out of RAM because we're designing our webapps as if client RAM was no longer a finite resource, we simply forgot it even existed in the first place.
I'm on a 2GB Laptop with XP typing this and the fan is silent. SILENT. The second I turn noscript off it overloads, then of course, crashes. Some scripts are running constantly in the background, with click tracking and even mouse motion tracking becoming more commonplace.
I've been doing this since years and I seriously do not understand when people complain about extreme RAM usage of their browser. I have 4GB of RAM and often sit at 50-100+ tabs for weeks and don't really have any issues - Firefox Nightly is currently hogging ~16% of my memory according to htop - which is perfectly okay.
How about once those 50-100 tabs crash, do you have an open link backups strategy or just hope restore catches it all in time?
Concerning crashes (which became incredibly rare after I decided to just not have Flash installed anymore[1]), Firefox usually is able to just restore the session. I can't actually remember if I ever had the issue of "loosing" all my opened tabs.
A lot of my open tabs are children (I use Tree Style Tabs) of sites like HN, which I just keep open and refresh every now and then to see if the discussion has progressed, or a tree of tabs dealing with a specified topic (for example if I am researching something, or a web-shop where I just left open some interesting articles). And then there are tabs which are just open because I forgot to close them. I clean up every now and then.
[1] This is seriously baffling. I'd wager that >90% of all browser crashes can, in some way or another, be attributed to Flash. I have nothing to back up that claim, it's just anecdotal experience, but I thought this was worth mentioning.
I'm running ubuntu 11.10 with awesome wm, I have chrome open with 11 tabs, thunderbird, and pidgin and rhythmbox running. a number of terminals.
I'm using 1 gig of ram according to htop.
It is a disappointment for me to see the link get so many points in this community where you would expect the baseline of knowledge of technology to be 'above the average'.
EDIT: and the cost of programmer hours required to save a gig of RAM is orders of magnitude more.
1) Freeing memory is expensive. In order to do it properly in a running process, you need to check that each piece of memory is not in use, and that typically requires traversing large graphs. It's only something you want to do when absolutely necessary. (Hint: it's not necessary when you have 16 GB of RAM.)
2) RAM is really fast. Things in memory can be read in a few CPU clock cycles. On disk, you need to wait for a plate of metal moving at 50 mph to swing by.
3) It's totally free! It's not like unused RAM can enter power-saving mode or anything. If your RAM usage isn't at 100% you are wasting valuable space. This isn't a checkout lane where lower usage is a win for everyone. It's like an everlasting candy fountain: it all rots if you leave it, so get the whole neighborhood in for a piece.
So honestly, what are you complaining about? So chrome wants to cache hour-old pictures just in case you hit the back button 45 times. What's the big deal? YOU HAVE 6,600,000,000 FREAKIN' BYTES LEFT! You could probably fit Project Gutenberg in there. If and when you run out of memory, and pages take 15 seconds to load because your hard drive is swapping like crazy, then you have a problem. But you won't have that problem. And just because chrome is taking up 3 GB at the moment doesn't mean it won't be a nice citizen when the operating system starts to cry about low memory, which it won't do because YOU HAVE 6 MORE GIGABYTES.
Good luck with that.
1) RAM is a finite resource 2) Managing excessive RAM usage (even without swapping) requires OS effort 3) indifferent memory management is a sign of sloppy coding and usually is a memory leak 4) casual memory usage does not fly in the mobile power constrained world that is now upon us.
#1: Ram is finite
If a computer with 1 terabyte memory with all but 1 gb consumed by a memory-indifferent program, is going to behave poorly. The OS is spending a lot of time trying to find free space every time the program or OS needs new memory for any reason.
#2 Managing excessive Ram usage is not free:
The OS is constantly shuffling stuff around in memory, updating pointers, etc. If the OS has more active memory to manage then the CPU is spending more time to do this overhead work that is not available to actually run the program.
#3 Excessive memory usage = sloppy programming (ers)
Whenever I have found code that is memory-indifferent I have found bugs beyond memory usage: 1) massive http-session size. On a web server if each session takes "only" 2 megabytes, that means 1 high-end server can handle about 4000 users (8gb of session data) before bogging down. 2) memory leaks - temporary data created but never released to the OS.
#4 : mobile
Memory uses power. Remember this is a key reason why Flash is dead on mobile. Adobe could never reduce the memory/power hungriness of Flash.
As a Java developer, I routinely constrain the JavaVM's memory during testing to just above the target footprint. I want to know if there is a memory issue before it blows up in my face on a busy day.