EDIT: I try to not degrinitgrate the illiterate, but as the first commenter on this thread (you), I've gotta go "gosh!" Please tell me how I am wrong. (It's possible that I am both wrong and having a bad day)
The author argues that the number of meaningful operations a device can do per second will not keep growing exponentially. The reason he gives is that while we can add more parts that work in parallel, this has limited benefit:
The speed up starts to disappear as silicon is left idle
because there just aren’t enough different things to do.
I think there will be enough 'things to do'. Looking at real world applications of more processing power like AI and simulations, I expect them to be perfectly parallelizable.
Only a very tiny proportion of problems fit into that category. AI and simulations tend to be more easily parallelizable, but CPU-CPU communication is slow at any scale and imposes a limit to how much parallelism can be exploited.
The original statement is accurate:
> The speed up starts to disappear as silicon is left idle because there just aren’t enough different things to do
Efficient parallel algorithms require a problem that can be divided into smaller independent components and solved separately. Not every problem can be divided in this way. Crypto algorithms are the classic example; there's no way to perform round N+1 without first performing round N, and this is very much by design.
A parallel algorithm is always less efficient (per unit of computational resources) due to the extra communication overhead. That's the tradeoff for being able to utilise more total computational resources.
Sure; I meant a different algorithm. e.g. inefficient by recomputing the same result in some cases; perhaps like parallel searches that may overlap. Such strategies would not have been researched much because they seem not just inefficient but stupid. But when you have cores to burn, they may turn out to be clever...
I know of a case where similar happened when silicon became much cheaper several decades ago... the inventor said it just wouldn't have occurred to anyone before then, because it required an infeasible amount of silicon. People couldn't (or wouldn't) think in those terms.
> Only a very tiny proportions fit into that category
Actually, it is exactly the other way round: most problems in the real world are about local communications of cells in a 3 dimensional lattice. Including, relativistic limits to communication. So an ideal architecture for that is a 3D cellular automaton. With bigger cells, you've got nodes on a 3d mesh (torus). Incidentally, the topology of most supercomputers.
> Only a very tiny proportion of problems fit into that category
I think more correctly, only a tiny proportion of solutions to problems fit into that category, and I think that's largely because of the street light effect. The entire history of math and science has been written by people working out problems in their head, or on paper, which is more or less a serial process. People just don't think well in parallel, and until recently, neither did computers. We have a lot of serial solutions because that's where the light was, basically.
I think we're going to start finding a large variety of problems that turn out to have parallel solutions -- including a lot of problems that we didn't even know had solutions.
Sadly, in the real world most silicon for users will be busy rendering a cat trying to fit into a box and then falling over or alternatively attempting and failing to properly layout a word document while also accommodating macros inside said word document.
Word documents and tumbling cats don't really benefit from more than a few cores.
That's gotta finally happen SOME day, no?? I speak seriously.... Of course, by the time it happens, I wouldn't be surprised if we come up with a solution that people like better than IPv6?
We've shifted a lot of computing to the back-end where we can use more servers. We're using alternative architectures that can be built into chips that are more efficient for some workloads, e.g. ARM. We're using GPUs and now TPUs for compute-intensive tasks like machine learning that can benefit from them. It's not clear to what degree we'll be able to replace process scaling with other approaches but certainly things will change with chips unable to just get denser/faster at a predictable rate any longer.
Out of curiosity, what do people dislike about IPv6? I know next to nothing about networking, but it all I've observed from people against IPv6 is that it looks weird. It's downfall seems to be that we're so embedded into IPv4.
So ignoring IPv4, what is the problem with IPv6?
I was hoping we were still slowly making the switch. In the same way that the modern web took ages, but eventually adapted to update browsers more than once a decade and now it's reasonable to use modern JS features. I didn't give up hope on the "modern" web features.. should I give up hope on IPv6?
It's above most people's memorisation threshold, more than just looking weird. But the real limitation seems to be that nobody wants to make a change which has no benefit for them - and so long as you can still access all you need, there's no benefit to switching over to IPv4.
Nothing particularly golden about rearranging a given number of beads on a table. The only gains from cellular architectures is how many cells on a mesh you can get on a given piece of real estate.
What's left is wafer scale integration and stacking. And, making the cells smaller, which is not a silicon design problem but computer science brain damage problem.
I dunno, it's very different than things were. I built my current desktop in 2011, and aside from an ssd and upgrading from a 560 to a 970 gtaphics card, haven't made any changes. I'm running everything I want, multiple VMs all the time, Visual Studio, games at max graphics. Ten years ago, using a six year old machine meant you wouldn't have a snowball's chance in hell of running new software, but this one is chugging along admirably, and probably will for the foreseeable future. Unless everything moves to Electron, but if that happens, it really is the end times.
> Ten years ago, using a six year old machine meant you wouldn't have a snowball's chance in hell of running new software [...]
Do you have some evidence for this claim? I'm not saying I disagree -- indeed, I suspect it might be true -- but I'd appreciate some evidence to back it up.
Two that come to mind are Windows Vista upgrades, where many XP-capable machines were just not powerful enough to handle it. The other would be video games, which were constantly pushing the envelope. Consoles may have a lot to do with the stagnation in the latter case, however.
Right. I bought a Pentium 3 800/GeForce 2 GTS in the summer of 2000. And by 2003 Homeworld 2 and Knights of the Old Republic had a Pentium 3 833mhz and a Pentium 3 1ghz respectively.
I do not have statistically significant evidence, but as another data point I have the same experience: both my home and work machines are 7-8 years old and work just fine (30" monitors, decent performance with numpy, MATLAB, compiling, perfectly fine graphics, etc.).
But around 2000 I remember always carefully looking at the system requirements section of games to check which of those would work on my 2-3 year old computer and for which my machine is too old -- and find a significant rejection rate (25-50%).
I don't know if that needs any evidence of the person was simply around back then. Basically this is saying that 2001 hardware wouldn't be able to run the new software in 2007, which at least to my recollection, is absolutely true. Not only is there 32bit vs 64bit, we also started to see more cores rather than the standard single core in 2001
I remember receiving the game as a gift and having to tell my grandma that my single core PC wouldn't be able to play it. I think my PC was only 3 years old at the time, not even 6.
I think you're miss-remembering. Starcraft 2 Wings of Liberty was released in 2010. I played it on release on my Core 2 Quad/GeForce 8800 GTS machine that I built in 2009.
Ten years ago, in 2007, using a machine from 2001. Probably a second generation Pentium 4 (https://www.intel.com/pressroom/kits/quickrefyr.htm#2001) or Athlon XP, with less than a gig of RAM or if you had upgraded it by 2007, probably 2 gigs or less. A
Yeah, that would be pretty crummy in 2007 even if you'd bought a high end system in 2001. If it was entry level, forget it.
I mean you'd still be able to compute things in 2007, but the overall experience would be sub-par. And a lot of things would run technically, but it would be painful enough you wouldn't even want to bother.
My take on it is that multi-core CPU's caused the amount of computing power available on even budget systems to grow exponentially faster than software demands for that power. Not only is it more cores, but the power of a single core has also increased steadily. And the first Core Duo's were easily faster per core than the previous fastest single core Pentiums on top of having two cores. More power, better multi-tasking, it's a no brainer. The amount of computing power has increased much faster while software demands just haven't kept pace, and most software just doesn't require 3.5ghz and four cores for an optimal experience. The result is five, six, seven year old PC's are still perfectly serviceable. The same just generally wasn't true in the era of single core PC's.
Recently I fished out my Toshiba Libretto 30, which I bought in 1998. It had just about enough processing power to play MP3s using the Fraunhofer codec, but not run WinAmp.
Over the next couple of years the use of MP3 exploded: Napster, the iPod, iTunes. The boundary for "can this system decode mp3 in real time" was somewhere around the late 90s. Now of course it's not an issue on any device.
Like others have said, it is something that was extremely apparent to anyone around 10+ years ago that made use of computers. Back then, a computer was simply 2x as fast every 18 months. If you had a game that ran at 30fps, and you bought a new computer 18 months later, it would run at 60fps. Likewise, a new game would run at half speed on a computer that was 18 months old.
While there are limits, the scaling largely held up over time. A game that ran at 15 fps (60 fps wasn't really a thing back then, usually you hit 15-30 fps) would be 32x faster 7.5 years later. Ditto with most software.
The vast majority of speedup, from a processing standpoint, these days is all from increased parallelism (via more cores, or more SIMD-like instructions in the core), and because of Amdahl's "law", it can be very difficult to see benefits from increased parallelism.
We're still seeing Moore's law for drive space, RAM, and somewhat bus speeds, and that's also where you get a lot more speedup.
Not really. 2Tb became a standard a long time ago, and it didn't change afterwards. Speed did not improve much either, especially latency (it cannot really improve), except by switching to SSDs, but then you lose capacity, so it is not a continuous improvement.
> RAM, and somewhat bus speeds, and that's also where you get a lot more speedup.
SDRAM is slow as molasses, as far as latency is concerned (the "Random Access" in R.A.M.). The improvements have been incredibly small in this domain. DDR4 access times are slower than SRAM access time from perhaps 20 years ago. It has however much larger capacity and larger throughput, indeed.
> and somewhat bus speeds, and that's also where you get a lot more speedup.
Same problem with buses, their frequency raised a lot, but that happened with switching from parallel to serial buses, leading to a lot of latency (and overload), not only on the bus itself because of the protocol, but also before and after because you need to serialise and then deserialise the data for you need it in a parallel form anyway.
For example, the first SATA, despite its impressive frequency of 1.5 GHz, did not do better than the good old IDE/PATA with the latest UDMA.
Same for PCI Express, it showed 2.5 GHz frequency, but its performance was not better than AGP or top PCI with their < 100 Mhz frequencies.
Then to improve the throughput you had to add multiple lines of serial buses, which again add some need for synchronisation and thus extra latency.
Sorry, I flipped values on you - I meant capacity more than speed. Drives are still doubling in capacity at roughly the same rate, ditto with RAM, and also buses. "Speed" is still improving, but as you point out, not in a raw "2x faster every 18 months" manner. All of that extra space and throughput results in zippier machines, in practice. Sure the RAM isn't faster, but being able to keep me 20+ tabs I seem to chronically have open makes everything faster because I'm not swapping from drive. My games load faster because they are sucking more from disk through fatter buses, and the additional latency isn't noticeable then.
But can you play PUBG at max settings?
It's not just snark: PUBG is a new kind of game, with a level of realism, open-worldness and massive player numbers that hasn't been a hit before. Though it's said to be "not optimized", I think many of its performance issues won't be easily optimized away, because they are intrinsic to this new type of game. (Fortnite runs smoother, but has far less realism, amd its netcode has comparable latency),
The Battlefield series are heavily optimized and have much much better graphics than PUBG. They are in the same domain space (huge maps, 50+ players concurrently, etc.)
Battlefield does use hardware very much, but that PUBG is not well optimized is fact.
Agree that battlefield games look amazing and play smooth. And I'll allow 50 as close enough to 100.
But the maps are not only smaller (largest I found was 1.5x1.5km), but very different in nature: they are designed for traditional fps gameplay, and incorporate techniques to enable fast rendering (e.g. limited visibility so submaps can be streamed in as you move between regions; mountain ranges seem impassable, again forming a gameplay element and aiding rendering).
The PUBG map is less "designed", and more like a real place. All areas are accessible, and although terrain undulations do eventually close-off areas, they are far larger areas, and they don't seem designed for rendering but for realism and a different kind of gameplay.
Finally, a single figure to dramatize the map contrast: Battlefield 1 shipped with 9 maps.
PUBG shipped with 1.
massive player numbers? what about wow? and not the currect wow which has zones but old one from vanilla to wotlk
I think there were a lot more players than 100
Sounds like anecdotal evidence that Moore's law has ended: If Moore's law was still in play then your 2011 computer would struggle to run 2017 software which is geared towards 2017 computers. But if Moore's law had ended then 2017 computers wouldn't be much faster than 2011 computers, so 2017 software would work fine on 2011 computers
Moore's law isn't based on a machine ability to run software though. One could argue Moore's law isn't relevant anymore but that's different from whether it's dead or not.
Either because adoption of new hardware has slowed down, or because we hit a tipping point where CPU power available is much greater than most software's need to utilize it means that performance gains exist. Modern hardware is faster than 2011 or 2013 hardware but the pressure to upgrade because things run crummy on old hardware just isn't there like it used to be.
Which makes it a 32nm Sandy Bridge, likely a i7 2600K, which were really popular. I had one too. That's a 3.4GHz part with a single-core turbo to 3.8GHz, and an IPC maybe 5-10% slower than Kaby/Coffee Lake (which are at 14nm). So that puts it about the same performance level as the lowest end (i3 8100) offering on the current process, and at about 60-70% higher power.
So it's not that things have stopped. It's just much slower progress than you're used to. The desktop world is power-agnostic, so you're missing most of the innovation. Comparing a Sandy Bridge laptop to modern equivalents is a much more striking contrast: my current laptop (with a high performance CPU) will literally do a full day of work on a single battery, where nothing in 2011 was available to do that.
The end of Moore's Law doesn't necessarily have to imply a deceleration in technological evolution, as Ray Kurzweil and others have pointed out. In their view, Moore's Law is just one single phase in the "law of accelerating returns".
Which is what this article also argues. As we can no longer rely on Moore's exponential increases we will start looking elsewhere to evolve technology. A new "golden age" as he calls it.
Maybe that'll mean looking beyond von neumann for general purpose architecture. Maybe it'll mean designing a lot more specialised architectures, similar to how the GPU is specialised for graphics processing (and useful for some other tasks) or how most mobile devices come with specialised processors. Maybe it'll mean something completely different.
That makes little sense. We've been riding this particular technology called transistors we've stumbled on for the past 60 or so years. Now the technology is mature, and to make serious progress, we have to get lucky again and stumble on another technology. It doesn't just happen on its own.
As soon as a plot like this shows a "stop", there will pop up lots of possibilities for research and funding thereof. Physical restrictions predict this stop within the next years.
> Once you get down to 5nm features they are only about 20 silicon atoms wide.
This is a complete misunderstanding of modern processes.
It’s conflating feature size with some measure like “wire width”. Feature size is best thought of as the tightest corner radius you can cut in a piece of wood. Actual logic (and wires) are far larger than this. There’s actually a shocking amount of room “down there”, still. The problem is that lithography probably won’t scale down after some magic number (say 2 or 3 nm), and we’ll have to do something else clever. It’s been a while since I looked into this, but IBM’s smallest AFM-created transistor is ~10 atoms, which is something like 4 orders of magnitude (or more!) smaller than, say, 10nm.
Moreover, the node numbers do no longer relate to any specific geometric feature, but are computed from how many standard (memory) cells fit a given area.
Sort of: thinner wires have higher resistance, so what tends to happen is you get slightly more heat but in a much smaller space. Heat dissipation has become a major limiting factor.
Yes. Most of the power goes into charging and discharging the capacitance inherent in each wire and transistor gate every time a signal changes. When you shrink by 2x, the capacitance goes down by about 4x (because it's proportional to area) and power goes down 4x also.
(It's not exactly square-law, because the sides of wires have capacitance too, but at least at larger sizes it was close.)
Could you clarify the "4 orders of magnitude" comment?
~10 atoms is ballpark 2nm. If you are referring to a length dimension with your comment, four orders of magnitude larger would have some dimension on the order of 20um, corresponding to a 1970s process.
One point that I wish the article had expanded on is the impact on software development. In a recent speech, John Carmack talks of how the power of the desktop will never reach mobile.
This is incredibly exciting. I see the end of Moores law as the return of the era of performance-oriented software development. The past decade was ruled by software methodologies which optimised for developer time at the cost of performance. Moores law was the crutch on which such latency ridden software limped into userland.
With the stabilization of hardware performance, developers can once again put dedicated time and effort into crafting performant software with the assurance that hardware gains will not render their efforts useless in 2 years.
> This is incredibly exciting. I see the end of Moores law as the return of the era of performance-oriented software development.
And I see this as the exact opposite. Optimizing for developer time is one of the largest productivity enhancements we have. A true/pure positive feedback loop in technological progress. Optimizing for performance is getting little real gain for large human cost.
The end game of optimized for developer efficiency is we can design and implement "anything" we can think of in almost no time flat. The end game for optimizing for performance is it takes significantly longer to develop a program just to run it on the equivalent of 5 years ago hardware.
An analogy: it's early in human writing. People have trouble expressing average thoughts. We can either teach society to have a better grasp on writing and expressing complex ideas precisely. Or we can teach people to write really small because we're running out of paper.
The positive feedback loop of improved development is enormous.
The software methodology that has evolved over the last two decades has focused on development time. As such, there has not been much focus/research on performance-first methodologies. I see no reason that developer time cannot be optimized in a performance-first methodology, it's just that a developer time oriented methodology has years of fine tuning in it.
An example for this is video game development. If you look at some talks given by Mike Acton, you can see that it is possible to ship a game while optimizing for developer time. A triple A game is arguably one of the most intricate and complex projects under the label of "Software Engineering". I see no reason why we can't learn from such sub-fields and apply it to general software development.
I also disagree with you saying that "Optimizing for performance is getting little real gain for large human cost". Not optimizing for performance has had led to the creation of unresponsive text editors, slow word processors and a plethora of other latency-ridden software. This in turn affects the user and unfortunately alters their expectations for software.
There is no technological reason why Visual Studio 2017 should not run on a 5 year old laptop, yet that is the state we are in.
Of course - when it's performing well, it's what this site shines at.
> I see no reason that developer time cannot be optimized in a performance-first methodology
Well clearly it can but it won't be nearly as much of a priority. Just like performance isn't as much of a priority when focusing on developer time. It's not clear what you're specifically rebutting in this section. We're both agreeing that developer-performance will improve significantly less in this model.
And unresponsive text editor gives at most what a 3x slowdown in performance. Very little of a programmer's time is actually inputting text or data. While let's take the other extreme to show a point: Building a distributed transactional relational data driven application from scratch (because our tools weren't built because people would have been optimizing performance of late 90s hw) would be a 1,000x loss of developer productivity.
The scaling factor of productivity just utterly dwarfs human interface performance.
> There is no technological reason why Visual Studio 2017 should not run on a 5 year old laptop, yet that is the state we are in.
True, but almost everyone would rather 2017 VS not run on a 5 year old laptop, then the best available product right now be Visual Studio 2005 era functionality highly optimized so it can still run on a 5 year old laptop.
If text editors slowed down by 3x, I'd be happy. The latency I am complaining about is orders of magnitude higher than 3x. This is the most rigorous benchmarking that I am aware of:
Each of those tables show that the difference in performance is not 3x but at least an order of magnitude, if not more. I think this is where our disagreement stems from.
Your next point is valid, it would not make sense to develop such an application from scratch. However, I'm not talking about rebuilding from scratch merely prioritizing performance. This can be done by learning from Data Oriented Design where developers give a lot of importance to performance critical issues such as cache locality. This can be done without rewriting our tools from scratch, it just means that the developer will have to understand how memory works. It's not a tools issue, it's a knowledge issue.
Your last point on Visual Studio makes me want to re-iterate the core of my argument: Performance and developer time optimizations need not be mutually exclusive.
> Performance and developer time optimizations need not be mutually exclusive
Completely agreed. But since they're not mutually-exclusive why aren't you happy with the status quo? As stated we already can work on some perf improvements while focusing on developer productivity. I believe the reason why you aren't is logically, you'd rather the balance be shifted more towards performance. Which makes sense, but is illustrative of why saying "they aren't mutually exclusive" isn't a counterpoint. We're both discussing where the focus should be - not that the only options are all are nothing. If non mutually-exclusive were the solution, you wouldn't be happy to see this change and a shift towards performance.
> Each of those tables show that the difference in performance is not 3x but at least an order of magnitude, if not more. I think this is where our disagreement stems from.
Agreed. But taking sublime text as an example. I see peak ide/editor is roughly 7x faster across various tests, and slowest ide/editor is roughly 7x slower. Even combining both we see at most a 50x change in technical measures. Not in human productivity. The majority of the time people are sitting waiting. I think most people would argue that it doesn't take user 50x as long to write any program in Eclips/IDEA as it does in GVim. So that technical slowdown while annoying has a significantly lower impact on user performance.
Any my argument isn't that users shouldn't have to know about performance, or that performance is bad, it's that resources spent on performance come from being spent on productivity.
Here is an alternate view: Writing most software improves developer performance (text editors, communication tools, file system drivers, languages, compilers...). That better software ecosystem allowed developers to then write more / better software.
Over some timescale we can approximate this as P(t2) = P(t1) * e^(rt). Where P(x) is developer productivity at time x. And t is a timespan. And r is the "rate of return" or productivity improvement by writing software. Saying a developer now needs to spend 30% of their time on performance means we now have: P(t2) = P(t1) e^(.7 * r * t). Due to exponential growth this means significantly less productivity over the long term. This is no different than annually getting 20% returns or 13% returns in the stock market. I sounds like a little, but over a long time scale (or a high scaling rate) this adds up significantly. Meaning right now in 2017 we'd be using a decade or older technology (If peak performance slowdown had hit in some year in the past).
Overall I appreciate the debate and enjoy hearing different opinions. In this one, I feel we may just have different values here in the software space. I feel in an ideal world a developer would spend zero-time on performance. Since we don't live in an ideal world we have to spend some, but every bit of time spent on performance is time not spent on building a product. So we should want to minimize the amount of engineering time focused performance. My take is that you feel differently.
I think your last sentence sums it up perfectly. It does seem that we have different views on software. Rebutting your points would just rehash my above comment.
The only new point I'd like to add is one that requires imagination. Imagine if we had adopted performance oriented mindset 20 years ago, instead of today and we solved the developer productivity conundrum that we both care about so much. What a wonderful state software would be in, where IDE's can be installed in seconds, each action is instant and the engineer is completely intimate with every aspect of their baby. This depth of knowledge offers a certain creativity that results in amazing things. It lets engineers rise above what is good for the business and gives them the freedom to push software to it's utter limit. That's evolution. That's progress. That's engineering.
PS: The reason I quoted John Carmack specifically in my first comment is because he is such an engineer.
How do we think about AI/ML stack moving into GPUs and custom chips? Maybe Moore's law as envisioned is ending but it is probably changing in new ways to help scale the jobs and tasks we have based on the type of task.
It may not be the end of Moore's Law, but the increase in resources needed to run modern software has definitely slowed down.
In 2017, my mom is using my old 2006 era Core Duo 1.66Ghz Mac Mini running Windows 7 with only 1.5GB of RAM. She mostly uses it for tutoring when she doesn't want her students on her main computer. It still runs Chrome and Office surprisingly well. It has plenty of USB2 ports, Bluetooth and Gig E Ethernet.
i cant imagine getting any use out of the computer I used in 1995 - a PowerMac 6100/60 with a PPC 601/60Mhz processor and a 486Dx/2 66Mhz Dos Compatibility Card in 2006.
My Plex Server is a 2008 era Core 2 Duo 2.66Ghz Dell Laptop running Windows 10 with 4Gb of RAM.
90 comments
[ 5.3 ms ] story [ 147 ms ] threadEDIT: I try to not degrinitgrate the illiterate, but as the first commenter on this thread (you), I've gotta go "gosh!" Please tell me how I am wrong. (It's possible that I am both wrong and having a bad day)
Unfortunately, the real world does not meet your expectation. You're talking about a class of problems called 'embarrassingly parallel': https://en.wikipedia.org/wiki/Embarrassingly_parallel
Only a very tiny proportion of problems fit into that category. AI and simulations tend to be more easily parallelizable, but CPU-CPU communication is slow at any scale and imposes a limit to how much parallelism can be exploited.
The original statement is accurate:
> The speed up starts to disappear as silicon is left idle because there just aren’t enough different things to do
Efficient parallel algorithms require a problem that can be divided into smaller independent components and solved separately. Not every problem can be divided in this way. Crypto algorithms are the classic example; there's no way to perform round N+1 without first performing round N, and this is very much by design.
I know of a case where similar happened when silicon became much cheaper several decades ago... the inventor said it just wouldn't have occurred to anyone before then, because it required an infeasible amount of silicon. People couldn't (or wouldn't) think in those terms.
Actually, it is exactly the other way round: most problems in the real world are about local communications of cells in a 3 dimensional lattice. Including, relativistic limits to communication. So an ideal architecture for that is a 3D cellular automaton. With bigger cells, you've got nodes on a 3d mesh (torus). Incidentally, the topology of most supercomputers.
Uh?
I think more correctly, only a tiny proportion of solutions to problems fit into that category, and I think that's largely because of the street light effect. The entire history of math and science has been written by people working out problems in their head, or on paper, which is more or less a serial process. People just don't think well in parallel, and until recently, neither did computers. We have a lot of serial solutions because that's where the light was, basically.
I think we're going to start finding a large variety of problems that turn out to have parallel solutions -- including a lot of problems that we didn't even know had solutions.
Word documents and tumbling cats don't really benefit from more than a few cores.
Better than ideological proselytizing and political propaganda.
https://zwischenzugs.com/2017/11/11/towards-a-national-compu...
In there is an image from 1965 of a graph similar to Moore's Law:
https://zwischenzugs.files.wordpress.com/2017/11/20171105_11...
I'd be interested to know if anyone has the first edition of this book, or whether there were other similar graphs floating around at this time.
Note that the image also shows a storage graph alongside computing power. Does anyone know if this graph was continued, or extended into a 'law' also?
http://www.sciencedirect.com/science/article/pii/S0375960102...
It sort of has.
We've shifted a lot of computing to the back-end where we can use more servers. We're using alternative architectures that can be built into chips that are more efficient for some workloads, e.g. ARM. We're using GPUs and now TPUs for compute-intensive tasks like machine learning that can benefit from them. It's not clear to what degree we'll be able to replace process scaling with other approaches but certainly things will change with chips unable to just get denser/faster at a predictable rate any longer.
So ignoring IPv4, what is the problem with IPv6?
I was hoping we were still slowly making the switch. In the same way that the modern web took ages, but eventually adapted to update browsers more than once a decade and now it's reasonable to use modern JS features. I didn't give up hope on the "modern" web features.. should I give up hope on IPv6?
And ISPs are holding us up too.
What's left is wafer scale integration and stacking. And, making the cells smaller, which is not a silicon design problem but computer science brain damage problem.
Do you have some evidence for this claim? I'm not saying I disagree -- indeed, I suspect it might be true -- but I'd appreciate some evidence to back it up.
But around 2000 I remember always carefully looking at the system requirements section of games to check which of those would work on my 2-3 year old computer and for which my machine is too old -- and find a significant rejection rate (25-50%).
Starcraft 2 came out in 2007. These are the original system requirements. https://www.lifewire.com/starcraft-ii-wings-of-liberty-requi...
I remember receiving the game as a gift and having to tell my grandma that my single core PC wouldn't be able to play it. I think my PC was only 3 years old at the time, not even 6.
Blizzard announced they were making SC2 in 2007. https://en.wikipedia.org/wiki/StarCraft_II:_Wings_of_Liberty... there wasn't even a closed beta until early 2010.
Regardless, the minimum requirements you posted are single core CPU's.
* 1998: 32-bit, 500 MHz, single core, 128MiB of RAM IIRC (we later upgraded it to ~500 MiB). 7 GB of disk.
* 2007: 64-bit, ~2.5 MHz, dual core, 4, later 8 GiB of RAM. Hundreds of GB of disk.
There's essentially an order of magnitude improvement…
The systems from 2000 that I was familiar with had 512MB of less.
Not to mention the 32 bit to 64 bit transition that allowed us to use more RAM in Windows.
IMHO, the upgrade from HDD to SSD was the most noticeable speed jump.
Yeah, that would be pretty crummy in 2007 even if you'd bought a high end system in 2001. If it was entry level, forget it.
I mean you'd still be able to compute things in 2007, but the overall experience would be sub-par. And a lot of things would run technically, but it would be painful enough you wouldn't even want to bother.
My take on it is that multi-core CPU's caused the amount of computing power available on even budget systems to grow exponentially faster than software demands for that power. Not only is it more cores, but the power of a single core has also increased steadily. And the first Core Duo's were easily faster per core than the previous fastest single core Pentiums on top of having two cores. More power, better multi-tasking, it's a no brainer. The amount of computing power has increased much faster while software demands just haven't kept pace, and most software just doesn't require 3.5ghz and four cores for an optimal experience. The result is five, six, seven year old PC's are still perfectly serviceable. The same just generally wasn't true in the era of single core PC's.
Over the next couple of years the use of MP3 exploded: Napster, the iPod, iTunes. The boundary for "can this system decode mp3 in real time" was somewhere around the late 90s. Now of course it's not an issue on any device.
While there are limits, the scaling largely held up over time. A game that ran at 15 fps (60 fps wasn't really a thing back then, usually you hit 15-30 fps) would be 32x faster 7.5 years later. Ditto with most software.
The vast majority of speedup, from a processing standpoint, these days is all from increased parallelism (via more cores, or more SIMD-like instructions in the core), and because of Amdahl's "law", it can be very difficult to see benefits from increased parallelism.
We're still seeing Moore's law for drive space, RAM, and somewhat bus speeds, and that's also where you get a lot more speedup.
> We're still seeing Moore's law for drive space,
Not really. 2Tb became a standard a long time ago, and it didn't change afterwards. Speed did not improve much either, especially latency (it cannot really improve), except by switching to SSDs, but then you lose capacity, so it is not a continuous improvement.
> RAM, and somewhat bus speeds, and that's also where you get a lot more speedup.
SDRAM is slow as molasses, as far as latency is concerned (the "Random Access" in R.A.M.). The improvements have been incredibly small in this domain. DDR4 access times are slower than SRAM access time from perhaps 20 years ago. It has however much larger capacity and larger throughput, indeed.
> and somewhat bus speeds, and that's also where you get a lot more speedup.
Same problem with buses, their frequency raised a lot, but that happened with switching from parallel to serial buses, leading to a lot of latency (and overload), not only on the bus itself because of the protocol, but also before and after because you need to serialise and then deserialise the data for you need it in a parallel form anyway.
For example, the first SATA, despite its impressive frequency of 1.5 GHz, did not do better than the good old IDE/PATA with the latest UDMA.
Same for PCI Express, it showed 2.5 GHz frequency, but its performance was not better than AGP or top PCI with their < 100 Mhz frequencies.
Then to improve the throughput you had to add multiple lines of serial buses, which again add some need for synchronisation and thus extra latency.
Battlefield does use hardware very much, but that PUBG is not well optimized is fact.
But the maps are not only smaller (largest I found was 1.5x1.5km), but very different in nature: they are designed for traditional fps gameplay, and incorporate techniques to enable fast rendering (e.g. limited visibility so submaps can be streamed in as you move between regions; mountain ranges seem impassable, again forming a gameplay element and aiding rendering).
The PUBG map is less "designed", and more like a real place. All areas are accessible, and although terrain undulations do eventually close-off areas, they are far larger areas, and they don't seem designed for rendering but for realism and a different kind of gameplay.
Finally, a single figure to dramatize the map contrast: Battlefield 1 shipped with 9 maps. PUBG shipped with 1.
Either because adoption of new hardware has slowed down, or because we hit a tipping point where CPU power available is much greater than most software's need to utilize it means that performance gains exist. Modern hardware is faster than 2011 or 2013 hardware but the pressure to upgrade because things run crummy on old hardware just isn't there like it used to be.
Which makes it a 32nm Sandy Bridge, likely a i7 2600K, which were really popular. I had one too. That's a 3.4GHz part with a single-core turbo to 3.8GHz, and an IPC maybe 5-10% slower than Kaby/Coffee Lake (which are at 14nm). So that puts it about the same performance level as the lowest end (i3 8100) offering on the current process, and at about 60-70% higher power.
So it's not that things have stopped. It's just much slower progress than you're used to. The desktop world is power-agnostic, so you're missing most of the innovation. Comparing a Sandy Bridge laptop to modern equivalents is a much more striking contrast: my current laptop (with a high performance CPU) will literally do a full day of work on a single battery, where nothing in 2011 was available to do that.
Maybe that'll mean looking beyond von neumann for general purpose architecture. Maybe it'll mean designing a lot more specialised architectures, similar to how the GPU is specialised for graphics processing (and useful for some other tasks) or how most mobile devices come with specialised processors. Maybe it'll mean something completely different.
The future of computing is exciting again.
Meanwhile, I like the idea of physically massive GPUs if they can't get smaller (also doing ML, CFD etc)
It's not so easy to increase clock speed.
[0] https://en.wikipedia.org/wiki/Dennard_scaling [1] https://en.wikipedia.org/wiki/Koomey%27s_law
Moore's law is the observation that the number of transistors in a dense integrated circuit doubles approximately every two years.
https://en.wikipedia.org/wiki/Moore%27s_law
As soon as a plot like this shows a "stop", there will pop up lots of possibilities for research and funding thereof. Physical restrictions predict this stop within the next years.
I imagine it is this: https://en.wikipedia.org/wiki/Transistor_count
This is a complete misunderstanding of modern processes.
It’s conflating feature size with some measure like “wire width”. Feature size is best thought of as the tightest corner radius you can cut in a piece of wood. Actual logic (and wires) are far larger than this. There’s actually a shocking amount of room “down there”, still. The problem is that lithography probably won’t scale down after some magic number (say 2 or 3 nm), and we’ll have to do something else clever. It’s been a while since I looked into this, but IBM’s smallest AFM-created transistor is ~10 atoms, which is something like 4 orders of magnitude (or more!) smaller than, say, 10nm.
(It's not exactly square-law, because the sides of wires have capacitance too, but at least at larger sizes it was close.)
See https://en.wikipedia.org/wiki/Dennard_scaling for details.
~10 atoms is ballpark 2nm. If you are referring to a length dimension with your comment, four orders of magnitude larger would have some dimension on the order of 20um, corresponding to a 1970s process.
https://m.youtube.com/watch?v=vlYL16-NaOw&t=14m20s
This is incredibly exciting. I see the end of Moores law as the return of the era of performance-oriented software development. The past decade was ruled by software methodologies which optimised for developer time at the cost of performance. Moores law was the crutch on which such latency ridden software limped into userland.
With the stabilization of hardware performance, developers can once again put dedicated time and effort into crafting performant software with the assurance that hardware gains will not render their efforts useless in 2 years.
And I see this as the exact opposite. Optimizing for developer time is one of the largest productivity enhancements we have. A true/pure positive feedback loop in technological progress. Optimizing for performance is getting little real gain for large human cost.
The end game of optimized for developer efficiency is we can design and implement "anything" we can think of in almost no time flat. The end game for optimizing for performance is it takes significantly longer to develop a program just to run it on the equivalent of 5 years ago hardware.
An analogy: it's early in human writing. People have trouble expressing average thoughts. We can either teach society to have a better grasp on writing and expressing complex ideas precisely. Or we can teach people to write really small because we're running out of paper.
The positive feedback loop of improved development is enormous.
The software methodology that has evolved over the last two decades has focused on development time. As such, there has not been much focus/research on performance-first methodologies. I see no reason that developer time cannot be optimized in a performance-first methodology, it's just that a developer time oriented methodology has years of fine tuning in it.
An example for this is video game development. If you look at some talks given by Mike Acton, you can see that it is possible to ship a game while optimizing for developer time. A triple A game is arguably one of the most intricate and complex projects under the label of "Software Engineering". I see no reason why we can't learn from such sub-fields and apply it to general software development.
I also disagree with you saying that "Optimizing for performance is getting little real gain for large human cost". Not optimizing for performance has had led to the creation of unresponsive text editors, slow word processors and a plethora of other latency-ridden software. This in turn affects the user and unfortunately alters their expectations for software.
There is no technological reason why Visual Studio 2017 should not run on a 5 year old laptop, yet that is the state we are in.
Of course - when it's performing well, it's what this site shines at.
> I see no reason that developer time cannot be optimized in a performance-first methodology
Well clearly it can but it won't be nearly as much of a priority. Just like performance isn't as much of a priority when focusing on developer time. It's not clear what you're specifically rebutting in this section. We're both agreeing that developer-performance will improve significantly less in this model.
And unresponsive text editor gives at most what a 3x slowdown in performance. Very little of a programmer's time is actually inputting text or data. While let's take the other extreme to show a point: Building a distributed transactional relational data driven application from scratch (because our tools weren't built because people would have been optimizing performance of late 90s hw) would be a 1,000x loss of developer productivity.
The scaling factor of productivity just utterly dwarfs human interface performance.
> There is no technological reason why Visual Studio 2017 should not run on a 5 year old laptop, yet that is the state we are in.
True, but almost everyone would rather 2017 VS not run on a 5 year old laptop, then the best available product right now be Visual Studio 2005 era functionality highly optimized so it can still run on a 5 year old laptop.
https://pavelfatin.com/typing-with-pleasure/
Each of those tables show that the difference in performance is not 3x but at least an order of magnitude, if not more. I think this is where our disagreement stems from.
Your next point is valid, it would not make sense to develop such an application from scratch. However, I'm not talking about rebuilding from scratch merely prioritizing performance. This can be done by learning from Data Oriented Design where developers give a lot of importance to performance critical issues such as cache locality. This can be done without rewriting our tools from scratch, it just means that the developer will have to understand how memory works. It's not a tools issue, it's a knowledge issue.
Your last point on Visual Studio makes me want to re-iterate the core of my argument: Performance and developer time optimizations need not be mutually exclusive.
Completely agreed. But since they're not mutually-exclusive why aren't you happy with the status quo? As stated we already can work on some perf improvements while focusing on developer productivity. I believe the reason why you aren't is logically, you'd rather the balance be shifted more towards performance. Which makes sense, but is illustrative of why saying "they aren't mutually exclusive" isn't a counterpoint. We're both discussing where the focus should be - not that the only options are all are nothing. If non mutually-exclusive were the solution, you wouldn't be happy to see this change and a shift towards performance.
> Each of those tables show that the difference in performance is not 3x but at least an order of magnitude, if not more. I think this is where our disagreement stems from.
Agreed. But taking sublime text as an example. I see peak ide/editor is roughly 7x faster across various tests, and slowest ide/editor is roughly 7x slower. Even combining both we see at most a 50x change in technical measures. Not in human productivity. The majority of the time people are sitting waiting. I think most people would argue that it doesn't take user 50x as long to write any program in Eclips/IDEA as it does in GVim. So that technical slowdown while annoying has a significantly lower impact on user performance.
Any my argument isn't that users shouldn't have to know about performance, or that performance is bad, it's that resources spent on performance come from being spent on productivity.
Here is an alternate view: Writing most software improves developer performance (text editors, communication tools, file system drivers, languages, compilers...). That better software ecosystem allowed developers to then write more / better software.
Over some timescale we can approximate this as P(t2) = P(t1) * e^(rt). Where P(x) is developer productivity at time x. And t is a timespan. And r is the "rate of return" or productivity improvement by writing software. Saying a developer now needs to spend 30% of their time on performance means we now have: P(t2) = P(t1) e^(.7 * r * t). Due to exponential growth this means significantly less productivity over the long term. This is no different than annually getting 20% returns or 13% returns in the stock market. I sounds like a little, but over a long time scale (or a high scaling rate) this adds up significantly. Meaning right now in 2017 we'd be using a decade or older technology (If peak performance slowdown had hit in some year in the past).
Overall I appreciate the debate and enjoy hearing different opinions. In this one, I feel we may just have different values here in the software space. I feel in an ideal world a developer would spend zero-time on performance. Since we don't live in an ideal world we have to spend some, but every bit of time spent on performance is time not spent on building a product. So we should want to minimize the amount of engineering time focused performance. My take is that you feel differently.
The only new point I'd like to add is one that requires imagination. Imagine if we had adopted performance oriented mindset 20 years ago, instead of today and we solved the developer productivity conundrum that we both care about so much. What a wonderful state software would be in, where IDE's can be installed in seconds, each action is instant and the engineer is completely intimate with every aspect of their baby. This depth of knowledge offers a certain creativity that results in amazing things. It lets engineers rise above what is good for the business and gives them the freedom to push software to it's utter limit. That's evolution. That's progress. That's engineering.
PS: The reason I quoted John Carmack specifically in my first comment is because he is such an engineer.
You can't improve productivity in the same way.
For personal projects I'll rather take 10x better performance over 100% more productivity.
Why? Because my time is valuable enough to not waste it on slow software.
In 2017, my mom is using my old 2006 era Core Duo 1.66Ghz Mac Mini running Windows 7 with only 1.5GB of RAM. She mostly uses it for tutoring when she doesn't want her students on her main computer. It still runs Chrome and Office surprisingly well. It has plenty of USB2 ports, Bluetooth and Gig E Ethernet.
i cant imagine getting any use out of the computer I used in 1995 - a PowerMac 6100/60 with a PPC 601/60Mhz processor and a 486Dx/2 66Mhz Dos Compatibility Card in 2006.
My Plex Server is a 2008 era Core 2 Duo 2.66Ghz Dell Laptop running Windows 10 with 4Gb of RAM.