This is so true - so many bad Perl, Python and R jobs “requiring” supercomputer resources that a decent programmer would allow to run at better speed on a single modern server!
There was a post on HN back in 2016 pointing out that some tasks in Hadoop could be executed more quickly by just using standard Unix tools and pipes. It annoys me to no end that very few languages provide out of the box mechanisms to be able to emulate that sort of simple parallelism. Although I did find a Google proposal for the C++ standard library that, at the very least, made it clear how to roll this functionality myself should I so choose.
Yes, but it was more than that in the difference. It's that for many (maybe most) tasks, a simple pipeline-style parallelization makes much more sense than going through all the pain of map-reduce. If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
> If you've got five steps and five threads and a big pile of data, it can make a lot more sense to try to have each step running in its own thread and passing a processed chunk of data to the next step rather than having five threads trying to each take some chunk of data and process it through the five steps itself. Factories have done this since the 19th century, but most programming languages make it hard.
You can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.
"Scalability! But at what COST?" is another classic in the category (though it's not precisely about on-box parallelism if I recall; the authors' acronym "COST" is the "Configuration that Outperforms a Single Thread").
Part of the reason behind this is that it's really, really hard to hire good programmers in academia -- salaries are difficult to put into grants and equipment is easier. It may well be the case that an übernerd could get your algo a significant runtime improvement, but sometimes you're massively unlikely to actually be able to get an übernerd. On the other hand, saying "I and doing big science, give me time on this cluster" is often more of a positive thing to write.
And at the same time we give indecent salaries to PIs, Deans and basketball coaches. We waste large amount of money on inefficient processes and we take 5 years to develop projects that would have taken 6 months with a good dev and in the end would have cost less in maintenance and improvement. I needed to hire devs for a project, it was impossible to do.
What I've seen is that low-salaried ops people are viewed as a substitute for capital expenditure in this environment. It strongly discourages purchasing software and even non-white box hardware. "Just have IT cobble together something open source", is the message from administration and PIs. The result is ramshackle, inefficient systems running on low bidder hardware from often dubious "HPC" resellers.
I, too, am a software engineer dealing with researcher code on a daily basis, and I'm also the person that oversees most of our research compute hardware [sysadmin in a past life]. Some of my other favorite gems:
* "I need a terabyte of RAM"
* "50k samples ran in the blink of an eye. 100k ran in an hour. 1 million took a week. I'm about to do the full 50 million observations, I expect it'll take ten days"
* "This code is always slow... that's a lot of math it does"
* "Profiler? Never heard of such a thing"
* "Big-O? What's that?"
* "I only have ten days' money, and it took me that long. Afraid I can't afford your time to dig into this"
* "AWS will fix this"
Generally I'm dealing with the intersection of two or more of these.
I do think, though, that the last paragraph is a bit unfair. Literally everyone I work with is incredibly smart; just their background isn't the same as mine. Things that I find incredibly tough are idiotic to them ["Can't I just take the average and be done with it?" "no"].
So, for most of the people I work with, it's not "common sense"; they literally lack the frame of reference to even know what's awry. I never hold it against anyone, and nor should you.
My favorite example is from many years ago, a guy who had some code that took a month, "because that's just complicated math, it's how long that takes". I ran it through a profiler, and after about ten minutes work, it ran in seconds. I came out looking like a hero.
Terabyte of RAM is my favorite because I choose to believe he had a perfect data model and was literally telling you exactly what he needed :D
In my recent work doing financial modeling (options, cryptos, other nonsense), I pretty much just try to cache absolutely everything with some arbitrary 1 week TTL and then continue to cache new data every day. Cache hits cause a 10% bump in the current TTL time delta. Kinda heuristic but whatever, works in most cases.
Any other funny stories? I always love these engineering horror stories. Have you looked at Julia at all? Would be interesting to hear if there's been an uptick in usage from the research community from a veteran research code surgeon!
The mathematician that designed the model I referred to at the end ended up passing it on to me. I've now built something of an empire of people and projects using it, "just" by making better tools to manage its inputs, outputs, visualisation and automation.
I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do.
Not often I can empirically show traditional CS101 stuff in a real-world chart.
No, that's the point; the previous algorithm was O(1) [a simple queue] but that led to doing a huge amount of extra work. By using a smarter queue that "cost" an extra O(log(n)), we could skip work that didn't need doing in the first place.
Oh, and not to tar with toooo broad a brush: One guy contacted me the other day wanting a terabyte of memory. He'd actually done all his homework and had a weirdly interesting use case.
I'm actually pretty excited to explore getting his problem solved; it's likely that a terabyte is about the right amount to get him over the finish line, and he's converged on this algorithm precisely because it has fairly horrifying memory needs, but trades off to give a nice big-O runtime.
I've heard that some financial institutions use a YARC supercomputer with 256TB of RAM to do daily "full market watch" of client activity to determine insider trading for compliance purposes.
I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?
Finance guys tend to be able to afford to have the right people to be sure of that on hand; they also are specifically in the business of cost-benefit analysis.
I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
> I can't even imagine the power draw of such a system or the heat generation.
I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations.
Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour.
I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB):
If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.
If you're allowed to call a multi-node setup a "supercomputer", then how about 3 40 machine racks of Xeons where each one is dual-socketed, has 12 DIMMs per socket, and is using 128 GB DIMMs?
3 racks * 40 boxes * 2 sockets * 12 DIMMs per socket * 128 GB = 368640 GB or 360 TB raw. Remove some of that for overhead (kernel, OS gunk, network buffers, whatever code you'd have to run to make it into a cluster), and there's your 256 TB with a bunch left over.
Stuff like this is why I sometimes say that a lot of these startups could be done with "three Xeon racks" if they actually cared about doing it right.
[Note: assumes 2nd generation Xeons (Skylakes), with 24 DIMMs. Ice Lake = more/bigger memory, fewer actual machines.]
- For Java, personally I like the netbeans profiler
- Octave's built in profiler gets the job done: https://octave.org/doc/v6.2.0/Profiling.html
- I use JetBrain's tooling a lot.
- I like the profiler in Rider for VB.NET and C#
- The one in PyCharm is ... fine. Python profiling is ugly, PyCharm does a clean job handling it.
- For R, Rprof is ... fine: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/Rprof.html
- STATA's built in profiler does as good a job as possible given it's a weird language
For various other things, where a profiler doesn't really exist per se:
- SAS can be configured to output runtime after each proc. There's no cost, you should do it
- SAS proc SQL has an undocumented _method flag: http://support.sas.com/techsup/technote/ts553.html
- Past "EXPLAIN QUERY PLAN", one of the tools I build ends up doing a simple visualisation of SQLite's explain query plan output for *every* query it runs. There's no way to *not* get that estimate
Those are the tools I can think of that I've used in the last few months.
I appreciate you taking the time to write this out.
I'm curious about the lack of any mention of C/C++ tooling - I thought there are some mathematicians who write stuff in C/C++ and leverage libs like GNU SL (Scientific Library) etc. I've written a few custom pricers and needed to use those libs and they were pretty efficient, and didn't need to resort to anything crazy to run them in production.
Curious if you ever find stuff written in those languages, as opposed to the gamut you just mention, which are classical "research tools" (language/IDE combinations that let you not need to think about the language specifically.
I don't cross paths with a lot of C or C++ where profiling is a concern. I use gprof for that, but I haven't needed it in a few years. None of the folks I work with reach for C or C++ as a weapon in their quiver.
I do cross paths with a fair bit of FORTRAN, but in my experience, generally if someone's writing FORTRAN, they already know the deal with performance.
Fun fact: When I interviewed for my job [12 years ago], I interviewed with a guy using SQLite and FLTK in a C++ model. I'd been using SQLite and FLTK for some recent personal projects; in the interview, I solved a bunch of performance problems he'd been having for a couple years. I interviewed on Monday, and started working there two days later.
This place has been an astonishingly good fit for me, yeah. I was laid off from my last job, and a friend passed along my resume at an opportune moment. That was twelve years ago, and I'm still here.
I share the same feeling. I am a PhD student with working knowledge of application development. My skills have been quite useful to the rest of the lab as they transitioned to using HPC systems.
> * "I need a terabyte of RAM"
A lot of our work involve running algorithms such as PCA and ICA on thousands of biomedical imaging data. It is totally normal for us to allocation 1TB RAM. We have 3 nodes with 1.5TB RAM in our cluster. Aapproximation algorithms do exist and are used for resource optimization though.
To be clear, sometimes a terabyte of RAM is the right solution. But it's pretty common to find someone needing that simply needs a better algorithm. [or, recently, a mathematical identity that meant the task could be done streamed from disk]
Tbh if you need a terrabyte of RAM to simply run your experiment then you aren’t doing it right. We haven’t reached a level where an algorithm would require at least a terrabyte of RAM just to get baseline results.
You know what would be fun? A website like Project Euler, but you're given programs that are less than optimally efficient and your job is to make them as fast as possible.
Took a secure programming course in college that was basically just that for each assignment except the provided code wasn’t just inefficient but broken in interesting ways that prevented you from googling an error message. Incredibly fun course!
0.1 seconds is still too many. There is probably another 1000x speed up possible there. We just have to do a merge on two rather small arrays. The most expensive operation here would actually be writing out to the output array.
I remember one time, there was code that took 3 hours to run. Then printing the output took another 3 hours. A huge problem was that it printed out a stack of output about 3 feet high, and if the printout jammed, as was often the case, you had to start all over, even if it was 10 pages short of completion of the print job. Another 6 hours. Another printing error, yet another 6 hours. So, as you probably can see, it could take 18 hours or more to get a correct printout.
I took one look at the code, and changed it. It then took about literally 5 seconds to run the code. Then I added some code that would allow the entry of a page number, and the users could enter that page number and print from that page. So, it would then take always a maximum of 3 hours to print out the output. Then I told the night operator to run this program for the operations department at midnight. So, as far as the operations department was concerned, that brought the time down to zero, and the printouts were on everyone's desk when they walked in the door in the morning.
The manager of that specific department literally came up to me after it was rolled out and was crying hard, and hugged me for about a minute, crying the whole time, as her job was on the line, no fault of her own, because she couldn't get the job done, because she couldn't get the printouts.
There was no way that the most powerful computer in the world that would have got that code to run any faster, because of the horrible way it was written. Going from maybe 18 hours to zero time, on the same exact computer....what is that improvement in percentage anyways? What's 18 hours/0 hours? haha...
Interesting. My first thought was to print to a PDF, then print the PDF, since you can restart it from whatever page you want, and you don't have to restart it.
You solve the actual problem, but this would have been quicker.
36 comments
[ 4.3 ms ] story [ 89.5 ms ] threadYou can do that with an actor model by giving each actor thread affinity, but it's rarely what you want since it's normally a lot more efficient to move the code to the data than move the data to the code. In a factory the machines are heavier than the products they're working on so it makes sense to move the machines to the materials; for most code tasks it's much more like harvesting a field or something where it's much more efficient to move the code to the data.
https://www.usenix.org/system/files/conference/hotos15/hotos...
http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
I do think, though, that the last paragraph is a bit unfair. Literally everyone I work with is incredibly smart; just their background isn't the same as mine. Things that I find incredibly tough are idiotic to them ["Can't I just take the average and be done with it?" "no"].
So, for most of the people I work with, it's not "common sense"; they literally lack the frame of reference to even know what's awry. I never hold it against anyone, and nor should you.
My favorite example is from many years ago, a guy who had some code that took a month, "because that's just complicated math, it's how long that takes". I ran it through a profiler, and after about ten minutes work, it ran in seconds. I came out looking like a hero.
In my recent work doing financial modeling (options, cryptos, other nonsense), I pretty much just try to cache absolutely everything with some arbitrary 1 week TTL and then continue to cache new data every day. Cache hits cause a 10% bump in the current TTL time delta. Kinda heuristic but whatever, works in most cases.
Any other funny stories? I always love these engineering horror stories. Have you looked at Julia at all? Would be interesting to hear if there's been an uptick in usage from the research community from a veteran research code surgeon!
I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do.
Not often I can empirically show traditional CS101 stuff in a real-world chart.
(And you probably mean O(n) and not O(1), otherwise O(log(n)) isn't really an improvement).
I'm actually pretty excited to explore getting his problem solved; it's likely that a terabyte is about the right amount to get him over the finish line, and he's converged on this algorithm precisely because it has fairly horrifying memory needs, but trades off to give a nice big-O runtime.
I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?
I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations.
Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour.
I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB):
https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39...
If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.
3 racks * 40 boxes * 2 sockets * 12 DIMMs per socket * 128 GB = 368640 GB or 360 TB raw. Remove some of that for overhead (kernel, OS gunk, network buffers, whatever code you'd have to run to make it into a cluster), and there's your 256 TB with a bunch left over.
Stuff like this is why I sometimes say that a lot of these startups could be done with "three Xeon racks" if they actually cared about doing it right.
[Note: assumes 2nd generation Xeons (Skylakes), with 24 DIMMs. Ice Lake = more/bigger memory, fewer actual machines.]
Curious if you ever find stuff written in those languages, as opposed to the gamut you just mention, which are classical "research tools" (language/IDE combinations that let you not need to think about the language specifically.
I do cross paths with a fair bit of FORTRAN, but in my experience, generally if someone's writing FORTRAN, they already know the deal with performance.
Fun fact: When I interviewed for my job [12 years ago], I interviewed with a guy using SQLite and FLTK in a C++ model. I'd been using SQLite and FLTK for some recent personal projects; in the interview, I solved a bunch of performance problems he'd been having for a couple years. I interviewed on Monday, and started working there two days later.
My group has a perpetually open posting and are always looking for people: https://rand.wd5.myworkdayjobs.com/en-US/External_Career_Sit...
[If the link doesn't work, go to rand.org, search jobs, then search for "research software engineer"]
EDIT: That link is to a specific posting. Our perpetual one is going back up any day now, once we get through the machinations.
> * "I need a terabyte of RAM"
A lot of our work involve running algorithms such as PCA and ICA on thousands of biomedical imaging data. It is totally normal for us to allocation 1TB RAM. We have 3 nodes with 1.5TB RAM in our cluster. Aapproximation algorithms do exist and are used for resource optimization though.
http://james.hiebert.name/blog/work/2015/09/14/CS-FTW.html
I took one look at the code, and changed it. It then took about literally 5 seconds to run the code. Then I added some code that would allow the entry of a page number, and the users could enter that page number and print from that page. So, it would then take always a maximum of 3 hours to print out the output. Then I told the night operator to run this program for the operations department at midnight. So, as far as the operations department was concerned, that brought the time down to zero, and the printouts were on everyone's desk when they walked in the door in the morning.
The manager of that specific department literally came up to me after it was rolled out and was crying hard, and hugged me for about a minute, crying the whole time, as her job was on the line, no fault of her own, because she couldn't get the job done, because she couldn't get the printouts.
There was no way that the most powerful computer in the world that would have got that code to run any faster, because of the horrible way it was written. Going from maybe 18 hours to zero time, on the same exact computer....what is that improvement in percentage anyways? What's 18 hours/0 hours? haha...
You solve the actual problem, but this would have been quicker.