Ask HN: Can use the #1 supercomputer for any project I want. What should it be?
The professor recommended choosing something that relates to published research in physics or our own research field (mine is chemical engineering -> molecular dynamics). This freedom to choose whatever is really exciting, and I've got some interesting ideas, but I imagine there a lot of experts in their field who post on HN, and so someone may have a good idea for a new and exciting project.
My initial idea is to contribute to the effort of porting Quantum Monte Carlo code to GPUs. Titan's processing power is unique among supercomputers in that most of it comes entirely from nVidia Tesla K20X GPUs. QMC is among the most accurate methods that exist in predicting physical phenomena; the problem is that the methods are incredibly computationally demanding, something which highly parallelized GPUs are well-suited to handling. But I don't know. Maybe that's too much to do in a semester.
95 comments
[ 4.9 ms ] story [ 155 ms ] threadSorry if it's not as sexy as you thought, but I think this could make a pretty big impact in the security community.
(I also had the evil idea of seeing how many bitcoin I could generate on Titan, but I don't think the ORNL staff would even find that funny.)
http://freerainbowtables.com/
ATI cards have traditionally had better hashing (in general, rainbow table gen, jtr, BTC) because they have a larger number of Execution Units per core, however clocked slower, than the nvidia. Higher number of EU allows better exploitation of parallelism important for the performance of hashing.
This is the fastest damn computer, its not a brand loyalty GM vs Ford, Coke vs Pepsi, Android vs iOS duality. Oh it's got nvidia, not optimized for hashing. It's going to kick the arse of any consumer or professional grade GPU on the market...
Each block generate 25 BTC reward. At ~14.00 USD/BTC on MtGox right now that about $34k.
So the maximum reward in USD if you could totally control the block chain per day is $34,000. Compare hashing rate of your super computer to total mining hash rate of the btc mining swarm and that is your fraction of 34,000.
The greater the hashrate of your miners, the greater probability that your miners will find a valid nonce that satisfies the difficulty equation; it doesn't magically give you the ability to find a valid nonce before any other party. As long as there is one other party mining, there is a non-zero chance that their block will be accepted before yours.
What ratio of malicious/non-useful blocks vs valid, legitimate BTC transaction containing blocks would be required before the economy fails, that is an open question.
At best you can drop all transactions that are broadcast to the network and generate blank blocks, thus freezing all future transactions.
Anyway, Titan is not powerful enough to perform a majority attack. First of all, there is no Bitcoin miner optimized for the latest GK104/GK110 Tesla and GeForce GPUs. With current miners, people report speeds of barely 110 Mhash/s on the GeForce GTX 680, which should translate to ~140 Mhash/s on the Tesla K20X (which is only 30% faster in terms of 32-bit integer ops per second.) Titan has 18688 K20X, so that's only 2.6 Thash/s total. Far from the 20 Thash/s required. It would still bring 400 coins/day, or $5600/day :)
I theorize here [1] that based on the number of ALUs and clock frequency, that a properly optimized miner should be in theory 4x-6x faster than this on GK104/GK110, bringing Titan to 10-16 Thash/s. (Nobody bothered optimizing Nvidia because everybody is mining on more efficient Radeon GPUs or FPGAs.) Still not enough to match 20 Thash/s. Disclaimer: I have only programmed AMD GPUs, never Nvidia. Maybe there is a reason this 4x-6x theoretical perf gain has never been realized, such as the inability to execute 1 32-bit integer instruction per ALU per clock due to instruction latencies greater than 1 clock, or throughput being less than 1 instruction per clock... I don't know. Theoretical FLOPS numbers published by Nvidia indicate that floating point instructions, as opposed to integer instns, can run at 1 instruction per ALU per clock. Which makes it even weirder that there would be such a huge perf gap between int and fp.
[1] https://bitcointalk.org/index.php?topic=129292.msg1381510#ms...
Quote, Wikipedia: "Titan has 18,688 nodes (4 nodes per blade, 24 blades per cabinet), each containing a 16-core AMD Opteron 6274 CPU with 32 GB of DDR3 ECC memory and an Nvidia Tesla K20X GPU with 6 GB GDDR5 ECC memory."
So it's pretty much standard hardware just a lot of it. Except that the graphics card is basically a consumer tesla with some extra juice, but it's still just a card built with what's possible today. Nothing extra fancy.
Real-time zooming on Mandelbrot was already done 20 years ago.
Sure it was using a "trick" in that at each frame only a few vertical and horizontal lines were recomputed, and the others simply re-used from the previous frame but still... (and at every frame each line was guaranteed to be "at most" x frames old).
It looked really nice and it was cool to see a real-time zoom on a Mandelbrot... In the nineties!
It was going for quite a while too...
This was my 1995 final year Physics project with inadequate hardware and poor language options.
Of course back then the assumptions were considered a bit radical but now much more accepted as reflecting something approaching reality.
I think it would be an interesting project to do it again with modern tools and some real computing horsepower.
Edit: Another one - model the human circulatory system - trying to find optimal postures for maximum heart efficiency. Also simulate blood-flow keeping track of anti-coagulant factor concentration to detect high-risk areas for blood-clot formation.
http://en.wikipedia.org/wiki/Computational_fluid_dynamics http://en.wikipedia.org/wiki/99942_Apophis http://en.wikipedia.org/wiki/Goldstone_Deep_Space_Communicat...
If we knew their final protein structure, it would be much easier to work out what biological processes they may be involved in.
Also, taking known protein structures and learning more about how they may interact with other proteins would be extremely useful. For example, a large number of molecular pathways in cancer are fairly well described in a general sense, but lack specific information on the actual reactions.
Finally, being able to put it all together in the context of mutated genes (which we can now screen for fairly easily) and being able to determine what impact a mutation may have on the protein (can it still be formed, how will it interact in a pathway?). For example, some mutations break a protein completely, while others may impact "switch" genes, leaving them in a permanently "on" state preventing them for reacting to external signals [2].
This kind of thing is really important in cancer research (and one day treatment), where the function of specific proteins may be the difference between life and death.
[1] http://www.genecards.org/cgi-bin/carddisp.pl?gene=C1orf186 [2] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1891745/
For some proteins with post-transcriptional (ex. alternative splicing) or post-translational modifications (ex. attachment of functional groups, formation of disulfide bridges, proteolytic cleavage) it is difficult or impossible to arrive at the correct folded structures from computation on nucleic acid sequences alone. For such proteins, the structures can be determined experimentally. That means coaxing pure samples of protein to form single crystals and then subjecting the crystals to X-ray diffraction (XRD). Backing out the structural information from the XRD patterns takes serious horsepower (though admittedly not necessarily power with Titan-class interconnects). XRD does not give information about the location of hydrogen atoms in the structure though. For that information you need to do neutron diffraction (the de Broglie wavelength of the neutrons is short). ORNL is actually one of the few places that do neutron diffraction. The OP could connect with some of the neutron folks there and assist them with their research. If not for proteins, maybe for superconductors.
I had a chance to run some Monte Carlo code on some of the machines at ORNL as part of a supercomputing workshop there during the reign of Jaguar. It was an amazing experience. Enjoy it, OP! I hope you get to take a trip there and see all the racks too.
The IUPAC InChI keys are a nomenclature to turn chemical structures, like morphine, into a unique linear representation, like InChI=1S/C17H19NO3/c1-18-7-6-17-10-3-5-13(20)16(17)21-15-12(19)4-2-9(14(15)17)8-11(10)18/h2-5,10-11,13,16,19-20H,6-8H2,1H3/t10-,11+,13-,16-,17-/m0/s1 .
This string is too long, so there is a way to use SHA-256 to convert the above into a hashed string BQJCRHHNABKAKU-KBQPJGBKSA-N, where the first 14 characters are the basic topology, and the successive 9 characters contain the other information.
Some people believe that this can be used to query the web for "secret" information. That is, you work at a pharma and want to know if others know about compound X. If they don't know about X then you don't want to tell the about it. Otherwise it reveals information about new compounds that you are working on.
You instead search for hash(X). If others have hash(X) then you're revealing less information, since this is likely a publicly known structure. If others don't have hash(X) than you might conclude that you have a proprietary structure and haven't revealed enough information for others to know what you are working on.
I don't believe that the hash key is appropriate for this. I think it is open to a brute force attack.
While 26 * * 14 is very large, most people work inside "chemical space" of reasonable drug-like compounds. The Reymond group has enumerated, in GDB-13, all drug-like containing up to 13 atoms of C, N, O, S, and Cl, which is 977 million compounds. If you convert these to InChI hashes, then you might be able to guess the core scaffold given an unknown InChI hash.
Once you have the core structure, you might then be able to brute force the bond and hydrogen assignment.
This can be tested by taking GDB-13 and finding the InChI hashes. You may need to enumerate over a range of hydrogens for each one, giving some several billion keys. Then take, say, the ChEMBL data set (for N<=13 heavy atoms) as the source for the "secret" keys, and generate their hashes. Are there matches? What percentage of topologies can be found this way?
If you can find the topologies, can you then as phase 2 deduce the overall structure, and if so, what percentage?
As a quick estimate, phase one (enumerating GDB-13 InChI keys) would take over 10 years on my desktop, and 5 TB of disk. That's about perfect for a supercomputer job, and you can select subsets as a appropriate based on the available time. (Eg, pick only the C, N, O, S subset of GDB-13 and ChEMBL and you've reduced your space by a lot.)
I don't know what's needed for phase 2 and can provide no estimate.
this is an embarassingly parallel problem. The supercomputer hardware (interconnect) would be simply wasted. If you provide C++ source code that can be compiled with Native Client, and a list of tasks to run (binary command lines), Exacycle (http://googleresearch.blogspot.com/2012/12/millions-of-core-...) is more appropriate. Based on our publicized results this wouldn't take very long (10 years on a desktop = less than an hour on Exacycle; you'd spend more time writing the program and the data analysis than the actual runtime).
however, I don't really see this problem as a high priority. pharma has bigger problems than to probe remote websites to find out what their competitors might be interested in.
Not insinuating anything, but a NaCl-compiled distributed task would be an excellent fit for farming out to unsuspecting browser clients. (Reminds me about that sneaky javascript-based bitcoin miner that was injected on a few web sites some time ago)
We've talked with Vijay Pande (of Folding@Home fame) about making "Folding@Chrome", but that would be opt-in (IE, you'd install a Chrome App).
If you have an alternative sandbox and security framework, I'd be happy to hear about it.
So you are saying it is a better solution than any combination of Java, Ruby, SELinux, any other regular OS ACL, Xen, or any other virtualization platform?
BTW, I definitely think seccomp http://en.wikipedia.org/wiki/Seccomp and linux containers http://lxc.sourceforge.net/ or perhaps OpenVZ: http://wiki.openvz.org/Main_Page
could be engineered into a secure sandbox, without having to resort to virtualization (which I'd like to emphasize isn't really a security solution). This is called "container virtualization" (as compared to hardware/CPU-based), and has a number of very nice properties.
There are easier ways to make it more complex. For example, enumerate the first 15 atoms instead of the first 13. I believe this requires coordination between the enumeration nodes in order to reduce duplicate generation, but I don't know that subfield and labeled graph enumeration gets further away from hands-on chemical work. And stage 2 requires a combinatorial approach taking maybe 1,000x more time than the first stage. (That's a jazz hands level of waviness - I really don't know.)
I once did work in molecular dynamics. In fact, I was one of the co-authors of NAMD. That's a rather intently explored field, and progress in it can feel rather incremental. What I proposed is one that no one has explored, to my knowledge. There's maybe 50 people in the world who would be able to do this work now, if they had time and hardware access (which they don't), and perhaps ~5,000 people who would be affected by knowing the result; mostly to know if they could do certain searches on the public web.
So it's one with little competition and where the results would be immediately publishable. Not bad for a semester project, I think! I think it would be a good Master's thesis. And a lot different from the usual set of MD, docking/screening, folding projects that have consumed massive amounts of supercomputer time for the last three decades.
Given that Xcelerate was talking about porting some code to the GPU, I suspect that that person already expect to "spend more time writing the program and the data analysis than the actual runtime." When I was doing MD work, I expected to spend about 6 months in simulation time and about 2 years in analysis for a PhD. The previous generation to me in the group had to build their own hardware before doing simulations, so I though that it's usually the case that the non-simulation time is longer than the simulation time. :)
You said: "I don't really see this problem as a high priority. pharma has bigger problems than to probe remote websites to find out what their competitors might be interested in."
So they should only work on the big problems that everyone else is doing? What about smaller problems which might help solve big problems?
For example, 10 hours ago here on HN, mtgx posted a link titled "Patents Are Making Us Lose The Race Against Antibiotic-Resistant Bacteria" (to http://www.techdirt.com/articles/20130110/09590621628/world-... ), which reports that the "World Economic Forum's 8th Global Risks Report" suggests that "Rather than today's monopolistic hoarding [of data], what we need is more sharing of [pharmaceutical] knowledge."
Let us suppose that they are correct, and we need to have more sharing of some knowledge between different companies, perhaps with "public or philanthropic funding to incentivize academic collaboration."
One of the questions you could ask, if you had more information about what goes on inside of the companies, is: what's the overlap of the chemical space being tested by the different companies? Are they too similar? A recent paper on the topic - and one of the few such papers - is the recent "Big pharma screening collections: more of the same or unique libraries? The AstraZeneca–Bayer Pharma AG case". (See http://pipeline.corante.com/archives/2012/12/06/four_million... for a summary.)
In it, they conclude that there is a "low overlap between both collections in terms of compound identity and similarity."
They did it by using 2D ECFP4 fingerprints. A fingerprint is very much like a Bloom f...
With regards to sharing scientific information related to pharma, I (and the rest of the Exacycle team) believe strongly that the results the Exacycle Visiting Faculty achieve must be made public- ideally all the raw data in addition to the published analysis. If you look at the blog post I referenced, you'll notice that three of the faculty are working on protein/drug related problems, and we'll be sharing the bulk of the results with the scientific community.
The work we've done has moved protein folding and drug design far beyond incrementalism. The scale we make available also turns things that were 1-10 year projects and makes them an overnight job. I also spent a year running MD simulations on a supercomputer, so I can appreciate the higher level of scientific productivity that a system like Exacycle provides.
Some of the questions you raise about cloaked IP sharing are interesting, mainly from a theoretical perspective, but I think pharma would be more likely to use existing IP protection mechanisms, or they would simply not collaborate at all, before adopting stuff as "complicated" as this.
Beyond that though, I don't know where this conversation should go. I could be cynical and point out that the proposals you all accepted, while worthy, look similar to the proposals from 20 years ago but with "SP-2" or "cluster" scratched out and replaced with "Exacycle." Even some of the names (Russ Altman, GROMOS/GROMACS) have little changed in the last 15 years.
I could observe, as you did in passing, that this project does not turn "1-10 year projects" into "an overnight job" because writing the software and understanding the results dominates the overall time - otherwise they would be done, no? And realistically, few people embark on 10 year projects; they estimate 10 years then work on another project which is more tractable, with the hope that in 10 years it will become a 2 year project.
I could go the Greg Wilson route, and argue that the "Real Bottleneck in Scientific Computing" is not the CPU cycles but the need for better training in computational scientists in how to write and organize software, and a cultural change in how journals treat software development and testing. Yet supercomputers gets a large amount of funding compared to training.
Or I could quizzically wonder why it is that "interesting", "theoretical" work isn't exactly the sort of work that researchers should be doing. For example, "InChIKey collision resistance: an experimental testing", doi:10.1186/1758-2946-4-39 shows that related work is easily publishable.
And I further wonder how my conversations with people at pharmas, about how they might use the InChIKey, is "complicated." It was someone at AstraZeneca who told me they were evaluating the use of InChI keys in order to link internal databases to external web search engines and data sets without triggering the usual reactions against revealing internal structure information. They thought to generate the hash for each compound, make a hyperlink to the external site using that hash, and they are done. That seemed reasonable to me then. Over time I've wondered just how safe it actually is. Can pharmas actually use InChI keys to do things that they can't do with "existing IP protection mechanisms"? Are they leaking data already, without realizing it? If so, what is the level of leakage?
But realistically, or perhaps back to cynically, justifications like "could cure cancer" or the newer "could help identify new antibiotics", or like "help understand disease" or "reveal insights into fundamental biochemistry" are ever so much easier to justify funding than the mundane "train computational chemists in how to program" or "see if there are new ways to exchange proprietary data between distrusting organizations".
What, me bitter? Perhaps slightly, but I've decided that there's much more interesting diversity in the sorts of problems that one person or a small group can work on, without having to justify its practicality or how it fits into some long-term Grand Challenge vision. It just means that there are some interesting projects which need high-end compute power that I just can't work on. Yet.
Generating the hash is, as you say, trivial given the InChI string. Computing the InChI string in the first place is not. It requires molecular perception of the input graph. For example, which of the hydrogen positions are mobile (eg, can be identical under tautomerization) and which are fixed? It also requires generating the canonical graph representation, which is done via a variation of the nauty algorithm. Usually it's quite fast. Some structures can take longer. By default there's a 60 second timeout.
I haven't worked with the InChI code for a while, so I don't know what the performance numbers are. An output log from 2009 shows "Finished processing 2186 structures: 0 errors, processing time 0:00:05.93." or 369/second = 2.7ms. Perhaps its under 1.0 ms now?
You estimated 323 milliseconds/record based on the numbers I gave. However, I did not describe the step which turns 977 million compounds into the actual number to evaluate. I would also include the combinations replacing Cl with F and with Br, and I would replace at most two of the nitrogens with a phosphorous. I may also have to try different hydrogen counts, depending on the types of structures in GDB-13. I guessed that this would require about 100x as much compute time as the base 977 million structures.
This is all meant as very rough estimates. It's only been a background thought until now, and I hadn't written it up before. In any case, the parameters are all quite tunable, in that I could crank up or down the problem size to match the available CPU time and researcher patience.
Our research group was determined to run the largest hydrodynamic cosmological simulation ever done, using a new code. Over a year ago the code was "basically ready" and had been used on "small" runs of ~hundreds of k CPU hours. They still haven't completed a big run...
In general, if you scale by more than one or two orders of magnitude then you're going to run into new, unexpected problems.
In one project I worked on, the code was fast on all of the test sets, but when scaled up to the entire data set it was unexpectedly slow. We traced that down to a logger which we hadn't realized was enabled. In your code you might find that a minor piece of code used order n-squared time but because of data partitioning, it has minimal impact.
Or you might find that you have a barrier, where everything is supposed to be synchronized. (Eg, once every 1,000 time steps you rebalance the system load.) A few stragglers might not affect small system sizes, but cause 50% slowdowns with large numbers of CPUs. You might even have to rewrite the code to remove the barrier and come up with another way to do dynamic load balancing.
Bram Cohen, in his Stanford EE380/2005 talk on BitTorrent (linked from http://bramcohen.livejournal.com/11025.html?nojs=1 , at 8:45) mentioned how TCP's checksum isn't good enough once you start swapping terabytes of data over the Internet, so BitTorrent has to work around that.
On the topic of flakiness, Google's own Big Table/MapReduce, etc. is designed to be able to handle machine faults, which can happen with large numbers of cores. How well does your code handle those sorts of rare failures?
In another project I worked on, there was a bug in the code. It didn't conserve energy as well as we thought it should. That was eventually tracked down to a trigonometry error where we didn't compute one of the energy terms correctly when the angle was exactly 0.0. This doesn't happen that frequently, which is why we only saw it with large simulations run for a long time.
In our case, the most annoying problem we had was that the entire run would sometimes just hang. I'm not aware that the problem was tracked down, but the author suspected some race condition in the MPI library that caused some message to get lost and thus some task to wait indefinitely.
Normally the most difficult part is adapting your code (if exists) to the supercomputer anyway.
What about cross-referencing non-coding parts of all existing genomes? With RNA secondary structures? I bet there would be a big benefit, if only finding a lots of non-coding RNA that does something, since it is known that this part of genomes does most of the job.
Obligatory xkcd: http://xkcd.com/683/
Quantitative Chemical Analysis (Daniel C. Harris) Principles of Instrumental Analysis (Skoog, Holler, Nieman)
The following one, which you should read first because it's free, is also good:
http://www.asdlib.org/onlineArticles/ecourseware/Analytical%...
Just ask questions if you're interested.
The reality is that a broad range of techniques (e.g. GC-MS) could reliably identify the chemical composition of products if competitors really wanted to know, and creating a recipe from the chemical composition isn't that hard (the chemical composition of many flavouring herbs and spices used commercially are widely available), and assuming no chemical transformations during processing, solving for the levels of ingredients from the levels of plants is simple linear algebra (solve Transpose(A)Y=Tranpose(A)AX for X, Y is a column vector giving the concentrations of the dominant species, A is a matrix giving the concentrations of each compound in each plant, and X is the unknown column vector giving the amount of each plant to use). Assuming thousands of species and ingredient possibilities, this could realistically be solved in less than a second on my home computer - no need for the supercomputer.
The real reason that big companies like Pepsi don't make an exact copy of Coca-Cola is not that they don't know what is in Coca-Cola, it is that it would be bad business - it is far better to make a product with its own distinctive taste for customers to like that is nevertheless close enough to be a substitute.
Once you have your recipe you can go to a company like Cott who will manufacture it for you: http://en.wikipedia.org/wiki/Cott and then you have the problem of selling it. In many stores it is actually Coca Cola employees who deliver the product and put it up on the shelves. All the store does is ring the drinks up at the till. And the supplier often has to purchase the shelf space http://en.wikipedia.org/wiki/Slotting_fee
There are "open source" cola recipes: http://en.wikipedia.org/wiki/Open_source_cola http://en.wikipedia.org/wiki/OpenCola_(drink)