I continue to be impressed by how quickly DeepMind has managed to progress in such a short time.
CASP13 was a shocker to all of us I think, but many were skeptical as to the longevity of the performance DeepMind was able to achieve. I believe with CASP14 rankings now released, it's safe to say that they've proven themselves.
Congratulations to the team! This work will have far reaching impacts, and I hope that you continue to invest heavily in this area of research.
Which part of genetics are you thinking of? Much of genetics isn’t amenable to this kind of ML, because it isn’t some kind of optimisation problem. And many other parts don’t require ML because they can be modelled very closely using exact methods. ML does get used here, and sometimes to great effect (e.g. DeepVariant, which often outperforms other methods, but not by much — not because DeepVariant isn’t good, but rather because we have very efficient approximations to the exact solution).
Genetics is amenable because the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network.
There are plenty of inferences that you would want to do on genetic sequences that we can't model exactly and there is some past work on doing stuff like this, although biology is usually a few years behind.
I cannot give constructive feedback to something which is incomprehensible.
"the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network"
The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important.
Language modeling in the context of machine learning typically means NLP methods. Genetics is nothing like natural language.
Auto-regression is using (typically time series) information to predict the next codon. This makes very little sense in the context of genetics since, again, the genetic code is not an information carrying medium in the same sense as human language. Being able to predict the next codon tells you zilch in terms of useable information.
"Depth of understanding by the network" ... what does that even mean???
The above sentence is a bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way. AKA word salad.
> The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important.
aka a sequence. "a book is not a sequence so much as a discrete set of chapters which are themselves sequences of paragraphs which are themselves sequences of sentences" -> still a sequence
these techniques are already being used, such as in the paper I just linked.
> Being able to predict the next codon tells you zilch in terms of useable information.
You have absolutely no way of knowing that apriori. And autogressive tasks can be more sophisticated than just next codon.
> bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way
Okay, feel free to think that.
There's always this assumption of it "will never work on my field." I've done work on NLP and on proteins and read others' work on genetics. I think you will end up being surprised, although it might take a few years.
It is incomprehensible to you, because you just simply do not understand what your parent is talking about. You are the ignorant one here and indeed quite rude. Doesn't matter that genetics is not natural language. The point is we can train large transformers auto regressively and the representation it learns turns out to be useful for a) all kinds of supervised downstream tasks with minimal fine-tuning and b) interpreting the data by analysing the attention weights. There is a huge amount of literature on this topic and what your parent says is quite sensible.
That statement you quote is completely understandable.
Let's say you have discrete sequences that are a product of a particular distribution.
Unsupervised methods are able, by just reading these sequences, to construct a compact representation of that distribution.
The model has managed to untangle the sequences into a compact representation (weights in a neural network) that allows you to use it for other, higher level supervised tasks.
For example, the transformer model in NLP allowed us to not have to do part-of-speech tagging, dependency parsing, named entity recognition or entity relationship extraction for a successful language-pair translation system. The compact transformer model managed to remap the sequences into a representation that allows direct translation (people have inspected these models and figured out the internal workings of it and realized it does have latent information about a parse tree of a sentence or part-of-speech of a word).
Another interesting note is that designers of the transformer architecture did not incorporate any prior linguistic knowledge when they were designing it (meaning that the model is not designed to model language but just a discrete sequence).
Not sure what you mean by that. Genetics is a field of research. The genome is a sequence. And yes, that sequence can be modelled for various purposes but without a specific purpose there’s no point in doing so (and furthermore doing so without specific purpose is trivial — e.g. via markov chains or even simpler stochastic processes — but not informative).
> There are plenty of inferences that you would want to do on genetic sequences
I’m aware (I’m in the field). But, again, I was looking for specific examples where you’d expect ML to provide breakthroughs. Because so far, the reason why ML hasn’t provided many breakthroughs in less about the lack of research and more because it’s not as suitable here as for other hard questions. For instance, polygenic risk scores (arguably the current “hotness” in the general field of genetics) can already be calculated fairly precisely using GWAS, it just requires a ton of clinical data. GWAS arguably already uses ML but, more to the point, throwing more ML at the problem won’t lead to breakthroughs because the problem isn’t compute bound or vague, it’s purely limited by data availability.
I could imagine that ML can help improve spatial resolution of single-cell expression data (once again ML is already used here) but, again, I don’t think we’ll see improvements worthy of called breakthroughs, since we’re already fairly good.
I spoke loosely, my mind skipped ahead of my writing, and I didn't realize that we were parsing so closely. "Genetics (the field) is amenable because the object of its study (the genome) is a sequence" would have been more correct but I thought it was implied.
> without a specific purpose there’s no point in doing so
Well yes, prior to the success of transfer learning I could see why you would think that is the case, but if you've been following deep sequence research recently then you would know there are actually immense benefits to doing so because the embeddings learned can then be portably used on downstream tasks.
> it’s purely limited by data availability.
Yes, and transfer learning on models pre-trained on unsupervised sequence tasks provides a (so-far under-explored) path around labeled data availability problems.
I already linked to a paper showing a task that these sorts of approaches outperform, and that is without using the most recent techniques in sequence modeling.
Maybe read the paper in Nature that uses this exact LM technique to predict the effect of mutations before assuming that it doesn't work: https://sci-hub.do/10.1038/s41592-018-0138-4
I am not directly in the field, you are right - but I think you are also being overconfident if you think that these approaches are exactly the same as the HMM/markov chain approaches that came before.
Thanks for the paper, I’ll check it out; this isn’t my speciality so I’m definitely learning something. Just one minor clarification:
> Maybe read the paper … before assuming that it doesn't work
I don’t assume that. In fact, I know that using ML works on many problems in genetics. What I’m less convinced by is that we can expect a breakthrough due to ML any time soon, partly because conventional techniques (including ML) already have a handle on some current problems in genetics, and because there isn’t really a specific (or flashy) hard, algorithmic problem like there is in structural biology. Rather, there’s lots of stuff where I expect to see steady incremental improvement. In fact, in Wikipedia’s list of unsolved biological problems [1] there isn’t a single one that I’d characterise specifically as a question from the field of genetics (as a geneticist, that’s slightly depressing).
But my question was even more innocent than that: I’m not even that sceptical, I’m just not aware of anything and genuinely wanted an answer. And the paper you’ve posted might provide just that, so go and do my research now.
Not being in the field, I would term what I see in this story as a ‘bottom up’ approach to understanding genetics/molecular biology. More akin to applied sciences than medicine or health. This, for example, seems to be very important but it still leaves us with a jello jigsaw puzzle with 200 million pieces and probably far removed from immediate utility in health outcomes.
Then there’s the more clinically oriented approaches of looking at effects, trying to find associated genes/mutations whatever mechanisms exist in between to cause a desirable or undesirable outcome. I’d call that ‘top down’.
I’m sure the lines get blurred more every day, but is there a meaningful distinction into these and/or more categories that are working the problem from both ends? If so, are there associated terms of art for them?
FWIW, transformers is to sequences what convnets is to grids, modulo important considerations like kernel size and normalization. Think of transformers as really wide (N) and really short (1) convolutions. Both are instances of graphnets with a suitable neighbor function. Once normalization was cracked by transformers, all sort of interesting graphnets became possible, though it's possible that stacked k-dimensional convolutions are sufficient in practice.
I work in the field, I don't need the difference explained to me.
> Think of transformers as really wide (N) and really short (1) convolutions
Modern transformer networks are not "really short" and you're also conflating the difference between intra- and inter- attention.
There is still a pitched battle being waged between convnets and transformers for sequences, although it looks like transformers have the upper hand accuracy wise right now, convnets are competitive speed-wise.
> but many were skeptical as to the longevity of the performance DeepMind was able to achieve
For a non-biologist, on what is this skepticism based?
Just purely based on following ML news it looks like the trend for ML solutions has been that they've overtaken expert-systems once they've gained a solid foodhold in a field.
Maybe this is some perception bias. Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?
There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.
This is likely because linear regression meets most widely accepted definitions of machine learning. [0][1] It is simple and very effective when learning in linear space.
It's because for many researchers ML is just to take a standard keras or scikitlearn model shove their data in and get some table or number out, and see if that solves their problem. If that's your only ML experience then I suppose this is how sceptical you'd be of ML in general.
It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods.
That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest.
"It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods. That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest."
My PhD (now over a decade ago...yikes) was in applying much simpler ML methods to these kinds of problems (I started in protein folding, finished in protein / nucleic acid recognition, but my real interest was always protein design). Even back then, it was clear that ML methods had a lot more potential for structural biology (pun unintended) than for which they were being given credit. But it was hard to get interest from a research community that cared little about non-physical solutions. No matter how well you did, people would dismiss it as a "black box solution", and that pretty much limited your impact.
Some of this is understandable: even today, it's not at all clear that a custom-built ML model for protein folding is of much use to anyone -- particularly a model that doesn't consider all of the atoms in the protein. The traditional justification for research in this area is that if you could develop a sufficiently general model of protein physics, it would also allow you to do all sorts of other stuff that is much more interesting: rational protein design, drug binding, etc.
The alphafold model is not really useful for any of this, so in a way, it's kind of like the weinermobile of science: cool and impressive when done well ("hey! a giant hot dog on wheels!"), but not really useful outside of the niche for which it was designed. So it's hard to blame researchers in this field -- who generally have to chase funding and justify their existence -- from pursuing the application of deep learning to this one, narrow problem domain.
Obviously there will now be a wave of follow-on research, and it's impossible to know what methods this will spawn. Maybe this will revolutionize computational structural biology, maybe not. But I think it's a little unfair to demonize the entire field. Protein folding just traditionally hasn't been a very useful or interesting area, and like all "pure science", it leads to a lot of small-stakes, tribal thinking amongst the few players who can afford to compete. This is right out of Thomas Kuhn: a newcomer sweeps into a field, glances at the work of the past, then bashes it over the head, dismissively.
We don't know too much about the exact model they made but it looks sufficiently generalizable to be able to give a candidate protein structure for any given sequence. It doesn't automatically cure cancer and inject the drug but that by itself is an amazing tool that if available to everyone will revolutionize biology experimentation.
I will definitely blame the protein structure field in multiple levels though. It was always frustrating to me to open up Nature or Science and see it filled with papers about structure - like they are innovating so much that half of the top science magazines every week have papers in that field, yet it's not going anywhere? Or is it simply just a bunch of professors tooting their own horns about ostensible progress in a field that's archaic by decades if not years? The overall protein structure field internalised some dogmas in self defeating ways to everyone's detriment and finally events like this (and Cryo em, maybe) will jolt them out or make them fully irrelevant so we can move on. it's only doubly ironic that this came from a team in a company with minimal academic ties showing how toxic that entire system is. I only feel pity for the graduate students still trying to crystallize proteins in this day and age.
"We don't know too much about the exact model they made but it looks sufficiently generalizable to be able to give a candidate protein structure for any given sequence. It doesn't automatically cure cancer and inject the drug but that by itself is an amazing tool that if available to everyone will revolutionize biology experimentation."
They say on their own press-release page that side-chains are a future research problem, and nothing about their method description makes me believe they've innovated on all-atom modeling. This software seems able to generate good models of protein backbones; these kinds of models certainly have uses, but a backbone model is not enough for drug design.
This is certainly an advancement, but you're exaggerating the scope of the accomplishment.
" I only feel pity for the graduate students still trying to crystallize proteins in this day and age."
Nothing about this changes the fact that protein crystallography is a gold-standard method for determining a protein structure. CryoEM has made it possible to obtain good structures for classes of proteins we could never achieve before, and it's certainly interesting if we can run a computer for a few days to get a 1Å ab initio model for a protein sequence, but we could already do that for a large class of proteins with homology modeling. These predicted structures still aren't generally that useful for drug design, where tiny details of molecular interactions matter.
To put it in perspective: protein energetics are measured on the scale of tens of kcal / mol. Protein-drug interactions are measured in fractions of a kcal. A single hydrogen bond or cation-pi interaction or displaced water molecule can make the difference between a drug candidate and an abandoned lead. Tiny changes in backbone position make the difference between a good structure and a bad one. Alphafold isn't doing that kind of modeling.
Of course, they havent solved everything, but you seem to be doing exactly what I accuse that entire field (and academia in general) of doing - which is to insist a problem is intractable or hard and undermine someone potentially challenging that. When they released the 2018 results tbey field did embrace it (for sure I'd consider the groups organizing CASP as at least forward thinking) but was still skeptical on how much more progress it can make; now they blow everyone's minds again by a monumental leap and again people want to come say of course this is the last big jump!
I understand the self preservation instincts that kick in when there's a suggestion that the entire field has been in a dark age for a while, but I hope you can see that there might be something fundamentally wrong with how research is done in academia and that is to blame for why this didn't happen sooner, and why it's so hard for many to embrace it.
Regarding your comments on the inapplicability of this current solution for docking, I'm sure that's the next project they're taking up, and let's see where that goes.
This is exactly the same type of progression that happened with Go, where when their software bet a professional player everyone's like "yeah but I bet he wasn't that good". A few years later and Lee Sedol just decided to retire. I am interested to see what happens to that entire academic field in a similar vein, though my interests are more in knowing how science can advance from more people thinking this way.
> Nothing about this changes the fact that protein crystallography is a gold-standard method for determining a protein structure.
Yes it does. Protein crystallography is/was the gold-standard. Once this result is verified and accepted by the scientific community as a whole, that changes.
The reason for your second paragraph is pretty straightforward. There has been an immense amount of support for proteins as "the workhorses of the cell" for hundred+ years. I call it the "protein bias". We've seen in many times- for example when it was first hypothesized and then proved that DNA, rather than protein, is the heredity-encoding material, and seen many times, for example in the denial that RNA could act as an enzyme or the functional core of the ribosome could be a ribozyme.
I think what basically happened is a very influential group of scientists mainly in Cambridge around the 50s and 60s convinced everytbody that reductionist molecular biology would be able to crystallize proteins and "understand precisely how they function" by inspecting the structures carefully enough.
I learned, after reading all those breathless papers about individual structures and how they explain the function of protein is that in the vast majority of cases, they don't have enough data to speculate responsibility about the behavior of proteins and how they implement their functions.
There are definiteyl cases of where an elucidated structure immediately led to an improved understanding of function:
"It has not escaped our notice (12) that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material."
but most papers about how cytochrome "works" aren't really illuminating at all.
What's interesting is sometimes it boils down to the words effect or an affect in modern English. I think it's actually more interesting when you zoom out and look at bias caused to a lingua franca of standard modern discourse (it's probably counted in units relative to pi and Euler's consts on modern computers). If you switch individual words from roman (latin-1) to greek (I only was englightened to this by reading history book about the Byzantine empire, which the author had tried de-latinifying many words). You'll see the huge ae-bias, the large effect on greek logic thought its effect on math. Now, what's even more interesting is to also take a look at the east. Indo-European languages are all subclassed together in a morphological sense. Most classic Indic languages are heavily prosodic. Sanskrit is a very syntaxic heavy language. Actually the Dali Llama's take on it was quite interesting (see: Universe in a Single Atom) about how ancient wisdom, and indeed Buddhism spread from India to Tibet in the first place. It was over the himilaways so the trek was akin people trying to invade the Swiss during the Middle Ages. The vedas seemed to be like early particle physists (and where basically describing the world using all possible of the word were equipossible with basic afterlife being measurable via some abstract karma) who wiped out the memory but indirectly mentioned other Decartes (mind/body)-like sects. I wonder if it had anything to do with humidity needing that much 'state space'. Celsus the greek school of thought (so). The rate at our technology reinvention or more weakly put our transmission of knowledge about nature and what words mean relative to the experiments and phenomenon they describe has rapidly gotten better, but words cannot simply express intent content in and between n-ary multiple brains in a global enough semantic way. I think various IEEE/W3C and even Unicode will help debias and preserve aboriginal cultural behavior.
> Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?
Yes, this describes entire history of AI including several boom-bust cycles. In particular the 80's come to mind. Yes the practitioners think that there's no technical barriers stopping them from eating the world, but that's exactly what people thought about other so-called revolutionary advances.
Although to be pedantic, "expert systems" is the technology behind AI boom of the 80's. At the time people were saying expert systems can't be as good as existing algorithms (including what we would now call "machine learning" techniques), then suddenly the expert systems were better and there was rampant speculation real AI was around the corner. Then they plateaued.
We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.
I think the disconnect this time around is in productionization. We're getting breakthroughs in a wide range of problems, and translating those gains in the problem space into 'real' stable, practical solutions we can use in the world is the remaining gap, and often takes years of additional effort. It's still really expensive to launch this stuff, and often requires domain expertise that the ML research team doesn't have.
We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful.
I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.
Next time you fly through a busy airport, think about the system which assigns planes to gates in realtime based on a large number of variable factors in order to maximize utilization and minimize waits. This is an expert system design in the 80's and which allowed a huge increase in the number of planes handled per day at the busiest airports.
Or when you drive your car, think about the lights-out factory that built-it, using robotics technologies developed in the 80's and 90's, and the freeways which largely operate without choke points again due to expert system models used by city planners.
These advances were just as revolutionary before, and people were just as excited about AI technologies eating the world. Still, it largely didn't happen. To continue the example of robotics, we don't have an equivalent of the Jetson's home robot Rosey. We can make a robot assemble a $50,000 car, but we can't get it to fold the laundry.
These rapid successes you see aren't literally "any problem from any field" -- it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve. Don't expect them to have as much success on every problem they put their minds to.
No, machine learning is not trivially solving the hardest problems in every field. Not even close. In biomedicine, for example, protein folding is probably one of the easiest challenges. It's a hard problem, yes, but it's self-contained: given an amino acid sequence, predict the structure. Unlike, say, predicting the metabolism of a drug applied to a living system, which requires understanding an extremely dense network of existing metabolic pathways and their interdependencies on local cell function. There's no magic ML pixie dust that can make that hard problem go away.
Well, we can agree that world peace is off the table!
Beyond that, let's notice that expert systems did indeed change how airports and freeways work: They improved the areas where they solved problems. Deployment happened.
What we're seeing now is new classes of previously unsolvable problems falling. Deployment in medicine is known to be particularly hard, but not impossible. My read on the situation is that there have been a number of ML applications in the current round that have been kinda-successful 'in vitro' and failed in deployment. That doesn't mean that all deployments will fail.
Furthermore... Neil Lawrence points out that in most cases we change the world to fit new technologies. For example, mechanized tomato pickers suck, so we develop a more machine-resistant tomato. Cars break easily on dirt roads, so we pave half the planet. ML/AI somehow flips people's expectations of how technology works, and expect the algorithms to adapt to the world. This is almost certainly wrong.
"it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve."
I'm actually not sure this is at all true. Protein folding is a long-standing grand challenge on which no current methods were working. My guess is that it was initially chosen for potential impact, and chased with more resources after some initial success.
> We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.
What a take. Neural networks just took a huge bite out of protein folding and your take is: This just in, the Deep Learning boom is about to go bust! Asinine.
Considering the resource requirements for this AI approach mentioned in the article, its unlikely that its been tested on more than a few tens to hundreds of proteins. This may only work on a subset of the proteome so I would think it worth it to continue playing if you find it to be a fun past-time.
fold.it was always more geared towards being edutainment than actually contributing solutions. Of the ~20 publications made related to fold.it over a decade, ~5 of them seem to have contributed to solving structures, while the rest of them are about the game itself.
Not knowing a lot about biotechnology, I read the article and it sounds great, but how big is this as a gamechanger? Can someone comment on how big are the implications of this in, let’s say, 5 years from now, on day to day life? Does this mean that biotech is going to explode? Or just that drugs will come to market faster, perhaps cheaper for rare diseases, but from the same industry structure as always?
This will allow us to discover much more about the structure of the cell (of "life") at a before this unprecedented speed. We should find many, many more mechanisms and targets for medicine, but it takes 10-20 years to bring a new medicine to market.
So in 5 years you'll see exactly zero new medicines pop up.
No new medicines, but way more biotech tools. Higher yield GMO plants, foundational research into disease, science backed recommendations for lifestyle changes to avoid disease that previously eluded us, some crazy stuff happening in animal models. The progress in biotech the past 20 years makes moore's law look slow.
I agree. The main inhibitor of speed that products of this advancement will be deployed at will likely be determined by local policies. Though, given just how profound some of the impacts on medicine might be, the speed at which they can be deployed might become a matter of national security (a healthier population bodes well for a healthier economy which in turn strengthens national security). Hopefully this competition shortens the time-to-market for all these new medicines.
Protein folding is a big and important problem, so this is certainly big news if it works as well as it seems. But I wouldn't assume that this changes everything, we can already determine how proteins fold by experimental work. The disadvantage is that this is a lot of work, though the methods there also improved a lot.
One question is how robust the predictions are that DeepMind produces. I would also assume that right now it can't e.g. determine protein structures in the present of other small molecules, or protein complexes. A lot of the interesting stuff lies in the interactions between molecules.
And in general in life sciences any new development will take at least a decade until it hits day to day life, likely even more. We're living with a exception to this rule right now due to the pandemic, but in general things take quite a bit of time in that space.
We can already determine how a few proteins (170k — which sounds like a lot, but which is only 0.09% of all currently-catalogued protein sequences) fold by experimental work.
What an accurate model of protein folding allows us to do, is to take our big database of DNA, predict protein foldings for all of it, and then stand up a search index for this database, keying each amino-acid "row" by the "words" of its predicted protein's structural features.
We could then, with a simple search query that executes in O(log n) time, find DNA targets that produce molecules with interesting structures that might be worthy of study.
This would, for example, be a game-changer in how biopharmaceutical macromolecule-therapy R&D is conducted. Right now we have to notice that some bacterium or another produces some interesting protein, and then engineer a bioreactor to get more of that protein. With this tech, we can work backward from an entirely hyothetical, under-specified "interesting protein", to figure out what catalogued-but-unstudied DNA sequences produce never-before-catalogued proteins that fit that particular functional "shape", and therefore might do the interesting thing. Then we can either directly synthesize that same DNA, or find the organism we originally sampled it from and study it more.
"A few" does appear quite dismissive of the enormous amounts of effort in structural biology so far. There are more than 170,000 structures in the PDB right now.
To determine potential targets for drugs we have to understand what the proteins do. Having the structure is not really enough for that, it doesn't tell you the purpose of the protein (though it certainly can give you some hints).
In most cases the proteins were determined to be interesting by other experiments, and then people decided to try and solve their structure. So the structures we already solved are also biased towards the more biologically relevant proteins.
170,000 is three orders of magnitude less than the number of recorded protein sequences. I don't think it's dismissive to describe that as comparatively few.
I also don't know the field and the opposite concern is that 170,000 sounds like a lot, but, apparently, it's a relatively small amount compared to the number of proteins there are. It makes sense to me to refer to it as a small number - e.g. "That hard drive is tiny." "No, it stores several million bytes..."
Structure is much, much more conserved than sequence. In other words, protein sequences with low sequence identity can fold similarly due to the physical constraints that guide protein folding.
170k is "a few" compared to 180 million (i.e. the size of the PDB as soon as someone runs AlphaFold over everything in the UniProt.)
> In most cases the proteins were determined to be interesting by other experiments, and then people decided to try and solve their structure.
Yes, that's what we're doing right now, because structure is not a useful predictor, because we don't have structure available in advance of studies on the protein itself. There was no point to a "functional taxonomy" of proteins, because we were never trying to predict with protein-structure as the only data available.
In a world where protein structure is "on tap" in a data warehouse, part of the game of bioinformatics will become "structural analysis" of classes of known-function proteins, to find functional sub-units that do similar things among all studied proteins, allowing searches to be conducted for other proteins that express similar functional sub-units.
It's a step forward for sure, but structures change over time to perform their function. The method described here only returns a static structure. Much more research and development is needed to be able to predict the dynamic behavior and interplay with other proteins or RNA.
Determining what a protein structure does might be even harder than folding. Right now we can't really do that ab initio, you have determine the activity in the lab and then look at the structure. And that allows you to potentially identify this motif in other proteins.
If someone produces an AI that you give a sequence and it tells you what the protein does exactly, I'd be extremely impressed. I don't see that happening soon.
The specifics matter a lot here. We can often determine rough functions for subdomains by homology alone. But that really doesn't tell you the full story, it only gives you some hints on what that protein actually does.
My understanding of this is not perfect, but wouldn't answering the "actually does" question require a full biomolecular model of the cell, or even the whole organism? If so I see what you mean. I suppose that it might be possible to get around this by improving the theory of catalysts so that you could look at a site and say, "oh, this will act in such a way..." Dynamic quantum simulation of a few atoms at the active site is hardly easy but a far sight easier than the other.
"If someone produces an AI that you give a sequence and it tells you the protein conformation, I'd be extremely impressed".
Sure there are many more things to solve in this space; but that doesn't take away that this is an impressive achievement and does unlock quite a few things (including making more tractable the problem you just brought up). I'm excited to see what DeepMind works on now and what the new state of the world will be just five years from now.
I think I have to clarify that my response was to a large part to the "this will change all our lives" part, and might look too negative on its own. I'm very, very impressed by these results, but that still doesn't mean that we just solved biology. If this works that well on folding, this could mean that a lot of other stuff that simply didn't work well in silico might come into reach.
I'm maybe overcompensating for the tech-centric population here, with some comments speculating for very near and drastic impacts from discoveries like this. Biology and life sciences are much slower, and there's always more complexity below every breakthrough. That does tend to push me towards commenting with the more skeptical and sober view here.
> as soon as someone runs AlphaFold over everything in the UniProt
It'll take a while before those results can be trusted, though, right? There's probably a selection bias in the training data for proteins which are easy to crystallize, so many proteins probably aren't well represented by the training examples.
Considering that this system "uses approximately 128 TPUv3 cores (roughly equivalent to ~100-200 GPUs) run over a few weeks" to determine a single protein structure, making predictions for all proteins encoded in a human genome seems impractical at this stage. With luck, this advance will help lead to discovery and definition of new folding rules and optimizations that will make protein folding predictions for the whole human genome more tractable.
I think it is possible to make predictions for all proteins encoded in the human genome. Perhaps you misread a very long and confusing sentence?
Background, Neural networks have two modes 1) training - where you learn all the model weights and 2) inference - where you run the model once on new data. Training takes takes a long time, because you're computing derivatives to implement updates rules on millions or billions of parameters based on iteratively examining massive datasets. Inference is extremely fast because you're just running matrix multiplies of those parameters on new data. And TPUs/GPUs are specially designed to compute matrix multiplies.
The article said: "We trained this system [...] over a few weeks." I searched for, but did not see them identify the inference time. I do expect inference time to be well under one second, though I'm not personally experienced with running inference on this type of network architecture.
For comparison, GPT-3 and AlphaStar have month long training times and real-time (sub-second) inference times.
Still much faster than synthesizing the protein and then doing NMR or cristallography to solve the structure puzzle what easily takes half a year or more (and very expensive equipment).
We can already determine how a few proteins fold by experimental work.
Where "a few" is around 0.1% of the known 180 million proteins. So a relative few and a whole lot.
But the catch is which proteins could we figure out by experiment, and which not. In particular membrane proteins are hard to experimentally determine. But knowing how they fold is very important for figuring out how to get things to react with or get through membranes such as cell walls. Which is an important problem for everything from understanding how viruses work to targeted delivery of drugs. We now have a way to find those structures.
There are post-translational modifications to proteins. This means that for many (most?) proteins, the amino acid chain sequence is different from what you would predict from the DNA. These modifications are dependent on the state of the cell at the time of translation, and so cannot be predicted from the DNA alone. Even with a 100% accurate folding model, we cannot simply know the shapes of all the proteins inside the human body based on the genome.
Entire classes of diseases may become history. Creutzfeldt-Jakob and other prion diseases can now be completely understood. Precision targeting of cancerous cells will become trivial (in theory). Minimal life projects (simplest cell possible) will require less trial and error. In general, it will provide a magnitude level improvement to biotechnology, akin to moving from Aristotelian physics to Newtonian.
Some possibilities: artificial muscles for robots, man-made blood substitutes, designer enzymes to break down plastic and other compounds. Software defined biology, where the pipeline from DNA code to actual protein can now be modeled in silico ahead of time. The biology classes of the future may be less observation of animals and more training in usage of whatever the equivalent of Autodesk for biology will be. Healthcare economics in developing nations will be changed as biochemistry itself may finally become deterministic (to some extent). Orphan drug development price would drop (and if you take into account right to try laws and ignore ethics in favor of progress, then people with rare disease may be cured en masse without bankrupting the health insurance company).
I never worked directly with protein folding or structure, but worked a bit in proteomics on teams measuring gene expression (which you could roughly think of as how much of each protein is found in this cell). IIRC there are 50,000 - potentially millions of "kinds" of proteins found in a human, and the "shape" of most of them is unknown, and that determines a lot about how they work.
So imagine you gave an iPhone to someone in the 1800's, they wouldn't understand how most of it works, but this may be analogous to them finally figuring out some key aspects of the transistor. So it's another tool in the toolbelt and like all good tools will be used in all sorts of unpredictable ways.
Someone else I'm sure could do a lot better at explaining how important shape is to understanding the function and behavior of proteins.
It seems unlikely there will be any large changes in life from solving protein folding. Knowing the structure of a protein (or really, its dynamics) is useful for identifying drugs that bind, but the real bottlenecks n drug discovery and biotech are elsewhere.
If folding and docking, alongwith dynamics simulations, start getting commodified, that might change things significantly though. I can already start imagining project workflows that are significantly streamlined without much thought, god knows what other scientists would dream up when we reach those steps
The most accurate technique in computational drug discovery is protein-ligand binding prediction (https://blogs.sciencemag.org/pipeline/archives/2015/02/23/is...). Given the protein structure, you can predict which molecules will bind with it, even for molecules which have never been sythesized. Many protein targets have not been amenable to this because we don't know what the potential binding pockets look like. That set of proteins will now drastically shrink. We're going to have a lot of new drug candidates, and with any luck new drugs, come out of this.
Getting from DNA structure from tissue samples is relatively straight forward.
DNA -> RNA -> unfolded protein is basically one-to-one mapping in most cases.
How protein functions depends on how it folds into itself. Once you solve protein folding, you can take DNA sample and see the structure of the molecule without working in lab using crystallography techniques.
Solving protein folding is huge, Nobel in chemistry scale achievement. It would be massive leap for biochemistry.
It seems that Deep Mind solved competition benchmark and made huge leap, but it's just partial solution that works on limited set.
After you have solved protein folding, there is still problem of solving chemical interactions between molecules accurately. Quantum chemistry is extremely compute intensive.
IMO, this is huge. One of the biggest applications of ML to science that I know of for sure. People used to manually crystallize proteins at great effort to solve for structures.
Of course, there is a caveat. The static, crystallized structure is only one aspect of a protein. The dynamic behavior dissolved in H2O, at different pH, different ionic strength, with different ligands/cofactors are all also important, and not (afaik) directly addressed by this research.
One young lady I knew worked on neural algos recognition of X-ray images.
They always had single digit, bizarre artifacts, where the program can't sometimes recognise the very data it was trained on with most minute differences.
Other artifact was that the most "stereotypical cases" were least reliably recognised, and they hot a lot of flak for screwed up live demos, where a radiologist put a very, very obvious tumor shot onto the scanner, and it didn't work without a half an hour of wiggling the film, and a camera.
The "bruteforce" solution may well be always, 80-85% off, but off consistently, and always. NN algo so far beat them, but fail with double digit frequencies on "artifacts" which they themselves can't do anything about.
How well it deals with the later, is what I believe will measure its real world usefullness.
May well be, but if you spend more compute, and human time checking for those corner cases than if you went with another, more consistent exhaustive search algorith, then the method looses to it economically.
This is more the case the more close to bruteforce you come, like encryption cracking. Imagine, spending years of HPC cluster time, trying to break a password, while knowing you have a single digit chance to miss the right key, in a way which would be completely impossible with with a conventional solution.
I find this disingenuous. Yes, its important that the algos can perform well on real world data, but the framing of this post begins with an anecodote about one person who had a bad model, and implicitly extrapolates that these problems are generalized throughout all neural nets.
One could say the same thing about programmers automating a task, or a number of other trivial examples. I would lean towards assuming deep mind has competent model validation teams vs. not, even if data science is hard.
She can still work on complexes, binding modes, and engineered biomolecules (eg, protein–drug conjugates and antisense oligonucleotide dimers) where the training data isn't really there.
The industry process will not change. You still need industrial biologists to generate and validate AphaFold structures, interpret the results as part of the bigger picture, and to finally design the drugs. And, then, of course you still need to validate the drugs in experimental systems (first the test tube, then mice, then humans).
So your second guess is correct - one of the steps is much cheaper now, which marginally improves the entire pipeline. As a result, drugs should now arrive to the market faster.
As a side note, I am curious what happens to the field of structural biology in 10 to 15 years from now. Every research university has a large structural biology department with super expensive Xray/NRM/Cryo-EM machines, and armies of students who routinely spend 4-6 years of their PhD trying to solve a structure of a single protein. If AlphaFold works as advertised, NIH will gradually shift funding to other problems.
(It was predicted that it'd be taxi drivers, not professors, that AI got first. Ironic.)
> "armies of students who routinely spend 4-6 years of their PhD trying to solve a structure of a single protein"
Back in the 1990s, when I worked on structure data, I remember that at least some crystallizations were easy enough they could be done as a rotation project.
> Macromolecular crystallography evolved enormously from the pioneering days, when structures were solved by “wizards” performing all complicated procedures almost by hand. In the current situation crystal structures of large systems can be often solved very effectively by various powerful automatic programs in days or hours, or even minutes. Such progress is to a large extent coupled to the advances in many other fields, such as genetic engineering, computer technology, availability of synchrotron beam lines and many other techniques, creating the highly interdisciplinary science of macromolecular crystallography. Due to this unprecedented success crystallography is often treated as one of the analytical methods and practiced by researchers interested in structures of macromolecules, but not highly competent in the procedures involved in the process of structure determination.
Certainly some proteins are extremely hard to crystallize, and the new single-atom EM work will help a lot. But are there really "armies of students who routinely spend 4-6 years of their PhD trying to solve a structure of a single protein" these days?
I honestly don't know. I'm sure some do. But if so, that army is pretty small compared to the vast numbers who more routinely use crystallography.
Also, one important thing to realize is that AlphaFold was trained largely on proteins that we were able to crystallize. I'd be very curious to see how its performance fares as a function of 'ease of crystallization'.
You aren't wrong. I got caught up making the comparison between structural biologists and taxi drivers being ran out of business by AI, so I ended up exaggerating the work load that's addressed by AlphaFold. I should been more precise.
I had a friend who solved the structure of 2 or 3 new proteins pretty much by himself his senior year of college. I also had an acquaintance who was a PhD student in the same lab, who said (jokingly) that she hated him because she had spent 5 years on a single protein and got way worse results than he did. I got the sense from talking to them that the process of figuring out how to get a protein to crystallize is basically just trial and error over and over—my friend himself said he basically got very lucky several times in a row (though he is also a brilliant biochemist).
Anyway that anecdote is pretty much the entire sum of my protein crystallography knowledge, but perhaps it explains how your experience and GP's statement can both be true?
In short, a core problem of biochem (the wagon) was just hitched to Moore's law (the horse). Our understanding of proteins will now grow exponentially not linearly, helping us to move up a level of abstraction to higher level biochemistry and biology problems.
Given the DNA code for one of the "machines" that run cells, we can generate an atomic model of that machine. This means we can "compile" (one part of) the DNA code. It was already possible, but so slow that entire datacenters would spend months calculating this for a single protein and even then we can't use them on the really complex ones at all, necessitating things like neutron spectroscopy which are totally insane, and only work on like 1% of proteins.
This is useful because for example chemical simulation tools don't run on DNA code, but on atomic models. And also to produce "images" of the molecules (images between quotes because most proteins are too small to interact with reasonable photons, and no interaction with photons means you can't see them in any way)
DNA has other parts that are really important but we don't understand at all yet, where this doesn't help at all. This applies to sections of DNA sent to ribosomes, to produce actual molecules. Besides that, there are pieces of DNA that "index" the DNA, pointers (from one gene to another), triggers (that for instance start production of an enzyme based on some external influence, like detection of a marker molecule) and export markers (that tell you what to do once the protein is produced, for example, mark a protein to be removed from the cell, incorporated into the cell membrane, or for instance used inside the cell nucleus, and there's also one that essentially says "at this point stop producing a protein and instead couple the rest of the DNA code to the end of the protein you just made").
So what? The DNA only codes for the RNA and amino acid sequence. Structure determination is yet another topic. When we determine the protein structure we already know the sequence. Neither DeepMind has to look at the DNA to train their DNN.
Have you read the article? It's about protein structure determination. The DNA only determines the RNA and amino acid sequence. But who cares. I will get a bit less work and citations because http://cara.nmr.ch/doku.php will be less used in future.
The full chain is DNA -> mRNA -> Ribosome -> tRNA combinations -> amino acid chain -> protein.
It's true that in nature there are many steps between DNA and proteins (this list doesn't even include the steps that mediate the translation, ie. start it, stop it, slow it down, ...), but the structure of a protein is fully determined by the DNA code.
Protein folding is about you start from the DNA code that is fed into the ribosome ignoring all the meta information, and come up with an atomic model (VERY long list like "H atom at 3.27,2.17,12.18, C atom at 2.87, 2.19, 12.33, ..."). Now there's a million niceties we've discovered to make this problem simpler and nicer looking, but that's what it boils down to.
Thank you very much; almost forgot I did a Phd on the subject ;-)
But anyway your answer does not contradict my statement. What you say belongs to the basics of molecular biology, but does not justify that DNA should be considered when determining the structure of proteins. In practice, the amino acid sequence is always already present.
For the sceptics: if you read the referenced article, you will see that it is about protein structure determination by means of deep neural networks. It's not about gene expression, which is a different topic. What benefit does it have to respond to the question "What are the immediate real-world applications of this" (see above) by reciting some molecular biology dogmas from text books mixed with misconceptions, instead of responding to the real question?
Nobody is suggesting that this research has anything to do with gene expression or anything like that. Their point was simply that we now have better tools to actually see the meaning/effect of a given DNA sequence.
Also, there is no need to passive-agressively highlight your credentials. I already researched them before replying.
I rather think most people comment without even having a look at the referenced article. And since when is the reference to a qualification considered aggressive? If your doctor hangs his doctor's certificate on the wall, is he "passive-aggressive"? Pretty weird.
> that we now have better tools to actually see the meaning/effect of a given DNA sequence
Note that the "meaning/effect" of a DNA segment encoding a protein is known and unrelated to the protein folding process. The protein gets its conformation after the translation process.
> Note that the "meaning/effect" of a DNA segment encoding a protein [...]
The "meaning" of a DNA segment is not to encode a protein. The "meaning" is to describe a mechanism in the host organism (by way of encoding a protein). That is a complex process which involves gene expression AND protein folding.
For example would you say that the "meaning" of some Java code is to generate bytecode? Of course not, the "meaning" is to run some algorithm on the computer that executes it
> What are the immediate real-world applications of this?
A protein is actually a linear sequence of amino acids, but in a cell this sequence has a three-dimensional arrangement like a clew of thread. The arrangement is not random, but dependent on the specific composition of the sequence (i.e. selection and order of amino acids) and some other factors. To understand the function of a protein, we need to know this three-dimensional arrangement (i.e. structure). Up to now the structure determination process was mostly manual, complex, time-consuming (several months up to more than a year) and error prone. If structure determination by DNN is reliable, this is a big win for life science. There are still a lot of problems open: e.g. the structure is not constant over time but there are "moving parts" in the structure which are important for its function.
For-profit corporations that value protein engineering will beat a path to DeepMind's door ASAP, like pharmas.
Protein conformation prediction is essential when engineering new small-molecule drug compounds that must 'dock' with the specific proteins that regulate disease. Knowing how to create a protein with the precise shape to become biologically active has soaked up a lot of R&D funding toward pie-in-the-sky techniques that promise to advance that agenda (like quantum or DNA computing).
If this method works as DeepMind says, it will immediately be adopted by every pharma to assess and tweak the shape of candidate proteins.
you give pharma too much credit. I had built a previous system to do something similar to this that produced excellent results and tried to give it away for free to Genentech, which ignored me. They said it didn't work for their purchasing department.
I don't believe you, but I look forward to you showing proof of this with some links (and if you tried giving it for free, I assume you just open sourced the whole deal, so I look forward to a repo link or the like).
we offered the service for free to Genentech since I used to work there and knew they could probably use it to get some good publications.
We didn't open source the distributed computing framework, but the underlying technology (Folding@Home) is based on gromacs, which is open source. It's the scale at which it ran, and the processing pipeline for filtering the results that had the real value.
I feel that the "produced excellent results" has a lot of unpack there.
It obviously wasn't scoring 90+ in CASP.
Actually, after reading your linked blog post, it's pretty obvious why they weren't exactly chomping on the bit:
"To gain insights into the receptor’s dynamics, Kai performed detailed molecular simulations using hundreds of millions of core hours on Google’s infrastructure, generating hundreds of terabytes of valuable molecular dynamics data."
Hmm, yes, hundreds of millions of hours of cpu time, hundreds of terabytes of data, who says no to that? It doesn't even seem seem to attack generalized protein folding in general. It really seems like the plan was, "let's attack this problem with a Google-sized firehose" rather than created a fundamentally different algorithm that had game-changing results.
Comparing your system to AlphaFold seems like your really bending the truth here.
If you note, the paper has an enormous number of citations from pharma, since modelling protein dynamics, rather than static structure, is key to understsanding ligand binding firehose.
You can see another paper we published where attacking the problem with a firehose helped unlock a long-standing problem: https://pubmed.ncbi.nlm.nih.gov/24265211/ in this case, showed that bond angles need to be 'free' to move rather than fully constrained,to build the most accurate models. This paper is also heavily cited amongst protein modellers.
It is correct that the MD simulations don't directly work for CASP- in a sense, the results they produce directly disagree with CASP's mental model of protein structure and function.
I think this is the interesting part because there aren't going to be the same regulatory hurdles for using ribosomes to manufacture technology as there are for medicines. Synthetic organelles that weave fibers, build metamaterials, etc could lead to pretty magical advances in our capability.
Far from an expert here, but your comment makes me think of Michael Crichton's 'Prey', if you've not already read it.
Not that I wish to add to your apprehension.
There is probably a team at DeepMind working on cracking simple crypto.
Problem is, it can be difficult to cast the problem properly/“correcty”. How does a one way function get represented?
Far from an expert on complexity theory, but NP-hard problems can be approximated in polynomial time. With Deep Learning you are doing approximation. So this is nothing ground breaking in that respect.
That actually isn't totally true. Approximate methods, in the formal sense, require a guarantee that they perform within X of the optimal solution. Not all NP-hard problems have polynomial approximations and the methods shown here are likely not approximations because they very likely provide no guarantees on performance. They provide zero guarantees.
No. It really is just heuristic building. A core problem with using ML in this sort of use case is that it is often brittle. Once it gets outside of the context it was trained in it may or may not be able to generalize it's training to new contexts. We may have difficulty knowing when it is very wrong.
I think ML in research science could be viewed as a very good intuitive oracle. Even if they are right 95% of the time, you have to do this work prove the long way every time because that 5% matters. The real utility is in "scanning the field" to better focus research on things likely to bear fruit.
I think I'm almost as uninformed as you, but I believe it comes down to the difference between perfect solutions and close enough solutions. Consider the classic NP problem of the traveling salesman problem.
"[Modern heuristic and approximation algorithms] can find solutions for extremely large problems (millions of cities) within a reasonable time which are with a high probability just 2–3% away from the optimal solution." [0]
When close enough is enough, NP problems can often be solved in P time, and I suspect this is one of those cases. For crypto however, close enough is not enough.
> But wasn't protein folding supposed to be NP-hard?
Yeah, at least some variations of it are NP-hard. SAT is THE NP-complete problem, but there are some really good SAT solvers around. This basically means: They have a solution that mostly does very well on most instances. But because (probably) P != NP, you will never have a polynomial time algorithm for this.
I think that this is a heuristic "near optimal" method rather than an exact analytic method (I have little to no idea of what that would be in protein folding). A domain I do understand a bit which is np-hard is the travelling sales man. Computing an exact solution is unrealistic, but doing heuristic searches that get you to 99% of the optimal 99% of the time is relatively doable.
But - you don't know that you are 1% from the solution... even if you are pretty confident that you are. It's quite possible (unlikely) that you are way off the optimal, but if you have a decent solution that's ok.
NP-hard doesn’t say how hard it is to solve finite problems. Even for n = 1,000,000, O(e^n) isn’t necessarily problematic, if the constant is small enough, or if you throw enough hardware at it.
This “uses approximately 128 TPUv3 cores (roughly equivalent to ~100-200 GPUs) run over a few weeks”. That is a moderate amount of hardware for this kind of work, so it seems they have a more efficient algorithm.
Also, this algorithm doesn’t solve protein folding in the mathematical sense; it ‘just’ produces good approximations.
Been out of the field for a while, could someone currently in it qualify these results? Hyperbolic title notwithstanding, they approach 90% median free modeling accuracy. The "other 90%" still remains to be solved...
I don't think anyone on HN is going to have more authority to qualify the results than the independent experts quoted in the linked article. Among whom are numbered a Nobel laureate, the president of the group that designs the tests of protein folding systems, and the former CEO of Genentech+current CEO of Calico.
I would imagine that he is not assessing this advancement merely using his own personal expertise, but rather the combined expertise of the resources he represents. CEOs don't just look at problems and potential solutions. They have people who look at those things, and then tell them their opinion. In any case, you've picked a nit with one of the three people quoted. Any objections to the other two?
My main objection to Vivek (the Nobel Prize winner) is the prize in that case should have gone to my advisor, Harry Noller. John Moult... he's a nice guy but I think he's being a bit breathless here.
CASP is not "the organization that tests protein folding". It's an organization that every two years does a blind prediction and publishes the results (I've competed, some 20 years ago). John's a protein expert, no question about it.
I knew him moderately well back in the day because our advisors moved in similar circles.
The method relies on multiple-sequence-alignment (MSA) of homologous proteins. This cannot fold arbitrary proteins, only biologically relevant ones that have high quality MSAs available. It's also worth pointing out that the gold-standard for validating MSAs relies on PDBs of folded proteins. This is exciting work that will assist NMR and XRay crystallographers, but it's not a panacea of protein folding.
It doesn't matter so much how they perform the feature extraction, so much as what their inputs to the feature extraction are.
This model requires a collection of wild-type proteins in an accurate MSA. Producing an accurate MSA is hard even if you have many homologs.
They require protein homologs which means they can "only" do this for wild-type proteins. This work is useless with mutant and synthetic proteins. This is a big advancement that will assist crystallographers and NMR structural biologists with difficult wild-type proteins, but it doesn't "solve protein folding" by any stretch of the imagination.
> Producing an accurate MSA is hard even if you have many homologs.
To assess co-evolutionary couplings the amount of homologs in the MSA is not as important as the number of effective sequences (i.e. sequence depth and diversity) in it.
> They require protein homologs which means they can "only" do this for wild-type proteins.
Even remote homologs work, as shown by the widespread use of HHM-based methods in the prediction pipelines.
> This work is useless with mutant and synthetic proteins.
Unless you generate a flurry of data with them using deep mutational scanning for example. As long as correlated mutations are present in the MSA the technique should work as expected no matter where the protein sequences originated.
I'm honestly not familiar with "deep mutational scanning." Can you share a link? I'm first author on papers related to the structural biology of coevolution and I competed in CASP about a decade ago, but I haven't kept up much since then.
Has anyone got any good other references for this? After some of the dodgy experiments related to alpha zero (comparing to purposefully degraded chess systems), I'd love to see some independent analysis.
True, but I haven't seen an independent discussion of the CASP results. There is a good chance this is great, but I don't trust deepmind press releases.
The article in Science implies that we have independent confirmation of predictions yielding useful results, beyond the challenge itself:
> The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”
> But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”
The "dodgy experiments" (setting the per-turn computation time to a fixed value) in the chess system were only in the pre-print. In the actual publication, they allowed for full time control of the most up-to-date version of stockfish.
Yes, and then Leela Chess Zero, an open source implementation of AlphaZero, beat the latest Stockfish in the de facto engine championship (TCEC). Since AlphaZero, the TCEC finals have been traded back and forth between Stockfish and LCZero.
This last season, Stockfish won by using NNUE, a neural network based evaluation function.
This sounds big, like really really big. At least from my old times providing my idle computing resources to Folding@Home and following that project, this seems like the major golden milestone for protein folding.
but experimental methods have not solved protein folding either. AlphaFold has'nt solved protein folding but I can't wait to see their progress for ALphaFold 3.
What would be informatively useful would be to know how much accuracy is needed on average for drug engineers, I'd say that 99% is more likely to be the minimum to make solid inferences
> but experimental methods have not solved protein folding either.
I might be missing something here, but isn't "experimental methods" just shorthand for "our best knowledge of a protein's structure, obtained via NMR or X-ray crystallography"? In that case, I'm not sure what "solving" protein folding even means - literally zero mean error? We can't know/solve anything beyond our best knowledge, that's tautological.
> What would be informatively useful would be to know how much accuracy is needed on average for drug engineers.
Yeah that would be interesting, but:
> I'd say that 99% is more likely to be the minimum to make solid inferences
It's pretty clear what solving means, it means to have an exact representation of the 3D structure. Our partial knowledge obtained from such techniques is what it is, partial. We need new metrology that increase the observability accuracy and completeness OR better deterministic models from sequences.
"We can't know/solve anything beyond our best knowledge, that's tautological." yes it is indeed tautological if you assume that experimental methods can't get better then guess what? It follows that they can't get better!
"what are you basing this on?" on nothing solid, that's why I say it would be interesting.
99% is a non negligible error rate given that proteins have generally a not very high atom count and they the protein will be produced an enormous amount of time, then the 1% error progagate and can a priori easily break the system.
But this guess is not solid as I'm not an expert.
99% accuracy for simple (low atom count) proteins is a sensitive error and could be negligible for very high atom count proteins.
> It's pretty clear what solving means, it means to have an exact representation of the 3D structure.
That's not clear at all, because perfect measurement doesn't exist. I agree that improving is always a worthy goal, but clearly we don't need 100% accuracy to consider something "solved" for the purposes of science. Also, "3D structure" of a protein is not a fixed truth, the parts are in motion all the time and may even have multiple semi-stable conformations. Rather than focusing on X,Y,Z perfection, I would imagine getting the angles between bonds, or the general topological conformation right would be more valuable.
> if you assume that experimental methods can't get better ...
I'm saying that if your definition for "solved" is "perfect knowledge", then we might as well not discuss whether method X or Y solves the problem, because they obviously do not.
The more I think about it, the more I think we should just drop the whole debate over the word "solved". Clearly different experiments and different proteins will have different requirements which may or may not be met by this or by other techniques - I agree that I would be interested to hear an expert weigh in on those requirements.
If you can explain how gravity works in a quantum level you'd deserve a Nobel. It's not 100% solved, Newton's Laws of Motion are a model, not a solution. Just like the vast majority of science.
No, they are very crude (but useful!) models of reality. General relativity and quantum electrodynamics are much better corresponding models, respectively, and even those are just approximations.
If there was a headline, "Company X with Product Y cured cancer" and it turns out that product Y actually only cured 90% of cancer, I'm pretty sure most people would be happy the headline.
Oh, and to be a true parallel example, in this case the remaining 10% of cancers might not even be cancers, as experimental accuracy of protein structures is only ~90% accurate, the model could very well be more accurate than our current ability to experimentally detect protein structure.
I really interpreted that headline as "found a general solution to the protein-folding question" not as the also interesting but not that much "can be used to solve protein-folding problems".
After I had some time to think about it, I come to a different conclusion. Contrary to my first assumption, Bio NMR (in contrast to crystallography) will become more and more important, since the method allows to study the dynamic properties of proteins. With the structure predicted by DNNs, the chemical shifts to be expected in the NMR spectra can be calculated; the assignment problem is thus largely eliminated. Bio NMR can then be used specifically to study the "parts that move".
Sometimes announcements like this are a bit over-the-top. But what really, to me, cements the 'big-deal' of this is the "Median Free-Modelling Accuracy" graph half way down the page.
Scores of 30-45 for 15 years. Now scores of 87-92.
This isn't a minor improvement, it's a leap forward.
Not to mention the fact that two years ago they took it from 45% to >60%. If they can continue improving, even with an exponential decay in rate of improvement, this is certainly a stunning example of technological disruption.
Even without any improvement, the amount of grunt-work the AI can pre-do and get down to a short-list - that in itself will see changes in progress speeding research up.
That is an impressive improvement, but I think you've missed the most important point:
>a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods
So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure.
I have a related question about this. If experimental methods produce results around a score of 90, what is the baseline we are comparing the DeepMind results against? If the experimental error is equal to the observed DeepMind error, how can we say which one is actually more erroneous?
you really can't compare stats like that. Those are independent, uncorrelated measurements. When you take RMSD measurements on a molecule they are not independent (for example, atoms near the core are less likely to be "inaccurate").
Finding the energy of each configuration should be much easier than finding the lowest-energy configuration. Can that be calculated ab-initio or it is still too expensive?
The problem with ab-initio methods in this context is the sheer number of non-covalent interactions present in these large proteins. A simple protein would require a hybrid quantum mechanic/molecular mechanics simulation to even approximate the vibrational energy required to validate equilibrium.
These proteins are so massive that we often use Daltons [1] as an averaged measure of molecular weight.
Conceptually one of the most promising applications of quantum computing is theoretical chemistry, and we are only now starting to make progress in this avenue [2]. I anticipate it would require quantum computing to explicitly optimise large folded proteins.
The "experiments" here use X-Ray Crystallography. Like most methods of measuring anything, we have a pretty good idea of its accuracy under various conditions.
Think of it like satellite imagery of a tree: A score of zero would be a single green-ish pixel, while a score of 100 would show each leaf within the range it naturally moves in due to wind etc. (proteins tend to wiggle quite a bit under natural conditions, as well)
Something like this comes up in assessing the accuracy of automated segmentation results of brain regions e.g. the hippocampus. Human-machine reliability is approaching the human to human reliability, so it becomes harder to improve the automated methods.
I don’t think you can say DeepMind could ever be more accurate to the true physical structure since it was built on the same experimental structures that it is being compared to. The limit of accuracy is the experimental data. However, I think we can say that a DeepMind prediction could at least be as good as a new experimental structure.
This seems like an obvious assumption to make, but it isnt always true. It is easier to see why if you are measuring a single value multiple times in order to get a more accurate estimate of the true value. In that case your "model" is simply the mean of all measurements made and can exceed the accuracy of a single measurement.
In this case, the model is predicting values of multiple structures, but patterns could still theoretically be found which allow for predictions beyond the accuracy of a single measurement.
Is that true? I thought fundamentally, the simulation tries to find the state of lowest energy, which is defined by physics. So, your result can be better than the data set used for training.
But DeepMind could be used to find errors in the training set.
Let’s say you have 100000 proteins in the training set. Now remove #1 and train on 99999, and then check that it still predicts the same protein result for #1 as the experimental result.
Or remove from training whole sets of proteins by particular teams to find systematic errors made by teams?
DM is merging several experimental data: known x-ray structures, and evolutionary data. The experimental method (xray) doesn't take advantage of the evolutionary data. And it also doesn't model the underlying protein behavior accurately (xray basically assumes a single static model with atoms fluctuating in little gaussian "puffs" around the atomic centers, but that's not how most proteins behave).
I'm not a biologist but I'm not sure that follows. It could be that the experimentally-derived structure is 100% accurate to the actual physical structure but getting 90% of your predicted residues to match that is enough to get an accurate prediction of protein behavior and hence "competitive."
"So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure."
While this is an accomplishment, nobody is going to be confusing these models for structures produced experimentally. The CASP metric is for backbone atoms. To have a useful model of protein structure, you really need to have the positions of the protein side-chain atoms modeled correctly. Experimental methods will do that, but this method, as I understand it, does not.
So it's a really good start, but nobody is going to be throwing these structures into molecular docking simulations for drug discovery or etc just yet. But hopefully those details can be worked out soon enough.
Yeah, there's a huge difference between a 1Å all-atom RMSD structure, and a 1Å backbone RMSD structure. The non-backbone atoms in a protein make up most of the mass and volume. When structural biologists talk about RMSD, this is what they mean.
Then we get the really fun question: if the experimentally determined structure is only 90% accurate, can machine learning actually reach 100%? Can you learn exact truth from inexact examples?
Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works. If so, can we somehow extract that truth, or is it truly a black box that does the thing we want?
I'm reminded of a sci-fi book I read long ago in which humans are discussing the fact that the science they are utilizing is beyond the scope of a human mind to comprehend- only the AIs can intuitively deal with 12-dimensional manifolds (or something to that extent). Maybe we've reached the doorstep of that future.
If you have an experimental error that is somewhat normally distributed around the mean, the the AI should, with enough examples, learn what the rules are that are closest to the mean. Because it will minimize the sum of errors.
So i do think the results could be more accurate than measurement.
I don’t think we can assume the errors are normally distributed. It’s possible researchers are biased in a particular “direction”, away from 0 on all dimensions of this problem.
That's fine. It's still a normal distribution with a different mean. The Gaussian is characterized by having only the first two moments: mean and variance.
> Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works.
Well I think that the results speak for themselves; ultimately the question you raise is one of semantics. ML models don't think in terms of "conceptual ideas" like humans do, these models simply perform at such a massive statistical scale that they can identify patterns far beyond any human conception. Clearly, the model embodies some verifiably reliable information about the way the world works, but this is "just" a trick of statistics not anything resembling actual "understanding" in the way the word is typically used when referring to human understanding.
I don't have a background in biology, and that quote confused me.
What's an experimental method for protein folding and why is it so good? Are they talking about creating an actual, physical protein in a lab and observing how it folds?
> Are they talking about creating an actual, physical protein in a lab and observing how it folds?
Exactly. Researches purify the folded protein and then use methods such as X-ray crystallography, nuclear magnetic resonance, and cryo-electron microscopy to determine its three-dimensional atomic structure.
I’m not a protein crystallographer, but here’s my generalist take.
We understand the physics of e.g. X-ray diffraction pretty well, so we can fit pretty decent forward models for the x-ray data given a proposed structure. The hardest task here is getting a good enough guess at the structure to optimize the physical model, and it’s my impression that people use an iterative model refinement workflow. At least that’s how it’s done in condensed matter materials.
There are many sources of experimental uncertainty, like the non-ideal nature of the x-ray source and optics, and the fact that the atoms in the protein are not static but have some thermal fluctuations. so at the end of the refinement you still have some uncertainty on your model parameters (the interatomic distances for proteins I guess), but if you are careful you can calibrate these uncertainties pretty well.
X-ray diffraction is pretty nutty, too. You're taking the diffraction pattern, which is the fourier transform of the electron density. Fourier transform results are complex-valued data. Unfortunately, we don't really have X-Ray lasers, so you can only get the intensities and not the phases of those diffraction spots. Since mother science hates us, it of course the case that, in a fourier transform "more information" is contained in the phases than in the intensities.
So you "make guesses at what the phases are", the best choice is to bootstrapping these phases measured with another technique (you can introduce crystal defects that do allow you to guess at what the phases are).
Less scrupulous is to use a computer generated model, like fitting another protein "that you guess is related", then you model the electron density, take the phases of that.
In any case you take these "phase" guesses, and then apply it to your intensities, re-run the fourier transform, refine your electron densities, twiddle the location where you think the atoms, are, then repeat with your new model. This process repeats until you converge on a structure that you're happy with.
Now alarm bells should be screaming in your head right now: Yes, it's entirely possible to converge on a wrong structure, especially if you're a young up-and-comer professor seeking tenure that has no ethical problems with "suggesting" their grad students to sleep in the lab and work 100 hour weeks and willing to do slipshod work to get you tenure: https://www.sciencedirect.com/science/article/pii/S002228360...
Best is an orthogonal process (like NMR). Cryo-EM is getting better too so maybe that will start to be viable. Sometimes that's not possible, but you can use secondary evidence: "we know these three amino acids are important band hey look they touch in our model".
This reminds me of AlphaGo and AlphaZero. DeepMind was able to produce a very solid model on their first attempt, at both protein folding and at Go (and Starcraft2 as well). Their second models, however, seemed to blow their first out of the water.
This bodes extremely well for the future of computational biology, I'm very excited thinking about the prospects. If we know how a protein folds, we know its shape, meaning we know which shaped/charged molecules are needed to act as suppressors/enhancers of those proteins.
One difference to AlphaZero though, if my understanding is correct, is that AlphaFold is trained on a predetermined data set and hence didn’t learn how “arbitrary” proteins fold in general, but just how the kinds of proteins fold for which we already know how they fold. To work more like AlphaZero, AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions. Therefore it’s conceivable that AlphaFold is biased by the existing training data and doesn’t fully generalize to all proteins we would want to apply it to. Maybe that won’t be a problem in practice, but nevertheless it makes for a significant difference from what AlphaZero was about, being solely self-trained.
> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions.
Could this lead to a virtuous cycle where AlphaFold is used generate a ton of random sequences where it has low confidence, those are then screened for ease of synthesis, measured and the results used to improve the model?
Edit: nevermind, according to another comment[0] there are still plenty of real proteins without experimental data left to explore.
> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions.
It can verify how much it minimizes the potential energy, which may not always line up with how it would fold in the real world but is a strong indicator.
Why is the graph not monotonically increasing? Does the complexity of the problem to be solved increase each time? If so, does that make the relative improvement from the previous result even more impressive?
That's quite interesting ... I believe the test set size is not constant year to year but rather a function of how many new structures have been experimentally discovered since the last contest?
Does seem like the contest structure could include quite a bit of risk for hiding the effect of overfitting ... I wonder if there is anything inherent about the problem that reduces that risk ...?
My understanding is, that it's always 100 new structures, which is a small fraction of the total structures identified in that year.
The reason why the top score in one year, can be lower than in the previous year, is that the test (the 100 structures to guess) is always new and different, so it can end up being 'harder' than the year before. Luck will also play a small role.
Another explanation for a reduction in the top score would be, that previous winners are not re-submitted unchanged. For instance AlphaFold v1 seems to not have been submitted to the latest competition.
Only 100 new structures each test cycle? That seems a very small test set size ...
Is it really possible to select 100 new structures which together are likely to represent a meaningful increase in the sample generalization versus the prior years test set ...?
Given that we only know the structure of on the order of 100k proteins, we might only get another 10k new ones per year. I guess.
Using 1% of those (presumably from the more-often-reproduced subset) for this challenge seems reasonable? Note that the structures have to remain secret up until the challenge, and presumably all those teams uncovering the structures don't want to have to wait up to 2 years every time to actually make their results public.
Interesting ... plenty of opportunity then potentially for the 100 samples to have prediction similarity to the set of published discoveries (for expected or unknown reasons)?
I suppose it will take a few more years of repetition for the challenge to confirm that the problem has been been solved -- but I wonder if a new version of the contest is going to be needed as well? Maybe the model accuracy is now high enough to invert the contest to a form where models generate predictions for randomly selected unknown samples -- and experimental teams are then expected to make observations for those particular sequences over the next two years as part of their otherwise research agenda selected experimental workload?
There are different categories of samples, namely FM and TBM targets. FM targets don't have any similarity to known structures. Roughly a quarter were FM targets. I think the more interesting thing to look at is the size of the multiple sequence alignments (MSAs) which is the basis of this and essentially all methods. They seem to do very well with few MSAs, which bodes well for other targets, although there are families of proteins with few MSAs.
100 structure with 100+ amino acids each, so it's not quite as bad. Part of the folding information is contained within a distance of a few amino acids, while some (the harder part and crux of the problem) is farther away.
But yeah, compared to other fields, the size of training/test sets is sometimes pretty small in ML for life sciences.
This is a huge jump forward. Last year's performance already was a big step up over the previous, and this seems to go much further. So big kudos to the research team.
Nonetheless, I'd like to hear more from specialists outside the context of a marketing blog post before I fully buy into a claim of a solution.
There's also a rabbit hole about what 'solution' actually means. Is the performance sufficient for any protein folding prediction application that might arise in the future?
Man, I remember running folding@home years ago on my terrible laptop. Now this was done with what they say is equivalent to only 100-200 GPUs. Crazy to see how far we've come in just a short amount of time.
The forward folding problem lets you determine structures from a known genetic sequence. So for example you could very quickly sequence the genome of a virus and figure out how it worked much faster than current methods allow.
The reverse folding problem lets you specify a structure and then make a genetic sequence to produce it. For example you could look at this virus to see how it infects its host, then design a custom protein to act as an anti-body stopping it, which is a capability we don't currently have.
Forward folding is certainly useful, but reverse folding would be revolutionary.
The set of all proteins which can potentially be expressed in an organism is known. Now maybe we also get decent (static) structure information for these. But the interaction of a virus with the host cell is much more complex. There is much more than just an amino acid sequence involved. And these parts are all moving, so a static picture as we now can create faster than before does not contain all the information necessary to fully understand the functions.
No, the genome of the host is much smaller than the theoretical number of combinations. There are about 20 to 30k different proteins in a human cell (about 20k directly encoded on the DNA).
Right, but you made the example with the virus docking at a known organism. If you do synthetic biology and modify bacteria to produce any proteins then the situation is different of course.
The other comment mentioned the example of making proteins that bind a structure. Heres an extension - a general understanding of how an enzyme works to catalyze chemical reactions, is that it binds the reaction intermediate with higher affinity than the two substrates; thus if we have this reverse ability, we can start inventing enzymes that can catalyze any arbitrary chemical reaction, even ones that need energy input, so you could imagine for example enzyme systems that can convert plastic to fuel!
Ok, then this is about enzymes which do not yet exist in the organism. You could then modify bacteria so they produce this enzyme and feed on plastic, I see.
But producing fuel as the fellow suggested would then be another function to be added to the bacterium; and maybe it should work on different kinds of plastic.
Of course, that's why I focused on degradation. There's plenty of room for improvement. For instance, PETase is not very efficient actually, and many research groups are working on its engineering.
I think you have this backwards in practice. It was in the 80s that I first read a paper about a de-novo protein design engineered for a specific stable conformation. Natural proteins have no reason to be particularly predictable, just as genetic programming produces hard-to-understand programs relative to human-written ones. In fact making the structure especially stable against perturbations seems like it'd make it less responsive to changing evolutionary pressures.
This is a big step forward, but the outstanding question as far as to whether or not this is useful for evaluating novel proteins, is going to be how good is the confidence metric at telling the user to trust or not trust the results. You can see from their examples, that AlphaFold is very good but not perfect. I imagine for some proteins it will still give misleading or erroneous results and if you can’t tell when that happens without verifying the structure experimentally then this will likely not be that useful for new science.
> the outstanding question as far as to whether or not this is useful for evaluating novel proteins
That is not an outstanding question. The test on which DeepMind scored high marks is a test of how well the algorithm folds novel proteins -- proteins whose ground-truth structure has not yet been published.
We’d have to see the distribution of GDT scores evaluated on unknown proteins to say anything about how confident we can be. If the distribution is tightly distributed around the median then great, this works really well. If the variance is large though then you’re going to have a hard time using this for meaningful predictions.
According to the article there's a confidence score as well. As long as this is sufficiently predictive of errors either a tight or wide distribution is likely acceptable.
We need to see the relationship between confidence and GDT score. If you have a nice relationship then again everything is great. But... most confidence metrics from neural networks do not have a nice relationship to the primary metric.
You don't generally look at neural network output like that.
There is generally a threshold, less than X, not the class, equal or more, is the class. Then you run the network with the same threshold on a known data set and compute a confusion matrix, which tells you about the error, I don't even want to know what a confusion matrix analogue for 3D geometry would look like but I'm sure they have something.
This is literally the process that one does in taking part of the this. And the error rate (specifically the lack of errors) is what is everybody is talking about. 90 is just as accurate as we can get with experimental measurement. It's likely at this point the source of error is in the data set (we can only train on data we experimentally measure and these are not perfect measurements). It's also possible, at this point, the model generalized so well that when it deviates from experimental measurements it's actually correct and the experimental value was the one that was wrong.
So no, the outstanding question is not "is going to be how good is the confidence metric at telling the user to trust or not trust the results.". Nobody is going to be looking confidence values when it model is giving an output, they are going to be looking at the overall error rate across a broad spectrum of proteins to get a sense of it's accuracy.
Every simulator is going to have error. In this case this biennial challenge represents the computational state of the art with scores of 30-40 over the last decade. The AlphaFold2 model sends that score up to 87 with errors about than the width of the atom. You can actually see the difference between their prediction and the actual result and it’s stunning. This is all on the blog site so I recommend reading before throwing shade.
There's a difference between being a random commentator on HN and being one of the several experts in the field quoted in the article, among other things, predicting a mass exodus from the computational biology field as the major problem of that field is now solved.
I was wondering the same thing. But I also wonder if having good guesses makes the x-ray crystallography and other experiments to verify a given protein easier/cheaper/quicker? I don't know enough about the actual techniques to have an informed opinion but I would think it would be helpful.
Scientists can verify that an AlphaFold-predicted structure is correct, or at least useful, without being able to get the structure experimentally. For instance, we could use the AlphaFold-predicted structure to do protein-ligand binding calculations for a bunch of known molecules. If these calculations agree with experimental protein-ligand binding (which they generally do for proteins with known structures), then we can say with high confidence that we've got a good structure.
The way computer scientists do it, yes, it is. In the CS situation you define an energy function (in this case representing the physical behavior of the protein in water) and find a heuristic to approximate the coordinates of the lowest energy configuration; done, problem solved.
in reality, that's not how it works at all. The energy functions we have are crappy and require too much sampling before we can find the lowest energy configuration. And more importantly, it doesn't look like proteins typically fold to their lowest energy configuration (with the exception of some small fast two state folders), but rather explore a kinetically accessible region around there (or even somewhere else entirely, if the energy cost to transition is too high).
Methods like AF depend heavily on large amount of information correlation from evolutionary data, which has historically been of the highest value for making decisions about protein structure.
There are other experimental methods that are much cheaper that can be used to assist validation. Also the models look damn impressive, even down to the sidechain packing.
It’s a good question, and I’m not a domain expert here.
The article did claim:
> According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods.
So perhaps their score of 87 GDT is pretty significant. But “competitive with” is not the same as “always in agreement with”, as you point out. Could be the failure modes are problematic.
Fascinating work. I wonder if this approach works to model interactions (no reason it shouldn’t).
The interactions of proteins with other proteins and well as as molecules like lipids, water and electrolytes form the basis for cellular processes. If that can be inferred correctly, you are looking at the building blocks of a “human simulator”.
At Sun back in the day our workstations tended to have fairly promiscuous login settings, so one of my coworkers took the liberty to launch folding@home on every machine in the org. Listing running processes one day, I saw this thing pegging my CPU; asked around and others had it too. A virus!?! Then he fessed up. Kinda miffed at first but ultimately really cool, so we let the thing keep running. That was my introduction to the whole protein folding problem, and it's really great to see this milestone!
I ran Folding@Home at Google on hundreds of thousands of fast Xeon cores for over a year. I concluded at the end that unbiased MD simulations are not an effective use of computer time.
682 comments
[ 3.1 ms ] story [ 316 ms ] threadCongratulations to the team! This work will have far reaching impacts, and I hope that you continue to invest heavily in this area of research.
It'll be genetics next.
e: although AlphaFold appears to be convolutionally based! I suspect that'll change soon.
Which part of genetics are you thinking of? Much of genetics isn’t amenable to this kind of ML, because it isn’t some kind of optimisation problem. And many other parts don’t require ML because they can be modelled very closely using exact methods. ML does get used here, and sometimes to great effect (e.g. DeepVariant, which often outperforms other methods, but not by much — not because DeepVariant isn’t good, but rather because we have very efficient approximations to the exact solution).
Genetics is amenable because the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network.
There are plenty of inferences that you would want to do on genetic sequences that we can't model exactly and there is some past work on doing stuff like this, although biology is usually a few years behind.
https://www.nature.com/articles/s41592-018-0138-4
e: for clarity
"the genome is a sequence that can be language modeled/auto-regressed for depth of understanding by the network"
The genome is not a sequence so much as a discrete set of genes which are themselves sequences which specify construction plans for proteins. That distinction is important.
Language modeling in the context of machine learning typically means NLP methods. Genetics is nothing like natural language.
Auto-regression is using (typically time series) information to predict the next codon. This makes very little sense in the context of genetics since, again, the genetic code is not an information carrying medium in the same sense as human language. Being able to predict the next codon tells you zilch in terms of useable information.
"Depth of understanding by the network" ... what does that even mean???
The above sentence is a bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way. AKA word salad.
aka a sequence. "a book is not a sequence so much as a discrete set of chapters which are themselves sequences of paragraphs which are themselves sequences of sentences" -> still a sequence
these techniques are already being used, such as in the paper I just linked.
> Being able to predict the next codon tells you zilch in terms of useable information.
You have absolutely no way of knowing that apriori. And autogressive tasks can be more sophisticated than just next codon.
> bunch of popular technical jargon from an unrelated field thrown together in a nonsensical way
Okay, feel free to think that.
There's always this assumption of it "will never work on my field." I've done work on NLP and on proteins and read others' work on genetics. I think you will end up being surprised, although it might take a few years.
Let's say you have discrete sequences that are a product of a particular distribution.
Unsupervised methods are able, by just reading these sequences, to construct a compact representation of that distribution. The model has managed to untangle the sequences into a compact representation (weights in a neural network) that allows you to use it for other, higher level supervised tasks.
For example, the transformer model in NLP allowed us to not have to do part-of-speech tagging, dependency parsing, named entity recognition or entity relationship extraction for a successful language-pair translation system. The compact transformer model managed to remap the sequences into a representation that allows direct translation (people have inspected these models and figured out the internal workings of it and realized it does have latent information about a parse tree of a sentence or part-of-speech of a word).
Another interesting note is that designers of the transformer architecture did not incorporate any prior linguistic knowledge when they were designing it (meaning that the model is not designed to model language but just a discrete sequence).
> Genetics is amenable because it is a sequence
Not sure what you mean by that. Genetics is a field of research. The genome is a sequence. And yes, that sequence can be modelled for various purposes but without a specific purpose there’s no point in doing so (and furthermore doing so without specific purpose is trivial — e.g. via markov chains or even simpler stochastic processes — but not informative).
> There are plenty of inferences that you would want to do on genetic sequences
I’m aware (I’m in the field). But, again, I was looking for specific examples where you’d expect ML to provide breakthroughs. Because so far, the reason why ML hasn’t provided many breakthroughs in less about the lack of research and more because it’s not as suitable here as for other hard questions. For instance, polygenic risk scores (arguably the current “hotness” in the general field of genetics) can already be calculated fairly precisely using GWAS, it just requires a ton of clinical data. GWAS arguably already uses ML but, more to the point, throwing more ML at the problem won’t lead to breakthroughs because the problem isn’t compute bound or vague, it’s purely limited by data availability.
I could imagine that ML can help improve spatial resolution of single-cell expression data (once again ML is already used here) but, again, I don’t think we’ll see improvements worthy of called breakthroughs, since we’re already fairly good.
I spoke loosely, my mind skipped ahead of my writing, and I didn't realize that we were parsing so closely. "Genetics (the field) is amenable because the object of its study (the genome) is a sequence" would have been more correct but I thought it was implied.
> without a specific purpose there’s no point in doing so
Well yes, prior to the success of transfer learning I could see why you would think that is the case, but if you've been following deep sequence research recently then you would know there are actually immense benefits to doing so because the embeddings learned can then be portably used on downstream tasks.
> it’s purely limited by data availability.
Yes, and transfer learning on models pre-trained on unsupervised sequence tasks provides a (so-far under-explored) path around labeled data availability problems.
I already linked to a paper showing a task that these sorts of approaches outperform, and that is without using the most recent techniques in sequence modeling.
Maybe read the paper in Nature that uses this exact LM technique to predict the effect of mutations before assuming that it doesn't work: https://sci-hub.do/10.1038/s41592-018-0138-4
I am not directly in the field, you are right - but I think you are also being overconfident if you think that these approaches are exactly the same as the HMM/markov chain approaches that came before.
> Maybe read the paper … before assuming that it doesn't work
I don’t assume that. In fact, I know that using ML works on many problems in genetics. What I’m less convinced by is that we can expect a breakthrough due to ML any time soon, partly because conventional techniques (including ML) already have a handle on some current problems in genetics, and because there isn’t really a specific (or flashy) hard, algorithmic problem like there is in structural biology. Rather, there’s lots of stuff where I expect to see steady incremental improvement. In fact, in Wikipedia’s list of unsolved biological problems [1] there isn’t a single one that I’d characterise specifically as a question from the field of genetics (as a geneticist, that’s slightly depressing).
But my question was even more innocent than that: I’m not even that sceptical, I’m just not aware of anything and genuinely wanted an answer. And the paper you’ve posted might provide just that, so go and do my research now.
[1] https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_b...
Then there’s the more clinically oriented approaches of looking at effects, trying to find associated genes/mutations whatever mechanisms exist in between to cause a desirable or undesirable outcome. I’d call that ‘top down’.
I’m sure the lines get blurred more every day, but is there a meaningful distinction into these and/or more categories that are working the problem from both ends? If so, are there associated terms of art for them?
> Think of transformers as really wide (N) and really short (1) convolutions
Modern transformer networks are not "really short" and you're also conflating the difference between intra- and inter- attention.
There is still a pitched battle being waged between convnets and transformers for sequences, although it looks like transformers have the upper hand accuracy wise right now, convnets are competitive speed-wise.
“For the latest version of AlphaFold, used at CASP14, we created an attention-based neural network system”
?
For a non-biologist, on what is this skepticism based?
Just purely based on following ML news it looks like the trend for ML solutions has been that they've overtaken expert-systems once they've gained a solid foodhold in a field. Maybe this is some perception bias. Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?
There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.
[0] https://en.wikipedia.org/wiki/Machine_learning
[1] https://www.cs.cmu.edu/~tom/mlbook.html
It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods.
That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest.
My PhD (now over a decade ago...yikes) was in applying much simpler ML methods to these kinds of problems (I started in protein folding, finished in protein / nucleic acid recognition, but my real interest was always protein design). Even back then, it was clear that ML methods had a lot more potential for structural biology (pun unintended) than for which they were being given credit. But it was hard to get interest from a research community that cared little about non-physical solutions. No matter how well you did, people would dismiss it as a "black box solution", and that pretty much limited your impact.
Some of this is understandable: even today, it's not at all clear that a custom-built ML model for protein folding is of much use to anyone -- particularly a model that doesn't consider all of the atoms in the protein. The traditional justification for research in this area is that if you could develop a sufficiently general model of protein physics, it would also allow you to do all sorts of other stuff that is much more interesting: rational protein design, drug binding, etc.
The alphafold model is not really useful for any of this, so in a way, it's kind of like the weinermobile of science: cool and impressive when done well ("hey! a giant hot dog on wheels!"), but not really useful outside of the niche for which it was designed. So it's hard to blame researchers in this field -- who generally have to chase funding and justify their existence -- from pursuing the application of deep learning to this one, narrow problem domain.
Obviously there will now be a wave of follow-on research, and it's impossible to know what methods this will spawn. Maybe this will revolutionize computational structural biology, maybe not. But I think it's a little unfair to demonize the entire field. Protein folding just traditionally hasn't been a very useful or interesting area, and like all "pure science", it leads to a lot of small-stakes, tribal thinking amongst the few players who can afford to compete. This is right out of Thomas Kuhn: a newcomer sweeps into a field, glances at the work of the past, then bashes it over the head, dismissively.
I will definitely blame the protein structure field in multiple levels though. It was always frustrating to me to open up Nature or Science and see it filled with papers about structure - like they are innovating so much that half of the top science magazines every week have papers in that field, yet it's not going anywhere? Or is it simply just a bunch of professors tooting their own horns about ostensible progress in a field that's archaic by decades if not years? The overall protein structure field internalised some dogmas in self defeating ways to everyone's detriment and finally events like this (and Cryo em, maybe) will jolt them out or make them fully irrelevant so we can move on. it's only doubly ironic that this came from a team in a company with minimal academic ties showing how toxic that entire system is. I only feel pity for the graduate students still trying to crystallize proteins in this day and age.
They say on their own press-release page that side-chains are a future research problem, and nothing about their method description makes me believe they've innovated on all-atom modeling. This software seems able to generate good models of protein backbones; these kinds of models certainly have uses, but a backbone model is not enough for drug design.
This is certainly an advancement, but you're exaggerating the scope of the accomplishment.
" I only feel pity for the graduate students still trying to crystallize proteins in this day and age."
Nothing about this changes the fact that protein crystallography is a gold-standard method for determining a protein structure. CryoEM has made it possible to obtain good structures for classes of proteins we could never achieve before, and it's certainly interesting if we can run a computer for a few days to get a 1Å ab initio model for a protein sequence, but we could already do that for a large class of proteins with homology modeling. These predicted structures still aren't generally that useful for drug design, where tiny details of molecular interactions matter.
To put it in perspective: protein energetics are measured on the scale of tens of kcal / mol. Protein-drug interactions are measured in fractions of a kcal. A single hydrogen bond or cation-pi interaction or displaced water molecule can make the difference between a drug candidate and an abandoned lead. Tiny changes in backbone position make the difference between a good structure and a bad one. Alphafold isn't doing that kind of modeling.
I understand the self preservation instincts that kick in when there's a suggestion that the entire field has been in a dark age for a while, but I hope you can see that there might be something fundamentally wrong with how research is done in academia and that is to blame for why this didn't happen sooner, and why it's so hard for many to embrace it.
Regarding your comments on the inapplicability of this current solution for docking, I'm sure that's the next project they're taking up, and let's see where that goes.
This is exactly the same type of progression that happened with Go, where when their software bet a professional player everyone's like "yeah but I bet he wasn't that good". A few years later and Lee Sedol just decided to retire. I am interested to see what happens to that entire academic field in a similar vein, though my interests are more in knowing how science can advance from more people thinking this way.
Yes it does. Protein crystallography is/was the gold-standard. Once this result is verified and accepted by the scientific community as a whole, that changes.
I think what basically happened is a very influential group of scientists mainly in Cambridge around the 50s and 60s convinced everytbody that reductionist molecular biology would be able to crystallize proteins and "understand precisely how they function" by inspecting the structures carefully enough.
I learned, after reading all those breathless papers about individual structures and how they explain the function of protein is that in the vast majority of cases, they don't have enough data to speculate responsibility about the behavior of proteins and how they implement their functions. There are definiteyl cases of where an elucidated structure immediately led to an improved understanding of function:
"It has not escaped our notice (12) that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material."
but most papers about how cytochrome "works" aren't really illuminating at all.
Yes, this describes entire history of AI including several boom-bust cycles. In particular the 80's come to mind. Yes the practitioners think that there's no technical barriers stopping them from eating the world, but that's exactly what people thought about other so-called revolutionary advances.
Although to be pedantic, "expert systems" is the technology behind AI boom of the 80's. At the time people were saying expert systems can't be as good as existing algorithms (including what we would now call "machine learning" techniques), then suddenly the expert systems were better and there was rampant speculation real AI was around the corner. Then they plateaued.
We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.
We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful.
I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.
Or when you drive your car, think about the lights-out factory that built-it, using robotics technologies developed in the 80's and 90's, and the freeways which largely operate without choke points again due to expert system models used by city planners.
These advances were just as revolutionary before, and people were just as excited about AI technologies eating the world. Still, it largely didn't happen. To continue the example of robotics, we don't have an equivalent of the Jetson's home robot Rosey. We can make a robot assemble a $50,000 car, but we can't get it to fold the laundry.
These rapid successes you see aren't literally "any problem from any field" -- it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve. Don't expect them to have as much success on every problem they put their minds to.
No, machine learning is not trivially solving the hardest problems in every field. Not even close. In biomedicine, for example, protein folding is probably one of the easiest challenges. It's a hard problem, yes, but it's self-contained: given an amino acid sequence, predict the structure. Unlike, say, predicting the metabolism of a drug applied to a living system, which requires understanding an extremely dense network of existing metabolic pathways and their interdependencies on local cell function. There's no magic ML pixie dust that can make that hard problem go away.
Beyond that, let's notice that expert systems did indeed change how airports and freeways work: They improved the areas where they solved problems. Deployment happened.
What we're seeing now is new classes of previously unsolvable problems falling. Deployment in medicine is known to be particularly hard, but not impossible. My read on the situation is that there have been a number of ML applications in the current round that have been kinda-successful 'in vitro' and failed in deployment. That doesn't mean that all deployments will fail.
Furthermore... Neil Lawrence points out that in most cases we change the world to fit new technologies. For example, mechanized tomato pickers suck, so we develop a more machine-resistant tomato. Cars break easily on dirt roads, so we pave half the planet. ML/AI somehow flips people's expectations of how technology works, and expect the algorithms to adapt to the world. This is almost certainly wrong.
"it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve."
I'm actually not sure this is at all true. Protein folding is a long-standing grand challenge on which no current methods were working. My guess is that it was initially chosen for potential impact, and chased with more resources after some initial success.
What a take. Neural networks just took a huge bite out of protein folding and your take is: This just in, the Deep Learning boom is about to go bust! Asinine.
So in 5 years you'll see exactly zero new medicines pop up.
One question is how robust the predictions are that DeepMind produces. I would also assume that right now it can't e.g. determine protein structures in the present of other small molecules, or protein complexes. A lot of the interesting stuff lies in the interactions between molecules.
And in general in life sciences any new development will take at least a decade until it hits day to day life, likely even more. We're living with a exception to this rule right now due to the pandemic, but in general things take quite a bit of time in that space.
What an accurate model of protein folding allows us to do, is to take our big database of DNA, predict protein foldings for all of it, and then stand up a search index for this database, keying each amino-acid "row" by the "words" of its predicted protein's structural features.
We could then, with a simple search query that executes in O(log n) time, find DNA targets that produce molecules with interesting structures that might be worthy of study.
This would, for example, be a game-changer in how biopharmaceutical macromolecule-therapy R&D is conducted. Right now we have to notice that some bacterium or another produces some interesting protein, and then engineer a bioreactor to get more of that protein. With this tech, we can work backward from an entirely hyothetical, under-specified "interesting protein", to figure out what catalogued-but-unstudied DNA sequences produce never-before-catalogued proteins that fit that particular functional "shape", and therefore might do the interesting thing. Then we can either directly synthesize that same DNA, or find the organism we originally sampled it from and study it more.
To determine potential targets for drugs we have to understand what the proteins do. Having the structure is not really enough for that, it doesn't tell you the purpose of the protein (though it certainly can give you some hints).
In most cases the proteins were determined to be interesting by other experiments, and then people decided to try and solve their structure. So the structures we already solved are also biased towards the more biologically relevant proteins.
Anyone uninitiated with think the same, and thise already informed. Well, they are already informed.
> In most cases the proteins were determined to be interesting by other experiments, and then people decided to try and solve their structure.
Yes, that's what we're doing right now, because structure is not a useful predictor, because we don't have structure available in advance of studies on the protein itself. There was no point to a "functional taxonomy" of proteins, because we were never trying to predict with protein-structure as the only data available.
In a world where protein structure is "on tap" in a data warehouse, part of the game of bioinformatics will become "structural analysis" of classes of known-function proteins, to find functional sub-units that do similar things among all studied proteins, allowing searches to be conducted for other proteins that express similar functional sub-units.
If someone produces an AI that you give a sequence and it tells you what the protein does exactly, I'd be extremely impressed. I don't see that happening soon.
The specifics matter a lot here. We can often determine rough functions for subdomains by homology alone. But that really doesn't tell you the full story, it only gives you some hints on what that protein actually does.
"If someone produces an AI that you give a sequence and it tells you the protein conformation, I'd be extremely impressed".
Sure there are many more things to solve in this space; but that doesn't take away that this is an impressive achievement and does unlock quite a few things (including making more tractable the problem you just brought up). I'm excited to see what DeepMind works on now and what the new state of the world will be just five years from now.
I'm maybe overcompensating for the tech-centric population here, with some comments speculating for very near and drastic impacts from discoveries like this. Biology and life sciences are much slower, and there's always more complexity below every breakthrough. That does tend to push me towards commenting with the more skeptical and sober view here.
It'll take a while before those results can be trusted, though, right? There's probably a selection bias in the training data for proteins which are easy to crystallize, so many proteins probably aren't well represented by the training examples.
Background, Neural networks have two modes 1) training - where you learn all the model weights and 2) inference - where you run the model once on new data. Training takes takes a long time, because you're computing derivatives to implement updates rules on millions or billions of parameters based on iteratively examining massive datasets. Inference is extremely fast because you're just running matrix multiplies of those parameters on new data. And TPUs/GPUs are specially designed to compute matrix multiplies.
The article said: "We trained this system [...] over a few weeks." I searched for, but did not see them identify the inference time. I do expect inference time to be well under one second, though I'm not personally experienced with running inference on this type of network architecture.
For comparison, GPT-3 and AlphaStar have month long training times and real-time (sub-second) inference times.
https://science.sciencemag.org/content/369/6502/440.abstract
Where "a few" is around 0.1% of the known 180 million proteins. So a relative few and a whole lot.
But the catch is which proteins could we figure out by experiment, and which not. In particular membrane proteins are hard to experimentally determine. But knowing how they fold is very important for figuring out how to get things to react with or get through membranes such as cell walls. Which is an important problem for everything from understanding how viruses work to targeted delivery of drugs. We now have a way to find those structures.
Some possibilities: artificial muscles for robots, man-made blood substitutes, designer enzymes to break down plastic and other compounds. Software defined biology, where the pipeline from DNA code to actual protein can now be modeled in silico ahead of time. The biology classes of the future may be less observation of animals and more training in usage of whatever the equivalent of Autodesk for biology will be. Healthcare economics in developing nations will be changed as biochemistry itself may finally become deterministic (to some extent). Orphan drug development price would drop (and if you take into account right to try laws and ignore ethics in favor of progress, then people with rare disease may be cured en masse without bankrupting the health insurance company).
So imagine you gave an iPhone to someone in the 1800's, they wouldn't understand how most of it works, but this may be analogous to them finally figuring out some key aspects of the transistor. So it's another tool in the toolbelt and like all good tools will be used in all sorts of unpredictable ways.
Someone else I'm sure could do a lot better at explaining how important shape is to understanding the function and behavior of proteins.
Solving protein folding is huge, Nobel in chemistry scale achievement. It would be massive leap for biochemistry.
It seems that Deep Mind solved competition benchmark and made huge leap, but it's just partial solution that works on limited set.
After you have solved protein folding, there is still problem of solving chemical interactions between molecules accurately. Quantum chemistry is extremely compute intensive.
Of course, there is a caveat. The static, crystallized structure is only one aspect of a protein. The dynamic behavior dissolved in H2O, at different pH, different ionic strength, with different ligands/cofactors are all also important, and not (afaik) directly addressed by this research.
They always had single digit, bizarre artifacts, where the program can't sometimes recognise the very data it was trained on with most minute differences.
Other artifact was that the most "stereotypical cases" were least reliably recognised, and they hot a lot of flak for screwed up live demos, where a radiologist put a very, very obvious tumor shot onto the scanner, and it didn't work without a half an hour of wiggling the film, and a camera.
The "bruteforce" solution may well be always, 80-85% off, but off consistently, and always. NN algo so far beat them, but fail with double digit frequencies on "artifacts" which they themselves can't do anything about.
How well it deals with the later, is what I believe will measure its real world usefullness.
This is more the case the more close to bruteforce you come, like encryption cracking. Imagine, spending years of HPC cluster time, trying to break a password, while knowing you have a single digit chance to miss the right key, in a way which would be completely impossible with with a conventional solution.
One could say the same thing about programmers automating a task, or a number of other trivial examples. I would lean towards assuming deep mind has competent model validation teams vs. not, even if data science is hard.
So your second guess is correct - one of the steps is much cheaper now, which marginally improves the entire pipeline. As a result, drugs should now arrive to the market faster.
As a side note, I am curious what happens to the field of structural biology in 10 to 15 years from now. Every research university has a large structural biology department with super expensive Xray/NRM/Cryo-EM machines, and armies of students who routinely spend 4-6 years of their PhD trying to solve a structure of a single protein. If AlphaFold works as advertised, NIH will gradually shift funding to other problems.
(It was predicted that it'd be taxi drivers, not professors, that AI got first. Ironic.)
Back in the 1990s, when I worked on structure data, I remember that at least some crystallizations were easy enough they could be done as a rotation project.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6287266/ suggests that life is now a lot easier than the 1990s. Quoting the abstract:
> Macromolecular crystallography evolved enormously from the pioneering days, when structures were solved by “wizards” performing all complicated procedures almost by hand. In the current situation crystal structures of large systems can be often solved very effectively by various powerful automatic programs in days or hours, or even minutes. Such progress is to a large extent coupled to the advances in many other fields, such as genetic engineering, computer technology, availability of synchrotron beam lines and many other techniques, creating the highly interdisciplinary science of macromolecular crystallography. Due to this unprecedented success crystallography is often treated as one of the analytical methods and practiced by researchers interested in structures of macromolecules, but not highly competent in the procedures involved in the process of structure determination.
Certainly some proteins are extremely hard to crystallize, and the new single-atom EM work will help a lot. But are there really "armies of students who routinely spend 4-6 years of their PhD trying to solve a structure of a single protein" these days?
I honestly don't know. I'm sure some do. But if so, that army is pretty small compared to the vast numbers who more routinely use crystallography.
Anyway that anecdote is pretty much the entire sum of my protein crystallography knowledge, but perhaps it explains how your experience and GP's statement can both be true?
https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp...
This is useful because for example chemical simulation tools don't run on DNA code, but on atomic models. And also to produce "images" of the molecules (images between quotes because most proteins are too small to interact with reasonable photons, and no interaction with photons means you can't see them in any way)
DNA has other parts that are really important but we don't understand at all yet, where this doesn't help at all. This applies to sections of DNA sent to ribosomes, to produce actual molecules. Besides that, there are pieces of DNA that "index" the DNA, pointers (from one gene to another), triggers (that for instance start production of an enzyme based on some external influence, like detection of a marker molecule) and export markers (that tell you what to do once the protein is produced, for example, mark a protein to be removed from the cell, incorporated into the cell membrane, or for instance used inside the cell nucleus, and there's also one that essentially says "at this point stop producing a protein and instead couple the rest of the DNA code to the end of the protein you just made").
Structure determination is what allows you to see the purpose/effect of the sequence that the DNA encoded.
It's true that in nature there are many steps between DNA and proteins (this list doesn't even include the steps that mediate the translation, ie. start it, stop it, slow it down, ...), but the structure of a protein is fully determined by the DNA code.
Protein folding is about you start from the DNA code that is fed into the ribosome ignoring all the meta information, and come up with an atomic model (VERY long list like "H atom at 3.27,2.17,12.18, C atom at 2.87, 2.19, 12.33, ..."). Now there's a million niceties we've discovered to make this problem simpler and nicer looking, but that's what it boils down to.
But anyway your answer does not contradict my statement. What you say belongs to the basics of molecular biology, but does not justify that DNA should be considered when determining the structure of proteins. In practice, the amino acid sequence is always already present.
Also, there is no need to passive-agressively highlight your credentials. I already researched them before replying.
> that we now have better tools to actually see the meaning/effect of a given DNA sequence
Note that the "meaning/effect" of a DNA segment encoding a protein is known and unrelated to the protein folding process. The protein gets its conformation after the translation process.
The "meaning" of a DNA segment is not to encode a protein. The "meaning" is to describe a mechanism in the host organism (by way of encoding a protein). That is a complex process which involves gene expression AND protein folding.
For example would you say that the "meaning" of some Java code is to generate bytecode? Of course not, the "meaning" is to run some algorithm on the computer that executes it
A protein is actually a linear sequence of amino acids, but in a cell this sequence has a three-dimensional arrangement like a clew of thread. The arrangement is not random, but dependent on the specific composition of the sequence (i.e. selection and order of amino acids) and some other factors. To understand the function of a protein, we need to know this three-dimensional arrangement (i.e. structure). Up to now the structure determination process was mostly manual, complex, time-consuming (several months up to more than a year) and error prone. If structure determination by DNN is reliable, this is a big win for life science. There are still a lot of problems open: e.g. the structure is not constant over time but there are "moving parts" in the structure which are important for its function.
Protein conformation prediction is essential when engineering new small-molecule drug compounds that must 'dock' with the specific proteins that regulate disease. Knowing how to create a protein with the precise shape to become biologically active has soaked up a lot of R&D funding toward pie-in-the-sky techniques that promise to advance that agenda (like quantum or DNA computing).
If this method works as DeepMind says, it will immediately be adopted by every pharma to assess and tweak the shape of candidate proteins.
we offered the service for free to Genentech since I used to work there and knew they could probably use it to get some good publications.
We didn't open source the distributed computing framework, but the underlying technology (Folding@Home) is based on gromacs, which is open source. It's the scale at which it ran, and the processing pipeline for filtering the results that had the real value.
It obviously wasn't scoring 90+ in CASP.
Actually, after reading your linked blog post, it's pretty obvious why they weren't exactly chomping on the bit:
"To gain insights into the receptor’s dynamics, Kai performed detailed molecular simulations using hundreds of millions of core hours on Google’s infrastructure, generating hundreds of terabytes of valuable molecular dynamics data."
Hmm, yes, hundreds of millions of hours of cpu time, hundreds of terabytes of data, who says no to that? It doesn't even seem seem to attack generalized protein folding in general. It really seems like the plan was, "let's attack this problem with a Google-sized firehose" rather than created a fundamentally different algorithm that had game-changing results.
Comparing your system to AlphaFold seems like your really bending the truth here.
You can see another paper we published where attacking the problem with a firehose helped unlock a long-standing problem: https://pubmed.ncbi.nlm.nih.gov/24265211/ in this case, showed that bond angles need to be 'free' to move rather than fully constrained,to build the most accurate models. This paper is also heavily cited amongst protein modellers.
It is correct that the MD simulations don't directly work for CASP- in a sense, the results they produce directly disagree with CASP's mental model of protein structure and function.
I'm willing to bet it will be staggeringly different than what most people are expecting.
Can deep learning find the cracks in P vs NP?
Perhaps making clever guesses at prime factors because it learned some weird structural fact that has eluded mathematicians.
If we break crypto, there goes the modern world. Banks, bitcoin, privacy, Internet, the whole shebang.
(I obviously am not an expert in computational complexity and hope that some domain experts can chime in and assuage my fears.)
No. It really is just heuristic building. A core problem with using ML in this sort of use case is that it is often brittle. Once it gets outside of the context it was trained in it may or may not be able to generalize it's training to new contexts. We may have difficulty knowing when it is very wrong.
I think ML in research science could be viewed as a very good intuitive oracle. Even if they are right 95% of the time, you have to do this work prove the long way every time because that 5% matters. The real utility is in "scanning the field" to better focus research on things likely to bear fruit.
"[Modern heuristic and approximation algorithms] can find solutions for extremely large problems (millions of cities) within a reasonable time which are with a high probability just 2–3% away from the optimal solution." [0]
When close enough is enough, NP problems can often be solved in P time, and I suspect this is one of those cases. For crypto however, close enough is not enough.
[0] https://en.wikipedia.org/wiki/Travelling_salesman_problem#He...
Yeah, at least some variations of it are NP-hard. SAT is THE NP-complete problem, but there are some really good SAT solvers around. This basically means: They have a solution that mostly does very well on most instances. But because (probably) P != NP, you will never have a polynomial time algorithm for this.
But - you don't know that you are 1% from the solution... even if you are pretty confident that you are. It's quite possible (unlikely) that you are way off the optimal, but if you have a decent solution that's ok.
This “uses approximately 128 TPUv3 cores (roughly equivalent to ~100-200 GPUs) run over a few weeks”. That is a moderate amount of hardware for this kind of work, so it seems they have a more efficient algorithm.
Also, this algorithm doesn’t solve protein folding in the mathematical sense; it ‘just’ produces good approximations.
I'd love to hear more. Specifically, what do you think that computational biology can do that you think Art doesn't understand or credit?
https://github.com/deepmind/deepmind-research/issues/18
[1]: https://predictioncenter.org/casp14/doc/CASP14_Abstracts.pdf...
This model requires a collection of wild-type proteins in an accurate MSA. Producing an accurate MSA is hard even if you have many homologs.
They require protein homologs which means they can "only" do this for wild-type proteins. This work is useless with mutant and synthetic proteins. This is a big advancement that will assist crystallographers and NMR structural biologists with difficult wild-type proteins, but it doesn't "solve protein folding" by any stretch of the imagination.
To assess co-evolutionary couplings the amount of homologs in the MSA is not as important as the number of effective sequences (i.e. sequence depth and diversity) in it.
> They require protein homologs which means they can "only" do this for wild-type proteins.
Even remote homologs work, as shown by the widespread use of HHM-based methods in the prediction pipelines.
> This work is useless with mutant and synthetic proteins.
Unless you generate a flurry of data with them using deep mutational scanning for example. As long as correlated mutations are present in the MSA the technique should work as expected no matter where the protein sequences originated.
And another one about its application in structure prediction: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7295002/
> The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”
> But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”
Did they also restore the opening and endings, and use the latest Stockfish?
This last season, Stockfish won by using NNUE, a neural network based evaluation function.
https://en.wikipedia.org/wiki/Top_Chess_Engine_Championship#...
Now I'm waiting for the equivalent news about SETI@Home ;-)
Are you a domain expert? Because:
> According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods.
What would be informatively useful would be to know how much accuracy is needed on average for drug engineers, I'd say that 99% is more likely to be the minimum to make solid inferences
I might be missing something here, but isn't "experimental methods" just shorthand for "our best knowledge of a protein's structure, obtained via NMR or X-ray crystallography"? In that case, I'm not sure what "solving" protein folding even means - literally zero mean error? We can't know/solve anything beyond our best knowledge, that's tautological.
> What would be informatively useful would be to know how much accuracy is needed on average for drug engineers.
Yeah that would be interesting, but:
> I'd say that 99% is more likely to be the minimum to make solid inferences
...what are you basing this on?
"We can't know/solve anything beyond our best knowledge, that's tautological." yes it is indeed tautological if you assume that experimental methods can't get better then guess what? It follows that they can't get better!
"what are you basing this on?" on nothing solid, that's why I say it would be interesting. 99% is a non negligible error rate given that proteins have generally a not very high atom count and they the protein will be produced an enormous amount of time, then the 1% error progagate and can a priori easily break the system. But this guess is not solid as I'm not an expert. 99% accuracy for simple (low atom count) proteins is a sensitive error and could be negligible for very high atom count proteins.
That's not clear at all, because perfect measurement doesn't exist. I agree that improving is always a worthy goal, but clearly we don't need 100% accuracy to consider something "solved" for the purposes of science. Also, "3D structure" of a protein is not a fixed truth, the parts are in motion all the time and may even have multiple semi-stable conformations. Rather than focusing on X,Y,Z perfection, I would imagine getting the angles between bonds, or the general topological conformation right would be more valuable.
> if you assume that experimental methods can't get better ...
I'm saying that if your definition for "solved" is "perfect knowledge", then we might as well not discuss whether method X or Y solves the problem, because they obviously do not.
The more I think about it, the more I think we should just drop the whole debate over the word "solved". Clearly different experiments and different proteins will have different requirements which may or may not be met by this or by other techniques - I agree that I would be interested to hear an expert weigh in on those requirements.
Oh, and to be a true parallel example, in this case the remaining 10% of cancers might not even be cancers, as experimental accuracy of protein structures is only ~90% accurate, the model could very well be more accurate than our current ability to experimentally detect protein structure.
Scores of 30-45 for 15 years. Now scores of 87-92.
This isn't a minor improvement, it's a leap forward.
There's no reason to believe the list will contain all solutions, however.
>a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods
So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure.
The threshold for "real" in particle physics is +5 sigma. Which takes a lot of data.
These proteins are so massive that we often use Daltons [1] as an averaged measure of molecular weight.
Conceptually one of the most promising applications of quantum computing is theoretical chemistry, and we are only now starting to make progress in this avenue [2]. I anticipate it would require quantum computing to explicitly optimise large folded proteins.
1. https://en.m.wikipedia.org/wiki/Dalton_(unit) 2. https://arxiv.org/abs/2004.04174
Think of it like satellite imagery of a tree: A score of zero would be a single green-ish pixel, while a score of 100 would show each leaf within the range it naturally moves in due to wind etc. (proteins tend to wiggle quite a bit under natural conditions, as well)
In this case, the model is predicting values of multiple structures, but patterns could still theoretically be found which allow for predictions beyond the accuracy of a single measurement.
Let’s say you have 100000 proteins in the training set. Now remove #1 and train on 99999, and then check that it still predicts the same protein result for #1 as the experimental result.
Or remove from training whole sets of proteins by particular teams to find systematic errors made by teams?
While this is an accomplishment, nobody is going to be confusing these models for structures produced experimentally. The CASP metric is for backbone atoms. To have a useful model of protein structure, you really need to have the positions of the protein side-chain atoms modeled correctly. Experimental methods will do that, but this method, as I understand it, does not.
Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works. If so, can we somehow extract that truth, or is it truly a black box that does the thing we want?
I'm reminded of a sci-fi book I read long ago in which humans are discussing the fact that the science they are utilizing is beyond the scope of a human mind to comprehend- only the AIs can intuitively deal with 12-dimensional manifolds (or something to that extent). Maybe we've reached the doorstep of that future.
So i do think the results could be more accurate than measurement.
Well I think that the results speak for themselves; ultimately the question you raise is one of semantics. ML models don't think in terms of "conceptual ideas" like humans do, these models simply perform at such a massive statistical scale that they can identify patterns far beyond any human conception. Clearly, the model embodies some verifiably reliable information about the way the world works, but this is "just" a trick of statistics not anything resembling actual "understanding" in the way the word is typically used when referring to human understanding.
What's an experimental method for protein folding and why is it so good? Are they talking about creating an actual, physical protein in a lab and observing how it folds?
Exactly. Researches purify the folded protein and then use methods such as X-ray crystallography, nuclear magnetic resonance, and cryo-electron microscopy to determine its three-dimensional atomic structure.
We understand the physics of e.g. X-ray diffraction pretty well, so we can fit pretty decent forward models for the x-ray data given a proposed structure. The hardest task here is getting a good enough guess at the structure to optimize the physical model, and it’s my impression that people use an iterative model refinement workflow. At least that’s how it’s done in condensed matter materials.
There are many sources of experimental uncertainty, like the non-ideal nature of the x-ray source and optics, and the fact that the atoms in the protein are not static but have some thermal fluctuations. so at the end of the refinement you still have some uncertainty on your model parameters (the interatomic distances for proteins I guess), but if you are careful you can calibrate these uncertainties pretty well.
This paper looks like a really good detailed discussion: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4080831/
So you "make guesses at what the phases are", the best choice is to bootstrapping these phases measured with another technique (you can introduce crystal defects that do allow you to guess at what the phases are).
Less scrupulous is to use a computer generated model, like fitting another protein "that you guess is related", then you model the electron density, take the phases of that.
In any case you take these "phase" guesses, and then apply it to your intensities, re-run the fourier transform, refine your electron densities, twiddle the location where you think the atoms, are, then repeat with your new model. This process repeats until you converge on a structure that you're happy with.
Now alarm bells should be screaming in your head right now: Yes, it's entirely possible to converge on a wrong structure, especially if you're a young up-and-comer professor seeking tenure that has no ethical problems with "suggesting" their grad students to sleep in the lab and work 100 hour weeks and willing to do slipshod work to get you tenure: https://www.sciencedirect.com/science/article/pii/S002228360...
I guess my question is, how do you know if you’ve converged on the right structure or not? Is there a different experiment you could do?
This bodes extremely well for the future of computational biology, I'm very excited thinking about the prospects. If we know how a protein folds, we know its shape, meaning we know which shaped/charged molecules are needed to act as suppressors/enhancers of those proteins.
Could this lead to a virtuous cycle where AlphaFold is used generate a ton of random sequences where it has low confidence, those are then screened for ease of synthesis, measured and the results used to improve the model?
Edit: nevermind, according to another comment[0] there are still plenty of real proteins without experimental data left to explore.
[0] https://news.ycombinator.com/item?id=25255601
It can verify how much it minimizes the potential energy, which may not always line up with how it would fold in the real world but is a strong indicator.
Does seem like the contest structure could include quite a bit of risk for hiding the effect of overfitting ... I wonder if there is anything inherent about the problem that reduces that risk ...?
The reason why the top score in one year, can be lower than in the previous year, is that the test (the 100 structures to guess) is always new and different, so it can end up being 'harder' than the year before. Luck will also play a small role.
Another explanation for a reduction in the top score would be, that previous winners are not re-submitted unchanged. For instance AlphaFold v1 seems to not have been submitted to the latest competition.
Is it really possible to select 100 new structures which together are likely to represent a meaningful increase in the sample generalization versus the prior years test set ...?
Using 1% of those (presumably from the more-often-reproduced subset) for this challenge seems reasonable? Note that the structures have to remain secret up until the challenge, and presumably all those teams uncovering the structures don't want to have to wait up to 2 years every time to actually make their results public.
I suppose it will take a few more years of repetition for the challenge to confirm that the problem has been been solved -- but I wonder if a new version of the contest is going to be needed as well? Maybe the model accuracy is now high enough to invert the contest to a form where models generate predictions for randomly selected unknown samples -- and experimental teams are then expected to make observations for those particular sequences over the next two years as part of their otherwise research agenda selected experimental workload?
But yeah, compared to other fields, the size of training/test sets is sometimes pretty small in ML for life sciences.
Nonetheless, I'd like to hear more from specialists outside the context of a marketing blog post before I fully buy into a claim of a solution.
There's also a rabbit hole about what 'solution' actually means. Is the performance sufficient for any protein folding prediction application that might arise in the future?
The reverse folding problem lets you specify a structure and then make a genetic sequence to produce it. For example you could look at this virus to see how it infects its host, then design a custom protein to act as an anti-body stopping it, which is a capability we don't currently have.
Forward folding is certainly useful, but reverse folding would be revolutionary.
Sure, "known", but it's on the order of 20^10000. It won't fit in the entire visible volume of the universe.
(Am not a structural biologist.)
Added: 2019 article on de novo design: https://www.nature.com/articles/d41586-019-02251-x Not to say that better prediction won't also make design easier -- of course I expect it will.
That is not an outstanding question. The test on which DeepMind scored high marks is a test of how well the algorithm folds novel proteins -- proteins whose ground-truth structure has not yet been published.
> the outstanding question ... is going to be how good is the confidence metric at telling the user to trust or not trust the results.
There is generally a threshold, less than X, not the class, equal or more, is the class. Then you run the network with the same threshold on a known data set and compute a confusion matrix, which tells you about the error, I don't even want to know what a confusion matrix analogue for 3D geometry would look like but I'm sure they have something.
This is literally the process that one does in taking part of the this. And the error rate (specifically the lack of errors) is what is everybody is talking about. 90 is just as accurate as we can get with experimental measurement. It's likely at this point the source of error is in the data set (we can only train on data we experimentally measure and these are not perfect measurements). It's also possible, at this point, the model generalized so well that when it deviates from experimental measurements it's actually correct and the experimental value was the one that was wrong.
So no, the outstanding question is not "is going to be how good is the confidence metric at telling the user to trust or not trust the results.". Nobody is going to be looking confidence values when it model is giving an output, they are going to be looking at the overall error rate across a broad spectrum of proteins to get a sense of it's accuracy.
in reality, that's not how it works at all. The energy functions we have are crappy and require too much sampling before we can find the lowest energy configuration. And more importantly, it doesn't look like proteins typically fold to their lowest energy configuration (with the exception of some small fast two state folders), but rather explore a kinetically accessible region around there (or even somewhere else entirely, if the energy cost to transition is too high).
Methods like AF depend heavily on large amount of information correlation from evolutionary data, which has historically been of the highest value for making decisions about protein structure.
The article did claim:
> According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods.
So perhaps their score of 87 GDT is pretty significant. But “competitive with” is not the same as “always in agreement with”, as you point out. Could be the failure modes are problematic.