a purely random sequence of letters has greater Shannon information than a string of words that make up a sentence, because the words and the sentence have some higher-order patterns in them (like statistics of letters that typically follow each other, such as a “u” following a “q”), which can be used to compress the message. A random sequence has no such patterns and thus cannot be compressed.
Thinking in those terms naturally leads to the kind of “counting arguments” that Bengio makes. These seem to take each gene as a bit of information, and ask whether there are enough such bits to specify all the bits in the brain, usually taken as the number of connections. Obviously the answer is there are not enough such bits. (There aren’t even enough bits if you take individual bases of the genome as your units of information).
For me the most enlightening summary regarding information encoding in genomes is the following quote from the article:
Being amazed that you can make a human with only 20,000 genes is thus like being amazed that Shakespeare could write all those plays with only 26 letters. It’s totally missing where the actual, meaningful information is and how it is decoded.
Can you answer the question of where the meaningful information is in the genome? In Shakespeare or Dr. Seuss it’s in the combinatorial power of language to express concepts. With just 200 words Seuss could write stories like the Cat in the Hat. Is genetics similar?
My interpretation is that just as with words that in a different order may mean different things, genes expressing in a different order or at different times could have radically different effects.
The simplified cartoon image that is associated with DNA, where phenotypes are directly connecting to specific genes - i.e. this mutation makes you smart, or that mutation causes cancer - is not at all how biology works.
I can do a lot better than that with just 1's and 0's.
The sentence makes exactly the error that it accuses others of making, the '20,000 genes' is not the encoding, that's the codons of which there are 64 some of which encode the same amino acid. The 20,000 really is like a single play of Shakespeare.
Genes are regions on the genome, now whether or not are translated by 3 (as codons) does not seem to be relevant.
The quote is a criticism of a common way of thinking in biology, where many (if not most) practitioners think in terms "one gene does one thing" (or perhaps few things at most).
When one follows that mentality it becomes hard to explain how just 20K genes could make a living organism.
The analogy I often use is houses. Think of all the components used to build a complete house. One can use the same bricks, wood, tile, etc, to build many different houses. But the location of these components in the house is not enough, the order in which they are added is critical.
The genome must encode all the components used to make an organism, as well as the instructions on how and when to use these components.
The attempt to make analogies with biology from writing and computing will always fail. Biology is messy, lots of repeated use of the same data, circular references and other things that we find hard to interpret. It's as if starting on the second letter of a book instead of the first will give you a different story.
The 'one gene, one thing' or just a few things at most meme died long ago for all biology practitioners that are with the times, the problem is more with IT people attempting to understand biology trying to make computing analogies to make sense of something that evolved rather than that it was designed.
very interesting article, and intriguing conclusion towards the end, make me feel like knowing more
> You don’t need to specify where every synapse is to get the machine to work right. You just need to specify roughly the numbers of different cell types, their relative positions, the other types of neurons they tend to connect to, etc. The job of building the brain is accomplished statistically, and, crucially, probabilistically. This is why there is lots of variation in brain structure and function even between monozygotic twins and why intrinsic developmental variation is such a crucial (and overlooked) source of differences in people’s psychology and behaviour (the subject of my book Innate).
I agree with the author re: the relative importance of coding vs. non-coding regions, but disagree with the section on Shannon information.
Shannon information provides a trivial upper bound on Kolmogorov complexity: imagine a tiny program with a huge data segment, which just copies the contents of the data segment to standard output. In the case of the human genome, this value is on the order of 6 billion bits. The key takeaway is that a much larger fraction of these bits than one might expect from the phrase "20000 genes" are potentially relevant, not that Shannon information somehow underestimates complexity.
Perhaps along these lines, when considering compression, there is a tradeoff of sorts between the size of the compressed result and the size of the decompressor. If I get to choose the decompressor for a file, the answer to the question of how small I can make the compressed file is "all the way" (i.e. zero bits).
It seems likely that the amount of information in the DNA "decompressor" (in mammals, the mother and her environment) dwarfs the amount actually present in the DNA itself.
It is indeed proper to use that sum as a metric. It's not necessarily the case that the decompressor is encoded in the genome, though. It may very well be that crucial biological information is passed from mother to child in other ways, and it could turn out to be a lot of information.
It's a great article, loaded with information, and very insightful.
But I'm left disappointed that there's no actual estimate of maximum information content. Just this:
> *[Note from Tony Zador: The length of the genome, which in humans is around 3 billion letters, represents an upper bound on the Kolmogorov complexity. Only a fraction of that carries functional information, however, so the upper KC value may be quite a bit lower than that].
That must be 3 x 10^9 base pairs, in the haploid genome. Wikipedia tells me that's 800MB.[0] However, only 1%-2% of that codes for proteins.[1] Which is basically what Zador's note says.
So what's possible for the rest? While 800MB doesn't seem that much, maybe it's enough for basic "firmware" and "software". Including instinctual behaviors.
But even all that is clearly not enough for a full blueprint and assembly instructions. That, as TFA explains, must flow from interactions during development.
An important thing to remember/ consider about DNA, is that in its somatic state it is 3 dimensional, so while its informational content may be ~3 billion letters, those letters can and are part of a dynamical system when encoded in DNA. I'm not sure if what this means in regards to its total informational content.
thinking of DNA as some kind of abstract turing tape is wildly misleading, here's a graphic that summarizes some of the ways DNA is conformed/stored/managed in the cell.
each of these have huge implications for what can be done with the DNA
there's tons more information that's attached to the strands of DNA, too, like methylation or promoters, etc. that the simple ATGC doesn't cover at all
if we want to stretch the turing tape metaphor, it's more like marking the symbols as various kinds of magnets on a string, then putting a bunch of them in a sack and shaking them up. does that change the amount of information?
if you're interested, you really can just pick up an intro bio textbook, skim it, then do an intro genetics text (and skip the classical genetics crap).
looking at the first step in that graphic, for example, a histone is made of 4 distinct types of parts, and in humans each of those has a dozen variants, and the final assembled histone can additionally be tagged with other pieces. and the physical interaction between the histone and the DNA is influenced by the local sequence.
I don't know how I would quantify that, but there's a lot going on.
Information is a log of the possibilities. Adding epigenetics and such will add to the number of bits, not multiply them. If I give you another decimal digit to stick on to a number, it may give you another factor of 10 to play with on its sum, but it'll still be just one more digit, not a factor of magnitude more digits.
Layman here. I agree, viewing it as a "turing tape" highly misleading - however, I believe the general idea of viewing a cell as a sort of microprocessor with the DNA being the "firmware" is still useful - at least it gave me a better understanding of what kind of information is stored in the DNA at all.
Of course, if we stay with that analogy, DNA would not be a "series of instructions" but would more resemble a huge, unordered set of "if/then" rules or "trigger/action" pairs - the "actions" being protein-coding sequences and the "triggers" being their regulatory sequences.
The actual behavior of the cell is then determined by the interation of those rules and the effects of the coded proteins.
"so while its informational content may be ~3 billion letters, those letters can and are part of a dynamical system when encoded in DNA. I'm not sure if what this means in regards to its total informational content."
It means nothing, in terms of information content.
This is a common misunderstanding, which the author of the piece appears to share, but there's no getting "more" information out of something by talking about its interactions. Suppose we write a computer language where every byte is a meaningful opcode, and all possible strings of bytes are valid and do something. If we consider the set of 8-byte programs, there are 2^64 of them. It takes 64 bits to encode which one you are choosing from that set, and there are 2^64 options. The fact the they are "programs" and thus that many of them will "interact" does not mean that you suddenly have more than 64 bits of information. You still have just 2^64 options. None have appeared from the ether by the virtue of some of them "interacting". In that set of programs will be everything from programs that immediately terminate, to infinite loops, to programs that use staggering amounts of RAM and CPU before terminating, but there's just 2^64 of them.
Information is really just a count of possibilities. (Weighted, technically, but that's not a terribly helpful detail at the moment.)
Since the question is explicitly does DNA itself encode innate knowledge, it is relevant to discuss the contents of the DNA in this strict counting argument. (Add the epigenetics and whatever else you like; all it does is add more bits proportionally to the possibilities they add. But they don't make more bits than that due to "interactions".)
You, yourself, are not the sum total of information in your DNA. Even at the one-cell beginning of your life that would be an oversimplification, and since then the universe has poured far, far, far more "bits of information" through you than you could possibly have integrated into yourself. You would require a great deal more information to describe than your raw genome (just as identical twins are still, you know, distinct people). But in terms of asking what can be in the DNA alone, the information theory is relevant, and citing "interactions" buys nothing. The DNA has the bits the DNA has and it doesn't get any more by "interactions".
The organism with the DNA will interact with its environment, of course, and it's essentially proved that a lot of our "innate knowledge" requires interacting with the environment to develop. In terms of understanding the whole process, that will all be critical to understand. But in terms of literally what is in the DNA, that's not particularly relevant.
(My personal belief is that a non-trivial component of the confusion is indeed that we look at the "final product" of this innate knowledge and over-attribute it to DNA. Babies naturally recognize faces... except they don't. "Face recognition" is a huge amount of bits. On the other hand, encoding some instructions that will result in embedding an eigenface [1] into the visual system and giving a hint of dopamine when it recognizes one, and leaving the baby to eventually work out what "faces" are and why they are interesting isn't that many bits at all. The end result is "face recognition" and that system would consist of a huge number of bits' worth of information, but the vast bulk would be from the environment. What was actually in the DNA is just a tiny seed crystal of information, which fits perfectly comfortably in the DNA. Especially since it wouldn't even necessarily be a "thing that matches faces" but just a weird streak through the hyperspace of possible visual inputs that happen to privilege faces but will also privilege and disfavor any number of other inputs too, which reduces the requis...
> That, as TFA explains, must flow from interactions during development.
That's true for any encoding of information. KC can only be defined relative to some computational model, and the actual number depends on the details of that model. It just so happens that there is a large family of models whose KCs are equal to within a multiplicative constant. But the actual number is never meaningful in isolation.
OK, but TFA title is "How much innate knowledge can the genome encode?".
So is the answer that we have no clue? Or that it's a meaningless question? Given that we don't know the computational model, I mean.
To me the interesting question is how much life memory could be encoded in the genome. Maybe top points from human evolution? Or nothing more than really simple basic stuff?
Leaving aside how it might get stored in the germline, and later accessed.
> So is the answer that we have no clue? Or that it's a meaningless question?
Both. The fundamental problem here is that the phrase "innate knowledge" hasn't been defined well enough to allow one to quantify it. How do you measure "innate knowledge"? In what units would you express the results?
Here is an analogy: consider movies. A movie is the result of a collaboration between many people: writers, directors, actors, editors, stunt people, special-effects folks... how would you go about measuring how much of the quality of the final product is due to each individual contribution? It's a very similar situation with the nature-vs-nurture question (which is really what this is). The sum total of what we are is a result of very complex interactions/collaborations between our genes and the environment in which they exist. Quantifying the contribution of our genes is not that far removed from quantifying the contribution of an individual on a film production team.
Answering these questions is not impossible in principle. It's just really, really hard, and at the moment we aren't anywhere close to being able to do so.
> How do you measure "innate knowledge"? In what units would you express the results?
Useful knowledge might include sucking on nipples, being afraid of snakes and spiders, being wary of fire, sexual drives after puberty, and so on.
As far as units, why not bits?
> Here is an analogy: consider movies.
I agree that going from a zygote to an adult animal is a
complicated process. There's the genome, the process of development, and a complex mixture of sensory data and actions. And yes, that doesn't do justice to the complexity.
But the question here is how much innate data. Not how substantial a role that data plays. So as a first approximation, wouldn't that be the script? Or maybe the script plus the video file?
> It's a very similar situation with the nature-vs-nurture question (which is really what this is).
I wasn't seeing it that way. There's no question that the environment plays a crucial role in development. But I don't see how any of that can create knowledge of past events. Which is what "innate knowledge" means to me.
Because bits are a unit of information, not knowledge. Not the same thing.
But even if you're really interested in information and not knowledge that's still a problem. You can easily measure the information content of the human genome because we know how it's encoded. It's much harder to measure the information content of the human brain because we don't yet have a complete grip on how it works.
> I don't see how any of that can create knowledge of past events.
That depends on what you mean by "past events". If you really mean individual one-off events, there is no known mechanism for that kind of information to get encoded in a genome. If you mean general information about the world acquired through "experience" (where having natural selection act on your population counts as "experience"), that could be encoded in a combination of information that resides in the genome and information that resides in the environment, where the structure of the brain is part of the environment (of the genome). We just don't know.
Well, it is, but it is not just a subset of information. The distinction between knowledge and information is very tricky to get a handle on. Philosophers argue about it all the time. For example, here is some information:
"I am wearing a white hat."
Is that information also knowledge? Well, it depends. It depends, for example, on whether or not I am in point of fact wearing a white hat, because if I'm not then you cannot possibly know that I am wearing a white hat despite the fact that I have just given you that information. That information may or may not correspond to reality, and one of the things that distinguishes knowledge from information is that knowledge has to correspond to reality in order to qualify as knowledge, and information need not correspond to reality ("false knowledge" is an oxymoron but "false information" is not.)
But there's more to the distinction between knowledge and information than mere correspondence to reality. If I am in point of fact not wearing a white hat, the information above does not magically transform itself into knowledge if I suddenly put on a white hat.
Thinking of the encoded genetic information developing over time as simply "training" the neural network of the brain doesn't quite capture development; proteins are manufactured and move in space as the brain is growing in space over time.
It sort of feels like training a neural network is like rewiring an adult brain which has already fully grown. Are there neural architectures which grow over time, or "age", emulating the early development of the brain?
(Not a biologist by any stretch, so the following is speculation)
As for reflexes, don't we kow how they work on an anatomical/neural level in a grown adult? From my high school biology lessions, I remember that for certain reflexes, there are nerves that directly connect "sensors" to motor neurons in the spinal cord, bypassing the brain altogether.
If you stick to the "genes as a program" metaphor, "encoding" this reflex in the genome would mean that the genome contains the correct triggers and "subroutines" (i.e. as cascades of gene expression) so that a particular group of cells will form that nerve and connect it to the correct other neurons.
the scientific community has discussed this for some time. A lot of discussions I see on HN about this start from "let's understand this from de-novo first order principles". But to make rapid progress, scientists in these fields use evidence from many different sources rather than first order principles.
One of the interesting things that we've learned over the years, painfully and repeatedly, is that fairly simple encodings of fairly simple data can repeatedly generate extremely complex visible outcomes. A great example of this is patterning in animal fur- there isn't a section that encodes a pixel map of the resulting spots or their X/Y locations. INstead a collection of genes operate in a feedback loop to generate the patterns. Turing's theory of morphogenesis has turned out to be highly prescient application of very simply theories that does a good job of explaining observed patterning.
What I've seen repeatedly over the years is that the proteins themselves aren't that interesting- all the interesting complex repeatable phenotypes come from careful regulation of the expression of proteins with feedback loops and spatial gradients. The best work on this has been done in evo-devo field on model organisms (not humans). C Elegans was an exceptionally good choice for a model organism for many reasons, not the least of which is that every instead of the organism has exactly the same cells in exactly the same location, and that appears to be completely genetically encoded (again, not like a pixel map, but as generative rules).
I've wondered in the past if humans have traded off hard-coded instincts for more cerberal cortex or more connectivity between brain regions. Culture and communication is now part of our environment and so our genes don't have to carry as much information about the environment itself; our parents/society can add it later.
Trivial sci-fi comment here - would love to see the 10,000 year endgame of this reseaech: near death, all your memories are encoded into a new genome, which becomes cloned into a new person that is you and also retains all of your life experiences. You die, and a new one of you regenerates, then picks up where you left off.
To piggyback on the sci-fi theme, some of the best sci-fi from the past few years has been Adrian Tchaikovsky's novel Children of Time and its follow-up Children of Ruin, which explore the idea of an artificial virus that interacts with genomes and allows for a sort of genetic memory.
How much confidence do we have that the literal base pair encoding of DNA encodes the entirety (or even the majority) of the information needed to construct an individual, anyway? As the author mentions, you have things like histone dynamics as well as other potentially relevant variations in the many organelles that make up the cell. Eggs in particular are chock full of proteins that are directly passed down from the parent that could have important implications for gene expression frequency. These effects would either have to be self-propagating in some way via further protein synthesis or their effect would be primarily early-developmental, but they could still have large effects on grown individual.
As evidence for this we know there are all sorts of heritable epigenetic effects. It would also partially explain the partial failure of GWAS to pinpoint all an individual's features as was previously hoped before the human genome project completed.
It seems like there's a process called Somatic Cell Nuclear Transfer that might make possible to evaluate aspects of this, but it doesn't appear to have been performed sufficiently many times with different individuals to draw major conclusions.
Interesting article, but I think the "information" metaphor sort of misses the point. DNA is not a store of information but rather the result of computation to solve a hard problem (survival/reproduction in a given environment). Think about it this way: it is really hard to crack a SHA256 hash -- but the information contained in the solution is very small. DNA is the same thing. Using information is not really the appropriate metaphor to describe DNA.
Geneticists and biologists have a concept called conservation [1] that sort of gets at this point. If we think of an environment as an encryption algorithm, then certain bits in the hashes ("genomes") that provide suitable solutions tend to be stable over time.
I think the article sort of misses the real effect of genome on "innate" knowledge.
There's a new soon to be hot (I predict) topic in AI/ML called inductive bias. It turns out that the way your neural net is structured, i.e. the overall shape and the way the neurons are connected, can strongly bias which information the neural net tends to learn, and how quickly. To the point that there was a paper (can't find it now) where IIRC the authors were able to develop an algorithm which creates neural nets which can solve basic problems with little to no training. Point being, you don't need to encode, say, the image of a nipple and the action of suckling in DNA to create innate suckling knowledge - you simply need to structure the brain such that it is biased to perform and learn certain sequences of behaviors.
I think this may be the origin of much if not all innate knowledge and I believe that research in AI/ML will lead to better understanding of human (and animal) cognition.
eh... Information theory wise, at best you can say you've compressed the knowledge to the action to the steps required to construct the geometry. But you've still encoded it on some level. So there is an information-theoretical limit that is still worth thinking about. I think they kind of miss the boat on their discussion of Shannon. You've encoded the structure, and the actual behavior then arises from actually running the network. I think they got a little confused to be honest.
And indeed, what people will find is exactly as you say: The DNA encodes how to grow the network such that it will respond with the evolutionary advantageous instinct.
I don't know if I'd quite say that no one is looking into that. Obviously we know that there are NN that use deep networks. But also we can see that we can increase the computation power by removing nodes and edges that don't contribute to the correct answer. In evolution, this is a literal cost savings in that you need to feed less neurons.
I understand your point, and I'm having trouble articulating this thought, but I think with inductive bias you can structure your net such that it is primed to naturally learn certain behaviors that are more complex than the Shannon entropy required in the encoding DNA. The minimal structural information combined with environmental interaction can produce far richer information content, past the limit of the original DNA, but from a design perspective your actual code is very compact because you gain the rest of the knowledge required for innate behaviors naturally from the environment.
It's interesting to think about all this. Everything DNA encodes is responses to the environment.
On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels.
On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself.
On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.
As the article points out, Shannon information really is the wrong benchmark. Kolmogorov complexity and the busy beaver sequence should demonstrate that there's very little reason to argue for any sort of genetic bottleneck on gene count grounds.
"So there is an information-theoretical limit that is still worth thinking about"
So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe.
I think there is perhaps a dozen reasons why this is an incorrect comparison.
If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms...
Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt.
So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.
Could, say, certain groups of people be more predisposed to easily learning, for example, complex math, or more disposed to certain types of behaviors than other groups of people?
Inductive biases are not a new topic in ML. Also your examples have little to do with inductive biases as inductive biases are about generalization on out-of-sample data through human based restriction of the model hypothesis space.
> Point being, you don't need to encode, say, the image of a nipple and the action of suckling in DNA to create innate suckling knowledge - you simply need to structure the brain such that it is biased to perform and learn certain sequences of behaviors.
How do you think "structuring the brain such that it performs suckling" differs from "encoding the action of suckling"?
No, not quite. This is a little hard to describe outside of ML terminology and it's pretty new, but I'll try.
The structure I'm describing is at a different scale from the structures of the individual neurons. Actually, come to think of it, baby motion may be an even better example:
When a baby is born, it (possibly) has no understanding of how to move it's arms and legs. But the newborn brain is structured in such a way as to encourage movement of limbs, such that the movement required for learning becomes a high probability event°. In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.
If you connect neurons in a particular large scale pattern, and wire some up to ears, some to eyes, some to touch receptors, two things may happen:
1. Your brain is more likely to perform the initial actions required to learn
2. Your brain may more easily learn certain actions
The second thing is inductive bias, and I think it will be extremely powerful in the world of ML once we figure out how to harness it. We know that neural nets are hugely wasteful now, there have been papers published on neural network optimizers which can shrink nets by some 80% while maintaining accuracy. Inductive bias is probably such an optimization.
> In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.
That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary.
What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.
No, it's not compression at rest. What GP is talking about is the encoding of recipes for acquiring skills rather than encoding the skill itself. The end result is that the skill is acquired, but the skill is not latent in the genome, so your .gz analogy doesn't hold. It's a compression that has access to a vast existing dictionary and makes better use of it then some other.
The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.
> The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome.
Why not?
Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally.
So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song.
This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.
That's just one, particularly limited, way of compressing information by using the environment. The information is merely a fixed point of the "function" that is encoded. But what about all the other sources of signal that don't originate from the organism's own genome? If there are other ways to compress the information even further, should we not expect Nature to use all of them?
Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.
(Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)
> Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.
Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds.
The hypothetical would be, as a thought experiment, what if Mars was colonized by a bunch of newborn human babies? Leaving aside the question of how they'd survive, would they develop languages like the ones we have today within a few generations? I think the very idea is ridiculous, but you may disagree.
Nicaraguan sign language was not developed in a vacuum, it was developed by children living in a completely normal human society with the full use of all their senses but one. Far from "nothing".
It was developed by children living in a completely normal human society with zero exposure to language. So yes, ignoring the question of how they survive, a community of children will develop its own language without the need for external input.
Why do you think differently? "I think the very idea is ridiculous" isn't much of an argument.
> It was developed by children living in a completely normal human society with zero exposure to language.
If they are living in a normal society, they are exposed to language nearly continuously, from visible printed language to visible body language to the visual evidence of language being spoken all around them.
I personally had not seen the eyeless gene information. While I don't find the outcome surprising, I do find the images breathtaking. It's also very interesting that including the similar (but not identical) gene from the mouse has the same effect of inducing eye growth.
The brain is the computer, not the genome, no doubt about that. But the genome was not designed but evolved, as a consequence it contains many strange and overly complex mechanisms that just settled somewhere in a minimum in some highly complex landscape. It would be better to ask what it would take to store everything we learn and act upon it, in the most simple way.
If we are looking for the innate information, we should be asking the question: In how much information can we store the computer, and all machinery that builds it from basic components (molecules basically). It does become easy then to imagine that there is some kind of suckling response, a computer also starts doing things when it is powered up.
Given two genes, A and B, you could list out the information encoded by them as “A”,”B” and “AB”. Which is basically the Shannon estimate on complexity, right?
However in reality, the gene dose makes a difference - so it’s more like the genome (+genome architecture) encodes for capacity to produce different amounts of A and B. So, with high A expression you will see things like “AAAAB”,”AAAAAA” etc.
The point is that rather than encoding for different functions, the genome encodes for the diversity of different message queues - and it is the length of these queues and rate of message firing that encodes (as much as you can in a probabilistic sense) for our cell programming. The messages could be “build a transcription factor” or “put a receptor on the surface”, or “make a crank that increases the capacity of these queues”.
68 comments
[ 3.8 ms ] story [ 112 ms ] threadThinking in those terms naturally leads to the kind of “counting arguments” that Bengio makes. These seem to take each gene as a bit of information, and ask whether there are enough such bits to specify all the bits in the brain, usually taken as the number of connections. Obviously the answer is there are not enough such bits. (There aren’t even enough bits if you take individual bases of the genome as your units of information).
Being amazed that you can make a human with only 20,000 genes is thus like being amazed that Shakespeare could write all those plays with only 26 letters. It’s totally missing where the actual, meaningful information is and how it is decoded.
The simplified cartoon image that is associated with DNA, where phenotypes are directly connecting to specific genes - i.e. this mutation makes you smart, or that mutation causes cancer - is not at all how biology works.
The sentence makes exactly the error that it accuses others of making, the '20,000 genes' is not the encoding, that's the codons of which there are 64 some of which encode the same amino acid. The 20,000 really is like a single play of Shakespeare.
The quote is a criticism of a common way of thinking in biology, where many (if not most) practitioners think in terms "one gene does one thing" (or perhaps few things at most).
When one follows that mentality it becomes hard to explain how just 20K genes could make a living organism.
The genome must encode all the components used to make an organism, as well as the instructions on how and when to use these components.
The 'one gene, one thing' or just a few things at most meme died long ago for all biology practitioners that are with the times, the problem is more with IT people attempting to understand biology trying to make computing analogies to make sense of something that evolved rather than that it was designed.
Otherwise excellent, if short, article.
> You don’t need to specify where every synapse is to get the machine to work right. You just need to specify roughly the numbers of different cell types, their relative positions, the other types of neurons they tend to connect to, etc. The job of building the brain is accomplished statistically, and, crucially, probabilistically. This is why there is lots of variation in brain structure and function even between monozygotic twins and why intrinsic developmental variation is such a crucial (and overlooked) source of differences in people’s psychology and behaviour (the subject of my book Innate).
Shannon information provides a trivial upper bound on Kolmogorov complexity: imagine a tiny program with a huge data segment, which just copies the contents of the data segment to standard output. In the case of the human genome, this value is on the order of 6 billion bits. The key takeaway is that a much larger fraction of these bits than one might expect from the phrase "20000 genes" are potentially relevant, not that Shannon information somehow underestimates complexity.
It seems likely that the amount of information in the DNA "decompressor" (in mammals, the mother and her environment) dwarfs the amount actually present in the DNA itself.
But I'm left disappointed that there's no actual estimate of maximum information content. Just this:
> *[Note from Tony Zador: The length of the genome, which in humans is around 3 billion letters, represents an upper bound on the Kolmogorov complexity. Only a fraction of that carries functional information, however, so the upper KC value may be quite a bit lower than that].
That must be 3 x 10^9 base pairs, in the haploid genome. Wikipedia tells me that's 800MB.[0] However, only 1%-2% of that codes for proteins.[1] Which is basically what Zador's note says.
So what's possible for the rest? While 800MB doesn't seem that much, maybe it's enough for basic "firmware" and "software". Including instinctual behaviors.
But even all that is clearly not enough for a full blueprint and assembly instructions. That, as TFA explains, must flow from interactions during development.
0) https://en.wikipedia.org/wiki/Human_genome
1) http://sitn.hms.harvard.edu/flash/2012/issue127a/
https://upload.wikimedia.org/wikipedia/commons/4/4b/Chromati...
each of these have huge implications for what can be done with the DNA
there's tons more information that's attached to the strands of DNA, too, like methylation or promoters, etc. that the simple ATGC doesn't cover at all
if we want to stretch the turing tape metaphor, it's more like marking the symbols as various kinds of magnets on a string, then putting a bunch of them in a sack and shaking them up. does that change the amount of information?
if you're interested, you really can just pick up an intro bio textbook, skim it, then do an intro genetics text (and skip the classical genetics crap).
How many orders of magnitude?
I don't know how I would quantify that, but there's a lot going on.
Of course, if we stay with that analogy, DNA would not be a "series of instructions" but would more resemble a huge, unordered set of "if/then" rules or "trigger/action" pairs - the "actions" being protein-coding sequences and the "triggers" being their regulatory sequences.
The actual behavior of the cell is then determined by the interation of those rules and the effects of the coded proteins.
DNA shape certainly matters for gene expression. Just simply accessibility of sequences. But also, as TFA notes, through binding of RNAs and proteins.
But still, it's sequence that determines shape. So I'm not sure how that could increase coding capacity.
It means nothing, in terms of information content.
This is a common misunderstanding, which the author of the piece appears to share, but there's no getting "more" information out of something by talking about its interactions. Suppose we write a computer language where every byte is a meaningful opcode, and all possible strings of bytes are valid and do something. If we consider the set of 8-byte programs, there are 2^64 of them. It takes 64 bits to encode which one you are choosing from that set, and there are 2^64 options. The fact the they are "programs" and thus that many of them will "interact" does not mean that you suddenly have more than 64 bits of information. You still have just 2^64 options. None have appeared from the ether by the virtue of some of them "interacting". In that set of programs will be everything from programs that immediately terminate, to infinite loops, to programs that use staggering amounts of RAM and CPU before terminating, but there's just 2^64 of them.
Information is really just a count of possibilities. (Weighted, technically, but that's not a terribly helpful detail at the moment.)
Since the question is explicitly does DNA itself encode innate knowledge, it is relevant to discuss the contents of the DNA in this strict counting argument. (Add the epigenetics and whatever else you like; all it does is add more bits proportionally to the possibilities they add. But they don't make more bits than that due to "interactions".)
You, yourself, are not the sum total of information in your DNA. Even at the one-cell beginning of your life that would be an oversimplification, and since then the universe has poured far, far, far more "bits of information" through you than you could possibly have integrated into yourself. You would require a great deal more information to describe than your raw genome (just as identical twins are still, you know, distinct people). But in terms of asking what can be in the DNA alone, the information theory is relevant, and citing "interactions" buys nothing. The DNA has the bits the DNA has and it doesn't get any more by "interactions".
The organism with the DNA will interact with its environment, of course, and it's essentially proved that a lot of our "innate knowledge" requires interacting with the environment to develop. In terms of understanding the whole process, that will all be critical to understand. But in terms of literally what is in the DNA, that's not particularly relevant.
(My personal belief is that a non-trivial component of the confusion is indeed that we look at the "final product" of this innate knowledge and over-attribute it to DNA. Babies naturally recognize faces... except they don't. "Face recognition" is a huge amount of bits. On the other hand, encoding some instructions that will result in embedding an eigenface [1] into the visual system and giving a hint of dopamine when it recognizes one, and leaving the baby to eventually work out what "faces" are and why they are interesting isn't that many bits at all. The end result is "face recognition" and that system would consist of a huge number of bits' worth of information, but the vast bulk would be from the environment. What was actually in the DNA is just a tiny seed crystal of information, which fits perfectly comfortably in the DNA. Especially since it wouldn't even necessarily be a "thing that matches faces" but just a weird streak through the hyperspace of possible visual inputs that happen to privilege faces but will also privilege and disfavor any number of other inputs too, which reduces the requis...
That's true for any encoding of information. KC can only be defined relative to some computational model, and the actual number depends on the details of that model. It just so happens that there is a large family of models whose KCs are equal to within a multiplicative constant. But the actual number is never meaningful in isolation.
So is the answer that we have no clue? Or that it's a meaningless question? Given that we don't know the computational model, I mean.
To me the interesting question is how much life memory could be encoded in the genome. Maybe top points from human evolution? Or nothing more than really simple basic stuff?
Leaving aside how it might get stored in the germline, and later accessed.
Both. The fundamental problem here is that the phrase "innate knowledge" hasn't been defined well enough to allow one to quantify it. How do you measure "innate knowledge"? In what units would you express the results?
Here is an analogy: consider movies. A movie is the result of a collaboration between many people: writers, directors, actors, editors, stunt people, special-effects folks... how would you go about measuring how much of the quality of the final product is due to each individual contribution? It's a very similar situation with the nature-vs-nurture question (which is really what this is). The sum total of what we are is a result of very complex interactions/collaborations between our genes and the environment in which they exist. Quantifying the contribution of our genes is not that far removed from quantifying the contribution of an individual on a film production team.
Answering these questions is not impossible in principle. It's just really, really hard, and at the moment we aren't anywhere close to being able to do so.
Useful knowledge might include sucking on nipples, being afraid of snakes and spiders, being wary of fire, sexual drives after puberty, and so on.
As far as units, why not bits?
> Here is an analogy: consider movies.
I agree that going from a zygote to an adult animal is a complicated process. There's the genome, the process of development, and a complex mixture of sensory data and actions. And yes, that doesn't do justice to the complexity.
But the question here is how much innate data. Not how substantial a role that data plays. So as a first approximation, wouldn't that be the script? Or maybe the script plus the video file?
> It's a very similar situation with the nature-vs-nurture question (which is really what this is).
I wasn't seeing it that way. There's no question that the environment plays a crucial role in development. But I don't see how any of that can create knowledge of past events. Which is what "innate knowledge" means to me.
Because bits are a unit of information, not knowledge. Not the same thing.
But even if you're really interested in information and not knowledge that's still a problem. You can easily measure the information content of the human genome because we know how it's encoded. It's much harder to measure the information content of the human brain because we don't yet have a complete grip on how it works.
> I don't see how any of that can create knowledge of past events.
That depends on what you mean by "past events". If you really mean individual one-off events, there is no known mechanism for that kind of information to get encoded in a genome. If you mean general information about the world acquired through "experience" (where having natural selection act on your population counts as "experience"), that could be encoded in a combination of information that resides in the genome and information that resides in the environment, where the structure of the brain is part of the environment (of the genome). We just don't know.
How can knowledge not be a subset of information?
"I am wearing a white hat."
Is that information also knowledge? Well, it depends. It depends, for example, on whether or not I am in point of fact wearing a white hat, because if I'm not then you cannot possibly know that I am wearing a white hat despite the fact that I have just given you that information. That information may or may not correspond to reality, and one of the things that distinguishes knowledge from information is that knowledge has to correspond to reality in order to qualify as knowledge, and information need not correspond to reality ("false knowledge" is an oxymoron but "false information" is not.)
But there's more to the distinction between knowledge and information than mere correspondence to reality. If I am in point of fact not wearing a white hat, the information above does not magically transform itself into knowledge if I suddenly put on a white hat.
So the answer to TFA's question is "It depends".
It sort of feels like training a neural network is like rewiring an adult brain which has already fully grown. Are there neural architectures which grow over time, or "age", emulating the early development of the brain?
have labs successfully changed these? I'm familiar with a prenatal brain surgery that made chickens behave like ducks, but it wasn't genetic
As for reflexes, don't we kow how they work on an anatomical/neural level in a grown adult? From my high school biology lessions, I remember that for certain reflexes, there are nerves that directly connect "sensors" to motor neurons in the spinal cord, bypassing the brain altogether.
If you stick to the "genes as a program" metaphor, "encoding" this reflex in the genome would mean that the genome contains the correct triggers and "subroutines" (i.e. as cascades of gene expression) so that a particular group of cells will form that nerve and connect it to the correct other neurons.
One of the interesting things that we've learned over the years, painfully and repeatedly, is that fairly simple encodings of fairly simple data can repeatedly generate extremely complex visible outcomes. A great example of this is patterning in animal fur- there isn't a section that encodes a pixel map of the resulting spots or their X/Y locations. INstead a collection of genes operate in a feedback loop to generate the patterns. Turing's theory of morphogenesis has turned out to be highly prescient application of very simply theories that does a good job of explaining observed patterning.
What I've seen repeatedly over the years is that the proteins themselves aren't that interesting- all the interesting complex repeatable phenotypes come from careful regulation of the expression of proteins with feedback loops and spatial gradients. The best work on this has been done in evo-devo field on model organisms (not humans). C Elegans was an exceptionally good choice for a model organism for many reasons, not the least of which is that every instead of the organism has exactly the same cells in exactly the same location, and that appears to be completely genetically encoded (again, not like a pixel map, but as generative rules).
As evidence for this we know there are all sorts of heritable epigenetic effects. It would also partially explain the partial failure of GWAS to pinpoint all an individual's features as was previously hoped before the human genome project completed.
It seems like there's a process called Somatic Cell Nuclear Transfer that might make possible to evaluate aspects of this, but it doesn't appear to have been performed sufficiently many times with different individuals to draw major conclusions.
[1] https://en.wikipedia.org/wiki/Conserved_sequence
There's a new soon to be hot (I predict) topic in AI/ML called inductive bias. It turns out that the way your neural net is structured, i.e. the overall shape and the way the neurons are connected, can strongly bias which information the neural net tends to learn, and how quickly. To the point that there was a paper (can't find it now) where IIRC the authors were able to develop an algorithm which creates neural nets which can solve basic problems with little to no training. Point being, you don't need to encode, say, the image of a nipple and the action of suckling in DNA to create innate suckling knowledge - you simply need to structure the brain such that it is biased to perform and learn certain sequences of behaviors.
I think this may be the origin of much if not all innate knowledge and I believe that research in AI/ML will lead to better understanding of human (and animal) cognition.
And indeed, what people will find is exactly as you say: The DNA encodes how to grow the network such that it will respond with the evolutionary advantageous instinct.
I don't know if I'd quite say that no one is looking into that. Obviously we know that there are NN that use deep networks. But also we can see that we can increase the computation power by removing nodes and edges that don't contribute to the correct answer. In evolution, this is a literal cost savings in that you need to feed less neurons.
On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels.
On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself.
On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.
So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe.
https://en.wikipedia.org/wiki/Busy_beaver
If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms...
Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt.
So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.
How do you think "structuring the brain such that it performs suckling" differs from "encoding the action of suckling"?
You just said the same thing two different ways.
The structure I'm describing is at a different scale from the structures of the individual neurons. Actually, come to think of it, baby motion may be an even better example:
When a baby is born, it (possibly) has no understanding of how to move it's arms and legs. But the newborn brain is structured in such a way as to encourage movement of limbs, such that the movement required for learning becomes a high probability event°. In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment.
If you connect neurons in a particular large scale pattern, and wire some up to ears, some to eyes, some to touch receptors, two things may happen:
1. Your brain is more likely to perform the initial actions required to learn
2. Your brain may more easily learn certain actions
The second thing is inductive bias, and I think it will be extremely powerful in the world of ML once we figure out how to harness it. We know that neural nets are hugely wasteful now, there have been papers published on neural network optimizers which can shrink nets by some 80% while maintaining accuracy. Inductive bias is probably such an optimization.
That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary.
What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.
The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.
Why not?
Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally.
So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song.
This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.
Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not.
(Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)
Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds.
https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language
This is an exact match to your hypothetical question if you're willing to overlook the fact that sign languages are not "spoken".
Nicaraguan sign language was not developed in a vacuum, it was developed by children living in a completely normal human society with the full use of all their senses but one. Far from "nothing".
Why do you think differently? "I think the very idea is ridiculous" isn't much of an argument.
If they are living in a normal society, they are exposed to language nearly continuously, from visible printed language to visible body language to the visual evidence of language being spoken all around them.
https://en.wikipedia.org/wiki/Epigenetics
If we are looking for the innate information, we should be asking the question: In how much information can we store the computer, and all machinery that builds it from basic components (molecules basically). It does become easy then to imagine that there is some kind of suckling response, a computer also starts doing things when it is powered up.
However in reality, the gene dose makes a difference - so it’s more like the genome (+genome architecture) encodes for capacity to produce different amounts of A and B. So, with high A expression you will see things like “AAAAB”,”AAAAAA” etc.
The point is that rather than encoding for different functions, the genome encodes for the diversity of different message queues - and it is the length of these queues and rate of message firing that encodes (as much as you can in a probabilistic sense) for our cell programming. The messages could be “build a transcription factor” or “put a receptor on the surface”, or “make a crank that increases the capacity of these queues”.
One form of this is culture and species taught behaviors.