HyenaDNA: Long-Range Genomic Sequence Modeling (context length of 1M tokens) (arxiv.org)

This is really cool...these LLM-like models in technical disciplines are going to open up he new frontiers in science.

And if you accidentially created a monster, because it switched some chars, you can always tell it: That was good, but you can do better and it will finish the job correctly.

Part of what’s interesting here is that there haven’t been any robust featurizers for DNA in the same sense that we have robust featurizers for proteins (like ProtBERT et al.) that just work out of the box for amino acid sequences. Since many genes have >100k bp vs. proteins that are often less than 1k AAs, the much longer context window is needed. ProtBERT gets ~200k downloads per month on huggingface and DNABERT gets ~10s.