Training the model, although it is small with only 15M parameters, takes quite some compute power. I wonder if the resources would have been sufficient at the time. Also, I wonder if disk space would have been sufficient to store the input corpus.
It looks like these machines run IRIX which is a unix variant on a mips processor and has some parallelization features. I wonder if the compiler he's using is taking advantage of these, or if there's potentially some speedup possible. From what I remember, llama2.c uses OMP to optionally parallelize some of the matrix-vector products which wouldn't be available on that machine.
It's interesting to think about some of the earliest machines we could have run a pretty great LLM on. A Cray XMP-EA from 1986 could address up to 2GW, which would have been 16GB. The XMP or earlier Cray-1 lines could also use multiple solid-state SRAM disks that were up to 1GB in size and could stream at ~1GB/sec.
It is not typically used outside of L1 cache. I imagine a computer using bistable memory cells as being more resilient, and a more elegant design than memory that needs circuitry to constantly refresh it.
answering the question in the xweet: researcher 28 years ago would have decent reproducible peer reviewed metrics so they would not even look at most of this.
I wonder how much RAM is needed to run this – for example, would this work on the Indigo2's minimal config of 16 MB? (If so, I'll be running towards THINK Pascal to get this running on an SE/30...)
10 comments
[ 2.6 ms ] story [ 8.2 ms ] threadTraining the model, although it is small with only 15M parameters, takes quite some compute power. I wonder if the resources would have been sufficient at the time. Also, I wonder if disk space would have been sufficient to store the input corpus.
https://en.m.wikipedia.org/wiki/IRIX