Running Llama3 Locally

Follow this: https://ollama.com/library/llama3

for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory

what kind of memory? there are 3 types of memory: GPU memory: medium capacity | high speed || Apple Unified memory: high capacity | high speed || CPU memory: high capacity | low speed ||

rtx 3090 has 24GB of memory at 935GB/s memory bandwidth || rtx 4060ti has 16GB of memory at 288GB/s memory bandwidth ||

m2 mac mini has 8GB/16GB of memory with 100GB/s memory bandwidth || m3 macbook pro has 32-128GB of memory with 400GB/s memory bandwidth ||

new x86 consumer cpu have up to 256GB of DDR5 at 100GB/s memory bandwidth || old cpu with ddr4 and ddr3 are much slower ||

||

Remember, there is a simple formula for tokens/s: you need to divide memory bandwidth by model size, so for example:

mac mini runs llama3-8b at 18.8 tps using 5GB of memory 100GB/s/5GB = 20 https://twitter.com/awnihannun/status/1781345824611680596

rtx 3090ti runs llama2-7b at 179 tps also using around 5GB 1000GB/s/5gb = 200 https://github.com/turboderp/exllamav2