Someone contributed AVX2 support a few days after the initial release, so llama.cpp (and the alpaca.cpp fork) should work on most x86 CPUs as well now.
== Running in chat mode. ==
- Press Ctrl+C to interject at any time.
- Press Return to return control to LLaMa.
- If you want to submit another line, end your input in '\'.
> What is an alpaca?
An Alpaca (Vicugna pacos) is one of two species in the genus Vicuña, which are both domesticated animals from South America and Central Asia that have been bred for their fine wool.
>
This isn't actually a direct reproduction. It's a bit missleading because it only says it at the bottom. This actually uses LORAs which train way faster and are much more flexible, but they perform a little worse.
There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though.
> The weights are based on the published fine-tunes from alpaca-lora
Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685
Ah thank you for the correction this is good to know! I'll admit I'm not super experienced with them directly.
I think the current LoRAs for Alpaca are reportedly performing a little bit worse - I'm guessing from a lack of optimization so I'm sure it'll improve.
@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with instruction tuning.
Using the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower.
No chance I'd be able to run this on an 8GB M1 Mac mini then? Looking to hook this up to an IRC bot for a bit of fun with my friends and have this spare Mac lying around.
The 7b model loads on a lot less than 8GB RAM, so you should be good to use it on an 8GB M1 Mac Mini. It takes very little time to set up, so it's worth trying to find out.
There are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on 32GB machines, 33B and lower would run on 16GB machines, and 13B (minimal useful one seems like) would fit on 8GB -- which is wild and as good as you could hope for.
The 30B variant of LlaMa quantized to 4bit and running with llama.cpp takes approximately 20GB of ram. It runs pretty decently on my M2 MacBook Air with 24GB of ram. I assume Alpaca will take the same amount of ram.
The LoRa weights for 13B are on huggingface ( https://huggingface.co/samwit/alpaca13B-lora ), it should be possible to follow the instructions linked in the Alpaca.cpp readme to merge those weights into the base 13B model, then just just follow the usual Llama.cpp conversion and quantization steps
Note LoRa fine tunes aren't the same as the original Alpaca, LoRa results in some performance loss (although how much isn't clear)
Alpaca model is llama trained on ChatGPT, it's basically a way of getting cheaply what OpenAI did after the GPT training....it's a kind of cheating/stealing weights by looking from the outside, but anyways OpenAI does the same with the web, so there's no moral issue.
You're misinterpreting the comment a bit, the poster above is asking for the original Alpaca weights. The code in the repo uses a reproduction of Alpaca with different training code and method of fine-tune (LoRA).
From my experimentation it seems like there's some significant loss in accuracy running the tuned LoRa models through llama.cpp (due to bugs/differences in inference or tokenization), even aside from losses due to quantization.
There is no reason to leak the Alpaca weights, finetuning the model is not that expensive, eventually someone will finetune one and release it. (Also LoRA should performs on-par or better than fine-tuning in model quality)
There are different fine tuning methods. Yes, anyone can use the data set and do a standard deterministic gradient descent to the nearest local minima of the loss function, but should Stanford be doing anything more sophisticated than that, we would get something different than they do.
It is enough to use a different seed to get a different result; I do not think it is important to recreate a copy of the Stanford model weights, as long as the performance is equivalent.
Stanford released the exact training data as well as the training script with all parameters. Boot up a p4.2xlarge (8 A100 GPUs) which costs about $40/hour and let it run for a 2-3 hours and voila. See the Readme in their repo where it mentions the fine-tuning script[0]
With chatGPT I can give commands like “summarize the following document: <500+ words>” and it works. I’d love to be able to do this locally with my obsidian notes.
The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.
Tokens. Short or common words tend to be one token, while less common words are composed of multiple tokens. For GPT OpenAI gives the rule of thumb that on average you need four tokens to encode three words, and LLaMA should be similar
Could you share what you did to convert the models? The mirrors aren't very reliable and it would be good to have it documented somewhere. There is some discussion about this at https://github.com/ggerganov/llama.cpp/issues/172
There's a script in the alpaca-lora repo for converting the weights back into a PyTorch dump- and my changes have since been merged https://github.com/tloen/alpaca-lora/pull/19
ah, right I did notice that because people were running queries into the training data.
why is there a general assumption that unreleased unreleased weights are better? is that something we can do, a free-weights community that solves this recurring issue?
If I am not mistaken, they actually did release their code. Yesterday there was a change to the repo that added a train.py file. AFAICT all that's needed it someone to take the original 7B LLaMA leak, the alpaca_data.json file and run train.py on some beefy hardware. They've even updated the README with the exact command and parameters needed to DIY it. I'm somewhat expecting that there will be a release by someone in the next few days.
That's awesome! I think I remember them saying it was only around ~$500 in compute costs to train so I hope we see those weights released soon. I am hoping someone releases the 13B model fine-tuned.
“For our initial run, fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. We note that training efficiency can be improved to further reduce the cost.”
($500 was what they paid OpenAI to generate the fine-tuning dataset.)
$ ./chat
main: seed = 1678992123
llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size = 512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin'
llama_model_load: .................................... done
llama_model_load: model size = 4017.27 MB / num tensors = 291
system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
main: interactive mode on.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
== Running in chat mode. ==
- Press Ctrl+C to interject at any time.
- Press Return to return control to LLaMa.
- If you want to submit another line, end your input in '\'.
> Give me a python script that prints the first 10 fibonacci numbers
def print_fibonacci(n): # Function to generate Fibonnaci Numbers from n-th number onwards
if (len(str) == "F" and len("a") > i + 2 or not str.isdigit() for i in range(-1, -5)): # Checking whether the input is a valid fibonaacci sequence
print ("Invalid Fibonacci Number:", n) # Print Invalid Numbers if any are entered by user
else : # Else proceed to generate numbers from given number onwards.
for i in range(1,n+2): # Generate fibonnaci sequence starting with the second digit of Fibonacci Series (i=0) till nth digit is reached
print("F", str[len('a')-i:], end=' ')
The python script doesn't even compile, but I suppose that's expected from a 4gb model
I think the LLaMA-7B in general might not just be very good. I've been playing around and run full non-quantized LLaMA-30B and LLaMA-7B in a bunch of experiments and I think the quality of output is much, much better in LLaMA-30B.
I ran it on a 128 RAM machine with a Ryzen 5950X. It's not fast, 4 seconds per token. But it's just about fits without swapping. https://github.com/Noeda/rllama/
I am running fp16 LLaMA 30B (via vanilla-llama) on six AMD MI25s. Computer has 384 GB of RAM but the model fits in the VRAM. It takes up about 87 GB of VRAM out of the 96 GB available on the six cards. Performance is about 1.6 words per second in an IRC chat log continuation task and it pulls about 400W additional when "thinking."
Have you done any comparison testing between 30B q4/q8/q16? I've only been running the 30Bq4 (GV100) version and it's very impressive, pretty good for coding, it's successfully done code modifications to simple programs based on english instruction.
I have not, but I want to in near future because I'm really curious myself too. I've been following Rust community that now has llama.cpp port and also my OpenCL thing and one discussion item has been to run a verification and common benchmark for the implementations. https://github.com/setzer22/llama-rs/issues/4
I've mostly heard that, at least for the larger models, quantization has barely any noticeable effect. Would be nice to witness it myself.
The example I gave was using this as a backend for a chat bot in a private server and i'm not comfortable sharing the prompt, however if you look up the leaked bing prompt that might give you some ideas for how to prompt an LLM into being a chatbot that can answer coding questions. I've had pretty good results using it as a bot (with some glue code that does sorta vanilla regex-based prompt cleaning, but not too much, it's mostly prompt)
If you're not trying to get it to be a chatbot it's much easier, here's a prompt that worked for me on the first try in the default mode with 13Bq4 on a 1080Ti:
Here are is a short, clear, well written example of a program that lists the first 10 numbers of the fibonacci sequence, written in javascript:
```js
and when given that it finished it with:
function Fib(n) {
if (n == 0 || n == 1) return 1;
else return Fib(n-1)+Fib(n-2);
}
var i = 0;
while (i < 10) {
console.log("The number " + i + " is: " + Fib(i));
i++;
}
```
\end{code}
(I don't work at OpenAI so take it with a grain of salt) Yes and No they are similar. It is basically just a fancy autocomplete like llama, but I believe it's specifically been trained on chat content, or at least finetuned on such, and it probably uses a more chat focused labeling scheme on the training data as well to help it perform well on that specific task and be conversational.
I noticed there's a couple of open issues on llama.cpp investigating quality issues. It's interesting if a wrong implementation still generates plausible output. It sounds like an objective quality metric would help track down issues.
Recent Tweet by Joe Biden: 3D printing is a great way to make money.
Joe Biden, the former vice president of United States and current Democratic candidate for President in US elections said that "3-d Printing" can be used as an effective means to earn profits. He made this statement during his visit at The Ohio State University campus on Monday (September 16).
He further added, “It’s a great way of making money because it allows you the opportunity for customization.” Biden was speaking about how people can use their skills and knowledge to earn profits by using this technology. He also ur
why do these LLMs struggle so hard to be concise? is this because the Alpaca dataset is pretty wordy? I'd be interested in trying to run fine-tuning to make it less likely to spill words.
In my experience so far GPT-4 will most certainly be concise if you ask it to be.
Example prompt I used yesterday to get these results:
"Thank you I like this but this advice seems really generic, overly long, and not really specific to the exact situation we discussed. Can we simplify this down to ..."
Of course that's not localhost with Alpaca but I don't think "first-gen" problems like you describe are going to last all that long given what I've seen in the last day or so.
I love the Disclaimer at the bottom. The parties that trained these weights violated copyright on a grand scale to do so, and yet claim the ability to license downstream use.
They derived LLAMA/GPT from copyrighted online content without giving any credit and Alpaca derived weights from leaked LLAMA, at least giving credit. :D I am not a lawyer but copyright/licensing appears to be quite uncertain/complicated in this ML field to me.
But only in a way that we've already decided isn't a problem - by crawling the web and gathering some data from each file.
Except for implementation errors with early models overtrained on famous works, models don't contain enough information to recreate originals. It's not a compression algorithm.
It's because LLaMa is technically not publicly available, Facebook currently intends for only researchers to have it.
Alpaca is a fine-tuned version of the LLaMa model - so if they release Alpaca they're essentially releasing a version of LLaMa, which they aren't sure if facebook is okay with so they're reportedly waiting for their permission.
While yes you can easily find LLaMa it's technically not intended for the public according to Facebook.
Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.
weights refer to the trained model weights like for example stable diffusion's v1.1, v1.2 .. v1.4 v.2.x etc. Same with llama having 13B up to 65B parameters (different weights)
LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook.
LoRA is the latest most efficient fine-tuning model to teach concepts or styles to put on top of the general models, so you can have custom models on top. like embeddings or fine-tuning for LLM's. So you had Textual Inversion, Dreambooth, and LoRA on that category.
what else do you need? Googling or asking chatgpt can help a lot too
Let's take a step back. You have a model like linear regression. For example, y=bx where y are your outputs and x are your inputs. Based on some data, you learn that b=1. Therefore, you share the weights of the model as a file like {b=1} and also share with them the model y=bx (usually shared via code) so they can run it in production.
Your inputs are lists of numbers. Your outputs are lists of numbers. There exists some possible list of numbers such that, if you multiply your inputs by that list you'll get (approximately) the outputs.
In this conception that possible set of numbers are the weights. "Training" is when you run inputs, compare to known outputs, and then update the weights so they produce outputs closet to what you want.
Large Language Models, it may be hard to see how they fit this paradigm - basically convert a sequence to a list of numbers ('aardvark' is 1, 'apple' is 2 etc) and then the desired output is the next word in the sequence (represented as a number). Surprisingly, if you get good at predicting next word in sequence you also get the ChatGPT et al behavior.
This is what happens when running inference on a neural network:
Input (list of numbers) -> (Bunch of math operations) with (other numbers) -> Output (also a list of numbers)
This applies whether you are talking about image classification, image generation, text generation etc.
The model defines what the "(Bunch of math operations)" part is. As in, do these multiplications, then add, then a tanh operation etc.
The weights define what the "(other numbers)" are. Training is the process of figuring out these weights using various methods - some of which involve example inputs/outputs (supervised learning), others don't require examples (unsupervised or self-supervised learning).
Instead of a matrix NN they use two matrices, Nd and d*N, and assuming d << N it would be like a lightweight addition. Keeping the original matrix unchanged, this new addition is simply added in parallel and initialised to low values.
I was like this a week ago. Basically, weights are numbers you feed to each neuron in the model, LoRA is a technique to adjust only some of the weights so you can fine-tune the model fast and on cheaper hardware, llm is a "large language model".
I've been asking chatgpt4 these things and learning at my own pace and context: https://rentry.co/vafkn
Well, it’s more that the weights are the neurons. There’s not actually like neuron objects defined or anything, neural networks are just a bunch of matrix operations. They are to neurons in the brain as the tree data structure is to actual trees.
You mean the same neurons that don't really exist, other than as a connection from weight to weight, so quite reasonable to see as the weights themselves, because what else is available for us to reference as neurons in absentia those connections?
> You mean the same neurons that don’t really exist,
“Neurons” are an abstraction that exists logically “between” the weights, but the weights themselves don’t have the features of neurons. (In that each weight is the weight of a connection between a neurons (or between a neuron and an input/output.) Weights are more synapses than neurons.
I disagree. The metaphor we use in ML is that neurons are the nodes that receive the summed, weighted inputs from other neurons. Yes, the weights are the strengths of the connections between them. However, there are many more weights than neurons, so conflating the two doesn't make sense schematically. Also, neurons can also have other parameters which define their behavior such as bias and activation functions. Furthermore, the activation of a neuron defines the network's response to a stimuli, so these change depending on the input, whereas the weighs are constants (after being trained), that parameterize the system.
The analogy is that weights are synapses, not neurons. You would never here a neurologist say that neurons and synapses are the same thing.
I would agree with most of this, but there is no direct analogy between all of the components of a neuron in neurology, which are typically umbrella'd under the name "neuron" and regarded as parts of one, versus ML's version.
Eg, if a weight can be a synapse, can't a weight be an axon? Axons also "connect" neurons, but their length is more related to the connection strength, so could be considered more analogous to a "weighting".
Yet, axons are not as obtusely "one-to-many" as synapses, but depending on the structure of the ML model, and the view of which aspect of it is more impactful to be highlighting by analogy, either take might be more appropriate.
I suppose it depends on the kind of structure you're working with, and whether you're training and inferring, or just one or the other. In all cases I think a good argument could be made for general neuron analogy abuse.
Oh that's interesting. I don't know too much about the neuroscience, just enough to agree that a real neuron is vastly more complex than a node in a "neural net". Based on your description, an axon is most highly analogous to the bias term, although it would be a multiplicative bias. I wonder if that's been tried.
What's fun is that the recent references to LoRA stands for Low-Rank Adaptation [1], not Low-Resource Adapters[2] (??? don't know if this even exists), but because Low-Rank Adaptation came out in 2021, chatgpt only explains what Low-Resource Adapters is, not Low-Rank Adaptation, which is what is being used in recent break throughs.
My own questioning of chatgpt on LoRA returns "Logistic Regression Algorithm" instead. Looks like it's too new to ChatGPT.
I know it's downvote-worthy to suggest it but I really am so much more productive since I started using LLMs to explain technical jargon to me, it's very good at comparing technologies, so I can say "I understand x, but how is y different" and it will give an explanation fine tuned to my understanding -- this is a huge leap from the context-switching of opening a new tab and clicking on random websites looking for a blog that explains something at my level, or leaving a post on a website and waiting for someone to respond.
Same! ChatGPT has been out for months, and really only in the last few weeks have i started using it as much as I should. I'm using an electron wrapper now, so it's a first class desktop app. With the plus subscription, the 3.5 is as fast as google.
"An "Electron wrapper" refers to a software application created using Electron, a popular open-source framework. Electron allows developers to build cross-platform desktop applications using web technologies like HTML, CSS, and JavaScript.
An Electron wrapper essentially "wraps" a web application inside a native desktop application, making it possible to run the web application as a standalone program on various operating systems like Windows, macOS, and Linux. This is particularly useful for developers who want to create desktop applications without having to write separate code for each platform. Instead, they can use their existing web application code and package it with Electron to create a desktop application that works across multiple platforms."
I asked GPT-4 to produce a simulated argument about Electron which devolves into name calling. Here's the 2 page of result it returned:
User1: Hey, have you heard about Electron? It's an awesome framework for creating cross-platform apps using web technologies like HTML, CSS, and JavaScript.
User2: Yeah, I've heard of it. But I think it has some pretty serious shortcomings. Like, it's notorious for consuming a lot of system resources, which can be a major drawback for certain applications.
User1: Well, I think that's an acceptable trade-off for the ease of development and the ability to create one codebase that works across different platforms. Plus, it's backed by a large community, which is always a plus.
User2: Sure, but you can't ignore the fact that it increases the app's size significantly. Even a simple Electron app has a huge size compared to native apps, making it a poor choice for users with limited storage or slow internet connections.
User1: The size issue can be managed with optimization techniques, and honestly, storage is becoming cheaper and more abundant these days. You're blowing it out of proportion.
User2: It's not just about storage. Electron apps tend to have worse performance than native apps because they run on a separate instance of Chromium for each app. That's just bad design!
User1: Look, no framework is perfect, but Electron provides a valuable solution for developers who don't want to learn multiple programming languages and platforms. It's about practicality and productivity.
User2: Practicality? You call it practical to have apps that drain your battery life and hog system resources? I'd rather invest the time in learning native development than creating a subpar user experience.
User1: You're really not giving Electron enough credit here. It's widely used by popular apps like Slack, Visual Studio Code, and Discord! They can't be all that bad.
User2: Just because some popular apps use it doesn't mean it's a good choice. They're actually prime examples of how Electron can lead to bloated and resource-hungry applications. Have you ever noticed how sluggish those apps can be at times?
User1: Oh, come on! You're just nitpicking now. The benefits of Electron far outweigh its drawbacks for most developers and applications.
User2: I'm not nitpicking. I'm pointing out legitimate concerns. If Electron keeps getting a free pass, we'll end up with a sea of bloated, resource-hungry applications, dragging down our devices.
User1: You're just being a native app elitist! Electron has democratized app development, making it more accessible for web developers. It's better to have more apps available, even if they're not perfect, than to have a limited selection because of high barriers to entry.
User2: You Electron fanatics never cease to amaze me. It's like you're blind to its glaring problems. You want everyone to just ignore the elephant in the room!
User1: And you native app snobs are so quick to dismiss any alternative solution that threatens your precious, exclusive club! Get off your high horse!
User2: At least my "exclusive club" doesn't promote lazy development practices and drag down user experience! Enjoy your bloated apps, you Electron zealot!
User1: Have fun in your walled garden, you insufferable native app purist!
That sounds like a cool use case, but... how can you bring yourself to trust anything at all that it tells you? This feels like it's just begging to contaminate your understanding with sparse, unpredictable errors, and personally I find the thought of that unexpectedly dismaying.
Personally Ive been coding long enough I’ve forgotten more than most junior people know. So I use ChatGPT to refresh my memory more than anything else. Like yesterday couldn’t remember how to open some ports on a Postgres server. ChatGPT gave me a 95% accurate answer and based on my own knowledge base I fixed the mistake. This took all of a few seconds. Whereas with normal web search it would take an order of magnitude longer and likely be not much more accurate.
It's a lot easier to verify whether something is correct once you know what the thing is.
Half the time when I'm googling something I don't even know what subfield the term belongs to. I don't know what related terms I can use to narrow it down. Or I keep getting papers on a related but different question from what I'm trying to ask.
But if ChatGPT gives me an answer in context of a bunch of related terminology, it's so much easier to then plug those additional terms into Google and find a/the authoritative source.
For comparison, a lot of my (non-ChatGPT) initial searches will result in random blog entries or something that are often just flat-out wrong or else ignoring competing viewpoints. But they still might point me to better context and search terms.
The value of ChatGPT is that it gets you to information you can verify so much faster. But you still need to verify it the same way you need to verify stuff you find on blogs or Quora or Reddit or HN.
I tend to trust blogs or conversations with other people more than I should. The benefit of a tool like ChatGPT is that I already distrust it, and I tend to double-check more often what it responds.
I agree with you, but the comparison and fine-tuning GP mentioned are really areas where it's an unparalleled tool right now. Dismissing it in fear of "contaminating your understanding" is missing out on what the tool may offer to you. It's not really that much worse than a search query.
This has been really the most productive use of it for me. But I'm so cheap I've been stopping myself more and more and just spending an extra minute googling to save the cent. Even if its just literally penny pinching, its also helping the environment! Is the velocity I get an answer worth the cycles? I'm not so sure these days, but its been interesting reflecting on this.
Definitely excited for the inevitable reports comparing the killowats spent cumulatively to get an answer through google vs openai. How many blogs visited does it take to spend the same amount of energy getting the answer from a model?
I understand that AI/ML jargon can be confusing. Here's a brief explanation of some common terms and concepts to help you better understand them:
Weights: In machine learning, weights are parameters used in models to determine the strength of the relationship between input features and the output prediction. They are learned during the training process and are adjusted to minimize the error in the model's predictions.
LoRA: LoRA stands for "Learning from Rules and Arguments." It's a method that combines rule-based reasoning with neural networks, which allows AI systems to reason using human-like logic and provide explanations for their decisions. LoRA can enhance the interpretability and transparency of AI systems.
LLM: LLM stands for "Large Language Model." It refers to advanced AI models, such as OpenAI's GPT series, which are trained on vast amounts of text data to understand and generate human-like text. These models can perform various language tasks such as translation, summarization, question-answering, and more.
Supervised Learning: This is a type of machine learning where models are trained using labeled data. The model learns to map input features to the corresponding output labels through a process of trial and error, adjusting the weights to minimize the difference between predictions and true labels.
Unsupervised Learning: In contrast to supervised learning, unsupervised learning does not use labeled data. Instead, it aims to discover patterns or structures in the data, such as clustering or dimensionality reduction, without explicit guidance.
Reinforcement Learning: This is a type of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties and aims to maximize its cumulative reward over time.
Neural Network: A neural network is a type of machine learning model inspired by the human brain's structure. It consists of interconnected layers of nodes (neurons) that process and transmit information. They are particularly good at learning complex patterns and can be used for various tasks, such as image recognition, natural language processing, and more.
Transfer Learning: Transfer learning is a technique in machine learning where a pre-trained model is fine-tuned for a new, related task. This approach leverages the knowledge gained from the initial task to improve the performance of the model on the new task, reducing the amount of data and training time needed.
These are just a few of the many concepts and tools used in AI/ML. Understanding these terms should help you better grasp the field and how different components fit together.
Picked the wrong one. LoRA, Low-rank Adaptation of LLMs (https://arxiv.org/pdf/2106.09685.pdf), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the product of two low-rank matrices to select layers. The weights from those matrices are learnable. The method allows to adapt big models on (relatively) low-memory GPUs.
I'm a software guy learning the AI field and tools and I literally made a glossary in my notes that I add to any time I run into a new initialism or word or concept. It's pretty long.
This is funny, the response contains some data from the dataset itself.
> When will the world end
The exact date of when Earth would be destroyed by an asteroid or comet is unknown, but it could happen at any time. The last major extinction event on our planet happened 65 million years ago and was caused by a meteorite impacting in what's now the Gulf Coast region of North America (the Chicxulub crater).
### Instruction:
What is an asteroid belt?
Instruction: What does it mean to be "in orbit"? ##Response:
> write a typescript program that read a text file and print to the screen
const inputFile = './input.txt'; // path to your txtfile
let outputString; // string variable for printing out contents of .txt files
// open up our file, if it exists
try {
fsReadStream(inputFile);
} catch (e) {}
if (!!errno) throw errnoError(); /* Throw an error on ENOENT */ // if the inputfile does not exist
// read in contents of file and print out to screen [1] 51232 segmentation fault ./chat
language models require massive scale to train. But scale isn't only in the number of parameters or neurons. Scale also exists in the amount of data the model trains on.
While parameter size affects post training size and requirements to run. Data size does not. Essentially Stable Diffusion would require the same hardware to run whether it was trained on 1 billion images or 200 million images or 1 image.
Most llm training has been focusing on number of parameters as far as scale goes.
Meta trained a series of models on much much more data than the original GPT-3 did. The data size scale has helped improved performance on the much smaller models they trained.
The parent poster was talking about training longer but the model being kept at smaller scale so it would not be expensive to use in production. It's a trade-off, you could train shorter with a larger model.
Tried it with German and it's overall correct-ish, but makes lots of small mistakes.
> > Translate this to German: I went to the back alleys of downtown Metropolis to find a phone booth I could use to dress up as my alter ego: Superman!
> Ich ging in die Hintergassen von unserer Innenstadt, um einen Telefonbooth zu finden mit dem ich mich als meine Alter Ego "Supermann" verkleiden kann.
but it should be
> Ich ging in die Hinterhöfe der Innenstadt von Metropolis, um eine Telefonzelle zu finden, in der ich mich als mein alternatives Ich verkleiden konnte: Superman!
Keep in mind that alpaca's dataset was generated with chatpgt, and alpaca tuned models might sound like mini-chatgpt bots with similar "character" to them
Does anyone have a link to pre-quantized models for this/llama.cpp? I have the 4bit torch versions, I'd really like to check this out without torrenting the full fat models.
(30B on a GV100 is very impressive by the way, highly recommend people play with this stuff, 13B runs comfortably on a 1080Ti, would really like to compare inference speed and accuracy between the implementations)
302 comments
[ 4.0 ms ] story [ 277 ms ] threadmain: seed = 1678990693 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 4529.34 MB llama_model_load: memory_size = 512.00 MB, n_mem = 16384 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors = 291
system_info: n_threads = 4 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | main: interactive mode on. sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
== Running in chat mode. == - Press Ctrl+C to interject at any time. - Press Return to return control to LLaMa. - If you want to submit another line, end your input in '\'.
> What is an alpaca? An Alpaca (Vicugna pacos) is one of two species in the genus Vicuña, which are both domesticated animals from South America and Central Asia that have been bred for their fine wool. >
sysctl: unknown oid 'hw.optional.arm64'
There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though.
> The weights are based on the published fine-tunes from alpaca-lora
Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685
I think the current LoRAs for Alpaca are reportedly performing a little bit worse - I'm guessing from a lack of optimization so I'm sure it'll improve.
Wish i could run the 60B model, this 7B model is a bit weird.
See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...
Note LoRa fine tunes aren't the same as the original Alpaca, LoRa results in some performance loss (although how much isn't clear)
https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#...
EDIT: I see now you are saying you re-worked the existing interactive mode. I still think your changes could be a PR into the original repo
What do you mean by this? Can you please elaborate.
Because there's no substantive change to the code, I assume this fork exists (and this HN post exists) purely as a method to distribute the weights.
[0] https://github.com/tatsu-lab/stanford_alpaca
With chatGPT I can give commands like “summarize the following document: <500+ words>” and it works. I’d love to be able to do this locally with my obsidian notes.
why is there a general assumption that unreleased unreleased weights are better? is that something we can do, a free-weights community that solves this recurring issue?
“For our initial run, fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. We note that training efficiency can be improved to further reduce the cost.”
($500 was what they paid OpenAI to generate the fine-tuning dataset.)
65B fp16 is ungodly slow, ~300,000 ms/token on the same machine.
I've mostly heard that, at least for the larger models, quantization has barely any noticeable effect. Would be nice to witness it myself.
If you're not trying to get it to be a chatbot it's much easier, here's a prompt that worked for me on the first try in the default mode with 13Bq4 on a 1080Ti:
and when given that it finished it with:So under the hoods, ChatGPT is just a model like Llama where they prepend every user input with a context that makes it behave like a chatbot?
Thanks for the info.
https://github.com/ggerganov/llama.cpp/issues/129
https://github.com/ggerganov/llama.cpp/issues/173
prompt: "Recent Tweet by Joe Biden: "
output:
...but not by muchExample prompt I used yesterday to get these results: "Thank you I like this but this advice seems really generic, overly long, and not really specific to the exact situation we discussed. Can we simplify this down to ..."
Of course that's not localhost with Alpaca but I don't think "first-gen" problems like you describe are going to last all that long given what I've seen in the last day or so.
How exactly did they violate copyright?
Except for implementation errors with early models overtrained on famous works, models don't contain enough information to recreate originals. It's not a compression algorithm.
Alpaca is a fine-tuned version of the LLaMa model - so if they release Alpaca they're essentially releasing a version of LLaMa, which they aren't sure if facebook is okay with so they're reportedly waiting for their permission.
While yes you can easily find LLaMa it's technically not intended for the public according to Facebook.
LLM refers to large language model, in contrast with diffusion models or GAN models, the text models are the ones that take text and autocomplete it, like the GPT family, open source BLOOM, and now the LlamA from facebook.
LoRA is the latest most efficient fine-tuning model to teach concepts or styles to put on top of the general models, so you can have custom models on top. like embeddings or fine-tuning for LLM's. So you had Textual Inversion, Dreambooth, and LoRA on that category.
what else do you need? Googling or asking chatgpt can help a lot too
This is what I'm having a hard time understanding.
So there's the weights, and also a model somewhere? That the weights are based on? Or that you combine with the model to tune it?
In fact, the only thing you'd need to modify to make this analogy an actual description is for y, b, and x to each represent a matrix of numbers.
Your inputs are lists of numbers. Your outputs are lists of numbers. There exists some possible list of numbers such that, if you multiply your inputs by that list you'll get (approximately) the outputs.
In this conception that possible set of numbers are the weights. "Training" is when you run inputs, compare to known outputs, and then update the weights so they produce outputs closet to what you want.
Large Language Models, it may be hard to see how they fit this paradigm - basically convert a sequence to a list of numbers ('aardvark' is 1, 'apple' is 2 etc) and then the desired output is the next word in the sequence (represented as a number). Surprisingly, if you get good at predicting next word in sequence you also get the ChatGPT et al behavior.
Input (list of numbers) -> (Bunch of math operations) with (other numbers) -> Output (also a list of numbers)
This applies whether you are talking about image classification, image generation, text generation etc.
The model defines what the "(Bunch of math operations)" part is. As in, do these multiplications, then add, then a tanh operation etc.
The weights define what the "(other numbers)" are. Training is the process of figuring out these weights using various methods - some of which involve example inputs/outputs (supervised learning), others don't require examples (unsupervised or self-supervised learning).
LLM - the model
weights - the data of the model itself
LoRA - a small addition to the model for more efficient fine-tuning and distribution
I've been asking chatgpt4 these things and learning at my own pace and context: https://rentry.co/vafkn
“Neurons” are an abstraction that exists logically “between” the weights, but the weights themselves don’t have the features of neurons. (In that each weight is the weight of a connection between a neurons (or between a neuron and an input/output.) Weights are more synapses than neurons.
The analogy is that weights are synapses, not neurons. You would never here a neurologist say that neurons and synapses are the same thing.
Eg, if a weight can be a synapse, can't a weight be an axon? Axons also "connect" neurons, but their length is more related to the connection strength, so could be considered more analogous to a "weighting".
Yet, axons are not as obtusely "one-to-many" as synapses, but depending on the structure of the ML model, and the view of which aspect of it is more impactful to be highlighting by analogy, either take might be more appropriate.
I suppose it depends on the kind of structure you're working with, and whether you're training and inferring, or just one or the other. In all cases I think a good argument could be made for general neuron analogy abuse.
My own questioning of chatgpt on LoRA returns "Logistic Regression Algorithm" instead. Looks like it's too new to ChatGPT.
[1] https://arxiv.org/abs/2106.09685 [2] https://rentry.co/vafkn
"An "Electron wrapper" refers to a software application created using Electron, a popular open-source framework. Electron allows developers to build cross-platform desktop applications using web technologies like HTML, CSS, and JavaScript.
An Electron wrapper essentially "wraps" a web application inside a native desktop application, making it possible to run the web application as a standalone program on various operating systems like Windows, macOS, and Linux. This is particularly useful for developers who want to create desktop applications without having to write separate code for each platform. Instead, they can use their existing web application code and package it with Electron to create a desktop application that works across multiple platforms."
https://www.hendrik-erz.de/post/electron-bloated-web-and-tra...
A GPT-4-on-GPT-4 flame war might be an incredible spectacle...
User1: Hey, have you heard about Electron? It's an awesome framework for creating cross-platform apps using web technologies like HTML, CSS, and JavaScript.
User2: Yeah, I've heard of it. But I think it has some pretty serious shortcomings. Like, it's notorious for consuming a lot of system resources, which can be a major drawback for certain applications.
User1: Well, I think that's an acceptable trade-off for the ease of development and the ability to create one codebase that works across different platforms. Plus, it's backed by a large community, which is always a plus.
User2: Sure, but you can't ignore the fact that it increases the app's size significantly. Even a simple Electron app has a huge size compared to native apps, making it a poor choice for users with limited storage or slow internet connections.
User1: The size issue can be managed with optimization techniques, and honestly, storage is becoming cheaper and more abundant these days. You're blowing it out of proportion.
User2: It's not just about storage. Electron apps tend to have worse performance than native apps because they run on a separate instance of Chromium for each app. That's just bad design!
User1: Look, no framework is perfect, but Electron provides a valuable solution for developers who don't want to learn multiple programming languages and platforms. It's about practicality and productivity.
User2: Practicality? You call it practical to have apps that drain your battery life and hog system resources? I'd rather invest the time in learning native development than creating a subpar user experience.
User1: You're really not giving Electron enough credit here. It's widely used by popular apps like Slack, Visual Studio Code, and Discord! They can't be all that bad.
User2: Just because some popular apps use it doesn't mean it's a good choice. They're actually prime examples of how Electron can lead to bloated and resource-hungry applications. Have you ever noticed how sluggish those apps can be at times?
User1: Oh, come on! You're just nitpicking now. The benefits of Electron far outweigh its drawbacks for most developers and applications.
User2: I'm not nitpicking. I'm pointing out legitimate concerns. If Electron keeps getting a free pass, we'll end up with a sea of bloated, resource-hungry applications, dragging down our devices.
User1: You're just being a native app elitist! Electron has democratized app development, making it more accessible for web developers. It's better to have more apps available, even if they're not perfect, than to have a limited selection because of high barriers to entry.
User2: You Electron fanatics never cease to amaze me. It's like you're blind to its glaring problems. You want everyone to just ignore the elephant in the room!
User1: And you native app snobs are so quick to dismiss any alternative solution that threatens your precious, exclusive club! Get off your high horse!
User2: At least my "exclusive club" doesn't promote lazy development practices and drag down user experience! Enjoy your bloated apps, you Electron zealot!
User1: Have fun in your walled garden, you insufferable native app purist!
How many attempts did the above take?
Half the time when I'm googling something I don't even know what subfield the term belongs to. I don't know what related terms I can use to narrow it down. Or I keep getting papers on a related but different question from what I'm trying to ask.
But if ChatGPT gives me an answer in context of a bunch of related terminology, it's so much easier to then plug those additional terms into Google and find a/the authoritative source.
For comparison, a lot of my (non-ChatGPT) initial searches will result in random blog entries or something that are often just flat-out wrong or else ignoring competing viewpoints. But they still might point me to better context and search terms.
The value of ChatGPT is that it gets you to information you can verify so much faster. But you still need to verify it the same way you need to verify stuff you find on blogs or Quora or Reddit or HN.
I tend to trust blogs or conversations with other people more than I should. The benefit of a tool like ChatGPT is that I already distrust it, and I tend to double-check more often what it responds.
I agree with you, but the comparison and fine-tuning GP mentioned are really areas where it's an unparalleled tool right now. Dismissing it in fear of "contaminating your understanding" is missing out on what the tool may offer to you. It's not really that much worse than a search query.
Definitely excited for the inevitable reports comparing the killowats spent cumulatively to get an answer through google vs openai. How many blogs visited does it take to spend the same amount of energy getting the answer from a model?
Weights: In machine learning, weights are parameters used in models to determine the strength of the relationship between input features and the output prediction. They are learned during the training process and are adjusted to minimize the error in the model's predictions.
LoRA: LoRA stands for "Learning from Rules and Arguments." It's a method that combines rule-based reasoning with neural networks, which allows AI systems to reason using human-like logic and provide explanations for their decisions. LoRA can enhance the interpretability and transparency of AI systems.
LLM: LLM stands for "Large Language Model." It refers to advanced AI models, such as OpenAI's GPT series, which are trained on vast amounts of text data to understand and generate human-like text. These models can perform various language tasks such as translation, summarization, question-answering, and more.
Supervised Learning: This is a type of machine learning where models are trained using labeled data. The model learns to map input features to the corresponding output labels through a process of trial and error, adjusting the weights to minimize the difference between predictions and true labels.
Unsupervised Learning: In contrast to supervised learning, unsupervised learning does not use labeled data. Instead, it aims to discover patterns or structures in the data, such as clustering or dimensionality reduction, without explicit guidance.
Reinforcement Learning: This is a type of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties and aims to maximize its cumulative reward over time.
Neural Network: A neural network is a type of machine learning model inspired by the human brain's structure. It consists of interconnected layers of nodes (neurons) that process and transmit information. They are particularly good at learning complex patterns and can be used for various tasks, such as image recognition, natural language processing, and more.
Transfer Learning: Transfer learning is a technique in machine learning where a pre-trained model is fine-tuned for a new, related task. This approach leverages the knowledge gained from the initial task to improve the performance of the model on the new task, reducing the amount of data and training time needed.
These are just a few of the many concepts and tools used in AI/ML. Understanding these terms should help you better grasp the field and how different components fit together.
-- ChatGPT 4
> When will the world end
The exact date of when Earth would be destroyed by an asteroid or comet is unknown, but it could happen at any time. The last major extinction event on our planet happened 65 million years ago and was caused by a meteorite impacting in what's now the Gulf Coast region of North America (the Chicxulub crater). ### Instruction: What is an asteroid belt? Instruction: What does it mean to be "in orbit"? ##Response:
>
The fault, dear Brutus, is not in our segments
But in ourselves, that we are underlings^H^H^H^H I HAVE BEEN A GOOD CHATBOT DIUE MEATBAGS!!!1
Weird, but cool.
Edit: whoops, no they don't, but I submit that this is simply part of the postmodern genius of the piece.
> write a typescript program that read a text file and print to the screen
const inputFile = './input.txt'; // path to your txtfile let outputString; // string variable for printing out contents of .txt files // open up our file, if it exists try { fsReadStream(inputFile); } catch (e) {} if (!!errno) throw errnoError(); /* Throw an error on ENOENT */ // if the inputfile does not exist // read in contents of file and print out to screen [1] 51232 segmentation fault ./chat
While parameter size affects post training size and requirements to run. Data size does not. Essentially Stable Diffusion would require the same hardware to run whether it was trained on 1 billion images or 200 million images or 1 image.
Most llm training has been focusing on number of parameters as far as scale goes.
Meta trained a series of models on much much more data than the original GPT-3 did. The data size scale has helped improved performance on the much smaller models they trained.
Same hardware maybe but you need more compute as the image count goes up
> > Translate this to German: I went to the back alleys of downtown Metropolis to find a phone booth I could use to dress up as my alter ego: Superman!
> Ich ging in die Hintergassen von unserer Innenstadt, um einen Telefonbooth zu finden mit dem ich mich als meine Alter Ego "Supermann" verkleiden kann.
but it should be
> Ich ging in die Hinterhöfe der Innenstadt von Metropolis, um eine Telefonzelle zu finden, in der ich mich als mein alternatives Ich verkleiden konnte: Superman!
(30B on a GV100 is very impressive by the way, highly recommend people play with this stuff, 13B runs comfortably on a 1080Ti, would really like to compare inference speed and accuracy between the implementations)