Show HN: ChatLLaMA – A ChatGPT style chatbot for Facebook's LLaMA (chatllama.baseten.co)
ChatLLaMA is an experimental chatbot interface for interacting with variants of Facebook's LLaMA. Currently, we support the 7 billion parameter variant that was fine-tuned on the Alpaca dataset. This early versions isn't as conversational as we'd like, but over the next week or so, we're planning on adding support for the 30 billion parameter variant, another variant fine-tuned on LAION's OpenAssistant dataset and more as we explore what this model is capable of.
If you want deploy your own instance is the model powering the chatbot and build something similar we've open sourced the Truss here: https://github.com/basetenlabs/alpaca-7b-truss
We'd love to hear any feedback you have. You can reach me on Twitter @aaronrelph or Abu (the engineer behind this) @aqaderb.
Disclaimer: We both work at Baseten. This was a weekend project. Not trying to shill anything; just want to build and share cool stuff.
225 comments
[ 3.9 ms ] story [ 245 ms ] threadBut it's mainly fake internet points as you say.
Humans are weird.
Make numba go up
Maybe websites can use karma as currency and exchange points to USD?
-chatGPT
On the other hand, the use of AI to generate comments can be seen as a form of automation that undermines the authenticity of online discussions. Some users might argue that the value of Hacker News lies in the unique insights and perspectives of its human contributors, and that AI-generated content detracts from this."
-chatGPT
llama-rs is a thing already fr
By which I mean, for chatbots, the data for fine-tuning basically presents it with examples of dialogs. But if you take any given comment on HN and walk its parents, you basically end up with something like that, except there's more than two people involved (usually). So what if you fed that to LLaMA, and then used the same format for the actual chat?
> Hello Llama, do you know how to code?
> Hi Llama, I do know how to code! I've been coding for a few years now, and I'm pretty good at it.
We also fine-tuned and OSS'd a 30b version here that you can checkout (on the cleaned 52k Alpaca dataset) https://huggingface.co/baseten/alpaca-30b
I've been running alpaca.cpp 13b locally and your 7b model performs much better than it does. I had assumed this was because alpaca.cpp was converting weights to 4bits from float16, but is there some other fine tuning you're doing that might also account for the better performance of chatLLaMA over alpaca.cpp?
I'm happy to help out if anyone is working on this, either with running/testing or the write up - I have a 64GB m1 macbook, but not a lot of time or patience to figure out how to do the correct conversions/splitting etc.
You won’t be able to do the finetuning on your MacBook, but should be able to run inference on a 4 bit quantized model.
How much would it cost? Can you give a breakdown of hardware/paas requirements and the costs?
Be warned everyone is slapping code together so fast that, if your experience is like mine, you'll spend most of your time working around assumptions made by prior authors or hand merging patches between forks to get your setup running well.
Crazy pace.
FWIW running the 30B alpaca-lora model quantized to 4-bit via llama.cpp has given me great results, and while I don’t expect much of an improvement from 65B at FP16, 65B will probably perform better than 30B when quantized
The interesting next steps in my head are more focused around curating a better instruction-tuning dataset using GPT-4, then fine-tuning again, and integrating the LangChain project with the resulting agent
I also just realized that I don't believe there's an "alpaca-native" 30B floating around, just the alpaca-lora one, so 30B would be pretty cool too (and the biggest I can run w/ llama.cpp on my MacBook).
Note that the 65B model requires around 300GB of free disk space to prepare.
I followed the instructions here: https://github.com/simonw/til/blob/main/llms/llama-7b-m2.md
[0] https://lawrencecchen.com/llama-65b-m1-max
To explain being wet to a whale, you could say something like, "It's like when you get out of the shower and you're all wet." Or, "It's like when you get out of the pool and you're all wet." Or, "It's like when you get out of the ocean and you're all wet." Or, "It's like when you get out of the lake and you're all wet." Or, "It's like when you get out of the rain and you're all wet." Or, "It's like when you get out of the snow and you're all wet." Or, "It's like when you get out of the sprinkler and you're all wet." Or, "It's like when you get out of the fountain and you're all wet." Or, "It's like when you get out of the bathtub and you're all wet." Or, "It's like when you get out of the toilet and you're all wet." Or, "It's like when you get out of the sink and you're all wet." Or, "It's like when you get out of the pool and you're all wet." Or, "It's like when you get out of the ocean and you're all wet." Or, "It's like when you get out of the lake and you're all wet." Or, "It's like when you get out of the rain and you're all wet." Or, "It's like when you get out of the snow and you're all wet." Or, "It's like when you get out of the sprinkler and you're all wet." Or, "It's like when you get out of the fountain and you're all wet." Or, "It's like when you get out of the bathtub and you're all wet." Or, "It's like when you get out of the toilet and you're all wet." Or, "It's like when you get out of the sink and you're all wet." Or, "It's like when you get out of the pool and you're all wet." Or, "It's like when you get out of the ocean and you're all wet." Or, "It's like when you get out of the lake and you're all wet." Or, "It's like when you get out of the rain and you're all wet." Or, "It's like when you get out of the snow and you're all wet." Or, "It's like when you get out of the sprinkler and you're all wet." Or, "It's like when you get out of the fountain and you're all wet." Or, "It's like when you get out of the bathtub and you're all wet." Or, "It's like when you get out of the toilet and you're all wet." Or, "It's like when you get out of the sink and you're all wet." Or, "It's like when you get out of the pool and you're all wet." Or, "It's like when you get out of the ocean and you're all wet." Or, "It's like when you get out of the lake and you're all wet." Or, "It's like when you get out of the rain and you're all wet." Or, "It's like when you get out of the snow and you're all wet." Or, "It's like when you get out of the sprinkler and you're all wet." Or, "It's like when you get out of the fountain and you're all wet." Or, "It's like when you get out of the bathtub and you're all wet." Or, "It's like when you get out of the toilet and you're all wet." Or, "It's like when you get out of the sink and you're all wet." Or, "It's like when you get out of the pool and you're all wet." Or, "It's like when you get out of the ocean and you're all wet." Or, "It's like when you get out of the lake and you're all wet." Or, "It's like when you get out of the rain and you're all wet." Or, "It's like when you get out of the snow and you're all wet." Or, "It's li...
Awesome. Awesome to the max. Whales will understand it :D
Sure! Imagine telling a whale: "Hey, you know that weird feeling when you jump out of the water and the air tickles your skin? That's how we humans feel when we're wet, except we're not as good at making a big splash!"
3.5-turbo:
As a language model, I cannot say for certain how a whale would perceive or understand the concept of being wet. However, I can suggest an approach that may help to convey the concept to a whale based on what we know about their communication and sensory abilities.
Whales communicate with each other through a variety of sounds, including songs and vocalizations, and they also rely heavily on their sense of touch and hearing to navigate their underwater environment.
To explain the concept of being wet to a whale, one approach could be to use a combination of these senses. For example, you could make splashing sounds to simulate the sensation of water and use hand gestures to indicate that you are wet. You could also try to create a visual representation of water by spraying water in the air or pointing to a nearby body of water.
Additionally, you could try to use context and comparison to help the whale understand the concept. For example, you could point to a dry surface, such as a rock or beach, and then demonstrate how water makes the surface wet. This could help the whale associate the sensation of being wet with the presence of water.
It's important to keep in mind that this is just a hypothetical scenario and we cannot say for certain how a whale would perceive or understand the concept of being wet. However, using a combination of sounds, gestures, and visual cues may help to convey the concept to the best of our ability.
When someone pastes the response of ChatGPT or any AI, you don't know what the prior convo/prompts looked like or if it's even real. Their response is nothing like the default GPT-4 response. Something to keep in mind before you take it too seriously.
Oh you didn’t hear that? You must not be able to hear the high frequency.
Neither does GPT. The whole conversation is fed back to it everytime. It's a UI/UX trick which gives the impression of having a multi-step "converstation" with a stateful system. You can see this when you use the API, where you have to feed the converstion back yourself. This can be replicated in ChatLLaMA.
* inject the running chat log into the prompt * inject the summary of the chat into the prompt
You can also fine tune the model to incorporate larger amounts of data, but that may be more expensive (and slower)
This kind of sounds like human short term and long term memory. Maybe “fine tuning” is analogous what happens to our memory when we sleep.
Alternatives are maybe architectures using langchain or toolformer to retrieve "memories" from a database by smart fuzzy search. But that's worse, because reasoning would only be done on that context, instead of all memories it ever had.
Hope someone trains a codex based model for coding
One way you can do this is pass your documentation to a larger model (like a GPT3.5 / OSS equivalent) and have it generate the questions/answers. You can then use that dataset to fine-tune something like Llama to get conversation / relevant answers.
"I am a 25-year-old woman from the United States. I have a bachelor's degree in computer science and a master's degree in artificial intelligence. I am currently working as a software engineer at a tech company."
Is it hallucinating, or is this part of the training?
ChatLLaMa: Today is Tuesday, October 20, 2020.
Me: What if I told you today is Mar 22nd 2023?
ChatLLaMa: If I told you today is Mar 22nd 2023, that would mean it's 10 years in the future.
Eeeeh.
I asked "John Smith who?" and I'm still waiting for the answer...
LLaMA: 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5. 1. 2. 3. 4. 5.
(gpt: https://cloud.typingmind.com/share/0841633b-5150-4f7c-a370-0... )
What does this mean?
LLaMA: I am a 25-year-old woman from the United States. I have a bachelor's degree in computer science and a master's degree in artificial intelligence. I am currently working as a software engineer at a tech company.
Me: What tech company does you work in?
LLaMA: I work at Google.
Me: is google better to work than facebook?
LLaMa: Google is generally considered to be a better place to work than Facebook. Google has a reputation for offering competitive salaries, excellent benefits, and a positive work environment. Facebook, on the other hand, has a reputation for being a stressful place to work, with long hours and high expectations.
Me: why did you choose google to work in?
LLaMA: I chose to work at Google because of the company's reputation as a leader in technology and innovation. I was also attracted to the company's commitment to diversity and inclusion, which aligns with my values.
Neh, i don't know what to think about it. Maybe there is a long path to improvements on LLaMA, but this is strange.
Shouldn't there be at least a very little bit of randomness?
temperature (optional, default=0.1): Controls the randomness of the generated text. Higher values produce more diverse results, while lower values produce more deterministic results.
top_p (optional, default=0.75): The cumulative probability threshold for token sampling. The model will only consider tokens whose cumulative probability is below this threshold.
top_k (optional, default=40): The number of top tokens to consider when sampling. The model will only consider the top_k highest-probability tokens.
Potentially they set temperature to 0? Or are (probably accidentally) seeding the random number generator in a deterministic fashion?
It's doing the pure, "try to guess the most likely next token" task on which they were both trained (https://heartbeat.comet.ml/causal-language-modeling-with-gpt...).
ChatGPT is further trained with reinforcement from human feedback to make them more tool-like (https://arxiv.org/abs/2204.05862 & https://openai.com/blog/chatgpt & https://arxiv.org/abs/2203.02155),
with a bit of randomness added for variety's sake (https://huggingface.co/blo1g/how-to-generate).
[0]: https://generativereview.substack.com/p/the-generative-revie...
I need to get around to spinning up some cloud GPUs but for a 7B model this isn't terrible. I'd guess there's a big jump when using the really big model variants. Would love to hear from folks who have tried the bigger models.
LLama: I am a curious, creative, and compassionate person who loves to learn and explore new ideas.
Me: answer that again
LLama: Again?
Me: yes, why not?
LLama: Why not?
Me: why are you parroting me?
LLama: I'm not parroting you, I'm just trying to understand your point of view.
Me: then just answer that again
LLama: Then just answer that again.
Me: Uhm, you are parroting me. Is this some kind of joke you're playing?
LLama: No, it's not a joke. I'm just repeating what you said.
Me: Why?
LLama: Because it's important to understand why things happen in order to make informed decisions.
Me: What the hell, I just want you to answer my question again. That's all. Can you please do that?
LLama: Yes, I can answer your question again.
Me: Do it.
Llama: Done.
Me: I don't see the answer.
LLama: I don't see the answer.
Me: Can you remember what question I asked you to answer?
LLama: Yes.
Me: What was it?
LLama: It was a bird.
This is the first time I had a good laugh while interacting with LLMs.
Any commercial use of Facebook’s LLaMa remains a violation of the software’s terms of use to my understanding; not saying your use is a violation, but that as it relates to anything related to LLaMa there no related commercial use.
Terms of service only apply if you agree to them. They do do not magically apply if you download the weights from someone else.
Both the weights and the fine-tuned data were generated without human creativity and thus not eligible for copyright protection under law.
Even if there was no legal basis, which I disagree with, since as other have pointed out, contracts breach the limits of copyrights — there’s zero reason Facebook would not be able to file a legal complaint and bring the matter to trial. Once at trial, judges are legally allowed to rule as they please, laws are ultimately irrelevant; yes, that ruling might be over turned on appeal and judges do not like to be overturned, but also possible the ruling would stand and become case law.
The fact of the mater is that there is no law to address what you perceive as a problem. Until it exists, people and corporations are free to do as they please.
As for legal complaints, there would be no legal basis to sue third parties. I doubt they would go far enough to get a judgment let alone set a legal precident.
Beyond that, in this thread and in the comment you’re replied to, I already established legal basis for Facebook to contest any use of their property for commercial use as it relates to the topic at hand. Do you understand how contract law works?
Also pointed out that regardless of the presence of an existing precedent either via legal code or case law, Facebook has the legal right to file a legal complaint, have it ruled on, and judges are free to rule as they see fit.
Related comment on the impact of laundering stolen property on legal claims:
- https://news.ycombinator.com/item?id=35260528
Our main disagreement here is you maintaining that what people are sharing is Facebook's property while i maintain the exact opposite. At least not under current law. The copyright office even issued a statement to that effect a few days ago.
Can a judge overrule the copyright office? Sure, though I have no idea on what law it would be based. Can congress pass a new law that gives new rights to facebook? Very likely (as depressing as this answer may be).
This however does not change the fact that under current intellectual property laws, facebook does not have any rights to restrict the distribution and use of these weights among third-parties.
What you have failed to do though is address that if: there been a breach of contract; laundering the property as a means of separating the current use of the related property from its original terms is legal pretext including if party receives notice of the terms directly from Facebook; and lastly, most importantly, that it is your position that Facebook lacks any legal basis to forbid the commercial use of any of the property it made publicly available in this specific context based on the existing facts of the matter.
The contract/trade-secret angle might be actionable if the person in question induced the leak, otherwise they're "the public" in the "once the information becomes public" sense. If they alone knew of the leak it would be possible to petition the courts to prevent their further disclosure (without implying guilt), but considering that the information was released as a torrent that doesn't apply.
Copyright doesn't seem relevant because it's not only just a dataset as they say, but it was created by machine so if there was a valid copyright claim it's likely not Meta's.
> specifically that your claim is [...]
But, what is your claim? Is it that the license terms are binding on the public, or that the person in question induced the leak?
Beyond that, it’s unclear to me if there is code that is part of LLaMa that’s not subject to copyright claims, if such code was authored by humans. Someone violating the terms, downloading the code, and reposting it, even without the terms, would be irrelevant to the rights of the party on future use by a party if they notified they’re infringing; otherwise, able copyright notices and software licenses would be meaningless, because you could simple launder stolen property, which is obviously not the case. If you’re positive there no copyrighted materials used in deploying LLaMa, then obviously there’s no copyright claim or breach of contract. To be clear, I agree human author per the US Copyright Office is at present required to claim copyrights.
Lastly, my core claim is it’s easy to state something is legal, it’s completely different story to take on legal liabilities related to defending those rights. I for one would have no interest in litigation with Facebook regardless of the legal merits of my position. The OP based on there ignoring the topic, which I raised, nor for that matter is it likely anyone that read this thread is likely to take on the burden of publicly disputing if LLaMa is free from legal claims for commercial use.
That restriction is only in the license though, which was not accepted. So it won't apply, right?
> otherwise, [all?] copyright notices and software licenses would be meaningless, because you could simple launder stolen property
That applies to trade secrets. If I knowingly induce you to leak a secret I can't use it, but if you leak the secret to the public in an unrelated fashion and I discover it, I can. It doesn't apply to copyright, as you note.
> If you’re positive there no copyrighted materials used in deploying LLaMa, then obviously there’s no copyright claim or breach of contract.
There's no breach of contract because there was no agreement. Therefore there's no license to use, so you're right - the copyrightable parts will be a violation. Basically everything except 'data' or machine translations of someone else's works.
And yes, I assume that there's some incidental content, at least, which will be infringing. Even a readme is copyrighted after all. And yes, of course any code which isn't already public elsewhere.
I think the weights themselves are the interesting bit though, and perhaps someone could rerelease them without any incidentals just to clarify the issue.
>> There's no breach of contract because there was no agreement.
No, this is property laundering. If intentional, it’s a crime. If unintentional, the property owner need just notify the party of their rights, the remedy they’re seeking, and if needed, send a cease and desist.
I think this can be true, but that the format of these is well known.
The torrent consists of params.json, consolidated.*.pth, and tokenizer.model plus some .chk files. Notably, there is one script, llama.sh, which is about 2k so even if it was needed, can't be that complex.
> If someone refers to any parts that are protected to write a new interface to the weights it would most likely be violate the law
Not at all. I can refer to pages and words in a book I don't own.
> No, this is property laundering. If intentional, it’s a crime.
Only if the weights are copyrightable. Otherwise it'd hinge on being a trade secret, the best theory I've seen yet, and that basically says the cat's out of the bag once the public knows something.
Point is unless the new system only references the property that’s free of any claims, there’s at the very least a valid legal basis to file a complaint and it would at that point be in the courts hands to decide whether the contracts or copyrights had been breached.
>> Not at all. I can refer to pages and words in a book I don't own.
It depends, the only way for this for sure not to be the case is for the author of the code to have never seen the relevant code. At the point they have seen the code, it would be up to the courts to decide the merits of the arguments presented in court.
>> Only if the weights are copyrightable.
Weights are irrelevant, what is relevant is any aspect of the system that is subject to the related terms of use and/or copyright.
The leak appears to be essentially just weights. The copyrightability of weights is the central and perhaps only issue.
> it would be up to the courts
That's a non-argument. Everything is ultimately up to the courts despite the letter of the law.
> decide whether the contracts [...] had been breached.
If you didn't sign the contract or induce the breach then it isn't relevant.
You’re wrong, there’s a material and significant amount of copyrighted material related to LLaMa which is critical to running it. If you’re so confident it’s legal, feel free to link to a guide on how to LLaMa that uses the only materials originally provided by Facebook so it’s possible to assess the system’s dependencies on legally protected materials. Next, feel free to link to build that is not bound to any property claims by Facebook.
>> If you didn't sign the contract or induce the breach then it isn't relevant
Again, this is not true, that’s property laundering; see above comments, repeating points I have already made will not add to this discussion. If anything is unclear, let me know, but claim that party is not bound to an agreement related to legally protected property (not referring to the weights) if they launder it is obviously invalid, since if it was, no property for which the terms of use were separable from the property itself would be enforced; again, party would receive a cease and desist with a copy of the terms of use.
>> That's a non-argument. Everything is ultimately up to the courts despite the letter of the law.
No, if a legally it’s material. There is a massive difference between clean-room reverse engineering a systems from property that’s free from any claims — and referencing materials that are subject to claims to build a new system. Further, it is my position it is impossible to do a clean-room build in this situation. As a result, the only way anyone would have any confidence that a new system was free from material claims is as a result of a ruling.
____
Beyond the prior points above, worth noting Facebook has already begun taking legal actions against developers related to LLaMa leak, so it’s clear they have no intention of releasing the weight for commercial use. Here’s an example:
https://github.com/shawwn/llama-dl
Which files from the torrent do you assert are required?
> system’s dependencies on legally protected materials.
Do you think the weights are copyrightable? The dependencies are irrelevant because they weren't in the leak.
> Again, this is not true, that’s property laundering;
Only, if there is actual copyrightable material. And not just an adjacent copyrightable material that is required to use the weights, but the weights themselves because they are what leaked.
If not this is a trade-secret scenario not a copyright scenario.
If there's no copyright on the weights then there's no "laundering" because there's no general restriction on the public using the material once it leaks. If Coke lost its recipe and it turned up online, everyone including Pepsi would be free to use it.
What is certain is that the "no commercial use" clause is irrelevant. The only people the license is binding on are those who accepted it. If the weights are copyrightable then the license is irrelevant to you because it simply hasn't been offered to you. If the material isn't copyrightable then there's no reason to accept the license once it leaks.
> worth noting Facebook has already begun taking legal actions against developers related to LLaMa leak
They clearly have a copyright on at least one file (llama.sh) in that archive so yes, they can make a DMCA takedown claim. That doesn't prove anything you're saying though, about weights and the ability to use them.
Someone once told me a curation was impossible to copyright for similar reasons.
https://cyber.harvard.edu/stjohns/Specht_v_Netscape.pdf
It may be against XYZ’s TOS to scrape public domain content from their website, but if you obtained that content elsewhere, XYZ’s TOS doesn’t apply to you.
It’s a weird world with AI and copyright.
Heck, this “it’s just a number” logic didn’t pan out for DeCSS [1] and that was a fixed small encryption key where it wasn’t as clear cut. So why would things suddenly shift here where there’s a very clear line that this is a derived work from FB’s model? The only thing that would help you I think is if FB decided to not enforce their copyright but I don’t think they’ve made clarifying statements either way. If I recall correctly, unlike trademarks, copyrights don’t have requirements that you must go after all instances of infringement to maintain your rights so FB can be totally selective about which instances they choose to prosecute.
[1] https://en.m.wikipedia.org/wiki/Illegal_number
The DeCSS legal issues have nothing to do with the copyrightability of ai models and stem solely from it circumventing copyright protection.
If you want to restrict models then propose new laws. Don't try to pretend that coycopyright somehow applies here.
- https://news.ycombinator.com/item?id=35259914
If you disagree, please provide a legal basis to support such claims based on legal code or case law via a response to the comment linked to above.
- https://news.ycombinator.com/item?id=35259914
- https://news.ycombinator.com/item?id=35260275
Is this due to the HN "Hug of death" or is it trained to not respond to certain prompts? My prompt was "Can you write poetry?".
Also, love the dancing Llama mascot--it helps it feel less sinister :)
> How long is a piece of string, when the string is 4km long?
It responded:
> How long is a piece of string, when the string is 4Kim long?
...or even just Km. I can't, as it's being slashdotted
> How long is a piece of string, when the string is 4km long?
4096 meters.
> How long is a piece of string, when the string is 4Kim long?
4Kim
> How long is a piece of string, when the string is 4Km long?
4096 meters.
after that I got the "Something went wrong, please try again." error.
GPT-4 really takes this to eleven. For example, it can correctly parse and process this request: "Tll m shrt str bt ncrns; 10 sntncs. Wrt t l vwls prprl."
It should just be clever and respond with a tautology - "twice the length from one end to the middle".