Thanks, so from that PyTorch doc it seems that pickle format has the filenames of the model classes, but not the classes themselves. I'm sure someone will figure it out though!
I already got the 7B model to generate text using my GPU! The 1st example prompt generated this:
[I believe the meaning of life is] to be happy, and it is also to live in the moment. I think that is the most important thing.
I'm not really a party girl. I'm not a girl's girl. I have a really small group of close girlfriends and that's all I need.
I believe in equal rights for everyone.
I'm not a rebel. I don't really rebel against anything. I'm a very traditional girl, very loyal. I'm a mum's girl and I'm a dad's girl.
People have a right to know what's going on.
I don't care about the haters, because at the end of the day they're just going to have to
deal with themselves.
I've been getting more and more into fashion since I was about 16.
I know I'm a little different, but so what? I think that's good. I don't think you should be like everyone else.
It's my birthday, and I'll cry if I want to.
I've always been a huge fan of fashion, and I've always liked to dress up
I could drop the batch size to 5, then the VRAM use seemed to be around 15GB. Some of that I'm sure is not necessary, and if you rewrite the outer products to use less VRAM you might get away with even less. Eventually someone will make a library so you can run it without extra work.
Yeah true, do you think that it a realistic expectation though? I ask this given the events that have led to the leaking of the models. I am genuinely not sure what the optics / real world ramifications are of being publicly associated with projects that leverage models obtained via torrents through either hacking or negligence.
If you look at how much infrastructure was quickly developed around Stable Diffusion, the same might repeat here. This also depends on how useful the model is but from the scores it looks like it's quite useful, and it's "uncensored" unlike commercial "online" models which is valuable on it's own. I suspect Facebook won't care and will be happy to get people to use an offline model since that means Microsoft and Google will make less money from online models. The model itself is licenced under the GPL, but I have no idea what that means when it comes to model weights.
Edit: It looks like it can code, I tried to autocomplete the first 2 lines and it wrote the rest. Local Github Copilot here we come?:
//find index of element in sorted array in O(log(N)) time using binary search
int find_idx(int a[N], int element) {
int low = 0, high = N-1;
while (low <= high) {
int mid = (low + high) / 2;
if (a[mid] == element)
return mid;
else if (a[mid] < element)
low = mid + 1;
else
high = mid - 1;
}
return -1;
}
I wouldn’t jump to that conclusion — the first and last names are not uncommon, and the GitHub user has some attributes (eg Haskell; geographic ties) they do not share with the LinkedIn profile you link.
Unless you have strong evidence of their identity I’d suggest rethinking this.
The Christopher King user who submitted the pull request has a git repo in C++ called “Final - My Homework” made in 2015; the Christopher King you link to completed his BA in Computer Science in 2005. I strongly suspect they’re different people who happen to share the same name.
The user who submitted the pull request is not part of Meta or Facebook Research, and the users who signed off on reviewing the changes don’t appear to be either. I highly doubt Meta will approve the pull request. The models are being distributed by torrent by someone with access to the models, not by Meta themselves as far as we know. They likely still intend to distribute via the form. This is just someone publicizing the torrent link by being cheeky on GitHub.
(As they didn’t reply to my request for the model - I specified it was for personal use and my use case was “I think it would be fun to run it on my own hardware” - I appreciate this little stunt a great deal!)
old school opensource, which is a bit surprising from meta. I wonder how they managed to square that with legal. Someone must have been very good friends with Zuck.
Getting anything that could produce, look like, or smell anything like misinformation out of meta is very hard (for good reason!)
My friends have had repeated push back for various papers because they are ML based and could be in the same room as something that could possible be used by miscreants.
And here we have a LLM that can spit out all sorts of things that are misinformation like.
If their department tried to launch something like Galactica they would have been slapped down and told to think again about what they were doing in life.
This comment took me back, haven't been to astalavista.box.sk in a LONG time - looks like the site is still online in some form (but without the old black and green color scheme)
It seems like the model has been leaked (not by Meta) and is being distributed via a torrent. Someone has created a PR to the repo as a joke, suggesting that instead of filling out a form and waiting to be granted access (which is the official way to get access to the model), that you could just download it via the torrent.
In case it's not clear what's happening here (and from the comments it doesn't seem like it is), someone (not Meta) leaked the models and had the brilliant idea of advertising the magnet link through a GitHub pull request. The part about saving bandwidth is a joke. Meta employees may have not noticed or are still figuring out how to react, so the PR is still up.
(Disclaimer: I work at Meta, but have no relationship with the team that owns the models and have no internal information on this)
The folder structure definitely looks like model weights, I didn't download or run it though so for all I know it only generates the words to "Never Gonna Give You Up".
> Meta employees may have not noticed or are still figuring out how to react
Given that the cat is out of the bag, if I were them, I would say that it is now publicly downloadable under the terms listed in the form. It is great PR, which if this was unintentional, is a positive outcome out of a bad situation.
Considering the opt-models that they've previously released publicly, intentionally, I'd say that facebook is better open source AI than OpenAI even without fumbling.
It's fairly easy to obtain the weights. I've had two of my friends downloading these weights and sharing them with me, so it's probably not surprising that the weights got leaked.
Anyone can submit a request form and enter their email address to request access to the weights. People who have a .edu email and are involved in deep learning are likely to succeed and get a download link sent to their email.
Since the point seems to be lost on some of the early commenters, this appears to be a cheeky PR by someone unaffiliated with Facebook, suggesting that they put a magnet link to (what seems to be) a leak of the model weights along with the previously existing invitation to apply to receive them on their own page.
Funny. iirc some of the big tech (I think it was Google?) use torrents internally to deploy very large images to servers. Piracy is not the only use case!
Now that I think about it I wonder why we don't see it being used to distribute packages for linux distros. Seems more flexible than the current mirror system.
More overhead, torrents being blocked or disliked because of their association with piracy, difficulty to distribute updated versions of files (package indexes)?
Modern AAA games have hundreds of GBs worth of content, and a game is a single unified package. Linux distros have tens of thousands of packages, many of them in less than a MB in size, with different update frequencies and different users. You would need to generate massive amounts of torrents.
There seem to be a lot of confused commenters here. This is the content of an as-yet-unmerged pull request, and presumably not something that Facebook approves of.
Is there anything stopping anyone from using this for commercial purposes? I know that when you fill in the google form you need to agree to noncommercial use, but someone downloading this will never have agreed to that licence agreement.
I don't know. Is there anything stopping you using the latest Miley Cyrus album for commercial purposes if you downloaded it via torrent and never agreed to any licencing terms?
IANAL, but I imagine it's a legal grey area if the weights can be copyrighted? Works produced by purely mechanical means don't normally meet the threshold of originality.
and, copyright rarely bites if you use something without publishing/redistributing it.
It would be like playing copyrighted music in your office without permission. Perhaps technically illegal, but your customers will never know what music your Devs were listening to...
I am quite confident that using this model for commercial purposes will, if detected, land you in quite a legal quagmire that almost certainly sides in favor of Meta.
And even if it did not, Meta certainly has a more capable legal team with more cash to spend than the average HN user.
I don't think weights can be copyrighted (unless overfit on your own other copywritten work), and I don't think weights shared broadly with the research community can be considered trade secrets. And even if they were trade secrets despite the wide sharing, only the people that leaked them could get into trouble, right? They aren't trade secrets anymore once you didn't keep them secret and they are out on torrents.
IDK, its more like finding recipes to many great restaurant chains all mushed together by a 5th grader whose uncle stole it from them, on the sidewalk. looks like a grey area to me legally but IANAL.
That's what Facebook and OpenAI are doing. They consumed tons of copyrighted content without permission and are now using it for commercial purposes. So using their model seems fair game.
Generally, you'll need multiply model size by two to get required amount of video RAM. There are 4 sizes, so you might get away with even smaller GPU for say 13B model.
opt-175B weights are already openly available as I understand. Hugging-face also has openly available weights for a 176B parameter LLM called Bloom. Is LLAMA offering something over and above these?
Yeah, their recent papers show the smaller LLAMA models outperforming the major LLMs today, and they also have bigger models. This isn't just an alternative, it's a multi order of magnitude optimization.
In principal you can run it on just about any hardware with enough storage space. It's just a question of how fast it will run. This readme has some benchmarks with a similar set of models (and the code has support for even swapping data out to disk if needed): https://github.com/FMInference/FlexGen
As the models proliferate, I guess we'll be finding out soon. The torrent has been going pretty slow for me for the past couple hours, but it looks like there are a couple seeders, so eventually it'll hit that inflection point where there are enough seeders to give all the leechers full speed downloads.
Looking forward to the YouTube videos of random tinkerers seeing what sort of performance they can squeeze out of cheaper hardware.
The 30B is 64.8GB and the A40s have 48GB NVRAM ea - so does this mean you got it working on one GPU with an NVLink to a 2nd, or is it really running on all 4 A40s?
Is there a sub/forum/discord where folks talk about the nitty-gritty?
> so does this mean you got it working on one GPU with an NVLink to a 2nd, or is it really running on all 4 A40s?
it's sharded across all 4 GPUs (as per the readme here: https://github.com/facebookresearch/llama). I'd wait a few weeks to a month for people to settle on a solution for running the model, people are just going to be throwing pytorch code at the wall and seeing what sticks right now.
> people are just going to be throwing pytorch code at the wall
The pytorch 2.0 nightly has a number of performance enhancements as well as ways to reduce the memory footprint needed.
But also, looking at the README, it appears that model alone needs 2x the model size, eg 65B needs 130GB NVRAM, PLUS the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes = 17GB for the 7B model (not sure if it needs to increase for the 65B model), but maybe a total of 147GB total NVRAM for the 65B model.
That should fit on 4 Nvidia A40s. Did you get memory errors, or you haven't tried yet?
So since making that comment I managed to get 65B running on 1 x A100 80GB using 8-bit quantization. Though I did need ~130GB of regular RAM on top of it.
It seems to be about as good as gpt3-davinci. I've had it generate React components and write crappy poetry about arbitrary topics. Though as expected, it's not very good at instructional prompts since it's not tuned for instruction.
People are also working on adding extra samplers to FB's inference code, I think a repetition penalty sampler will significantly improve quality.
The 7B model is also fun to play with, I've had it generate Youtube transcriptions for fictional videos and it's generally on-topic.
Yes, LLaMA is state of the art in several domains. The model was trained on a much larger data set than most models which is why it is higher scoring vs other models with similar numbs of parameters. This represents millions of dollars in compute time alone for the training.
This should lead to quite a lot of innovation and it’s inevitable that someone will get these working slowly on your average MacBook.
opt-175B doesn't exist; the largest one is opt-66B. And, at least in the tests I've run (not with the biggest one, but only up to a dozen billion parameters), all the opt models severely underperform with respect to even much smaller models. To the point that the launch of OPT (before BLOOM) was literally advertised as "the biggest OpenSource language model released to date", because they couldn't push on much else.
BLOOM goes indeed up to 175B parameters, and is certainly better than OPT. However, at least in my specific tests, it's still significantly inferior to OpenAI models, and actually on par with a few smaller models. There's also a "newer" fine-tuned model, called BLOOMZ, but at least in my tests it's even worse. Of course, that depends a lot on what you ask the model to do...
If LLAMA can indeed match OpenAI products, and do so with much fewer parameters, then it would be really great, and I'd really like to test it. However, even if the weights are now in the wild, using them would be clearly against the user agreement, and there's no way I'm going to do that in my work time :-) so let's hope Meta will come to sense and release them with a more friendly set of terms...
If you want to do it the cheap way by buying used stuff, the most expensive parts are:
- $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard
- $5000 for 6x RTX 3090
- $350 for two 1500W PSUs
- $700 for 256GB RAM
You will also need PCIe extenders and perhaps some watercooling. And find a suitable case.
The 2-card NVLink bridges are between $100 and $300 each (you nay want 3).
All in all i think less than $10k.
Are there any official checksums available? I'm happy to see this, even if it's an unsanctioned stunt, because I think it's really pathetic of meta to want to gatekeep their "open" model. But ML models generally can execute arbitrary code, I'd want to make sure it's the real version at least.
They could contain arbitrary code... But typically do not. That means that with the right viewer application it will be trivial to know for sure.
It isn't like a multi gigabyte game for example, where knowing if there is any malicious code could easily be a multi-month reverse engineering project to get to the answer of 'probably not, but we don't have time to check every byte with a fine tooth comb'
In practice, who's going to bother checking the language model? All the code that runs Stable Diffusion or other Hugging Face models that I've seen just downloads the model dynamically, then uses it without asking question. That's a pretty low-hanging supply chain attack waiting to happen, I believe.
Anything that loads pickles from sources your unsure of can contain executable code. There were a few samples a couple month ago showing distribution on huggingface.
Because human-readable text-based formats are really inefficient to both download and load, especially when in the hundreds of GB range. And no human cares to read billions of weights.
Agreed. However, there are much better formats than Python pickles for exchanging binary data. As it is, using PyTorch means that you force your users to also use PyTorch, which is a shame, as libtorch (which is what makes PyTorch work) offers a much more portable format (which I suspect might also be more efficient at least in terms of raw size, but I haven't checked).
Sounds like this should be the default. Maybe you can submit a PR to the official Torch repo? There is no reason why a static model checkpoint should be potentially dangerous to run.
I’m aware that they exist. I figured if someone inserted a hack they wouldn’t bother with docker escapes as they would catch plenty of people who run it without docker. I figured it was a calculated risk.
I wonder what the memory requirements would be to run such a large model. I'd love to be able to run this model, alas my MacBook can barely run toy models.
Yep. It’s expensive to spin up an A100 80GB instance but not THAT expensive. Oracles cloud offering (first thing to show up in google search I know you probably won’t use them and it seems extra expensive) is $4.00 per hour. If you are motivated to screw around with this stuff there’s definitely options.
With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typically need to read all the weights per 'word' output, so if your model is 20GB and you have not enough ram or vram, but have an SSD that reads 1GB/sec, expect 3 words per minute output speed.
However, code changes are necessary to achieve that, although they won't be crazy complex.
They are saying you can run it on a CPU by doing this:
> However, code changes are necessary to achieve that, although they won't be crazy complex.
This is technically true. It will be very slow though.
However, give it 6 months and I think we might see an order of magnitude increase in speed on CPUs. This will still be too slow to be very useful though.
Should be like an order of magnitude faster than trying to run it from a NVMe still, no? I've ran some small flan models from RAM and it was fine, but yeah it's not exactly realtime.
There is a neat potential speedup here for the case where the bandwidth to your model weights is the limiting factor.
If you have a guess what the model will output, then you can verify that your guess is correct very cheaply, since you can do it in parallel.
That means there is the possibility to have a highly quantized small model in RAM, and then use the big model only from time to time. You might be able to get a 10x speedup this way if your small model agrees 90% of the time.
It looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course.
The potential for speedup according to their paper is closer to 2x than 10x however.
Why 3 words per minute as opposed to second? Is that a typo? If you have enough RAM (but not VRAM), does it basically become limited by the PCIE lanes? So for the 112GB model with a Gen 5 GPU (64 GB/s PCIE bandwidth) that would be roughly 2 seconds per word right?
I just tried this on the 7B model. Steady state single threaded CPU performance of 23 seconds/token on a Ryzen 5800x (I'm not sure why it's only using a single thread... usually these libraries automatically use more) and 14GB of ram. It used more than double that amount of ram while loading the model, and the first token took 183 seconds (potentially it's doing more work to parse the prompt that I'm not measuring properly).
you can - slowly - run Bloom 3b and 7b1 on the free (trial) tiers of Google Cloud Compute if you use the low_cpu_mem_usage parameter of from_pretrained
FWIW this information was already freely available via DHT scrapers like btdig [1] I think everyone at Facebook knows that torrents aren't secret and the Google form is basically a legal tool to shield them from liability while making litigation against anyone misusing the model easier.
Nope, nor are the actual trackers, it's only the website used for search that's blocked in reality, slight flaw in Bittorrent, not having p2p search, although one could argue that lead to its success, with lists of new torrents, incentives for seeding more, and the fact that most people with slow connections don't seed everything forever, unlike previous UI designs that made that the default
The fun question is anyway if a ML model is copyright protectable. Probably not as it is produced by an algorithm (which even is GPL'ed). So the only tool would have been watermarking and pulling NDA type clauses, however a Google form seems not the best way in the first place also it is close to impossible to identify the leak (if they are not as stupid as it seems). Or am I missing anything? One backdoor would be if they included copyrighted material in the training and show how this can be extracted from the model. Maybe it the whole stunt was about trying out how the legal system works in those cases :)
commercial derivative works have always been legal when you did not agree to other terms.
one person broke their agreement with Meta, they're the only person that has a problem and the only person who gets to find out if the agreement was applicable at all.
if you released a chat bot that could be prompted to regurgitate some copyrighted information, so what? it just proves that you didn't need the $30 million in funding yet to train your own because you are using an existing model. So either use the funding for that or don't sell shares or a product based on that pretext. Nobody else has a problem.
Anything I missed? Now I wouldn't reshare the model, but aside from use and commercial use of its output? Not everyone gets their way, that's not controversial.
photos are copyrightable by the person taking the photo only because they decided where and when to press a button. the rest are algorithms and hardware.
I believe the AI models would also be copyrightable as such, subject to arguments that the underlying data was protected and thus it was subject to prior copyrights instead
It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent.
This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing you could usefully run on your own PC or server VM.
But these ML models are like neutron stars of computation density. You can’t really peek inside to see what’s going on either. An unknown stolen model’s properties would need to be discovered by experimentation.
I guess you have never compiled a Linux/BSD distribution from scratch and supported it alongside its infrastructure and lead its maintenance process.
Even without that, if you want downloadable and runnable software platforms, look to public Git repositories. Some of the people who have no financial motivation will release what they do alongside installation procedures and quality of life scripts and architecture documentation.
Most of the open source platforms doesn't publish this documentation and doesn't make installation easy to keep a sizeable moat and protect the platform they have developed, hence this is why we have a division between "Free Software" and "Open Source".
In short, "The bulk of really valuable commercial code" is self contained, but not open source, or if open source it's not Free Software and made deployable for other parties. Otherwise it loses monetary value in the eyes of the people who develop that for the monies.
Otherwise we have have Elasticsearch incident, where they pivot and move to "Source Available" model to protect their castle.
"I guess you have never compiled a Linux/BSD distribution from scratch"
You guess, and so perhaps do other things as well, with poor acuity.
The incalculable value of open source software has approximately no bearing on this assertion.
Yes I love linux and bsd too I'm not defacing your religion. I'm actually quite Stallmanesque in making my own life harder by only using ooen source software as much as possible and being super fun a family gatherings talking about it.
I don't think that's right - even if you had the full source code for either of those, it's extremely unlikely you'd be able to build them on your own machine.
Building them would be a challenge, but definitely not an insurmountable one. I’ve worked on a couple of C++ projects at a similar scale to Windows (millions of LOC) and the build systems were a major pain. But a determined engineer with the readme file and no other help could get it building in a week or so.
(This probably says more about how hard it is to build C++ than anything else)
Well, it kind of is a language thing. Many newer languages (Rust and Go come to mind) are much more consistent in the way you interact with them as your project scales.
Some years ago someone that worked at Microsoft told me he didn't think any individual engineer who already works on Windows could ever get Windows building by themselves with just the code.
You’re also not trying to get a full CICD pipeline complete with unit/integration tests, crypto signatures, ability to flip features on and off with a click , monitoring of the cicd pipeline, scaled so 1000s of engineers can work at once etc
Plus it already has been done before with Windows XP, without even any documentation and there is a guide on the internet on how to build Windows Server 2003.
This was done (if I remember right) when governments and big customers had access to the Windows source-code.
> But a determined engineer with the readme file and no other help could get it building in a week or so.
That's probably true, but I wouldn't be surprised if something like windows doesn't have a README file. And it does have build instructions they may well be in some wiki separated from the source code.
There are even nice timelapse videos of that process: https://vimeo.com/464644850 (you’ll need a Vimeo account to see it, because Vimeo is weird like that. This was on YouTube originally, but it was taken down.)
If you were to steal a chunk of source code or a binary from meta/Google, you could probably get it running inside a few weeks effort.
Sure, the binary probably depends on a lot of internal proprietary infrastructure, but also most of that infrastructure is easy to write a mock implementation of, as long as you are happy for it to be in-ram, not multi-homed and don't need it to scale to billions of users.
Most of the binaries have a standalone mode for running on a developers PC with few/no dependencies anyway.
-1: as an ex-googler, I can say it was hard enough for Google itself to get its code to run, given gonzo infrastructure assumptions, proprietary libraries/languages, etc.
sorry, but that's not how code works. It's true that code quality could be terrible but in fact Google is famous/notorious for extreme code review at the line-by-line granularity, plus comments, design docs and more.
The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it on.
Assume you got source for a game written in a proprietary game engine. If you don't have access to the game engine itself, nor the API documentation, etc, how long will it take to get this game running in your manner of choosing?
The infrastructure in these companies is a huge amount of scaffolding that's non-trivial to replicate.
Google has no incentives to allow an arbitrary component run standalone. Quite the contrary.
What they do get in return for the coupling is that they can evolve the common libraries and code patterns across the board (there are even automated code refactoring tools that help you do massive code changes, automating code review sessions across hundreds and hundreds of teams, with all changes tested against all reverse dependencies etc etc). All this allows for a level of internal code quality that is hard to see elsewhere.
Unless you really care a lot about that one requirement you seem to care about. In that case, yeah, you'd choose a different tradeoff.
Arguably it's not low quality code, but low quality system. Code can be correct, clear, and documented, and still be fragile and sensitive to platform configuration changes. E.g. "how many switches do I have to change in the build system before the code no longer builds?", "how many network jacks can I move this server over before I lobotomize the system?"
You ought to be able to arrive at the same conclusion, then, with these LLMs. Without arrays of GPUs, it would take thousands of years to train one. Without a corpus of billions or trillions of words, one would produce output of very limited utility.
I think you have to consider that some things are systems, and it is the assembly of their components that imparts the true quality.
Good luck even getting a google3-based Hello World to compile. I don't remember the exact numbers, but just #including the most basic libs resulted in a O(100M) binary.
And anything more complex than that would probably have dependencies on so many fat client libs, so much infrastructure, and so many external services, that you'll need months-years to even make sense of them, let alone mock them up.
> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent.
I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissioning and building a skyscraper in Manhattan. Is that correct approximately?
“Brute forcing a really inefficient approximation/estimator” is a good way to summarize it.
It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near.
They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.
Can "something simpler", for example, code correct function bodies from comments describing functions in natural language? I think people are too quick to dismiss the power of these models.
I am by no means dismissing the power. They are created very chaotically, however. Spaghetti thrown at a wall. They are brute force approximations.
They are wasteful. If LLaMa 13B is as powerful as previous 65B models, that's a significant amount of unnecessary paramaters lost/pruned in just this iterative upgrade alone. How small can they go? The fewest parameters that get the job done 99% as well is the way to go.
There is also the difference between the rules and use of language being directly compressed into the model, vs all the information known to humans compressed into the model. A smaller model that ingests relevant information on the fly (more like Bing, that supplements itself with search), may be less wasteful and perform better.
The current models being released are chosen because "they work" not because they are least fit and most performant optimized.
I don't think the term "brute forcing" is an adequate term to describe gradient descent. Brute forcing would be to try all random weights with no system imo.
No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.
> No you are probably overestimating the cost by 1-2 orders of magnitude.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
So do they estimate how much computing power/time they will need and then find some upper tier minimum $ amount to get the maximum discount possible or getting a certain resource availability commitment from Google? That's an interesting accounting problem.
For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount.
That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
$30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.
At the same time I guarantee you they didn’t get it right the first time. I’m sure there were multiple (both serially and in parallel) runs as they worked out kinks and tuned hyper parameters.
Not to mention, the kind of expertise to run this for a major corporation doesn't come for free either? Facebook employs quite a few high profile ML researchers who undoubtedly make mid-high six figure salaries.
The point was that if you only need to train once, then it's cheaper to rent the GPUs than to buy them. If you need to train it multiple times, then the cost of buying the GPUs is amortized among runs.
In any case the cost per run is going to be lower than 30m
I'm sure that's the case. The latest sku I'm responsible for QC testing now contains 4x A100's in a 2U chassis. And oh man the number of QSFP ports it utilizes..
Azure is generally a pretty terrible cloud (poor UX, very slow for anything, multiple highly critical cross-tenant security issues, etc.) far behind the market leader, AWS, so they have to compensate with pricing (same reason why Oracle Cloud is so reasonably price, they're already so far behind their usual pricing wouldn't make any sense).
Electricity costs are basically irrelevant because the cards are so expensive.
A100 cards consume 250w each, with datacenter overheads we will call it 1000 kilowatts for all 2048 cards. 23 days is 552 hours, or 552,000 kilowatt hours total.
Most dataceneters are between 7 and 10 cents per kilowatt hour for electricity. Some are below 4. At 10 cents, that's $53,000 in electricity costs, which is nothing next to $30 million in capital costs.
No, I'm willing to bet the CO2 cost of the cards is also way higher than the electricity. Those things are built on the global supply chain, with materials potentially making multiple thousands of kms journeys between each step.
Long term I also imagine it's much cheaper to run these large model trainings on renewables. It's a very centralized process that doesn't necessarily need 100% availability.
The manufacturing process, however, is totally decentralized, and NVIDIA mostly manufactures in China where coal is cheap.
I always feel there is an opportunity cost here though. If that green energy wasn’t being used for compute it could be available to heat someone’s home instead of them using dirty sources.
US grid mix produces about 0.855 pounds of CO2 per kWh[0]. So 552,000 kWh 452,640 pounds of CO2 which is 205.31 metric tons. At a cost of $40 per tonne[1] of CO2 that works out to $8,212.40 which is still small compared to the capital cost of the cards.
AWS us-west-2 is housed in The Dalles and Prineville, Oregon. Not only are they near a massive wind farm in the Columbia Gorge, but also quite near the Columbia river's many hydro-electric dams. Facebook and Apple also have Prineville data centers. They are built there intentionally. Electricity at many data centers is quite carbon-lean.
There's no reasonable way to get an estimate of what it actually costs FB.
1) The GPU's are not single use, they will amortize it over 3 yrs and there are other things that it will be used for that generate revenue.
2) The cost of the servers for these GPU's to run in with massive CPU, RAM, and storage requirements.
3) The overhead of building and operating all of that infrastructure in terms of people, electricity, cooling, etc.
4) The overhead of having dozens or hundreds of engineers & scientists who contributed to this.
One way you can distill the first three is to use AWS/Azure/GCP costs. But then you are still missing a major factor which is the humans that worked on it, and the human may very well exceed the hardware cost.
Plus there's a lot of highly specialized engineers required to keep all those GPUs up and running during training and the ML engineers who are skilled in deep learning + hardware, plus the systems for gathering/cleaning/labelling data. Gather enough engineers and now you need managers, PMs, etc.
I fail to see how this is different from other software in that regard. If you have parameters but not the network architecture, then it's not very useful.
You do need to guess things like activation functions, number of attention heads, order of attention layers, etc. Often the parameter names reveal something about these.
Finding a sha256 hash with N leading zeros is basically arbitrarily computationally expensive but could be written on a piece of paper. I don't see training an ML model as an egregious example of concentrating compute power
Certainly they retain not just information but compute capacity in a way that other expensive transformations don’t. I’m hard pressed to think of another example where compute spend now can be banked and used to reduce compute requirements later. Rainbow tables maybe? But they’re much less general purpose.
Not only can we bank computation, speed up physical simulations by 100x but I also saw some work on being able to design outcomes in GoL (game of life).
There was a paper on using a NN to build or predict arbitrary patters in GoL, but I can't find it right now.
It would be interesting to see an analysis of this. I see your point - otoh is there a reason to believe that more computation is being "banked" than say matrix inversion, or other optimizations that aren't gradient descent based?
The large datasets involved let us usefully (for some value of useful) bank lots of compute, but it's not obvious to me that it's done particularly efficiently compared to other things you might precompute.
For converged model training, training is often quite inefficient because the weight updates decay to zero and most epochs are having a very small individual effect. I think for e.g. stable diffusion, they dont train to anywhere near convergence so weight updates have a bigger average effect. Not sure if that applies to llms
Back when wavelet compression was still being developed, there was a joke that the best compression algorithm is "give an image to a grad student and tell them to figure out the best transform".
Under Feist Publications, Inc., v. Rural Telephone Service Co. ... it gets tricky.
From Wikipedia:
> The ruling of the court was written by Justice Sandra Day O'Connor. It examined the purpose of copyright and explained the standard of copyrightability as based on originality.
> The case centered on two well-established principles in United States copyright law: that facts are not copyrightable, and that compilations of facts can be.
> "There is an undeniable tension between these two propositions", O'Connor wrote in her opinion. "Many compilations consist of nothing but raw data—i.e. wholly factual information not accompanied by any original expression. On what basis may one claim a copyright upon such work? Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The ″Sine qua non of copyright is originality."
> ...
> The standard for creativity is extremely low. It need not be novel; it need only possess a "spark" or "minimal degree" of creativity to be protected by copyright.
> In regard to collections of facts, O'Connor wrote that copyright can apply only to the creative aspects of collection: the creative choice of what data to include or exclude, the order and style in which the information is presented, etc.—not to the information itself. If Feist were to take the directory and rearrange it, it would destroy the copyright owned in the data. "Notwithstanding a valid copyright, a subsequent compiler remains free to use the facts contained in another's publication to aid in preparing a competing work, so long as the competing work does not feature the same selection and arrangement", she wrote.
> The court held that Rural's directory was nothing more than an alphabetic list of all subscribers to its service, which it was required to compile under law, and that no creative expression was involved. That Rural spent considerable time and money collecting the data was irrelevant to copyright law, and Rural's copyright claim was dismissed.
---
And so, my (I am not a lawyer) take on this is that the numbers of the model are not copyrightable. The selection of the source material is... kind of. This gets into a "a recipe is not copyrightable, yet a recipe book is"
Whole new vistas open up to possible retaliation for piracy. Imagine how a bootlegged AI could have been set up to not just steal your info but manipulate you into ruining your life as revenge for bootlegging it...
I don't know about about this model, but usually with these ML models you download the static weights, but nothing is stopping you from fine tuning them to your needs/new information.
It's not automatic, would require some ML Engineering, but nothing is stopping you if you have the Pytorch graph and weights.
I mean the answer to the life is 42 but it took 7.5 million years for an advanced alien tech computer with the size of a building to calculate that
/s
Calculating things takes time and unrelated to output size. There are NP problems that simply outputs true or false yet requires more computational power than the universe can support
Anyhow I do remember a post of a person stating this will never happen but it's just a web form and request for describing of what type of research you do
Yeah, Meta must have had a plan for "when this gets leaked" because they put up only the flimsiest of foils. As per other comments the most likely is simply that they could shield themselves (and plausibly litigate with grounds) while ensuring that the model escapes into the wild to wreak its chaos against MS (OAI) and big G. This way they can see what's what from the safety of their shielded bubble and make a more informed call about changing the license to something more permissive if it looks like the strategic wins against their enemies would be worthwhile. Win win win. (Except for the leaker, that was an unfortunate own goal, they're going down).
718 comments
[ 4.1 ms ] story [ 354 ms ] threadhttps://pytorch.org/tutorials/beginner/saving_loading_models...
.chk file is am md5 hash of the file, the .json file contains this for the 7B model:
https://github.com/facebookresearch/llama
I already got the 7B model to generate text using my GPU! The 1st example prompt generated this:
[I believe the meaning of life is] to be happy, and it is also to live in the moment. I think that is the most important thing. I'm not really a party girl. I'm not a girl's girl. I have a really small group of close girlfriends and that's all I need. I believe in equal rights for everyone. I'm not a rebel. I don't really rebel against anything. I'm a very traditional girl, very loyal. I'm a mum's girl and I'm a dad's girl. People have a right to know what's going on. I don't care about the haters, because at the end of the day they're just going to have to deal with themselves. I've been getting more and more into fashion since I was about 16. I know I'm a little different, but so what? I think that's good. I don't think you should be like everyone else. It's my birthday, and I'll cry if I want to. I've always been a huge fan of fashion, and I've always liked to dress up
I have a 2060 and I am too afraid and poor to buy a 4090 after import duties and taxes in a tropical country
Edit: It looks like it can code, I tried to autocomplete the first 2 lines and it wrote the rest. Local Github Copilot here we come?:
Unless you have strong evidence of their identity I’d suggest rethinking this.
(As they didn’t reply to my request for the model - I specified it was for personal use and my use case was “I think it would be fun to run it on my own hardware” - I appreciate this little stunt a great deal!)
The worst offender is AMZ, all the rest big tech are pretty open-source friendly.
0:https://opensource.fb.com/projects/
Aren't you a cheeky lad? Metea turned out lots of open-source database systems:
* RocksDB
* Hive
* Presto
* Cassandra
* Velox
LFP
Getting anything that could produce, look like, or smell anything like misinformation out of meta is very hard (for good reason!)
My friends have had repeated push back for various papers because they are ML based and could be in the same room as something that could possible be used by miscreants.
And here we have a LLM that can spit out all sorts of things that are misinformation like.
If their department tried to launch something like Galactica they would have been slapped down and told to think again about what they were doing in life.
TeamMysticAvengers-meta-llm-x-cars-movie-model-x-angelina-jolie-naked-xxx-2023.zip.exe.torrent
Or am I understanding it all wrong
Edit: maybe in this case it's a leak though
(Disclaimer: I work at Meta, but have no relationship with the team that owns the models and have no internal information on this)
I had to look up the legality of downloading torrents of copyrighted IP - and let me just say, don’t do it.
Given that the cat is out of the bag, if I were them, I would say that it is now publicly downloadable under the terms listed in the form. It is great PR, which if this was unintentional, is a positive outcome out of a bad situation.
Did you mean to write "someone (possibly Meta)" ?
But you just said it was not Meta. Is that based on internal information? ;-)
Good times.
It would be like playing copyrighted music in your office without permission. Perhaps technically illegal, but your customers will never know what music your Devs were listening to...
And even if it did not, Meta certainly has a more capable legal team with more cash to spend than the average HN user.
Not that I could run it if I downloaded it.
https://aibusiness.com/meta/meta-s-llama-language-model-outp...
And here are some benchmarks running OPT-175B purely on (a very beefy) CPU machine. Note that the biggest llama model is only 65.2B: https://github.com/FMInference/FlexGen/issues/24
Looking forward to the YouTube videos of random tinkerers seeing what sort of performance they can squeeze out of cheaper hardware.
I only got the 30b model running on a 4 x Nvidia A40 setup though.
Is there a sub/forum/discord where folks talk about the nitty-gritty?
it's sharded across all 4 GPUs (as per the readme here: https://github.com/facebookresearch/llama). I'd wait a few weeks to a month for people to settle on a solution for running the model, people are just going to be throwing pytorch code at the wall and seeing what sticks right now.
The pytorch 2.0 nightly has a number of performance enhancements as well as ways to reduce the memory footprint needed.
But also, looking at the README, it appears that model alone needs 2x the model size, eg 65B needs 130GB NVRAM, PLUS the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes = 17GB for the 7B model (not sure if it needs to increase for the 65B model), but maybe a total of 147GB total NVRAM for the 65B model.
That should fit on 4 Nvidia A40s. Did you get memory errors, or you haven't tried yet?
People are also working on adding extra samplers to FB's inference code, I think a repetition penalty sampler will significantly improve quality.
The 7B model is also fun to play with, I've had it generate Youtube transcriptions for fictional videos and it's generally on-topic.
EDIT: correcting the type of GPU
This should lead to quite a lot of innovation and it’s inevitable that someone will get these working slowly on your average MacBook.
BLOOM goes indeed up to 175B parameters, and is certainly better than OPT. However, at least in my specific tests, it's still significantly inferior to OpenAI models, and actually on par with a few smaller models. There's also a "newer" fine-tuned model, called BLOOMZ, but at least in my tests it's even worse. Of course, that depends a lot on what you ask the model to do...
If LLAMA can indeed match OpenAI products, and do so with much fewer parameters, then it would be really great, and I'd really like to test it. However, even if the weights are now in the wild, using them would be clearly against the user agreement, and there's no way I'm going to do that in my work time :-) so let's hope Meta will come to sense and release them with a more friendly set of terms...
It doesn't exist for practical purposes because it is gatekept behind the same Facebook application process
https://ai.facebook.com/blog/democratizing-access-to-large-s...
Looks like it needs 14gb for weights and it isn't clear what the minimum size for the decoding cache is, but it defaults to settings for 30gb GPUs.
6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.
- $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard
- $5000 for 6x RTX 3090
- $350 for two 1500W PSUs
- $700 for 256GB RAM
You will also need PCIe extenders and perhaps some watercooling. And find a suitable case. The 2-card NVLink bridges are between $100 and $300 each (you nay want 3). All in all i think less than $10k.
Privacy costs money.
One of the reasons I'm not a huge fan of PyTorch.
It isn't like a multi gigabyte game for example, where knowing if there is any malicious code could easily be a multi-month reverse engineering project to get to the answer of 'probably not, but we don't have time to check every byte with a fine tooth comb'
[0] - https://github.com/mmaitre314/picklescan
In practice, who's going to bother checking the language model? All the code that runs Stable Diffusion or other Hugging Face models that I've seen just downloads the model dynamically, then uses it without asking question. That's a pretty low-hanging supply chain attack waiting to happen, I believe.
Some solutions for checking: https://huggingface.co/docs/hub/security-pickle
or run them in an isolated env.
But seriously, why not something more human readable and text-based if it's just weights?
$ fickling --check-safety consolidated.00.pth
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 63: ordinal not in range(128)The usage boils down to
import safer_unpickle from safer_unpickle
safer_unpickle.patch_torch_load()
This overrides default torch unpickler with a relatively safe one. Hope this helps.
Erm, for inference that is. Training is definitely out of question for individuals I believe (unless you use much smaller models?).
However, code changes are necessary to achieve that, although they won't be crazy complex.
> However, code changes are necessary to achieve that, although they won't be crazy complex.
This is technically true. It will be very slow though.
However, give it 6 months and I think we might see an order of magnitude increase in speed on CPUs. This will still be too slow to be very useful though.
If you have a guess what the model will output, then you can verify that your guess is correct very cheaply, since you can do it in parallel.
That means there is the possibility to have a highly quantized small model in RAM, and then use the big model only from time to time. You might be able to get a 10x speedup this way if your small model agrees 90% of the time.
The potential for speedup according to their paper is closer to 2x than 10x however.
0: https://arxiv.org/abs/2302.01318
[1]: https://btdig.com/b8287ebfa04f879b048d4d4404108cf3e8014352/l...
one person broke their agreement with Meta, they're the only person that has a problem and the only person who gets to find out if the agreement was applicable at all.
if you released a chat bot that could be prompted to regurgitate some copyrighted information, so what? it just proves that you didn't need the $30 million in funding yet to train your own because you are using an existing model. So either use the funding for that or don't sell shares or a product based on that pretext. Nobody else has a problem.
Anything I missed? Now I wouldn't reshare the model, but aside from use and commercial use of its output? Not everyone gets their way, that's not controversial.
I believe the AI models would also be copyrightable as such, subject to arguments that the underlying data was protected and thus it was subject to prior copyrights instead
This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing you could usefully run on your own PC or server VM.
But these ML models are like neutron stars of computation density. You can’t really peek inside to see what’s going on either. An unknown stolen model’s properties would need to be discovered by experimentation.
So, like movies or software
Where is the torrent with a runnable copy of paypal, or amazon?
Even without that, if you want downloadable and runnable software platforms, look to public Git repositories. Some of the people who have no financial motivation will release what they do alongside installation procedures and quality of life scripts and architecture documentation.
Most of the open source platforms doesn't publish this documentation and doesn't make installation easy to keep a sizeable moat and protect the platform they have developed, hence this is why we have a division between "Free Software" and "Open Source".
In short, "The bulk of really valuable commercial code" is self contained, but not open source, or if open source it's not Free Software and made deployable for other parties. Otherwise it loses monetary value in the eyes of the people who develop that for the monies.
Otherwise we have have Elasticsearch incident, where they pivot and move to "Source Available" model to protect their castle.
You guess, and so perhaps do other things as well, with poor acuity.
The incalculable value of open source software has approximately no bearing on this assertion.
Yes I love linux and bsd too I'm not defacing your religion. I'm actually quite Stallmanesque in making my own life harder by only using ooen source software as much as possible and being super fun a family gatherings talking about it.
You sure nailed that one.
https://news.ycombinator.com/item?id=34525936
Looking down that page, there are some valid magnet links.
(This probably says more about how hard it is to build C++ than anything else)
May not take hours, but determined engineer should be able to figure it out.
Just getting the code to compile, link, baking in assets, etc. For a single architecture is a much more reasonable goal.
I'd imagine it'd be on the scale of 1-3months for an engineer to get working full time, but large error bars around this figure
This was done (if I remember right) when governments and big customers had access to the Windows source-code.
That's probably true, but I wouldn't be surprised if something like windows doesn't have a README file. And it does have build instructions they may well be in some wiki separated from the source code.
Video games build systems are a thing onto themselves even taking the c++ issues out of the equation.
Hell even comparatively simole microservices in modern CRUD apps have resorted to docker to do away with pains of rebuilding stuff.
Sure, the binary probably depends on a lot of internal proprietary infrastructure, but also most of that infrastructure is easy to write a mock implementation of, as long as you are happy for it to be in-ram, not multi-homed and don't need it to scale to billions of users.
Most of the binaries have a standalone mode for running on a developers PC with few/no dependencies anyway.
The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it on.
The infrastructure in these companies is a huge amount of scaffolding that's non-trivial to replicate.
Google has no incentives to allow an arbitrary component run standalone. Quite the contrary.
What they do get in return for the coupling is that they can evolve the common libraries and code patterns across the board (there are even automated code refactoring tools that help you do massive code changes, automating code review sessions across hundreds and hundreds of teams, with all changes tested against all reverse dependencies etc etc). All this allows for a level of internal code quality that is hard to see elsewhere.
Unless you really care a lot about that one requirement you seem to care about. In that case, yeah, you'd choose a different tradeoff.
I think you have to consider that some things are systems, and it is the assembly of their components that imparts the true quality.
And anything more complex than that would probably have dependencies on so many fat client libs, so much infrastructure, and so many external services, that you'll need months-years to even make sense of them, let alone mock them up.
I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissioning and building a skyscraper in Manhattan. Is that correct approximately?
It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near.
They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.
They are wasteful. If LLaMa 13B is as powerful as previous 65B models, that's a significant amount of unnecessary paramaters lost/pruned in just this iterative upgrade alone. How small can they go? The fewest parameters that get the job done 99% as well is the way to go.
There is also the difference between the rules and use of language being directly compressed into the model, vs all the information known to humans compressed into the model. A smaller model that ingests relevant information on the fly (more like Bing, that supplements itself with search), may be less wasteful and perform better.
The current models being released are chosen because "they work" not because they are least fit and most performant optimized.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...
That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
In any case the cost per run is going to be lower than 30m
Edit: todays pricing looks like about 20% higher, still. How are these prices so different.
If an A100 costs $15k and is useful for 3 years, that’s $5k/year, $425/mo. 2048 A100’s cost $870k for a month.
A100 cards consume 250w each, with datacenter overheads we will call it 1000 kilowatts for all 2048 cards. 23 days is 552 hours, or 552,000 kilowatt hours total.
Most dataceneters are between 7 and 10 cents per kilowatt hour for electricity. Some are below 4. At 10 cents, that's $53,000 in electricity costs, which is nothing next to $30 million in capital costs.
You mean in terms of money. I think this is exactly the problem that we have in CS, nobody really cares about CO2.
The manufacturing process, however, is totally decentralized, and NVIDIA mostly manufactures in China where coal is cheap.
https://www.cnbc.com/2022/09/01/nvidia-says-us-government-al...
All consumer SKUs that I know of are manufactured in China. By volume this is certainly the majority of manufacturing.
[0] https://www.eia.gov/tools/faqs/faq.php?id=74&t=11
[1] https://www.pnas.org/doi/full/10.1073/pnas.1609244114
I believe capex <> opex is more 1:1 nowadays, so something feels off here...
A100 costs $2/h, so it is $2M to train biggest model. Easily kikstart crowdfundable project.
One way you can distill the first three is to use AWS/Azure/GCP costs. But then you are still missing a major factor which is the humans that worked on it, and the human may very well exceed the hardware cost.
At least $10 million/yr just for the talent.
The crazy thing about these models is that the compute power going into them is at least somewhat reversible.
How Well Can DeepMind's AI Learn Physics? https://www.youtube.com/watch?v=2Bw5f4vYL98 https://arxiv.org/abs/2002.09405 https://sites.google.com/corp/view/learning-to-simulate/home
Discovering Symbolic Models from Deep Learning (Physics) https://www.youtube.com/watch?v=HKJB0Bjo6tQ
Scientific Machine Learning: Physics-Informed Neural Networks with Craig Gin https://www.youtube.com/watch?v=RTPo6KgpvBA
Steve Brunton's channel is even more mind blowing than Two Minute Papers, https://www.youtube.com/@Eigensteve
Not only can we bank computation, speed up physical simulations by 100x but I also saw some work on being able to design outcomes in GoL (game of life).
There was a paper on using a NN to build or predict arbitrary patters in GoL, but I can't find it right now.
https://btm.qva.mybluehost.me/building-arbitrary-life-patter...
The large datasets involved let us usefully (for some value of useful) bank lots of compute, but it's not obvious to me that it's done particularly efficiently compared to other things you might precompute.
For converged model training, training is often quite inefficient because the weight updates decay to zero and most epochs are having a very small individual effect. I think for e.g. stable diffusion, they dont train to anywhere near convergence so weight updates have a bigger average effect. Not sure if that applies to llms
Fractal Image Compression, https://en.wikipedia.org/wiki/Michael_Barnsley https://www.abebooks.com/servlet/BookDetailsPL?bi=3131987970...
Are models like this copyrightable? It seems like this falls under the realm of "fact", which can't be copyrighted.
From Wikipedia:
> The ruling of the court was written by Justice Sandra Day O'Connor. It examined the purpose of copyright and explained the standard of copyrightability as based on originality.
> The case centered on two well-established principles in United States copyright law: that facts are not copyrightable, and that compilations of facts can be.
> "There is an undeniable tension between these two propositions", O'Connor wrote in her opinion. "Many compilations consist of nothing but raw data—i.e. wholly factual information not accompanied by any original expression. On what basis may one claim a copyright upon such work? Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The ″Sine qua non of copyright is originality."
> ...
> The standard for creativity is extremely low. It need not be novel; it need only possess a "spark" or "minimal degree" of creativity to be protected by copyright.
> In regard to collections of facts, O'Connor wrote that copyright can apply only to the creative aspects of collection: the creative choice of what data to include or exclude, the order and style in which the information is presented, etc.—not to the information itself. If Feist were to take the directory and rearrange it, it would destroy the copyright owned in the data. "Notwithstanding a valid copyright, a subsequent compiler remains free to use the facts contained in another's publication to aid in preparing a competing work, so long as the competing work does not feature the same selection and arrangement", she wrote.
> The court held that Rural's directory was nothing more than an alphabetic list of all subscribers to its service, which it was required to compile under law, and that no creative expression was involved. That Rural spent considerable time and money collecting the data was irrelevant to copyright law, and Rural's copyright claim was dismissed.
---
And so, my (I am not a lawyer) take on this is that the numbers of the model are not copyrightable. The selection of the source material is... kind of. This gets into a "a recipe is not copyrightable, yet a recipe book is"
The model may, however, be a trade secret. ( https://en.wikipedia.org/wiki/Trade_secret )
It's not automatic, would require some ML Engineering, but nothing is stopping you if you have the Pytorch graph and weights.
/s
Calculating things takes time and unrelated to output size. There are NP problems that simply outputs true or false yet requires more computational power than the universe can support
The bar to competition is far lower, as already evidenced by the plethora of AI products being put forward. Its a race to the bottom on pricing
There's an ass ton of hardware that might otherwise be idle.
I really like that expression.
Anyhow I do remember a post of a person stating this will never happen but it's just a web form and request for describing of what type of research you do
Of course it will be leaked
That's quite big!