Ask HN: Can you crowdfund the compute for GPT?
I'm just curious to know whether it's possible to crowdfund the compute costs for GPT models? It seems like this stuff is going to start to get beyond what any one individual can run, meaning it's in the hands of corporations or people with deep pockets. Can groups of people pool together money to run shared models? Because the alternative is that the companies just run away with the technology and leave the rest of us to wait for APIs or use whatever they give us.
156 comments
[ 3.9 ms ] story [ 208 ms ] threadSeems to work well, albeit very slowly…
It depends on what part of what you're building is your core intellectual property and such.
I'll ask back: what products would you build on top of a crowdfunded-compute GPT like model?
Separately, examination of religious and historic tests in alternate languages providing superior transliteration and translation into english than what humans could achieve.
I guess there's not much interest to develop something equivalent for the English-speaking market because it's difficult to monetize.
https://en.wikipedia.org/wiki/Xiaoice
https://arxiv.org/pdf/1812.08989.pdf (actually a better introduction than the wikipedia)
I'd be interested in something similar to this as well - but I think part of the problem is that depending on the language, the text might only be available as digital scans without OCR. In that case, given that it would be in difficult-to-OCR languages, how could it be fed to an AI model?
1. “AI” requires compute time for training (GPT, etc.) 2. If you use any “AI” service in the future, you could have to share idle computing power to improve that “AI.” 3. This may be tokenized via crypto. More contributions = more “AI” usage available for you.
Could the use of tokens or cryptocurrency incentivize participants to contribute their idle computing power to facilitate the process beyond those with deep pockets?
Don't you need the load the entire parameter set to backpropagate and potentially at quite high precision. Then probably load less than 8gb as typical max vram. And then finally send that all back.
I think you really need dedicated data centres if only to move the parameters around and load the entire thing at once.
https://docs.google.com/document/d/1NTsnUVRzBfK6y7WNX2JURogD...
Yeah, try rival Google's Street View data.
And Google have no equivalent to Wikipedia.
It's a 114 page document detailing the months of full-time work for multiple engineers that went into training a 175B language model at meta AI.
My contribution was around 19,000 books.
bmk was the magician there. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD...
what does this mean? not meaning to cross examine you, just curious how people contribute to The Pile since it seemingly appeared out of nowhere
I was convinced that a model needed to be able to read like we do. And what do we do when we read? Pick up a book.
That turns out to be surprisingly hard, at least for training data. Step one is to acquire the books. Step two is to turn them into a readable format for computers.
Both steps were very hard. I lucked out on step one because The Eye happened to host all of bibliotok, which came to around 30k books or so.
Trouble is, lots of those are PDFs. And although humans are great at reading those, they fucking suck for blind people. And a gpt is a blind person in a sense, because it needs to follow a linear sequence of words — something that PDFs are horrible at giving.
But one day I realized that epubs were merely html files, and aaronsw happened to write an amazing html to text converter. I had to hack it to fix a few corner cases. But after a few days, I ran it across all 19,000 epubs I spidered, then zipped the whole thing up and called it books3: https://twitter.com/theshawwn/status/1320282149329784833?s=4...
It’s one of the larger components of the pile, I think around 35%. Which is quite the hefty sum when it’s purely text. I still have a hard time wrapping my head around just how mindbogglingly big 800GB of text is.
this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who was curating the overall "mix" of the content. seems easily biased to, say, public domain books, since the corpus is easily available.
when people say things like "GPT3 has been trained on all of the internet" i suspect this is a gross exaggeration. In reality it's just C4/commoncrawl, so that's like 800GB of text.
Also bmk. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD...
They did the legwork of writing the paper and getting everything into a presentable format. A bunch of other people helped too; I wasn’t as involved as I could’ve been.
It was all discord-based. As far as I know it was the first serious research collaboration to happen solely via chatroom.
bmk also got the 50GB of code from GitHub, I think. So that’s where GPT-J’s coding ability likely came from.
could be a good story to write up, i think the people who do the hard work behind all the datasets dont get any of the glory of the ML models that get built on top of it
The real moat is more about the lack of concentrated compute, ML engineering, and, more generally, prosaic lack of political with outside a few orgs.
is there a crowdsourced list of text corpuses somewhere? i bet thats the starting point for all this. i'm only aware of C4 and The Pile.
- Company Alpha needs $40,000,000 worth of cloud computing for their training model - Company Beta provides them said cloud computing for $30,000,000 from their pool of connected graphics cards - Individuals can connect their computers to the Company Beta network and receive compensation for doing so. In total $20,000,000 is distributed.
Company Alpha gets their cloud computing done for cheap, Company Beta pockets the $10,000,000 difference for running a network, the individuals make money with their graphics cards, except this time it's actual United States Dollars. What am I missing here that would make this type of business unfeasible?
Sometimes the algorithm or the data is commercially sensitive.
Those are the two main reasons a 'rent out my GPU while I sleep' scheme wouldn't work.
Neither are insurmountable though.
What made this type of business feasible/attractive for cryptocurrency is that the miners were Company Beta. There wasn't two mouths to feed or two pockets to line, just a direct and transparent reward scheme for people donating compute. Projects like Folding@Home have leveraged world-scale networks before, but I'm not aware of anyone who's managed to monetize it.
You for example get a keyboards and a set of hands with each node, a webcam, a microphone.
I have no real idea but it seems the hand of cards isn't hopeless.
How? Definitionally, they wouldn't be turning a profit.
Company Beta likely started off with hardware for AI/ML and mined on the side when they didn't have enough customers.
Edit: I was probably thinking of vast.ai
It's not especially useful though because most companies (who are actually going to pay for this service) aren't going to want to send their training data to random people, and ML training needs high performance links between the cards. Plus you'd have to deal with the fact that you're running on 100 different GPU models.
* Servers in a datacenter are much more reliable than a network of PCs (power goes off, someone decides to play Crysis, etc)
* People will find ways to scam you (pretend like they’re doing the calculation while not actually doing it)
* Economies of scale means a datacenter will probably be cheaper than what you’d have to pay the PC owners (power consumption, network, wear, etc)
* The PC owners will have to trust this arbitrary code that they’re running (can you assure that there’s never a jailbreak resulting in a huge botnet?)
I’ve pitched this idea more than a decade ago and quickly realized it has many issues.
I'd contribute my GPU time to a Folding@Home style project if it meant that we had powerful, open LLMs that were free to use. I'm positive many others would as well.
As far as worrying about scammers, could you send the same compute task and training data to multiple clients and validate the results against each other? If they differed, you could throw the results and try again, or send it to a 3rd to break the tie.
By sending the work to 3 people each time, you’re effectively cutting your (already limited) resource pool by 66%.
But then how would crypto-scammers run ponzi after ponzi after ponzi if they did that?
There’s a whole field here and people exploring this problem, colloquially solving this would enable Federated Learning and whoever figures this out will far eclipse OpenAI (if it’s ever solved).
See also: https://github.com/learning-at-home/hivemind
and more to OP's incentive structure: https://docs.bittensor.com/
Latter two intend to beat latency with Mixture-of-Expert models (MoEs). If the results of the former hold, it shows that with a simple algorithmic transformation you can merge two independently trained models in weight-space and have performance functionally equivalent to a model trained monolithically.
https://en.wikipedia.org/wiki/MLOps
Armed with that term, we get (haven’t read):
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
https://arxiv.org/abs/2205.02302
[+ps]
Better resource: https://ml-ops.org/
Based on my (limited) exposure to date, there is tremendous opportunity for software engineers and architects to make impact in ML systems. There is a pronounced lack of seasoned engineering talent (outside of big players like DataBricks, et al) and this knowledge gap sits behind an experience curve that mere IQ can’t jump over. Our experience as software architects and engineers is very valuable.
Know this and recognize the value you will bring to the table.
Intel is doing some work with Penn on the subject now, if people want to read further: https://www.intel.com/content/www/us/en/newsroom/news/intel-...
A business paying USD is never going to be competitive with a decentralized crypto compute market. Better to build something just like mining BTC, except you do useful work, and are paid in a cryptocurrency you can exchange for USD. Then businesses can build on top of that to make it more user friendly.
The Golem network already lets you do this with CPU. It's way cheaper than centralized cloud computing could ever be, less than $0.003 per core per hour.
There are plans to add GPU support to Golem, but they have been on the backlog for a long time because there is a shortage of devs.
It's much safer to sign a contract with a real organization, especially the one which has a reputation to uphold. If someone like DigitalOcean steals data from servers, they can be sued, and there will be penalties. If some decentralized miner somewhere does that? Nothing you can do.
This only leaves decentralized compute to the applications where neither the data nor code does not matter. While I am sure there are applications like those (probably related to open-source or cryptocurrencies) I doubt they'll bring much money.
(And don't say "homomorphic encryption" -- the overhead there is so high it is much cheaper to just get a centralized server)
The same problem applies to things like Mechanical Turk and other croudsourcing. The way I've dealt with the issue in the past is to start with zero trust and to have them do computations that I already know the answer to. After that, they do computations that are matched with a random other participant (the two results should match, if they don't, compare against a third random participant).
Later, when trust has been developed, you can start assuming their work work is trustworthy, but still check it randomly with computations you know the answer to, or a second person doing the same computation.
Yes, this adds overhead (roughly 10% in aggregate) but it works fairly well. It works even better if there are penalties that you can impose for ffraudulent results (like cancelling ALL owed payouts).
By definition, this is not having them do any computation. The proper solution at this time would be some trapdoor function that is easily verifiable (proof of work), at least while P != NP
If I ask you to compute the first million digits of pi to the power of 1.23456 and I already know the answer to validate it, how is this "by definition" not computation?
There's a useful asymmetry we can exploit: finding weights that perform well is computationally intensive and takes time, but scoring a set of weights is fast and easy.
A number cruncher could spend 2 weeks training a model, and then when they submit the results it takes me 10 seconds to score the model - to verify the quality of the results, and calculate the performance-based payout. In the #1 or #3 scenario where they didn't do or didn't complete the computations, they wouldn't have a well-trained model to submit for payout. (The lost time in #3 is inconvenient in time-sensitive situations, but mechanisms exist to address that - SLAs, up-front collateral, etc)
Regarding privacy, that's an EXTREMELY good and important question. There's some really neat prior art for privacy-preserving machine learning that could be useful here, e.g. https://arxiv.org/abs/2106.07229 "Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network"
(note I'm approaching this as an interesting DistML thought experiment, not proposing it as an immediately viable or sensible initiative)
Any computation can be verified as having been done by, at a minimum, checking for reproducibility. This requires having each work unit be done twice and only issuing credit if both units match. For deep-learning applications "match" is relative: different compute accelerators are going to give different results. So, instead we can insist that all the floating-point outputs on the model have to match up to the first n bits of mantissa. Neural networks are actually really insensitive to small perturbations in their weights, and it's common to train on 16-bit floats to save time.
We can also exploit the loss function itself as a verification mechanism. Generally speaking training is more compute-heavy than inference[0], so we can just run the updated model on the training set and confirm that your trained model is better than the original you were provided with to start from. This will need upper bounds, too - if only to catch people trying to overfit the model to guarantee they get credit.
As for privacy and security... the answer is to not train on private data or things that people do not want to be trained. Period. This isn't even a problem solely with distributed computing. All AI training should be limited to either data provided with consent, or data that's so old that training on it would not cause harm.
Availability is a problem, but not necessarily one that most distributed computing projects actually have to deal with. There is a minor incentive to participate with the credit system; there's a leaderboard for the fastest/highest credit users and teams. And people do compete for those leaderboard slots, because that's effectively ad space.
[0] Model execution.
Physical goods vendors that sell through Amazon have found that once they sell something highly in demand, and profitable, Amazon uses all of their internal knowledge of the sales to create their own Amazon Basics version of the same good and then promotes that as a cheaper alternative, capturing most of the profit.
There is absolutely no reason not to believe that AWS will do the same thing and use their knowledge of the workloads and exact hardware requested to compete with you. The fact that they haven't done so yet isn't much evidence against them doing this in the future, since other parts of the same company already follow this approach.
Individuals renting out their home compute for things like this suffer from the problem of supporting so many different types of hardware profiles. BOINC is a prime example of this, but it doesn't have a funding mechanism.
https://rendertoken.com
I'm not sure whether or not they will be successful.
https://docs.google.com/document/d/1NTsnUVRzBfK6y7WNX2JURogD...
But the issue with large language models is that you need ms-latency access to TBs of data, or else you won't be able to saturate your GPUs with useful work.
Our biggest problem was finding customers on the demand side. What is the killer application which can take use of that kind of compute with those kinds of networking properties? What are the use cases? No one has written software that takes advantage of that kind of compute, because it's essentially a solution looking for a problem.
- Fluidstack: https://fluidstack.io
- Vast: https://vast.ai
- QBlocks: https://qblocks.cloud
- RunPod: https://runpod.io
- Sonm: https://sonm.com (blockchain)
- Golem: https://golem.network (blockchain)
- Rentaflop: https://rentaflop.com (rendering specific, blockchain)
- RNDR: https://rendertoken.com (rendering specific, blockchain)
If you want HPC specific cloud providers:
- Crusoe Cloud: https://crusoecloud.com
- Coreweave: https://coreweave.com
- Lambda Labs: https://lambdalabs.com
- Paperspace: https://paperspace.com
As others have pointed out, the decentralized clouds can't offer high performance interconnects (e.g. InfiniBand) that a lot of folks are using for LLM training. There are definitely initiatives underway to reduce dependence on these interconnects and build performant distributed training (again, some threads below mention this), but I think it's mostly academic at this point.
Disclosure: I run product at Crusoe Cloud, which aims to provide ML training at half the cost of a hyperscaler, while also being carbon reducing (https://crusoecloud.com/climate-impact/).
If there was a feasible crowdfunded solution to this and putting it in the hands of the people - I would certainly be prepared to lay down up to £5K.
[0] https://www.sheepit-renderfarm.com/home
You start a kickstarter, and when we get to our target of $10M, we can make a start on the computation!
https://github.com/Kindelia/Kindelia-Chain
Yeah, I think we could crowdfund a billion dollars for this, but we'd need some really competent people making sure it gets used optimally.
I think that will help get it started. Then swap to a non-profit monthly sub model where people are simply paying for use (pay out the cost for labor and hardware at that point only). A utility bill essentially.
Just for individuals, no promises of deep data / model privacy.
Has anyone implemented something like this?
https://akash.network/
The larger issue here though is the amount of bandwidth and memory required. Downloading billions of images to train a model just isn't going to work.
People are working on splitting the training into smaller chunks (together.xyz is one example), but they aren't quite there yet.
Like how people mirror the initial Stable Diffusion model, only to find better, faster and smaller new versions later.
Then share the profit among the group of people? It’s called a company.
https://www.reddit.com/r/AiCrowdFund/
https://docs.google.com/document/d/1NTsnUVRzBfK6y7WNX2JURogD...