Very happy to see oobabooga on the list. Everyone's contributing in different ways, and ooba serves as a fantastic portal to try out various models which I appreciate. Its readme even says it wants to be like the automatic 1111 of text generation. For beginners like me who are curious about this new world, ooba is a great entry point.
Well, at least this time a16z is giving money to actually useful technologies instead of bloviating about the world changing potential of cryptocurrencies. I imagine they've written down most of their cryptocurrency investments at this point.
That being said, these are some great grants for local LLM builders, local computation for such powerful tools is needed, and will continue to be needed, as industry incumbents like OpenAI continue to not release their models.
I had become honestly cynical of their actions in the past, as they felt hype-driven and externalities-blind. Yet I don’t see how this grant could become rapacious. Hopefully it is an attempt at cleaning the brand which can work as making amends.
Don't VCs basically chase everything because they only rely on 10% or less to succeed but, that 10% needs to have at least 10x or more profit and then they do another round?
Hi HN, we are super excited to announce this new initiative!!
Please note, this program was designed to support individuals, teams, and hackers who are not pursuing commercial companies. Nevertheless, these projects push the state of the art in open source AI and help provide us with a more robust and comprehensive understanding of the technology as it is developed.
We are really proud to be contributing in this small fashion and grateful to the first cohort and all others contributing in this space!
Interesting, hadn’t thought of that, thank you! If you want to host end points Replicate is a great option, they also have a newer fine tuning api and solution! for raw VMs with GPUs right now it’s a bit situational and you have to try multiple different vendors tbh, also really depends on the capacity you need and which machines!!
A lot of people are using RunPod for experimental/small-scale workloads. They have good network and disk speeds and you can generally find availability for a latest-gen GPU like an L40 or 4090 if your workload can fit on a single GPU. One GPU is plenty for fine-tuning Llama 2 7B or 13B with LoRA or QLoRA. They also sometimes have availability for multi-GPU servers like 8xA100s, but that's more hit-or-miss.
If you want to go even cheaper vast.ai is a popular option. It's a P2P marketplace for individuals to rent out their GPUs. You can generally get a ~20-30% discount vs RunPod prices by using Vast, but network speeds and perf are much more variable and there's always the possibility that the host will just shut you off without warning. I also wouldn't recommend using it if you're training with proprietary data since they can't guarantee the host isn't logging it, but most of the OSS fine-tuning community publishes their datasets anyway.
1. I've updated the section now: https://gpus.llm-utils.org/cloud-gpu-guide/#so-which-gpus-sh... - that should answer it. Basically 1x 3090 or 1x 4090 is an ideal set up for stable diffusion, 1x A100 80GB is an ideal setup for llama 2 70b GPTQ (and you can use much smaller GPUs or even CPUs if needed, for smaller llama 2 models)
2. No, they don't have access to the internet unless you build something that gives them access
If an individual is looking to contribute to the field with different training data ideas - would they need to first establish themselves and get your attention, or would there be a way to submit a proposal?
For myself compute is rather expensive, I could likely afford a few test runs for proofs of concept but beyond that it would be difficult. I've got a single 4090 so I can't run llama70b faster then 1it/s.
Thank you so much for your work and enthusiasm here!! Unfortunately, that’s roughly the best way to approach it and how most of these have worked so far, demonstrations on smaller models (e.g. 7/13B), with slightly smaller datasets, catching the eye of the community, etc. in general it’s not a bad approach to research too, prove out concepts at smaller scale before scaling up!
Definitely. This was kicked off in the last month or two when there just seemed to be a bit more happening in OSS LLMs, but we are prioritizing diversifying this for the next cohort!
This is great, but I'm not seeing anything about how to apply, or really anything about the application and selection process at all. Is this by design?
My guess is that they'll prefer to get nominations.
Having once done a grant application process - I think their approach is smart.
If they opened grant applications, they'd get thousands of (sadly mostly very low quality) applications. It's extremely hard to vet.
It's a much safer bet to wait until projects are already popular in the community, already being relied on by people, and where those devs would love to spend more time on their open source projects but they're limited by money.
Yes, the downside of this is that some people could make things happen if they were given money first. But, if I was them, I'd still prefer to do what they're doing and pick projects that are already successful and give grants to those.
(This isn't a comment about your project specifically, for all I know your project might be a popular open source LLM project that lots of people already rely on, in which case they probably already know about your project and will probably reach out to you for the second round, is my guess)
It's interesting to note that there's particular emphasis on "removing censorship from LLM outputs" given a16z's investments outside the AI space in some ideological social media companies.
> We believe artificial intelligence has the power to save the world—and that a thriving open source ecosystem is essential to building this future.
Thank you for understanding the fundamental flaws of the World and human consciousness, and thank you for spending multiple lifetimes as Buddhist monks (not to mention lamas) to understand the intricacies of the human mind from a scientific perspective. This is exactly what we need right now and this initiative looks to be the answer to all of our problems.
“We are disguising this round of seed funding as a grant, accomplishing all of the below:
1) brand-washing for people who don’t do the math
2) avoiding angering our LPs and/or violating our fund theses and
3) getting in on the ground floor of things that might follow OpenAI through the non-profit-to-VC-darling door
Thanks for reading our press release, feel free to try to apply using your personal network (somehow) because we don’t care as we’ve already landed our positions.”
They state the nature of their support in very clear terms:
"We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
Some of the recipients of financial support have described the nature of the agreement as well.
> a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant.
You could choose to interpret it as marketing spend or "buying relationships", sure. For me, there's enough negativity in the world - I don't want to spend time imagining more.
And why would I? A16Z have already explicitly stated their intent: "a thriving open source ecosystem is essential to building [ai's] future" and "We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
There's not a lot of spin on this. They want to be a leader in AI so they're investing in the people who build the tools that they use in pursuit of their commercial goals.
This is insurance for their already huge investments (and future rounds) in AI businesses. It's a pittance relative to everything else they have at risk. It's prudent since failure of small projects like these can delay work on profitable corporate efforts.
if the work is OSS/public and you aren't trying to start a VC-backed company, it could be! We usually want to see some early work already published, community reception and validation, but feel free to DM Matt or myself and we can take a look!
> if the work is OSS/public and you aren't trying to start a VC-backed company, it could be!
Yes, it is a company that often applies and receives grants. For example, we are working on adding AI to security tools such as https://github.com/CoinFabrik/scout
I want to congratulate you based on your current grant recipients. They are all very impactful individuals/teams. It shows that you clearly did your research and put the money in the right places.
Is there a way we can apply? I am building https://atomictessellator.com, and since I am 100% technical I am not sure about commercialising or open sourcing it, I definitly would like to open source some of the components (e.g. turnkey dockerized scientific computing environments, the molecular force viewer, and also open up a hub for the environments and models) - when I was in Stanford meeting with some comp chemists a few months ago they got very excited for this.
I negotiated with my day job to work 2AM to 10AM so that I can work on this passion project in the afternoons, some grant funding would really help me focus on this more. There's quite a bit of interesting stuff not on the website in catalyst discovery and molecular dynamics. Contact me if you'd like to chat / demo.
In my ideal world I would work on this full time and totally open source it.
feel free to DM on twitter or linkedin. In a nutshell, if you want to pursue a company now in or in the near future, we would likely need to consider you through our normal investing process for equity funding. If you are considering just working on it OSS, we would encourage taking first steps in that direct,
and testing community-project fit validation before quite going full or near-full time!
"We believe artificial intelligence has the power to save the world"
I don't agree with the side that says AI is about to destroy everything, it seems very hyperbolic... but neither do I agree with this sentiment that it's going to save the world.
It's all relative. Compared to the huge companies spending thousands per month/week/day on GPU usage, having two 4090s (which you could acquire for like ~$3000) could be considered on the lower end for sure.
I interpret "GPU poor" the same way as "house poor", i.e. you're poor (lack cash) because you've use it to purchase GPUs, not that you don't have enough GPUs.
No deeper reason. I think there just a lot in LLMs happening right now which skewed it towards them. We would love to do something in the SD ecosystem.
I'm still not sure what I'm supposed to be using AI for. I already know how to code, tried for a decade to make ACR (auto content recognition) work with CNNs, and as best I can tell, chatGPT can write a news article for you if you write it first and then hire someone else to edit it.
Even knowing this is partly motivated by branding/marketing, it's great to see a16z getting more aligned with solving real pain points (vs crypto and churning out shallow media in recent years). Hope they can keep it up and hopefully more "thought leaders"/VCs follow suit. Best of luck.
Agreed, but IMO it's their job to do stuff like this (and prob unrealistic to expect a world where it doesn't exist). I'd rather see a higher percentage of marketing for things that have more real value vs. less
Of course - there will always be investors/influencers pushing narratives to hype their bets. IMO everyone is better off when the hype is grounded in solving real pain points, which the AI projects I've seen seem to be closer to than most of the crypto projects I've seen.
Certainly there are plenty of grifters in AI too (as with any gold rush) and many AI efforts will fizzle out. But it seems there is more real value being created here than in crypto, which is the main thing I'm excited about and hope to see more of
I agree this initiative helps solve real pain points, and I'm not trying to defend "Crypto" with my question here but isn't solving a "real pain" kinda subjective? For example, many people/companies have the real pain/need to improve ad click-through-rates and reduce crypto/gas fees but maybe these aren't problems you care about.
I guess sometimes a pain can exist without a profit, there's only a small number of people willing to solve those pains without a profit incentive though (pro bono lawyers, doctor's without borders, etc).
Agreed, "real pain" is a subjective/vague term and even crypto is a broad term. My intended use here is to describe solutions that are more grounded/integrated into reality vs. more grounded in a fantasy of how the world works. It is hard to describe well though and if you keep unrolling most perceptions of reality are subjective
Leftfield thought, imagining a tax (vaguely like carbon credits) on energy/cpu cycles/modelling (just my vague idea). With the proceeds going to other projects to diversify the outcomes.
Always seems to me our economy is being less based on money and more on energy and productivity, basically being rounded down to physics as a healthy market economy should.
You won't be able to limit it to a particular set of clients who want to use it, but you're basically supporting the "GPU Poor" by doing this, as I think it's mostly smaller companies, individuals and researchers using vast.ai, rather than huge companies.
25K will either get you money for renting servers, or allow you to jerry rig yourself something like a 8x4090 machine.
The servers that most people would like to run (8xA100, 8xH100) are over 200k, and even if you had the money they're probably very hard to get right now.
8x4090 machines should be a lot cheaper than that. 1599*8=12,792. I.e. closer to 16x4090 machines. Sure labor is worth something but he has it in cash, and also there's barely a lot of people (yet) paying high prices for research (non ML ops) labor hours for this yet so the salary wouldn't be that high anyway.
Just a pet peeve when ppl under exaggerate how many gpus/flops a dollar can buy (if you're slighly smart with your money).
What about PSU to power all of this, GPU motherboards, SSDs, CPU to not bottleneck this and sufficient RAM? You could argue it's more than 8, but definitely not 16. To utilize those GPUs to the max you need memory bandwidth to even feed it enough data.
Yeh I built a dual 3090 machine and just to have those 2 cards run well cost a lot for all the other stuff. Workstation spec mobo, top tier PSU, CPU and memory with headroom for the system, not to mention storage and case and cooling. And this is just a hobbyist build. Cost would have easily doubled if I actually got server grade CPU and memory etc.
Meh, I was just ballparking. But add a threadripper, compatible motherboard, lots of RAM, fast storage, a few beast PSUs, and tax, and you're probably around 18k. So you ain't getting 2 of these machines for 25k.
Plus you'll need to set aside some money for the power bill lol
honestly would recommend using spot/rented instances on that budget vs. buying! A100s run around ~$2/hour, the hard part is getting a large allocation... training workloads are quite spikey anyways, usually running lots of smaller experiments before actually doing a larger single run...
No intention to attribute malice, but wouldn't this be more like buying a condo and renting it out to pay your mortgage rather than donating resources?
123 comments
[ 3.4 ms ] story [ 232 ms ] threadThat being said, these are some great grants for local LLM builders, local computation for such powerful tools is needed, and will continue to be needed, as industry incumbents like OpenAI continue to not release their models.
Please note, this program was designed to support individuals, teams, and hackers who are not pursuing commercial companies. Nevertheless, these projects push the state of the art in open source AI and help provide us with a more robust and comprehensive understanding of the technology as it is developed.
We are really proud to be contributing in this small fashion and grateful to the first cohort and all others contributing in this space!
If you want to go even cheaper vast.ai is a popular option. It's a P2P marketplace for individuals to rent out their GPUs. You can generally get a ~20-30% discount vs RunPod prices by using Vast, but network speeds and perf are much more variable and there's always the possibility that the host will just shut you off without warning. I also wouldn't recommend using it if you're training with proprietary data since they can't guarantee the host isn't logging it, but most of the OSS fine-tuning community publishes their datasets anyway.
Let me know what you'd want to see added!
One thing I cant glean ; What GPu/kit are preferred for which type of output?
Like chat vs imaging...
Do locally run models/agents have access to the internet?
Whats the best internet connected crawler version on can use?
2. No, they don't have access to the internet unless you build something that gives them access
3. I'm not sure what you're asking
If an individual is looking to contribute to the field with different training data ideas - would they need to first establish themselves and get your attention, or would there be a way to submit a proposal?
For myself compute is rather expensive, I could likely afford a few test runs for proofs of concept but beyond that it would be difficult. I've got a single 4090 so I can't run llama70b faster then 1it/s.
Are you interested in non-LLMs as well? Stable diffusion for example?
Having once done a grant application process - I think their approach is smart.
If they opened grant applications, they'd get thousands of (sadly mostly very low quality) applications. It's extremely hard to vet.
It's a much safer bet to wait until projects are already popular in the community, already being relied on by people, and where those devs would love to spend more time on their open source projects but they're limited by money.
Yes, the downside of this is that some people could make things happen if they were given money first. But, if I was them, I'd still prefer to do what they're doing and pick projects that are already successful and give grants to those.
(This isn't a comment about your project specifically, for all I know your project might be a popular open source LLM project that lots of people already rely on, in which case they probably already know about your project and will probably reach out to you for the second round, is my guess)
Thank you for understanding the fundamental flaws of the World and human consciousness, and thank you for spending multiple lifetimes as Buddhist monks (not to mention lamas) to understand the intricacies of the human mind from a scientific perspective. This is exactly what we need right now and this initiative looks to be the answer to all of our problems.
1) brand-washing for people who don’t do the math
2) avoiding angering our LPs and/or violating our fund theses and
3) getting in on the ground floor of things that might follow OpenAI through the non-profit-to-VC-darling door
Thanks for reading our press release, feel free to try to apply using your personal network (somehow) because we don’t care as we’ve already landed our positions.”
what math should i be doing here?
The bigger math is significance_of_contribution = $size_ai_grant_program/$size_ai_vc_program (or maybe $size_ai_grant_program - $size_ai_vc_program)
[ 0 https://link.springer.com/chapter/10.1007/978-3-030-72428-3_... ]
"We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
Some of the recipients of financial support have described the nature of the agreement as well.
> a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant.
And why would I? A16Z have already explicitly stated their intent: "a thriving open source ecosystem is essential to building [ai's] future" and "We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
There's not a lot of spin on this. They want to be a leader in AI so they're investing in the people who build the tools that they use in pursuit of their commercial goals.
Is this approach covered in the grants?
Yes, it is a company that often applies and receives grants. For example, we are working on adding AI to security tools such as https://github.com/CoinFabrik/scout
I negotiated with my day job to work 2AM to 10AM so that I can work on this passion project in the afternoons, some grant funding would really help me focus on this more. There's quite a bit of interesting stuff not on the website in catalyst discovery and molecular dynamics. Contact me if you'd like to chat / demo.
In my ideal world I would work on this full time and totally open source it.
I don't agree with the side that says AI is about to destroy everything, it seems very hyperbolic... but neither do I agree with this sentiment that it's going to save the world.
That said, software of all kinds (including machine learning) does make the world more interesting and exciting, so happy to see more investment.
Less than 10k H100s makes you GPU poor. The author is presumably using it as a proxy for "possible to achieve AGI".
Curious why there are no image ai projects receiving grants here?
Even knowing this is partly motivated by branding/marketing, it's great to see a16z getting more aligned with solving real pain points (vs crypto and churning out shallow media in recent years). Hope they can keep it up and hopefully more "thought leaders"/VCs follow suit. Best of luck.
https://a16z.com/2023/06/06/ai-will-save-the-world/
Certainly there are plenty of grifters in AI too (as with any gold rush) and many AI efforts will fizzle out. But it seems there is more real value being created here than in crypto, which is the main thing I'm excited about and hope to see more of
Plus A16Z has some questionable motives in general based on previous comments by them...
Always seems to me our economy is being less based on money and more on energy and productivity, basically being rounded down to physics as a healthy market economy should.
I don't want to donate the cash but instead set up a handful of GPUs(Somehow?) and let people pay at cost to use them.
You won't be able to limit it to a particular set of clients who want to use it, but you're basically supporting the "GPU Poor" by doing this, as I think it's mostly smaller companies, individuals and researchers using vast.ai, rather than huge companies.
The servers that most people would like to run (8xA100, 8xH100) are over 200k, and even if you had the money they're probably very hard to get right now.
Just a pet peeve when ppl under exaggerate how many gpus/flops a dollar can buy (if you're slighly smart with your money).
Plus you'll need to set aside some money for the power bill lol