I've been using Stable Diffusion to generate cover images for the music I release & produce for others. It's a massive time saver compared to comping together the release art using image editing software, and a lot cheaper than working with artists, which just doesn't make sense financially as an independent musician.
It's a little bit difficult to get what you want out of the models, but I find them very useful! And while the output resolution might be quite low, things are improving & AI upscaling also helps a lot.
Everyone is getting disrupted by AI sooner or later.
The trick is to use AI to do things it would take you five lifetimes to learn. It's a tool to lower opportunity cost, financial capital, and human capital. That gives anyone leveraging it a much bigger platform, and the ability to dream big without resources.
If you can become your own "studio", you're the indie artist of the future. You don't need Disney or Universal Music backing.
Anyone can step up and do this. The artists being threatened can use these tools to do more than they've ever done by themselves.
The problem isn’t capabilities, it’s having a market that’s saturated with supply twice over - once by the ability to make infinite copies of the product, the other where there’s an infinite supply of distinct high-quality products.
Subcultures used to provide a counterbalancing force here, but they aren’t doing so well these days.
I watch films and media, listen to music, and I'm only truly fully satisfied a single digit number of times a year. That's a consequence of not enough being created and experimented with.
The long tail is longer than you can imagine, and that's what form fits to your personal interest graph.
I don't think that will do a whole lot to protect you from the economic harm. If everyone is producing more, the value of the works are reduced. At best, nobody will will make more money, they'll just be working harder to stay at the same place. More likely, there will simply be no room in the market for as many people and most will be out of work.
In the music industry that appears to have been the case since iTunes hit the scene. The ease of distribution has enabled countless artists that nobody has ever heard of and will never listen to. Yet some have risen up and become hits despite this.
I disagree. Youtube model has shown that multiple people can produce videos and still earn profit from it. There are thousands of niches that creators can target which big studios don't even touch because masses might not be interested in it.
We can a Big Bang without all the stupid romantic stuff. We can have different ending versions of Game of Thrones. So much stuff is never made because it takes so resources to produce them.
I think the market will only grow when this technology is available to everybody.
If we'd prevented new technologies from influencing our artwork, our paintings would never have left the cave wall. I'm a musician with live published albums as well; if there comes a time when I think AI will help with my creative process, you can bet that I'll be using it.
"I think we should ban ATMs, online banking, and direct deposit so I can get work as a bank teller" said no one ever. (Well, maybe someone when these were new.)
Displaced workers need support to ensure they can weather these transitions, but it doesn't make sense to artificially create demand by fighting new conveniences. If we want to ensure people have money, the solution is to give them money, not give them money in exchange for busywork.
Are single mom's a special class we should treat differently from others? You have single fathers.. childless couples, singles, parents with kids who have a disability, healthcare workers, transgendered singles, frail elders, mute single males..
I'm already using AI to make my music production process more efficient! Namely, I'm using a program called Sononym which listens to my tens of thousands of audio samples and lets you search by audio similarity through the entire library, as well as sort by various sonic qualities.
I think I'd still go for a human artist for a bigger release such as an album! It's a lot less hassle than sorting through (often rubbish) AI output & engineering your prompts, though it does cost £££ which is the main thing making it prohibitive for single releases.
This is such an interesting read. It makes a compelling case, though how the likes of Google should react feels less like an adjustment and more like a revolution.
Cringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!
If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor.
A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns.
A model doesn't have to be better at everything to be better at something.
Yes, price = quality because they require supercomputing resources to train. GPT-3 required hundreds of Tesla GPUs running for several weeks. That's millions of dollars just for hardware, not including power (the GPUs cost $15k each)
"April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT
Berkeley launches Koala, a dialogue model trained entirely using freely available data.
They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100."
This is hugely misleading. If your bot just memorizes Shakespeare and output segments from memorization, of course nobody can tell the difference. But as soon as you start interacting with them the difference can't be more pronounced.
>With these two evaluation sets, we conducted a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. In the ratings interface, we present each rater with an input prompt and the output of two models. They are then asked to judge which output is better (or that they are equally good) using criteria related to response quality and correctness.
No, it's not just memorising shakespeare, real humans interacted with the models and rated them.
That's not what I meant by interaction. The evaluator had to ask the models to do tasks for them that they thought of by their own. Otherwise there are just too many ways that information could have leaked.
OpenAI's model isn't immune from this either, so take any so-called evaluation metrics with a huge grain of salt. This also highlights the difficulties of properly evaluating LLMs: any metrics, once set up, can become a memorization target for LLMs and lose their meaning.
Exciting to see open source catching up, but seems to still be a big difference in factuality and common sense between ChatGPT and Koala. Eg:
Who built AlexNet on the ImageNet dataset?
ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer.
Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.
I'd agree they aren't close, but they are way better than I expected to see in a short few months. At this rate they'll be approaching "good enough" for me pretty soon. I don't always need a dissertation out of it unless I'm fooling around. I want quick facts and explainers around difficult code and calculations. Been playing with Vicuna-7b on my iPhone through MLC Chat and it's impressive.
I use DDG over Google for similar reasons. It's good enough, more "free" (less ads), and has better privacy.
If all you’ve done is download the model and perform basic prompts then I understand why you think this. There is a lot more going on behind Bard and GPT than a chat window passing the inputs to the model.
Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform.
In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases.
Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model.
Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident.
This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.
All the things you mentioned make it a platform, but even as a model, none of the smaller open-source models come close to GPT 3.5 or 4 in my experience. You can test it by using the GPT3.5 or 4 with their API. They outputs are waaaay better than anything I get from the open source models.
I am not doubting you and my experience has been the same. My current home lab has a pretty good Jupyter server where I experiment with different local models vs GPT using LangChain and the simple chains can achieve some impressive parity with GPT3.5 depending on the use case and local model. Things do break down when we I do more complex things due to compute capacity. Im still running all of the local models on CPU mind you. I have not gotten to the point of testing on a high end GPU yet but based on what ive seen so far, it won't take much more to run smaller local models that are good enough. This is they key. On the client side, we want smaller more focused models. This is what the post linked in this thread hints at and I agree. We are months, if not weeks, if not days...and maybe hours (at this pace!) where those smaller more domain specific models are common. Still, they won't solve the issues I mentioned above. You will likely need to build your own platform around it, or pay exorbitant fees to host it in the Cloud.
Are you sure? I have yet to see any evidence that anyone at all (including Google) has built a model (or a "platform" as you prefer to refer to them) that can follow instructions as well as 50% of ChatGPT, let alone GPT-4. I don't think any amount of work in LangChain and vector databases is enough to fix this: you really need a strong base model that is trained to align with human intentions well. Of course if you just want a bot that can answer free-form simple questions, then maybe people can't tell the difference. Just give them some real work to do and it becomes glaringly obvious.
Vector databases such as Milvus are only there to help reduce/minimize hallucinations rather than get rid of them completely. Until we have a model architecture that can perform completion from the _prompt only_ rather than pre-training data, hallucinations will always be present.
I think that analogy is flawed to try to undercut OpenAI’s lead. The reason it’s flawed is that the search business is really lucrative and OpenAI is trying to completely disrupt Google’s business there. So while the AI isn’t a moat, establishing a lead in search is because you obviously will use that to inject ads in the commercial space and capture the market.
Great read, but I don't agree with all of these points. OpenAI's technological moat is not necessarily meaningful in a context where the average consumer is starting to recognize ChatGPT as a brand name.
Furthermore, models which fine-tune LLMs are still dependent on the base model's quality. Having a much higher quality base model is still a competitive advantage in scenarios where generalizability is an important aspect of the use case.
Thus far, Google has failed to integrate LLMs into their products in a way that adds value. But they do have advantages which could be used to gain a competitive lead:
- Their crawling infrastructure could allow their to generate better training datasets, and update models more quickly.
- Their TPU hardware could allow them to train and fine-tune models more quickly.
- Their excellent research divisions could give them a head start with novel architectures.
If Google utilizes those advantages, they could develop a moat in the future. OpenAI has access to great researchers, and good crawl data through Bing, but it seems plausible to me that 2 or 3 companies in this space could develop sizeable moats which smaller competitors can't overcome.
Consumers recognizing ChatGPT might just end up like vacuum cleaners; at least in the UK, people will often just call it a "hoover" but the likelihood of it being a Hoover is low.
It is difficult to see where the moat might exist if it's not data and the majority of the workings are published / discoverable. I don't think the document identifies a readily working strategy to defend against the threats it recognises.
This reminds me of Lego's constant campaign about "don't call them Legos" that was similar and it always made me think the Lego company is very pretentious and I avoid them. I don't think that was their desired effect.
I'll also mark myself as skeptical of the brand-as-moat. I think AskJeeves and especially Yahoo probably had more brand recognition just before Google took over than ChatGPT or openai has today.
> in a context where the average consumer is starting to recognize ChatGPT as a brand name.
That brand recognition could hurt them, though. If the widespread use of LLMs results in severe economic disruption due to unemployment, ChatGPT (and therefore OpenAI) will get the majority of the ire even for the effects of their competition.
You're forgetting the phenomenon of the fast follower or second to market effect. Hydrox and Oreos, Newton and Palm, MySpace and Facebook, etc. Just because you created the market doesn't necessarily mean you will own it long term. Competitors often respond better to customer demand and are more willing to innovate since they have nothing to lose.
> context where the average consumer is starting to recognize ChatGPT as a brand name.
Zoom was once that brand name which was equated to a product. Now, people might say "Zoom call", but may use Teams or Meet or whatever. Similarly, people call a lot of robot vacuum cleaners Roombas, even though they might be some other brand.
Brand recognition is one thing, but the actual product used will always depend on what their employer uses, what their mobile OS might use, or what API their products might use.
For businesses, a lot will be about the cost and performance vs "the best available".
Completely agree. It is interesting, but the gravitas of it seems lower than of course if an executive said this and corroborated it. I do feel that opensource for AI is going to be really interesting and shake things up.
And (probably) through no fault of their own they'll get totally thrown under the bus for this--whether directly but when raises/promotions come around or not.
FWIW I posted Simon's summary because it's what I encountered first, but here's the leaked document itself[0].
Some snippets for folks who came just for the comments:
> While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.
> A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other.
> This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them?
> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.
> And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
> OpenAI can't build a moat because OpenAI isn't a new vertical, or even a complete product.
> Right now the magical demo is being paraded around, exploiting the same "worse is better" that toppled previous ivory towers of computing. It's helpful while the real product development happens elsewhere, since it keeps investors hyped about something.
> The new verticals seem smaller than all of AI/ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That's a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.
ML dominating education seems pretty realistic to me. E.g. this series of prompts for example:
> "Please design a syllabus for a course in Computer Architecture and Assembly language, to be taught at the undergraduate level, over a period of six weeks, from the perspective of an professor teaching the material to beginning students."
> "Please redesign the course as an advanced undergraduate six-month Computer Architecture and Assembly program with a focus on the RISC-V ecosystem throughout, from the perspective of a professional software engineer working in the industry."
> "Under the category of Module 1, please expand on "Introduction to RISC-V ISA and its design principles" and prepare an outline for a one-hour talk on this material"
You can do this with any course, any material, any level of depth - although as you go down into the details, hallucinations do become more frequent so blind faith is unwise, but it's still pretty clear this has incredible educational potential.
Fortunately, what I said was that a single company becomes the sole source for the ML in education; not the same thing and thus I have no conflict with your statement.
> The premise of the paper is that while OpenAI and Google continue to race to build the most powerful language models, their efforts are rapidly being eclipsed by the work happening in the open source community.
Another magnificent unsurprising set of correct prediction(s) [0] [1] [2] and as triumphantly admitted by Google themselves on open source LLMs eating both of their (Google) and OpenAI's lunch.
"When it is the race to the bottom, AI LLM services, like ChatGPT, Claude (Anthropic), Cohere.ai, etc are winning the race. Open source LLMs are already at the finish line."
So, in perfect theory land, people get paid because they provide value. That obviously breaks down at the extremes.
But, for sake of example, let's take Uber, super easy to hate on them, but they have had a measurable impact on reducing deaths from drunk driving. That obviously provides a lot of value to people.
Likewise, it is hard to overstate the value people have gained from smartphones, Apple has made a lot of money but they have also provided a lot of value. Arguments over if the individual value brought is long term good or bad for society are a separate topic, but people value their iPhones and therefor they pay for them. No way could something as complicated as an iPhone have been made with 1970s software engineering technology.
I’m not arguing that. I’m saying the bar is higher and pay relative to value has decreased for all other than at the upper end. Easiest way to think about this is look at percentage revenue paid to engineers.
In that situation it would be reasonable to expect to be paid twice as much while also being able to devote half the working day to personal/open-source projects.
I disagree with this. It's too expensive to train high quality models. For example I don't see how anyone would make an open-source GPT4 unless OpenAI leaks their model to the public.
Can someone dumb this down for me because I don’t understand why this is a surprise… people are getting excited and collaborating to improve and innovate the same models that these larger companies are milking to death
Basically, if I understand correctly, the "status quo" was that the big models by OpenAI and Google that are much better (raw) than anything that was open source recently, would remain the greatest, and the moat would be the technical complexity of training and running those big models.
However, the open sourcing led to tons of people exploring tons of avenues in an extremely quick fashion, leading to the development of models that are able to close in on that performance in a much smaller envelope, destroying the only moat and making it possible for people with limited resources to experiment and innovate.
Note that this is a personal manifesto, which doesn't really represent Google's official stance. Which is unfortunate because I'm largely aligned with this position.
I think OpenAI has a defensible moat by having the first movers' advantage. As the ease of producing written content declines, we can expect the amount of written content to increase in a consummate fashion. Due to OpenAI's position, a vast majority of the newly generated data will come from OpenAI's models. When it comes time to train new models with superior network structures or with new data, no one else is going to be able to differentiate human-generated text from LLM generated text. OpenAI's training data should become far superior to others.
Having enough scale to perpetually offer free/low-cost compute is a moat.
The primary reason ChatGPT went viral in the first place was because it was free, with no restrictions. Back in 2019, GPT-2 1.5B was made freely accessible by a single developer via the TalkToTransformers website, which was the very first time many people talking about AI text generation...then the owner got hit with sticker shock from the GPU compute needed to scale.
AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.
There's "immediately profitable" and "eventually profitable". Vast compute scale allows collection of customer generated data so the latter is possible, AI as of yet is not the former.
So GP point still stands. FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.
The amount of valuable data generated from professionals using these services to work through their problems and find solutions to industry problems is immense. It essentially gives these companies the keys to automating many industries by just...letting people try and make their jobs easier and collecting all data.
All this talk that every investment pays off in the end is faulty and dangerous. Many investments don't pan out, 95% of the firms you see in the ticker this decade might be gone, and yet everyone is very confident is underwriting these "losses for future gains" but really it's economies of scale. It doesn't cost MSFT much more to run the GPU than to turn it on in the first place.
> . FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.
This assumes that there is a corner-able market. Previously, the cost of training was the moat. That appears to have been more of a puddle under the gate than an actual moat.
There was some indication recently that OpenAI was spending over $500k/day to keep it running. Not sure how long thats going to last. AGI is still a pipe dream. Sooner or later , they’re going to have to make money.
Assuming you're talking about the free ChatGPT product, it's important to consider the value of the training data that users are giving them.
Beyond that, they are making a lot of money from their enterprise offerings (API products, custom partnerships, etc.) with more to come soon, like ChatGPT for Business.
I know there are use cases out there, so it's not a dig. I'm curious how many enterprises are actually spending money with OpenAI right now to do internal development. Have they released any figures?
Oh no, they're going belly up in 20,000 days! (i.e. $10B / 500k) Compute is going to keep getting cheaper and they're going to keep optimizing it to reduce how much compute it needs. I'm more curious about their next steps rather than how they're going to keep the lights on for ChatGPT.
The post mentions that they consider "Responsible Release" to be an unsolved hard problem internally. It's possible that they are culturally blind to agents.
They're basically saying that Pandora's Box, assuming it exists, has already been open. Even if OpenAI, Facebook AI Research and Google DeepMind all shut down tomorrow, research capable of producing agents will continue worldwide.
This really ought to mention https://github.com/oobabooga/text-generation-webui, which was the first popular UI for LLaMA, and remains one for anyone who runs it on GPU. It is also where GPTQ 4-bit quantization was first enabled in a LLaMA-based chatbot; llama.cpp picked it up later.
> Having enough scale to perpetually offer free/low-cost compute is a moat.
Its a moat for services, not models, and its only a moat for AI services as long as that compute isn’t hobbled by being used for models which are so inefficient compared to SOTA as to waste the advantage, which underlines why leaning into open source the way this piece urges is in Google’s interests, the same way open source has worked to Google and Amazon’s benefits as service providers in other domains.
(Not so much “the ability to offer free/low-cost compute” but “the advantages of scale and existing need for widely geographically dispersed compute on the cost of both marginal compute and having marginal compute close to the customer where that is relevant”, but those are pretty close to differenly-focussed rephrasings of the same underlying reality.)
Yes there will, that's the problem HN User Minimaxir is talking about.
It will only get less and less expensive for Microsoft in terms of cost. And more and more effective for Microsoft in terms of results delivered.
How do you compete with free? That's the question. The previous internet experience has already shown us that "also be free" is not really a sustainable or even effective answer. You have to be better in some fundamental dimension.
What makes you think open ai won't look at the FOSS improvements, include them in their tech, and make their GPU farm way cheaper, rendering their service even more competitive?
Not to mention it's easy to run stable diffusion, but midjourney is still a good business. I can run sd on my laptop, I still pay for midjourney because it's convenient, the out of the box experience is better than any competition, and it keeps improving.
that's like saying that apple and MS can look into linux and steal ideas. Yes they can do that but it doesnt make linux any less useful. If anything they learned to contribute back to the common pile, because everyone benefits from it. It would be a problem if this was a one-way relationship , which it doesnt seem to be. If Open source is making them more money, why kill it
You are making my point: linux, mac and windows coexist, despite the overwhelming strength of open source, and the proprietary platforms are quite profitable.
I mean, read the article, the author is concerned about that, and wants Google to open source more so it's not just Facebook's lama that gets open source building on it.
The reason why proprietary software ever had a moat simply comes down to: software startups could dump investment capital onto the development process and achieve results much faster, with better user interfaces, allowing them to achieve path dependence in their customer base. Thus we had a few big application verticals that were ultimately won by MS Office, Adobe Photoshop, etc.
If the result here is as marginal as it seems - a few months of advantage in output quality and a slightly more sleek UI - the capital-intensive play doesn't work. The featuresets that industrial users want the most depend on having more control over the stack, not on UI or output quality. The open source models are stepping up to this goal of "cheap and custom". Casual users can play with the open models without much difficulty either, provided they take a few hours to work through an installation tutorial - UI isn't a major advantage when the whole point is that it's a magic black box.
> Casual users can play with the open models without much difficulty either, provided they take a few hours to work through an installation tutorial
That can be quite a barrier for entry for non-powerusers. I wouldn't underestimate serving casual users, considering that the alternative is OSS i.e. giving your shit away for free.
And ChatGPT has a super low barrier to entry while open source alternatives have a high one.
Creating a service that can compete with it on that regard implies you can scale GPU farms in a cost effective way.
It's not as easy as it sounds.
Meanwhile, openai still improves their product very fast, and unlike google, it's their only one. It's their baby. It has their entire focus.
Since for most consumers, AI == ChatGPT, they have the best market share right now, which mean the most user feedback to improve their product. Which they do at a fast pace.
They also understand that to get mass adoption, they need to censor the AI, like MacDonald and Disney craft their family friendly image. Which irritate every geeks, including me, but make commercially sense.
Plus, despite the fact you can torrent music and watch it with VLC, and that Amazon+Disney are competitors, netflix exists. Having a quality service has value in itself.
I would not count open ai as dead as a lot of people seem to desperately want it to be. Just because Google missed the AI train doesn't mean wishful thinking the market to be killed by FOSS is going to make it so.
As usual with those things it's impossible to know in advance what's going to happen, but odds are not disfavoring chatgpt as much as this article says.
> AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.
Anthropic already does, with its models. They are same price or cheaper than OpenAI, with comparable quality.
> Having enough scale to perpetually offer free/low-cost compute is a moat.
Rather than a moat it is a growth strategy. At one point in time you need to start to monetize and this is the moment when rubber hits the road. If you can survive monetization and continue to grow, now you have a moat.
>Research institutions all over the world are building on each other’s work, exploring the solution space in a breadth-first way that far outstrips our own capacity.
what this proves to me without a doubt is that silo'd and proprietary iteration i still very clearly also a massive disadvantage. i really hope companies internalize that. if they just keep scooping up and hiding open-source improvements they very well may still be left behind.
the final quote from the doc:
> And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
1,355 comments
[ 0.19 ms ] story [ 611 ms ] threadIt's a little bit difficult to get what you want out of the models, but I find them very useful! And while the output resolution might be quite low, things are improving & AI upscaling also helps a lot.
The trick is to use AI to do things it would take you five lifetimes to learn. It's a tool to lower opportunity cost, financial capital, and human capital. That gives anyone leveraging it a much bigger platform, and the ability to dream big without resources.
If you can become your own "studio", you're the indie artist of the future. You don't need Disney or Universal Music backing.
Anyone can step up and do this. The artists being threatened can use these tools to do more than they've ever done by themselves.
Subcultures used to provide a counterbalancing force here, but they aren’t doing so well these days.
I watch films and media, listen to music, and I'm only truly fully satisfied a single digit number of times a year. That's a consequence of not enough being created and experimented with.
The long tail is longer than you can imagine, and that's what form fits to your personal interest graph.
We can a Big Bang without all the stupid romantic stuff. We can have different ending versions of Game of Thrones. So much stuff is never made because it takes so resources to produce them.
I think the market will only grow when this technology is available to everybody.
If we'd prevented new technologies from influencing our artwork, our paintings would never have left the cave wall. I'm a musician with live published albums as well; if there comes a time when I think AI will help with my creative process, you can bet that I'll be using it.
Except the single mom in a studio apartment trying to get some pay from her art gigs.
Displaced workers need support to ensure they can weather these transitions, but it doesn't make sense to artificially create demand by fighting new conveniences. If we want to ensure people have money, the solution is to give them money, not give them money in exchange for busywork.
Who should you protect?
I think I'd still go for a human artist for a bigger release such as an album! It's a lot less hassle than sorting through (often rubbish) AI output & engineering your prompts, though it does cost £££ which is the main thing making it prohibitive for single releases.
If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor.
A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns.
A model doesn't have to be better at everything to be better at something.
Because surely price = quality. Solid argumentation there.
"April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT
Berkeley launches Koala, a dialogue model trained entirely using freely available data.
They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100."
>With these two evaluation sets, we conducted a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. In the ratings interface, we present each rater with an input prompt and the output of two models. They are then asked to judge which output is better (or that they are equally good) using criteria related to response quality and correctness.
No, it's not just memorising shakespeare, real humans interacted with the models and rated them.
OpenAI's model isn't immune from this either, so take any so-called evaluation metrics with a huge grain of salt. This also highlights the difficulties of properly evaluating LLMs: any metrics, once set up, can become a memorization target for LLMs and lose their meaning.
Who built AlexNet on the ImageNet dataset?
ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer.
Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.
I use DDG over Google for similar reasons. It's good enough, more "free" (less ads), and has better privacy.
Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform.
In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases.
Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model.
Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident.
This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.
Furthermore, models which fine-tune LLMs are still dependent on the base model's quality. Having a much higher quality base model is still a competitive advantage in scenarios where generalizability is an important aspect of the use case.
Thus far, Google has failed to integrate LLMs into their products in a way that adds value. But they do have advantages which could be used to gain a competitive lead: - Their crawling infrastructure could allow their to generate better training datasets, and update models more quickly. - Their TPU hardware could allow them to train and fine-tune models more quickly. - Their excellent research divisions could give them a head start with novel architectures.
If Google utilizes those advantages, they could develop a moat in the future. OpenAI has access to great researchers, and good crawl data through Bing, but it seems plausible to me that 2 or 3 companies in this space could develop sizeable moats which smaller competitors can't overcome.
It is difficult to see where the moat might exist if it's not data and the majority of the workings are published / discoverable. I don't think the document identifies a readily working strategy to defend against the threats it recognises.
The term of art is Generic Trademark
https://en.m.wikipedia.org/wiki/Generic_trademark
In US common law (and I'd imagine UK too), it's usually something companies want to avoid if at all possible.
Relevant case for Google itself: https://www.intepat.com/blog/is-google-a-generic-trademark/
[1] https://m.youtube.com/watch?v=rRi8LptvFZY
https://www.adrants.com/2005/09/lego-gets-pissy-about-brand-...
Can't really genericise that which was never made specific...
https://tsdr.uspto.gov/#caseNumber=97733261&caseType=SERIAL_...
That brand recognition could hurt them, though. If the widespread use of LLMs results in severe economic disruption due to unemployment, ChatGPT (and therefore OpenAI) will get the majority of the ire even for the effects of their competition.
You're forgetting the phenomenon of the fast follower or second to market effect. Hydrox and Oreos, Newton and Palm, MySpace and Facebook, etc. Just because you created the market doesn't necessarily mean you will own it long term. Competitors often respond better to customer demand and are more willing to innovate since they have nothing to lose.
Zoom was once that brand name which was equated to a product. Now, people might say "Zoom call", but may use Teams or Meet or whatever. Similarly, people call a lot of robot vacuum cleaners Roombas, even though they might be some other brand.
Brand recognition is one thing, but the actual product used will always depend on what their employer uses, what their mobile OS might use, or what API their products might use.
For businesses, a lot will be about the cost and performance vs "the best available".
> [...] It originates from a researcher within Google. [...] The document is only the opinion of a Google employee, not the entire firm. [...]
Some snippets for folks who came just for the comments:
> While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.
> A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other.
> This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them?
> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.
> And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
[0]: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
> OpenAI can't build a moat because OpenAI isn't a new vertical, or even a complete product.
> Right now the magical demo is being paraded around, exploiting the same "worse is better" that toppled previous ivory towers of computing. It's helpful while the real product development happens elsewhere, since it keeps investors hyped about something.
> The new verticals seem smaller than all of AI/ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That's a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.
> "Please design a syllabus for a course in Computer Architecture and Assembly language, to be taught at the undergraduate level, over a period of six weeks, from the perspective of an professor teaching the material to beginning students."
> "Please redesign the course as an advanced undergraduate six-month Computer Architecture and Assembly program with a focus on the RISC-V ecosystem throughout, from the perspective of a professional software engineer working in the industry."
> "Under the category of Module 1, please expand on "Introduction to RISC-V ISA and its design principles" and prepare an outline for a one-hour talk on this material"
You can do this with any course, any material, any level of depth - although as you go down into the details, hallucinations do become more frequent so blind faith is unwise, but it's still pretty clear this has incredible educational potential.
Another magnificent unsurprising set of correct prediction(s) [0] [1] [2] and as triumphantly admitted by Google themselves on open source LLMs eating both of their (Google) and OpenAI's lunch.
"When it is the race to the bottom, AI LLM services, like ChatGPT, Claude (Anthropic), Cohere.ai, etc are winning the race. Open source LLMs are already at the finish line."
[0] https://news.ycombinator.com/item?id=34201706
[1] https://news.ycombinator.com/item?id=35661548
[2] https://news.ycombinator.com/item?id=34716545
Nothing new there.
What productivity improvements have opened up is more opportunities for developers. Larger and more complex systems can be built using better tooling.
to what end, make rich people richer?
So, in perfect theory land, people get paid because they provide value. That obviously breaks down at the extremes.
But, for sake of example, let's take Uber, super easy to hate on them, but they have had a measurable impact on reducing deaths from drunk driving. That obviously provides a lot of value to people.
Likewise, it is hard to overstate the value people have gained from smartphones, Apple has made a lot of money but they have also provided a lot of value. Arguments over if the individual value brought is long term good or bad for society are a separate topic, but people value their iPhones and therefor they pay for them. No way could something as complicated as an iPhone have been made with 1970s software engineering technology.
However, the open sourcing led to tons of people exploring tons of avenues in an extremely quick fashion, leading to the development of models that are able to close in on that performance in a much smaller envelope, destroying the only moat and making it possible for people with limited resources to experiment and innovate.
When you say "models" do you mean TRAINED models?
Wouldn't the best training and supervised/feedback learning still be in the hands of the big players?
An open source "model" of all content in the open internet is great, but it has the garbage-in/garbage-out problem.
AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.
So GP point still stands. FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.
This assumes that there is a corner-able market. Previously, the cost of training was the moat. That appears to have been more of a puddle under the gate than an actual moat.
Beyond that, they are making a lot of money from their enterprise offerings (API products, custom partnerships, etc.) with more to come soon, like ChatGPT for Business.
Feb 24, 2023: Meta launches LLaMA, a relatively small, open-source AI model.
March 3, 2023: LLaMA is leaked to the public, spurring rapid innovation.
March 12, 2023: Artem Andreenko runs LLaMA on a Raspberry Pi, inspiring minification efforts.
March 13, 2023: Stanford's Alpaca adds instruction tuning to LLaMA, enabling low-budget fine-tuning.
March 18, 2023: Georgi Gerganov's 4-bit quantization enables LLaMA to run on a MacBook CPU.
March 19, 2023: Vicuna, a 13B model, achieves "parity" with Bard at a $300 training cost.
March 25, 2023: Nomic introduces GPT4All, an ecosystem gathering models like Vicuna at a $100 training cost.
March 28, 2023: Cerebras trains an open-source GPT-3 architecture, making the community independent of LLaMA.
March 28, 2023: LLaMA-Adapter achieves SOTA multimodal ScienceQA with 1.2M learnable parameters.
April 3, 2023: Berkeley's Koala dialogue model rivals ChatGPT in user preference at a $100 training cost.
April 15, 2023: Open Assistant releases an open-source RLHF model and dataset, making alignment more accessible.
Its a moat for services, not models, and its only a moat for AI services as long as that compute isn’t hobbled by being used for models which are so inefficient compared to SOTA as to waste the advantage, which underlines why leaning into open source the way this piece urges is in Google’s interests, the same way open source has worked to Google and Amazon’s benefits as service providers in other domains.
(Not so much “the ability to offer free/low-cost compute” but “the advantages of scale and existing need for widely geographically dispersed compute on the cost of both marginal compute and having marginal compute close to the customer where that is relevant”, but those are pretty close to differenly-focussed rephrasings of the same underlying reality.)
It will only get less and less expensive for Microsoft in terms of cost. And more and more effective for Microsoft in terms of results delivered.
How do you compete with free? That's the question. The previous internet experience has already shown us that "also be free" is not really a sustainable or even effective answer. You have to be better in some fundamental dimension.
Not to mention it's easy to run stable diffusion, but midjourney is still a good business. I can run sd on my laptop, I still pay for midjourney because it's convenient, the out of the box experience is better than any competition, and it keeps improving.
If the result here is as marginal as it seems - a few months of advantage in output quality and a slightly more sleek UI - the capital-intensive play doesn't work. The featuresets that industrial users want the most depend on having more control over the stack, not on UI or output quality. The open source models are stepping up to this goal of "cheap and custom". Casual users can play with the open models without much difficulty either, provided they take a few hours to work through an installation tutorial - UI isn't a major advantage when the whole point is that it's a magic black box.
That can be quite a barrier for entry for non-powerusers. I wouldn't underestimate serving casual users, considering that the alternative is OSS i.e. giving your shit away for free.
Creating a service that can compete with it on that regard implies you can scale GPU farms in a cost effective way.
It's not as easy as it sounds.
Meanwhile, openai still improves their product very fast, and unlike google, it's their only one. It's their baby. It has their entire focus.
Since for most consumers, AI == ChatGPT, they have the best market share right now, which mean the most user feedback to improve their product. Which they do at a fast pace.
They also understand that to get mass adoption, they need to censor the AI, like MacDonald and Disney craft their family friendly image. Which irritate every geeks, including me, but make commercially sense.
Plus, despite the fact you can torrent music and watch it with VLC, and that Amazon+Disney are competitors, netflix exists. Having a quality service has value in itself.
I would not count open ai as dead as a lot of people seem to desperately want it to be. Just because Google missed the AI train doesn't mean wishful thinking the market to be killed by FOSS is going to make it so.
As usual with those things it's impossible to know in advance what's going to happen, but odds are not disfavoring chatgpt as much as this article says.
Anthropic already does, with its models. They are same price or cheaper than OpenAI, with comparable quality.
> Having enough scale to perpetually offer free/low-cost compute is a moat.
Rather than a moat it is a growth strategy. At one point in time you need to start to monetize and this is the moment when rubber hits the road. If you can survive monetization and continue to grow, now you have a moat.
BroadMind beats DeepMind!
the final quote from the doc:
> And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.