I think that it would also make sense to have a diagram that has open source achievements in Ai for 2023 (at least open weight & inference code). A lot of the announcements in the chart are behind an api so can’t be run locally.
Just thinking off the top of my head, Segment Anything, Llama 1 and 2, Mistral, Stable diffusion XL, ControlNet, Whisper are all open source AI releases this year.
I appreciate the sentiment. It definitely seems possible, but it doesn't ever seem as easy as copy + paste if you ever want performance or to go outside the same generic tutorial.
> Just like this:
Just is doing a lot of heavy lifting there.
Is this the first model you came across?
We're there any dependencies you had to install?
Did you have to check video card compatibility?
Where did you get the command arguments. It looks awfully complicated for a "just", as if there's a lot of options that aren't straightforward to use.
Ye ... with "just like this" I actually mean "proficient with compiling C projects on an Unixy system" which is like years of dev, "power user" or admin experience (not being sarcastic here). For the audience here I would say that "just" is about right though.
I have no clue about anything LLM related. I just made it run after reading some comment on HN pointing in its direction.
My point is that these locally run LLMs seems way "underreported". I even tried to make a "Show HN" post about it but it got zero interest.
With almost every AI story I read on here (granted, I don't read that many, I find them quite boring), the top comment seems to be something like "you can do the same on your machine, see ollama". Case in point, your comment was the first one listed for me. So the interest for running LLMs on here seems quite high. Don't know why your post didn't catch anyone.
no command line involved, has a huggingface browser built in to load in the model of the day, has a chat-like interface for chatGPT like use, and can create a local server to run your local model for your programs to interact with it using an API that is identical to OpenAI's
No they haven't. It's actually quite weird how the whole hype-cycle is working this time; I think tech journalists are so caught up in the drameh of OAI and whatever they haven't a clue that most people can run most of these novelty UCs on their own. You don't even need a speedy rig!
Ye it is strange. And in the tail around Sam Altman there seems to be these semi-religious lunatics believing that only some government mandated big tech monopoly can save mankind from the LLMs.
Meanwhile amateurs and uni researchers post really interesting work on a site named after a hugging emoji ...
Ye well at some point getting a sausage from Deliveroo or Uber Eats was a no-brainer since it was cheaper than in the actual fast food joint, due to some VC selling a dollar for pennies.
I followed your instructions, and it works without a hitch. macOS-13.6 on an Intel iMac with a Radeon Pro 5700 GPU. My first direct contact with an LLM, and I don't understand most of the command line options yet, but it's certainly interesting.
A key difference is that for basic use cases, yes, this is a comparable experience.
Where it stops being comparable is general application across literally millions of use cases. The ChatGPT system has proven itself a valuable utility across industries, people, and use cases. No open model I know of can match it yet.
They don't have to match the market leader, they just have to be "good enough".
There are oodles of use-cases where sending your data to an outside provider is a complete no-go. In these cases OpenAI/Google/whoever-products aren't relevant competition.
I keep hearing this, but which company has this policy? The privacy policy of openAI's enterprise(or Azure's) is not a lot different than say AWS, which everyone uses.
"These actions clearly put confidential information at risk, prompting Samsung to warn its employees about the dangers of using ChatGPT. Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."
"Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."
Part of the problem is that "good enough" can be really difficult to figure out, whereas GPT-3.5 (and of course GPT-4) are almost a guaranteed success with just basic prompting and context fed in via the prompt.
And yes, there are indeed use cases where sending data to an outside provider is a no-go. The bet OpenAI is making is that they can solve for that later while building their business on use cases where it's fine to send data to an outside parameter. It may also simply not be something they care about. In my own work I know of a massive financial enterprise that has prioritized ~30 or so features where it's fine to send that data. OpenAI is not struggling to get their money.
It remains to be seen if OpenAI will also capture this market, or if fine-tuning open models to be "good enough" wins out over time. The point isn't that, though. The point is that their models are so broadly applicable that _anyone_ can get some value quickly without much work.
Imagine some old sci-fi movie where the computer write out the answer to a query, as if it actually was some scene worker typing it to some terminal. On this:
A few years ago peak HN comment was complaining about wget | sh.
Nowadays we just run full on opaque models directly from huggingface without thinking twice about reading anything. Interesting how times change. I wonder what supply chain attacks will come from huggingface, must not be long now.
Never underestimate the value of usability. I appreciate being able to run these locally - I do! - but for a lot of people a website is going to be infinitely more convenient.
Something I find interesting is that in the last year or so, the talk around AI shifted from the model architecture to the trained model. People talk about Mistral 7B e.g., not transformer with rotary position embedding and gelu feed forward network (I don't know Mistral's architecture).
Contrast this to a few years ago we'd talk about Resnet or Retina-net or whatever, not so much about the facebook pertain on Image-net when describing the model.
In 2018 I remember hearing "architecture is the new feature engineering" (mostly meaning over-fitting I undetstood). Now it's all (mostly) about the dataset and training, the architecture, in 2023, was a minor detail. I personally think architecture will make a comeback soon.
It’s because weights are where the magic lies, for the most part.
Annnd nowadays tools like transformers can automatically select the architecture based on the name of the weights, so when interacting with LLMs, you generally just name the weights.
Also MANY models are just using the llama or llama2 architecture.
And a fitting architecture can save orders of magnitude in numbers of weight and training time.
There isn’t one that’s more important than the other. There are no weights at all without an architecture. And any architecture improvements that pay off have a multiplier effect on efficiency.
Nobody on Earth had the resources to get the performance they are getting on these big models with just a default fully connected layer architecture.
I think people talk about the models because the architectural details are generally inaccessible for us since we don't have sufficient training (which it requires quite a lot to have an intuitive understanding I believe). Whereas the models are easily downloaded and tested.
For all we know, Mamba could be overhyped and could only work for small models. Remember retnet[1] which said the same thing, equal to transformer in linear time.
Even in Resnet era, data was the most relevant for accuracy. Google had this internal JFT data, training on which improves imagenet score much larger than different architecture. Also most of the gains after that came from data augmentation techniques or bigger model rather than new architecture.
I've delivered consulting projects using RAG and Mistral 7B. Not saying it's easy to get the prompts right but I've been able to build systems that work well. Just trying to counter the myth that paid APIs are the only way to do this.
I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language.
Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case.
Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.
How do you define "real tasks"? Even small models that can be run on consumer CPU can produce a coherent summary of an email, as an example. Isn't that a real task?
I think that might have been more true several 8-12 months ago. But now it feels like the momentum has swung towards open source. Multiple models are close to or exceeding GPT 3.5 now. They can do a lot of useful things.
I have come to the conclusion that as much as possible I should try to wean myself off of OpenAI immediately. Because it's just not necessary or desirable to be tied to a single vendor anymore for many tasks. And in 2024 the open source capabilities will continue to increase. Soon everyone with a relatively new computer will be running things like LLMs locally. Within a couple of years it will be integrated into every OS or browser.
Why are people benchmarking against 3.5? To me, the real race started with GPT-4. 3.5 and 4 are completely different beasts and that open models are catching up to 3.5 doesn't mean much. I've yet to see anything come close to 4.
AI has been gradually improving for decades, but this is the year we finally noticed.
The big thing was huge progress in natural language understanding. 2023 was the year the Turing test was smashed.
Seeing computers win games, drive cars, optimize systems, even design things wasn’t as subjectively impressive to most of us as being able to talk to them. This was the year we first saw AI that could sort of communicate with us the way we can with each other.
The usual "use cases" mentioned usually seem to boil down to:
- Acting as a lossy text decompressor (harmful)
- Acting as a lossy text compressor (mostly trying and failing to undo the damage from above)
- Acting as an outright bullshit generator (harmful)
- Acting as a poor substitute for a search engine with a propensity for trying to bullshit you (harmful)
- Acting as a poor substitute for a parser (which, even if it worked would be dumb because it doesn't understand structured output, so now you have two parsing problems)
Anyone testing GPT4, without having heard of it or other recent progress, would be flabbergasted to find out it wasn’t human.
One problem with the Turing Test we might have predicted: if people have context on how models have improved, they narrow their expectations to match.
But I suppose the Strong Turing Test is when a model is so much smarter than us, that it can convincingly hide its differences and superiority, even when we have good reason to suspect it’s an AI.
Things are already changing for early adopters. Yesterday, I used a PDF GPT to review an 80+ page contract and GPT4 to explain specific parts of it in depth using layman's terms. I could have hired a lawyer like I have done in the past but that itself is a slow, laborious process and I often feel like my questions don't get thoroughly answered. I feel very confident of my understanding of the contract now.
I also used StableDiffusion to mock-up landscaping designs for our back patio and make ceramic art ideas for my wife last month.
I was finally able to use super resolution to improve the quality of an old video CD of a VHS tape of a theater play my dad starred in with his friends in the 80s. I tried without super resolution many times in the past but it was never good enough.
You are right that it will take a few years before it hits but AI in 2023 is not hype if you know which tool to use when.
When we use it to solve real problems, make some real progress on climate change, then it will be something else from hype and it will be the year of AI.
When you as an individual use it to sort PDFs or something it’s really hype.
83 comments
[ 3.9 ms ] story [ 143 ms ] threadJust thinking off the top of my head, Segment Anything, Llama 1 and 2, Mistral, Stable diffusion XL, ControlNet, Whisper are all open source AI releases this year.
> Just like this:
Just is doing a lot of heavy lifting there.
Is this the first model you came across?
We're there any dependencies you had to install?
Did you have to check video card compatibility?
Where did you get the command arguments. It looks awfully complicated for a "just", as if there's a lot of options that aren't straightforward to use.
Etc
I have no clue about anything LLM related. I just made it run after reading some comment on HN pointing in its direction.
My point is that these locally run LLMs seems way "underreported". I even tried to make a "Show HN" post about it but it got zero interest.
But maybe I am missing something?
This was a few months ago and things may be better now.
You can also use ollama which makes it into a one line install and then a simple command to run any popular model.
no command line involved, has a huggingface browser built in to load in the model of the day, has a chat-like interface for chatGPT like use, and can create a local server to run your local model for your programs to interact with it using an API that is identical to OpenAI's
Meanwhile amateurs and uni researchers post really interesting work on a site named after a hugging emoji ...
Lots, lots of people do not care.
Should limit the stock price prospects though.
Where it stops being comparable is general application across literally millions of use cases. The ChatGPT system has proven itself a valuable utility across industries, people, and use cases. No open model I know of can match it yet.
There are oodles of use-cases where sending your data to an outside provider is a complete no-go. In these cases OpenAI/Google/whoever-products aren't relevant competition.
Have you ever worked in any sector that has security policies?
Even if you haven't, perhaps spend 2 minutes using a search engine?
Here is a first page result for you: https://www.tomshardware.com/news/samsung-fab-workers-leak-c...
Your own link shows Samsung using ChatGPT. Not sure what point you're trying to make with it.
"These actions clearly put confidential information at risk, prompting Samsung to warn its employees about the dangers of using ChatGPT. Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."
Obnoxious comment considering your link shows the opposite of what you're claiming.
Opposite of what I claim? No.
"Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."
And yes, there are indeed use cases where sending data to an outside provider is a no-go. The bet OpenAI is making is that they can solve for that later while building their business on use cases where it's fine to send data to an outside parameter. It may also simply not be something they care about. In my own work I know of a massive financial enterprise that has prioritized ~30 or so features where it's fine to send that data. OpenAI is not struggling to get their money.
It remains to be seen if OpenAI will also capture this market, or if fine-tuning open models to be "good enough" wins out over time. The point isn't that, though. The point is that their models are so broadly applicable that _anyone_ can get some value quickly without much work.
Nowadays we just run full on opaque models directly from huggingface without thinking twice about reading anything. Interesting how times change. I wonder what supply chain attacks will come from huggingface, must not be long now.
I’m reminded of the classic Dropbox HN comment (https://news.ycombinator.com/item?id=8863): why use Dropbox when any Linux user can just use curlftpfs?
Contrast this to a few years ago we'd talk about Resnet or Retina-net or whatever, not so much about the facebook pertain on Image-net when describing the model.
In 2018 I remember hearing "architecture is the new feature engineering" (mostly meaning over-fitting I undetstood). Now it's all (mostly) about the dataset and training, the architecture, in 2023, was a minor detail. I personally think architecture will make a comeback soon.
Annnd nowadays tools like transformers can automatically select the architecture based on the name of the weights, so when interacting with LLMs, you generally just name the weights.
Also MANY models are just using the llama or llama2 architecture.
What's that supposed to mean? Weights are not independent of the underlying architecture.
There isn’t one that’s more important than the other. There are no weights at all without an architecture. And any architecture improvements that pay off have a multiplier effect on efficiency.
Nobody on Earth had the resources to get the performance they are getting on these big models with just a default fully connected layer architecture.
The name is a little confusing, since the package supports non transformers models as well (like RWKV)
I think HN took out the huggingface emoji from my post
I think people talk about the models because the architectural details are generally inaccessible for us since we don't have sufficient training (which it requires quite a lot to have an intuitive understanding I believe). Whereas the models are easily downloaded and tested.
I would definitely include Mamba state space models and of course would prefer a technical review over a corporate review of the year.
[1]: Retentive Network: A Successor to Transformer for Large Language Models: https://arxiv.org/abs/2307.08621
Very curious about the type of projects you've delivered.
I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language.
Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case.
Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.
I have come to the conclusion that as much as possible I should try to wean myself off of OpenAI immediately. Because it's just not necessary or desirable to be tied to a single vendor anymore for many tasks. And in 2024 the open source capabilities will continue to increase. Soon everyone with a relatively new computer will be running things like LLMs locally. Within a couple of years it will be integrated into every OS or browser.
Only in benchmarks as GP said. All the models I have tried either hallucinates like crazy or refuse to answer everything. Nowhere close to GPT 3.5.
The big thing was huge progress in natural language understanding. 2023 was the year the Turing test was smashed.
Seeing computers win games, drive cars, optimize systems, even design things wasn’t as subjectively impressive to most of us as being able to talk to them. This was the year we first saw AI that could sort of communicate with us the way we can with each other.
- Acting as a lossy text decompressor (harmful)
- Acting as a lossy text compressor (mostly trying and failing to undo the damage from above)
- Acting as an outright bullshit generator (harmful)
- Acting as a poor substitute for a search engine with a propensity for trying to bullshit you (harmful)
- Acting as a poor substitute for a parser (which, even if it worked would be dumb because it doesn't understand structured output, so now you have two parsing problems)
- Generating broken and/or extremely poorly factored code (harmful)
- Hiding plagiarism (harmful)
GPT (and the AI hype in general) is the wrong solution to the wrong problem. Look at the effort that Google wasted on Duplex instead of OpenTable.
One problem with the Turing Test we might have predicted: if people have context on how models have improved, they narrow their expectations to match.
But I suppose the Strong Turing Test is when a model is so much smarter than us, that it can convincingly hide its differences and superiority, even when we have good reason to suspect it’s an AI.
I also used StableDiffusion to mock-up landscaping designs for our back patio and make ceramic art ideas for my wife last month.
I was finally able to use super resolution to improve the quality of an old video CD of a VHS tape of a theater play my dad starred in with his friends in the 80s. I tried without super resolution many times in the past but it was never good enough.
You are right that it will take a few years before it hits but AI in 2023 is not hype if you know which tool to use when.
When you as an individual use it to sort PDFs or something it’s really hype.
In fact, I would say thinking that AI is going to solve 'real' issues like that, is more 'hype' than the above commenter.
https://www.nature.com/articles/s41586-021-04301-9
Personally speaking, anything that eliminates and reduces mundane but complex work is high magic. The real deal.
If “hype” pays for itself, it isn’t hype. It just became a necessity.