Ask HN: Is GenAI heading for a crypto-esque bubble pop?
Many of these finetunes or base models are dumped with little info and private training datasets. How much of those datasets are questionably legal or ethical? How many are accidentally or intentionally contaminated with test data? No 3rd party ever seems to test for contamination outside the rare social media post.
I see a whole lot of frameworks, papers and APIs claiming SOTA transformers/diffusion performance in this or that, but they seem barebones and janky when one actually goes and investigate them, if you can get them to work at all. Or they are grand promises asking for money now and usability later.
...I am no data scientist, but I am a straight up AI fan. I was finetuning ESRGAN 1x models for img2img/vid2vid years ago. GenAI is the best thing since sliced bread as far as I am concerned. Yet I am getting a serious crypto fraud vibe from everything that is happening, albeit one more targeted at businesses and investors than the crypto wave.
54 comments
[ 2.5 ms ] story [ 201 ms ] threadMost models on HF are like CrapCoins of the crypto era, so forget about 99.5% of them. There are only a few that can be useful (mostly +70b params).
Don't forget that some of this hype can be driven by nVidia to sell more GPUs. Some of it is driven by crypto enthusiasts who now use their GPUs for GenAI.
Forget about benchmarks and SOTA performance claims. Data leaks and you'll never be 100% sure that the team didn't mess with training data to manipulate the benchmarks.
Is GenAI a hype? Most likely, yes. Can it still be useful? Yes. You can still use GPT-4 and Llama 2 70b models for various tasks. Just don't use Langchain or any frameworks. Make your own API calls. And the smaller models (3-7b)? They are useful for when you don't want to chat with the model, like sentiment analysis.
I had modest success in using LLMs to achieve productivity improvements and results for my clients that are otherwise not possible. I used more direct API calls to invoke LLMs so far but am now considering switching to LangChain to make it more maintainable and scalble to turn more of my ideas into products and service offerings for my clients.
At the outset, a standard library that warps around popular LLMs and makes calling them easier seems needed and inevitable.
I suppose if LangChain doesnt find a good path to becoming more mature -- then some other equivalent open-source library will replace it in coming months.
Thanks for the heads-up.
def embed_documents(self, texts: List[str]) -> List[List[float]]:
```I have some transcription correction needs, and I had a set of prompts that could get GPT4 to do it...but boy was it expensive, and I had to break the problem down into multiple steps to get accurate results, increasing costs further.
My 13b Llama2 fine-tune Inkbot ( https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 ) is able to do those same tasks with a single shot prompt. I have some examples of the specific tasks I trained that model on in the readme. But it isn't intended to be a general purpose model. It has a specific skill-set, and it's best to stick to those types of tasks. I'm also able to do batch inference at over 1k tokens/sec with this model on my desktop. That is something simply not possible with a 70b model on consumer hardware.
AI doesn’t have any rivals, perhaps faster and lower-power chips but we haven’t seen all the implications yet.
Humans have desires, and that make them sometimes way more interesting to listen to than AIs, until at least the AI has desire functions worth talking about. They also understand what the other side might have experienced, and give answers that will be more relevant to a human. For instance I asked "Is crypto a scam, fundamentally ?" to chatGPT and he gave me a well meaning very nuanced "no", synthetizing all sources. I would have answered myself: "it sounds useless, it feels dangerous, it may not be a sincere scam in all instances, but you should stay away if you don't have surplus money": am I not more useful ?
Yes, you are more useful to validate your own opinion and disregard ChatGPT as worse because it didn't agree with you. That's mainly the experience of people that fights against the tech in place of learning how to use it.
With crypto, the product is speculation. There are some potentially interesting applications for crypto, but it's never gotten large-scale traction as anything other than a form of gambling. AI doesn't have speculation built in like crypto does. It has to actually solve problems and do useful things to make money, and this is already happening.
There may be a speculative bubble in the VC realm, but when isn't there? It happens literally every time a new startup trend comes around.
My primary concern is the ratio of AI-infrastructure products being built relative to the average utility delta of current end-user products. There are some shining counterexamples to this like Github copilot, but for the most part the utility gain from most text-based generative-AI seems incremental.
Current language models are bad to okay at most tasks - and great at a select few. The capabilities in unstructured ETL and production of labeled datasets fall in the latter category.
I think the next generation of really powerful capabilities to be unlocked with generative-AI will come with 1) an increase in compute availability (training even larger models) 2) advances in multimodal models, and 3) better interfaces to interact with them. Spatial computing will be a really big deal here, and people seem to have forgotten that Apple may have unlocked the next version of interactive computing earlier this year.
In my opinion the significant innovation is the design and implementation of the precision gesture control interface.
I can very much see that in use a decade from now with some future devise iterations be they laptop/watch/glasses/iPad.
#horselesscarriage
Like any tech bubble, you have crowds of investors pouncing on the “next big thing”, which will inevitably be overhyped and oversold, leading to the bubble popping when expectations outpace actual returns; but the core technology is still valid. The dotcom bubble popping didn’t mean the Internet was a fad, the video game crash of 1983 didn’t permanently doom the industry, the railway mania in the 1840s didn’t mean no one ever took a train after it crashed… any new and exciting development attracts grifters, but it eventually stabilises once people learn to separate the hype from the real applications of the tech. I remember when VR went through this - it was absolutely overhyped, but today there’s still a solid community around it even if it didn’t change the world.
With open source models and fine-tunes being uploaded on Huggingface, there’s no fraud, they’re perfectly free, and you can test them out and see for yourself that some work better than others even if benchmarks only tell part of the story. There’s absolutely a glut of thin veneers over APIs and models offering questionable benefits, but there’s also real value in having a solid UI on top of a basic model; we’ll see the ecosystem stabilise eventually.
Literally VCs.
Just to clarify, I'm not talking about LLMs in general, which I do believe have value, but about the current wave of low quality AI startups in the bubble.
Personally, I haven't found a tool like ChatGPT+ that instantly improved my productivity overnight. Brainstorming, writing code, documentation, marketing copy, transforming data etc all became easier with this tool. I'd rather spend a shorter amount of time providing clear instructions to a language model than writing a lot these things from scratch.
Anecdotally, I've tried all the major models from Google, Meta, Anthropic, Stability etc nothing comes close to GPT-4 in understanding instructions properly.
The competition for AI is simply any solution that gets a higher quality of answer than a trained approximation. That's actually a lot of things, but often there's a chain of dependencies in the information lifecycle that lead to combinations of technologies, and so our total solutions frontier expands by having the tech as a "drop-in" option that would otherwise be some custom engineered thing. But the AGI hype, to name one "unlikely AI thing", is silly. Based on what we know, larger genai models are likely to be more normal, not more novel. And normal isn't right or good, it just tells us what we want to hear.
Crypto is actually covering a polar-opposite technological context: it lets you get very high quality answers within a very narrow space. Done right, it tells you what you DON'T want to know.
Prior to LLaMA in February there were almost no openly licensed language models that were any good at all - BLOOM and GPT-J were pretty much the best of the bunch.
Then LLaMA happened and the flood gates opened - there was now a base model which people could fine-tune against to get really great results.
Llama 2 accelerated things even more because it's licensed for commercial use - so now it's worth spending real money building on top of it.
Llama 2 isn't even 2.5 months old at this point. I'm not at all surprised at the rate of innovation around it - there is SO MUCH scope for improvement, and so many new techniques to be explored.
I expect we'll see a slow down in the release of models at some point, once the low hanging fruit has all been picked off - but it's going to be a while away yet.
Bitcoin is a solution in search of a much bigger set of problems than it can currently solve now. The ML Renaissance we've hit right now has clear, immediate utility to many. The moves made by the stock art heavyweights is proof enough of how disruptive it is.
Contrast this with how banks and financial institutions have responded to bitcoin. To my knowledge, there's yet to be a Bank of America Coin, a Wells Fargo blockchain solution, or non-fungible tokens offered by RBC. They clearly felt at least somewhat threatened, but not in any meaningful way.
1. If you have a large amount of unlabeled data, labeling it using LLMs(even a costly XXXB parameter model) can be significantly cheaper than using humans.
2. Text classification works far better with LLMs compared to usual techniques like random forests.
3. We are exploring using LLMs on structured data(tables) for tasks like clustering, where it is difficult to tune other unsupervised approaches. Similarly we are exploring using LLM embeddings for similarity on unsupervised datasets.
4. LLMs could explain their decisions to users(e.g. why was your comment removed?) although work needs to be done on verifying it's correctness.
I wouldn't be surprised if LLM takes the spot of Random forest as the default go to for supervised ML.
The downrounds and devaulations for the ChatGPT wrappers are coming in [0] and realizing that most of these 'AI startups' are overhyped. Especially the 'Chat with PDF' web apps and basic AI copywriting apps using AI APIs.
To suggest otherwise is pretty naive, and the large AI providers (OpenAI, Google, etc) do compete against their own partners.
[0] https://www.theinformation.com/articles/jasper-an-early-gene...
There are a bunch of models because it's easy to grab llama2 etc and do a little bit of fine tuning. There is nothing wrong or bubbly about that.
It has adoption that might not be every person in the world, but the depth and breadth and variety of users is hard to ignore.
Crypto wallets and getting up and running is the failure for most crypto. Yes, there have been improvements to that.
LLM's that took a chat interface went as universal as they could - just a computer anyone can talk to.
One issue with looking at HuggingFace and expecting maturity in 10-12 months is I'm not sure what can be compared as an equal in that number of months in other technologies. I agree it's far from perfect, but also it seems to be improving.
LLM's are just one kind of GenAI as you have pointed out. The AI you were working with before all of this remains relevant.
There's more to this than wanting a prompt to do everything magically. Having to "fine tune" and get each piece working may be the work for a while, and still be usable and beneficial.
Does it need to evolve and get better? It is.
With each passing month are more and more models running on more accessible hardware? It is.
Are people quietly doing a lot with it and not sharing it yet? Sure feels like it.
There’s still a lot of evolution to be had with AI; for example gen AI to interactive AI that uses all the bits we’ve seen so far to compose experiences with more than chatbot interaction.
If anything is going to die in AI it’s all the bespoke projects, not AI as a whole.
Am working on a Linux distro that ditches userspace as we know for a local LLM (root login is still available). The goal is no user accounts, /home and the AI encodes user data as vectors to disk. Need to successfully unlock the vectors rather than a user account.
So far it’s Linux From Scratch, Wayland, hyprland which displays the GPU viewport.
There is still a lot of weird stuff to do with AI. What do I use blockchain for except as yet another fiat currency
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
If you mean the bubble of excitement for AI, then I highly doubt it. LLMs are paradigm shift in how we use computers and what we can accomplish with them, and we've barely scratched the surface of them.