You're not wrong, but most of the big players will take a while to switch, at least in my experience you have to put more effort into making sure your prompts result in what you want, and that's annoying especially if GPT4 is already working for you. Claude historically has really bad refusals for safe prompts.
Also, GPT4 is cheaper 10/30 $/m vs 15/75 $/m for claude 3 opus - I'm not sure that price hike is worth the _slight_ benchmark improvement.
People don't use GPT-4 because it was created 3 years ago, or because it's pink, or because it has a 4 in the name.
They use it because it's better than any other publicly available model for most people.
If this is better, and people can access it, they'll use it instead of GPT-4.
People would already be using gemini ultra instead if they could access it, but google fucked the rollout out by telling everyone about it and then saying no one could play with it.
> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.
Sounds pretty good.
If OpenAI want to stay in the game, they need something more than offering 'GPT-TEAM, all the features you already had!' or 'We made this 3 years ago'.
Sora was really fantastic. No one has access.
> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.
Tell me this doesn't sound a littllllle bit more exciting than anything OpenAI has been releasing recently?
I look forward to their response... but I agree with sentiment that they better not sit around twiddling their thumbs; the world is moving fast.
"However, all three models are capable of accepting inputs exceeding 1 million tokens and we may make this available to select customers who need enhanced processing power."
Right, and it's seems very doable. We've been getting little bells and whistles like "custom instructions" have felt like marginal addons. Meanwhile huge context windows seem like they are a perfect overlap of (1) achievable in present day and (2) substantial value add.
Firms are hiring from each other all the time. Plus there’s the fact that the base pertaining is being done at higher context lengths, so then the context extending fine tuning is working from a larger base
A paper describing how you might do it published in December last year. The paper was "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". To be clear I don't know if Claude and Gemini actually use this technique but I would not be surprised if they did something similar:
This is indeed huge for Anthropic. I have never been able to use Claude as much simply because of how much it wants to be safe and refuses to answer even for seemingly safe queries. The gap in reasoning (GPQA, MGSM) is huge though, and that too with fewer shots. Thats great news for students and learners at the very least.
They claim that the new models "are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models", looks like about a third of "incorrect refusals" compared to Claude 2.1. Given that Claude 2 was completely useless because of this, this still feels like a big limitation.
The guard rails on the models make the llm-market a complete train wreck. Wish we could just collectively grow up and accept that if a computer says something bad that doesn't have any negative real world impact - unless we let it - just like literally any other tool.
They're not there to protect the user, they're they're to protect the brand of the provider. A bot that spits out evil shit easily screenshotted with the company's brand right there, isn't really great for growth or the company's brand both.
True and this is also the reason why open source models are commonly uncensored.
It's frustrating though because these companies have the resources to do amazing things, but it's been shown that censoring an LLM can dumb it down in general, beyond what it was originally censored for.
Also, this of course. It's just a cheap bandaid to prevent the most egregious mistakes and embarrasing screenshots.
I don't disagree but on the other hand, I never run into problems with the language model being censored because I am not asking it to write bad words just so I can post online that it can't write bad words.
Yeah I spent a lot of time with Claude 2 and if I hadn’t heard online that it’s “censored,” I wouldn’t have even known. It’s given me lots of useful answers in close to natural human language.
Hm, I don't buy this. The statistics shown in the blog post revealing the new Claude models (this submission) show a significant tendency to refuse to answer benign questions.
Just the fact that there's a x% risk it doesn't answer complicates any use case unnecessarily.
I'd prefer if the bots weren't antrophomized at all, no more "I'm your chatbot assistant". That's also just a marketing gimmick. It's much easier to assume something is intelligent if it has a personality.
Imagine if the models weren't even framed as AI at all. What if they were framed as 'flexi-search' a modern search engine that predicts content it hasn't yet indexed.
Now this looks really promising, the only question is if they've taken the constant ridicule by the open LLM community to heart and made it any less ridiculously censored than the previous two.
At this point I wonder how much of the GPT-4 advantage has been OpenAI's pre-training data advantage vs. fundamental advancements in theory or engineering. Has OpenAI mastered deep nuances others are missing? Or is their data set large enough that most test-cases are already a sub-set of their pre-training data?
Most other model fail on basic stuff like the python creator on stack overflow question, they identify Guido as the python creator, so the knowledge is there, but they don't make the connection.
You linked to a lesswrong post with prompts asking the AI to explain jokes (among other tasks?) and said only Openai models can do it, didn't you? I'm confused why you said only OpenAI models can do it?
Ah sorry if it wasn't clear below the jokes there are a few inferring posts and so far yeah didn't see Claude or other to reason the same way as palm or gpt4, (gpt3.5 did got some wrong), haven't had time tho to test mistral large yet. Mixtral didn't get the right. Tho.
This may explain the substantial performance increase in proprietary models over the last 6 months. It also may explain why open-air and others had to drop open models. Distributing copyrighted material via model weights would be problematic.
More than pretraining data, I think the advantage was ChatGPT and how quickly it grew. Remember it was 3.5, and within a month or two, it generated so many actual q&a pairs with rating, feedback, and production level data of how a model will be used by actual users. Those queries and subsequent RLHF + generating better answers for the questions meant the model would have been improved a lot at the SFT stage. Think this is the reason why Anthropic, Google, and Mistral, all three launched their own chatbots, all providing it to users for free and getting realtime q&a data for them to finetune the models on. Google did it with bard too, but it was so bad that not many used it.
My understanding is that GPT-4 had been almost fully trained before ChatGPT was released - they spent around six months testing GPT-4 before making it available to the public, ChatGPT came out 31st November 2022, GPT-4 came out March 14th 2023.
But maybe that was still enough time for them to instruction tune it based on ChatGPT feedback, or at least to focus more of their fine tuning iteration in the areas they learned were strong or weak for 3.5 based on ChatGPT usage?
I don't think it was pretrained on knowledge gaps. A version was already available in testing w select customers. The version released to the public would definitely have feedback from those customers, and finetuned/instruction tuned on the data from ChatGPT.
Training data is publicly available internet (and accessible to everyone). It's the SFT step w high quality examples which determines how well a model is able to answer questions. ChatGPT's virality played a part in that in the sense that OAI got the real world examples + feedback others did not have. And yeah, it would have been logical to focus on 3.5's weaknesses too. From Karpathy's videos, it seems they hired a contractual labelling firm to generate q&a pairs.
I'd guess a bit of both, perhaps more on the data side. One could also flip the question and ask how is this new Anthropic model able to beat GPT-4 in some benchmarks?
As far as data, OpenAI haven't just scraped/bought existing data, they have also on a fairly large scale (hundreds of contractors) had custom datasets created, which is another area they may have a head start unless others can find different ways around this (e.g. synthetic data, or filtering for data quality).
Altman has previously said (on Lex's podcast I think) that OpenAI (paraphrasing) is all about results and have used some ad-hoc approaches to achieve that, without hinting at what those might be. But, given how fast others like Anthropic and Google are catching up I'd assume each has their own bag of tricks too, whether that comes down to data and training or architectural tweaks.
There was a period of time where data was easily accessible, and Open AI suctioned up as much of it as possible. Places have locked the doors since then realizing someone was raiding their pantry.
To get that dataset now would take significantly more expense.
I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output?
Claude isn't available in EU yet, else i'd try it myself. :(
> AFAIK there's no strict EU ban but no EU country is listed here
That's really weird, I just signed up with no issues and my country together with some other EU countries was listed. Now when I try to signup a new account, it says that my region is not supported.
I still have the sms verification from them as proof.
One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.
On the other hand, programmers are very expensive.
At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative.
And first glance at numbers, LLMs are still way underpriced relative to humans.
The value/competency may approach that of a human but the price won't necessarily follow. Price will be determined by market forces. If compute is cheap and competition is fierce then the price can be near free even if it is at human-level intelligence. Then there will be a lot of surplus value created because buyers would be happy to pay $50/million tokens but only have to pay $0.1/million tokens thanks to competition. Frontier models will probably always be expensive though, because frontier by definition means you're sucking up all the available compute which will probably always be expensive.
I have a suspicion that greenfield science will be the last thing automated, at least the non-brute-force kind. AI assistants to do the drugery (smart search agents), but not pick the directions to proceed in.
Too little relevant training data in niche, state of the art topics.
But to the broader point, isn't this progress in a nutshell?
(1) Figure out a thing can be done, (2) figure out how to manufacture with humans, (3) maximize productivity of human effort, (4) automate select portions of the optimized and standardized process, (5) find the last 5% isn't worth automating, because it's too branchy.
From that perspective, software development isn't proceeding differently than any other field historically, with the added benefit that all its inputs and outputs are inherently digital.
I think that picking a direction is not that hard, and I don't know that AI couldn't do it better. I'm not sure mid-tier CEO's won't be on their way out, just like middle management.
On the people-direction side, I expect the span of control will substantially broaden, which will probably lead to fewer manager/leader jobs (that pay more).
You'll always need someone to do the last 5% that it doesn't make sense to data engineer inputs/outputs into/from AI.
Yeah. Right now, its been helping me be more productive in my science by writing code quicker...mainly on the data management side of things.
I do however wonder, at what point do I just describe the hypothesis, point to the data files, and have it design an analysis pipeline, produce the results, interpret the results, then suggest potential follow-up hypotheses, do a literature search on that, then have it write up the grant for it.
It'll probably be like automating most other tasks: the effort is dominated by finding the right data, transforming it into a standardized input stream, then transforming the output back into actions.
Programming became high-level programming (of compilers) became library-glueing/templating/declarative programming... becomes data engineering.
This is also why I have about negative sympathy for artists who are crying about AI taking their jobs.
Programmers (specifically AI researchers) looked at their 300K+ a year salaries and embraced the idea of automating away the work despite how lucrative it would be to continue to spin one's wheels on it. The culture of open source is strong among SWEs, even one's who would lose millions of unrealized gains/earnings as a result of embracing it.
Artists looked at their 30K+ a year salaries from drawing furry hentai on furaffinity and panic at the prospect of losing their work, to the point of making whole political protest movements against AI art. Artists have also never defended open source en mass, and are often some of the first to defend crappy IP laws.
Why be a luddite over something so crappy to defend?
(edit to respond)
I grew up poor as shit and got myself out of that with code. I don't need a lecture about appearing as an elitist.
I'm more than "poking fun" at them - I'm calling them out for lying about their supposed left-wing sensibilities. Artists have postured as being the "vanguard" of the left wing revolution for awhile (i.e. situationalist international and may 68), but the moment that they had a chance to implement their tactics in the art world (open source AI art), they shunned it and cried and embraced ludditism.
Compare this to the world of AI right now. AI has somehow "legally circumvented" copyright laws and we are living in a de-facto post-copyright world. Huggingface and Richard Stallman as an entity/community and individual have done more to democratize access to and give the poors real access to social and economy mobility than any artists have done in the last 10 years, anywhere in the entire world.
You should embrace shit jobs going away, especially in a world where the speed to "re-skill" is often on the orders of hours when AI is involved. I am pointing out that the well-paid AI professional had much to lose and embraced losing it anyway, while the furry artist acted greedily over their pretty awful situation.
Group A making 300K embraces risk more readily than group B making 30k
Wow who would've thought a large income allowed you to take risks and embrace change?
Imagine being a copywriter for 25 years, on 30k, paying a mortgage, running a car, feeding a family, trying to save on what's left... And all your clients dry up. You've got no other skills, you invested your career in copywriting. You don't have the savings to pivot and your kids need new school uniforms now, not when you reskill to a new career.
You lost your clients. Now your home. Maybe your wife and kids too.
Money is a buffer from risk most don't have.
I hope you never feel this and get to keep the luxury of poking fun at other people for being risk averse without the buffer. Maybe bring some compassion to the table tho? Furry art or copywriting, it isn't anyone's place to judge the merit of the income.
Same, although they are helpful for setting expectations for me. I have some use cases (I'm hesitant to call them evals) related to how we use GPT for our product that are a good "real world" test case. I've found that Claude models are the only ones that are up to par with GPT in the past.
I've also seen the opposite, where tiny little 7B models get real close to GPT4 quality results on really specifically use cases. If you're trying to scale just that use case it's significantly cheaper, and also faster to just scale up inference with that specialty model. An example of this is using an LLM to extract medical details from a record.
That continent wants their citizens to be safe. So, their citizens are going to pay the price of not having access to these developments as they are happening. I really doubt any of these big players will willingly launch in EU given how big the fines are from EU.
I hate that they require a phone number but this might be the only way to prevent abuse so I'll have to bite the bullet.
> We’ve made meaningful progress in this area: Opus, Sonnet, and Haiku are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models.
Finally someone who takes this into account, Gemini and chatGPT is such an obstacle sometimes with their unnecessary refusal because a keyword triggered something.
https://openrouter.ai/ lets you make one account and get API access to a bunch of different models, including Claude (maybe not v3 yet - they tend to lag by a few days). They also provide access to hosted versions of a bunch of open models.
Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)
I could only send one message, after that I had to add more credits to my account. I don't really think it's worth paying if I already get Gemini, chatGPT and Claude for free.
I think it's just to get free credits that you need to give a phone number?
To the other point, yes it's crazy that "When inside kitty, how do I get my python inside latex injected into Julia? (It somehow works using alacritty?)"
Despite the question being pretty underspecified or confusing, it still shouldn't read as inappropriate.
Unfortunately, many image generation systems will refuse prompts with latex in them (I assumed it was a useful term for styling).
My best guess is that it thinks latex is more often used as a clothing item or something, and it's generally associated with inappropriate content. Just unfortunate for scientists :/.
Less refusals than "previous generations of models" presumably means that is has less refusals than _their_ previous generations of models (= Claude 2), which was notorious for being the worst in class when it came to refusals. I wouldn't be surprised if it's still less permissive than GPT-4.
Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive.
Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models - of course some team members had the know-how of being ex. OpenAI) and are, temporarily at least, now ahead of OpenAI in some of these benchmarks.
I'd assume that OpenAI's next-gen model (GPT-5 or whatever they will choose to call it) has already finished training and is now being fine tuned and evaluated for safety, but Anthropic's cause d'etre is safety and I doubt they have skimped on this to rush this model out.
They are funded mostly by Microsoft, and dependent on them for compute (which is what this funding is mostly buying), but I'd hardly characterize that as meaning they are "Microsoft in a trenchcoat". It's not normal to identify startups as being their "VC in a trenchcoat", even if they are dependent on the money for growth.
Satya Nadella during the OpenAI leadership fiasco: “We have all of the rights to continue the innovation, not just to serve the product, but we can, you know, go and just do what we were doing in partnership ourselves. And so we have the people, we have the compute, we have the data, we have everything.”
Doesn’t sound like a startup-investor relationship to me!
Sure, but that's just saying that Microsoft as investor has some rights to the underlying tech. There are limits to this though, which we may fairly soon be nearing. I believe the agreement says that Microsoft's rights to the tech (model + weights? training data? -- not sure how specific it is) end once AGI is achieved, however that is evaluated.
But again, this is not to say that OpenAI is "Microsoft in a trenchcoat". Microsoft don't have developers at OpenAI, weren't behind the tech in any way, etc. Their $10B investment bought them some short-term insurance in limited rights to the tech. It is what is is.
“We have everything” is not “some underlying rights to the tech.” I dunno what the angle is on minimizing here, but I’ll take the head of Microsoft at his word vs. more strained explanations about why this isn’t the case.
It's also explicitly mentioned in Musk's lawsuit against OpenAI. Much as Musk wants to claim that OpenAI is a subsidiary of Microsoft, even he has to admit that if in fact OpenAI develop AGI then Microsoft won't have any IP rights to it!
The context for Nadella's "We have everything" (without of course elaborating on what "everything" referred to) is him trying to calm investors who were just reading headlines about OpenAI imploding in reaction to the board having fired Altman, etc. Nadella wasn't lying - he was just being coy about what "everything" meant, wanting to reassure investors that their $10B investment in OpenAI had not just gone up in smoke.
OpenAI has not and will likely never develop AGI, so this is akin to saying “Microsoft doesn’t own OpenAI because they have a clause in their contract that’s says they stop owning it when leprechauns exist.” Musk is trying to argue leprechauns exist because he’s mad he got outmaneuvered by Altman, which I imagine will go as well as you’d expect that argument to go in a court of law.
What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyone can push the limits of research, but once the cat’s out of the bag, anyone with a few $B and dozen engineers can replicate a model that’s indistinguishably good from best in class to most users.
Barrier to entry with "few $B" is pretty high.
Especially since the scaling laws indicate that it's only getting more expensive.
And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...
You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers.
Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash.
In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances.
Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea.
I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
Also worth keeping in mind the lock in for the big tech firms is due to business decisions not the technology per se. If we had say micropaynents in http1 headers in 1998 we might have a much more decentralized system supported by distributed subscriptions rather than ads. To this day I cannot put up $50 to mastodon and have it split amongst the posts I like or boost or whatever. Instead we have all the top content authors trying to get me to subscribe to their email subscriptions which Isa vastly inferior interface and too expensive to get money to all the good writers out there.
There is no meaningful network effect or vendor lock-in - which is like the #1 thing that prevents companies from competing. That's the real problem for these AI companies.
Yes, it seems that AI in form of LLMs is just an idea whose time has come. We now have the compute, the data, and the architecture (transformer) to do it.
As far as different groups leapfrogging each other for supremacy in various benchmarks, there might be a bit of a "4 minute mile" effect here too - once you know that something is possible then you can focus on replicating/exceeding it without having to worry are you hitting up against some hard limit.
I think the transformer still doesn't get the credit due for enabling this LLM-as-AI revolution. We've had the compute and data for a while, but this breakthough - shared via a public paper - was what has enabled it and made it essentially a level playing field for anyone with the few $B etc the approach requires.
I've never seen any claim by any of the transformer paper ("attention is all you need") authors that they understood/anticipated the true power of this model they created (esp. when applied at scale), which as the title suggests was basically regarded an incremental advance over other seq2seq approaches of the time. It seems like one of history's great accidental discoveries. I believe there is something very specific about the key-value matching "attention" mechanism of the transformer (perhaps roughly equivalent to some similar process used in our cortex?) that gives it it's power.
> We now have the compute, the data, and the architecture (transformer) to do it.
It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different wiring.
The merit for our current wave is 99% on the training data, its quality and size are the true AI heroes. And it took humanity our whole existence to build up to this training set, it cost "a lot" to explore and discover the concepts we put inside it. A single human, group or even a whole generation of humans would not be able to rediscover it from scratch in a lifetime. Our cultural data is smarter than us individually, it is as smart as humanity as a whole.
One consequence of this insight is that we are probably on an AI plateau. We have used up most organic text. The next step is AI generating its own experiences in the world, but it's going to be a slow grind in many fields where environment feedback is not easy to obtain.
> It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify.
My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience.
The ultimate intelligent architecture, for now, is our own cortex, which can be architecturally analyzed as a prediction machine - utilizing masses of perceptual feedback to correct/update predictions of how the perceptual scene, and results of our own actions, will evolve.
With prediction as the basis of intelligence, any model capable of predicting - to varying degrees of success - will be perceived to have a commensurate degree of intelligence. Transformer-based LLMs of course aren't the only possible way to predict, but they do seem significantly better at it than competing approaches such as Mamba or the RNN (LSTM etc) seq2seq approaches that were the direct precursor to the transformer.
I think the reason the transformer architecture is so much better than the alternatives, even if there are alternatives, is down to this specific way it does it - able to create these attention "keys" to query the context, and the ways that multiple attention heads learn to coordinate such as "induction heads" copying data from the context to achieve in-context learning.
If you invented the transformer but didn't have trillions of tokens to train it with, no chatGPT. But if you had Mamba/RWKV/SSSM and trillions of tokens you would have almost the same thing with chatGPT.
The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intelligence is a social process, even though we like to think about keys and queries, or synapses and neurotransmitters, in fact it is the work of many people that made it possible.
And language is that central medium between all of us, an evolutionary system of ideas, evolving at a much faster rate than biology. Now AI have become language replicators like humans, a new era in the history of language has begun. The same language trains humans and LLMs to achieve similar sets of abilities.
I agree about language - which might be though of as "thought macros". Human experience has taught us what things (objects, actions, etc) are worth labelling, what thought patterns are useful to reason about them, etc. Being able to reason about things in the realm of, and using the patterns of, human language is tremendously powerful.
Are there any Mamba benchmarks that show it matching transformer (GPT, say) benchmark performance for similiar size models and training sets?
I don’t think we are at a plateau. We may have fed a large amount of text into these models, but when you add up all other kinds of media, images, videos, sound, 3D models, there’s a castle more rich dataset about the world. Sora showed that these models can learn a lot about physics and cause and effect just from video feeds. Once this is all combined together into multimodal mega models then we may be closer to the plateau.
MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.
And yet it’s still a good indicator of general performance. Any model that scores under GPT-4 on that benchmark, but above it in other, tends to be worse overall.
"In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"
Yeah the output pricing I think is really interesting, 150% more expensive input tokens 250% more expensive output tokens, I wonder what's behind that?
That suggests the inference time is more expensive then the memory needed to load it in the first place I guess?
I'm more curious about the input/output token discrepancy
Their pricing suggests that either output tokens are more expensive for some technical reason, or they're trying to encourage a specific type of usage pattern, etc.
Or that market research showed a higher price for input tokens would drive customers away, while a lower price for output tokens would leave money on the table.
That's quite expensive indeed. At full context of 200K, that would be at least $3 per use. I would hate it if I receive a refusal as answer at that rate.
You are not going to take the expensive human out of the loop where downside risk is high. You are likely to take the human out of the loop only in low risk low cost operations to begin with. For those use cases, these models are quite expensive.
Just a note that the 67.0% HumanEval figure for GPT-4 is from its first release in March 2023. The actual performance of current ChatGPT-4 on similar problems might be better due to OpenAI's internal system prompts, possible fine-tuning, and other tricks.
There’s a market for that though. If I am running a startup to generate video meeting summaries, the price of the models might matter a lot, because I can only charge so much for this service. On the other hand, if I’m selling a tool to have AI look for discrepancies in mergers and acquisitions contracts, the difference between $1 and $5 is immaterial… I’d be happy to pay 5x more for software that is 10% better because the numbers are so low to begin with.
My point is that there’s plenty of room for high priced but only slightly better models.
Once you’ve taken all the data in the world and trained a sufficiently large model on it, it’s very hard to improve on that base. It’s possible that GPT-4 basically represents that benchmark, and improvements will require better parsing/tokenization, clever synthetic data methods, building expert datasets. Much harder than just scraping the internet and doing next token after some basic data cleaning.
Did some quick tests and Claude 3 Sonnet responses have been mostly wrong compared to Gemini :/ (was asking it to describe certain GitHub projects and Claude was making stuff up)
Does any of those LLM-as-a-service companies provide a mechanism to "save" a given input? Paying only for the state storage and the extra input when continuing the completion from the snapshot?
Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity.
I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient.
Right now, only ChatGPT (the webapp) seems to be using such those snapshots.
I think the answer's in the original question: the provider has to pay for extra storage to cache the model state at the prompt you're asking to snapshot. But it's not necessarily a net increase in costs for the provider, because in exchange for doing so they (as well as you) are getting to avoid many expensive inference rounds.
The "cost" is storing the state of the LLM after processing the input. My back-of-the-envelop guesstimate gives me 1GB to capture the 8bit state of 70B parameters model (I might be wrong though, insights are welcome), which is quite manageable with NVMe storage for fast reload. The operator would charge per pay per "saved" prompt, plus maybe a fix per call fee to re-load the state.
My calculation of kv cache gives 1GB per 3000 tokens for fp16. I am surprised openAI competitors haven't done this. This kind of features have not so niche uses, where prefix data could be cached.
That's a great idea! It would also open up the possibility for very long 'system prompts' on the side of the company, so they could better fine-tune their guardrails
FWIW the use case you're describing is very often achievable with RAG. Embedding models are deterministic, so while you're still limited by the often-nondeterministic nature of the LLM, in practice you can usually get the same answer for the same input. And it's substantially cheaper to do.
With 1M tokens, if snapshotting the LLM state is cheap, it would beat out-of-the-box nearly all RAG setups, except the ones dealing with large datasets. 1M tokens is a lot of docs.
Yeah, but latency is still a factor here. Any follow-up question requires re-scanning the whole context, which often takes a long time. IIRC when Google showed their demos for this use case each request took over 1 minute for ~650k tokens.
> I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient.
If you don't care about latency or can wait to set up a batch of inputs in one go there's an alternative method. I call it batch prompting and pretty much everything we do at work with gpt-4 uses this now. If people are interested I'll do a proper writeup on how to implement it but the general idea is very straightforward and works reliably. I also think this is a much better evaluation of context than needle in a haystack.
Example for classifying game genres from descriptions.
I attempted similar mechanics multiple times in the past, but always ditched them, as there was always a non-negligable amount of cross-contamination happening between the individual instances you are batching. That caused so much of a headache that it wasn't really worth it.
Yeah that's definitely a risk with language models but it doesn't seem to be too bad for my use cases. Can I ask what tasks you used it for?
I don't really intend for this method to be final. I'll switch everything over to finetunes at some point. But this works way better than I would have expected so I kept using it.
One thing I tried using it for was for a summarization/reformulation tasks, where it did RAG of ~3-4 smallish (~single sentence) documents per instance where each should be in the end form a coherent sentence. There, batching either caused one of the facts to slip into an adjacent instance or two instances to be merged into one.
Another thing I used it for was data extraction, where I extracted units of measurements and other key attributes out of descriptions from classifieds listings (my SO and me were looking for a cheap used couch). Non-batched it performed very well, while in the batched mode, it either mixed dimensions of multiple listings or after the summary for the initial listing it just gave nulls for all following listings.
Yes: That's essentially their fine-tuning offerings. They rewrite some weights in the top layers based on your input, and save+serve that for you.
It sounds like you would like a wrapped version tuned just for big context.
(As others write, RAG versions are also being supported, but they're less fundamentally similar. RAG is about preprocessing to cut the input down to relevant bits. RAG + an agent framework does get closer again tho by putting this into a reasoning loop.)
The problem is that it’s probably often not a lot cheaper. Most of the high end gpus have comparatively little bandwidth over pcie (that you’d need to use to store the context on a nvme for example). The cost there would scale with length too so you wouldn’t necessarily save more in that situation either. I think if you used a small enough gqa ratio and you knew for sure you would reuse the weights it could work, but my suspicion is that in general it would just be cheaper to recalculate.
Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon.
From my early tests this seems like the first API alternative to GPT4. Huge!
No catch. We're pretty early tbh so mostly looking to get some early power users and make the product great before doing a big launch. It's been popular with yc founders in the latest batches thus far but we haven't really shared publicly. We'll charge when we launch. If you try it now, I hope you'll share anything you liked and didn't like with us!
Probably not data so much as growth numbers to appease investors. Such offerings typically don’t last forever. Might as well take advantage while it lasts.
Hey Wesley, I just checked Double. Do you plan to support open source models hosted locally or on a cloud instance? Asking out of curiosity as I am building a product in the same space and have had a few people ask this. I guess since Double is an extension in IDEs, it can connect to AI models running anywhere.
it's an interesting idea. We asked our users this as well but at least for those we talked to, running their own model wasn't a big priority. What actually mattered to them is being able to try different (but high performance) models, privacy (their code not being trained on), and latency. We have some optimizations around time-to-first-token latency that would be difficult to do if we didn't have information about the model and their servers.
I see. Thanks Wesley for sharing and great to know it is not a priority. Also, the Mistral situation kinda makes me feel that big corps will want to host models.
Although, I feel Apple will break this trend and bring models to their chips rather than run them on the cloud. "Privacy first" will simply be a selling point for them but generally speaking cloud is not a big sell for them.
I am not at the level to do much optimizations, plus my product is a little more generic. To get to MVP, prompt engineering will probably be my sole focus.
Thank you for working on gptel, it's an excellent package. I'm still using the copilot more because of the pure speed (competing with company mode/LSP), but I never use it if it suggests more than one line. The quality is just not there. But having access to gpt4 from gptel has been very useful. Can't wait to play around with Claude 3.
Thanks for the feedback. I was actually reworking the default shortcuts and the onboarding process when I got pre-empted by claude. I was planning to change the main actions to alt-j, alt-k to minimize conflicts.
Are you asking because it conflicts with an existing shortcut on your setup? Or another reason?
Yes, it conflicts with some of my other shortcuts, but more generally, I think it'd be better to have consistent shortcuts, like CMD-CTRL-i for inline completion, CMD-CTRL-c for chat, etc.
for future readers, the setting is now shipped in >v0.49. The default is now back to GPT-4 as it has lower latency but you can manually change it to Claude 3 in settings if you wish to try out Anthropic's new model.
I noticed it might actually be a little more censored than the lmsys version. Lmsys seems more fine with roleplaying, while the one on Double doesn't really like it
I think the tldr would be that they have more products (for example, their agent to write git commit messages). In the products we do have (autocomplete, chat), we spend a lot of time to get the details right. For example for autocomplete:
* we always close any brackets opened by autocomplete (and never extra brackets, which is the most annoying thing about github copilot)
* we automatically add import statements for libraries that autocomplete used
* mid-line completions
* we turn off autocomplete when you're writing a comment to avoid disrupting your train of thought
Wow you are right. That is confusing! I was asking about Codeium (with an e) but I linked the wrong one in my post. The vim plugin link is correct though.
Explain the following article in smaller points and in very simple words and in 750 words for better understanding.Huawei Chip Breakthrough Used Tech From Two US Gear Suppliers
SMIC used Applied Materials and Lam equipment to make 7nm chip
US wants to further limit China’s access to foreign chip tech
Bloomberg has learned that Huawei and its partner SMIC relied on gear from Applied Materials Lam Researchto produce an advanced chip.
Huawei Technologies Co. and its partner Semiconductor Manufacturing International Corp. relied on US technology to produce an advanced chip in China last year, according to people with knowledge of the matter.
Shanghai-based SMIC used gear from California-based Applied Materials Inc. and Lam Research Corp. to manufacture an advanced 7-nanometer chip for Huawei in 2023, the people said, asking not to be named as the details are not public.
The previously unreported information suggests that China still cannot entirely replace certain foreign components and equipment required for cutting-edge products like semiconductors. The country has made technological self-sufficiency a national priority and Huawei’s efforts to advance domestic chip design and manufacturing have received the backing of Beijing.
Representatives of SMIC, Huawei and Lam did not respond to requests for comment. Applied Materials and the US Commerce Department’s Bureau of Industry and Security, which is responsible for implementing export controls, declined to comment.
Lauded in China as a major leap in indigenous semiconductor fabrication, last year’s SMIC-made processor powered Huawei’s Mate 60 Pro and a wave of patriotic smartphone-buying in the Asian country. The chip is still generations behind the top components from global firms, but ahead of where the US hoped to stop China’s advance.
The machinery used to make it, however, still had foreign sources including technology from Dutch maker ASML Holding NV as well as the gear from Lam and Applied Materials. Bloomberg News reported in October that SMIC had used equipment from ASML for the chip breakthrough.
Leading Chinese chip equipment suppliers including Advanced Micro-Fabrication Equipment Inc. and Naura Technology Group Co. have been trying to catch up with their American peers, but their offerings are still not as comprehensive or sophisticated. China’s top lithography system developer Shanghai Micro Electronics Equipment Group Co. still lags a few generations behind what industry leader ASML is capable of.
SMIC obtained the American machinery before the US banned such sales to China in October 2022, some of the people said. Both firms were among the American suppliers that began pulling their staff from China after those rules went into effect and prohibited US engineers from servicing some machines in the Asian country. ASML also told American employees to stop working with Chinese customers in response to the US curbs, but Dutch and Japanese engineers are still able to service many machines in China — much to the chagrin of their American rivals.
Companies are now prohibited from selling cutting-edge, US-origin technology to either SMIC or Shenzhen-based Huawei. Both tech firms have been blacklisted by the US for alleged links to the Chinese military, while Washington has been tightening China’s overall access to chipmaking equipment and advanced semiconductors.
Those trade curbs pushed Huawei and SMIC to pursue avenues for building a domestic chip supply chain, and the Mate 60 Pro marked a surprising advance in that effort.
After Huawei released the new phone, Washington launched a probe into its processor and US Commerce Secretary Gina Raimondo vowed the “strongest possible” actions to ensure national security. Meanwhile, Republican lawmakers have called for the Biden administration to completely cut off Huawei and SMIC’s access to US technology.
Department of Commerce officials have said they haven’t seen evidence that SMIC can make the 7nm chips “at scale,” a point echoed by ASML’s Chief Executive O...
Huawei Chip Breakthrough Used Tech From Two US Gear Suppliers
SMIC used Applied Materials and Lam equipment to make 7nm chip
US wants to further limit China’s access to foreign chip tech
Bloomberg has learned that Huawei and its partner SMIC relied on gear from Applied Materials Lam Researchto produce an advanced chip.
Huawei Technologies Co. and its partner Semiconductor Manufacturing International Corp. relied on US technology to produce an advanced chip in China last year, according to people with knowledge of the matter.
Shanghai-based SMIC used gear from California-based Applied Materials Inc. and Lam Research Corp. to manufacture an advanced 7-nanometer chip for Huawei in 2023, the people said, asking not to be named as the details are not public.
The previously unreported information suggests that China still cannot entirely replace certain foreign components and equipment required for cutting-edge products like semiconductors. The country has made technological self-sufficiency a national priority and Huawei’s efforts to advance domestic chip design and manufacturing have received the backing of Beijing.
Representatives of SMIC, Huawei and Lam did not respond to requests for comment. Applied Materials and the US Commerce Department’s Bureau of Industry and Security, which is responsible for implementing export controls, declined to comment.
Lauded in China as a major leap in indigenous semiconductor fabrication, last year’s SMIC-made processor powered Huawei’s Mate 60 Pro and a wave of patriotic smartphone-buying in the Asian country. The chip is still generations behind the top components from global firms, but ahead of where the US hoped to stop China’s advance.
The machinery used to make it, however, still had foreign sources including technology from Dutch maker ASML Holding NV as well as the gear from Lam and Applied Materials. Bloomberg News reported in October that SMIC had used equipment from ASML for the chip breakthrough.
Leading Chinese chip equipment suppliers including Advanced Micro-Fabrication Equipment Inc. and Naura Technology Group Co. have been trying to catch up with their American peers, but their offerings are still not as comprehensive or sophisticated. China’s top lithography system developer Shanghai Micro Electronics Equipment Group Co. still lags a few generations behind what industry leader ASML is capable of.
SMIC obtained the American machinery before the US banned such sales to China in October 2022, some of the people said. Both firms were among the American suppliers that began pulling their staff from China after those rules went into effect and prohibited US engineers from servicing some machines in the Asian country. ASML also told American employees to stop working with Chinese customers in response to the US curbs, but Dutch and Japanese engineers are still able to service many machines in China — much to the chagrin of their American rivals.
Companies are now prohibited from selling cutting-edge, US-origin technology to either SMIC or Shenzhen-based Huawei. Both tech firms have been blacklisted by the US for alleged links to the Chinese military, while Washington has been tightening China’s overall access to chipmaking equipment and advanced semiconductors.
Those trade curbs pushed Huawei and SMIC to pursue avenues for building a domestic chip supply chain, and the Mate 60 Pro marked a surprising advance in that effort.
After Huawei released the new phone, Washington launched a probe into its processor and US Commerce Secretary Gina Raimondo vowed the “strongest possible” actions to ensure national security. Meanwhile, Republican lawmakers have called for the Biden administration to completely cut off Huawei and SMIC’s access to US technology.
Department of Commerce officials have said they haven’t seen evidence that SMIC can make the 7nm chips “at scale,” a point echoed by ASML’s Chief Executive Officer Peter Wennink.
If SMIC wants to advance its technology without ASML’s state-of-the-art extreme ultraviol...
Just a comment about the first chart: having the X axis in log scale to represent the cost and a Y axis without any units at all for the benchmark score seem intentionally misleading.
I don't understand the need to do that when your numbers look promising.
722 comments
[ 3.0 ms ] story [ 411 ms ] threadGPT-4 has little "lock-in" and isn't "good enough" the keep users via inertia.
You're not wrong, but most of the big players will take a while to switch, at least in my experience you have to put more effort into making sure your prompts result in what you want, and that's annoying especially if GPT4 is already working for you. Claude historically has really bad refusals for safe prompts.
Also, GPT4 is cheaper 10/30 $/m vs 15/75 $/m for claude 3 opus - I'm not sure that price hike is worth the _slight_ benchmark improvement.
They use it because it's better than any other publicly available model for most people.
If this is better, and people can access it, they'll use it instead of GPT-4.
People would already be using gemini ultra instead if they could access it, but google fucked the rollout out by telling everyone about it and then saying no one could play with it.
> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.
Sounds pretty good.
If OpenAI want to stay in the game, they need something more than offering 'GPT-TEAM, all the features you already had!' or 'We made this 3 years ago'.
Sora was really fantastic. No one has access.
> Opus and Sonnet are available to use today in our API, which is now generally available, enabling developers to sign up and start using these models immediately.
Tell me this doesn't sound a littllllle bit more exciting than anything OpenAI has been releasing recently?
I look forward to their response... but I agree with sentiment that they better not sit around twiddling their thumbs; the world is moving fast.
Now this is interesting
It's like every AI researcher had an epiphany all at once
https://arxiv.org/abs/2312.00752
https://github.com/state-spaces/mamba
It's frustrating though because these companies have the resources to do amazing things, but it's been shown that censoring an LLM can dumb it down in general, beyond what it was originally censored for.
Also, this of course. It's just a cheap bandaid to prevent the most egregious mistakes and embarrasing screenshots.
https://twitter.com/iliaishacked/status/1681953406171197440
Both sides in this to me need to get a life.
Just the fact that there's a x% risk it doesn't answer complicates any use case unnecessarily.
I'd prefer if the bots weren't antrophomized at all, no more "I'm your chatbot assistant". That's also just a marketing gimmick. It's much easier to assume something is intelligent if it has a personality.
Imagine if the models weren't even framed as AI at all. What if they were framed as 'flexi-search' a modern search engine that predicts content it hasn't yet indexed.
Llms are an illusion of general intelligence. What is different about these models that leads to such a claim? Marketing hype?
Most other model fail on basic stuff like the python creator on stack overflow question, they identify Guido as the python creator, so the knowledge is there, but they don't make the connection.
You're saying that when Mistral Large launched last week you tested it on (among other things) explaining jokes?
But maybe that was still enough time for them to instruction tune it based on ChatGPT feedback, or at least to focus more of their fine tuning iteration in the areas they learned were strong or weak for 3.5 based on ChatGPT usage?
Training data is publicly available internet (and accessible to everyone). It's the SFT step w high quality examples which determines how well a model is able to answer questions. ChatGPT's virality played a part in that in the sense that OAI got the real world examples + feedback others did not have. And yeah, it would have been logical to focus on 3.5's weaknesses too. From Karpathy's videos, it seems they hired a contractual labelling firm to generate q&a pairs.
As far as data, OpenAI haven't just scraped/bought existing data, they have also on a fairly large scale (hundreds of contractors) had custom datasets created, which is another area they may have a head start unless others can find different ways around this (e.g. synthetic data, or filtering for data quality).
Altman has previously said (on Lex's podcast I think) that OpenAI (paraphrasing) is all about results and have used some ad-hoc approaches to achieve that, without hinting at what those might be. But, given how fast others like Anthropic and Google are catching up I'd assume each has their own bag of tricks too, whether that comes down to data and training or architectural tweaks.
To get that dataset now would take significantly more expense.
Claude isn't available in EU yet, else i'd try it myself. :(
I'm currently in EU and I have access to it?
https://www.anthropic.com/claude-ai-locations
Perhaps you meant Europe the continent or using a VPN?
edit: They seem to have updated that list after I posted my comment, the outdated list I based my comment on: https://web.archive.org/web/20240225034138/https://www.anthr...
edit2: I was confused. There is another list for API regions, which has all EU countries. The frontend is still not updated.
I was just able to sign up, while not being able to a few weeks ago.
But it still errors out when trying to sign up from Germany:
https://i.imgur.com/rX0XA8d.jpeg
https://i.imgur.com/Xlyqm8D.jpeg
That's really weird, I just signed up with no issues and my country together with some other EU countries was listed. Now when I try to signup a new account, it says that my region is not supported.
I still have the sms verification from them as proof.
At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative.
And first glance at numbers, LLMs are still way underpriced relative to humans.
It would be an ironic thing that it was open source that killed the programmer; as how would they train it otherwise?
As a scientist, should I continue to support open access journals, just so I can be trained away?
Slightly tongue in check, but not really.
Too little relevant training data in niche, state of the art topics.
But to the broader point, isn't this progress in a nutshell?
(1) Figure out a thing can be done, (2) figure out how to manufacture with humans, (3) maximize productivity of human effort, (4) automate select portions of the optimized and standardized process, (5) find the last 5% isn't worth automating, because it's too branchy.
From that perspective, software development isn't proceeding differently than any other field historically, with the added benefit that all its inputs and outputs are inherently digital.
On the people-direction side, I expect the span of control will substantially broaden, which will probably lead to fewer manager/leader jobs (that pay more).
You'll always need someone to do the last 5% that it doesn't make sense to data engineer inputs/outputs into/from AI.
I do however wonder, at what point do I just describe the hypothesis, point to the data files, and have it design an analysis pipeline, produce the results, interpret the results, then suggest potential follow-up hypotheses, do a literature search on that, then have it write up the grant for it.
Programming became high-level programming (of compilers) became library-glueing/templating/declarative programming... becomes data engineering.
If science was reproducible form articles posted in open access journals, we wouldn’t have half the problems we have with advancing research now.
Slightly tongue in check, but not really.
Programmers (specifically AI researchers) looked at their 300K+ a year salaries and embraced the idea of automating away the work despite how lucrative it would be to continue to spin one's wheels on it. The culture of open source is strong among SWEs, even one's who would lose millions of unrealized gains/earnings as a result of embracing it.
Artists looked at their 30K+ a year salaries from drawing furry hentai on furaffinity and panic at the prospect of losing their work, to the point of making whole political protest movements against AI art. Artists have also never defended open source en mass, and are often some of the first to defend crappy IP laws.
Why be a luddite over something so crappy to defend?
(edit to respond)
I grew up poor as shit and got myself out of that with code. I don't need a lecture about appearing as an elitist.
I'm more than "poking fun" at them - I'm calling them out for lying about their supposed left-wing sensibilities. Artists have postured as being the "vanguard" of the left wing revolution for awhile (i.e. situationalist international and may 68), but the moment that they had a chance to implement their tactics in the art world (open source AI art), they shunned it and cried and embraced ludditism.
Compare this to the world of AI right now. AI has somehow "legally circumvented" copyright laws and we are living in a de-facto post-copyright world. Huggingface and Richard Stallman as an entity/community and individual have done more to democratize access to and give the poors real access to social and economy mobility than any artists have done in the last 10 years, anywhere in the entire world.
You should embrace shit jobs going away, especially in a world where the speed to "re-skill" is often on the orders of hours when AI is involved. I am pointing out that the well-paid AI professional had much to lose and embraced losing it anyway, while the furry artist acted greedily over their pretty awful situation.
Wow who would've thought a large income allowed you to take risks and embrace change?
Imagine being a copywriter for 25 years, on 30k, paying a mortgage, running a car, feeding a family, trying to save on what's left... And all your clients dry up. You've got no other skills, you invested your career in copywriting. You don't have the savings to pivot and your kids need new school uniforms now, not when you reskill to a new career.
You lost your clients. Now your home. Maybe your wife and kids too.
Money is a buffer from risk most don't have.
I hope you never feel this and get to keep the luxury of poking fun at other people for being risk averse without the buffer. Maybe bring some compassion to the table tho? Furry art or copywriting, it isn't anyone's place to judge the merit of the income.
Same, although they are helpful for setting expectations for me. I have some use cases (I'm hesitant to call them evals) related to how we use GPT for our product that are a good "real world" test case. I've found that Claude models are the only ones that are up to par with GPT in the past.
The one for chat: https://www.anthropic.com/claude-ai-locations
The one for API: https://www.anthropic.com/supported-countries
The latter has Norway in it, while the former does not. One wonders why.
Also, Mistral is in Europe. By the time they enter the EU there will only be breadcrumbs left.
> We’ve made meaningful progress in this area: Opus, Sonnet, and Haiku are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models.
Finally someone who takes this into account, Gemini and chatGPT is such an obstacle sometimes with their unnecessary refusal because a keyword triggered something.
https://openrouter.ai/ lets you make one account and get API access to a bunch of different models, including Claude (maybe not v3 yet - they tend to lag by a few days). They also provide access to hosted versions of a bunch of open models.
Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)
To the other point, yes it's crazy that "When inside kitty, how do I get my python inside latex injected into Julia? (It somehow works using alacritty?)" Despite the question being pretty underspecified or confusing, it still shouldn't read as inappropriate.
Unfortunately, many image generation systems will refuse prompts with latex in them (I assumed it was a useful term for styling).
My best guess is that it thinks latex is more often used as a clothing item or something, and it's generally associated with inappropriate content. Just unfortunate for scientists :/.
Less refusals than "previous generations of models" presumably means that is has less refusals than _their_ previous generations of models (= Claude 2), which was notorious for being the worst in class when it came to refusals. I wouldn't be surprised if it's still less permissive than GPT-4.
But then again...GPT4 is a year old and OpenAI has not yet revealed their next-gen model.
Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models - of course some team members had the know-how of being ex. OpenAI) and are, temporarily at least, now ahead of OpenAI in some of these benchmarks.
I'd assume that OpenAI's next-gen model (GPT-5 or whatever they will choose to call it) has already finished training and is now being fine tuned and evaluated for safety, but Anthropic's cause d'etre is safety and I doubt they have skimped on this to rush this model out.
Elon Musk seems to think that, based on his recent lawsuit.
I wouldn't agree but the argument has some validity if you look at the role Microsoft played in reversing the Altman firing.
Doesn’t sound like a startup-investor relationship to me!
But again, this is not to say that OpenAI is "Microsoft in a trenchcoat". Microsoft don't have developers at OpenAI, weren't behind the tech in any way, etc. Their $10B investment bought them some short-term insurance in limited rights to the tech. It is what is is.
https://cryptoslate.com/agi-is-excluded-from-ip-licenses-wit...
It's also explicitly mentioned in Musk's lawsuit against OpenAI. Much as Musk wants to claim that OpenAI is a subsidiary of Microsoft, even he has to admit that if in fact OpenAI develop AGI then Microsoft won't have any IP rights to it!
The context for Nadella's "We have everything" (without of course elaborating on what "everything" referred to) is him trying to calm investors who were just reading headlines about OpenAI imploding in reaction to the board having fired Altman, etc. Nadella wasn't lying - he was just being coy about what "everything" meant, wanting to reassure investors that their $10B investment in OpenAI had not just gone up in smoke.
This is not an uncommon tactic for companies to use.
Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash.
In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.
I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.
Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea.
I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.
As far as different groups leapfrogging each other for supremacy in various benchmarks, there might be a bit of a "4 minute mile" effect here too - once you know that something is possible then you can focus on replicating/exceeding it without having to worry are you hitting up against some hard limit.
I think the transformer still doesn't get the credit due for enabling this LLM-as-AI revolution. We've had the compute and data for a while, but this breakthough - shared via a public paper - was what has enabled it and made it essentially a level playing field for anyone with the few $B etc the approach requires.
I've never seen any claim by any of the transformer paper ("attention is all you need") authors that they understood/anticipated the true power of this model they created (esp. when applied at scale), which as the title suggests was basically regarded an incremental advance over other seq2seq approaches of the time. It seems like one of history's great accidental discoveries. I believe there is something very specific about the key-value matching "attention" mechanism of the transformer (perhaps roughly equivalent to some similar process used in our cortex?) that gives it it's power.
It's really not the model, it's the data and scaling. Otherwise the success of different architectures like Mamba would be hard to justify. Conversely, humans getting training on the same topics achieve very similar results, even though brains are very different at low level, not even the same number of neurons, not to mention different wiring.
The merit for our current wave is 99% on the training data, its quality and size are the true AI heroes. And it took humanity our whole existence to build up to this training set, it cost "a lot" to explore and discover the concepts we put inside it. A single human, group or even a whole generation of humans would not be able to rediscover it from scratch in a lifetime. Our cultural data is smarter than us individually, it is as smart as humanity as a whole.
One consequence of this insight is that we are probably on an AI plateau. We have used up most organic text. The next step is AI generating its own experiences in the world, but it's going to be a slow grind in many fields where environment feedback is not easy to obtain.
My take is that prediction, however you do it, is the essence of intelligence. In fact, I'd define intelligence as the degree of ability to correctly predict future outcomes based on prior experience.
The ultimate intelligent architecture, for now, is our own cortex, which can be architecturally analyzed as a prediction machine - utilizing masses of perceptual feedback to correct/update predictions of how the perceptual scene, and results of our own actions, will evolve.
With prediction as the basis of intelligence, any model capable of predicting - to varying degrees of success - will be perceived to have a commensurate degree of intelligence. Transformer-based LLMs of course aren't the only possible way to predict, but they do seem significantly better at it than competing approaches such as Mamba or the RNN (LSTM etc) seq2seq approaches that were the direct precursor to the transformer.
I think the reason the transformer architecture is so much better than the alternatives, even if there are alternatives, is down to this specific way it does it - able to create these attention "keys" to query the context, and the ways that multiple attention heads learn to coordinate such as "induction heads" copying data from the context to achieve in-context learning.
The training set is magical. It took humanity a long time to discover all the nifty ideas we have in it. It's the result of many generations of humans working together, using language to share their experience. Intelligence is a social process, even though we like to think about keys and queries, or synapses and neurotransmitters, in fact it is the work of many people that made it possible.
And language is that central medium between all of us, an evolutionary system of ideas, evolving at a much faster rate than biology. Now AI have become language replicators like humans, a new era in the history of language has begun. The same language trains humans and LLMs to achieve similar sets of abilities.
Are there any Mamba benchmarks that show it matching transformer (GPT, say) benchmark performance for similiar size models and training sets?
https://trendingpapers.com/search?q=mamba
Keep in mind that Antropic was founded by former OpenAI people (Dario Amadei and others). Both companies share a lot of R&D "DNA".
It's genuinely outperforming GPT4 in my manual tests.
"In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"
GPT-4-1106-preview GPT-4-0125-preview
See: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
i) There is no calculator and it's hallucinating the whole thing
ii) There is a calculator but it's terrible. This seems hard to believe
iii) It does a bad job of copying the numbers into and out of the calculator
Pricing (input/output per million tokens):
GPT4-turbo: $10/$30
Claude 3 Opus: $15/$75
That suggests the inference time is more expensive then the memory needed to load it in the first place I guess?
Probably that and what you mentioned.
Their pricing suggests that either output tokens are more expensive for some technical reason, or they're trying to encourage a specific type of usage pattern, etc.
Nitpick: It's 50% and 150% more respectively.
My point is that there’s plenty of room for high priced but only slightly better models.
Why can we still not easily surpass a (relatively) ancient model?
Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity.
I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient.
Right now, only ChatGPT (the webapp) seems to be using such those snapshots.
Am I missing something?
If you don't care about latency or can wait to set up a batch of inputs in one go there's an alternative method. I call it batch prompting and pretty much everything we do at work with gpt-4 uses this now. If people are interested I'll do a proper writeup on how to implement it but the general idea is very straightforward and works reliably. I also think this is a much better evaluation of context than needle in a haystack.
Example for classifying game genres from descriptions.
Default:
[Prompt][Functions][Examples][game description]
- >
{"genre": [genre], "sub-genre": [sub-genre]}
Batch Prompting:
[Prompt][Functions][Examples]<game1>[description]</game><game2>[description]</game><game3>[description]</game>...
- >
{"game1": {...}, "game2": {...}, "game3": {...}, ...}
I don't really intend for this method to be final. I'll switch everything over to finetunes at some point. But this works way better than I would have expected so I kept using it.
Another thing I used it for was data extraction, where I extracted units of measurements and other key attributes out of descriptions from classifieds listings (my SO and me were looking for a cheap used couch). Non-batched it performed very well, while in the batched mode, it either mixed dimensions of multiple listings or after the summary for the initial listing it just gave nulls for all following listings.
It sounds like you would like a wrapped version tuned just for big context.
(As others write, RAG versions are also being supported, but they're less fundamentally similar. RAG is about preprocessing to cut the input down to relevant bits. RAG + an agent framework does get closer again tho by putting this into a reasoning loop.)
From my early tests this seems like the first API alternative to GPT4. Huge!
Would be great to have an FAQ for this type of common question
Although, I feel Apple will break this trend and bring models to their chips rather than run them on the cloud. "Privacy first" will simply be a selling point for them but generally speaking cloud is not a big sell for them.
I am not at the level to do much optimizations, plus my product is a little more generic. To get to MVP, prompt engineering will probably be my sole focus.
Thank you again for the great tool.
But could you let the users choose their keyboard shortcuts before setting the default ones?
Are you asking because it conflicts with an existing shortcut on your setup? Or another reason?
I noticed it might actually be a little more censored than the lmsys version. Lmsys seems more fine with roleplaying, while the one on Double doesn't really like it
[0] - https://www.codium.ai
[1] - https://github.com/Exafunction/codeium.vim
* we always close any brackets opened by autocomplete (and never extra brackets, which is the most annoying thing about github copilot)
* we automatically add import statements for libraries that autocomplete used
* mid-line completions
* we turn off autocomplete when you're writing a comment to avoid disrupting your train of thought
You can read more about these small details here: https://docs.double.bot/copilot
As you noted we don't have a vim integration yet, but it is on our roadmap!
p50: 2.14s p95: 3.02s
And these aren't super long prompts either. vs gpt4 ttft:
p50: 0.63s p95: 1.47s
(Chromium 87.0.4280.144 (Jan. 2021), plus security patches up to 119.0.6045.160 (Nov. 2023).)
What about Ultra?
SMIC used Applied Materials and Lam equipment to make 7nm chip
US wants to further limit China’s access to foreign chip tech
Bloomberg has learned that Huawei and its partner SMIC relied on gear from Applied Materials Lam Researchto produce an advanced chip.
Huawei Technologies Co. and its partner Semiconductor Manufacturing International Corp. relied on US technology to produce an advanced chip in China last year, according to people with knowledge of the matter.
Shanghai-based SMIC used gear from California-based Applied Materials Inc. and Lam Research Corp. to manufacture an advanced 7-nanometer chip for Huawei in 2023, the people said, asking not to be named as the details are not public.
The previously unreported information suggests that China still cannot entirely replace certain foreign components and equipment required for cutting-edge products like semiconductors. The country has made technological self-sufficiency a national priority and Huawei’s efforts to advance domestic chip design and manufacturing have received the backing of Beijing.
Representatives of SMIC, Huawei and Lam did not respond to requests for comment. Applied Materials and the US Commerce Department’s Bureau of Industry and Security, which is responsible for implementing export controls, declined to comment.
Lauded in China as a major leap in indigenous semiconductor fabrication, last year’s SMIC-made processor powered Huawei’s Mate 60 Pro and a wave of patriotic smartphone-buying in the Asian country. The chip is still generations behind the top components from global firms, but ahead of where the US hoped to stop China’s advance.
The machinery used to make it, however, still had foreign sources including technology from Dutch maker ASML Holding NV as well as the gear from Lam and Applied Materials. Bloomberg News reported in October that SMIC had used equipment from ASML for the chip breakthrough.
Leading Chinese chip equipment suppliers including Advanced Micro-Fabrication Equipment Inc. and Naura Technology Group Co. have been trying to catch up with their American peers, but their offerings are still not as comprehensive or sophisticated. China’s top lithography system developer Shanghai Micro Electronics Equipment Group Co. still lags a few generations behind what industry leader ASML is capable of.
SMIC obtained the American machinery before the US banned such sales to China in October 2022, some of the people said. Both firms were among the American suppliers that began pulling their staff from China after those rules went into effect and prohibited US engineers from servicing some machines in the Asian country. ASML also told American employees to stop working with Chinese customers in response to the US curbs, but Dutch and Japanese engineers are still able to service many machines in China — much to the chagrin of their American rivals.
Companies are now prohibited from selling cutting-edge, US-origin technology to either SMIC or Shenzhen-based Huawei. Both tech firms have been blacklisted by the US for alleged links to the Chinese military, while Washington has been tightening China’s overall access to chipmaking equipment and advanced semiconductors.
Those trade curbs pushed Huawei and SMIC to pursue avenues for building a domestic chip supply chain, and the Mate 60 Pro marked a surprising advance in that effort.
After Huawei released the new phone, Washington launched a probe into its processor and US Commerce Secretary Gina Raimondo vowed the “strongest possible” actions to ensure national security. Meanwhile, Republican lawmakers have called for the Biden administration to completely cut off Huawei and SMIC’s access to US technology.
Department of Commerce officials have said they haven’t seen evidence that SMIC can make the 7nm chips “at scale,” a point echoed by ASML’s Chief Executive O...
SMIC used Applied Materials and Lam equipment to make 7nm chip
US wants to further limit China’s access to foreign chip tech
Bloomberg has learned that Huawei and its partner SMIC relied on gear from Applied Materials Lam Researchto produce an advanced chip.
Huawei Technologies Co. and its partner Semiconductor Manufacturing International Corp. relied on US technology to produce an advanced chip in China last year, according to people with knowledge of the matter.
Shanghai-based SMIC used gear from California-based Applied Materials Inc. and Lam Research Corp. to manufacture an advanced 7-nanometer chip for Huawei in 2023, the people said, asking not to be named as the details are not public.
The previously unreported information suggests that China still cannot entirely replace certain foreign components and equipment required for cutting-edge products like semiconductors. The country has made technological self-sufficiency a national priority and Huawei’s efforts to advance domestic chip design and manufacturing have received the backing of Beijing.
Representatives of SMIC, Huawei and Lam did not respond to requests for comment. Applied Materials and the US Commerce Department’s Bureau of Industry and Security, which is responsible for implementing export controls, declined to comment.
Lauded in China as a major leap in indigenous semiconductor fabrication, last year’s SMIC-made processor powered Huawei’s Mate 60 Pro and a wave of patriotic smartphone-buying in the Asian country. The chip is still generations behind the top components from global firms, but ahead of where the US hoped to stop China’s advance.
The machinery used to make it, however, still had foreign sources including technology from Dutch maker ASML Holding NV as well as the gear from Lam and Applied Materials. Bloomberg News reported in October that SMIC had used equipment from ASML for the chip breakthrough.
Leading Chinese chip equipment suppliers including Advanced Micro-Fabrication Equipment Inc. and Naura Technology Group Co. have been trying to catch up with their American peers, but their offerings are still not as comprehensive or sophisticated. China’s top lithography system developer Shanghai Micro Electronics Equipment Group Co. still lags a few generations behind what industry leader ASML is capable of.
SMIC obtained the American machinery before the US banned such sales to China in October 2022, some of the people said. Both firms were among the American suppliers that began pulling their staff from China after those rules went into effect and prohibited US engineers from servicing some machines in the Asian country. ASML also told American employees to stop working with Chinese customers in response to the US curbs, but Dutch and Japanese engineers are still able to service many machines in China — much to the chagrin of their American rivals.
Companies are now prohibited from selling cutting-edge, US-origin technology to either SMIC or Shenzhen-based Huawei. Both tech firms have been blacklisted by the US for alleged links to the Chinese military, while Washington has been tightening China’s overall access to chipmaking equipment and advanced semiconductors.
Those trade curbs pushed Huawei and SMIC to pursue avenues for building a domestic chip supply chain, and the Mate 60 Pro marked a surprising advance in that effort.
After Huawei released the new phone, Washington launched a probe into its processor and US Commerce Secretary Gina Raimondo vowed the “strongest possible” actions to ensure national security. Meanwhile, Republican lawmakers have called for the Biden administration to completely cut off Huawei and SMIC’s access to US technology.
Department of Commerce officials have said they haven’t seen evidence that SMIC can make the 7nm chips “at scale,” a point echoed by ASML’s Chief Executive Officer Peter Wennink.
If SMIC wants to advance its technology without ASML’s state-of-the-art extreme ultraviol...
I don't understand the need to do that when your numbers look promising.