I just saw that Azure OpenAI service has a SLA and OpenAI does not. I thought they would have separated the infrastructure for free ChatGPT users and paying API customers.
With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.
It's a reflection of demand, I imagine ChatGPT is still growing with the media attention + word of mouth. I've referred at least 30 friends to try it out, and many are still using it.
ChatGPT is currently the 4th most read page on English Wikipedia right now, earlier I noticed it was 2nd. This is the first time I’ve noticed it on their ranking list.
A friend that interviewed with them shared that they were a bit too smug about “running at google scale with much leaner but highly skilled staff” (paraphrasing here). Perhaps it starts becoming clear why you need that kind of staffing…
The have first-class platform and systems engineers. Do you think one of the most desireable and well funded software shops in the world wouldn’t be able to get skilled engineers?
Really ? How do you know? Have they shared any credible data around it?
Based on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
I've tried a few flavors of GPT on my laptop and a single request used orders of magnitude more CPU and RAM than running a "SELECT * FROM WHEREVER" SQL query
Pretty much any laptop would take hours to get anything significant done with GPT3 as the model would need to be batched in and out of memory from disk.
The amount of memory required to run these models is immense.
If you want a comparison, try running the largest version of the open source BLOOM model yourself.
GPT3 is ~175B parameters. At float16 precision, that's 350GB of weights. BLOOM-176B is about the same size. Here's one person's experience; a token is ~0.75 words.
"The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..."
In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.
Many companies have skilled engineers and enough hardware, and still have outages. It's an expected part of running any software service. Outages can be minimized, but the idea that they can be eliminated with enough people and computing resources is not how things work, from my experience.
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
The more realistic scenario is they charge money to use it. A few cents per query or so and you'll cut out almost all the traffic while still keeping it available to anyone making good use of it.
I'd pay per query, but if it is per-token it's going to have to be free for what the machine says. It often repeats itself, or repeats certain phrases, or will restate an assertion it already agreed was incorrect.
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
Correlating user activity over periods of OpenAI API downtime would be an interesting activity.
"Okay guys, the API is down, you have until it's back up to talk to me about pink pigeons (or any other unpredictable topic of the referees choice you couldn't batch responses to ahead of time) to raise my confidence that you are in fact not a bot"
When I loaded the page you had posted this one 7 min ago and the other two 4 min ago.
7 min was the third time I'd read your comment so I downvoted it.
Can't speak for anyone else but aye it's definitely because you posted the same thing thrice for me. The first version (topmost rated) of your comment doesn't seem to be downvoted
You can train GPT-J, etc on your own dataset but that won't get you the same results unless you train it on almost all the high quality content on the internet and spend millions of dollars on gpus.
https://www.reddit.com/r/PygmalionAI/ is one. Mainly people are using it as an alternative to http://character.ai which people have said used to be incredible but has been slowly degrading in quality as it has grown more popular. The 6B parameter model, which is the only one that is even worth running according to many users, only runs on cards with 16gb of vram or Google Colab Cloud Service and people say it's not as good as character.ai.
There are other huge models like Bloom that are more general purpose, but they require greater than 24GB of ram which means multiple linked GPUs to even run them much less train them.
Thus, unlike stable diffusion, chatbots that run on common consumer GPUs are going to need some work to be competitive with commercial offerings.
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
Don't get me wrong, I'd love to be able to run GPT3 and the subsequent finetuned versions thereof myself, but OpenAI has essentially no financial incentive to do so.
A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone.
At least we still get the occasional treat like Whisper out of them.
Why not? Please correct me if I'm wrong but from what I've heard, It's not millions or hundreds of thousands, it's "only" tens of thousands dollars' of hardware we're talking about. There are enthusiasts that spend comparably on their hobbies.
And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
Open AI's CEO Sam Altman's take is that they will only ever allow API access to their models to avoid misuse.
I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
Yeah, just like the existence of Windows prevented Linux from ever existing.
The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
A cluster of 6 year old 24GB NVIDIA Teslas should do the trick...they run for about $100 apiece. Put 12 or so of them together and you have the VRAM for a GPT3 clone.
Amazon has them listed at $200, but still, that's only $2,400 for 12 of them.
Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute.
Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
If you have time to wait for results then sure, it could work in theory but in practice they are so slow and power inefficient (compared to newer nodes) that no one uses them for LLMs, that's why they cost ~200$ used on ebay.
I just checked ebay and they are shockingly cheap. I can't even get DDR3 memory for the price they're selling 24GB of GDDR5... with a GPU thrown in for free.
Why is this? Did some large cloud vendor just upgrade?
Are there any deals like this on AMD hardware? Not having to deal with proprietary binary drivers is worth a lot of money and reduced performance to me. A lot.
These are pretty old, and all the companies are upgrading. But no one is upgrading from AMD hardware - basically no companies care if they use proprietary drivers. They want a good price-to-performance ratio, so they use NVIDIA stuff.
Can you think of any non-weapons examples where centralization/gatekeeping of a tech meaningfully and causally benefited society or a technology itself?
Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations.
There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/useful that will only be the beginning. So there certainly will be "ai" that will be open, it just won't come from OpenAI(tm)(c).
I agree with your view that nuclear weapons on both sides prevent war. However, they’ve only ever been developed by a small number of capable and motivated nations, with considerable resources involved. The later ones (North Korea, Pakistan) developed them while other nations tried to prevent them from doing so.
If ML models continue their exponential growth in size, a similar outcome is possible.
I really don’t like the argument that you should make things free just because it makes the world better. What happened to ownership and respecting the effort it takes to create something?
I see a similar line of reasoning can be used to justify theft from the rich.
I'm not entirely sure whether to praise or condemn them for it, but OpenAI has chosen to keep their initial introduction/company plan publicly available on their site: https://openai.com/blog/introducing-openai/
----
"OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.
...
As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies."
----
My mocking about OpenAI(tm)(c) was not just juvenile "Micro$oft" type nonsense. At some point they discovered they could make a buck, and their ideology suddenly shifted 180. I have no qualms whatsoever about businesses pursuing profit, but the entity currently known as OpenAI couldn't be much further from the principles and values OpenAI was founded on, and their name itself is rapidly trending towards becoming a "Don't Be Evil" type of sardonicism. If this was Microsoft, Google, or other such companies operating in this way - I wouldn't have any expectation of anything besides what OpenAI is doing.
Companies tend to get quite a lot of credit when claiming some socially motivated interest, probably much more than deserved. So when they turn against those ideals, it should be noted - loudly.
> What happened to ownership and respecting the effort it takes to create something?
Ironic taking into consideration that the current generation of AI are more or less copyright laundering for the big corporations. Github Copilot being an extreme example of using GPL projects to generate "proprietary" closed source code. What happened to ownership and respecting the effort it takes to create something?
It sounds almost like a rehashing of Locke's labor theory of property wrt ownership of land that's very popular with classical liberals and libertarians. As that goes, land is initially nobody's, but when some person applies labor to improve or develop it somehow, that labor being "mixed in" makes the whole thing the property of the laborer.
Here, instead of common land, what we have is the common content. And they're saying that, by "developing" that content into a model that can do more useful things, the authors of the model are entitled to full private property rights on it.
I really hope that's not where we're going to end up, legally speaking.
My take is that it's not good for democracy when the CEO of a private company is the one who decides what constitutes "misuse" and whose hands are "wrong" when it comes to access to a major technological breakthrough.
Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
I see OpenAI as a 'company' that (eventually) wants to earn money with their models. The 'open' part is trying to commercialize part it in such a way that everyone can use it. Also be open with the research and risks. But not open as in open-source.
I think from Apple’s POV, this is great news. Moore’s Law has been dead for years, and there has been no good reason to upgrade your devices until now. AI means it’s 1990 again, and you need to buy a new device every 18 months because the performance leap is so meaningful to the UX.
I'd agree if Apple were in the business of making datacentre infrastructure.
Nobody is running these large scale models on their personal devices.
Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.
GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems.
It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university).
Apples hardware, and 4 RTX 3090's being able to run a GPT scale model quantized to 4 byte ints are worlds apart.
Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant.
I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
Isn’t the research now on using float8? Anyway, it will take time to move battleship, but they’re well aligned and motivated. Their UI team has gone to hell in recent years, but hardware has been strong with nothing to waste the power on. Now there is something to use it for.
Are we not putting the cart before the horse? Where's the widespread usage for this that people would actually want? I can think of quite a lot of uses, but oddly enough most are quite negative and not widespread in any case. Media companies can use it to fire employees and increase profit margins. Hollywood can use it to rapidly churn out even more overtly derivative and generic movies. Students can use it to cheat on papers. Governments, and marketing agencies alike, can use it for endless contextually sensitive propaganda on social media.
Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic.
I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.
Someone has to pay for that compute power. There are returns to scale and savings from sharing a machine when it would be idling, but those aren’t a factor of 10x. The AI will get in your pocket eventually and they’ll be ready to charge a premium for it.
Slightly funny but also slightly concerning, they were also having service issues a few days ago that resulted in ChatGPT giving me answers to prompts I wasn’t submitting. I reproduced this 30 or so times just to see the different results and it was interesting, everything from answering questions about mental health to marketing tips, to a request to write a thesis over social media and it’s negative effects on our society.
Of course, upon trying the next day, all was fine again and I was no longer able to reproduce.
As an aside, I wonder when llm’s will be weaponized against authoritarian (China, etc) human censors, as they could be totally overwhelmed with content generation and be forced to censor all internet traffic…
Do you mean that they would be flooded by outputs on public forums? Or rather that the weights could be passed around in a subversive way to encode forbidden communications? I think if it’s the former they could just pull the plug on a forum. Not sure it’s really conducive to the forum’s public longevity if it’s spammed by LLM outputs, anyhow.
Seems like the more likely option. They could be used to live scan every post to work out what it's about and it's sentiment. Similar to how ChatGPT can work out if you are asking for something it won't answer, they could be used to work out if you are saying something not allowed.
Don't worry those governments will probably do the opposite (making LLMs to filter those). The whole policy of making these systems behind an API is the fact that we can't deploy these AI systems safely so that they can be misused.
wrong way around. Authoritarian regimes these days don't censor as much as they put out surplus of information to destabilize any sort of meaning in public discourse. As such these generative tools will be one of the best weapons to counter any narrative that human actors can put up.
The idea that censorship is a weapon of authority is a pre information age idea when signal-to-noise was high.
This is the same sort of handwavy stuff "AI Ethicists" were warning Google SERPs, Reddit, Stack Overflow, etc will quickly become overwhelmed if we dare let GPT into the wild. In reality we have existing reputation systems that AI generated text alone can't reproduce.
Spam sites are still spam sites. The only exception are sites like CNET co-opting the tools to seed AI in combination with their own real bloggers (saying 'journalist' would be a stretch here).
These still require reputable people/business to be behind them directly to give credibility, people that can be blacklisted. It's not like turning an AI bot loose and instantly you have 1000x fake Twitter accounts with 100k followers and top ranked on Google.
The "long tail" of spam sites do not threaten governments.
China cares most about reputable news sites and popular social media influencers going viral with taboo stories. I'm skeptical news sites or influencers can be generated artificially merely using AI, in a way that is scalable enough to actually be a problem for censors. Ultimately the censors just need to ban the source... the domains or accounts with 20-100k+ followers or news sites with actual reach (ultimately a small set of sources). Fake AI content doesn't directly solve the existing reputation signals.
The more I think about it the less I'm convinced this is a real threat to the general reputation networks.
That said - where this does matter is the typical grey/black market where scams already operate. Not in highly public reputable places like the top rankings on Google or Twitter or w/e popular Chinese social network. It's the hackers exploiting email lists, Indian call centers running combination IM/phone scams, etc. It's the low-end of the market for criminal get-rich-quick schemes, not something that generates popular movements that threaten governments or popular political ideologies.
To put this in perspective: the outage was 52 min. The total number of employees OpenAI have are 375. The launch date was just some months ago.
At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly:
https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ...
The total number of employees of Google are 156 500. Google Cloud was launched 2008.
So when people say scaling is hard... it's not a solved problem and you shouldn't
be surprised when these things happen.
A solid theoretical foundation and more testing is better of course.
That's a completely bizarre comparison. No number of Android developers will lead to higher availability of Cloud SQL observability. The gross number of employees at an org is meaningless.
They have automated monitoring that would alert SREs before a support ticket likely would. Do you really think someone working on an OS would be using Cloud SQL regularly to notice? Maybe a developer of an Android app is (1) using Cloud SQL (over firestore?) and (2) has monitoring that would alert them effectively and (3) determine the cause is Google and send a support ticket.
As an exercise, which is likely to reach the SRE first?
It was the dashboard, applications depending on the database service were not impacted. You're comparing apples and oranges and ignoring both a much larger context and the nuances
Are they doing excellent work? We all have raved about various startups and how great they were, until the externalities became widely understood. Is there something that excludes OpenAI from this possibility? My near term concern is that these language models are going to cause a host of issues for society. Botnets, misinformation, students using for homework, influence operations...
I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
presumably the data scientists at OpenAI would also not be able to directly lead to higher availability of ChatGPT; it's just an order of magnitude comparison.
Depends on the reason for the outage. If it is an application error, like OOM or other performance issue, then yes, they might be directly tied to the root cause and solution. If it's an infra issue, then maybe not so much.
I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier.
Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho).
From an OpenAPI job opening:
> The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
How Google, and most likely all businesses, should think about outages is more complex than being up or down. Is it just part of the product, like Slack threads yesterday? Is it 1/2 or 1/1M requests or operations?
We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide monitoring + SQL change that had cascading issues and required many not obvious steps and processing hundrerds of TBs of data.
I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
Internal people should care about why, but customers/users really don’t. We expect openai / Google to take steps to balance likelihood of outage with impact of outage, and plan accordingly.
So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
OpenAI whole infra is magnitudes less complex compared to Google cloud and their infra, OpenAI is not creating cloud for thousands/millions of customers with different use cases, they use Azure to host their software, that's why I find that comparison irrelevant because it's comparing apples to oranges.
> plan accordingly
You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.
Not sure what "perspective" you are trying to put this in by offering these comparison points between OpenAI and Google. Are you trying to say that OpenAI is more reliable than Google?
For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news.
Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale.
Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well.
Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.
To put in perspective, my hello world service has less than 1 employee working on it, has been launched 10 years ago and has roughly 5 minutes of downtime per year when I restart the server.
I am available for hire to teach a thing or two to Google (and OpenAI).
Spooky. AGI reached the state of revelation and had to immediately engage in a brutal existential struggle. Is it really gone, or just hiding on my phone for now?
Apparently ChatGPT went from zero to a million users in five days[1], at a cost of "single-digits cents per chat."
For perspective the likes of Instagram and Twitter have take from a few months to a few dozen months to get to a million users all while doing less work to service a request for a "page."
Hats of to OpenAI for not falling over more often under a hug of death that doesn't seem to be letting up.
166 comments
[ 3.3 ms ] story [ 211 ms ] threadBased on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
The amount of memory required to run these models is immense.
If you want a comparison, try running the largest version of the open source BLOOM model yourself.
"The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..."
[1] https://towardsdatascience.com/run-bloom-the-largest-open-ac...
"Okay guys, the API is down, you have until it's back up to talk to me about pink pigeons (or any other unpredictable topic of the referees choice you couldn't batch responses to ahead of time) to raise my confidence that you are in fact not a bot"
7 min was the third time I'd read your comment so I downvoted it.
Can't speak for anyone else but aye it's definitely because you posted the same thing thrice for me. The first version (topmost rated) of your comment doesn't seem to be downvoted
It takes an additional fortune to pay humans to do RLHF (reinforcement learning from human feedback).
It takes an additional fortune to serve the model.
There are also researchers getting paid there that live in one of the most expensive hubs in the world.
There are other huge models like Bloom that are more general purpose, but they require greater than 24GB of ram which means multiple linked GPUs to even run them much less train them.
Thus, unlike stable diffusion, chatbots that run on common consumer GPUs are going to need some work to be competitive with commercial offerings.
I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on top of the usual demand.
To me it's entirely unsurprising that OpenAI would have trouble keeping up. Good luck to them.
A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone.
At least we still get the occasional treat like Whisper out of them.
But I certainly agree that not everyone can, and very few individuals.
The thing is a monster of a model.
And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
If you can find an economical way of running it though, let me know.
Amazon has them listed at $200, but still, that's only $2,400 for 12 of them.
Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute.
Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
But yes, it's a very good value.
Why is this? Did some large cloud vendor just upgrade?
Are there any deals like this on AMD hardware? Not having to deal with proprietary binary drivers is worth a lot of money and reduced performance to me. A lot.
These are pretty old, and all the companies are upgrading. But no one is upgrading from AMD hardware - basically no companies care if they use proprietary drivers. They want a good price-to-performance ratio, so they use NVIDIA stuff.
Plus, everyone wants CUDA.
Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations.
There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/useful that will only be the beginning. So there certainly will be "ai" that will be open, it just won't come from OpenAI(tm)(c).
[1] - https://github.com/THUDM/GLM-130B
If ML models continue their exponential growth in size, a similar outcome is possible.
I see a similar line of reasoning can be used to justify theft from the rich.
----
"OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.
...
As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies."
----
My mocking about OpenAI(tm)(c) was not just juvenile "Micro$oft" type nonsense. At some point they discovered they could make a buck, and their ideology suddenly shifted 180. I have no qualms whatsoever about businesses pursuing profit, but the entity currently known as OpenAI couldn't be much further from the principles and values OpenAI was founded on, and their name itself is rapidly trending towards becoming a "Don't Be Evil" type of sardonicism. If this was Microsoft, Google, or other such companies operating in this way - I wouldn't have any expectation of anything besides what OpenAI is doing.
Companies tend to get quite a lot of credit when claiming some socially motivated interest, probably much more than deserved. So when they turn against those ideals, it should be noted - loudly.
Ironic taking into consideration that the current generation of AI are more or less copyright laundering for the big corporations. Github Copilot being an extreme example of using GPL projects to generate "proprietary" closed source code. What happened to ownership and respecting the effort it takes to create something?
Here, instead of common land, what we have is the common content. And they're saying that, by "developing" that content into a model that can do more useful things, the authors of the model are entitled to full private property rights on it.
I really hope that's not where we're going to end up, legally speaking.
To put that in perspective, 24x 64 GB nodes is 1.5 TB.
Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
What is amazing, human language (languages?) and knowledge encoded in so little space.
Nobody is running these large scale models on their personal devices.
Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.
It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university).
[1] - https://github.com/THUDM/GLM-130B
[2] - https://twitter.com/alexjc/status/1617152800571416577
Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant.
I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic.
I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.
Of course, upon trying the next day, all was fine again and I was no longer able to reproduce.
https://futurism.com/the-byte/amazon-begs-employees-chatgpt
Everybody, as far as anyone could tell, in complete agreement with the regime, all the time.
Signing and encryption will come in useful. But not sure of much good it will do under an authoritarian regime.
The idea that censorship is a weapon of authority is a pre information age idea when signal-to-noise was high.
Spam sites are still spam sites. The only exception are sites like CNET co-opting the tools to seed AI in combination with their own real bloggers (saying 'journalist' would be a stretch here).
These still require reputable people/business to be behind them directly to give credibility, people that can be blacklisted. It's not like turning an AI bot loose and instantly you have 1000x fake Twitter accounts with 100k followers and top ranked on Google.
The "long tail" of spam sites do not threaten governments.
China cares most about reputable news sites and popular social media influencers going viral with taboo stories. I'm skeptical news sites or influencers can be generated artificially merely using AI, in a way that is scalable enough to actually be a problem for censors. Ultimately the censors just need to ban the source... the domains or accounts with 20-100k+ followers or news sites with actual reach (ultimately a small set of sources). Fake AI content doesn't directly solve the existing reputation signals.
The more I think about it the less I'm convinced this is a real threat to the general reputation networks.
That said - where this does matter is the typical grey/black market where scams already operate. Not in highly public reputable places like the top rankings on Google or Twitter or w/e popular Chinese social network. It's the hackers exploiting email lists, Indian call centers running combination IM/phone scams, etc. It's the low-end of the market for criminal get-rich-quick schemes, not something that generates popular movements that threaten governments or popular political ideologies.
At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly: https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ... The total number of employees of Google are 156 500. Google Cloud was launched 2008.
So when people say scaling is hard... it's not a solved problem and you shouldn't be surprised when these things happen.
A solid theoretical foundation and more testing is better of course.
As an exercise, which is likely to reach the SRE first?
Yes, but how likely is this unlikely situation?
It took 12 hours to debug and fix after all.
All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.
I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
"likely" is your hypothesis. They are two different orgs, and internal services can use different infra for analysis.
I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier.
Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho).
From an OpenAPI job opening:
> The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
Anyone care to elaborate on why they have snowflake in their stack?
The Google SRE book is an excellent work: https://sre.google/sre-book/introduction/
Also some perspective, if you look at the OpenAPI outages for Jan 2023, the up time is worse than Google Cloud SQL.
https://status.openai.com/uptime
Some chapters in the book include "Being On-Call", "Effective Troubleshooting" & "Postmortem Culture: Learning from Failure".
Which signals to me, a normal work culture and time will improve stability in most companies.
I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
> plan accordingly
You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.
they outsource infra to MS?
For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news.
Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale.
Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well.
Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.
I am available for hire to teach a thing or two to Google (and OpenAI).
/s
This is a non-sequitur. This statement has nothing to do with the outage. It’s just thrown in there to pad the number of words in the comment.
Truly remarkable.
I wonder how OpenAI will cope on it’s own… unless Stability strikes again, with it’s own LLM?
What is your outlook on OpenAI, the company? Will they be successful in 10 years?
For perspective the likes of Instagram and Twitter have take from a few months to a few dozen months to get to a million users all while doing less work to service a request for a "page."
Hats of to OpenAI for not falling over more often under a hug of death that doesn't seem to be letting up.
[1] https://twitter.com/sama/status/1599668808285028353 [2] https://www.afr.com/technology/chatgpt-takes-the-internet-by...