Let's say 500 employees, earning above Bay Area average, along with benefits etc at $400K average per year. That's $200 million a year in wages, so in the same ballpark as the $200 million in infrastructure costs cited in the article, for a total of $400 million in expenses per year.
Isn't Microsoft too late to the chip game? Both their competitors AWS and GCP have their own ML training chips, while Azure has nothing. The article seems like they may have something by next year, but by that time the competition would have evolved even more. Nadella has executed on strategy brilliantly but on the cloud seems it seems like the tech innovation is a bit lagging. It hasn't mattered that much so far though due to how entrenched the MS stack is in the enterprise.
It feels like Microsoft realized that sales/support matters way more than technology when it comes to cloud services. See how Azure has a ton of users despite being much worse than AWS or GCP by every objective metric.
I would imagine that Google's reputation as a company where it is impossible to ever talk to a human (even when you have a 7 figure annual spend) hurts them in this space.
Yes, but that strategy works due to how entrenched they were in the enterprise already. It won't work for a brand new vendor. Almost every big company uses some enterprise Windows product (Windows,Office,AD, SQLServer).
Once people already are buying 20 products from you and have a good sales relationship of decades, selling them some cloud services is easy and might even lead to better deals on something else (Windows/Office)
Funny it seems both GCP and Azure meet the expectations you have of their parent in this space. With Google you always got the "latest and greatest" but support was lacking. With MS you got the support, but maybe on not the latest and greatest. That seems to have carried into their cloud philosophies.
The problem for both of them is AWS, which somehow manages to give you your cake and let you eat it too, even if it's a little more expensive.
But what is being optimized? Hardware sure isn't getting faster in a hurry, and I don't see anything on the horizon that will aid in optimizing software.
The various open source LLMs are doing things like reducing bits-per-parameter to reduce hardware requirements; if they're using COTS hardware it almost certainly isn't optimised for their specific models; Moore's Law is pretty heavily reinterpreted, so although we normally care about "operations per second at a fixed number of monies" what matters here is "joules per operation" which can improve a by a huge margin even before human level, which itself appears to be a long way from the limits of the laws of physics; and even if we were near the end of Moore's Law and there was only a 10% total improvement available, that's 10% of a big number.
Moore's law was an effect that stemmed from the locally exponential efficiency increase from designing computers using computers, each iteration growing more powerful and capable of designing still more powerful hardware.
10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law.
We're actually at an inflection point where this isn't the case anymore.
For a long time, GPU hardware basically became more powerful with each generation, but prices stayed roughly the same plus minus inflation. Last couple of years, this trend has broken. You pay double or even quadruple the price for a relatively tenuous increase in performance.
We said that in 1982, and 1987, and 1993, and 1995, and 2001, 2003, 2003.5
You get the point.
There's always local optimization that leads to improvements. Look at the Apple M1 chip rollout as a prime example of that. Big/Little processors, on die RAM, shared memory with the GPU and Neural Engine, power integration with the OS.
Big difference now is that we have a clear inflection point. Die processes aren't getting much smaller than they are. A sub-nanometer process would involve arranging single digit counts of atoms into a transistor. A sub-Å process would involve single atom transistors. A sub 0.5Å process would mean making them out of subatomic particles. This isn't even possible in sci-fi.
You can re-arrange them for minor boosts, double the performance a few times sure, but that's not a sustained improvement month upon month like we have in the past.
As anyone who has ever optimized code will attest, optimization within fixed constraints typically hits diminishing returns very quickly. You have to work harder and harder for every win, and the wins get smaller and smaller.
Current process nodes are mostly 5nm, with 3nm getting rolled out. Atomic is ~0.1nm, which is x30 linear and x900 by area.
However, none of that is actually important when the thing people care about most right now is energy consumed per operation.
This metric dominates for anything battery powered for obvious reasons; less obvious to most is that it's also important for data centres where all the components need to be spread out so the air con can keep them from being damaged by their own heat.
I've noticed a few times where people have made unflattering comparisons between AI and cryptocurrency. One of the few that I would agree with is the power requirements are basically "as much as you can".
Because of that:
> double the performance a few times sure, but that's not a sustained improvement month upon month like we have in the past.
"Doubling a few times" is still huge, even if energy efficiency was perfectly tied to feature size.
But as I said before, the maximum limit for energy efficiency is in the order of a billion-fold, not the x900 limit in areal density, and even our own brains (which have the extra cost of being made of living cells that need to stay that way) are an existential proof it's possible to be tens of thousands of times more energy efficient.
That's not true. You can buy Raspberry PI, which is 10x cheaper and 10x more powerful than the computers at the beginning of 2000s.
Ditto with mobile phones. iPhone may be more expensive than when it launched, but you can buy dirt-cheap chinese smartphones that have similar performance - if not higher to the first iPhones.
> 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law.
Missing the point, despite being internally correct: 10% of $700k/day is still $25M/y.
If you'd instead looked at my point about energy cost per operation, there's room for something like 46,000 improvement just to human level, and 5.3e9 to the Landauer limit.
There are a few avenues. Further specialization of hardware around LLMs, better quantization (3 bits/p seems promising), improved attention mechanisms, use of distilled models for common prompts, etc.
This would be optimizations, which is not really the same thing as moore's law-like growth which was absolutely mind-boggling, like it's hard to even wrap your head around how fast tech was moving in that period since humans don't really grok exponentials too well, we just think they look like second degree polynomials.
Probabilistic computing offers the potential of a return to that pace of progress. We spend a lot of silicon on squashing things to 0/1 with error correction, but using analog voltages to carry information and relying on parameter redundancy for error correction could lead to much greater efficiency both in terms of OPS/mm^2 and OPS/watt.
I am wondering about this as well - wondering how difficult it would be to build an analog circuit for a small LLM (7B?). And wondering if anyone's working on that yet. Seems like an obvious avenue to huge efficiency gains.
Seems very unrealistic when considering how electromagnetic interference works. Clamping the voltages to high and low goes some way to mitigate that problem.
What I mean is resources will be limited or models that are slightly worse will be released that will be much more cost effective but not quite as good.
This is often the case with these types of technologies.
Interesting numbers. That roughly equates to about $250 million per year plus I don't know how much training is costing them to keep the model up to date and suchlike.
The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million.
That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been at a company that had 1:1 costs for running servers vs employee costs but I could entirely be off base here.
Given Sam Altman's recent comments on the days of these LLM being over I think maybe Microsoft or whomever is basically saying that they can't spend that much money and they need to control costs much more heavily.
This, there's like a endless line of companies waiting to snatch OpenAI's employees right outside the door. $200k average comp at OpenAI would be laughable.
As a side, I am a bit shocked by these numbers. Is this an American thing? I understand myself to be good software engineer with good well rounded experience of 14+ years. Yet my income, in Europe, is really above 100k.
What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?
Honestly, depends. Some teams at FAAMNG are really stressful and if you work on a Tier 1 service even with loads of SRE support you have to be working a fair bit. That being said, the pay is for design decisions at the higher IC level (senior or staff) and most people at that level are very smart. I’m not saying this salary is for 10x engineers or anything.
I would say 50% the work is harder and consuming and then 50% they can just afford to pay you more and lock up talent because of the wild margins on their products.
Amazon has a terrible reputation for internal infrastructure issues, with "on call" being a truly shitty experience for employees. aka burn out over a year is common
Note that there's likely to be some variation per team, but Amazon is famously bad, so ... ;)
I'd argue it's the opposite. We're coming off a decade of free money driving a second tech boom.
If interest rates stay elevated, and value investing becomes valuable again, it will be interesting to see how the tech space transforms. When start-ups have to compete with money market funds or even treasuries for investor cash, things become orders of magnitude tighter and more challenging.
I’ve been through both horror (endless 100 hour weeks) and bliss (just attending meetings and not really stressing about much of anything) in that range. It’s highly variable.
Yes, though Switzerland approaches it. If you want to see how much people of various levels of experience get paid at different companies and in different locations go to levels.fyi
Americans get paid much, much more than anyone else.
Your standard of living might be comparable. Your retirement is taken care of, you have a reasonable amount of vacation, you have better job security, your health care, in most European countries, has much less hassle, and your property costs are lower.
I am seriously considering a move if my husband can find an academic job over there. The retirement won't be a great lure (fewer years in the system) but we almost have enough to coast from here, so it's about the rest.
Taxes in the bay area can be insane - ~40% if I remember correctly. On top of that you have crazy-expensive healthcare, and crazy expensive housing costs.
~100k€ in (western) Europe may be comparable to ~200k€ in Bay Area.
I'm 20 years into programming and a senior architect and lead on an enterprise project.
I don't even make that first number.
But I value certain things way more than other things, and my current job provides it. Fully remote, leaves me completely alone to accomplish what they need done (and I get it done), unlimited vacation, great benefits, zero pointless meetings (almost an empty calendar).
I'm sure these other companies offer some of that but 500k?! That is absurd.
They're interesting numbers, but the linked article's cite amounts to:
> ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.
which, pardon me, but no shit.
Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances.
The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.
that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.
Why is that ridiculous? Cloud services gives companies "coupons" and free usage for X hours for a bunch of other companies, why wouldn't they do that for a company they invested heavily in?
Because that's not how it works. Even company cars of General Motors employees have to be purchased from General Motors.
Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.
At best OpenAI has negotiated a near 0 profit margin for Microsoft when paying for the services. But even that is unlikely given how much money/resources are involved. There's no scenario where it's free at that scale.
The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here.
> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
I should clarify: training is not latency sensitive, so you can run your workloads at off-peak hours when people are asleep. Inference means you need to run your workloads at peak when people are using your service.
(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)
That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.
"“I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said.
He sees size as a false measurement of model quality and compares it to the chip speed races we used to see. “I think there’s been way too much focus on parameter count, maybe parameter count will trend up for sure. But this reminds me a lot of the gigahertz race in chips in the 1990s and 2000s, where everybody was trying to point to a big number,” Altman said.
As he points out, today we have much more powerful chips running our iPhones, yet we have no idea for the most part how fast they are, only that they do the job well. “I think it’s important that what we keep the focus on is rapidly increasing capability. And if there’s some reason that parameter count should decrease over time, or we should have multiple models working together, each of which are smaller, we would do that. What we want to deliver to the world is the most capable and useful and safe models. We are not here to jerk ourselves off about parameter count,” he said."
Interesting, the 2020 revenue and costs are significantly lower than previous years. Actually the prior years give a much better insight into salaries there. I wonder if this is because they switched from the non-profit model to the for-profit subsidiary at that time?
Sam Altman didn't say LLMs are over. (He's the CEO of OpenAI, so that would be a really strange thing for him to say, wouldn't it?)
What he actually said was that we've reached the point where we can't improve model quality simply by increasing its size (number of parameters). We'll need new techniques to continue to improve.
I suspect the number will go higher since right now you are limited to 25 queries every 3 hours. They will definitely introduce higher tiers which removes or expands that limit.
I expect a lot of optimization will come in the next few months (weeks?)... it may be hanging together by threads behinds the scenes - this is often the case when things are moving this quickly.
I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization.
It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at $20/month, it was such a no-brainer, that I had to do it. Even at $75/mo it would more than pay for itself.
It even understands the concept of a "shit post". So - its more social media savvy than I am, thats for sure.
You should give it a shot, keep an open mind. Yes, it will get things wrong and yes, it will disappoint you. But every conversation I've had with it holds some surprises for me. It's a tool you'll need to learn to use before it actually becomes useful, but give it a couple minutes a few times a week and you'll start to see the diamonds in the rough.
Nerd-snipe and useful? That’s the holy grail! We’re having fun and we’re solving real problems that we have. Amazing.
I’ve already incorporated it into my publishing process. My home-grown “cms” uses ChatGPT (via API) to write my article description, draft a twitter thread, and craft a “viral insight”. The latter is mostly useful to make sure my article even makes a point.
Hoping to use it for a related articles feature next. I’m also building a chatbot based on my content.
Yes. i think its a good software product! Maybe one of the best there ever was - I'm not surprised Sam Altman jumped the YC ship for this one, or that MS is particularly interested it. It gets ignored in all the other bru-haha but I'm excited to see what the really talented software teams of the world will be able to accomplish with an AI coding assisstent - and I don't mean just in the world of AI.
The activation energy for new software lowered by about 30% overnight, which is outrageously cool, and of course disquieting.
Especially for new software in a different language than you already know or haven't used years, in a realm other than your current area of expertise. It's pretty useful to be able to directly ask it how do to, eg basic string manipulation in $new_language, but even more convenient is to ask it to do the transformation for you. "Given a string of the format X, write a $language snippet that outputs X'." It'll do more advanced things than that, but that's one area I found it to be useful and time-saving in an unfamiliar language because it saves me from looking up that boilerplate before getting to the part where I throw a regexp at the problem.
I paid for it, then stopped paying for it. It seemed to be completely novel and fun to use but after a while it just became a pain. I tried writing a library to act as a code assistant for data science but trying to get it to write the codes that were useful to me took around the same amount of time as writing them myself. What you say is completely correct, it gets to around 70% of whatever you ask it to do then you have to finish the job for it. Which I guess is nice. But I'm not going to pay 20$/month for something that i can just use for free that amounts to slightly better intellisense.
For text generation, it is much better for tasks like
* letter writing
* rewriting my writing so i don't plaigerize myself
* summarizing several paragraphs into one
My habit is to cancel right away when I start a subscription. You can still use the service for the month you paid for, and it's easy to restart it if you do still find yourself attempting to use it a month later.
Substack disappointed me because you can't do this. They remove content access if you do not have an "active" subscription. You don't have to pay again to "reactivate", but the paid content is locked until you do.
This is obviously intended dark behavior because when I went to fully unsub later in the month they gave me a special dialog and a free month hoping for me to forget again this month.
Very lame.
Oh, you haven't seen darker behavior of many Japan based services:
- Payment cycle closes always on the 1st of the month (which means, if you sign up on the last day of the month, you get 1 day of service for full month of payment. No proration, or anything.)
- If you cancel the service, you'll immediately lose access, doesn't matter if it's 1st day of the month, or very end of the month, it's gone. To reenroll, you often have to pay again.
I've been using Privacy.com cards for years to avoid this. Set a limit of 1 month on the card, and if I'm actually using the sub I'll fix/update it when the sub runs out.
I had a boss I didn't like a while ago, but one of the things he did that I did like was set a calendar reminder to unsubscribe. Too bad; it would've been easier on me mentally and emotionally if he were completely irredeemable, but I admit to using this trick myself now.
I don't think it's cheap to pay for a service, test it out for a month, find that in the beginning you used it everyday, and by the end of the month you forgot it existed. Then decide to not keep paying for it.
Probably for the next proposal I write, I'll pay for it again. It's super useful to take care of all the bullshit things you have to write for science to not plaigerize yourself.
I also tried using it as a dungeon master like that blog post from a couple weeks ago. But gpt4 didn't seem to remember things with regularity enough to actually work. Basically there was an uncanny valley because gpt4 doesn't have a pad of paper to write things down on like any person would have.
>> I'm not going to pay 20$/month for something that I can just use for free
Well, it works until it doesn't because the website is overloaded. $20 gets you in through the VIP door and you don't need to wait along with the rest of the peasants.
You also get (throttled) GPT 4 which is much better at many tasks. That’s why I paid. We’ll see if the novelty lasts for very long but right now it’s a exciting product that I use multiple times a day and the most sci-fi invention of my life so far so I don’t mind paying
I didn't see any difference between 3.5 turbo and 4 for the data science app that I wrote. Another thing is that it's stochastic. So you could actually get a prompt that works well then you run it again and the identical prompt that generated codes that work now has generated codes that don't work. Of course, I could have my data science app cache codes that worked so if the prompt doesn't change it already has the generated code but I didn't want to have an ever growing repository of cached codes. Anyways.
In my experience 3.5 is so far away from 4 that it makes it makes 3.5 useless. I'd give another shot to gpt 4 if 3.5 was almost working for you (what did you use it for, anything for specific?)
phind.com claims they use GPT4 but you have turn Expert mode on. Otherwise it's their home baked model. It's explained here:
> Expert mode is our most advanced searching mode, powered by GPT-4. This mode hallucinates less and writes better code. We highly recommend that you use it for advanced questions. Whenever the "regenerate" button is pressed, Expert mode is used to increase the odds of a high-quality answer.
Yes, some of the transformation mechanisms are nice if you need to dump info into it and want it reorganized or processed into something else as well. But outside of that it's hard to justify spending $20 month so you can be bullshitted by some know-it-all toddling proto-skynet when you need factual information.
I've been using it for product ideas as well. Occasionally it's brilliant but sometimes it's so bad it's funny, like when it suggested Seven-Year Itch for a perfume brand.
I actually like to make the bad ones. It is fun. It is an exercise in creative engineering. Especially since my side project is making home goods for stoners.
Don't they say something about seemingly brilliant ideas being too obvious or usually not working out, and the terrible ones being the ones that are have more potential to be diamonds in the rough?
But also it would heavily depend on the context they gave it when asking it to generate that.
Because GPT is capable of: doing marketing, math, science stuff, programming and anything else to a junior-semi advanced level, it needs to be prompted and given context to get it into the right "mood" for doing what you want it to.
There's a huge difference between casually chatting to it about various topics before suddenly asking what a good perfume brand would be. Versus providing it with the fragrances used in the perfumes that the brand makes, the general goals/image that the company wants to set, their target demographic etc and _then_ asking for a brand name suggestion.
> It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at $20/month, it was such a no-brainer, that I had to do it.
Yup. I bought accounts for everyone here. We are using it as what I have been calling a "force multiplier". We are not and cannot use it for coding, yet, things like presentations, analyzing logs, creating lists of things, researching topics, etc. It's a great time saver.
Also, for a lot of things ChatGPT is a much better search engine than Google. It gets you great answers and almost always the first time you ask.
In case the question comes up: We don't use it for coding because of potential liability concerns. At this point I feel that is a space that has not been explored at all. I have no interest in being a pioneer in a lawsuit that claims negligence due to the use of AI-generated code, even if it is reviewed by a human. The combination of fear-mongering and tech-challenged juries could make for very expensive outcomes.
These costs are falling by factors of 2 every year based solely on hardware progress ( gpu tech has beat Moore’s law for DNNs the last several years). ChatGPT may become much cheaper to operate/train in the future.
interesting that the article mentions Microsoft is trying to build their own AI chips to help reduce cost, they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game
> they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game
Where do we think Apple is in regards to this? If anyone has the upper-hand here with hardware, I would think it'd be Apple. But there's been zero indication that Apple has been working on any sort of generative AI.
In terms of production pipeline, sure. But since there’s an established model architecture, the hardware doesn’t need to be super generic. Google had to build TPUs in a world where everything was still rapidly changing - deciding on things like precision, memory bandwidth, SRAM size etc. Microsoft could theoretically stamp out some GPT ASICS and call it a day.
They are actually as good or better depending on the use case. IIRC their compute is around the same, and the compute per watt is substantially better. They are also purpose built for transformers.
But google is fumbling so hard right now it's not surprising that they are squandering this too.
At the end of the day, all we are doing is create, read, update, and delete data.
I mean if you take away all the complexity associated with ranking and scale (not everything has to be Google scale), that is exactly what a web search is as well, right?
I remember reading a post by a maintainer saying their job is to manipulate strings or something to that effect. iirc it was a gofmt maintainer who said that but I can't find that post now.
That’s just on computing power. You are forgetting the human costs of ongoing development, maintenance of ChatGPT, power, SRE/operations, cooling costs, and et cetera.
Now M$ is planning to create a specialized chip for AI which comes with its own R&D budget and ongoing costs.
If it proves successful, ChatGPT will become a household brand and M$ could easily ask for $500 per month or more for professional/corporate usage.
my point is that it is a derogatory abbreviation commonly used by people who, in fact, do not have an unbiased view of Microsoft.
it's harder to type than just "MS" so when someone uses "M$" they go out of their way to signal that they are biased. Being biased is fine, so long as one understands that they are biased and that they are communicating their bias along with the rest of the message.
The problem with this model is that if ChatGPT really is "good enough" now - many others will be "good enough" soon, and then it's a race to the bottom.
Users or accounts? I made an account but I use it approximately once every two weeks for about 30 minutes at a time, as it hasn't been that useful for me (needs more up-to-date info after September 2021).
I imagine many people made an account but only a small percentage are using it meaningfully often.
Also users are limited in the number of requests they can make. I have a feeling ChatGPT is actually very expensive to run, and they are burning cash like crazy.
If they weren't burning cash to run the thing, it would be more widely available.
My main concern with AI is that I'm not so sure if there is a market for any kind of intelligence anymore. So much human talent and intelligence has been wasted over the past few years; so many potentially useful projects went unfunded, I struggle to think of what purpose AI would serve in such an economic system as ours. I can see its potential, but I can't see our current system facilitating that potential. It seems more likely that the technology will be applied towards controlling the masses rather than benefiting them; this has been the purpose of most major tech 'innovations' of the last decade.
Wealthy individuals these days hardly need any intelligence at all to stay rich. On the other hand, it seems like poor individuals have no chance no matter how much intelligence they have... Does anyone really need more intelligence? Most human intelligence seems to be wasted on bullshit jobs anyway.
What we should be asking ourselves is "how much more comfortable can we make our billionaires?" Because the entire concept of 'economic efficiency' appears to be about optimizing society towards that goal.
Agreed. I entered the workforce as a developer in 2012; it felt like the tail-end of the tech boom but intelligence was still valued. Nowadays, most people don't even recognize intelligence, let alone value it.
Intelligence briefly became useful again as the Internet spontaneously created a bunch of mental real estate. Plowing that took the skills that you would find in a person prone to mentalising. So you got a shot at selling that and you might've even been able to derive some small feeling of privilege from it for a while, even though true power and authority, after all, do not depend on any particular quality, but exist in defiance to all, especially intelligence.
Funny to think all that was a generation ago, that the economical conditions for our artifice are now history and it's once again economical to study the much more all-encompassing art of not thinking. (It's very cyclical but, since it involves not thinking, i.e. the termination of a coherent thread of thought, you think it's for the first time every time. But sure, plug in to the simulation at 110% and get 10 years of life for free, you won't even notice the difference, after all, you're already not noticing it.)
This is how I, a unit of human capital from the former global East, see the matter of intelligence and automating it at scale: "intelligence" is already something that was forced onto my society in the 1950s, and once again on the 1990s, so that we could be interfaced with the same kinds of systems that the conquering populations exported, under conditions of forcible termination of local culture at the cost of great atrocities. By the time we started building our own Borg cube we'd already been half Borged by the Western one. If the word in some cases denotes another kind of abstract human "intelligence", I have rarely if ever seen such cases exemplified in public. So now it's a bit distasteful to equate "intelligence" with "savoir faire". Having neuromorphic computing in the first place would've saved a lot of suffering in comparison with how we were "civilized" to support only the established market protocols and nothing of our own.
Which is something someone will eventually have to admit - probably after a Pontypool-class scenario wipes out their investment in predictable outcomes for a region and they're left with a huge mess on their hands to mop up. And then it turns out the consequences themselves have been thought in such inhumanly fine detail as to possess a rudimentary intelligence of their own and the mop up ends up embedding a sentient malevolent ghost into the nature of that branch of reality. Shit happens, at a certain saturation with Internet stream of thought an omnipresent autoritarian AI even becomes fun to imagine. Of course if one existed its cruelty would be much more banal, as is always the difference between mentalization and reality until the next brief Golden Age (for some of the people, some of the time) and then it's "oh no this time we're fucked" again (for some other people).
So sad because there are so many great ideas worth building and fighting for, even in the current limits of human knowledge, but in practice any form of innovation is only tolerated if it's backed by a major financial effort by the upholders of the status quo. For some reason so many times this has ended to the detriment of nonparticipants that we've basically stopped noticing or keeping count. As regulatory capture, so learned helplessness. The economic reality of already existing in a world of inscrutable mechanisms as that most fragile of creatures, the independent knowledge worker, is a psychological stressor, to the extent that everyone's so burnt by the effort on giving up on things that in this new ecosystem are "too smart to work". Help, the AI stole my role in the food chain and now all my thoughts lead nowhere.
I’ve never heard a more cynical AI take. There’s always a market for intelligence, whatever that means. The word intelligence is misleading anyway. No one said the AI needed to compete with the smartest person on the planet. It just needs to be smart enough to do some tasks that are “bullshit jobs”
>No one said the AI needed to compete with the smartest person on the planet. It just needs to be smart enough to do some tasks that are “bullshit jobs”
And then the smartest person on the planet figures out a better thing for the people who lost their "bullshit jobs" to do, right? No, gets sent to space to see how much even the smartest person on the planet really matters.
> There’s always a market for intelligence, whatever that means. The word intelligence is misleading anyway.
Agreed. What is decided here is to what extent there will continue to be a market for human intelligence. There are of course opportunity costs that humans who entered a market that they apparently believed to exist, will now have to, apparently, absorb. Or will we once again have to voluntarily undergo unsupervised radical mental restructuring - on the degree of your grandma having to learn how to navigate the Internet, and of her grandma presumably never having to do anything of the sort - and all that just to retain our Red Queen out of checkmate? It's always too early until it's too late.
There is definitely a market for an intelligence crutch or assisted critical-thinking aid.
Like people slowly getting more obese, the general IQ and CT have been sliding for the last few decades. Not sure if it’s the food, media or zanax but society needs something.
I’d wager the Altmans of the world know this and figured out a way to monetize it. Necessity is the mother of invention after all.
Having spent 11MM annually on infra to effectively (though obviously not efficiently despite a very favorable ROI) generate reports for <1000 customers, that sounds like a bargain.
This tech can more or less improve productivity in every sector of the economy, and that includes energy/climate change.
Personally, I've used it with success with R&D and due dil of projects related to climate change. LLMs (and progress in general) can help tremendously with switching to a sustainable economy.
This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly.
For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kind of traffic touch the expensive part of the architecture?
It's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).
> That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).
The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means.
"give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response.
Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?
Sure for Bing type interfaces it makes sense but if you consider what it is more holistically there are two questions: 1) do you want it to be non-deterministic and hope the first answer is the one you want everyone to see and 2) do you invalidate the entire cache every time there's a minor update to the model? How else do you tell which keys to remove?
You'd basically be removing the entire cache every release
You know some people used to use Google as a glorified spell checker? I use ChatGPT as a glorified stupidity checker. What I mean is I ask it the silliest of the doubts. Like how to set environment variable in Windows (because we are all used to EXPORT aren't we?), whether or not can we do X in K8S YAML in a given conversation.
Obviously I use it for other purposes as well, but it definitely has saved me a lot of hours getting the basics things right there in a prompt.
> explaining repos and apis instead of reading all the docs - help with debugging
You can only do this with something that existed in its training set, right? There's no way to point it at a random GitHub repo and say "digest this, so I can ask you questions about it"?
Using the GPT-4 32k context model I would guess you can just paste the entire docs in there, and ask it questions about it. 32k tokens is about 24k words.
222 comments
[ 3.8 ms ] story [ 273 ms ] threadLet's say 500 employees, earning above Bay Area average, along with benefits etc at $400K average per year. That's $200 million a year in wages, so in the same ballpark as the $200 million in infrastructure costs cited in the article, for a total of $400 million in expenses per year.
I would imagine that Google's reputation as a company where it is impossible to ever talk to a human (even when you have a 7 figure annual spend) hurts them in this space.
Once people already are buying 20 products from you and have a good sales relationship of decades, selling them some cloud services is easy and might even lead to better deals on something else (Windows/Office)
The problem for both of them is AWS, which somehow manages to give you your cake and let you eat it too, even if it's a little more expensive.
Sometimes it's better to be the fast follower.
You Z80 computer cost $700 in the lat 70's...they're now in sub-$1 embedded controllers.
10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law.
I don't really see anything like that here.
I can't confirm it, but I noticed this comment says "gpu tech has beat Moore’s law for DNNs the last several years":
https://news.ycombinator.com/item?id=35653231
For a long time, GPU hardware basically became more powerful with each generation, but prices stayed roughly the same plus minus inflation. Last couple of years, this trend has broken. You pay double or even quadruple the price for a relatively tenuous increase in performance.
You get the point.
There's always local optimization that leads to improvements. Look at the Apple M1 chip rollout as a prime example of that. Big/Little processors, on die RAM, shared memory with the GPU and Neural Engine, power integration with the OS.
LOTS of things that led to a big leap forward.
You can re-arrange them for minor boosts, double the performance a few times sure, but that's not a sustained improvement month upon month like we have in the past.
As anyone who has ever optimized code will attest, optimization within fixed constraints typically hits diminishing returns very quickly. You have to work harder and harder for every win, and the wins get smaller and smaller.
However, none of that is actually important when the thing people care about most right now is energy consumed per operation.
This metric dominates for anything battery powered for obvious reasons; less obvious to most is that it's also important for data centres where all the components need to be spread out so the air con can keep them from being damaged by their own heat.
I've noticed a few times where people have made unflattering comparisons between AI and cryptocurrency. One of the few that I would agree with is the power requirements are basically "as much as you can".
Because of that:
> double the performance a few times sure, but that's not a sustained improvement month upon month like we have in the past.
"Doubling a few times" is still huge, even if energy efficiency was perfectly tied to feature size.
But as I said before, the maximum limit for energy efficiency is in the order of a billion-fold, not the x900 limit in areal density, and even our own brains (which have the extra cost of being made of living cells that need to stay that way) are an existential proof it's possible to be tens of thousands of times more energy efficient.
Ditto with mobile phones. iPhone may be more expensive than when it launched, but you can buy dirt-cheap chinese smartphones that have similar performance - if not higher to the first iPhones.
Missing the point, despite being internally correct: 10% of $700k/day is still $25M/y.
If you'd instead looked at my point about energy cost per operation, there's room for something like 46,000 improvement just to human level, and 5.3e9 to the Landauer limit.
How is it not?
These LLMs were recently trained using NVidia A100 GPUs.
Now NVidia has H100 GPUs.
The H100 is up to nine times faster for AI training and 30 times faster for inference than the A100.
This is often the case with these types of technologies.
To not look far - gpt3.5 turbo.
The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million.
That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been at a company that had 1:1 costs for running servers vs employee costs but I could entirely be off base here.
Given Sam Altman's recent comments on the days of these LLM being over I think maybe Microsoft or whomever is basically saying that they can't spend that much money and they need to control costs much more heavily.
What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?
I would say 50% the work is harder and consuming and then 50% they can just afford to pay you more and lock up talent because of the wild margins on their products.
Note that there's likely to be some variation per team, but Amazon is famously bad, so ... ;)
I've said it before on here, but I live very comfortably in Philly for a lot less than that.
If interest rates stay elevated, and value investing becomes valuable again, it will be interesting to see how the tech space transforms. When start-ups have to compete with money market funds or even treasuries for investor cash, things become orders of magnitude tighter and more challenging.
Yes, though Switzerland approaches it. If you want to see how much people of various levels of experience get paid at different companies and in different locations go to levels.fyi
Americans get paid much, much more than anyone else.
I am seriously considering a move if my husband can find an academic job over there. The retirement won't be a great lure (fewer years in the system) but we almost have enough to coast from here, so it's about the rest.
I was expecting salaries to cool off a bit with the massive wave of layoffs across the industry, but from what I've seen, that hasn't happened.
~100k€ in (western) Europe may be comparable to ~200k€ in Bay Area.
I'm 20 years into programming and a senior architect and lead on an enterprise project.
I don't even make that first number.
But I value certain things way more than other things, and my current job provides it. Fully remote, leaves me completely alone to accomplish what they need done (and I get it done), unlimited vacation, great benefits, zero pointless meetings (almost an empty calendar).
I'm sure these other companies offer some of that but 500k?! That is absurd.
> ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.
which, pardon me, but no shit.
Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances.
The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.
Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.
Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"
They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.
I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well.
While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).
> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)
That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.
He sees size as a false measurement of model quality and compares it to the chip speed races we used to see. “I think there’s been way too much focus on parameter count, maybe parameter count will trend up for sure. But this reminds me a lot of the gigahertz race in chips in the 1990s and 2000s, where everybody was trying to point to a big number,” Altman said.
As he points out, today we have much more powerful chips running our iPhones, yet we have no idea for the most part how fast they are, only that they do the job well. “I think it’s important that what we keep the focus on is rapidly increasing capability. And if there’s some reason that parameter count should decrease over time, or we should have multiple models working together, each of which are smaller, we would do that. What we want to deliver to the world is the most capable and useful and safe models. We are not here to jerk ourselves off about parameter count,” he said."
via https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...
What he actually said was that we've reached the point where we can't improve model quality simply by increasing its size (number of parameters). We'll need new techniques to continue to improve.
If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even.
Now it makes sense why they chose to charge $20/m, I predicted much higher.
It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at $20/month, it was such a no-brainer, that I had to do it. Even at $75/mo it would more than pay for itself.
It even understands the concept of a "shit post". So - its more social media savvy than I am, thats for sure.
It would benefit her immensely to use it.
#include <windows.h>
then it is a must have.
I’ve already incorporated it into my publishing process. My home-grown “cms” uses ChatGPT (via API) to write my article description, draft a twitter thread, and craft a “viral insight”. The latter is mostly useful to make sure my article even makes a point.
Hoping to use it for a related articles feature next. I’m also building a chatbot based on my content.
Yes. i think its a good software product! Maybe one of the best there ever was - I'm not surprised Sam Altman jumped the YC ship for this one, or that MS is particularly interested it. It gets ignored in all the other bru-haha but I'm excited to see what the really talented software teams of the world will be able to accomplish with an AI coding assisstent - and I don't mean just in the world of AI.
The activation energy for new software lowered by about 30% overnight, which is outrageously cool, and of course disquieting.
For text generation, it is much better for tasks like
* letter writing * rewriting my writing so i don't plaigerize myself * summarizing several paragraphs into one
but all of that is available in the free version.
If you're like me you tend to forget you're paying for something, and have to do a yearly purge.
It's very easy to slowly but surely rack up hundreds of dollars a month in subscription services.
- Payment cycle closes always on the 1st of the month (which means, if you sign up on the last day of the month, you get 1 day of service for full month of payment. No proration, or anything.)
- If you cancel the service, you'll immediately lose access, doesn't matter if it's 1st day of the month, or very end of the month, it's gone. To reenroll, you often have to pay again.
Probably for the next proposal I write, I'll pay for it again. It's super useful to take care of all the bullshit things you have to write for science to not plaigerize yourself.
I also tried using it as a dungeon master like that blog post from a couple weeks ago. But gpt4 didn't seem to remember things with regularity enough to actually work. Basically there was an uncanny valley because gpt4 doesn't have a pad of paper to write things down on like any person would have.
Well, it works until it doesn't because the website is overloaded. $20 gets you in through the VIP door and you don't need to wait along with the rest of the peasants.
> Expert mode is our most advanced searching mode, powered by GPT-4. This mode hallucinates less and writes better code. We highly recommend that you use it for advanced questions. Whenever the "regenerate" button is pressed, Expert mode is used to increase the odds of a high-quality answer.
https://www.phind.com/tutorial
Yeah, what's wrong with that? I'm sure it would be catchy to a particular demographic. No need for a brand to cater to everyone.
Because GPT is capable of: doing marketing, math, science stuff, programming and anything else to a junior-semi advanced level, it needs to be prompted and given context to get it into the right "mood" for doing what you want it to.
There's a huge difference between casually chatting to it about various topics before suddenly asking what a good perfume brand would be. Versus providing it with the fragrances used in the perfumes that the brand makes, the general goals/image that the company wants to set, their target demographic etc and _then_ asking for a brand name suggestion.
I'd be keen to see how they asked.
better source : https://tech.slashdot.org/story/23/04/18/1326235/microsoft-r...
Yup. I bought accounts for everyone here. We are using it as what I have been calling a "force multiplier". We are not and cannot use it for coding, yet, things like presentations, analyzing logs, creating lists of things, researching topics, etc. It's a great time saver.
Also, for a lot of things ChatGPT is a much better search engine than Google. It gets you great answers and almost always the first time you ask.
In case the question comes up: We don't use it for coding because of potential liability concerns. At this point I feel that is a space that has not been explored at all. I have no interest in being a pioneer in a lawsuit that claims negligence due to the use of AI-generated code, even if it is reviewed by a human. The combination of fear-mongering and tech-challenged juries could make for very expensive outcomes.
I posed a question about this here:
https://news.ycombinator.com/item?id=35655521
Where do we think Apple is in regards to this? If anyone has the upper-hand here with hardware, I would think it'd be Apple. But there's been zero indication that Apple has been working on any sort of generative AI.
But google is fumbling so hard right now it's not surprising that they are squandering this too.
At the end of the day, all we are doing is create, read, update, and delete data.
I mean if you take away all the complexity associated with ranking and scale (not everything has to be Google scale), that is exactly what a web search is as well, right?
I remember reading a post by a maintainer saying their job is to manipulate strings or something to that effect. iirc it was a gofmt maintainer who said that but I can't find that post now.
Now M$ is planning to create a specialized chip for AI which comes with its own R&D budget and ongoing costs.
If it proves successful, ChatGPT will become a household brand and M$ could easily ask for $500 per month or more for professional/corporate usage.
it's harder to type than just "MS" so when someone uses "M$" they go out of their way to signal that they are biased. Being biased is fine, so long as one understands that they are biased and that they are communicating their bias along with the rest of the message.
Users or accounts? I made an account but I use it approximately once every two weeks for about 30 minutes at a time, as it hasn't been that useful for me (needs more up-to-date info after September 2021).
I imagine many people made an account but only a small percentage are using it meaningfully often.
Also users are limited in the number of requests they can make. I have a feeling ChatGPT is actually very expensive to run, and they are burning cash like crazy.
If they weren't burning cash to run the thing, it would be more widely available.
Wealthy individuals these days hardly need any intelligence at all to stay rich. On the other hand, it seems like poor individuals have no chance no matter how much intelligence they have... Does anyone really need more intelligence? Most human intelligence seems to be wasted on bullshit jobs anyway.
What we should be asking ourselves is "how much more comfortable can we make our billionaires?" Because the entire concept of 'economic efficiency' appears to be about optimizing society towards that goal.
Wasn't it always more or less like this?
Funny to think all that was a generation ago, that the economical conditions for our artifice are now history and it's once again economical to study the much more all-encompassing art of not thinking. (It's very cyclical but, since it involves not thinking, i.e. the termination of a coherent thread of thought, you think it's for the first time every time. But sure, plug in to the simulation at 110% and get 10 years of life for free, you won't even notice the difference, after all, you're already not noticing it.)
This is how I, a unit of human capital from the former global East, see the matter of intelligence and automating it at scale: "intelligence" is already something that was forced onto my society in the 1950s, and once again on the 1990s, so that we could be interfaced with the same kinds of systems that the conquering populations exported, under conditions of forcible termination of local culture at the cost of great atrocities. By the time we started building our own Borg cube we'd already been half Borged by the Western one. If the word in some cases denotes another kind of abstract human "intelligence", I have rarely if ever seen such cases exemplified in public. So now it's a bit distasteful to equate "intelligence" with "savoir faire". Having neuromorphic computing in the first place would've saved a lot of suffering in comparison with how we were "civilized" to support only the established market protocols and nothing of our own.
Which is something someone will eventually have to admit - probably after a Pontypool-class scenario wipes out their investment in predictable outcomes for a region and they're left with a huge mess on their hands to mop up. And then it turns out the consequences themselves have been thought in such inhumanly fine detail as to possess a rudimentary intelligence of their own and the mop up ends up embedding a sentient malevolent ghost into the nature of that branch of reality. Shit happens, at a certain saturation with Internet stream of thought an omnipresent autoritarian AI even becomes fun to imagine. Of course if one existed its cruelty would be much more banal, as is always the difference between mentalization and reality until the next brief Golden Age (for some of the people, some of the time) and then it's "oh no this time we're fucked" again (for some other people).
So sad because there are so many great ideas worth building and fighting for, even in the current limits of human knowledge, but in practice any form of innovation is only tolerated if it's backed by a major financial effort by the upholders of the status quo. For some reason so many times this has ended to the detriment of nonparticipants that we've basically stopped noticing or keeping count. As regulatory capture, so learned helplessness. The economic reality of already existing in a world of inscrutable mechanisms as that most fragile of creatures, the independent knowledge worker, is a psychological stressor, to the extent that everyone's so burnt by the effort on giving up on things that in this new ecosystem are "too smart to work". Help, the AI stole my role in the food chain and now all my thoughts lead nowhere.
If that's you, look for a bigger room.
And then the smartest person on the planet figures out a better thing for the people who lost their "bullshit jobs" to do, right? No, gets sent to space to see how much even the smartest person on the planet really matters.
> There’s always a market for intelligence, whatever that means. The word intelligence is misleading anyway.
Agreed. What is decided here is to what extent there will continue to be a market for human intelligence. There are of course opportunity costs that humans who entered a market that they apparently believed to exist, will now have to, apparently, absorb. Or will we once again have to voluntarily undergo unsupervised radical mental restructuring - on the degree of your grandma having to learn how to navigate the Internet, and of her grandma presumably never having to do anything of the sort - and all that just to retain our Red Queen out of checkmate? It's always too early until it's too late.
>I’ve never heard a more cynical AI take.
Sapienti sat.
Like people slowly getting more obese, the general IQ and CT have been sliding for the last few decades. Not sure if it’s the food, media or zanax but society needs something.
I’d wager the Altmans of the world know this and figured out a way to monetize it. Necessity is the mother of invention after all.
I kid, I kid.. or do I?
Yeah I guess having a brain is a privilege
I imagine people will be using this to create wasteful products, but also green solutions.
Personally, I've used it with success with R&D and due dil of projects related to climate change. LLMs (and progress in general) can help tremendously with switching to a sustainable economy.
For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kind of traffic touch the expensive part of the architecture?
The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means.
"give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response.
Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?
You'd basically be removing the entire cache every release
because of context. You can cache it if it is indeed the first sentence of a fresh dialogue, but that's it.
I'll start:
- making little scripts in shell / js / python that I'm not as fluent in. 5 min vs 1-3 hours
- explaining repos and apis instead of reading all the docs - help with debugging
- flushing out angles for new concepts that I did not previously consider (ex: how do you make a good decentralized exchange)
Obviously I use it for other purposes as well, but it definitely has saved me a lot of hours getting the basics things right there in a prompt.
You can only do this with something that existed in its training set, right? There's no way to point it at a random GitHub repo and say "digest this, so I can ask you questions about it"?
https://help.openai.com/en/articles/7127966-what-is-the-diff...