> For all enterprise customers, it offers:
> Customer prompts and company data are not used for training OpenAI models.
> Unlimited access to advanced data analysis (formerly known as Code Interpreter)
> 32k token context windows for 4x longer inputs, files, or follow-ups
I'd thought all those had been available for non enterprise customers, but maybe I was wrong, or maybe something changed.
Having conversations saved to go back to like in the default setting on Pro, that's disabled when a Pro user turns on the privacy setting, is another big difference.
What about prompt input and response output retention for x days for abuse monitoring? does it not do that for enterprise? For Microsoft Azure's OpenAI service, you have to get a waiver to ensure that nothing is retained.
" We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. "
Which part of that is new, because I was pretty sure they were saying "we do not train on your business data or conversations, and our models don’t learn from your usage" already. Maybe the SOC 2 and encryption is new?
Use a third party interface which uses the API directly like YakGPT or the OpenAI playgrounds, and you can save some costs that way along with a local chat history that’s not shared with OpenAI.
Prior to releasing the chat history "feature" there was an opt-out form that could be submitted, which did not have any impact on the webapp's functionality. I'm not current enough to know if that form 1) ever had any effect, and 2) if a form-submitted opt-out is still valid given they now have the aforementioned in-app feature.
And you’re going to spend the time reviewing every single commit to make sure the dev didn’t sell out without telling anyone? Or risk running a potentially outdated and vulnerable extension?
The obvious point being that the “fork your own code and write your own kernel” attitude is simply unworkable for 99.999% of the population.
If we had to waste that much time re-inventing the loaf of bread, and then making sure that my neighbors didn’t decide to throw some raisins in my loaf, that we never get around to figuring out the next best thing: slicing it.
>" We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. "
That's great. But can customer prompts and company data be resold to data brokers?
It's exactly opposite. The entire point of an enterprise option would be that you DO train it on corporate data, securely. So the #1 feature is actually missing, yet is announced as in the works.
Can't believe the pushback I'm getting here. The use case is stunningly obvious.
Companies want to dump all their Excels in it and get insights that no human could produce in any reasonable amount of time.
Companies want to dump a zillion help desk tickets into and gain meaningful insights from it.
Companies want to dump all their Sharepoints and Wikis into it that currently nobody can even find or manage, and finally have functioning knowledge search.
You absolutely want a privately trained company model.
None of the use cases you are describing require training a new model. You really don't want to train a new model, that's not a good way of getting them to learn reliable facts and do so without losing other knowledge. The fine tuning for GPT 3.5 suggests something like under a hundred examples.
What you want is to get an existing model to search a well built index of your data and use that information to reason about things. That way you also always have entirely up to date data.
People aren't missing the use cases you describe, they're disagreeing as to how to achieve those.
>>Companies want to dump all their Excels in it and get insights that no human could produce in any reasonable amount of time.
>>Companies want to dump a zillion help desk tickets into and gain meaningful insights from it.
>>Companies want to dump all their Sharepoints and Wikis into it that currently nobody can even find or manage, and finally have functioning knowledge search.
Mature organizations already have solutions for all of these things. If you can't mine your own data competently, you've got bigger problems than not having AI doing it for you. It means you don't have humans who understand what's going on. AI is not the answer to everything.
I think those are examples of prompting, not modeling. You'd use the API to develop an app where the end user's question gets prefaced with that stuff. Modeling is more like teaching it how to sensibly use language, which can be centralized instead of each enterprise having experts in that. It would be like having in-house English teachers instead of sending people to school, based on a desire to have a corporate accent -- interesting but probably not useful in most cases.
I wonder if corporations would train it on emails/Exchange as well, since they are often technically company property and could contain valuable information not found in tickets/wikis.
ChatGPT doesn’t work for this. There is a huge GIGO problem here that it’s missing the organizational knowledge to disambiguate. Unless you’ve pre-told it which excel sheets are correct, this is DOA.
ChatGPT only works as well as it does because it’s been trained on a corpus of “internet accepted” answers. It can’t fucking reason about raw data. It’s a language model.
But that's exactly the point, an enterprise offering should be able to provide guarantees like this while also allowing training - model per tenant. I think the reality is they are doing multi-tenant models which means they have no way guarantee your data won't be leaked unless they disable training altogether.
I'm imagining some corporate scenario where Coca Cola or Pepsi are purposefully training models on poisoned information so they can out each other for trying to use AI services like ChatGPT to glean information about competitors via brute force querying of some type
But, can they provide a comprehensive dump of all data it was trained on that we can examine? Otherwise my company may end up using IP that belongs to someone else.
I believe the API (chat completions) has been private for a while now. ChatGPT (the chat application run by OpenAI on their chat models) has continued to be used for training… I believe this is why it’s such a bargain for consumers. This announcement allows businesses to let employees use ChatGPT with fewer data privacy concerns.
Note that turning 'privacy' on is buried in the UI; turning it off again requires just a single click.
Such dark patterns, plus their involvement in crypto, their shoddy treatment of paying users, their security incidents... make it harder for me to feel good about OpenAI spearheading the introduction of (real) AI into the world today.
That's an important correction. Thanks, I got a bit carried away with the comment. There's enough hearsay on the internet, and I don't want to contribute.
While we're at it, another exaggeration I made is "security incidents"; in fact, I am only aware of one.
> Such dark patterns, plus their involvement in crypto, their shoddy treatment of paying users, their security incidents... make it harder for me to feel good about OpenAI spearheading the introduction of (real) AI into the world today.
Interesting. My opinion is it is a great product that works well for me, I don't find my treatment as a paying user shoddy, and their security incident gives me pause.
> I don't find my treatment as a paying user shoddy
I have never payed for a service with worse uptime in my life than ChatGPT. Why? So that OpenAI could ramp up their user-base of both free and paying users. They knowingly took on far more paying users than they could properly support for months.
There are justifications for the terrible uptime that are perfectly valid, but in the end, a customer-focused company would have issued a refund to the paying customers for the months during which they were shafted by OpenAI prioritizing growth.
That doesn't mean OpenAI isn't terrific in some ways. They're also lousy in others. With so many tech companies, the lousy aspects grow in significance as the years pass. OpenAI, because of all the reasons in my parent comment, is not off to a great start, imo.
Worse (IMO) than that is the fact that when the privacy mode is turned on, you can't access your previously saved conversations nor will it save anything you do while it's enabled. Really shitty behaviour.
If you turn off history and training, you as the user can no longer see your history, and OpenAI won't train with your data. But can customer prompts and company data still be resold to data brokers?
This assumes the portion of the enterprise fee related to this feature is only large enough to cover the cost of losing potential training data, which is an absurd assumption that can't be proven and has no basis in economic theory.
Companies are trying to maximize profit; they are not trying to minimize costs so they can continue to do you favors.
These arguments creep up frequently on HN: "This company is doing X to their customers to offset their costs." No, they are a company, and they are trying to make money.
The fact that companies want to maximise profits doesn't prove the point you think it does.
Nobody is arguing that there's an exact matching of value to the company between 1 user giving OpenAI permission to use their chat history for future training and 1 user paying $20/month. But based on your simplistic view, no company would ever offer a free tier because it's not directly maximising revenue.
It's very obvious that getting lots of real-world examples of users using ChatGPT is beneficial for multiple reasons - from using in future training runs (or fine tuning), to analysing what users what to use LLMs for, to analysing what areas ChatGPT is currently performing well or badly in, etc.
So it's not about blankly and entirely "offsetting costs", it's about the fact that both money into their bank account and this sort of data into their databases are both beneficial to the long-term profitability of the company even though only one of them is direct and instant revenue.
Before ChatGPT was released for the world to use, OpenAI were even paying people (both employees and not) to have lots of conversations with it for them to analyse. The exact same logic that justified that justifies allowing some users to pay some or all of the fee for the service in data permissions rather than money.
I'm speaking from experience making these sorts of business decisions, and to a company like OpenAI this is just basic common sense.
I'm going to see if the word "Enterprise" convinces my organization to allow us to use ChatGPT with our actual codebase, which is currently against our rules.
ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise.
I think this will have a solid product-market-fit. The product (ChatGPT) was ready but not enterprise. Now it is. They will get a lot of sales leads.
Just the SOC2 bit will generate revenue… If your organization is SOC2 compliant, using other services that are also compliant is a whole lot easier than risking having your SOC2 auditor spend hours digging into their terms and policies.
- GPT-4 API: has max 8K tokens (for most users atm)
- GPT-3.5 API: has max 16K tokens
I'd consider the 32K GPT-4 context the most valuable feature. In my opinion OpenAI shouldn't discriminate in favor of large enterprises. It should be equaly available to normal (paying) customers.
Thanks. I'm currently using the API models (even GPT 3.5 16K) for things that require a larger context. So much for "Priority access to new features and improvements" as advertised with Plus.
Seems like they are quite startled with LLama 2 and Code Llama, and how its rapid adoption is accelerating the AI race to zero. Why have this when Llama 2 and Code Llama exists and brings the cost close to $0?
This sound like a huge waste of money for something that should just be completely on-device or self-hosted if you don't trust cloud-based AI models like ChatGPT Enterprise and want it all private and low cost.
But either way, Meta seems to be already at the finish line in this race and there is more to AI than the LLM hype.
Maybe, but that's why things like ollama.ai are trying to fill the gap. It's simple, and you don't need all of the heavy weight enterprise crap if nothing ever leaves your system.
Less technical companies throw money at problems to solve them. Like mine, sadly... Even if it takes a small amount of effort, companies will throw money for zero effort.
Zero execution risk, rather than zero effort. There’s always a 10% chance that implementation goes on forever and spending some money eliminates that risk.
agreed, and I can't wait for gpt4 to have great competition in terms of ease, price and performance. I was responding to this
> something that should just be completely on-device or self-hosted if you don't trust cloud-based AI models like ChatGPT Enterprise and want it all private and low cost
I get the self-host part, but if you had a dedicated machine would the ram be an issue? Can you run it on a machine with like 128GB of ram or the GPU equivalent?
> This sound like a huge waste of money for something that should just be completely on-device or self-hosted
I can imagine this argument being made repeatedly over the past several decades whenever anyone makes a decision to use any paid cloud service. There is a value in self-hosting FOSS services and managing it in house and there is a value in letting someone else manage it for you. Ultimately it depends on the business use case and how much effort / risk you are willing to handle.
If you could offer stable 70B llama API at half the price of ChatGPT API I would pay for it. I know HN likes to believe everything is close to $0, but it is hardly the case.
Or maybe they got urged to offset more operational costs - and I would believe that companies already paying for Microsoft things Wille happily pay for OpenAI in addition just to be safe.
Sell the same product under two brands at the same time?
Optimal business strategy. Makes it look like there's more competition, and changes the decision from "do we use ChatGPT" to "Which GPT vendor do we use?"
Microsoft has a stake in OpenAI but they don't have a controlling interest in it. What they got instead was exclusive access to the models on Azure. So they benefit from OpenAIs success but they benefit more from their own success in the space and in a way they are competitors.
It’s pretty interesting to see both companies copying each other. Bing Chat has GPT4 with Vision, Chat History and some other goodies whereas OpenAI extends towards B2B.
Microsoft is primarily a mid-market company. They definitely sell to enterprise as well, but what makes Microsoft truly great is their ability to sell at enormous scale through a vast network of partners to every SMB in the world.
OpenAI is a tiny company, relative to Microsoft. They can’t afford to build a giant partner network. At best, they can offer a forum-supported set of products for the little guys and a richly supported enterprise suite. But the middle market will be Microsoft’s to own, as they always do.
At my work we've been using Bing Enterprise for a brief while now, which it states is based on ChatGPT4 and has the same promise of keeping data private. From what I've seen, Bing Chat is superior to ChatGPT4 in many respects.
Interesting, but I am a bit disappointed that this release doesn't include fine-tuning on an enterprise corpus of documents. This only looks like a slightly more convenient and privacy-friendly version of ChatGPT. Or am I missing something?
At the bottom, in their coming soon section: "Customization: Securely extend ChatGPT’s knowledge with your company data by connecting the applications you already use"
I saw it, but it only mentions "applications" (whatever that means) and not bare documents. Does this mean companies might be able to upload, say, PDFs, and fine-tune the model on that?
Yeah, I'll be curious to see what it means by this. Could be a few things, I think:
- Codebases
- Documents (by way of connection to your Box/SharePoint/GSuite account)
- Knowledgebases (I'm thinking of something like a Notion here)
I'm really looking forward to seeing what they come up with here, as I think this is a truly killer use case that will push LLMs into mainstream enterprise usage. My company uses Notion and has an enormous amount of information on there. If I could ask it things like "Which customer is integrated with tool X" (we keep a record of this on the customer page in Notion) and get a correct response, that would be immensely helpful to me. Similar with connecting a support person to a knowledgebase of answers that becomes incredibly easy to search.
Pretty unlikely. Generally you don't use fine-tuning for bare documents. You use retrieval augmented generation, which usually involves vector similarity search.
Fine-tuning isn't great at learning knowledge. It's good at adopting tone or format. For example, a chirpy helper bot, or a bot that outputs specifically formatted JSON.
I also doubt they're going to have a great system for fine-tuning. Successful fine-tuning requires some thought into what the data looks like (bare docs won't work), at which point you have technical people working on the project anyway.
Their future connection system will probably be in the format of API prompts to request data from an enterprise system using their existing function fine-tuning feature. They tried this already with plugins, and they didn't work very well. Maybe they'll come up with a better system. Generally this works better if you write your own simple API for it to interface with which does a lot of the heavy lifting to interface with the actual enterprise systems, so the AI doesn't output garbled API requests so much.
When I first started working with GPT I was disappointed in this. I thought like the previous commentor that I could fine tune by adding documents and it would add it to the "knowledge" of GPT. Instead I had to do what you suggest is vector similarity search, and add the relevant text to the prompt.
I do think an open line of research is some way for users to just add arbitrary docs in an easy way to the LLM.
Yes, this would definitely be a game changer for almost all companies. Considering how huge the market is, I guess it's pretty difficult to do, or it would be done already.
I certainly don't expect a nice drag-and-drop interface to put my Office files and then ask questions about it coming in 2023. Maybe 2024?
That would be the absolute game-changer. Something with the "intelligence" of GPT-4, but it knows the contents of all your stuff - your documents, project tracker, emails, calendar, etc.
Unfortunately even if we do get this, I expect there will be significant ecosystem lock-in. Like, I imagine Microsoft is aiming for something like this, but you'd need to use all their stuff.
There are great tools that do this already in a support-multiple-ecosystems kind of way! I'm actually the CEO of one of those tools: Credal.ai - which lets you point-and-click connect accounts like O365, Google Workspace, Slack, Confluence, e.t.c, and then you can use OpenAI, Anthropic etc to chat/slack/teams/build apps drawing on that contextual knowledge: all in a SOC 2 compliant way. It does use a Retrieval-Augmented-Generation approach (rather than fine tuning), but the core reason for that is just that this tends to actually offer better results for end users than fine tuning on the corpus of documents anyway!
Link: https://www.credal.ai/
What are the limitations on adding documents to your system? Your website doesn't particularly highlight that feature set, which it probably should if you support it!
Thanks for the feedback! Going to make some changes to the website to reflect that later today! Right now we support connecting Google Doc, Google Sheet, PDFs from Google Drive, Slack channel, or Confluence space. O365, Notion and a couple other sources integrations are in beta. We don't technically have restrictions on volume, the biggest customers we have have around 100 GB of data with us total. If you were trying to connect a terrabyte worth of data, that might be a conversation about pricing! :)
Your pricing seems to eliminate some use cases, including mine.
Rather than wanting to import N documents per month, I would want to import M documents all at once, then use that set of documents until at some future time I want to import another batch of K documents (probably a lot smaller than M) or just one document once in a while.
By limiting it to a fixed amount of documents per month, it eliminates all the applications where you need to import a complete corpus before the service is useful.
Totally agree. retrieval augmented generation is still the preferred way to give the LLM more knowledge. Fine-tuning is mostly useful for adapting the base model for another task. I wrote about this in a recent blog post: https://vectara.com/fine-tuning-vs-grounded-generation/.
Anyone knows how this new capability works in terms of where the model inference be done? Would it still be at the OpenAI side or is this going to be at the customer side?
After using RAG with pgvector for the last few months with temperature 0, it's been pretty great with very little hallucination.
The small context window is the limiting factor.
In principle, I don't see the difference between a bunch of fine-tuned prompts along the lines of "here is another context section: <~4k-n tokens of the corpus>", which is the same as what it looks like in a RAG prompt anyway.
Maybe the distinction of whether it is for "tone" or "context" is based on the role of the given prompts and not restricted by the fine-tuning process itself?
In theory, fine-tuning it on ~100k tokens like that would allow for better inference, even with the RAG prompt that includes a few sections from the same corpus. It would prevent issues where the vector search results are too thin despite their high similarity. E.g. picking out one or two sections of a book which is actually really long.
For example, I've seen some folks use arbitrary chunking of tokens in batches of 1k or so as an easy config for implementation, but that totally breaks the semantic meaning of longer paragraphs, and those paragraphs might not come back grouped together from the vector search. My approach there has been manual curation of sections allowing variations from 50 to 3k tokens to get the chunks to be more natural. It has worked well but I could still see having the whole corpus fine-tuned as extra insurance against losing context.
It's not impossible that fine-tuning would also help RAG. but it's certainly not guaranteed and hard to control. Fine-tuning essentially changes the weights of the model, and might result in other, potentially negative outcome, like loss of other knowledge of capabilities of the resulting fine-tuned LLM.
Other considerations:
(A) would you fine-tune daily? weekly? as data changes?
(B) Cost and availability of GPUs (there's a current shortage)
My experience is that RAG is the way to go, at least right now.
But you have to make sure your retrieval engine work optimally: getting the very most relevant pieces of text from your data: (1) using a good chunking strategy that's better than arbitrary 1K or 2K chars (2) using a good embedding model (3) Using hybrid search, and a few other things like that.
Certainly the availability of longer sequence models is a big help
On the other hand, there was a lot of knowledge in those documents that effectively got lost - while the relevant tech is still underpinning half the world. For example: DCOM/COM+.
Azure-hosted GPT already lets you "upload your own documents" in their playground; it seems to be similar to how ChatGPT GPT-4 Code Interpreter handles file uploads.
You don't fine-tune on a corpus of documents to give the model knowledge, you use retrieval.
They support uploading documents to it for that via that code interpreter, and they're adding connectors to applications where the documents live, not sure what more you're expecting.
Yes, but what if they are very large documents that exceed the maximum context size, say, a 200-page PDF? In that case won't you be forced to do some form of fine-tuning, in order to avoid a very slow/computationally expensive on-the-fly retrieval?
Fine-tuning the LLM in the way that you're mentioning is not even an option: as a practical rule fine-tuning the LLM will let you do style transfer, but you knowledge recall won't improve (there are edge cases, but none apply to using ChatGPT)
That being said you can use fine tuning to improve retrieval, which indirectly improves recall. You can do things like fine tune the model you're getting embeddings from, fine tune the LLM to craft queries that better match a domain specific format, etc.
It won't replace the expensive on-the-fly retrieval but it will let you be more accurate in your replies.
Also retrieval can be infinitely faster than inference depending on the domain. In well defined domains you can run old school full text search and leverage the LLMs skill at crafting well thought out queries. In that case that runs at the speed of your I/O.
We have >200 page PDFs at https://docalysis.com/ and there's on-the-fly retrieval. It's not more computationally expensive than something like searching one's inbox (I'd image you have more than 200 pages worth of emails in your inbox).
Interesting that they offer GPT-4 32k in the enterprise version while only giving very few people API access to it. I guess we'll see that more often in the future.
They do. If you use the entire context, a single request is like 30 cents. Very easy to rack up 10s of dollars very very quickly. Not an explanation/excuse, but additional context (no pun intended).
Is it really so hard for companies to provide a price range for Enterprise plan publicly on the pricing page?
Why can't I, as an individual, have the same features of an Enterprise plan?
What is the logic behind this practice other than profit maximization?
I'm willing to pay more to have unlimited high-speed GTP4 and Longer inputs with 32k token context.
EDIT: since I'm getting a lot of replies. Genuine question: how should I move to get a reasonable price as an individual for unlimited high-speed gpt4 and longer token context?
I'm sure that's the correct answer, and that their very best was invested in analyzing the max profit strategy (as they should).
What I'm wondering if it means that the minimal price they can offer the service with at profit, is likely to be too steep for anyone like me, who interpret "talk to us" as the online equivalent of showing him the door. The other explanation I see is that there's not many in the camp of users who react to "talk to us" button by closing the tab instead of a deal, but I find that implausible.
> I'm wondering if it means that the minimal price they can offer the service with at profit, is likely to be too steep for anyone like me
I think the answer to that is "no". The problem is that they don't want to reveal the minimal price to their initial round of customers.
There are two basic ways you can think about pricing: cost-plus and value-minus. We programmers tend to like the former because it's clear, rational, and simple. But if you've got something of unknown value and want to maximize income, the latter can be much more appealing.
The "talk to sales" approach means they're going to enter into a process where they find the people who can get the most out of the service. They're going to try to figure out the total value added by the service. And they'll negotiate down from there. (Or possibly up; somebody once said the goal of Oracle was to take all your money for their server software, and then another $50k/year for support.)
Eventually, once they've figured out the value landscape, they'll probably come back for users like you, creating a commoditized product offering that's limited in ways that you don't care about but high-dollar customers can't live without. That will be closer to cost-plus. For example, note Github's pricing, which varies by more than 10x depending on what they think they can squeeze you for: https://github.com/pricing
Because it’s often heavily negotiated. At the enterprise level, custom requests are entertained, and teams can spend weeks or months building bespoke features for a single client. So yeah, it’s kinda fundamentally impossible.
Oh yes. I'm willing to bet that it involves things like progressive discounts on # of tokens or # of seats, etc etc. This is just how you get access to the big bucks.
> What is the logic behind this practice other than profit maximization?
Why would it be something else than profit maximization? It's a for-profit company, with stakeholders who want to maximize the possible profits coming from it, seems simple enough to grok, especially for users on S̵t̵a̵r̵t̵u̵p̵ ̵N̵e̵w̵s̵ Hacker News.
Because the truth is, each deal is custom packaged and priced for each enterprise. It's all negotiated pricing. Call it "value pricing" or whatever you want, prices are set at the tolerance level of each company. A price-sensitive enterprise might pay $50k while another company won't blink at $80k for essentially the same services.
If it goes to the direction of Microsoft Copilot, then you can check out the recent announcement. Microsoft currently estimates that 30/user/month is a good list price to get „ChatGPT with all your business context“ to your employees.
This seems really cool, but I guess most companies in the EU won't dare to use this due to GDPR concerns and instead will opt-in for the Azure version, where you can choose to use GPT-models that are hosted in Azure's EU servers.
I used to be super hyped about ChatGPT and the productivity they could deliver. However the large amount of persistent bugs in their interface has convinced me otherwise.
Yes. If you re-open a chat from a while ago, typing a new message will typically result in an error causing me to having to start a new conversation. Happens both in the browser and on mobile for months now. Because no one else has it then it’s probably because I disabled history. It’s still a bug.
Another bug that I‘m having for weeks now is that pressing Stop Responding will indeed stop the stream but it will also cause a block on any new messages for about a minute. This one used to work just fine but started to fail a few weeks ago.
"Unlimited access to advanced data analysis (formerly known as Code Interpreter)"
Code Interpreter was a pretty bad name (not exactly meaningful to anyone who hasn't studied computer science), but what's the new name? "advanced data analysis" isn't a name, it's a feature in a bullet point.
Also I'd heard anecdotally on the internet (Ethan Mollick's twitter I think) that 'code interpreter' was better than GPT 4 even for tasks that weren't code interpretation. Like it was more like GPT 4.5. Maybe it was an experimental preview and only enterprises are allowed to use it now. I never had access anyway.
I still have access in my $20/m non-Enterprise Pro account, though it has indeed just updated its name from Code Interpreter to Advanced Data Analysis. I haven't personally noticed it being any better than standard GPT4 even for generation of code that can't be run by it (ie non-Python code).
I've been using it heavily for the last week - hopefully it doesn't become enterprise only... it's very convenient to pass it some examples and generate and test functions.
And it does seem "better" than standard 4 for normal tasks
There is just nothing out there, open source or otherwise, that even comes close to GPT-4. Therefore, the value proposition is clear, this is providing you with access to the SOTA, 2x faster, without restrictions.
I can actually see this saving a lot of time for employees (1-10% maybe?), so the price is most likely calculated on that and a few other factors. I think most big orgs will eat it like cake.
That was quick. Companies offering APIs end up competing with their developer base that built end-user facing products. Another example is Twilio that offers retail-ready products now such as Studio, prebuilt Flex, etc.
We (like many other companies) have deployed an internal UI[1] that integrates with our SSO and makes calls via the OpenAI API, which has better data privacy terms than the ChatGPT website.
We'd be potentially very interested in an official internal-facing ChatGPT, with a caution that the economics of the consumption-based model have so far been advantageous to us, rather than a flat fee per user per month. I can say that based on current usage, we are not spending anywhere close to $20 per user per month across all of our staff.
Interesting that they're still centered around Chat as the interface, with https://flowch.ai (our product) we're building it much more around projects and reports, which we think is often more suitable for businesses.
We're going after some of these use cases:
Want a daily email with all the latest news from your custom data source (or Google) for a topic?
How about parsing custom data and scores from your datasets using prompts with all the complicated bits handled for you, then downloading as a simple CSV?
Or even simply bulk generating content, such as generating Press Releases from your documents?
All easy with FlowChai :)
I think there's room for many different options in this space, whether that be Personal, Small Business or Enterprise.
Some feedback (it's clear you're just pitching FlowChai, but that's ok its HN):
I quick scrolled through your webpage and had no idea what it was. Extremely text heavy, and generic images that didn't communicate anything. I wanted to know what the product LOOKED like, especially as you're describing the difference between it and the chat interface of OpenAI.
I think you updated your comment (or I missed it) with the link to a "report" - it looked just like the output of one of the text bubbles except it had some (source) links (which I think Bing does as well)? It didn't seem all that different to me.
Very fair, we have demo videos, guides etc planned for the next week or so. As it's a tool that can do many things it's hard to describe. Still a lot to do :)
In terms of what makes the report different from Bing: this could be any source of data: scraped from the web, search, API upload, file upload etc, so there's a lot more power there. Also, it's not just one off reports, there's automation there which would allow for example a weekly report on the latest papers on GPT4 (or whatever you're interested in).
Doesn't seem to be in a usable state yet. I created an account and realised there's not actually any features to play with yet. I gave a URL for scheduled reports but I cannot configure anything about them.
You didn't offer me any way to delete my account and remove the email address I saved in your system. I hope you don't start sending me emails, after not giving me an ability to delete the account
If it was transferred, the /microsoft link would have redirected to it. Instead, it's the git commits re-uploaded to another repo - so the commits are the same but it didn't transfer past issues, discussions or PRs https://github.com/matijagrcic/azurechatgpt/pulls?q=
Seemed like a great project. Hope to see it come back!
There are some great open-source projects in this space – not quite the same – many are focused on local LLMs like Llama2 or Code Llama which was released last week:
The UI is relatively mature, as it predates llama. It includes upstream llama.cpp PRs, integrated AI horde support, lots of sampling tuning knobs, easy gpu/cpu offloading, and its basically dependency free.
yes. at first glance it looks like a windows app but it's actually very portable. it has some parameters for gpu offloading and extended context size that just work. it exposes an api endpoint. i use it on a workstation to serve larger llms locally and like the performance and ease of use.
Ollama is very neat. Given how compressible the models are is there any work being done on using them in some kind of compressed format other than reducing the word size?
There are different levels of quantization available for different models (if that's what you mean :). E.g. here are the versions available for Llama 2: https://ollama.ai/library/llama2/tags which go down to 2-bit quantization (which surprisingly still happens to work reasonably well).
No, what I mean is that it seems as though there is quite a bit of sparseness to the matrix and I was wondering if that can somehow be used to further shrink the model, quantization is another effect (it leaves the shape of the various elements as they are but reduces their bit-depth).
Ah, gotcha! I thought you probably meant something else. I've been wondering this too, and it's something I've been meaning to look at.
On a related note it doesn't seem like many local runners are leveraging techniques like PagedAttention yet (see https://vllm.ai/) which is inspired by operating system memory paging to reduce memory requirements for LLMs.
It's not quite what you mentioned, but it might have a similar effect! Would love to know if you've seen other methods that might help reduce memory requirements.. it's one of the largest resource bottlenecks to running LLMs right now!
That's a clever one, I had not seen that yet, thank you.
The hint for me is that the models compress so well, that suggests the information content is much lower than the size of the uncompressed model indicates which is a good reason to investigate which parts of the model are so compressible and why. I haven't looked at the raw data of these models but maybe I'll give it a shot. Sometimes you can learn a lot about the structure (built in or emergent) of data just by staring at the dumps.
That's quite interesting. I hadn't thought of sparsity in the weights as a way to compress models, although this is an obvious opportunity in retrospect! I started doing some digging and found https://github.com/SqueezeAILab/SqueezeLLM, although I'm sure there's newer work on this idea.
All activity stopped a couple of weeks ago. It was extremely active and had close to 5 thousand stars/watch events before it was removed/made private. Unfortunately I never got around to indexing the code. You can find the insights at https://devboard.gitsense.com/microsoft/azurechatgpt
It looks like your account has been using HN primarily (in fact exclusively) for promotion for quite some time. I'm not sure how we didn't notice this before but someone finally complained, and they're right: you can't use HN this way. Note this, from https://news.ycombinator.com/newsguidelines.html: Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
Normally we ban accounts that do nothing but promote their own links, but as you've been an HN member for years, I'm not going to ban you, but please do stop doing this! We want people to use HN to read and post things that they personally find intellectually interesting—not just to promote something.
If I go back far enough (a couple hundred comments are so), it's clear that you used to use HN in the intended spirit, so this should be fairly easy to fix.
Based on past discussion, my guess is it was removed because the name and description were wildly misleading. People starred it because it was a repo published by Microsoft called "azurechatgpt", but all it contained was a sample frontend UI for a chat bot which could talk to the OpenAI API.
It's helpful to think of OpenAI as Microsoft's R&D lab for AI without the political and regulatory burdens that MSR has to abide by. Through that lens, it's really all just the same thing. There is no endgame for OpenAI that doesn't involve being a part of Microsoft.
IIRC it is impossible for OpenAI to become part of Microsoft since the incorporation documents of the for-profit bit of OpenAI prevent anyone from having a majority of the shares (except the non-profit foundation, of course).
can you briefly explain the vulnerability here. I'm having difficulty understanding as all I see is him recalling already previously recorded chat history of his own session. thanks.
The vulnerability is the automatic insecure rendering of image markdown. One way to trigger it is with an indirect prompt injection payload. The scenario is that the user analyzes some text/data, which contains malicious instructions. The owner of the text doesn't have access to the chat history (it's just some random text somewhere), it could be a comment on a webpage, text inside a pdf file, copy/pasting, or even instructions hidden inside an image the user analyzes and sends to the LLM. You can find many examples of indirect prompt injections on my blog (e.g. analyzing YouTube transcripts,...). Just yesterday I put up a video explaining the various TTPs (and also fixes companies put in place): https://www.youtube.com/watch?v=L_1plTXF-FE Hope that helps.
Clicked on ChatGPT / Compare ChatGPT plans / Enterprise ...
> Contact sales
Oops. Scary.
I'm missing the Teams plan: transparent pricing with a common admin console for our team. Yes, fast GPT-4, 32k context, templates, API credits... they're all very nice-to-haves, but just the common company console would be crucial for onboarding and scaling-up our team and needs without the big-bang "enter-pricey" stuff.
Any "Contact sales" stuff has just been an instant "no" at any company I've ever worked at, because that always means that the numbers are always too high to include in the budget unless it's a directive coming down directly from the top.
That’s where directives for enterprise contracts usually come from. I’m sure they won’t even talk to anyone not willing to pay $100k+ per year. Salesforce’s AI Cloud starts at $365k a year.
> I’m sure they won’t even talk to anyone not willing to pay $100k+ per year.
Wouldn't surprise me. We had a vendor whose product we had used at relatively reasonable rates for multiple years suddenly have a pricing model change. It would have seen our cost go from $10k/yr to $100k/yr. As a small nonprofit we tried to engage them in any sort of negotiation but the response was essentially a curt "too bad." Luckily a different vendor with a similar product was more than happy to take our $10k.
We have a model that I see a lot of others do, even if they don't publicize. We have free, OSS, and cheap SaaS tiers fine for many of our academic users, and when a small group really wants the full enterprise version, we generally offer a heavily discounted pricing model to make that affordable too. The only exception here is when it is a true enterprise sale like a shared resource for a large number of users, and we'd still have to think there too.
The reason is it keeps their low budgets and thus their ROI in alignment, which is why this is pretty normal. So again, I'd recommend asking and just clarifying your are a NGO/EDU. No 100% guarantee, but should be common.
I don't know if that's a smart way to bypass pesky hidden information negotiations and suss out other party's upper bound or a really stupid way to do business...
Of course, I'd rather have 20K per customer
But an initial quote of 300K would likely lead to many instant rejections rather than engaging in negotiation, right? That's why I say it feels like a stupid practice, even though it could pay off really well if some company accepts outright (With the caveat that I've never been near this kind of business deal, so I'm just going off of common sense)
It depends on the company. It's kind of like a menu item that says "Market Price". You know it's not going to be cheap. You don't know until you ask if the price is less than the value offered.
It makes it impossible to access for bootstrapping, at least for people who have budget constraints. Which is just reality, it's a scarce resource and I appreciate what they have made available so far inexpensively.
But hopefully it does give a little more motivation to all of the other great work going on with open models to keep trying to catch up.
The jump to enterprise pricing suggests that they have enormous enterprise demand and don’t need to bother with SMB “teams” pricing. I suspect OpenAI is leaving the SMB part up to Microsoft to figure out, since that’s Microsoft’s forte through their enormous partner program.
How is this different from using GPT api on Azure? I thought that allowed you to keep you data corpus/documents private as well, ie not get sent to their servers for training
531 comments
[ 2.7 ms ] story [ 277 ms ] threadHaving conversations saved to go back to like in the default setting on Pro, that's disabled when a Pro user turns on the privacy setting, is another big difference.
" We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. "
So i have one primary profile logged in normally and a separate tab which i turn off chat history.
So now I get best of both worlds
They want chat history and no training on the same conversation.
Chrome extension is a no go.
What’s with the argumentative tone? Do you think that the replier doesn’t know this?
If we had to waste that much time re-inventing the loaf of bread, and then making sure that my neighbors didn’t decide to throw some raisins in my loaf, that we never get around to figuring out the next best thing: slicing it.
That's great. But can customer prompts and company data be resold to data brokers?
Companies want to dump all their Excels in it and get insights that no human could produce in any reasonable amount of time.
Companies want to dump a zillion help desk tickets into and gain meaningful insights from it.
Companies want to dump all their Sharepoints and Wikis into it that currently nobody can even find or manage, and finally have functioning knowledge search.
You absolutely want a privately trained company model.
What you want is to get an existing model to search a well built index of your data and use that information to reason about things. That way you also always have entirely up to date data.
People aren't missing the use cases you describe, they're disagreeing as to how to achieve those.
>>Companies want to dump a zillion help desk tickets into and gain meaningful insights from it.
>>Companies want to dump all their Sharepoints and Wikis into it that currently nobody can even find or manage, and finally have functioning knowledge search.
Mature organizations already have solutions for all of these things. If you can't mine your own data competently, you've got bigger problems than not having AI doing it for you. It means you don't have humans who understand what's going on. AI is not the answer to everything.
ChatGPT only works as well as it does because it’s been trained on a corpus of “internet accepted” answers. It can’t fucking reason about raw data. It’s a language model.
Such dark patterns, plus their involvement in crypto, their shoddy treatment of paying users, their security incidents... make it harder for me to feel good about OpenAI spearheading the introduction of (real) AI into the world today.
While we're at it, another exaggeration I made is "security incidents"; in fact, I am only aware of one.
Interesting. My opinion is it is a great product that works well for me, I don't find my treatment as a paying user shoddy, and their security incident gives me pause.
There are justifications for the terrible uptime that are perfectly valid, but in the end, a customer-focused company would have issued a refund to the paying customers for the months during which they were shafted by OpenAI prioritizing growth.
That doesn't mean OpenAI isn't terrific in some ways. They're also lousy in others. With so many tech companies, the lousy aspects grow in significance as the years pass. OpenAI, because of all the reasons in my parent comment, is not off to a great start, imo.
This is borderline extortion, and it's hilarious to witness as someone who doesn't have a dog in this fight.
Companies are trying to maximize profit; they are not trying to minimize costs so they can continue to do you favors.
These arguments creep up frequently on HN: "This company is doing X to their customers to offset their costs." No, they are a company, and they are trying to make money.
Nobody is arguing that there's an exact matching of value to the company between 1 user giving OpenAI permission to use their chat history for future training and 1 user paying $20/month. But based on your simplistic view, no company would ever offer a free tier because it's not directly maximising revenue.
It's very obvious that getting lots of real-world examples of users using ChatGPT is beneficial for multiple reasons - from using in future training runs (or fine tuning), to analysing what users what to use LLMs for, to analysing what areas ChatGPT is currently performing well or badly in, etc.
So it's not about blankly and entirely "offsetting costs", it's about the fact that both money into their bank account and this sort of data into their databases are both beneficial to the long-term profitability of the company even though only one of them is direct and instant revenue.
Before ChatGPT was released for the world to use, OpenAI were even paying people (both employees and not) to have lots of conversations with it for them to analyse. The exact same logic that justified that justifies allowing some users to pay some or all of the fee for the service in data permissions rather than money.
I'm speaking from experience making these sorts of business decisions, and to a company like OpenAI this is just basic common sense.
I don't think they're removing all instances of your company from their existing data sources, which would make sense to call "borderline extortion".
That's great. But can customer prompts and company data be resold to data brokers?
ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise.
I think this will have a solid product-market-fit. The product (ChatGPT) was ready but not enterprise. Now it is. They will get a lot of sales leads.
- GPT-4 (ChatGPT Plus): has max 4K tokens ?
- GPT-4 API: has max 8K tokens (for most users atm)
- GPT-3.5 API: has max 16K tokens
I'd consider the 32K GPT-4 context the most valuable feature. In my opinion OpenAI shouldn't discriminate in favor of large enterprises. It should be equaly available to normal (paying) customers.
This sound like a huge waste of money for something that should just be completely on-device or self-hosted if you don't trust cloud-based AI models like ChatGPT Enterprise and want it all private and low cost.
But either way, Meta seems to be already at the finish line in this race and there is more to AI than the LLM hype.
From the very beginning everyone knew data privacy & security would be one of the main issues for corporations.
> something that should just be completely on-device or self-hosted if you don't trust cloud-based AI models like ChatGPT Enterprise and want it all private and low cost
I can imagine this argument being made repeatedly over the past several decades whenever anyone makes a decision to use any paid cloud service. There is a value in self-hosting FOSS services and managing it in house and there is a value in letting someone else manage it for you. Ultimately it depends on the business use case and how much effort / risk you are willing to handle.
Optimal business strategy. Makes it look like there's more competition, and changes the decision from "do we use ChatGPT" to "Which GPT vendor do we use?"
The exclusive license refers to having access to the models, which is probably why they were able to finetune Bing AI.
OpenAI is a tiny company, relative to Microsoft. They can’t afford to build a giant partner network. At best, they can offer a forum-supported set of products for the little guys and a richly supported enterprise suite. But the middle market will be Microsoft’s to own, as they always do.
- Codebases
- Documents (by way of connection to your Box/SharePoint/GSuite account)
- Knowledgebases (I'm thinking of something like a Notion here)
I'm really looking forward to seeing what they come up with here, as I think this is a truly killer use case that will push LLMs into mainstream enterprise usage. My company uses Notion and has an enormous amount of information on there. If I could ask it things like "Which customer is integrated with tool X" (we keep a record of this on the customer page in Notion) and get a correct response, that would be immensely helpful to me. Similar with connecting a support person to a knowledgebase of answers that becomes incredibly easy to search.
Fine-tuning isn't great at learning knowledge. It's good at adopting tone or format. For example, a chirpy helper bot, or a bot that outputs specifically formatted JSON.
I also doubt they're going to have a great system for fine-tuning. Successful fine-tuning requires some thought into what the data looks like (bare docs won't work), at which point you have technical people working on the project anyway.
Their future connection system will probably be in the format of API prompts to request data from an enterprise system using their existing function fine-tuning feature. They tried this already with plugins, and they didn't work very well. Maybe they'll come up with a better system. Generally this works better if you write your own simple API for it to interface with which does a lot of the heavy lifting to interface with the actual enterprise systems, so the AI doesn't output garbled API requests so much.
I do think an open line of research is some way for users to just add arbitrary docs in an easy way to the LLM.
I certainly don't expect a nice drag-and-drop interface to put my Office files and then ask questions about it coming in 2023. Maybe 2024?
Unfortunately even if we do get this, I expect there will be significant ecosystem lock-in. Like, I imagine Microsoft is aiming for something like this, but you'd need to use all their stuff.
Rather than wanting to import N documents per month, I would want to import M documents all at once, then use that set of documents until at some future time I want to import another batch of K documents (probably a lot smaller than M) or just one document once in a while.
By limiting it to a fixed amount of documents per month, it eliminates all the applications where you need to import a complete corpus before the service is useful.
Anyone knows how this new capability works in terms of where the model inference be done? Would it still be at the OpenAI side or is this going to be at the customer side?
After using RAG with pgvector for the last few months with temperature 0, it's been pretty great with very little hallucination.
The small context window is the limiting factor.
In principle, I don't see the difference between a bunch of fine-tuned prompts along the lines of "here is another context section: <~4k-n tokens of the corpus>", which is the same as what it looks like in a RAG prompt anyway.
Maybe the distinction of whether it is for "tone" or "context" is based on the role of the given prompts and not restricted by the fine-tuning process itself?
In theory, fine-tuning it on ~100k tokens like that would allow for better inference, even with the RAG prompt that includes a few sections from the same corpus. It would prevent issues where the vector search results are too thin despite their high similarity. E.g. picking out one or two sections of a book which is actually really long.
For example, I've seen some folks use arbitrary chunking of tokens in batches of 1k or so as an easy config for implementation, but that totally breaks the semantic meaning of longer paragraphs, and those paragraphs might not come back grouped together from the vector search. My approach there has been manual curation of sections allowing variations from 50 to 3k tokens to get the chunks to be more natural. It has worked well but I could still see having the whole corpus fine-tuned as extra insurance against losing context.
Other considerations: (A) would you fine-tune daily? weekly? as data changes? (B) Cost and availability of GPUs (there's a current shortage)
My experience is that RAG is the way to go, at least right now.
But you have to make sure your retrieval engine work optimally: getting the very most relevant pieces of text from your data: (1) using a good chunking strategy that's better than arbitrary 1K or 2K chars (2) using a good embedding model (3) Using hybrid search, and a few other things like that.
Certainly the availability of longer sequence models is a big help
Sharing this relevant discussion from LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7101638...
They support uploading documents to it for that via that code interpreter, and they're adding connectors to applications where the documents live, not sure what more you're expecting.
Edit: spelling
That being said you can use fine tuning to improve retrieval, which indirectly improves recall. You can do things like fine tune the model you're getting embeddings from, fine tune the LLM to craft queries that better match a domain specific format, etc.
It won't replace the expensive on-the-fly retrieval but it will let you be more accurate in your replies.
Also retrieval can be infinitely faster than inference depending on the domain. In well defined domains you can run old school full text search and leverage the LLMs skill at crafting well thought out queries. In that case that runs at the speed of your I/O.
Why can't I, as an individual, have the same features of an Enterprise plan?
What is the logic behind this practice other than profit maximization?
I'm willing to pay more to have unlimited high-speed GTP4 and Longer inputs with 32k token context.
EDIT: since I'm getting a lot of replies. Genuine question: how should I move to get a reasonable price as an individual for unlimited high-speed gpt4 and longer token context?
That’s a real big “other than”…
I don't know, but I can't imagine any other logic.
Maybe posting the price they'd like to charge would scare away almost all interested parties.
Maybe the price they charge you depends more on how much money they think you have than it does on a market's "decision" on what the product is worth.
How much more? That’s the question that “talk to us” enterprise pricing is trying to answer.
What I'm wondering if it means that the minimal price they can offer the service with at profit, is likely to be too steep for anyone like me, who interpret "talk to us" as the online equivalent of showing him the door. The other explanation I see is that there's not many in the camp of users who react to "talk to us" button by closing the tab instead of a deal, but I find that implausible.
I think the answer to that is "no". The problem is that they don't want to reveal the minimal price to their initial round of customers.
There are two basic ways you can think about pricing: cost-plus and value-minus. We programmers tend to like the former because it's clear, rational, and simple. But if you've got something of unknown value and want to maximize income, the latter can be much more appealing.
The "talk to sales" approach means they're going to enter into a process where they find the people who can get the most out of the service. They're going to try to figure out the total value added by the service. And they'll negotiate down from there. (Or possibly up; somebody once said the goal of Oracle was to take all your money for their server software, and then another $50k/year for support.)
Eventually, once they've figured out the value landscape, they'll probably come back for users like you, creating a commoditized product offering that's limited in ways that you don't care about but high-dollar customers can't live without. That will be closer to cost-plus. For example, note Github's pricing, which varies by more than 10x depending on what they think they can squeeze you for: https://github.com/pricing
Why would it be something else than profit maximization? It's a for-profit company, with stakeholders who want to maximize the possible profits coming from it, seems simple enough to grok, especially for users on S̵t̵a̵r̵t̵u̵p̵ ̵N̵e̵w̵s̵ Hacker News.
Are you aware what the entire point of a business is?
https://blogs.microsoft.com/blog/2023/07/18/furthering-our-a...
Surprising it didn't make the initial launch announcement though.
Another bug that I‘m having for weeks now is that pressing Stop Responding will indeed stop the stream but it will also cause a block on any new messages for about a minute. This one used to work just fine but started to fail a few weeks ago.
Code Interpreter was a pretty bad name (not exactly meaningful to anyone who hasn't studied computer science), but what's the new name? "advanced data analysis" isn't a name, it's a feature in a bullet point.
And it does seem "better" than standard 4 for normal tasks
What a terrible name! They should have asked ChatGPT for suggestions.
I can actually see this saving a lot of time for employees (1-10% maybe?), so the price is most likely calculated on that and a few other factors. I think most big orgs will eat it like cake.
We'd be potentially very interested in an official internal-facing ChatGPT, with a caution that the economics of the consumption-based model have so far been advantageous to us, rather than a flat fee per user per month. I can say that based on current usage, we are not spending anywhere close to $20 per user per month across all of our staff.
[1] We used this: https://github.com/dotneet/smart-chatbot-ui
We're going after some of these use cases:
Want a daily email with all the latest news from your custom data source (or Google) for a topic? How about parsing custom data and scores from your datasets using prompts with all the complicated bits handled for you, then downloading as a simple CSV? Or even simply bulk generating content, such as generating Press Releases from your documents?
All easy with FlowChai :)
I think there's room for many different options in this space, whether that be Personal, Small Business or Enterprise.
Here's an example of automatically scraped arXiv papers on GPT4, turned into a report (with sources) generated by GPT4: https://flowch.ai/shared/6107d220-4e19-4bdc-a566-e84e8a60565...
I quick scrolled through your webpage and had no idea what it was. Extremely text heavy, and generic images that didn't communicate anything. I wanted to know what the product LOOKED like, especially as you're describing the difference between it and the chat interface of OpenAI.
I think you updated your comment (or I missed it) with the link to a "report" - it looked just like the output of one of the text bubbles except it had some (source) links (which I think Bing does as well)? It didn't seem all that different to me.
In terms of what makes the report different from Bing: this could be any source of data: scraped from the web, search, API upload, file upload etc, so there's a lot more power there. Also, it's not just one off reports, there's automation there which would allow for example a weekly report on the latest papers on GPT4 (or whatever you're interested in).
You didn't offer me any way to delete my account and remove the email address I saved in your system. I hope you don't start sending me emails, after not giving me an ability to delete the account
https://github.com/microsoft/azurechatgpt
Past discussion:
https://news.ycombinator.com/item?id=37112741
https://github.com/matijagrcic/azurechatgpt
There are some great open-source projects in this space – not quite the same – many are focused on local LLMs like Llama2 or Code Llama which was released last week:
- https://github.com/jmorganca/ollama (download & run LLMs locally - I'm a maintainer)
- https://github.com/simonw/llm (access LLMs from the cli - cloud and local)
- https://github.com/oobabooga/text-generation-webui (a web ui w/ different backends)
- https://github.com/ggerganov/llama.cpp (fast local LLM runner)
- https://github.com/go-skynet/LocalAI (has an openai-compatible api)
- https://github.com/trypromptly/LLMStack (build and run apps locally with LocalAI support - I'm a maintainer)
The UI is relatively mature, as it predates llama. It includes upstream llama.cpp PRs, integrated AI horde support, lots of sampling tuning knobs, easy gpu/cpu offloading, and its basically dependency free.
GPTQ has also been merged into Transformers library recently ( https://huggingface.co/blog/gptq-integration ).
GGML quantization format used by llama.cpp also supports (8,6,5,4,3, and 2 bit quantization).
On a related note it doesn't seem like many local runners are leveraging techniques like PagedAttention yet (see https://vllm.ai/) which is inspired by operating system memory paging to reduce memory requirements for LLMs.
It's not quite what you mentioned, but it might have a similar effect! Would love to know if you've seen other methods that might help reduce memory requirements.. it's one of the largest resource bottlenecks to running LLMs right now!
The hint for me is that the models compress so well, that suggests the information content is much lower than the size of the uncompressed model indicates which is a good reason to investigate which parts of the model are so compressible and why. I haven't looked at the raw data of these models but maybe I'll give it a shot. Sometimes you can learn a lot about the structure (built in or emergent) of data just by staring at the dumps.
Full Disclosure: This is my tool
Normally we ban accounts that do nothing but promote their own links, but as you've been an HN member for years, I'm not going to ban you, but please do stop doing this! We want people to use HN to read and post things that they personally find intellectually interesting—not just to promote something.
If I go back far enough (a couple hundred comments are so), it's clear that you used to use HN in the intended spirit, so this should be fairly easy to fix.
https://embracethered.com/blog/posts/2023/chatgpt-webpilot-d...
Seems like a no-go for companies if an attacker can steal stuff.
> Contact sales
Oops. Scary.
I'm missing the Teams plan: transparent pricing with a common admin console for our team. Yes, fast GPT-4, 32k context, templates, API credits... they're all very nice-to-haves, but just the common company console would be crucial for onboarding and scaling-up our team and needs without the big-bang "enter-pricey" stuff.
Wouldn't surprise me. We had a vendor whose product we had used at relatively reasonable rates for multiple years suddenly have a pricing model change. It would have seen our cost go from $10k/yr to $100k/yr. As a small nonprofit we tried to engage them in any sort of negotiation but the response was essentially a curt "too bad." Luckily a different vendor with a similar product was more than happy to take our $10k.
We have a model that I see a lot of others do, even if they don't publicize. We have free, OSS, and cheap SaaS tiers fine for many of our academic users, and when a small group really wants the full enterprise version, we generally offer a heavily discounted pricing model to make that affordable too. The only exception here is when it is a true enterprise sale like a shared resource for a large number of users, and we'd still have to think there too.
The reason is it keeps their low budgets and thus their ROI in alignment, which is why this is pretty normal. So again, I'd recommend asking and just clarifying your are a NGO/EDU. No 100% guarantee, but should be common.
A lot of value in some SaaS apps is in the initial investment it took to build it, not in the cost to host a customer's assets.
If the runtime costs of a new customer are negligible, would you rather have 0K or 20K?
If dodgy pricing/sales tactics didn't work then Oracle would be bankrupt instead of a 300 billion dollar company.
But hopefully it does give a little more motivation to all of the other great work going on with open models to keep trying to catch up.