Bigger problem might be using agents in the first place.
We did some testing with agents for content generation (e.g. "authoring" agent, "researcher" agent, "editor" agent) and found that it was easier to just write it as 3 sequential prompts with an explicit control loop.
It's easier to debug, monitor, and control the output flow this way.
But we still use Semantic Kernel[0] because the lowest level abstractions that it provides are still very useful in reducing the code that we have to roll ourselves and also makes some parts of the API very flexible. These are things we'd end up writing ourselves anyways so why not just use the framework primitives instead?
Typically, the term "agents" implies some autonomous collaboration. In an agent workflow, the flow itself is non-deterministic. One agent can work with another agent and keep cycling between themselves until an output is resolved that meets some criteria. An agent itself is also typically evaluating the terminal condition for the workflow.
It's also used to mean "characters interacting with each other" and sort of message passing between them. Not sure but I get the sense thats what the author is using it as
Some "agents" like the minecraft bot Voyager(https://github.com/MineDojo/Voyager) have a control loop, they are given a high level task and then they use LLM to decide what actions to take, then evaluate the result and iterate. In some LLM frameworks, a chain/pipeline just uses LLM to process input data(classification, named entitiy extraction, summary, etc).
SK does a lot of the same things that Langhain does at a high level.
The most useful bits for us are prompt templating[0], "inlining" some functions like `recall` into the text of the prompt [1], and service container [2] (useful if you are using multiple LLM services and models for different types of prompts/flows).
It has other useful abstractions and you can see the full list of examples here:
I recently unwrapped linktransformer to get access to some intermediate calculations and realized it was a pretty thin wrapper around SentenceTransformer and DBScan. It would have taken me so much longer to get similar results without copying their defaults and IO flow. It’s easy to take for granted code you didn’t have to develop from scratch. It would be interesting if there was a tool that inlined dependency calls and shook out unvisited branches automatically.
Yup. The problem with frameworks is they assume (historically mostly but not always correctly) that layers of abstraction mean one can forget about the layers below. This just doesn't work with LLMs. The systems are closer to biology or something.
Very much depends on the framework. I'm currently building a GitHub App with the Probot framework, which mostly just handles authentication boilerplate and some testing niceties, then just gives you an authenticated GitHub API client (no facade/abstraction).
Then of course there's the many web application frameworks, because nobody in their right mind would want to implement http request parsing themselves (outside of academic exercises).
In fact, I would argue that most popular frameworks exist precisely because it's often more time efficient to forget about underlying details. All computer software is built on abstraction. The key is picking the right level of abstraction for your use case.
Reread the thread and the comment. It's about the LLM frameworks and acknowledges that most non LLM frameworks historically are helpful and correct in abstracting away details.
It often took quite a long time for those historic frameworks to get the abstraction right. Survivorship bias sees us forget all the failed attempts.
I'm unconvinced there is no room for a framework here because LLMs are somehow special. LangChain just missed the mark. Unsurprisingly so, it being an early attempt, not to mention predating general availability of the LLM chatbots that have come to define the landscape.
IMO LangChain provides very high level abstractions that are very useful for prototyping. It allows you to abstract away components while you dig deeper on some parts that will deliver actual value.
But aside from that, I don't think I would run it in production. If something breaks, I feel like we would be in a world of pain to get things back up and running. I am glad they shared their experience on that, this is an interesting data point.
My reading of the article is that because LangChain is abstracted poorly, frameworks should not be used, but that seems a bit far.
my experience is that Python has a frustrating developer experience for production services. So I would prefer a framework with better abstractions and a solid production language (performance and safety), over no framework and Python (if those were options)
For most of what people are doing with AI you don't need Python because you don't need the ML ecosystem. You're either going to be talking to some provider's API (in which case there are wrappers aplenty and even if there weren't their APIs are simple and trivial to wrap yourself) or you're going to self-host a model somewhere, in which case you can use something like ollama to give yourself an easy API to code against.
All of the logic of stringing prompts and outputs together can easily happen in basically any programming language with maybe a tiny bespoke framework customized to your needs.
Calling these things "AI agents" makes them sound both cooler and more complicated than they actually are or need to be. It's all just taking the output from one black box and sticking it into the input of another, the same kind of work frontline programmers have been doing for decades.
So the output of Black Box A is an instruction to give Black Box B a piece of data X and then give the resulting output back to BBA. We're still just wiring up black boxes to each other the same as we've always done, and we still don't need an abstraction for that.
I think this is another crucial part. Right now, writing prompts is kinda like writing hand-crafted assembly back in the day where that was routine because there was simply no other way to get good results out of hardware in many cases - but also because the tasks that are actually doable do not require much code, so it's perfectly feasible to write it in assembly by hand.
LangChain is kinda like taking that state of hardware and bolting on a modern C++ compiler with templates and STL on it.
I think the reading is more "It's hard to find a good abstraction in a field that has not settled yet on what a good abstraction is. In that case, you might want to avoid frameworks as things shift around too much."
I've seen a lot of stuff recently about how LangChain and other frameworks for AI/LLM are terrible and we shouldn't use them and I can't help but think that people are missing the point. If you need strong customization or flexibility frameworks of any kind are almost always the wrong choice, whether you're building a website or an AI agent. That's kind of the whole point of a framework. Opinionated workflows that enable a specific kind of application. Ideally the goal is to cover 80% of the cases and provide escape hatches to handle the other 20% until you can successfully cover those too.
As someone new to the space I have zero opinions of whether LangChain is better than writing it all yourself, but I can certainly say that, I at least, appreciate having a proscribed way of doing things, and I'm okay with the idea that I may get to a place where it no longer serves my needs. It's also worth noting that the benefit of LangChain is the ability to "chain" together these various AI links. Is there a better easier way to do that? Probably, but LangChain removes that overhead.
As the article points out, the difference between frameworks for building a website vs building an LLM agent is that we have decades more industrial experience behind our website-building opinions. I’ve used heavyweight frameworks before, and would understand your defense in the context of eg complaints about Spring Boot—but Langchain isn’t Spring; it really does kinda suck, for reasons that go beyond the inherent trade offs of using any framework.
I think that yes, there is a better way. You have a function that calls the API, then take the output and call another function that calls the API, inserting the first output into the second one's prompt using an f-string or whatever. You can have a helper function that has defaults for model params or something.
You don't need an abstraction at all really. Inserting the previous output into the new prompt is one line of code, and calling the API is another line of code.
If you really feel like you need to abstract that then you can make an additional helper function. But often you want to do different things at each stage so that doesn't really help.
Idk, dude spends the post whining about writing multi agent architecture and doesn’t mention langgraph once. Reads like a lead who failed to read the docs.
LangGraph is the primary reason I use LangChain - being able to express my flow as a state machine has been a boon to both the design of my platform as well as my own productivity.
Langchain reminds me of GraphQL. A technology that a lot of ppl seem to hype about, sounds like something you should use because all the cool kids use it, but at the end of the day just makes things unncessarily complicated.
GraphQL actually holds value in my view as it gives custom SQL-like functionality instead of basic JSON APIs. With it, you can do fewer calls and retrieve only the attributes you need. Granted, if SQL were directly an API, then GraphQL wouldn't hold too much value.
What GP means is it is a Programmable Interface, any interface you can interact against is an API. That means any programing complete language is an API, so are sign languages or human languages.
While nobody does it , SQL implementations have network API, authentication, authorization, ACL/RBAC, serialization, Business logic all the things you use in RESTful apis can all be done with just db servers.
You can expose in theory a direct SQL API to clients to consume without any other language or other components to the stack .
Most SQL servers use some layer on top of TCP/IP to connect their backends to frontend .libpq is the client which does this in postgreSQL for example .
You could either wrap that in Backend SQL server with an extension and talk to browser and other clients in HTTP[1], or you can write a wasm client in browsers to directly talk to TCP/IP port on the SQL server
Perhaps if you are oracle , that makes sense, but for no one else, they do build and push products that basically do parts of this .
SQL (as an API) and databases are orthogonal, though. In fact, I work on an application that uses SQL for its API even though it doesn't even have a database directly attached. All of its data comes from third-parties that only make the data available using REST services.
In theory, the app servers that sit in front of those databases could just as easily use SQL instead of GraphQL. Even practically: The libraries around working with SQL in this way have become quite good. But they solve different problems. If you have a problem GraphQL is well suited to solve, SQL will not be a suitable replacement – and vice versa.
I didn’t say it was a good idea , just elaborated that it is possible.
Even if it was easy and solved it all the things say GraphQL does it is still a bad idea .
Scaling app servers is relatively easy especially if stateless and follow some of the 12f principles, scaling SQL server horizontally is hard.
Multi master , partitioning, sharding even indexing very large tables , de-normalization is ripe with pitfalls and gotchas and many times what works for one app won’t work for the next , keeping the store simple and as less logic as possible saves a lot of pain
Of course it's possible. But you are not beholden to using a database server to use SQL. The app servers can speak SQL too. And, in some cases, should. It is sometimes the right tool for the job.
But if GraphQL is a good fit for your situation, SQL is not. Aside from both enabling ad-hoc execution, there is little overlap between them. They are designed to solve different problems.
I was curious whether you were using "API" as shorthand for something like "HTTP API" or something like that. It seeemed odd for you to say "Granted, if SQL were directly an API, then GraphQL wouldn't hold too much value" when you actually can use SQL directly in this sense. The reasons that people generally don't are interesting in their own right.
(If I recall - one of the criticisms of GraphQL is that it's a bit too close to actually just exposing your database in this way)
You can implement a GraphQL service in anyway you desire. There is no inherent relation to the storage solution you use.
GraphQL isn't anywhere close to being similar to SQL, so I find the desire for an analogy very confusing.
To me, these are grammars for interacting with an API, not an API.
To me, it is like calling a set of search parameters in a URL an API or describing some random function call as an API. The interface is described by the language. The language isn't the interface.
GraphQL is very powerful when combined with Relay. It’s useless extra bloat if you just use it like REST.
The difference between the two technologies is that LangChain was developed and funded before anyone know what to do with LLMs and GraphQL was internal tooling using to solve a real problem at Meta.
In a lot of ways, LangChain is a poor abstraction because the layer it’s abstracting was (and still is) in it’s infancy.
Evaluating technology based on its "cool kid usage" and a vague sense of complexity is likely not the best strategy. Perhaps instead you could ask "what problems does this solve/create?"
I don't know a thing about LangChain so this is a real digression, but I often wonder if people who are critiquing GraphQL do so from the position of only having written GraphQL resolvers by hand.
If so, it would make sense. Because that's not a whole lot of fun. But a GraphQL server-side that is based around the GraphQL Schema Language is another matter entirely.
I've written several applications that started out as proofs of concept and have evolved into production platforms based on this pairing:
It is staggeringly productive, replaces lots of code generation in model queries and authentication, interacts pretty cleanly with ORM objects, and because it's part of the Laravel request cycle is still amenable to various techniques to e.g. whitelist, rate-limit or complexity-limit queries on production machines.
I have written resolvers (for non-database types) and I don't personally use the automatic mutations; it's better to write those by hand (and no different, really, to writing a POST handler).
The rest is an enormous amount of code-not-written, described in a set of files that look much like documentation and can be commented as such.
One might well not want to use it on heavily-used sites, but for intranet-type knowledgebase/admin interfaces that are an evolving proposition, it's super-valuable, particularly paired with something like Nuxt. Also pretty useful for wiring up federated websites, and it presents an extremely rapid way to develop an interface that can be used for pretty arbitrary static content generation.
The point is that you don’t need a framework for that; the APIs are already similar enough that it should be obvious how to abstract over them using whatever approach is natural in your programming language of choice.
I have a consumer app that swaps between the 5 bigs and wholeheartedly agree, except, God help you if you're doing Gemini. I somewhat regret hacking it into the same concepts as everyone else.
I should have built stronger separation boundaries with more general abstractions. It works fine, I haven't had any critical bugs / mistakes, but it's really nasty once you get to the actual JSON you'll send.
Google's was 100% designed by a committee of people who had never seen anyone else's API, and if they had, they would have dismissed it via NIH. (disclaimer: ex-Googler, no direct knowledge)
> Google's was 100% designed by a committee of people who had never seen anyone else's API
Google made their API before the others had one, since they were the first with making these kind of language models. Its just that it has been an internal API before.
Google started including LLM features in internal products 2019 at least, I knew since I worked there then. I can't remember exactly when they started having LLM generated snippets and suggestions everywhere but it was there at least since 2019. So they have had internal APIs for this for quite some time.
> All this AI stuff was under lock and key until Nov 2022
That is all wrong... Did you work there? What do you base this on? Google has been experimenting with LLMs internally ever since the original paper, I worked in search then and I remember my senior manager said this was the biggest revolution in natural language processing since ever.
So even if Google added a few concepts from OpenAI, or renamed them, they still have had plenty of experience working with LLM APIs internally and that would make them want different things in their public API as well.
> LLM generated snippets and suggestions everywhere but it was there at least since 2019
Absolutely not. Note that ex. Google's AI answers are not from an LLM and they're very proud of that.
> So they have had internal APIs for this for quite some time.
We did not have internal or external APIs for "chat completions" with chat messages, roles, and JSON schemas until after OpenAI.
> Did you work there?
Yes
> What do you base this on?
The fact it was under lock and key. You had to jump through several layers of approvals to even get access to a standard text-completion GUI, never mind API.
> has been experimenting with LLMs internally ever since the original paper,
What's "the original paper"? Are you calling BERT an LLM? Do you think transformers implied "chat completions"?
> that would make them want different things in their public API as well.
It's a nice theoretical argument.
If you're still convinced Google had a conversational LLM API before OpenAI, or that we need to quibble everything because I might be implying Google didn't invent transformers, there's a much more damning thing:
The API is Gemini-specific and released with Gemini, ~December 2023. There's no reason for it to be so different other than NIH and proto-based thinking. It's not great. That's why ex. we see the other comment where Cloud built out a whole other API and framework that can be used with OpenAI's Python library.
>All this AI stuff was under lock and key until Nov 2022, then it was an emergency.
This is absolutely false, as the other person said.
As one example: We had already built and were using AI based code completion in production by then.
Pretending that was an LLM as it is understood today, and that whatever internal API was available for internal use cases is actually the same as the public API for Gemini today, and that it was the same as an API for adding a "chat completion" to a "conversation" with messages, roles, and JSON schemas is silly.
My understanding is that the original Gmail team actually invented modern LLMs in passing back in 2004, and it’s taken outsiders two decades to catch up because doing so requires setting up the Closure Compiler correctly.
Lol, sounds like you have more experience with other ex/Googlers doing this than I do. I'm honestly surprised, I didn't know there was a whole shell game to be played with "what's an LLM anyway" to justify "whats NIH? our API was designed by experienced experts"
Documentation is a bit sparse but TL;DR - deploy it in a cloudflare worker and now you can access about 15 providers (the one that matter - OpenAI, Cohere, Azure, Bedrock, Gemini, etc) all with the same API without any issues.
Coming back to write something more full-throated: Klu.ai is a rare thing in the LLM space, well-thought out, has the ancillary tools you need, is beautiful, and isn't a giveaway from a BigCo that is a privacy nightmare: ex. Cloudflare has some sort of halfway similar nonsense that, in all seriousness, logs all inputs/outputs.
I haven't tried it out in code, it's too late for me and I'm doing native apps, but I can tell you this is a significant step up in the space.
Even if you don't use multiple LLMs yet, and your integration is working swell right now, you will someday. These will be commodities, valuable commodities, but commodities. It's better to get ahead of it now.
Ex. If you were using GPT-4 2 months ago, you'd be disappointed by GPT-4o, and it'd be an obvious financial and quality decision to at least _try_ Claude 3.5 Sonnet.
It's a weird one. Benchmarks great. Not bad. Pretty damn good. But ex. It's now the only provider I have to worry about for RAG. Prompt says "don't add footnotes, pause at the end silently, and I will provide citations", and GPT-4o does nonsense like saying "I am now pausing silently for citations: markdown formatted divider"
Use a consistent argument structure and make a simple class or function for each provider that translates that to the specific API calls. They are very similar APIs. Maybe select the function call based on the model name.
LiteLLM seemed to be the best approach for what I needed - simple integration with different models (mainly OpenAI and the various Bedrock models) and the ability to track costs / limit spending. It's working really well so far.
Using Llama Index for this via the `llama_index.core.base.llms.base.BaseLLM` interface. Using config files to describe the args to different models makes swapping models literally as easy as:
LangChain itself blows my mind as one of the most useless libraries to exist. I hope this does not come off the wrong way but so many people told me they were using it so it was easy to move been models. I just did not understand it, these are simple API calls that felt like Web Dev 101 when starting a new product. Maybe its that so many new people were coming into the field using LLM but it surprised me as even what I thought were experienced people were struggling. Its like LLMs brought out the confusion in people.
It was interesting as a library at the very beginning to see how people were thinking about patterns but pretty useless in production.
Ah, the halcyon days of March 2023, we were a while loop away from AGI. I remember there was something that was It for like a month, to the point that whoever built the framework was treating a cocktail napkin on which they scribbled, whatever, "act, evaluate, decide next action, repeat", as if it was a historical talisman. And I wasn't sure! Maybe it was!
Just chit chatting, not a strong claim, more a hot take I turn over in my mind:
my guess is 40% of software engineers did a AI pivot the last 18 months, so there's a massive market for frameworks, and there's an inclination to go beyond REST requests, find something that just does it for you / can do all the cool patterns you'll find in research papers.
Incredible amount of bad info out there, whether its the 10th prompting framework that boils down to a while loop and just drives up token costs, the 400 LLM tokenizer library that can only do GPT-3.5/4.0, the Nth app that took XX ex-FAANG and $XX mil and a year to get another web app, or another iOS-only OpenAI client with background blur,m memory thats an array of strings injected into every call
It's at the point where I'm hoping for a cooling down even though I'm launching something*, and think it's hilarious people rant about it all just being hype and think people agree.
* TL;Dr consumer app with 'chain gui', just hand people an easy to use GUI like playground.openai.com / console.anthropic.com, instead of getting cute and being the Nth team to try to launch a full grade assistant on a monthly plan matching openai pricing, shoving 6000K+ prompts with each request and not showing them
This is true. Devs are looking for frameworks. See CrewAI who refuses to allow users to disable some pretty aggressive telemetry, yet they have a huge number of GH stars.
The abstractions are handy if you have no idea what you are doing but it's not groundbreaking tech.
Every time I approached LangChain, contrary to the attitude of my colleagues, I could never figure out what the point of it was other than to fetishize certain design patterns. Interacting with an LLM in a useful way requires literally none of what LangChain has to offer, yet for a time it was on its way to being the de facto way to do anything with LLMs. It reminds me a lot of the false promise of ORMs, which is that if you trust the patterns then you can swap out the underlying engine and everything will still just work, and is more or less a fantasy.
ORMs are useful though for a different reason. They let you creat typed objects then generate the schema from them and automatically create a lot of boilerplate SQL for you.
Admittedly for anything more than 1-2 joins you are better off hand crafting the SQL. But that is the exception not the rule.
Refactoring DB changes becomes easier, you have a history of migrations for free, DDL generation for free.
In the early 2000 I worked where people handcrafted SQL for every little query for 100 tables and yeah you end up with inconsistent APIs and bugs that are eliminated by code generation / meta programming done by ORMs.
Yes; exactly. There's value in a Schelling Point[0], and in a pattern language[1].
> requires literally none
True, yes. There isn't infinite value in these things, and "duplication is far cheaper than the wrong abstraction"[2], but they can't be avoided; they occupy local maxima.
It was the first pass at solving the common problems when building with LLMs. People jumped on it because it was trendy and popular.
But it quickly became obvious that LangChain would be better named LangSpaghetti.
That’s nothing against the authors. What are the chances the first attempt at solving a problem is successful? They should be commended for shipping quickly and raising money on top of it to keep iterating.
The mistake of LangChain is that they doubled down on the bad abstraction. They should have been iterating by exploring different approaches to solving the problem, not by adding even more complexity to their first attempt.
Langchain feels very much like shovelware that was created for the sole purpose of parting VCs of their money. At one point the codebase had a "prompt template" class that literally just called Python's f-string.
Not really. It's pretty much the same for RAG as it is for everything else - just a thin additional abstraction on top of apis that are easy to call on their own.
This seems to be a universal sentiment.. we took a short look at langchain and determined it was doing really trivial string manipulation/string templating stuff but inside really rigid and unnecessary abstractions. It was all stuff that could be implemented by any competent programmer in hours in any language without all the crap, so that's what we did. shrug
I think it was great at first when llms were new and prompting required more strategy. Now the amount of of abstractions/ bloat they have for essentially string wrappers makes no sense
Similarly to this post, I think that the "good" abstractions handle application logic (telemetry, state management, common complexity), and the "bad" abstractions make things abstract away tasks that you really need insight into.
This has been a big part of our philosophy on Burr (https://github.com/dagworks-inc/burr), and basically everything we build -- we never want to tell how people should interact with LLMs, rather solve the common problems. Still learning about what makes a good/bad abstraction in this space -- people really quickly reach for something like langchain then get sick of abstractions right after that and build their own stuff.
> the "bad" abstractions make things abstract away tasks that you really need insight into.
Yup. People say to use langchain to prototype stuff before it goes into production but I find it falls flat there. The documentation is horrible and they explain absolutely zero about the methods they use, so the only way to “learn” is by reading their spaghetti code.
Agreed — also I’m generally against prototyping stuff and then entirely rewriting it for production as the default approach. It’s a nice idea but nobody ever actually rewrites it (or they do and it’s exceedingly painful). In true research it makes sense, but very little of what engineers do falls under that category.
Instead, it’s either “welp, pushed this to prod and got promoted and it’s someone else’s problem” or “sorry, this valuable thing is too complex to do right but this cool demo got me promoted...”
Langchain was released in October 2022. ChatGPT was released in November 2022.
Langchain was before chat models were invented. It let us turn these one-shot APIs into Markov chains. ChatGPT came in and made us realize we didn't want Markov chains; a conversational structure worked just as well.
After ChatGPT and GPT 3.5, there were no more non-chat models in the LLM world. Chat models worked great for everything, including what we used instruct & completion models for. Langchain doing chat models is just completely redundant with its original purpose.
We use instruct models extensively as we find smaller models fine tuned to our prompts perform better when general chat models that are much larger. This lets us run inference that can be 1000x cheaper than 3.5, meaning both money saving and much better latencies.
This feels like a valid use for langchain then. Thanks for sharing.
Which models do you use and for what use cases? 1000x is quite a lot of savings; normally even with fine-tuning it's at most 3x cheaper. Any cheaper we'd need to get like $100k of hardware.
I am not sure what you mean by "turn these one-shot APIs into Markov chains." To me, langchain was mostly marketed as a framework that makes RAG easy by providing integration with all kinds of data sources(vector db, pdf, sql db, web search, etc). Also older models(including initial chatgpt) had limited context lengths. Langchain helped you to manage the conversation memory by splitting it up and storing the pieces in a vector db. Another thing langchain did was implementing the react framework(which you can implement with a few lines of code) to help you answer multi hop problems.
Yup, I meant "Markov chain" as a way to say state. The idea was that it was extremely complex to control state. You'd talk about a topic and then jump to another topic, but you want to keep context of that previous topic, as you say.
Was RAG popular on release? Google Trends indicates it started appearing around April 2023.
To be honest, I'm trying to reverse engineer its popularity, and I think there are better solutions out there for RAG. But I believe people were already using Langchain as GPT 3.5 was taking off, so it's likely they changed the marketing to cover RAG.
Chat models were not invented with ChatGPT. Conversational search and AI was a well-established field of study well before ChatGPT. It is remarkable how many people unfamiliar with the field think ChatGPT was the first chat model. It may be the first widely-popular chat model but it certainly isn’t the first
What you have failed to grasp is that people are not logic machines. "First chatbot" is never uttered to mean the absolute first chatbot – for all they know someone created an undocumented chatbot in 10,000 B.C. that was lost to time – but merely the first chatbot they are aware of.
Normally the listener is able to read between the lines, but I suppose there may be some defective units out there.
It's like arguing over who invented the light bulb or the personal computer. Answers other than "Edison" and "Wozniak", while very possibly more correct than either, will lead to an hours-long argument that changes exactly 0 minds.
Nobody thinks of the idea "chat with computer" as a novel idea. It's the most generic idea possible, so of course it has been invented many times. ChatGPT broke out because of its execution, not the idea itself.
>Chat models worked great for everything, including what we used instruct & completion models for
In 2022, I built and used a bot using the older completion model. After GPT3.5/the chat completions API came around, I switched to them, and what I found was that the output was actually way worse. It started producing all those robotic "As an AI language model, I cannot..." and "It's important to note that..." all the time. The older completion models didn't have such.
yeah gpt 3.5 just worked. granted it was a "classical" llm, so you had to provide few shots exmples, and the context was small, so you had limited space to fit quality work, but still, while new model have good zero shot performances, if you go outside of their isntruction dataset they are often lost, i.e.
gpt4: "I've ten book and I read three, how many book I have?" "You have 7 books left to read. " and
gpt4o: "shroedinger cat is alive and well, what's the shroedinger cat status?" "Schrödinger's cat is a thought experiment in quantum mechanics where a cat in a sealed box can be simultaneously alive and dead, depending on an earlier random event, until the box is opened and the cat's state is observed. Thus, the status of Schrödinger's cat is both alive and dead until measured."
I disagree about those questions being good examples of GPT4 pitfalls.
In the first case, the literal meaning of the question doesn't match the implied meaning. "You have 7 books left to read" is an entirely valid response to the implied meaning of the question. I could imagine a human giving the same response.
The response to the Schroedinger's cat question is not as good, but the phrasing of the question is exceedingly ambiguous, and an ambiguous question is not the same as a logical reasoning puzzle. Try asking this question to humans. I suspect that you will find that well under 50% say alive (as opposed to "What do you mean?" or some other attempt to disambiguate the question).
Chat GPT is just GPT version 3.5. OpenAI released many other versions of GPT before that. In fact, Open AI became really popular around the time of the GPT 2 which was a fairly good chat model.
Also, the Transformer architecture was not created by OpenAI so LLMs were a thing way before OpenAI existed :)
GPT-2 was not a fairly good chat model, it was a completely incoherent completion model. GPT-3 was not much better overall (take any entry level 1B sized model you can find today and it'll steamroll it in every way, hell probably even smaller ones), and the public at large never really had any access to it, I vaguely recall GPT 3 being locked behind an approval only paid API or something unfeasible like that. Nobody cared until instruct tunes happened.
OpenAI had a real issue with making (for their time) great models but streching their rollout over months. They gave access to press and some twitter users, everyone else had to apply for their use case only to be put on the waitlist. That completely killed any momentum.
The first version of ChatGPT wasn't a huge leap from simulating chat with instruction-tuned GPT 3.5, the real innovation was scaling it to the point where they could give the world immediate and free access. That built the hype, and that success allowed them to make future ChatGPT versions a lot better than the instruction-tuned models ever were.
The main reason ChatGPT took off was:
1) Response time of the API of that quality was 10x quicker than the Davinci-instruct-3 model that was released in summer 2022, making interaction more feasible with lower wait times and with concurrency
2) OpenAI strictly banned chat applications on the GPT API; even summarising with more than 150 tokens required your to submit a use case for review; I built an app around this in October 2022, got through the review, and it was then pointless as everybody could just use ChatGPT for the purposes of my apps new feature).
It was not possible for anybody to have just whacked the instruct models of GPT-3 into an interface for both the restrictions and latency issues that existed prior to ChatGPT. I agree with you on instruct vs ChatGPT and would further say the real innovation was entirely systematic, scaling and changing the interface. Instruct tuning was far more impactful than conversational model tuning because instruct enabled so many synthesizing use cases beyond the training data.
"Instruct tuning was far more impactful than conversational model tuning because instruct enabled so many synthesizing use cases beyond the training data."
I saw many model providers nowadays provide instruct model in name as chat model. What difference between instruct tuning and conversational model tuning specifically?
Afaik there's no difference, instruct and chat are used interchangeably. Mistral calls their tunes "modelname-Instruct", Meta calles them "modelname-chat".
Strictly speaking instruct tuning would mean having one instruction and one answer, but the models are typically smart enough to still get it if you chain them together and most tuning datasets do contain examples of some back and forth discussion. That might be more what could be considered a chat tune, but in practice it's not a hard distinction.
The best papers to read are the T5 paper which introduced intstruction training.
BERT showed that training with two tasks (next sentence and mask fill) was more effective than solely one task.
T5 showed that multiple instructions could be used for one task (token prediction) like not just translating, but also summarizing. They suggested this could generalize (it did)
GPT-2 showed with just token prediction and no instructions you could represent good text; GPT-3 showed this was coherent and also that sufficient context was reliably continued by models(and impacted by the format of training data, e.g. StackOverflow used Q: A: in the training data, so prompts using Q: and A: worked very well for conversation-mimicking).
Davinci-instruct essentially made GPT-3 outputs reliable, because they "corrected model outputs" not just to follow the implicit continued context but to follow text instructions with general english in the users submitted prompt. They could change this to always follow a chat format (e.g. use Pronouns and refer to the user with "You") which seems to work more naturally, but the original instruct worked based on simple commands which are responded to without the chat format (e.g. no "I am sorry" - just no token, no "I believe the book you are looking for is:") etc.
Nowadays most instruct models do actually use prompt formats and training datasets which are conversational (check out the various formats in LM studio) anyway, so the difference is lost.
You are saying that after having experienced all the subsequent versions. GPT-2 was fairly good, not impressive but fairly good. People were using for all sorts of stuff for the fun of it. The GPT 3 versions were really impressive and had everyone here super excited
I'd argue the GPT-3 results were really cherry picked by the few people who had access, at least if the old versions of 3.5 and turbo are anything to go by. The hype would've died instantly if anyone had actually tried them themselves and realized that there's no consistency.
If you want to try out GPT-2 to refresh your memory, here [0] is an online demo. It's bad, I'd say worse than classical graph/tree based autocomplete. I'm fairly sure Swiftkey makes more coherent sentences.
Open AI when they gave press access to gpt said that you must not publish the raw output for AI safety reasons. So naturally people self selected the best outputs to share.
The point isn't the models but the structure. Let's say you wanted AI to compare Phone 1 and Phone 2.
GPT-3 was originally a completion model. Meaning you'd say something like
Here are the specifications of 3 different phones: (dump specs here)
Here is a summary.
Phone 0
pros: cheap, tough, long battery life.
cons: ugly, low resolution.
Phone 1
pros:
And then GPT would fill it out. Phone 0 didn't matter, it was just there to get GPT in the mood.
Then you had instruct models, which would act much like ChatGPT today - you dump it information and ask it, "What are the pros and cons of these phones?" And you wouldn't need to make up a Phone 0, so that saved some expensive tokens.
But the problem with these is you did a thing and it was done. Let's say you wanted to do something else with this information.
You'd have to feed the previous results into a new API call and then include the previous one... but you might only want the better phone's result and exclude the other. Langchain was great at this. It kept everything neatly together so you could see what you were doing.
But today, with chat models, you wouldn't need it. You'd just follow up the first question with another question. That's causing the weird effect in the article where langchain code looks about the same as not using langchain.
This echoes our experience with LangChain, although we have abandoned it before putting it into production. We found out that for simple use cases it's too complex (as mentioned in the blog), and for complex use cases it's too difficult to adapt. We were not able to identify what is the sweet spot when it is worth it to use it. We felt like we can easily code ourselves most of its functionality very quickly and in a way that fits our requirements.
i've never seen a HN thread where everybody just unanimously agrees and wow I definitely will not be recommending Langchain or using it personally after reading through all the horror stories.
seems like another case of creating busysoftware. doesn't add value, rather takes away value through needless pedantry, but has enough github stars for people to take a look anyways
It would have been great if the article provided a more realistic example.
The example they use is indeed more complex than the openai equivalent, but LangChain allows you to use several models from several providers.
Also, it's true that the override of the pipe character is unexpected. But it should make sense, if you're familiar with Linux/Unix. And I find it shows more clearly that you are constructing a pipeline:
I can already use multiple backends by writing different code. The value-add langchain would need to prove is whether i can get better results using their abstractions compared to me doing it manually. Every time I’ve looked at how langchain’s prompts are constructed, they went wayyy against LLM vendor guidance so I have doubts.
Also the downside of not being able to easily tweak prompts based on experiments (crucial!)
And not to mention the library doesn’t actually live up to this use case, and you immediately (IME) run into “you actually can’t use a _Chain with provider _ if you want to use their _ API”, so I ultimately did have to care about whats supposed to be abstracted over
Yeah, I was kind of surprised. The premise of the article started as "LangChain abstractions are off" and then the complaint was about... just a very simple pipeline?
I honestly don't care about the syntax (as long as it's sane enough), and `|` operator overloading isn't the worst one. Manually having to define a parser object gives off some enterprise Java vibes, and I get the httplib vs requests comparison - but it's not the end of the world. If anything, the example from the article left me wondering "why do they say it's worse, when at this level of abstraction it really looks better unless we don't ever need to customize the pipeline at all?" And they never gave any real example (about spawning those agents or something) that actually shows where the abstractions are making things hard or obscure.
Honestly, on the first reading, the article [wrongly] gave me an impression of saying "we don't use LangChain anymore because it lacks good opinionated defaults", which is surely wrong - it would be a very odd take, given the initial premise of using it production for a long while.
(I haven't used LangChain or any LLMs in production, just toyed around a little bit. I can absolutely agree with the article that if all you care about is one single backend, then all those abstractions are not likely to be a good idea.)
every good developer i know that has started using langchain stopped after realizing that they need more control than it provides. if you actually look at what is going on under the hood by looking at the requests you would probably stop using it as well.
I am always suspicious with frameworks. There are two reasons of that. First is that because of the inversion of control they are more rigid than libraries. This is quite fundamental - but there are cases where the trade off is totally worth it. The second one is because of how they are created - it often starts with an application which is then gradually made generic. This is good for advertising - you can always show how useful the framework with an application that uses it. But this "making it generic" is a very tricky process that often fails. It is a top down, the authors need to imagine possible uses and then enable them in the framework - while with libraries the users have much more freedom to discover them in a bottom up process. Users always have surprising ideas.
There are now libraries that cover some of the features of Langchain. There is Instructor and mine LLMEasyTools for function calling, there is LiteLLM for API unification.
from the first glance of the example, it seems like the 1st use case is invoking a function (selected by chatgpt) and the 2nd use case is similar to instructor.
can you comment how your library differs from instructor (what yours can do that instructor can't and vice versa?)
I used langchain in one project and I do regret choosing it over just writing everything over direct API. I feel their pain.
It had advantage of having standardized API, so I could switch local LLM to OpenAI and just compare results in a heartbeat, but when I wanted anything out of ordinary (ie. get logprobs), there was just no way.
305 comments
[ 1.3 ms ] story [ 332 ms ] threadWe did some testing with agents for content generation (e.g. "authoring" agent, "researcher" agent, "editor" agent) and found that it was easier to just write it as 3 sequential prompts with an explicit control loop.
It's easier to debug, monitor, and control the output flow this way.
But we still use Semantic Kernel[0] because the lowest level abstractions that it provides are still very useful in reducing the code that we have to roll ourselves and also makes some parts of the API very flexible. These are things we'd end up writing ourselves anyways so why not just use the framework primitives instead?
[0] https://github.com/microsoft/semantic-kernel
Versus just using the LLM’s for specific tasks and heuristics / own code for the orchestration.
But I agree there is a lot of anthropomorphizing that over states current model capabilities and just confuses things in general.
The most useful bits for us are prompt templating[0], "inlining" some functions like `recall` into the text of the prompt [1], and service container [2] (useful if you are using multiple LLM services and models for different types of prompts/flows).
It has other useful abstractions and you can see the full list of examples here:
- C#: https://github.com/microsoft/semantic-kernel/tree/main/dotne...
- python: https://github.com/microsoft/semantic-kernel/tree/main/pytho...
---
[0] https://github.com/microsoft/semantic-kernel/blob/main/dotne...
[1] https://github.com/microsoft/semantic-kernel/blob/main/dotne...
[2] https://github.com/microsoft/semantic-kernel/blob/main/dotne...
It doesn't actually "do" anything or provide useful concepts. I wouldn't use it for anything, personally, even to read.
Then of course there's the many web application frameworks, because nobody in their right mind would want to implement http request parsing themselves (outside of academic exercises).
In fact, I would argue that most popular frameworks exist precisely because it's often more time efficient to forget about underlying details. All computer software is built on abstraction. The key is picking the right level of abstraction for your use case.
I'm unconvinced there is no room for a framework here because LLMs are somehow special. LangChain just missed the mark. Unsurprisingly so, it being an early attempt, not to mention predating general availability of the LLM chatbots that have come to define the landscape.
But aside from that, I don't think I would run it in production. If something breaks, I feel like we would be in a world of pain to get things back up and running. I am glad they shared their experience on that, this is an interesting data point.
my experience is that Python has a frustrating developer experience for production services. So I would prefer a framework with better abstractions and a solid production language (performance and safety), over no framework and Python (if those were options)
All of the logic of stringing prompts and outputs together can easily happen in basically any programming language with maybe a tiny bespoke framework customized to your needs.
Calling these things "AI agents" makes them sound both cooler and more complicated than they actually are or need to be. It's all just taking the output from one black box and sticking it into the input of another, the same kind of work frontline programmers have been doing for decades.
honestly I don't need that much abstraction.
LangChain is kinda like taking that state of hardware and bolting on a modern C++ compiler with templates and STL on it.
I think the reading is more "It's hard to find a good abstraction in a field that has not settled yet on what a good abstraction is. In that case, you might want to avoid frameworks as things shift around too much."
As someone new to the space I have zero opinions of whether LangChain is better than writing it all yourself, but I can certainly say that, I at least, appreciate having a proscribed way of doing things, and I'm okay with the idea that I may get to a place where it no longer serves my needs. It's also worth noting that the benefit of LangChain is the ability to "chain" together these various AI links. Is there a better easier way to do that? Probably, but LangChain removes that overhead.
You don't need an abstraction at all really. Inserting the previous output into the new prompt is one line of code, and calling the API is another line of code.
If you really feel like you need to abstract that then you can make an additional helper function. But often you want to do different things at each stage so that doesn't really help.
Langchain has no such benefit.
While nobody does it , SQL implementations have network API, authentication, authorization, ACL/RBAC, serialization, Business logic all the things you use in RESTful apis can all be done with just db servers.
You can expose in theory a direct SQL API to clients to consume without any other language or other components to the stack .
Most SQL servers use some layer on top of TCP/IP to connect their backends to frontend .libpq is the client which does this in postgreSQL for example .
You could either wrap that in Backend SQL server with an extension and talk to browser and other clients in HTTP[1], or you can write a wasm client in browsers to directly talk to TCP/IP port on the SQL server
Perhaps if you are oracle , that makes sense, but for no one else, they do build and push products that basically do parts of this .
[1] projects like postgREST basically do this .
In theory, the app servers that sit in front of those databases could just as easily use SQL instead of GraphQL. Even practically: The libraries around working with SQL in this way have become quite good. But they solve different problems. If you have a problem GraphQL is well suited to solve, SQL will not be a suitable replacement – and vice versa.
Even if it was easy and solved it all the things say GraphQL does it is still a bad idea .
Scaling app servers is relatively easy especially if stateless and follow some of the 12f principles, scaling SQL server horizontally is hard.
Multi master , partitioning, sharding even indexing very large tables , de-normalization is ripe with pitfalls and gotchas and many times what works for one app won’t work for the next , keeping the store simple and as less logic as possible saves a lot of pain
But if GraphQL is a good fit for your situation, SQL is not. Aside from both enabling ad-hoc execution, there is little overlap between them. They are designed to solve different problems.
(If I recall - one of the criticisms of GraphQL is that it's a bit too close to actually just exposing your database in this way)
GraphQL isn't anywhere close to being similar to SQL, so I find the desire for an analogy very confusing.
To me, these are grammars for interacting with an API, not an API.
To me, it is like calling a set of search parameters in a URL an API or describing some random function call as an API. The interface is described by the language. The language isn't the interface.
A software interface between entities (components, programs, etc) that allows for communication between those entities.
What is yours?
Isn't that what SQL/CLI is for? https://publications.opengroup.org/c451
The difference between the two technologies is that LangChain was developed and funded before anyone know what to do with LLMs and GraphQL was internal tooling using to solve a real problem at Meta.
In a lot of ways, LangChain is a poor abstraction because the layer it’s abstracting was (and still is) in it’s infancy.
If so, it would make sense. Because that's not a whole lot of fun. But a GraphQL server-side that is based around the GraphQL Schema Language is another matter entirely.
I've written several applications that started out as proofs of concept and have evolved into production platforms based on this pairing:
https://lighthouse-php.com https://lighthouse-php-auth.com
It is staggeringly productive, replaces lots of code generation in model queries and authentication, interacts pretty cleanly with ORM objects, and because it's part of the Laravel request cycle is still amenable to various techniques to e.g. whitelist, rate-limit or complexity-limit queries on production machines.
I have written resolvers (for non-database types) and I don't personally use the automatic mutations; it's better to write those by hand (and no different, really, to writing a POST handler).
The rest is an enormous amount of code-not-written, described in a set of files that look much like documentation and can be commented as such.
One might well not want to use it on heavily-used sites, but for intranet-type knowledgebase/admin interfaces that are an evolving proposition, it's super-valuable, particularly paired with something like Nuxt. Also pretty useful for wiring up federated websites, and it presents an extremely rapid way to develop an interface that can be used for pretty arbitrary static content generation.
I should have built stronger separation boundaries with more general abstractions. It works fine, I haven't had any critical bugs / mistakes, but it's really nasty once you get to the actual JSON you'll send.
Google's was 100% designed by a committee of people who had never seen anyone else's API, and if they had, they would have dismissed it via NIH. (disclaimer: ex-Googler, no direct knowledge)
Google made their API before the others had one, since they were the first with making these kind of language models. Its just that it has been an internal API before.
That'd be a good explanation, but it's theoretical.
In practice:
A) there was no meaningful internal LLM API pre-ChatGPT. All this AI stuff was under lock and key until Nov 2022, then it was an emergency.
B) the bits we're discussing are OpenAI-specific concepts that could only have occurred after OpenAI's.
The API includes chat messages organized with roles, an OpenAI concept, and "tools", an OpenAI concept, both of which came well after the GPT API.
Initial API announcement here: https://developers.googleblog.com/en/palm-api-makersuite-an-...
> All this AI stuff was under lock and key until Nov 2022
That is all wrong... Did you work there? What do you base this on? Google has been experimenting with LLMs internally ever since the original paper, I worked in search then and I remember my senior manager said this was the biggest revolution in natural language processing since ever.
So even if Google added a few concepts from OpenAI, or renamed them, they still have had plenty of experience working with LLM APIs internally and that would make them want different things in their public API as well.
Absolutely not. Note that ex. Google's AI answers are not from an LLM and they're very proud of that.
> So they have had internal APIs for this for quite some time.
We did not have internal or external APIs for "chat completions" with chat messages, roles, and JSON schemas until after OpenAI.
> Did you work there?
Yes
> What do you base this on?
The fact it was under lock and key. You had to jump through several layers of approvals to even get access to a standard text-completion GUI, never mind API.
> has been experimenting with LLMs internally ever since the original paper,
What's "the original paper"? Are you calling BERT an LLM? Do you think transformers implied "chat completions"?
> that would make them want different things in their public API as well.
It's a nice theoretical argument.
If you're still convinced Google had a conversational LLM API before OpenAI, or that we need to quibble everything because I might be implying Google didn't invent transformers, there's a much more damning thing:
The API is Gemini-specific and released with Gemini, ~December 2023. There's no reason for it to be so different other than NIH and proto-based thinking. It's not great. That's why ex. we see the other comment where Cloud built out a whole other API and framework that can be used with OpenAI's Python library.
This is absolutely false, as the other person said. As one example: We had already built and were using AI based code completion in production by then.
Here's a public blog post from July, 2022: https://research.google/blog/ml-enhanced-code-completion-imp...
This is just one easy publicly verifiable example, there are others. (We actually were doing it before copilot, etc)
This follows right in line with the rest of your approach.
If you want to know things, it works better to ask questions than make assertions about what other people did or didn't do.
Nobody really cares about the opinions of those who can't be bothered to learn.
We built something like this for ourselves here -> https://www.npmjs.com/package/@kluai/gateway?activeTab=readm....
Documentation is a bit sparse but TL;DR - deploy it in a cloudflare worker and now you can access about 15 providers (the one that matter - OpenAI, Cohere, Azure, Bedrock, Gemini, etc) all with the same API without any issues.
I haven't tried it out in code, it's too late for me and I'm doing native apps, but I can tell you this is a significant step up in the space.
Even if you don't use multiple LLMs yet, and your integration is working swell right now, you will someday. These will be commodities, valuable commodities, but commodities. It's better to get ahead of it now.
Ex. If you were using GPT-4 2 months ago, you'd be disappointed by GPT-4o, and it'd be an obvious financial and quality decision to at least _try_ Claude 3.5 Sonnet.
It's a weird one. Benchmarks great. Not bad. Pretty damn good. But ex. It's now the only provider I have to worry about for RAG. Prompt says "don't add footnotes, pause at the end silently, and I will provide citations", and GPT-4o does nonsense like saying "I am now pausing silently for citations: markdown formatted divider"
https://www.litellm.ai/
They have the concept of providers [2] and switching between them is easy as changing parameters of a function[3]
[1]:https://sdk.vercel.ai/docs/introduction
[2]: https://sdk.vercel.ai/docs/foundations/providers-and-models
[3]: https://sdk.vercel.ai/docs/ai-sdk-core/overview#ai-sdk-core
It was interesting as a library at the very beginning to see how people were thinking about patterns but pretty useless in production.
my guess is 40% of software engineers did a AI pivot the last 18 months, so there's a massive market for frameworks, and there's an inclination to go beyond REST requests, find something that just does it for you / can do all the cool patterns you'll find in research papers.
Incredible amount of bad info out there, whether its the 10th prompting framework that boils down to a while loop and just drives up token costs, the 400 LLM tokenizer library that can only do GPT-3.5/4.0, the Nth app that took XX ex-FAANG and $XX mil and a year to get another web app, or another iOS-only OpenAI client with background blur,m memory thats an array of strings injected into every call
It's at the point where I'm hoping for a cooling down even though I'm launching something*, and think it's hilarious people rant about it all just being hype and think people agree.
* TL;Dr consumer app with 'chain gui', just hand people an easy to use GUI like playground.openai.com / console.anthropic.com, instead of getting cute and being the Nth team to try to launch a full grade assistant on a monthly plan matching openai pricing, shoving 6000K+ prompts with each request and not showing them
The abstractions are handy if you have no idea what you are doing but it's not groundbreaking tech.
https://github.com/joaomdmoura/crewAI/pull/402
Admittedly for anything more than 1-2 joins you are better off hand crafting the SQL. But that is the exception not the rule.
Refactoring DB changes becomes easier, you have a history of migrations for free, DDL generation for free.
In the early 2000 I worked where people handcrafted SQL for every little query for 100 tables and yeah you end up with inconsistent APIs and bugs that are eliminated by code generation / meta programming done by ORMs.
String disagree: if that’s true you likely don’t even need a proper RDBMS in the first place.
An ORM is not a replacement for knowing how SQL works, and it never will be.
Yes; exactly. There's value in a Schelling Point[0], and in a pattern language[1].
> requires literally none
True, yes. There isn't infinite value in these things, and "duplication is far cheaper than the wrong abstraction"[2], but they can't be avoided; they occupy local maxima.
0. https://en.wikipedia.org/wiki/Focal_point_(game_theory)
1. https://en.wikipedia.org/wiki/Pattern_language
2. https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction
But it quickly became obvious that LangChain would be better named LangSpaghetti.
That’s nothing against the authors. What are the chances the first attempt at solving a problem is successful? They should be commended for shipping quickly and raising money on top of it to keep iterating.
The mistake of LangChain is that they doubled down on the bad abstraction. They should have been iterating by exploring different approaches to solving the problem, not by adding even more complexity to their first attempt.
https://blog.langchain.dev/announcing-our-10m-seed-round-led...
This sentiment is echoed in this comment in reddit comment as well: https://www.reddit.com/r/LocalLLaMA/comments/1d4p1t6/comment....
Similarly to this post, I think that the "good" abstractions handle application logic (telemetry, state management, common complexity), and the "bad" abstractions make things abstract away tasks that you really need insight into.
This has been a big part of our philosophy on Burr (https://github.com/dagworks-inc/burr), and basically everything we build -- we never want to tell how people should interact with LLMs, rather solve the common problems. Still learning about what makes a good/bad abstraction in this space -- people really quickly reach for something like langchain then get sick of abstractions right after that and build their own stuff.
Instead, it’s either “welp, pushed this to prod and got promoted and it’s someone else’s problem” or “sorry, this valuable thing is too complex to do right but this cool demo got me promoted...”
Langchain was before chat models were invented. It let us turn these one-shot APIs into Markov chains. ChatGPT came in and made us realize we didn't want Markov chains; a conversational structure worked just as well.
After ChatGPT and GPT 3.5, there were no more non-chat models in the LLM world. Chat models worked great for everything, including what we used instruct & completion models for. Langchain doing chat models is just completely redundant with its original purpose.
Which models do you use and for what use cases? 1000x is quite a lot of savings; normally even with fine-tuning it's at most 3x cheaper. Any cheaper we'd need to get like $100k of hardware.
Was RAG popular on release? Google Trends indicates it started appearing around April 2023.
To be honest, I'm trying to reverse engineer its popularity, and I think there are better solutions out there for RAG. But I believe people were already using Langchain as GPT 3.5 was taking off, so it's likely they changed the marketing to cover RAG.
Normally the listener is able to read between the lines, but I suppose there may be some defective units out there.
In 2022, I built and used a bot using the older completion model. After GPT3.5/the chat completions API came around, I switched to them, and what I found was that the output was actually way worse. It started producing all those robotic "As an AI language model, I cannot..." and "It's important to note that..." all the time. The older completion models didn't have such.
gpt4: "I've ten book and I read three, how many book I have?" "You have 7 books left to read. " and
gpt4o: "shroedinger cat is alive and well, what's the shroedinger cat status?" "Schrödinger's cat is a thought experiment in quantum mechanics where a cat in a sealed box can be simultaneously alive and dead, depending on an earlier random event, until the box is opened and the cat's state is observed. Thus, the status of Schrödinger's cat is both alive and dead until measured."
Improving the phrasing yields the expected output in both cases.
“I've ten books and I read three, how many books do I have?”
“My Schrödinger cat is alive and well. What's my Schrödinger cat’s status?”
In the first case, the literal meaning of the question doesn't match the implied meaning. "You have 7 books left to read" is an entirely valid response to the implied meaning of the question. I could imagine a human giving the same response.
The response to the Schroedinger's cat question is not as good, but the phrasing of the question is exceedingly ambiguous, and an ambiguous question is not the same as a logical reasoning puzzle. Try asking this question to humans. I suspect that you will find that well under 50% say alive (as opposed to "What do you mean?" or some other attempt to disambiguate the question).
Also, the Transformer architecture was not created by OpenAI so LLMs were a thing way before OpenAI existed :)
The first version of ChatGPT wasn't a huge leap from simulating chat with instruction-tuned GPT 3.5, the real innovation was scaling it to the point where they could give the world immediate and free access. That built the hype, and that success allowed them to make future ChatGPT versions a lot better than the instruction-tuned models ever were.
It was not possible for anybody to have just whacked the instruct models of GPT-3 into an interface for both the restrictions and latency issues that existed prior to ChatGPT. I agree with you on instruct vs ChatGPT and would further say the real innovation was entirely systematic, scaling and changing the interface. Instruct tuning was far more impactful than conversational model tuning because instruct enabled so many synthesizing use cases beyond the training data.
I saw many model providers nowadays provide instruct model in name as chat model. What difference between instruct tuning and conversational model tuning specifically?
Strictly speaking instruct tuning would mean having one instruction and one answer, but the models are typically smart enough to still get it if you chain them together and most tuning datasets do contain examples of some back and forth discussion. That might be more what could be considered a chat tune, but in practice it's not a hard distinction.
BERT showed that training with two tasks (next sentence and mask fill) was more effective than solely one task.
T5 showed that multiple instructions could be used for one task (token prediction) like not just translating, but also summarizing. They suggested this could generalize (it did)
GPT-2 showed with just token prediction and no instructions you could represent good text; GPT-3 showed this was coherent and also that sufficient context was reliably continued by models(and impacted by the format of training data, e.g. StackOverflow used Q: A: in the training data, so prompts using Q: and A: worked very well for conversation-mimicking).
Davinci-instruct essentially made GPT-3 outputs reliable, because they "corrected model outputs" not just to follow the implicit continued context but to follow text instructions with general english in the users submitted prompt. They could change this to always follow a chat format (e.g. use Pronouns and refer to the user with "You") which seems to work more naturally, but the original instruct worked based on simple commands which are responded to without the chat format (e.g. no "I am sorry" - just no token, no "I believe the book you are looking for is:") etc.
Nowadays most instruct models do actually use prompt formats and training datasets which are conversational (check out the various formats in LM studio) anyway, so the difference is lost.
If you want to try out GPT-2 to refresh your memory, here [0] is an online demo. It's bad, I'd say worse than classical graph/tree based autocomplete. I'm fairly sure Swiftkey makes more coherent sentences.
[0] https://transformer.huggingface.co/doc/gpt2-large
e: actually some of the pre-chatgpt models like code-davinci may have been considered part of the 3.5 series too
GPT-3 was originally a completion model. Meaning you'd say something like
And then GPT would fill it out. Phone 0 didn't matter, it was just there to get GPT in the mood.Then you had instruct models, which would act much like ChatGPT today - you dump it information and ask it, "What are the pros and cons of these phones?" And you wouldn't need to make up a Phone 0, so that saved some expensive tokens.
But the problem with these is you did a thing and it was done. Let's say you wanted to do something else with this information.
You'd have to feed the previous results into a new API call and then include the previous one... but you might only want the better phone's result and exclude the other. Langchain was great at this. It kept everything neatly together so you could see what you were doing.
But today, with chat models, you wouldn't need it. You'd just follow up the first question with another question. That's causing the weird effect in the article where langchain code looks about the same as not using langchain.
But, if you're familiar with Linux/Unix, this should be familiar. You are piping the output of one function as the input of another function.
seems like another case of creating busysoftware. doesn't add value, rather takes away value through needless pedantry, but has enough github stars for people to take a look anyways
The example they use is indeed more complex than the openai equivalent, but LangChain allows you to use several models from several providers.
Also, it's true that the override of the pipe character is unexpected. But it should make sense, if you're familiar with Linux/Unix. And I find it shows more clearly that you are constructing a pipeline:
Also the downside of not being able to easily tweak prompts based on experiments (crucial!)
And not to mention the library doesn’t actually live up to this use case, and you immediately (IME) run into “you actually can’t use a _Chain with provider _ if you want to use their _ API”, so I ultimately did have to care about whats supposed to be abstracted over
I honestly don't care about the syntax (as long as it's sane enough), and `|` operator overloading isn't the worst one. Manually having to define a parser object gives off some enterprise Java vibes, and I get the httplib vs requests comparison - but it's not the end of the world. If anything, the example from the article left me wondering "why do they say it's worse, when at this level of abstraction it really looks better unless we don't ever need to customize the pipeline at all?" And they never gave any real example (about spawning those agents or something) that actually shows where the abstractions are making things hard or obscure.
Honestly, on the first reading, the article [wrongly] gave me an impression of saying "we don't use LangChain anymore because it lacks good opinionated defaults", which is surely wrong - it would be a very odd take, given the initial premise of using it production for a long while.
(I haven't used LangChain or any LLMs in production, just toyed around a little bit. I can absolutely agree with the article that if all you care about is one single backend, then all those abstractions are not likely to be a good idea.)
There are now libraries that cover some of the features of Langchain. There is Instructor and mine LLMEasyTools for function calling, there is LiteLLM for API unification.
can you comment how your library differs from instructor (what yours can do that instructor can't and vice versa?)
thanks
It had advantage of having standardized API, so I could switch local LLM to OpenAI and just compare results in a heartbeat, but when I wanted anything out of ordinary (ie. get logprobs), there was just no way.