Ask HN: What is the right way to fuse my enterprise data with an LLM?
Some stuff it into the prompt
Some fine tune llama
Some rag and then ask LLM to answer questions
What are the others?
What is the best?
Some fine tune llama
Some rag and then ask LLM to answer questions
What are the others?
What is the best?
13 comments
[ 4.3 ms ] story [ 46.7 ms ] threadImagine that the LLM portion is running as client-side browser JavaScript application: Nothing in the training data or prompt is reliably-secret, and a determined user can get it to emit almost anything they like to whatever is downstream.
It's kinda like the folks saying Google is useless when there are clearly cases where it is useful.
In the same way LLMs have enormous value and utility, but equally there are places it is not as useful. Simply dismissing it as "mostly hype" is to miss the potential value.
For example, I've found it to be good at writing SQL queries. I can see how a human interface to a sql backend would make reading data much easier for my users.
Equally I have a well documented data schema, and lots of canonical docs for the product. I'd like to integrate this specialised knowledge with the existing SQL knowledge. In that context I'm interested in actual answers to the original question.
Is it even possible?
We can revisit the graveyard of over-hyped technologies if you want, and catalog the failures and wasted money. Ride the wave if you like. Show the success stories of LLMs in terms of business value or innovation. Don’t just repeat the press releases and anecdotes. You won’t find many successes, though maybe they will come in time.
> I’ve found [an LLM] good at writing SQL queries.
I don’t mean snark, but how do you know it writes good queries? I will grant that it can write queries that execute, but do those give correct results? Will they look good on a large production database where performance starts to matter?
When I played with LLMs writing SQL they worked well enough on toy schemas that resemble those found in tutorials. But faced with a real schema that requires joining tables and applying complex conditions they failed to produce usable SQL. However they did (with prodding) produce SQL that executed. It just gave the wrong results. That’s not a gain along any axis and could cost a lot of money for an organization blindly trusting such tools.
LLMs can write “good” code for some constrained definition of “good.” If you also want correct, testable, maintainable, secure, etc. they don’t perform well enough. Not just my opinion, we’ve had time to study the output of coding LLMs and it’s not looking good.
Look out for security problems, because the bots certainly will.
Please report your experience.
Your typical LLM now days makes a lot of applications humans programmed thus far look silly.
There will definitely be a denial phase, but entire companies and products just look literally silly compared to an llm.
Almost unanimously CEOs/CTOs report no big gains in business value or measured productivity due to LLM adoption. The majority opinion seems to land at "We don't want to miss out, but we're not sure how to get any real benefit."
Fine tuning: Great if you have static data and deep pockets for compute.
RAG inclusive Vector DB: The gold standard. Think of it as having your data whisper the answers to the LLM.
With AI Squared, you can keep your data fresh, dynamic, and external because nobody wants to retrain a model every time the boss changes their mind. :D