It should be possible to write a simple script to link LLMStack's app with your IRC channel. API documentation for running the apps is at https://llmstack.ai/docs/apis/introduction#apps. Let me know if you need help with this.
Firstly, the amalgamation of custom data sources via RAG pipelines sounds enticing on paper, but there are inherent complications. when fusing disparate data silos, considerations around data normalization, entropy reduction, and the potential amplification of noise-to-signal ratio are paramount. simply bolting on data to an LLM doesn't guarantee efficacy.
Then there's the underlying premise of using LLM as a primary UI model. while GPT and its ilk have made waves, a chatbot interface has inherent limitations. there's the conversational depth challenge, context retention gaps, and the non-trivial issue of multi-turn dialogues throwing off the model. Not to mention, the computational overhead of on-the-fly, real-time retrievals needs to be closely monitored to avoid a latency nightmare, especially if the knowledge base grows.
Further, there's a certain laziness in simply bolting on an LLM chatbot interface, which isn't appropriate for all settings, and can present accessibility challenges.
Speaking of the KB, the two-stage process delineated – indexing and querying – while conceptually sound, often buckles under scale. Memory networks and transformer architectures have their limits, and I'm curious about how LlamaIndex circumvents these bottlenecks.
also, leaning heavily on Streamlit could be a double-edged sword. Can anyone speak to its adaptability in production environments, especially under high concurrency?
anyway, I think it's important to bear in mind that chatbots and LLMs are tools, not magic wands or universal hammers
The data questions are exactly the things we are working through right now!
I think Streamlit is a great way to get started quickly. Would love to talk more about your thoughts around data ingest for prod use cases. yiding@runllama.ai
RAG tools are usually very trivial w.r.t. extracting clean data for embeddings. Demos typically show clean data sources with orthogonal data which lends itself well to good performing embeddings based IR. Real-world data is mixed up, messy, and requires a lot of work to extract, clean, normalize, etc. Typically fine-tuned embedding model would be needed.
> Real-world data is mixed up, messy, and requires a lot of work to extract, clean, normalize, etc. Typically fine-tuned embedding model would be needed.
I'd think the former is required for whichever strategy you choose, although I haven't found it necessary to fine-tune a base LLM.
What use case are you thinking of when you say, "typically fine-tuning is required"?
I would have loved some elaboration on how "the data" came to be. It appears in the sample app that each document is some subset of technical docs? That can be helpful, but when each document is very large you run into token limitations and increased probability of hallucinations. That's what I ran into when I tried to build a similar application that offers "GPT over docs". Figuring out your strategy for splitting documents isn't trivial.
It's not and that's why most of these solutions utilize an indexing/search service to provide more targeted data to feed into "context" instead of just burning tokens with whole/chunked docs.
12 comments
[ 0.80 ms ] story [ 41.4 ms ] threadDoes anyone know something out there?
It should be possible to write a simple script to link LLMStack's app with your IRC channel. API documentation for running the apps is at https://llmstack.ai/docs/apis/introduction#apps. Let me know if you need help with this.
Not sure tho if this code is shareable at the moment tho.
But if I'm not mistaken (can double check later on) it's based on Sopel
Ref: https://sopel.chat/
Firstly, the amalgamation of custom data sources via RAG pipelines sounds enticing on paper, but there are inherent complications. when fusing disparate data silos, considerations around data normalization, entropy reduction, and the potential amplification of noise-to-signal ratio are paramount. simply bolting on data to an LLM doesn't guarantee efficacy.
Then there's the underlying premise of using LLM as a primary UI model. while GPT and its ilk have made waves, a chatbot interface has inherent limitations. there's the conversational depth challenge, context retention gaps, and the non-trivial issue of multi-turn dialogues throwing off the model. Not to mention, the computational overhead of on-the-fly, real-time retrievals needs to be closely monitored to avoid a latency nightmare, especially if the knowledge base grows.
Further, there's a certain laziness in simply bolting on an LLM chatbot interface, which isn't appropriate for all settings, and can present accessibility challenges.
Speaking of the KB, the two-stage process delineated – indexing and querying – while conceptually sound, often buckles under scale. Memory networks and transformer architectures have their limits, and I'm curious about how LlamaIndex circumvents these bottlenecks.
also, leaning heavily on Streamlit could be a double-edged sword. Can anyone speak to its adaptability in production environments, especially under high concurrency?
anyway, I think it's important to bear in mind that chatbots and LLMs are tools, not magic wands or universal hammers
I think Streamlit is a great way to get started quickly. Would love to talk more about your thoughts around data ingest for prod use cases. yiding@runllama.ai
I'd think the former is required for whichever strategy you choose, although I haven't found it necessary to fine-tune a base LLM.
What use case are you thinking of when you say, "typically fine-tuning is required"?
You create 'contexts' (like folders) of data to chat with, and your data can be any of the following:
- Text files, pdf files, etc - Audio files (automatic transcription) - Web pages - YouTube videos - Entire YouTube channels
Demo video on front page too.