HN Explorer

Locally hosted LLM use cases?

11 points by donji ↗ HN
I’m curious what interesting use cases people on here have thought of for locally hosted llms. I’m also curious what form factor will need to be supported to make you use case.

I think because of the large amount of ram that’s needed, something phone sized could not be that useful.

But I could be wrong, maybe there is a use case for a really dumb small model that can be used to great effect.

Thoughts?!

7 comments

[ 0.30 ms ] story [ 25.6 ms ] thread
quickthrower2
The advantages if local:

Privacy / Secrecy

Low Latency (as in network)

Low Latency (as in ttfb)

Can fine tune continuously and immediately.

No need for internet connection or just a “download only” type connection. Great for remote areas.

So where this might be useful?

A local NoT (network of things) that is privacy conscious.

An assistant for people who want to detox from being online.

A low latency code assistant for anyone who gets annoyed if they blink and nothing has happened yet.

A toy / pet type thing

LLM guidance for a submarine or space mission.

thensome
So, i have gpt4all on my laptop and i mostly use it for random fictional writing fun stuff.

Aside from that, it would be cool to have some kind of automatic inventory and info system for items in my home that can make minor assumptions about what you want, or do a "twenty questions" game to help find a word you forgot. My gf is dyslexic and can't remember words that she's only seen in writing, so she sometimes gets really irritated when she's trying to explain stuff and she can't even think of what letter the word starts with. I notice that when she wants to think of a specific word like "conglomeration" she has a very hard time thinking of how to describe the concept. So something that would be like "are you thinking of a noun? Is it a person, group, etc?" without her having to write a stating sentence would be helpful. It would have to just have a "start questions" button, bc she will get stressed by trying to think of how to start the writing lol

thiag0
May I ask you how fast is gpt4all running on your laptop? I've tried to run locally on my desktop and it worked really well, however the "response" time it's pretty slow.
thensome
Yeah, it's still pretty slow and can't take large inputs very well. I'd say it takes 30sec to a minute to process a 200-300 word selection.
kobalsky
pseudo googling without internet

you could wade through ffmpeg docs for a while to crop a video or you could ask the llm and get a quick response

shortrounddev2
You can have it generate things without the censorship. I saw an article in WaPo saying that some people use it for cybering