HN Explorer

3 comments

[ 3.8 ms ] story [ 7.8 ms ] thread
wanderingmind
This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?
tynmann
I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models.

But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization.

The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.

tynmann
However if one wants to run something like this in production, it would make more sense to use a server with at least an A10, which has better performance per power consumption.