Simple & straightforward, it's a quick spare-time project designed to enable automation and not as a standalone way to publicly host the models (no rate limiting, persistence or auth). The project README is here: https://github.com/edfletcher/llama.http/tree/master/example...
With the Docker image, it's just a matter of grabbing the model binaries you want, adding a small sidecar JSON [1] describing them and you're up and running.
Really just a porting/re-targeting onto the parent llama.cpp project of the HTTP I'd built on the alpaca.cpp fork awhile back [2] in order to take advantage of all the great new GGMLv3-encoded models, mmap'ed loading, etc.
All I'd ask if you build anything neat on top of it, please share it so I can use it too! Thanks & enjoy
1 comment
[ 3.4 ms ] story [ 10.4 ms ] threadWith the Docker image, it's just a matter of grabbing the model binaries you want, adding a small sidecar JSON [1] describing them and you're up and running.
Really just a porting/re-targeting onto the parent llama.cpp project of the HTTP I'd built on the alpaca.cpp fork awhile back [2] in order to take advantage of all the great new GGMLv3-encoded models, mmap'ed loading, etc.
All I'd ask if you build anything neat on top of it, please share it so I can use it too! Thanks & enjoy
[1] https://github.com/edfletcher/llama.http/tree/master/example...
[2] https://news.ycombinator.com/item?id=35417730