mmoskal

I wanted to share a project I've been working on for the past few weeks: llgtrt. It's a Rust implementation of a HTTP REST server for hosting Large Language Models using llguidance library for constrained output with…

AICI is a proposed common interface between LLM inference engines (llama.cpp, vLLM, HF Transformers, etc.) and "controllers" - programs that can constrain the LLM output according to regexp, grammar, or custom logic, as…