Show HN: Grammar Generator App for Llama.cpp (grammar.intrinsiclabs.ai)

Thanks for the feedback. There's an example in the linked GH discussion but let me provide a quick ETE example here, centered around parsing structured information from a hypothetical shipping company email.

1. In the app, write a TypeScript schema for this, one good example could be

  interface DeliveryInformation {
      /* Tracking number for the delivery */
      tracking_number: string;
      /* Status of the delivery, one of "preparing", "out-for-delivery", or "delivered" */
      status: string;
      /* Weight of the package, e.g. "2oz" or "3lb" */
      weight: string;
      /* submission date time representation */
      submitted_ts: string;
  }

If you click Generate you'll see it generate a context-free grammar looking text, which is what llama.cpp reads. Click the download file to save it as grammar.gbnf

2. Grab a model for llama.cpp, e.g. this quantized Llama2 chat model: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...

3. Grab the prompt in this gist (https://gist.github.com/a10y/d926039eee63cc2bcaf6345f9a419e3...) and save as prompt.txt

4. Compile a recent commit of llama.cpp and run the following command

  ./main -m ./models/llama-2-13b-chat/llama-2-13b-chat.ggmlv3.q8_0.bin -f prompt.txt -c 4096 -n 1000 -t 1 --temp 0 --grammar-file ./grammar.gbnf

EDIT: fixed some typos

It's a relatively new feature, see

https://github.com/ggerganov/llama.cpp/pull/1773

Fixed! Thanks.