Show HN: Ollama – Run LLMs on your Mac (github.com)

The Modelfile is an exciting abstraction, OP.

EDIT: We don't let you run stuff from HF, but we are trying to repackage the popular models. The plan is to let you upload your own in the future to share them.

Easier is to override SYSTEM which allows will customize the system prompt.

SYSTEM "You are an expert in writing python."

If you do want modify the template, you can start by looking at the "base images" https://github.com/jmorganca/ollama/blob/main/library/modelf...

For example with Llama 2 as a base model, you can do this, but YMWV:

  FROM llama2
  TEMPLATE """
  {{- if .First }}
  <<SYS>>
  A conversation between three friends
  <</SYS>>
  {{- end }}

  Friend 1: hello
  Friend 2: hey
  Friend 3: {{ .Prompt }}
  """

Regarding the disk space, the model itself is 3.8GB.

  NAME          SIZE   MODIFIED      
  llama2:latest 3.8 GB 3 hours ago

`CGO_ENABLED=1 go build . `

If you have the Go toolchain installed, there are some steps from building it from source here: https://github.com/jmorganca/ollama#building

It does use GGML under the hood. Will definitely make sure that's more obvious for compatibility. There's about to be an update as well that includes those recent changes. We do vendor the source code since it made building with Go a lot easier, but definitely don't want to be "forking" it (vs supporting it) – it's a brilliant piece of software!

1. Running the model: it's built on the open-source (and amazing) llama.cpp project for running quantized (i.e. compressed) models like Llama 2 (launched yesterday) that will fit in memory on even a commodity Mac. It's similar to their "server" example as a starting point.

2. Downloading and storing models: models are distributed in a way that ensures their integrity and re-usability as much as possible (since they are large files!). For this we use a similar approach as Docker (https://github.com/distribution/distribution)

3. Creating custom models: models can be extended with this new idea we're experimenting with: a Modelfile. What this will do is effectively add "layers" to a model so you can distribute model data together and keep them self-contained. This builds on what I mentioned in 2 – our hope is this will make it easier to extend models like Llama 2 to your own use cases (e.g. a character).

We have a fix in the works that will be released soon.

May I ask what mac & memory you're running this on?

  /usr/local/bin/ollama -> /Applications/Ollama.app/Contents/Resources/ollama

I added a PR that supports dynamic command output injection, among other interesting things: https://github.com/jmorganca/ollama/pull/132. So you can imagine taking the output of, say, top and having the LLM parse it into useful information for you. Or having a chat session where you roll a d20 between each user interaction to determine their luck. Then being able to share that around to other people generically.

I'm also hoping at some point this project or a similar one will enable easy passing of full blown pipelines/chains of stuff around, as well as a ChatGPT style conversation sync and sharing tool.

This project builds on llama.cpp in a few ways:

1. Easy install! Precompiled for Mac (Windows and Linux coming soon)

2. Run 2+ models: loading and unloading models as users need them, including via a REST API. Lots to do here, but even small models are memory hogs and they take quite a while to load, so the hope is to provide basic "scheduling"

3. Packaging: content-addressable packaging that bundles GGML-based weights with prompts, parameters, licenses and other metadata. Later the goal is to bundle embeddings and other larger files custom models (for specific use cases, a la PrivateGPT) would need to run.

edit: formatting

Surface gravity 1.622 m/s2 (0.1654 g; 5.318 ft/s2)

> …would remain suspended in mid-air

Aside from the no-gravity hallucination, this is also funny.

Also: thank you!!

Half Mario, half Joey from Friends?

> Sure! Here's one: Why did the Emacs user bring a ladder to work? Because they wanted to go up a level! (This joke plays on the idea that Emacs has a lot of levels and modes, and the user might want to switch between them using a ladder.) I hope you found that amusing! Do you have any other requests?

[1] https://llama.perplexity.ai/

Something that annoys me about AI bots is that the language is extremely American. This is no surprise. But polite in the USA is very different to the UK, where fake politeness is almost as rude as it is possible to get. Fake politeness followed by “let’s…” as the other party tries to wrestle control of the conversation from their self-proclaimed pedestal is rage-inducing.

Put simply, there should be a British mode for chatbots, where a request that it cannot comply with is responded to with profanity. “Nah, don’t be a c**, Emacs is awesome. If you want a joke, look in a mirror”. Now that’s a bot I could get along with.

I tried with the "Wizard Vicuna Uncensored" model, and it gave me:

>>> tell me a joke about emacs

Why did Emacs cross the road?

To edit the other side.

>>> another one

Why did the Emacs programmer quit his job?

He couldn't handle the backslashes.

...

I guess it's better than the llama one.

By 'add documents', can I assume you are asking about embeddings? Ollama doesn't yet support embeddings. We are looking into how we can support this in the future.