Localllm lets you develop gen AI apps on local CPUs (cloud.google.com)

Even Google is making a llama.cpp wrapper.

Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).

> We suggest using a machine type of e2-standard-32 (32 vCPU, 16 core and 128 GB memory), an admittedly beefy machine.

Average price per month $966.56

https://gcloud-compute.com/e2-standard-32.html

One thing I'm happy about is that the gguf file format is getting widely used, here endorsed by Google. I vastly prefer working with gguf and have adopted it for my project, hopefully it gets the critical mass to become a "native" format and we can stop with pytorch checkpoints.

> pip3 install openai

Why is that required?

I'm not thrilled about https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... calling their Python package "llm" and installing "llm" as a CLI command, when my similar https://llm.datasette.io/ project has that namespace reserved on PyPI already: https://pypi.org/project/llm/

How can you call something “local”LLM and have it rely on Google cloud?!

> “…within the Google Cloud Workstation”

They've really jumped the shark. Creating a wrapper for llama.cpp (a project they don't even credit here) and acting like that's even interesting, then somehow implying this is a feature or benefit of using Google Cloud Workstations? This is the most technically bankrupt product page I've ever seen in my life and for a company that used to mean something to people to do it is pathetic.

Hey that's my title ... almost :-) https://martincapodici.com/2023/07/15/no-local-gpu-no-proble...

This blog post has a distinct "written by an LLM" feel to it that I can't quite put my finger on.

As a former Google employee working on their OSS stack, this feels a bit off brand.

Here's a whole list of indie, OSS AI frameworks you can use: https://github.com/janhq/awesome-local-ai

Disclaimer: I'm one of the core devs on Jan.

The amount of obfuscation & head-in the-ground was nauseating to read

For the interested, I've been having fun with llama.cpp. I've got it running well, if a bit slowly, on my 2019 intel MacBook Pro. There's configs you can tweak that get it to run faster, mainly dealing with memory mapping. I've been working on documenting my findings but haven't published yet.

I say all that to encourage people to try it out with different models. It's not production quality but totally usable for testing things out even on older machines.

[dead]

A package called localllm (three l's in the middle, nice) which is focused on running models in GCP.

> run LLMs locally on CPU and memory, right within the Google Cloud Workstation

And in the code https://github.com/GoogleCloudPlatform/localllm/blob/main/qu... there's just a shebang halfway down?

    import sys

    
    def main(argv):
        pass


    if __name__ == '__main__':
        main(sys.argv)
    #!/usr/bin/env python3

This is awful. Did bard write this?

Looks like the repo (localllm) only contains a smallish Python project to do the model downloading. When it comes to serving the model, it calls out a Python module llama_cpp.server, which is not part of the repo, likely something inside the Docker image. A bit disappointing given how the article sounds.

Google really doesn’t seem to have much of a coherent plan on AI by the looks of it

The idea of making it easier to download modes from HuggingFace, and serve them via llama-cpp-python could be a good contribution, but unfortunately there are two different names that the "Localllm" project introduces for itself - Localllm and the 'llm' command, both of them problematic for quite a few of reasons, perhaps they could just drop naming it anything to do with local? E.g: Simon Willison's well known `llm` package on PyPI already uses the `llm` command. A name for the command that differs from the project name creates extra work for no good reason.

Look at this positioning Google use in the blog post: "localllm combined with Cloud Workstations revolutionizes AI-driven application development by letting you use LLMs locally on CPU and memory within the Google Cloud environment." - implying a local setup that's actually reliant on cloud resources is a contradiction.

Maybe they could rename it with a single name that has no conflicts or misleading terms that makes it clear it's to help setup and run inference without confusing the matter about how local things might sometimes be. It won't be nice to read the scathing critiques in some of the other comments! Maybe they could fix up the mess in querylocal.py!

Being able to run the same code locally as in a cloud platform is a nice development pattern but optimal inference setup for many ML situations is a tricky problem, are you really making good use of the CPU / GPU / TPU?

The underlying server tech: llama-cpp-python recently added support for draft models that could make inference much faster. How easy would using that from Localllm be? I'd be concerned `Localllm > llama-cpp-python > llama.cpp` might be too many layers of systems to optimise or code with. Milliseconds often matter.

Having a fast, well priced solution is not easy, people want as many tokens per second as they can get, ideally with no warm-up. How would they solve that?

People looking at fast inference options might like to see Banana's parting words, as they wind down their serverless GPU cloud, in their swansong, they suggest a few hosted inference options people might want to consider: https://news.ycombinator.com/item?id=39288915

I've run non-ECC systems at heavy load for uptimes in excess of a year and not had issues. It really is statistically rare... but of course common enough that if you run whole data centers you will see it.

What an awfully misleading name. This doesn't run locally, it runs on the cloud. What the fuck was that guy thinking?