A bit misleading, it's relying on onnxruntime for the actual inference. That said, given the state of python packaging and the fact that onnxruntime has APIs in C++ and Java, I don't know why more people don't distribute self-contained examples using it.
What are the implications of onnxruntime vs something like llama.cpp or those nn-512 models that seem closer to just C code for inference? What's different?
Any POSIX system that has a C++11 compiler could compile llama.cpp and run it, including all the BSDs. The OS supported by onnxruntime is very limited.
Yes, many are finally getting the point that Python "libraries" are actually bindings to native libraries, and any language can have bindings to the same libraries.
As mentioned in the README it's using onnxruntime for inferencing. Onnxruntime is a available on all platforms including Linux. See https://onnxruntime.ai/.
> The current codebase and the resulting Nuget packages target Windows and use DirectML, however only small sections of the code utilize Windows specific APIs, and thus could be ported to other platforms with minimal effort.
Well, that's a little uncharitable, isn't it? Assuming the developers primarily use Windows, they probably just don't have much priority for it; there's probably a lot of things they could do with the time. Perhaps they don't even have a setup to test it, and even as a staunch Linux user, I can't blame someone for not wanting to mess with all of the GPGPU drivers and non-sense.
The misunderstanding here is that I am referring to getting Linux GPGPU to work in the first place, not porting the code itself. Presumably they are suggesting that one of us Linux weirdos who already have gotten ROCm and/or CUDA working could do it with relative ease. That said, I have not attempted to verify the veracity of that assertion, but I think it's what they meant.
Though even if they could just do it, and have everything they need to do it, that doesn't mean they're lying. It really could just not be a priority.
There is sometimes a wide gulf between simple to implement and easy to verify. They may not have a Linux box set up with the hardware, drivers and development environment to build ML code. In that case even if the fix is simple, testing it and verifying it works is less simple. But anyone interested in using it on Linux would need to do that setup as a prerequisite anyway.
Someone please edit the title as it's really misleading. This linked project is using onnxruntime as opposed to implementing stable diffusion from scratch in C++. Meaning this is not in the same spirit as llama.cpp.
Assuming SD models are converted to the onnx format (or whatever its called) shouldn't current sd projects offer similar performance? The dominant cost is in generation is inference, not the project runtime I thought.
This. Conda and Venv in paper should be like docker plug and play... But in practice they can be a hit or miss. Still better than manual global library management.
Fwiw it's not impossible to package all this junk up in Docker. I did it recently. You have to run Docker w/ a whole bunch of flags though, like `--gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -p 7860:7860`. And the GPU isn't available during build time unless you do some hacks. But you can build the CUDA junk into the docker image along with Python and all the libs. Combine that with a .bat or .sh to contain the giant `docker run` command and you have a half-OK solution. Aside from having to start docker before being able to run it.
It's really nice to see that C++ implementations in the ML field are growing. Personally I do not like Python very much. C++ does not force you to do things in a specific (Pythonic) way.
38 comments
[ 5.1 ms ] story [ 91.9 ms ] threadLlama.cpp / ggml (while they support some hardware acceleration) is more focused on commodity hardware like x86 CPUs and Apple M-series silicon.
Also new apple metal implementation in progress in addition to apple accelerate, and is in baseline if you enable it.
Also a CUDA implementation.
Was hoping for something cross platform on to Linux.
If it were trivial, that statement wouldn't exist.
Just trying to be fair here.
Thanks for the downvote and for proving my point for me. That was literally exactly what I said. :)
But yeah, if you say something is "simple" and you can't do it yourself, then it's most likely not.
Though even if they could just do it, and have everything they need to do it, that doesn't mean they're lying. It really could just not be a priority.
Or at least has parts of that project copy pasted into its tree?
> This project is for educational purposes and serves as the main playground for developing new features for the ggml library.
(Edit, typo.)
And since C++17, it is quite easy to write Python like code.