The Nvidia container toolkit works pretty much out of the box on WSL these days as well.
Funny that some cuda stuff works better through Windows virtualizing Linux than Windows natively, but if we're being honest even as a native Linux user, WSL probably provides a better user experience (vs having to use Nvidia drivers on Linux anyways)
Is the container toolkit even necessary anymore in WSL2? I did a fresh install of windows 10 LTSC a few weeks ago and installed WSL2 and docker with WSL2 integration and I was able to use my nvidia rtx card (checked through nvidia-smi) without any issues.
Interesting. I'll have to check to be sure, but I think maybe something is happening automagically if you have reasonably up to date nvidia drivers on the host OS, because I was able to run the EmotiVoice TTS docker (which requires nvidia gpu) from WSL2.
As someone who has had a home ML server since 2016 with two TitanX GPUs, and has worked on and maintained numerous servers since then I can definitely echo that maintaining Nvidia drivers, along with CUDA, CUDNN, etc has always been a hassle. It's certainly gotten better over time, but it's still quite fragile.
What about trying something like enabling forward compatibility for CUDA using an older driver?
https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda...
(This issue was actually just posted within the last day, so clearly people still have problems.)
If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.
They did add support for the whole rdna2/3 series to windows. But only support for 7900xt/xtx were added to linux for some reason. And softwere in title seems actually ran on linux.
Besides that, I don't think I ever heard that they supports container at all.
It is kind of ironic that the Year of Linux on the Desktop is all based on having Linux VMs across all major mainstream desktop OSes, even on ChromeOS, Crostini runs inside something similar to WSL.
But can you get this to run in Linux? From the article I can see it's a VSCode extension or something like that but the name kind of implies it's Windows only.
Yep, didn't read that. In the meantime, kudos to Canonical (and other parties) who helped Microsoft with WSL. Now they can also use Linux to produce Windows-only software! :D
I don't know, once upon a time I would have agreed with you, but recently I'm optimistic that MS is genuinely trying to break out of that behavior. Time will tell.
The link takes me to marketplace.visualstudio.com which makes it seem like a Visual Studio extension, not a VSCode extension? Not sure, it's as confusing as the live/hotmail/outlook/authentication parts of Microsoft tends to be.
Besides, it says the following:
> Windows AI Studio will run only on NVIDIA GPUs for the preview, so please make sure to check your device spec. WSL Ubuntu distro 18.4 or greater should be installed and is set to default prior to using Windows AI Studio.
So seems it'll only run in Linux, as it's a requirement to have Ubuntu WSL running beforehand.
1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it?
2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?
That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.
Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting.
With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc.
Maybe if they make it exclusive to the mac studio?
Cuda would be pointless for Apple, because cuda is for Nvidia only,
no?
As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last
time
> Cuda would be pointless for Apple, because cuda is for Nvidia only, no?
That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose.
> it’s the cheapest setup on the market that can get you this much VRAM.
I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM.
Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM.
You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows.
Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard.
I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators.
The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable.
If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost).
Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
That sounds like an overly specific target audience:
- you need more than 16 or 32 GB of ram.
- but less than 100.
- speed is important.
- but losing a factor 4 compared to a GPU is fine.
- money is very limited.
- but paying 6000 for a mac studio is cheap.
- this is very important professional work.
- but I don't need servers, ECC ram, raid, another OS...
Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.
Shared vram is everything of you're running local inference.
My 3 x 3090s can't fit the 180b falcon even at 2 bits into vram. But people with a 128gb studio can, and it's cheaper
Companies of the size of Microsoft usually function more like a country with various companies within, rather than one company with a unified view and unified goals.
So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupting the landscape.
I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs.
Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project.
In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD.
I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL.
BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
Installed this, and it didn't let me do anything. Despite the fact the readme says stuff runs locally, first it asked me to link to my github account, and then all models required me to ask someone (I think meta?) for permission on github or use a huggingface token or whatever.
So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years.
Anyone knows when Microsoft will release a local OCR model officially? I haven't seen anyone talking about this, but the one they're shipping with Snipping Tool (OneOcr) is top-tier and beats everything out there like Tesseract, easyOCR etc.
The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.
Wow, very nice! Thank you, hadn't come across them in my search.
It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
Not sure if it's what's used in the Azure Document Intelligence service, but my experience with it is pretty good.
The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled.
This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).
68 comments
[ 1.8 ms ] story [ 54.3 ms ] threadhttps://marketplace.visualstudio.com/items?itemName=ms-windo...
Funny that some cuda stuff works better through Windows virtualizing Linux than Windows natively, but if we're being honest even as a native Linux user, WSL probably provides a better user experience (vs having to use Nvidia drivers on Linux anyways)
This clearly shows that GNU/Linux must be superior
https://github.com/netease-youdao/EmotiVoice
https://forums.developer.nvidia.com/t/545-drivers-have-bad-f...
https://forums.developer.nvidia.com/t/wayland-native-wayland...
https://gitlab.freedesktop.org/xorg/xserver/-/issues/1317
and literally me being a Linux Nvidia 1080ti user for years and having plenty of issues
https://i.imgur.com/lwHDZoq.png
Automatic kernel update? https://forums.developer.nvidia.com/t/nvidia-smi-not-working...
What about upgrading to a new version of CUDA? https://stackoverflow.com/questions/43022843/nvidia-nvml-dri...
What about trying something like enabling forward compatibility for CUDA using an older driver? https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda... (This issue was actually just posted within the last day, so clearly people still have problems.)
If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.
Besides that, I don't think I ever heard that they supports container at all.
For now, but seems to be planned in the future.
Besides, it says the following:
> Windows AI Studio will run only on NVIDIA GPUs for the preview, so please make sure to check your device spec. WSL Ubuntu distro 18.4 or greater should be installed and is set to default prior to using Windows AI Studio.
So seems it'll only run in Linux, as it's a requirement to have Ubuntu WSL running beforehand.
Yes, it'll only run in Linux, but only when that Linux is also running inside Windows.
I do hope they are considering going in that direction though.
1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it?
2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?
With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose.
> it’s the cheapest setup on the market that can get you this much VRAM.
shared VRAM is not everything.
Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
Those are not equivalent in speed. macs RAM are much slower than these GPUs.
You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows.
Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard.
I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators.
The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable.
If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost).
Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
- you need more than 16 or 32 GB of ram.
- but less than 100.
- speed is important.
- but losing a factor 4 compared to a GPU is fine.
- money is very limited.
- but paying 6000 for a mac studio is cheap.
- this is very important professional work.
- but I don't need servers, ECC ram, raid, another OS...
Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.
Year 0 of cool thing - Nothing and silence
Year 1-3 of cool thing - Maybe, if you're lucky, a mention on some hardware thing related to it
Year 3-5 of cool thing - Hardware or software launches that uses thing, no mentions of this besides the earlier one if any.
Year 5-6 of cool thing - Next part of hardware or software launches that uses previous launch, no mentions of this besides the earlier one if any.
Obviously, the time-frames differ, but that's generally how they do things.
I wonder why Microsoft helps nVidia, instead of using their own technology?
Here’s an example: https://github.com/Const-me/Cgml
So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs.
Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD.
I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL.
BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
https://blogs.windows.com/windowsdeveloper/2023/11/15/elevat...
https://blogs.windows.com/windowsdeveloper/2023/12/14/direct...
But it's easy to publish something on github, and you get an bug tracker. So .. I guess they do it because it's easy to do so?
So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years.
0/10 would not recommend.
Try: conda config --set auto_activate_base false
Coming soon!
=(
The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.
Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sor...
It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled.
This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).