HN Explorer

25 comments

[ 0.20 ms ] story [ 19.1 ms ] thread
refulgentis
Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
woadwarrior01
Yeah, both WebLLM and mlc-llm have been mostly abandoned since NVIDIA acquired OctoAI in late 2024.
TekMol
This seems to be the demo:

https://chat.webllm.ai/

I am getting:

    WebGPUNotAvailableError: WebGPU is not supported in
    your current environment, but it is necessary to
    run the WebLLM engine.
On both, FireFox and Chromium on Linux.
toto007
You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
gpugreg
I tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://flags/#enable-webgpu-developer-features and I can now request a GPU with 'high-performance' power preference, although there is no way to check whether this actually selected the correct GPU, and it is useless to do it like that anyway, since no website actually sets this (experimental) option. Fortunately, there is the flag chrome://flags/#force-high-performance-gpu which should solve this, but it is not available for my platform. I have a fairly standard RTX 3060, but apparently, the most common GPU (according to the steam hardware survey) is not supported.
rzzzt
Pretty much the same here with Edge,#enable-unsafe-webgpu got me the emulated swiftshader adapter, but only #force-high-performance-gpu made the dGPU appear in the adapter list of https://webgpureport.org/.
erichdongubler
Hey, Firefox Web GPU engineer here. We would love to hear about crashers on any platform! If you need help filing a Bugzilla bug, or want to email me so I can do it, please LMK.
gpugreg
Thanks, that is very kind of you. I have submitted bugs in the past and was enthusiastic for GPU support for over 15 years now, but I lost my faith.
noir_lord
For firefox you may need to enable `dom.webgpu.enabled` in `about:config`

It's on by default on Windows iirc - it's considered a potential security risk on Linux by Mozilla, so they ship it but it's turned off and it's up to the user to decide.

I think the reasoning is just because of how varied graphics drivers/stacks are on Linux compared to Windows/OSX and the attack surface been larger.

conceptme
Bake me a cake

responds with

> Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.

adastra22
A WebX technology that actually involves browsers!
MarioMan
I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.
mandeepj
It is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.
[deleted]
(comment deleted)
[dead] paidx
[flagged]
LoganDark
Oh nice, this actually works in Safari on my Mac! Prefill: 181.2 tok/s, Decode: 58.5 tok/s on my M4 Max -- not too impressive for 1B, but definitely impressive that it runs in my browser!
karuppiah7890
I love it! It works fine for me on Google Chrome on my MacBook Pro 14-inch M5 16GB RAM and 10 Cores CPU and 10 Cores GPU, with Metal 4 support
aitchnyu
Is there any use case/demo of lighter models like using embeddings for full text search, STT, OCR etc?