An intuition is to think of prefill (prompt processing) and decoding (text generation) phases. When you send a request with a 50K tokens document, there's a huge amount of work happening: all tokens must go through the…
Firefox's AI Chatbot feature only proposes one open model provider (Mistral) and zero local option. They're not walking the talk.
It was trained from scratch by Cohere. They're the only Canadian AI lab - I'm glad they're releasing open weights and I wish them luck catching up!
I am using the `voipmonitor/vllm:lucifer` docker from the RTX6K discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477
Here's a DeepSeek-V4-Flash benchmark on 2X RTX Pro 6000: - Prefill: ~10K tok/s - Decode: 190 | 375 | 980 tok/s (for 1 | 4 | 16 concurrent requests) - GPU power draw during benchmark: Average: 585W | Max: 849W | Limit:…
You're forgetting a critical factor: concurrency. If a given hardware serves a single request at 150 tokens/s, it can also serve 20-30 requests at 100 tokens/s. Suddenly your $5K becomes $100K/month, enough to recoup…
Thanks! There's a heading control to include rotation in link: https://earth.google.com/web/@3.63731074,-23.1618975,-2690.7...
> Does privacy of Netflix ratings matter? The issue is not “Does the average Netflix subscriber care about the privacy of his movie viewing history?,” but “Are there any Netflix subscribers whose privacy can be…
For this type of repetitive application I think it's common to "fine-tune" a model trained on your business problem to reach higher quality/reliability metrics. That might not be possible with this chip.
Good question. That was for my MS account/licenses and some Azure stuff. I use Google Authenticator for most things. Thanks for the link, I'll take a look. I might just move it to a secondary device first.
Just looked - Microsoft Authenticator doesn't appear to work. I might be able to get off of it but it will take some prep. My banks are supported so that's good.
> if you could exclude all of the R&D and training costs LLMs have a short shelf-life. They don't know anything past the day they're trained. It's possible to feed or fine-tune them a bit of updated data but its world…
Recently llama.cpp made a few common parameters default (-ngl 999, -fa on) so it got simpler: --model and --context-size and --jinja generally does it to start. We end up fiddling with other parameters because it…
Do you really need a H200 for this? Seems like something a consumer GPU could do. Smaller models might be ideal [0] as they don't require extensive world knowledge and are much more cost efficient/faster. Why can't you…
I use Readdle Documents to sync PDF folders with my server PC via FTP. Free version supports PDF highlighting & simple annotations, basic file management, and automatically syncs back everything.
Assuming it's correct, I think this answer explains it well: https://stackoverflow.com/a/31996121/283879 Basically, yes it may write snapshots per-file (never per-commit) locally but there is a separate routine to…
They're turing-complete and modular so it's not really about what they can or cannot do. Testability, tooling and the open-source ecosystem and either bad or non-existent. Writing PL/SQL is the worst environment I've…
Just upgraded to Windows 11 (for its HDR features) and weather is now part of the "Widgets" bombarding me with ads and poor news sources. I genuinely tried to customize my "feed" but it's all junk -no reputable sources…
We're not on 18 yet but things are clear, thanks!
I notice that ReactDOM.render is also deprecated. In our embedded/plugin component scenario where we are given a <div> to load in, it appears we should replace our current pattern ReactDOM.render(React.createElement(...…
> it has by far the best vertical tab implementation of all browsers (someone please copy this) TIL. I think it's still far behind TreeStyleTabs (TST) on Firefox. Missing for me: - No nested groups. I know it's called…
I close a door or put a box in its way. No app.
When I check my recommended feed it's all stuff very closely related stuff to my subscriptions and recent views, and some useless shorts. It's just an echo chamber at this point. If I'm looking for potentially…
Recently Akka [0], a 12K star project. [0]: https://github.com/akka/akka/pull/31561
Thanks. I really like the presentation/animation style.
An intuition is to think of prefill (prompt processing) and decoding (text generation) phases. When you send a request with a 50K tokens document, there's a huge amount of work happening: all tokens must go through the…
Firefox's AI Chatbot feature only proposes one open model provider (Mistral) and zero local option. They're not walking the talk.
It was trained from scratch by Cohere. They're the only Canadian AI lab - I'm glad they're releasing open weights and I wish them luck catching up!
I am using the `voipmonitor/vllm:lucifer` docker from the RTX6K discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477
Here's a DeepSeek-V4-Flash benchmark on 2X RTX Pro 6000: - Prefill: ~10K tok/s - Decode: 190 | 375 | 980 tok/s (for 1 | 4 | 16 concurrent requests) - GPU power draw during benchmark: Average: 585W | Max: 849W | Limit:…
You're forgetting a critical factor: concurrency. If a given hardware serves a single request at 150 tokens/s, it can also serve 20-30 requests at 100 tokens/s. Suddenly your $5K becomes $100K/month, enough to recoup…
Thanks! There's a heading control to include rotation in link: https://earth.google.com/web/@3.63731074,-23.1618975,-2690.7...
> Does privacy of Netflix ratings matter? The issue is not “Does the average Netflix subscriber care about the privacy of his movie viewing history?,” but “Are there any Netflix subscribers whose privacy can be…
For this type of repetitive application I think it's common to "fine-tune" a model trained on your business problem to reach higher quality/reliability metrics. That might not be possible with this chip.
Good question. That was for my MS account/licenses and some Azure stuff. I use Google Authenticator for most things. Thanks for the link, I'll take a look. I might just move it to a secondary device first.
Just looked - Microsoft Authenticator doesn't appear to work. I might be able to get off of it but it will take some prep. My banks are supported so that's good.
> if you could exclude all of the R&D and training costs LLMs have a short shelf-life. They don't know anything past the day they're trained. It's possible to feed or fine-tune them a bit of updated data but its world…
Recently llama.cpp made a few common parameters default (-ngl 999, -fa on) so it got simpler: --model and --context-size and --jinja generally does it to start. We end up fiddling with other parameters because it…
Do you really need a H200 for this? Seems like something a consumer GPU could do. Smaller models might be ideal [0] as they don't require extensive world knowledge and are much more cost efficient/faster. Why can't you…
I use Readdle Documents to sync PDF folders with my server PC via FTP. Free version supports PDF highlighting & simple annotations, basic file management, and automatically syncs back everything.
Assuming it's correct, I think this answer explains it well: https://stackoverflow.com/a/31996121/283879 Basically, yes it may write snapshots per-file (never per-commit) locally but there is a separate routine to…
They're turing-complete and modular so it's not really about what they can or cannot do. Testability, tooling and the open-source ecosystem and either bad or non-existent. Writing PL/SQL is the worst environment I've…
Just upgraded to Windows 11 (for its HDR features) and weather is now part of the "Widgets" bombarding me with ads and poor news sources. I genuinely tried to customize my "feed" but it's all junk -no reputable sources…
We're not on 18 yet but things are clear, thanks!
I notice that ReactDOM.render is also deprecated. In our embedded/plugin component scenario where we are given a <div> to load in, it appears we should replace our current pattern ReactDOM.render(React.createElement(...…
> it has by far the best vertical tab implementation of all browsers (someone please copy this) TIL. I think it's still far behind TreeStyleTabs (TST) on Firefox. Missing for me: - No nested groups. I know it's called…
I close a door or put a box in its way. No app.
When I check my recommended feed it's all stuff very closely related stuff to my subscriptions and recent views, and some useless shorts. It's just an echo chamber at this point. If I'm looking for potentially…
Recently Akka [0], a 12K star project. [0]: https://github.com/akka/akka/pull/31561
Thanks. I really like the presentation/animation style.