Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context (phind.com)
We’re excited to announce that Phind now defaults to our own model that matches and exceeds GPT-4’s coding abilities while running 5x faster. You can now get high quality answers for technical questions in 10 seconds instead of 50.
The current 7th-generation Phind Model is built on top of our open-source CodeLlama-34B fine-tunes that were the first models to beat GPT-4’s score on HumanEval and are still the best open source coding models overall by a wide margin: https://huggingface.co/spaces/bigcode/bigcode-models-leaderb....
This new model has been fine-tuned on an additional 70B+ tokens of high quality code and reasoning problems and exhibits a HumanEval score of 74.7%. However, we’ve found that HumanEval is a poor indicator of real-world helpfulness. After deploying previous iterations of the Phind Model on our service, we’ve collected detailed feedback and noticed that our model matches or exceeds GPT-4’s helpfulness most of the time on real-world questions. Many in our Discord community have begun using Phind exclusively with the Phind Model despite also having unlimited access to GPT-4.
One of the Phind Model’s key advantages is that it's very fast. We’ve been able to achieve a 5x speedup over GPT-4 by running our model on H100s using the new TensorRT-LLM library from NVIDIA. We can achieve up to 100 tokens per second single-stream while GPT-4 runs around 20 tokens per second at best.
Another key advantage of the Phind Model is context – it supports up to 16k tokens. We currently allow inputs of up to 12k tokens on the website and reserve the remaining 4k for web results.
There are still some rough edges with the Phind Model and we’ll continue improving it constantly. One area where it still suffers is consistency — on certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4.
We’d love to hear your feedback.
Cheers,
The Phind Team
357 comments
[ 3.3 ms ] story [ 278 ms ] thread> However, we’ve found that HumanEval is a poor indicator of real-world helpfulness.
https://news.ycombinator.com/item?id=38089888
https://news.ycombinator.com/item?id=38090442
If you want to show the warts of all these LLMs, ask it about ZFS if you know enough to spot the commonly parroted misinformation that plagues the internet.
IMHO, these systems look super useful if they're citing sources and they're worthless without.
When I ask it as one statement it performed ok, but if I made more specifications with follow-up statements, it kept trying to go down one path even though I told it to do it a different way. A solid start but it definitely needs some improvements, IMO.
E.g.: the efficient and robust file change monitoring on Windows is to read the NTFS change journal. For a single process lifetime there are other change notification APIs as well. Most software does neither and is either very slow or misses changes…
Is that with batching? If so, thats quite impressive.
> certain challenging questions where it is capable of getting the right answer, the Phind Model might take more generations to get to the right answer than GPT-4.
Some of this is sampler tuning. Y'all should look at grammar based sampling (https://github.com/ggerganov/llama.cpp/pull/1773) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and dynatemp: https://github.com/LostRuins/koboldcpp/pull/464
I think these should work with nvidia's implementation if you just swap the sampling out with the HF version.
BTW, all this is a great advantage of pulling away from OpenAI. You can dig in and implement experimental features that you just can't necessarily do through their API.
Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?
100 tokens/s on the user's end, on a host that is batching requests, is very impressive.
With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.
I suppose one could get decent utilization by prompt processing one user while generating tokens for another.
ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100:
https://github.com/turboderp/exllamav2#performance
I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my generation speed on a 3090.
100 tokens/s batched throughput (for each individual user) is much harder.
Especially for agents that do function calling.
And Cursor is just the start - there will be innovative workflows built on top of APIs you can't predict. You're missing out not having developers build an ecosystem for you.
[1] https://www.phind.com/search?cache=u3mnj3iwmjvgqlyf60bnbqo1
ChatGPT 4 does not take 50 seconds to answer, so I don't understand this comparison.
I've asked it how to scp a file on Windows 11 and it'll take a minute to tell me all the options possible.
If this takes 1/5th the time for equivalent questions, I'd consider switching
If so, consider adding one of the “just get to the point” prompts. GPT4’s defaults have been geared towards public acceptance through long-windedness which is imo entirely unnecessary when using it to do functional things like scp a file.
Also, use case thing. It is very likely the case that for certain coding use cases, Phind will always be faster because it's not designed to be general purpose.
By keeping the prompt short, it starts generating output quicker too.
Just trying to provide helpful feedback for you, this would have been a great comment, except for the "LOL" at the beginning that was unnecesary and demeaning.
ChatGPT4: 14 seconds
phind with "pair programmer" checked: 65 seconds
phind default: 16 seconds
https://imgur.com/a/iqxOJUV was 6.5 seconds.
https://imgur.com/a/pQFfWli was 15.
You can tell they're GPT-4 because the logo is purple (the logo is green when using 3.5).
It lets you specify verbosity from 1 to 5 (e.g. "V=1" in the prompt). Sometimes the model will just ignore that, but it actually does work most of the time. I use a verbosity of 1 or 2 when I just want a quick answer.
And I also can't wait to see how much Phind will improve further if the Glaive dataset is added onto it.
Edit: Contrastive search, dynamic temperatures.
Basically:
"How can I send network control commands to an AppleTV in C#"
They always make up some nonexistent library or gives an example using some nonexistent API.
“Make me a billionaire… I’m still poor! Bad AI!”
You need to collaborate with the AI, use it to help with each small step of the problem, with input references provided.
To a degree Phind can do the reference chasing for you, but it’s not magic.
Someone is doing it in python here:
https://pyatv.dev/
GPT-4 actually sent me here:
"Here is an example of a C# library that implements the HAP: CSharp.HomeKit (https://github.com/brutella/hkhomekit). You can use this library as a reference or directly use it in your project."
Which, to no surprise based on my experiences with LLMs for programming does not exist and doesn't seem to have ever existed.
I get that they aren't magic, but I guess I am just bad at trying to use LLMs to help in my programming. Apparently all I do are obscure things or something. Or I am just not good enough at prompting. But I feel like that's also a reflection of the weakness of an LLM in that it needs such perfect and specific prompting to get good answers.
Or you're good enough at using your tools that you can do all the low-hanging fruit. LLMs excel at working around inadequate tooling, but (at least at the moment) they can't help you if you're trying to do something actually tricky and get stuck enough that no rubber duck can save you.
Assuming a C# library even exists for what you’re doing (maybe not!) then still the best use of AI is to troubleshoot specific issues given an almost working piece of code as input.
Ask it to explain why something doesn’t work instead of asking it to do your job for you wholesale.
PS: GPT 4 (you are using the best coding AI, right? Right?) can get you going quickly:
“There are several libraries available for controlling Apple HomeKit from C#. One such library is *HapSharp* ². It is a .NET implementation of the HomeKit Accessory Server that allows you to create your own custom HomeKit accessory on a Raspberry Pi, Mac computer, or any other platform that can run Mono ².
Another option is *HomeKit* ¹. It is a native C# library for Apple's HomeKit Accessory Protocol. However, it is not a complete implementation and does not work ¹.
I hope this helps!
Source: Conversation with Bing, 31/10/2023 (1) netonjm/HapSharp: HomeKit Accessory Server .Net bridge! - GitHub. https://github.com/netonjm/HapSharp. (2) GitHub - ppumkin/HomeKit: Native C# Libary for Apple's HomeKit .... https://github.com/ppumkin/HomeKit. (3) homekit-accessory-protocol · GitHub Topics · GitHub. https://github.com/topics/homekit-accessory-protocol?o=asc&s...
AFAIK Apple generally does not allow arbitrary remote control (headless mode) for security reasons — it could be used for spam automation!
Really I just need to get an LLM to port pyatv to C# for me I guess.
I’m using it personally every day and while it still needs more work and polish, I’m finding it much better than ChatGPT or any other tools I’ve tried for bigger and more difficult tasks.
Please let me know if you (or anyone else reading this) would be interested to try a late alpha/early beta version: dane@envkey.com
But I want to control a unit like send navigation controls like 4 directions, back and select.
The only app I know that can do it is pyatv, (a python app) but I want to do it in my C# app.
It would be nice if an LLM could port pyatv to C# for me as I don't really know python at all.
Phind was really good, but still had a difficult time with library versions. Notably a lot of the search results it saw felt like they polluted it with incorrect assumptions about available methods on specific library versions. The web results felt like it made the LLM worse at some things. In the end i switched back to ChatGPT. Though i expect i'll retry Phind at some point, i do tend to ping pong on each respective release.
Does this version tackle that any better in your eyes?
The text box gets hidden after the conversation exceeds the page height
The results are impressive and things have been really progressing quickly, so kudos.
But even by your own description in this post, something like “rivals GPT-4 at coding” seems a more accurate appraisal.
https://www.phind.com/search?cache=ay8rx37gq8oy3z7uixftlqkt
https://chat.openai.com/share/a3a91dcc-a91a-4b04-8afd-40bd1a...
I think you might get a better answer if you rewrote your prompt using full sentences and more formal language.
> Llama 1 supports up to 2048 (2K) tokens, Llama 2 up to 4096 (4K), CodeLlama up to 16384 (16K). [0]
This is wild to me.
The token window is one of the limiting factors for having an AI that can actually remember you and past conversations. Having a large window is key for future AI applications that involve long running conversations (weeks, months, years). The tech is already very impressive, but imagine it as it becomes more like an actual pair programmer and remembers all the various things it's learned and worked on with you in the past.
[0] https://huggingface.co/docs/transformers/main/model_doc/llam...
It's just that, unlike these AIs, we're capable of online learning.
Something interesting that I heard from people trying to memorize things better is that memory “storage space” limits for people are essentially irrelevant. We’re limited by our learning and forgetting speeds. There’s no evidence of brains getting “full”.
Think of it like a giant warehouse of plants, with one employee. He can accept shipments (learning). He can take care of plants (remembering). Too long without care and they die (forgetting). The warehouse is big enough that it is not a limiting factor in how many plants he can keep alive. If it was 10x bigger it wouldn’t make a bit of difference.
For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring in relative obscurity.
HN, and the NLP community have some serious blindspots with knowing how to exploit their own technology. At least someone at Andreessen Horowitz got a clue and gave some funding to Oogabooga - still waiting for Automatic1111 to get any funding.
Also, I actually have several top NLP conference publications, so I'm not some charlatan when I say these things. I've actually physically used and seen these techniques improve LLM recall. It really actually works.
Here's more examples of low hanging fruit. The proof in that they work is in the implementations which I provide. You can run them, they work!: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
Check yourself before you try to check others.
You ain’t seen nuthin’ yet…
They do not. Sentence transformers aren't new, and have well-known trade offs. What source or line of reasoning misled you to believe otherwise?
> Here's more examples of low hanging fruit. The proof in that they work is in the implementations which I provide. You can run them, they work!: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
This...is your blog about prompt engineering. What do you believe this "proves"? How have you blown away current production encoding or attention mechanisms?
> Text Embeddings Reveal (Almost) As Much As Text. ... We find that although a naïve model conditioned on the embedding performs poorly, a multi step method that iteratively corrects and re embeds text is able to recover 92% of 32-token text inputs exactly. We train our model to decode text embeddings from two state of the art embedding models, and also show that our model can recover important personal information (full names) from a dataset of clinical notes.
https://arxiv.org/abs/2310.06816
On cast off/cheap workstation/server hardware.
There's an argument that music and the arts are dumbed down by the fact that, for instance, making an album worth $10 to millions of people pays way better than making an album worth a million dollars to tens of people, since the album is going to get priced at $10 one way or the other. It only just now occurred to me that the same phenomenon applies to tools.
And if you want vim/emacs to keep living, then you should spend time helping! Create your own extensions, maintain/contribute to existing ones, etc. They will only die out when the last person actively contributing to them stops, so keep the chain of people going :)
While I agree, at the very least IntelliJ stood up on its own as a good IDE. I cut my baby teeth on Eclipse, and as soon as I realised how good IntelliJ is, I jumped ship without looking back. The same can barely be said about VS Code.
https://github.com/huggingface/llm.nvim
:'<,'>y|call system('firefox <url>?q='.shellescape(@*).' &')
The only problem left is that the text is not urlencoded.
There probably is some elegant way to urlencode it. But I did not come up with one yet.
[1]: https://github.com/s-kostyaev/ellama
[2]: https://github.com/ahyatt/llm
Working with phind needs to be available in pycharm for me considering switching from gpt4 to phind. Chatting with phind on my local files is the feature I am looking for.