As someone who has utilized Nvidia Triton Inference Server for years it's really interesting to see people publicly disclosing use of TensorRT-LLM (almost certainly in conjunction with Triton).
Up until TensorRT-LLM Triton had been kind of an in-group secret amongst high scale inference providers. Now you can readily find announcements, press releases, etc of Triton (TensorRT-LLM) usage from the likes of Mistral, Phind, Cloudflare, Amazon, etc.
Every day now there are new AI models especially LLMs, which might warrant some consideration from a wide part of the human population. In a couple years we will have multiple new announcements per hour and we might need some earlier models to evaluate these new developments and test them. For Phind-70B in particular, I hope that lmsys will share a version that will be part of the human evaluation leaderboard so we get a rounded evaluation. But for code assistants there should be a totally separate impartial evaluation benchmark, ideally still human judged for another year or so but eventually maybe some way of having the models fighting out competitive coding battles that they can help create.
> In a couple years we will have multiple new announcements per hour
Models are research output. If 10 new models are being announced every day in a couple years, it would mean that generative AI research has failed to stabilize and produce a stable, reliable component ready for product engineering. And if that's where we are in a couple years, that's almost certainly a sign that the hype was misplaced and that money is chasing after itself trying to recoup sunk costs. That's a failure scenario for this technology, not what an AI-optimist (you otherwise seem to be one) should be anticipating.
That doesn't follow at all. It just means that there are still low-hanging fruits to pursue for better (smarter, faster, larger context etc) new models, but it doesn't say anything about the stability and usefulness of existing models.
That’s not true. Both good science and market-driven engineering favor continued iterations on existing ideas looking for improvements or alternatives. We’re often exploring a giant space of solutions.
Unlike many fields, the A.I. people are publicly posting many of their steps in this journey, their iterations, for review. While it brings lots of fluff, such openness dramatically increases innovation rate compared to fields where you only see results once or twice a year. Both people using cloud API’s and FOSS developers are steadily increasing effectiveness in both experimentation and product development. So, it’s working.
I referee for a lot of the top machine learning conferences and yes I am very optimistic about AI and its impact on humanity. The amount of exciting new papers in machine learning and AI was definitely on an exponential rise for a decade since about 2012 or so, and the total production has kept increasing even during the last couple of years when the submissions in some top annual conferences exceeded 10k. Not every paper results in a useable model but a higher fraction of papers come with code and pretrained weights over time. Many of these papers will never be read by many more than the reviewers and the group who wrote them and a couple friends, but it does not speak necessarily to the quality of the work itself or the potential impact it could have on every possible future if we found better ways to separate the useful information. As the exponential increase in total compute becomes more widely accessible there are exponentially more applications that are of broader interest and will have even bigger impact than nowadays. I don’t think that the model of reviewing 10s or 100s of thousands of papers in conferences, or playing the popularity contest on social media is going to be productive so we need better methods for advancing the useful ideas more quickly. (Case in point: the mamba state space model by Gu and Dao was rejected from a conference this winter, but it happened to be advertised enough at a keynote presentation by Chris Re with a packed audience at neurIPS23, so the model was picked up by a lot of people who used it and submitted applications that used it to the ICML conference already.) I also don’t think that some of the biggest companies have enough manpower, motivation and interest in going alone, though of course they can easily stay ahead of the game in specialized areas with their own resources.
> LLMs are more like apps being produced by different companies trying to capture walled gardens, and their open source counterparts.
I think the analogy to the web is stronger than that.
For now the LLMs are mostly separate, but it won't be long before LLMs emerge that make API calls to other LLMs, sometimes over the internet.
In due course, expect meta-LLMs to emerge that aggregate knowledge from other LLMs by talking to them, rather than by training on their data. Those meta-LLMs which optimise for competitive quality results will have to read the research as it comes out, and continually assess which other new LLMs are worth calling out to, and for which purposes. Eventually the API calls will become bi-directional requests to exchange knowledge and insights, i.e. multiple models talking to each other, continually learning.
Yup, that works (10 uses avail). Though i wasn't too concerned with actually using it, just thought it was interesting and wanted to expose that maybe-bug.
I understand why they’re doing this from a cost and dependency perspective, but I’ve pretty much stopped using Phind since they switched over to their own models. I used to use it in the past for thing like API docs summarization, but it seems to give mostly wrong answers for that now. I think this is mostly a “RAG doesn’t work very well without a very strong general model parsing the context” problem, which their prior use of GPT-4 was eliding.
I used it for awhile and it was pretty good at Bash or Emacs Lisp one-liners but it was wrong often enough that it was faster to just search on Kagi for the information that I want first, instead of performing N searches to check the answer from Phind after querying Phind.
Phind founder here. Thanks for the feedback -- I'd love to hear your thoughts on this new model. You can try it for free, without a login, by selecting it from the homepage: https://phind.com.
I just tried using the 70B model and the answer was listed as being returned using the 34B model instead of the 70B model and was wrong. Is there some logic that ignores user choice, depending on what the service thinks can be answered?
Fellow ST4 user checking in. It does everything VSCode does (minus remote development, which I don't need) with 1/4 of the resource usage. Just a quality piece of software that I'll keep using for as long as I can.
I also paid for ST3, but I switched to ST4 for the hardware accelerated rendering. I don't like their licensing policy anymore, so I just dismiss the purchase signs. I want to get my company to pay for it because paying 100 USD _again_ is just absurd for me.
I have no plans to switch off though. This is still a heck of a lot faster than the alternatives
There are dozens of us! Though for serious work I'll sometimes reluctantly switch to VSCode due to Sublimes language integrations always feeling hacked on.
And lately Sublime has been mysteriously freezing and crashing my other programs (though it might be Windows' fault, unclear) so I've reluctantly started developing my own editor...
VSCode used to be great, but now it feels garbage, or was it garbage all the time?
I used it because it was faster than WebStorm, but WebStorm was always just better. Now it seems VSCode is as slow as WebStorm, but is still garbage in everything.
They recently made it so you can drag tabs into their own windows (the issue was open for a decade), which makes it actually a respectable editor (despite the startup lag).
The one thing we've learnt from the past few months of LLM optimization is that model size is no longer the most important thing in determining LLM quality.
A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
It kinda is, if you want not just performance on synthetic benchmarks but a good coverage of the long tail. This is where GPT4 excels, and also why I pay for it. Transformers are basically fancy associative memories. A smaller model, much like a smaller search index, will not be able to contain as much nuanced information for some hard, immutable, information theoretic reasons.
I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.
I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
It's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination.
If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
I've found that GPT4 (via GitHub Copilot) and Gemini models are better at code tasks like reviewing for logical and functional errors, reasoning about structure and test/edge cases, and refactoring. Gemini is capable of devouring some very large files I've thrown at it.
Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
Phind-70B is a specialist model, unlike GPT-4. It optimizes for a different function than GPT-4 and therefore needs fewer parameters to learn it.
It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.
I mean, it could be as good or better at a lot of reasoning related tasks and just have less baked in general knowledge, in which case it'd make an amazing RAG model if the context length is reasonable.
We noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal damage. We were able to swap in a different node, but apparently the entire host server needed to be replaced.
We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.
Very nice. I've been working with GPT4 since it released, and I tried some of my coding tasks from today with Phind-70B. The speed, conciseness, and accuracy are very impressive. Subjectively, the answers it gives just feel better than GPT4, I'm definitely gonna give pro a try this month.
I prefer Phind's web search with LLM to both Google search and GPT-4. I have switched my default search engine, only using Google for finding sites, not for finding information anymore.
GPT-4 might be a better LLM but its search capability is worse, sometimes sends really stupid search keywords that are clearly not good enough.
I won’t steal phind’s thunder but kagi is another great modern tool to have, and much more reliable than google for a technical user IMO. Obviously Phind is irreplaceable for complex or chat-based technical questions, but Kagi sees much more use from me daily for syntax stuff, Wikipedia searches, finding and relating papers, etc.
I'd suggest logging in in that case -- you will still get your free uses. The Phind-70B counter for non-logged in users has carried over from when we offered GPT-4 uses without a login. If you've already consumed those uses, you'll need to log in to use Phind-70B.
I tried a question about Snobol4 and was impressed with what it said (it couldn't provide an exact example due to paucity of examples). When testing more mainstream languages I have found it very helpful.
Phindational models, phintech, Phinterest, phinder… it might be the best startup name of all time. Hell, startup a password manager and call it Phinders’ Keeper.
I don't use LLMs a lot, maybe once a week or so. But I always pick Phind as my first choice because it's not behind a login and I can use it without giving my phone number. Hopefully you'll keep it that way!
I think there’s room in the market to subsidize real users. Phind delivers absurd value, so I think the majority of paying users could account for the tech-averse or privacy-conscious
Since you're here: have you considered moving to other, better generalist base models in the future? Particularly Deepseek or Mixtrals. Natural language foundation is important for reasoning. Codellama is very much a compromise, it has lost some NLP abilities from continued pretraining on code.
Hello Michael, lovely to see this, congrats. Do you already have an API? I could not see it on the site. If not, then do you know around when we can expect it? I am building a desktop BI app with hosted and local LLMs (need schema inference and text to SQL). Would be nice to have Phind as an option for users. Thanks
I tried asking "What is the size of Phind-70B's context window?" and it couldn't answer the question. Strangely, it immediately found the page with the answer (https://www.phind.com/blog/introducing-phind-70b) but refused to acknowledge that the answer was there. I tried asking several ways. It even quoted the exact answer in the displayed snippet, but still said there was no answer!
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak.
If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.
- Hybrid offloading with llama.cpp, but with slow inference.
- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.
30B-34B is more of a sweetspot for 24GB of VRAM.
If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.
After running a bunch of models on my own PC (a pretty good one), I have to say by FAR the best results for coding has been with Deepseek models. However, I just spent 20 minutes playing with this Phind 70B model and it's totally nailing the questions I'm asking it. Pretty impressed.
I don't trust the code quality evalution. The other day at work I wanted to split my string by ; but only if it's not within single quotes (think about splitting many SQL statements). I explicitly asked for stdlib python solution and preferrably avoid counting quotes since that's a bit verbose.
GPT4 gave me a regex found on https://stackoverflow.com/a/2787979 (without "), explained it to me and then it successfully added all the necessary unit tests and they passed - I commited all of that to the repo and moved on.
I couldn't get 70B to answer this question even with multiple nudges.
Every time I try something non GPT-4 I always go back - it's feels like a waste of time otherwise. A bit sad that LLMs follow the typical winner-takes-it-all tech curve. However if you could ask the smartest guy in the room your question every time, why wouldn't you?
Thanks for the feedback, could you please post the cached Phind link so we can take a look?
It might also be helpful to try Phind Chat mode in cases like this.
EDIT: It seems like Phind-70B is capable of getting the right regex nearly every time when Chat mode is used or search results are disabled. It seems that the search results are polluting the answer for this example, we'll look into how to fix it.
You're right! It solved it. I didn't know about the Code/Search distinction. I still struggled for it to write me the unit tests. It does write them, they just don't pass. But this is definitely much closer to GPT4 than I originally thought.
I've tried it with a question which requires deeper expertise – "What is a good technique for device authentication in the context of IoT?" – and the Search mode is also worse than the Chat mode:
The search was heavily diluted by authentication methods that don't make any sense for machine-to-machine authentication, like multi-factor or biometric authentication, as well as the advice to combine several methods. It also falls into the, admittedly common, trap of assuming that certificate based authentication is more difficult to implement than symmetric key (i.e. pre-shared key) authentication.
The chat answer is not perfect, but the signal-to-noise ratio is much better. The multi-factor authentication advice is again present, but it's the only major error, and it also adds relevant side-topics that point in the right direction (secure credential storage, secure boot, logging of auth attempts). The Python example is cute, but completely useless, though (Python for embedded devices is rare and in any case you wouldn't want a raw TLS socket, but use it in a MQTTS / HTTPS / CoAP+DTLS stack, and last but not least, it provides a server instead of client, even though IoT devices mostly communicate outbound).
Agreed, though i'm _really_ interested in trying 1M token Gemini. The idea of uploading my full codebase for code assist stuff sounds really interesting. If i can ever get access to the damn thing...
I'm curious how they'll handle this. My understanding is that it takes quite a long time to get an answer, since there's no magic "semantic database" built for you behind the scenes.
That use-case seems inefficient to solve like that in the long run as well, like if you really would have to use a million tokens to do every small query you require on your data it would be prohibitively costly except doing as an experiment.
In my experience, Bing's image search is way better than Google's. Also, I'm not going to use a search engine that I have to log in or do a captcha for.
Gemini is much better than the free version of GPT 3.5 though. At least in my experience.
Microsoft’s enterprise co-pilot is also fairly decent. It’s really good at providing help to Microsoft related issues or helping you find the right parts of their ridiculously massive documentation site. Which probably isn’t too weird considering.
"zsh rename index.html.1 to image_1.png for many images"
Gemini
Here's a zsh script that will rename multiple files named index.html.1 to
image_1.png in your current directory and its subdirectories:
Bash
#!/bin/zsh
# Loop through all files in the current directory and subdirectories
for file in **/index.html.1; do
# Extract the base filename and extension
base_name="${file%.*}"
extension="${file##*.}"
# Rename the file with the new name and extension
mv "$file" "${base_name}_1.png"
# Print a message for each renamed file
echo "Renamed $file to ${base_name}_1.png"
done
ChatGPT3.5
To rename multiple files in zsh, you can use the zmv function. Assuming you
want to rename files from index.html.1 to image_1.png for many images, you
can create a pattern to match the filenames and then define the new names
using capturing groups. Here's an example:
zsh
autoload -U zmv
zmv -n '(*).html.(*)' 'image_$1_$2.png'
Not a great first impression of Gemini. ChatGPTs answer isn't perfect but its a lot closer to correct, only needing me to remove the extra 'index' capture of $1.
Curious if someone could commit some light copyright infringement and post what GPT4 says to the same prompt.
Edit: Also Phind-34B probably gives the best answer, with the correct capture.
To rename multiple files in zsh from index.html.1 to image_1.png for many
images, you can use the zmv command, which is a powerful batch renaming tool
available in zsh. If you encounter the error zsh: command not found: zmv, you
can load it using autoload -U zmv 1.
First, ensure zmv is available by adding autoload -U zmv to your ~/.zshrc file
or run it directly in your shell session 2.
Here's how you can use zmv to rename your files:
zmv -n 'index.html.(*)' 'image_$1.png'
Here's what gpt4-turbo-preview outputs (with max output of 256 tokens, so the result was truncated).
P.S. have you tried testing what happens when you clearly describe what you want? The prompt you're using is really low quality - more like a google search. If you asked me a question like that I'd tell you to clearly explain what it is you want.
If you have multiple files with names like `index.html.1`, `index.html.2`, etc., and you want to rename them to `image_1.png`, `image_2.png`, and so on, using Zsh, you can accomplish this with a simple loop and the `mv` command.
Here is how you could do it in a Zsh terminal:
```zsh
autoload -U zmv
zmv -W 'index.html.*' 'image_*.png'
```
The `autoload -U zmv` command loads the `zmv` function, which is a powerful utility in Zsh for batch renaming files. The `-W` option tells `zmv` to use wildcard patterns for matching and renaming files.
In the `zmv` command:
- `'index.html.*'` is the pattern to match. It matches any file that starts with `index.html.` followed by any characters.
- `'image_*.png'` is the replacement pattern. The asterisk `*` in the replacement pattern corresponds to the `*` in the match pattern, so each matched number after `index.html.` gets placed where the `*` is in `image_*.png`.
**Important:** Always make sure
The time complexity for all matching a string against any fixed regular expression is O(length of string).
If you want to talk about constant factors, we need to leave our comfortable armchairs and actually benchmark.
[Just to be clear, I am talking about real regular expressions, not Franken-xpressions with back-references etc here. But what the original commenter described is well within the realm of what you can do with regular expressions.]
You are right about escaped quotes etc. That's part of why parsing with regular expressions is hard.
The time complexity for deciding whether an N-letter string matches a regex or not, is O(N). The time complexity of finding all matches is not O(N) - which is needed in OPs case, because they want to split the string.
Also, OP's solution uses lookahead assertions, so it's not a real regular expression.
(I wonder if we can summon @burntsushi for expert opinion on this?)
You are right that the lookahead might be expensive.
(There's probably a way that a sufficiently smart compiler of (ir-)regular expressions can optimize this expression to be still matchable quickly; but Python's regular expression matcher is probably not that smart. I'm not sure if any real world matcher is.)
> The time complexity of finding all matches is not O(N) [...]
If you are happy find a maximal set of non-overlapping matches, you can still do it in O(N). By 'maximal' I mean that you can't greedily find another match (without removing any of the existing matches.)
A sketch of the technique is: take your pattern and wrap it up like this '.?{pattern.?}' (where ? means non-greedy repetition) and match that against your input string. You can do non-greedy repetition and the very limited form of sub-pattern capturing that you need to find all the matches of 'pattern' without breaking O(N) time.
I'm not sure whether you can find the global maximum number of non-overlapping matches, instead of a just a greedy maximum, in O(N) time.
I didn't take a look at the code, but to me it sounds quite dangerous to take an implementation AND the unit tests straight from an LLM, commit and move on.
You’ve still got to avoid prompting for questionable code in the first place, eg, splitting SQL statements on semicolons with an ad-hoc regex is going to fail in edge cases, but may be sufficient for a specific task.
Yes more than sufficient for an internal tool - we can assume good intentions of the users of the tool since people want for this to actually work and have no intention of hacking.
Except now it's a vector if anyone gets access to this internal tool.
I would be fine with this for one off scripts but absolutely can not consider anything less than full sql parsing or something equally robust if it is exposed over the network, even if only internally and behind authn and authz.
For this reason, I tend to ask LLMs additional questions like: "show me another way to do this" or specifically "how would someone with a higher need for security write this?"... knowing that I'm likely to get a more refined answer from different sources that have probably discussed deeper security implications around the same goals, for instance.
Yeap, I want a code-review bot that just says "this is very improbable; are you sure you didn't mean x instead?"
The old Coverity used to achieve similar results in a different way, spotting probable mistakes based on patterns its heuristics found in the rest of the same codebase.
Right on. These days my llm-assisted workflow feels very similar to the 20% of my day that I used to devote to code review, just now it’s more like 60% of my day.
I’m finding it’s more effective (and pleasurable) to write using GitHub CoPilot and CMD-RIGHT (accept next word). I put a detailed doc comment above and write in tandem with copilot. I’ve written the structure and I review as I write jointly with the model.
This way I don’t need to review a block of code I didn’t write.
<aside>I had an experience yesterday where CoPilot correctly freed all the memory in correct order at the end of a rather complicated C algorithm, even where there was nested mallocs.</aside>
I take it to mean that the code quality deserves more scrutiny because you can't guarantee what it has provided is quality code, without reviewing it first.
The same applies to brand new devs — it's normal to apply a little more scrutiny because they simply don't have the experience to make the right decisions as confidently (or frequently) as someone more senior.
It's an analogy and the natural fact that output reflects experience and practice over time.
Reminds me of a Facebook thread I saw a few days ago, on the topic of 3D printing houses. All the comments were angry dismissive "hurr durr that's clearly poor quality work" with no further justification of their position, and it struck me how similar the overall energy was to the "all AI image generation is bad and shit and is also heinous immoral theft and you're literally the worst person in the world and yous should feel bad" sort of raging that you see any time someone posts some SD or Midjourney or whatever pic of a cute puppy riding a tricycle. These comments originate from people who've spent their lives learning skills that are now largely replaceable by a few gigs of download and a Python tutorial. No wonder they're upset.
Right...because requiring developers to actually grasp the code they're outputting is gatekeeping.
If you want to pretend the rush to AI won't lead to more incompetent chefs in the kitchen than we already have (which is too many as it stands) then feel free, but acting like it's some kind of "party" people are being kept out of is daft.
Standards exist for a reason, not just to make people feel bad for not meeting them.
It's very powerful, I can enter implementations for any algorithm by typing 5 words and clicking tab. If I want the AI to use a hashmap to solve my problem in O(n), I just say that. If I need to rewrite a bunch of poorly written code to get rid of dead code, add constants, etc I do that. If I need to convert files between languages or formats, I do that. I have to do a lot more code review than before, and a lot less writing. It saves a huge amount of time, it's pretty easy to measure. Personally, the order of consultation is Github Copilot -> GPT4 -> Grimoire -> Me. If it's going to me, there is a high probability that I'm trying to do too many things at once in an over-complicated function. That or I'm using a relatively niche library and the AI doesn't know the methods.
Hopefully not, I feel it's a waste of time. The time spent on stupid minor mistakes by github copilot I didn't catch probably doesn't really compare to the time I would've spent typing on my own. (I only use that stuff for fancy code completion, nothing more. Every LLM is absolutely moronic. Yesterday I asked chatgpt to convert gohtml to templ, to no avail ...)
"Can you give me an approach for a pathfinding algorithm on a 2D grid that will try to get me from point A to point B while staying under a maximum COST argument, and avoid going into tiles that are on fire, except if no other path is available under the maximum cost?"
I've never found an AI that could solve this, because there's a lot of literature online about A* and tiles with cost, and solving this requires a different approach
I used to use Phind for couple of months. I liked the UI improvements but the slow limited free GPT4 and fast lackluster Phind model turned me off. I tried Bing and it wasn’t worse, had more free searches per day.
I tried the model and asked it to write a kubernetes operator with required DockerFiles, Resources, application code.. Asked it to migrate application to different languages. It looks like it's pretty capable and fast. It is impressive.
Thank you for your excellent work. Can you let us know when the 70B weights will be released? I'm really looking forward to trying them out with my own coding project.
Terrific stuff. I always enjoy using Phind for dev related questions.
Is it possible the chat history gets some product love? I would like to organize my conversations with tags, and folders. Make it easier to go back to what was said in the past instead of asking the question again.
Can we get a few accessibility fixed? The expandable button after the sign in button and the button after that are unlabeled. The image on the heading at level 1 has no Alt-text. The three buttons after the "Phind-34B" button are not labeled. The ones between that and the suggestions. On search results, there's an unlabeled button after each one, followed by a button labeled something like " search cache=tbo0oyn4s955gf03o…".
There's probably more, but hopefully that should get things started if you can fix these.
I was considering signing up for the pro plan. Now I won’t even give them my email. I tried the model and it is genuinely nice, but this is a huge red flag.
Phind makes impressive claims. They also claimed that their fine tune of codellama beat gpt4, but their finetune is miles behind gpt4 in open domain code generation.
Not impressed. Also this is a closed walled garden model.
Thank you for the reply, I'd like to congratulate you on the release, first. I'm a bit of a minimalist with regard to signups, unfortunately, so unless this is a known limit then I'd likely just spectate the thread and be happy for you from a distance.
They can be, but like with everything copyright-related for copyright to apply there need to be “creative work” involved. Which, for something that has been translated countless times in all possible directions, is going to be much harder than for a first translation.
I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs:
> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.
GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.
Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.
Thanks, can you send the cached link please? I'd also suggest trying Chat mode for questions like this, where there are unlikely to benefit from an internet search.
Just tried your query now and it seemed to work well -- what are your thoughts?
Thanks for the links. It seems like it switched to Phind-34B, which is worse.
Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.
I'm not sure what you mean that it "forgot" about POST? Even as an experienced Go developer, I looked at the code and thought it would probably work for both GET and POST. I couldn't easily see a problem, yet I had not forgotten about POST being part of the request. It's just not an obvious problem. This is absolutely what I would classify as a "brain teaser". It's a type of problem that makes an interviewer feel clever, but it's not great for actually evaluating candidates.
Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out.
I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized.
I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.
This isn't a brain teaser at all. It's a direct test of domain knowledge/experience.
There are countless well-documented RoundTripper implementations that handle this case correctly.
This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.
Does anyone outside the Go community call it a "RoundTripper"? I know what a retry is (and things like exponential backoff) and what GET and POST are, but not that, but I also hate Go, so...
EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing
A fun little challenge I like to give LLMs is to ask some basic logic puzzles, i.e. how can I measure 2 liters using a 3 liter and a 5 liter container? Usually if it's possible, they seem to do ok. When it's not possible, they produce a variety of wacky results. Phind-34B is rather amusing, and seems to get stuck in a loop: https://www.phind.com/agent?cache=clsxpravk0001la081cc9dl45
These are interesting tests. I wonder how far we are away from AIs solving these (the ones that have no solution) without any special programming to teach them how.
1. phind was by far the best - gave me solution in just 2 steps
2. Grok was second best - it did arrive at the solution but with additional non-sense step. But the solution was correct.
3. To my surprise GPT-4 could not solve the prompt and in fact gave a wrong answer in 4 steps - "Now you should have exactly 4 liters in the 5-liter container." which is not what I asked
4. As expected Gemini pro was the worst. It asks me to pour completely filled up 3L container into 5L and then you will be left with 2L in 3L container.. LOL that does not even make sense.
295 comments
[ 2.9 ms ] story [ 269 ms ] threadUp until TensorRT-LLM Triton had been kind of an in-group secret amongst high scale inference providers. Now you can readily find announcements, press releases, etc of Triton (TensorRT-LLM) usage from the likes of Mistral, Phind, Cloudflare, Amazon, etc.
I still see post of people running ollama on H100s or whatever, and that's just because its so easy to set up.
https://www.youtube.com/watch?v=dJ69gY0qRbg
Models are research output. If 10 new models are being announced every day in a couple years, it would mean that generative AI research has failed to stabilize and produce a stable, reliable component ready for product engineering. And if that's where we are in a couple years, that's almost certainly a sign that the hype was misplaced and that money is chasing after itself trying to recoup sunk costs. That's a failure scenario for this technology, not what an AI-optimist (you otherwise seem to be one) should be anticipating.
Unlike many fields, the A.I. people are publicly posting many of their steps in this journey, their iterations, for review. While it brings lots of fluff, such openness dramatically increases innovation rate compared to fields where you only see results once or twice a year. Both people using cloud API’s and FOSS developers are steadily increasing effectiveness in both experimentation and product development. So, it’s working.
LLMs are more like apps being produced by different companies trying to capture walled gardens, and their open source counterparts.
I think the analogy to the web is stronger than that.
For now the LLMs are mostly separate, but it won't be long before LLMs emerge that make API calls to other LLMs, sometimes over the internet.
In due course, expect meta-LLMs to emerge that aggregate knowledge from other LLMs by talking to them, rather than by training on their data. Those meta-LLMs which optimise for competitive quality results will have to read the research as it comes out, and continually assess which other new LLMs are worth calling out to, and for which purposes. Eventually the API calls will become bi-directional requests to exchange knowledge and insights, i.e. multiple models talking to each other, continually learning.
Context: I used to be a Phind Pro subscriber, but I've not used Phind in probably two months.
https://www.phind.com/agent?cache=clsxnhahk0006jn08zjvcgc9g
https://chat.openai.com/share/ec5bad29-2cda-48b5-9aee-da9149...
I have no plans to switch off though. This is still a heck of a lot faster than the alternatives
And lately Sublime has been mysteriously freezing and crashing my other programs (though it might be Windows' fault, unclear) so I've reluctantly started developing my own editor...
I used it because it was faster than WebStorm, but WebStorm was always just better. Now it seems VSCode is as slow as WebStorm, but is still garbage in everything.
It will be interesting to hear from other people why they do not like VSCode for data science related tasks.
My extensions is still there and I can access everything through shortcuts or the command palette.
Yeah, right.
A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.
I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
I pay for it and for chatGPT and I find copilot much worse.
For code review, I tend to engage Copilot Chat which probably uses GPT4 more often? https://github.com/orgs/community/discussions/58059#discussi...
It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.
We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.
https://forums.developer.nvidia.com/t/ada-geforce-rtx-4090-f...
GPT-4 might be a better LLM but its search capability is worse, sometimes sends really stupid search keywords that are clearly not good enough.
0 Phind-70B uses left
And I've never made any selection there.
And are there plans to release any more weights? Perhaps one or two revisions behind your latest ones?
[0] https://www.phind.com/plans
I'm not sure if it's really using the 34B model or if the UI is wrong about which one it used
Here are a couple screenshots:
https://imgur.com/a/u7iKOyw https://imgur.com/a/aHAto5H
And here's the link to the whole conversation:
https://www.phind.com/search?cache=zlaksmzkm0h5cpx8l95n62tl
Why is this happening? Does it generally have difficulty with reading web pages, or is there something strange about this particular question?
If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its just begging for a continued pretrain like that, and it will be significantly faster on 80GB cards.
What's best that can run fast on 4090 laptop?
- Hybrid offloading with llama.cpp, but with slow inference.
- Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context.
30B-34B is more of a sweetspot for 24GB of VRAM.
If you do opt for the high quantization, make sure your laptop dGPU is totally empty, and that its completely filled by the weights. And I'd recommend doing your own code focused exl2/imatrix quantization, so it doesn't waste a megabyte of your vram.
As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.
GPT4 gave me a regex found on https://stackoverflow.com/a/2787979 (without "), explained it to me and then it successfully added all the necessary unit tests and they passed - I commited all of that to the repo and moved on.
I couldn't get 70B to answer this question even with multiple nudges.
Every time I try something non GPT-4 I always go back - it's feels like a waste of time otherwise. A bit sad that LLMs follow the typical winner-takes-it-all tech curve. However if you could ask the smartest guy in the room your question every time, why wouldn't you?
---
Edit: USE CODE MODE and it'll actually solve it.
It might also be helpful to try Phind Chat mode in cases like this.
EDIT: It seems like Phind-70B is capable of getting the right regex nearly every time when Chat mode is used or search results are disabled. It seems that the search results are polluting the answer for this example, we'll look into how to fix it.
For writing/manipulating code, Chat mode might work better than Search.
- Search: https://www.phind.com/search?cache=s4e576jlnp1mpw73n9iy4sqc
- Chat: https://www.phind.com/agent?cache=clsyev95o0006le08b5pjrs14
The search was heavily diluted by authentication methods that don't make any sense for machine-to-machine authentication, like multi-factor or biometric authentication, as well as the advice to combine several methods. It also falls into the, admittedly common, trap of assuming that certificate based authentication is more difficult to implement than symmetric key (i.e. pre-shared key) authentication.
The chat answer is not perfect, but the signal-to-noise ratio is much better. The multi-factor authentication advice is again present, but it's the only major error, and it also adds relevant side-topics that point in the right direction (secure credential storage, secure boot, logging of auth attempts). The Python example is cute, but completely useless, though (Python for embedded devices is rare and in any case you wouldn't want a raw TLS socket, but use it in a MQTTS / HTTPS / CoAP+DTLS stack, and last but not least, it provides a server instead of client, even though IoT devices mostly communicate outbound).
Microsoft’s enterprise co-pilot is also fairly decent. It’s really good at providing help to Microsoft related issues or helping you find the right parts of their ridiculously massive documentation site. Which probably isn’t too weird considering.
"zsh rename index.html.1 to image_1.png for many images"
Gemini
ChatGPT3.5 Not a great first impression of Gemini. ChatGPTs answer isn't perfect but its a lot closer to correct, only needing me to remove the extra 'index' capture of $1.Curious if someone could commit some light copyright infringement and post what GPT4 says to the same prompt.
Edit: Also Phind-34B probably gives the best answer, with the correct capture.
P.S. have you tried testing what happens when you clearly describe what you want? The prompt you're using is really low quality - more like a google search. If you asked me a question like that I'd tell you to clearly explain what it is you want.
I see that the future is brighter than ever for the information security industry.
If you want to talk about constant factors, we need to leave our comfortable armchairs and actually benchmark.
[Just to be clear, I am talking about real regular expressions, not Franken-xpressions with back-references etc here. But what the original commenter described is well within the realm of what you can do with regular expressions.]
You are right about escaped quotes etc. That's part of why parsing with regular expressions is hard.
Also, OP's solution uses lookahead assertions, so it's not a real regular expression.
(I wonder if we can summon @burntsushi for expert opinion on this?)
(There's probably a way that a sufficiently smart compiler of (ir-)regular expressions can optimize this expression to be still matchable quickly; but Python's regular expression matcher is probably not that smart. I'm not sure if any real world matcher is.)
> The time complexity of finding all matches is not O(N) [...]
If you are happy find a maximal set of non-overlapping matches, you can still do it in O(N). By 'maximal' I mean that you can't greedily find another match (without removing any of the existing matches.)
A sketch of the technique is: take your pattern and wrap it up like this '.?{pattern.?}' (where ? means non-greedy repetition) and match that against your input string. You can do non-greedy repetition and the very limited form of sub-pattern capturing that you need to find all the matches of 'pattern' without breaking O(N) time.
I'm not sure whether you can find the global maximum number of non-overlapping matches, instead of a just a greedy maximum, in O(N) time.
Is this the new normal now?
You’ve still got to avoid prompting for questionable code in the first place, eg, splitting SQL statements on semicolons with an ad-hoc regex is going to fail in edge cases, but may be sufficient for a specific task.
Yes more than sufficient for an internal tool - we can assume good intentions of the users of the tool since people want for this to actually work and have no intention of hacking.
I would be fine with this for one off scripts but absolutely can not consider anything less than full sql parsing or something equally robust if it is exposed over the network, even if only internally and behind authn and authz.
The old Coverity used to achieve similar results in a different way, spotting probable mistakes based on patterns its heuristics found in the rest of the same codebase.
This way I don’t need to review a block of code I didn’t write.
<aside>I had an experience yesterday where CoPilot correctly freed all the memory in correct order at the end of a rather complicated C algorithm, even where there was nested mallocs.</aside>
The same applies to brand new devs — it's normal to apply a little more scrutiny because they simply don't have the experience to make the right decisions as confidently (or frequently) as someone more senior.
It's an analogy and the natural fact that output reflects experience and practice over time.
If you want to pretend the rush to AI won't lead to more incompetent chefs in the kitchen than we already have (which is too many as it stands) then feel free, but acting like it's some kind of "party" people are being kept out of is daft.
Standards exist for a reason, not just to make people feel bad for not meeting them.
Blindly copying code from any source and running it or committing it to your main branch without even the slightest critical glance is foolish.
But if there non-trivial logic in the code of the tests, I agree this is probably a risky approach.
"Can you give me an approach for a pathfinding algorithm on a 2D grid that will try to get me from point A to point B while staying under a maximum COST argument, and avoid going into tiles that are on fire, except if no other path is available under the maximum cost?"
I've never found an AI that could solve this, because there's a lot of literature online about A* and tiles with cost, and solving this requires a different approach
I suppose you are not releasing the weights, right? Anyway, good luck! I hope investors are already forming a nice queue before your door :)
We will eventually release the weights.
Is it possible the chat history gets some product love? I would like to organize my conversations with tags, and folders. Make it easier to go back to what was said in the past instead of asking the question again.
Thanks!
There's probably more, but hopefully that should get things started if you can fix these.
https://www.phind.com/agent?cache=clsxs6doj000wl008yk8wb4k8
It pointed out the lack of alt-text as well as a couple other issues. Some of the suggestions aren't applicable, but it's not bad as a starting point.
* https://www.phind.com/search?cache=rj4tpu6ut0jyzkf876e2fahh
The answer is 'lower' because the weight of the ball as a volume of water is larger than the volume of the ball.
Not impressed. Also this is a closed walled garden model.
the answer was good. two follow up answers were also fine.
just curious: what about the copyright status of the given sources?
the best result I received so far was with MS Bing app (android).
had reasonable results with my local llama2 13B.
cheers
OpenAI's leaked prompt literally encourages it to try harder[1]:
> Use high effort; only tell the user that you were not able to find anything as a last resort. Keep trying instead of giving up.
1: https://pastebin.com/vnxJ7kQk
> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.
GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.
Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.
Just tried your query now and it seemed to work well -- what are your thoughts?
https://www.phind.com/search?cache=tvyrul1spovzcpwtd8phgegj
https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x
I'll give chat mode a try. Didn't see that it existed until now.
EDIT
Chat mode didn't do much better:
https://www.phind.com/agent?cache=clsxpl4t80002l008v3vjqw5j
For the record, this is the interface I asked it to implement:
https://pkg.go.dev/net/http#RoundTripper
Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.
Phind still forgot about POST, but at least now it got the interface right.
https://www.phind.com/search?cache=ipu8z1tb3bnn7nfgfibcix38
Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out.
I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized.
I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.
Eg. Tree of thoughts, ...
There are countless well-documented RoundTripper implementations that handle this case correctly.
This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.
EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing
1. phind was by far the best - gave me solution in just 2 steps
2. Grok was second best - it did arrive at the solution but with additional non-sense step. But the solution was correct.
3. To my surprise GPT-4 could not solve the prompt and in fact gave a wrong answer in 4 steps - "Now you should have exactly 4 liters in the 5-liter container." which is not what I asked
4. As expected Gemini pro was the worst. It asks me to pour completely filled up 3L container into 5L and then you will be left with 2L in 3L container.. LOL that does not even make sense.