To me it would be even more responsible if they would define a default output license compatible to training input of the model. I guess they could simply refer to the input data set in there to fulfill attribution clauses of most permissive licenses. Or extract some really long list of copyright notices from the sources.
Well rather than "ethical" licensing, I'd argue this is "moral" licensing, and as usual, morals differ. For example:
> (m) To provide medical advice or medical results interpretation that is intended to be a substitute for professional medical advice, diagnosis, or treatment;
This seems particularly overly broad -- there is a lot of benefit for people with no access to a doctor of any kind that is left on the ground by excluding that use case.
The problem of bad advice is hard, but it's one that will likely become increasingly solved as the tech evolves, and this seems to exclude this model from that technological progression.
Does this work for second degree usecases? Since this a codegen model, can I output code that does something like fast-edge detection, put it under MIT, and then use it for medical analysis?
Can someone explain to me the "Model Formats" section at the end of that page. It makes sense to me as a description of how to format the training data, but it also says "Use these templates to explore the model’s capacities"
How would you make use of this prefix format in a prompt?
You may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion.
StarCoder is not just a code predictor, it is an assistant. It doesn’t just predict code; it can also help you review code and solve issues using metadata, thanks to being trained with special tokens.
The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others.
The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue.
The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor.
Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.
I'm excited to see all the model's implementations that are yet to come. Refact.ai seems to be working on code transformation tools and chat inside Jetbrains and VSCode powered by StarCoder https://refact.ai/blog/2023/self-hosted-15b-code-model/
There is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs.
But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.
You are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).
37 comments
[ 2.5 ms ] story [ 81.4 ms ] threadRelease thread: https://twitter.com/BigCodeProject/status/165417494197606811...
Model: https://huggingface.co/bigcode/starcoder
Paper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k...
Also available in HuggingChat: https://huggingface.co/chat
The performance of StarCoder seems superior on HumanEval pass@1:
• replit-code-v1-3b: 21.9%
• replit-finetuned-v1-3b: 30.5%
• StarCoder: 33.6%
• StarCoder prompted: 40.8%
Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.
Evaluation results are impressive!
Just curious what that is vs the main https://huggingface.co/bigcode/starcoder model
It could have other uses, but its not what you want code generation. For code generation, use StarCoder or StarCoderBase.
OK for some commercial uses but I wonder how many other models have restriction lists like this
To me it would be even more responsible if they would define a default output license compatible to training input of the model. I guess they could simply refer to the input data set in there to fulfill attribution clauses of most permissive licenses. Or extract some really long list of copyright notices from the sources.
> (m) To provide medical advice or medical results interpretation that is intended to be a substitute for professional medical advice, diagnosis, or treatment;
This seems particularly overly broad -- there is a lot of benefit for people with no access to a doctor of any kind that is left on the ground by excluding that use case.
The problem of bad advice is hard, but it's one that will likely become increasingly solved as the tech evolves, and this seems to exclude this model from that technological progression.
Can someone explain to me the "Model Formats" section at the end of that page. It makes sense to me as a description of how to format the training data, but it also says "Use these templates to explore the model’s capacities"
How would you make use of this prefix format in a prompt?
Page 30 of the TR has a few examples:
https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k...
Note that they were produced with StarCoderBase. E.g., here is one:
Input:
Output: (If you try this in the playground, dial down the repetition penalty to say 1.0 in Advanced Settings.)The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others.
The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue.
The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor.
Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.
https://github.com/bigcode-project/starcoder
Quantized versions here:
https://huggingface.co/mayank31398/starcoder-GPTQ
they will be benchmarked on humaneval and released soon—maybe tomorrow?
But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.