I think the critical thing is you need some ground truth way of evaluating the synthetic data. You can generate 100 programs with your LLM and filter to the 1-2 that solve the problem, but there's not an equivalent option for things like MRI.
Synthetic data is a big deal, essentially as a form of “knowledge distillation” from large models or for transforming high-quality text into training data (e.g. Q&A pairs). Almost everyone is using GPT-4 for this. Dunno about other domains, as it’s based on the mutability of text, relative to whatever ground truths are embedded therein. This seems less feasible for other kinds of inputs, but who knows.
Yes and no. In terms of LLMs, it's basically figuring out how to exfiltrate information from GPT4 to remove costs of data gathering. The limitations of that are that the model will never be better than gpt4, and when gpt4 produces incorrect information, the model trained on synthetic data will also do so.
In other fields like computer vision, synthetic data is useful for generating ground truth data, like for depth masks.
I’ve disputed that fact before. It depends what part of your data is generated from GPT4 if I have high quality code and I’m using gpt4 to synthetically generate variations on how some could ask for the code to be written it’s entirely possible because of the high quality code for the model to be better. It’s not all or nothing if you’re mixing synthetic data with quality sources.
In this case even though parts of the dataset are synthetic the bound is on the code not necessarily the 50 ways I got gpt4 to say “write me a script to do x” or modeled other interactions with that code data source.
If we’re talking about model distillation[0] I don’t think the student can ever be better than the teacher as optimising for speed and smaller model sizes inherently means that there will be precision loss. Even if the student is as big as the teacher, there is still data loss.
This is mistaken. You use GPT-4 to generate new data using other data sources, for example text books.
Asking GPT-4 to create 50 new conversations from a chapter of a textbook creates higher quality data than most of “the pile” and can extend beyond what GPT-4 has in its existing dataset. This is exactly what MS has done with Phi.
10 comments
[ 6.2 ms ] story [ 43.2 ms ] threadhttps://nitter.net/TeamCodeLLM_AI/status/1747652471714144702
In other fields like computer vision, synthetic data is useful for generating ground truth data, like for depth masks.
In this case even though parts of the dataset are synthetic the bound is on the code not necessarily the 50 ways I got gpt4 to say “write me a script to do x” or modeled other interactions with that code data source.
[0] https://arxiv.org/pdf/2210.17332.pdf
Asking GPT-4 to create 50 new conversations from a chapter of a textbook creates higher quality data than most of “the pile” and can extend beyond what GPT-4 has in its existing dataset. This is exactly what MS has done with Phi.