This article strongly resonates with me (thanks OP!). Models trained on huge datasets are truly very impressive, and it's easy to jump on the hype train of "BIG MODEL GO VROOM" and overlook the cool / useful things one can do with a little data and solid domain knowledge.
Transfer learning can be immensely useful when applicable, but many times it's not (e.g. imagine a medical domain where you track a patient through some process, recording procedure choices, measurements and outcomes, etc., where it can be difficult to find relevant data elsewhere).
Some approaches I've found useful:
* Get to know the domain really well
* It's not a lot of data - that's a potential for rich interactive visualizations that allow you to get to know the data quite well, and grok how it relates to the domain knowledge
* Following the advice that ML models in production could/should start with simple heuristics, view your model more as an augmented heuristic than a powerful model to solve everything - that means also figuring out how to catch and handle cases where it's wrong (which is something one ought to do anyway)
* Invest in tailoring priors suitable to the problem, based on your domain knowledge and understanding of the data. This can range from writing your own loss function to training an ad-hoc type of model, not based on DL, e.g. using metaheuristics (genetic algorithms, simulated annealing etc.). The advantage of small data is that evaluation on it can be relatively fast and ad-hoc models using nonstandard techniques can be realistically optimized (sometimes, depending on context of course).
I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.
It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.
I fully agree with your perspective, and I think there's a lot of cargo-culting in that area that explains your observations. Sure, if you're a FAANG collecting massive amounts of data comes almost for free, and it makes sense to find a way to properly utilize that data. But for many startups collecting that amount of data doesn't make sense (either because they don't have a lot of users yet or because their domain isn't high-freq digital activity or both), and in those cases it can be more worthwhile figuring out what to do with the little data they have than shoe-horning their way into Big Data (TM).
There's much more money to be had convincing startups to use big data infrastructure and train huge nets on GPU clusters etc. than convincing them to iterate on small ad-hoc models developed in-house. Not saying that big data or models are necessarily wrong for small startups, just that they don't have to be the default.
One question (please take in account that I am no more than a dabbler in this field) - you conclude your piece with ... the nature of the problem slowly changes over time and prediction quality deteriorates.
Isn't this a problem anyway, even with models based on much larger datasets?
You are right, data/concept drift affects both approaches.
What I meant to say was that retraining your model might not fix that if you baked strong assumptions into it. I edited the blog post to make it clearer.
I’ve found constraints in the loss functions are key to finding the correct solution space. With small amount of training data and SGD you can get a lot of mathematically “correct” answers, but a well informed constraint based on your problem space can eliminate 99.9% of the mathematically correct but practically incorrect answers.
I’ve only ever played with constraints in the loss at this point. Would love more time to also deal with constraining the optimization. Do you have any examples that come to mind that worked well when doing this? Thank you for your suggestion!
I've only seen (and used) this for linear / convex models and constraints, so that's actually linear / convex programming (highly recommend the cvxpy library [0]). I was curious if you've integrated that into DL models.
Here's a fun example I did for a uni project: say you have a small recipe dataset, where for each recipe you have the ingredient list ("3 tbsp sugar", ".5 kg flour" etc.) and macronutrients (carbs, proteins, fat), and you want to learn to predict the nutritional content given a list of ingredients.
With a bit of text manipulation you can split each ingredient text to "amount", "measurement unit" and "ingredient type". Then you can decompose the nutritional values into nutritional density of ingredients and the conversion of measurement unit -> actual mass for each ingredient type. Then you can introduce constraints in the form of known conversions between units, e.g. 1 cup = 16 x tbsp, known nutritional densities of some simple ingredients, and known unit conversions for specific ingredients.
It worked better than simple regression (that didn't take into account unit conversions) and a simple MLP, though not sure how it would compare if you actually tried to finetune a language model to the task. The overall predictions weren't too accurate, but for common ingredients it actually gave very close nutritional density values (without being constrained), which was cool.
Edit: another constraint I initially forgot about is that each gram of an ingredient cannot contain more than a gram of macronutrients (it can contain less because water), and of course nonnegative nutritional density.
Thank you too for this information and for the resource, I'm looking at it now it seems very interesting.
The breakdown you give above seems to me to be more akin to something DL types tend to call 'feature engineering'. I also have a fun example, in this case it would be identifying land cover from satellite imagery. You can obviously just feed the raw reflectance values (RGB etc.) into a DL model to create a semantic segmentation of classes. However, it's been well established in literature at this point that that is not the most effective way to create a classification. This is similar to my previous comment, where there are lots of solutions that can be found through SGD based on these raw values.
There's a lot of traditional satellite imagery analysis algorithms that are based on very simple 'band-ratios', i.e. NDVI (normalized difference vegetation index) is calculated by Near Infared - Red / Near Infared + Red. This index will visually highlight areas of vegetation that was extremely useful in human sight-based analysis to identify vegetative areas. Now, you'd expect a deep learning model that takes in all the bands to have this information already, it has the NIR band and the Red band. However, explicitly doing the NDVI calculation and using it as an input feature leads to increased accuracies for classification. The exact reasons for this are unknown, but I think you touched on some of this above. With machine learning, and DL even moreso, sometimes it's necessary to hand-hold the optimization to optimize for exactly what you want. It helps 'explainability' and oftentimes helps accuracy, at the cost of some preprocessing.
Could you elaborate a bit more on how you do this, or point to resources on it? Do you mean putting constraints on possible parameter values, or just on the loss function values itself? Thanks.
I have never dabbled with constraining the parameter values themselves. Ive mostly put the constraints into the loss function. This works well when working with regression, a super simple constraint that adds penalty when the regression goes outside of the possible solution space has been extremely helpful in our work. Heuristically, I think it’s more useful during the first few iterations to find the correct local minima. If you were already finding the “correct” local minima this might be less important, but if you’ve ever dealt with convolutional artifacts (boxes, lines, edge effects) in your predictions, well informed constraints tend to help avoid these, as they are a symptom of being in an incorrect local minima.
Another approach is to use Bayesian, and state some meaningful priors. This is implicit regularization (constraining) since the priors are "pulling" back the parameter values based on the data (likelihood), and in the end you get some kind of compromise between the two (posterior). One example I have is when estimating traffic flow, and I put a Poisson (or Half-Normal - works ok as well) distribution as my prior, with some gut-feeling/logically-reasonable parameters.
In my experience, overfitting also helps in case of small data problems. After all, there is a reason that you do only have few training data samples and the chance that any new prediction sample is close to the existing training ones is higher. In other words, for small data its more about specialization than generalization. Of course this has to be evaluated case specific.
Interesting, that advice is exactly the opposite of the common wisdom. You mean overfitting as in aggressively maximizing your crossvalidation scores? How do you decide for which problems that is a good approach and for which a more conservative approach is better?
Well, I mean not actively maximizing your CV score, just accepting normally insufficient CV scores.
For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.
In a lot of cases knn doesn't model systems appropriately unless you understand the space very well. At least knn offers some sort of theoretical assumption of the bayes error rate though so it's a decent tool.
I would take the opposite approach here. Instead of overfitting, bake probability into your model. Whether using a Bayesian weights or using an ad-hoc version (I.e. dropout and batch normalization), you can do your predictions in ensemble and look at the deviation of predictions. the combination of this method and a small dataset usually leads to wider ranges of prediction, which can be interpreted as a model uncertainty. When using these techniques we have found smaller datasets lead to more uncertainty, but it often will bound the error. I think this is much more useful than assuming a small dataset means an overfit model will do good on future data.
I teach Data / AI courses regularly and this topic is a recurring one with students. We the instructors like to make students start without even ML in the first place for them to construct a baseline model, usually using simple heuristics (e.g. conditions, regexes, etc.).
At first, students don't like it and tend to skip it because it is not that impressive, but in the end they learn so much about the data and the problem at hand! Then they are allowed to try simple ML models to see if that's a performance improvement and so on with more complex models. In some cases, improving baseline models with complex ML/DL models is really hard (e.g. time-series). Another benefit of simple models is explainability, for which the industry demand is growing everyday.
Just out of curiosity where do you see the demand? I've also heard from others that there's been a shift lately towards simpler models. Combined with domain knowledge linear/logistic regression can be really impressive!
I can agree this the comment. Linear models combined with advanced feature engineering gathered from domain knowledge can achieve great results in a white-box fashion!
A nice keynote by Vincent Warmerdam [1] talks about tips and tricking for advanced feature engineering combined with linear models.
A significant portion of ML workloads involve predicting or classifying something. Linear/logistic regression of the right variables/features typically gets a significant portion of the data's ability to predict /classify correctly, while being significantly easier to build, train, deploy, and understand.
Heck, in a large number of domains, simple ratios -- debt to income ratio in finance for example -- will dominate the feature weight for many models and can be used on their own as a pretty good heuristic.
Small non-FAANG companies usually, where they do not have the internal skills to maintain and explain models. And from what I've heard, big corporations under regulated domains (banking, healthcare, etc).
A powerful concept for thinking about small data is bias/variance tradeoff. The ELI5 is: data is how a learner selects confidently between one model and another. So if you have little data, you must choose between high bias (fewer models to choose from) or low confidence (can't be sure you picked the best one).
Bias in a lot of contexts is Bad, and large data techniques like DNN are basically about how you can have biases that are so vague the learner has plenty of room to surprise you. But when you add bias that truly aligns with the structure your learner is going to be exposed to, you enable it to reach confidence sooner. Many important techniques for small data are about allowing you to express specific biases. For example:
- PGMs and Probabilistic Programming are about giving you a specific, interpretable structure for how your data are related to each other.
- Picking informed bayesian priors
- Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset)
- feature engineering is about selectively adding and removing biases in terms of what kinds of data transformations are informative
With all that said, the large data ML community has produced an incredible tool for practical work with small data: transfer learning. If your dataset is related to a larger one (e.g. by including natural language), you can borrow informed biases from models that were trained on a much larger corpus.
I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios.
How much data is considered small? What is the complexity of the dataset itself?
Even in Machine Learning it is possible to learn from small datasets without transfer learning. See meta-learning for instance.
> I'd be careful of over applying the "bias-variance tradeoff." How to define the variance of a model is not a simple task. I wouldn't say it is immediately obvious how bias-variance relates to small data scenarios.
It's very important for anyone studying ML to understand how bias-variance relates to sample size, so I'd encourage finding more resources if this note didn't help clarify! Here's another shot at a summary: for a fixed size of training sample, you must trade off between sensitivity to randomness in the sample (variance) and assumptions that bias the model you train.
It's true that quantifying variance and bias can be hard, and you need systems like PAC learning to go further and actually estimate sufficient sample sizes for a task. But you can still reason usefully about any system that involves using data to select (train) among a class of potential output models!
For example, the statement about meta-learning is incorrect, at least as far as i've seen the term used. Meta learning involves learning hyperparameters (including functions) that are then used to train a model. The extra stage makes these models less biased, but actually require more data. (Of course, in some meta learning systems, hyper parameters are learned with the help of external data - a form of transfer learning.)
I occasionally work with data in the Humanities. The data here is often very, very small. I talk to other Humanities researchers and I often find that they really want to get on the ML bandwagon but they do not realize the sheer amount of data that they need to make ML as practiced today work. I have not looked into small dataset techniques in a long time (I have a day job so I do not get much chance to do this often) but I hope that one day we can find a technique that will work.
One side note, when I speak to other Humanities researchers about this, I always tell them that I have yet to find a technique that will give them novel insights. These techniques almost always tell the researchers things that they already know. I usually follow this up with a note that even formalizing Humanities knowledge in statistical or other computational terms is highly valuable and worth doing. Maybe someone else can take that formalism and build on top of it something truly new.
> how is it different from statistical forecasting
In statistical learning/forecasting, the researcher typically specifies the statistical model.
In machine learning, the statistical model is approximated by the algorithm.
Since a ML model needs to learn both the model form and the model parameters, it takes more data and also it does not allow for understanding (since it does not output the form of the model it learned).
You can fit a linear regression with just a few points, technically if you have one more data point than regressors, it works. Because you've assumed a linear relationship with normally distributed errors. And you can interpret the output, because the values of the regression coefficients tell you something. "having a high score on X doubles the odds of outcome Y", for example.
Also, because you've assumed a structure to the data, you can more easily test if the data has deviated from that structure. This can be data drift or single outliers-- for example, GARCH models (a type of regression) allow the normal distribution of the error to have a varying variance, so you can detect different variance regimes.
In short, they help a human understand and interpret data.
From what little I know, ML is not so good at that. But it has other advantages, and you don't always need or want the understanding. If your want to i.e. detect ground cover in satellite images, then all you care about is valid outputs, not necessarily the importance of near-infrared vs red band.
And ML (can) beat regression models by providing better interpolation, by better handling regions of the data space which violate the assumptions of the regression model, etc.
So it is a tradeoff. Both approaches are highly performant, just at different tasks.
An example I have used a few times to highlight the difference between "understanding how a model gives predictions" and "model uses predictors/features in a way that makes sense" is a linear regression model that Bondarchuk, a famous throwing coach of former USSR.
The linear regression was used to predict the distance expected in one of the throws, given the performances in other athletic tasks, say max squat, max power clean, max long jump and a few others.
Some of the regression coefficients were negative, which means that increasing performance in, say, long jump, leads to shorter shot-put (I don't remember which throw the model was for) distances.
The model and approach looked understandable and "weird" at the same time.
From a purely statistical perspective it makes sense, since that was the results coming from, I assume, maximum likelihood estimation. From a predictive performance, retrospectively it surely worked because it gave good prediction on past data, assuming there were training and test data (most likely, they were not, but let's assume).
But from a future prediction perspective, i.e. the forecasting and thus the manipulation of training to obtain a certain performance, did it make sense? I am very confident it did not, because, among other things, the performances of auxiliary lifts/feats were not independent (you cannot work on a heavier one rep max in the power clean and hope or work toward a shorter long jump performance).
The model by itself might have accurate, but considering that interpretable and thus guiding changes in the training program would have been a quite naive mistake. This kinda mistake is quite common among many who think too much about the machinery of the model and way too little about the domain.
> But from a future prediction perspective, i.e. the forecasting and thus the manipulation of training to obtain a certain performance, did it make sense?
Those are two different questions. For forecasting without manipulation of training it would still make sense. But it wouldn’t make sense for causal analysis.
That is a true difference and I should have been clearer. I should have specified that I don't believe they used any test data, the "study" had probably been done with no test data set and simply using all data for the estimation of regression parameters.
I believe that (1) the model made little "mechanistic" sense, (1) the forecasting accuracy of the model would have been low, (3) the model had good hind-casting accuracy through overfitting by modeling the "noise" in the data with too many predictors. (4) the model could have not guided any training.
> Because you've assumed a linear relationship with normally distributed errors.
Normality will give you stronger results, but generally linear regression doesn’t require normality. You only need errors to be uncorrelated for OLS to be the best linear unbiased estimator.
> These techniques almost always tell the researchers things that they already know.
Yes, but sometimes in surprising ways.
I build a simple decision-tree model for a medical study, looking at outcomes for acute pneumonia. Went with a single tree over a forest because the model had to be interpretable. Statistically it was almost as good as the forest; I built it using fields with high feature importance values. Thus there is a chance that any 'improvement' by the forest was overfitting. but I digress.
The tree said that blood CO2 levels were the most important factor. The doctors weren't surprised by this (though they had some internal debate if this was more or less important than some other factors). What did surprise them was the cutoff level.
They said they would be concerned if CO2 was above 7. My model had the cutoff at 9.5. Sorry, I forget the units.
Point is, it confirmed what they knew (CO2 levels matter when assessing lung function), but still surprised them (CO2 levels have to be much higher than normal before this becomes discriminant over other factors, such as age).
11. Reducing the scope of the problem so you don't even need that much data
12. Iterative model approaches that use submodels focused on different target problems (instead of trying to boil the ocean with a single model)
A lot of these are not what you do, but how you do it, something popularized by CPMAI methodology.
I would also add that I'm not sure the OP was tongue-in-cheek when saying that AGI is coming, so we don't need to worry about low data requirements. But needless to say, AGI is not coming any time soon, and if OpenAI's strategy for AGI is to be believed, it's anything but a low-data method. Unless you're counting on them to build the super-sized model that meets all needs.
I first read about 10-20 observations per feature in a statistical model (not ML), including interactions between features, in Frank Harrell's "Regression Modeling Strategies," an excellent book that many coming from the CS side of modeling would benefit greatly from studying.
There's also data in another source: domain knowledge and physical laws. This is the core tenant of scientific machine learning techniques (SciML). I have a talk that walks through how you can go from something without domain knowledge (like a neural ODE) and how as you add more prior model knowledge to the system the extrapolation accuracy improves even for small amounts of data (https://www.youtube.com/watch?v=FihLyzdjN_8). People have used these techniques in all kinds of places, like extrapolating black hole trajectories and building earthquake-safe buildings, off of like 20-40 data points (https://www.youtube.com/watch?v=eSeY4K4bITI). While there is a lot of work to be done in the domain still, there's a lot of empirical evidence that this approach does indeed decrease the data requirements (but is then no longer "pure" machine learning in some sense)
why would you build an earthquake-safe building using only 20-40 data points (sorry, I skimmed the video and I dont think it answered that question). That sounds irresponsible since you could collect far more data points, and unless you could demonstrate that collecting tiny amounts of data was "safe", you're always going to collect high quality data and lots of it.
The advantage of physical systems is that it's straightforward to collect data from natural systems.
> The advantage of physical systems is that it's straightforward to collect data from natural systems.
For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.
yes, but it's still unclear that for data-poor physics problems (the joke in grad school is that you could graduate when you collected your second data point, since that was enough to draw a straight line, and we all know any interesting physical phenomenon is linear in the engineering regime, right?) that NNs really provide anything more than a bayesian prior that physics folks already knew how to incorporate.
High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.
The general principle can be referred to as "inductive bias": out of the space of all functions that can be represented by computational means, how does one restrict the search space to only those functions which are relevant for the data/system of study? Developing physical theories is nothing more than the process of whittling down what space of functions can represent the data most accurately, using analytical and scientific techniques to get there.
Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.
I don't think there's any deep relationship between the functions we find that fit observed data well, and those functions being generally useful at making out of class predictions.
Really, I was making a comment about https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... which in my mind is still unresolved (that is: why do the math models we use for physical systems generalize so well, if we wouldn't expect them to do so?)
When you can model a system using physical or known principles the data points are mostly there to prove the model is valid. Also they usually aren't randomly chosen data points like in ML they are chosen via statistical theory.
The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD.
I've made models of physical systems that involve kind of life or death stuff with less than 30 samples.
It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!
Uh, OK. because if you were doing life and death stuff with 30 samples, people probably died due to false negatives or positives in your model, in a way that would have been better handled with more samples. Sorry to hear it- it's one of the reasons health care is so shitty.
oh, sorry- you mean "life and death" as in you had a murderbot (https://www.youtube.com/watch?v=xuFbxeDjRZg_ was my team's contribution) that could kill people, and your model prevented it?
Are you sure you can conclude your system is safe?
I think you'd be surprised at how few samples you need in order to get most deep learning algorithms learning at low loss. All you need are features with distributions that are roughly representative of the broader population.
The real problem that I see is that people don't pick a task that is granular enough.
Machine learning on small data, oh you mean statistics? I'm trolling a bit here, but this is what domain experts and scientists have been doing for about a century.
It's a much harder area to hang out in, and pays way less unless you are a world expert consultant. Stakes are often higher. I used to do this stuff, but there's so much hype around ML it's basically impossible to survive in industry even if you do very good work.
For example if you create a model catered to a system and it takes two weeks but is as correct as it can be given what's known about the system, your manager will turn their chin too it while trying to find startups to buy(yes buy the company) that claim they can solve it with deep learning. The startups run away once they learn each sample costs 10k USD or deliver a boosted decision tree model that under performs, but it got so old I changed careers.
Most older machine learning techniques work with "small data". Most of the literature pre deep learning is on those techniques and what problems they work well for.
69 comments
[ 4.8 ms ] story [ 155 ms ] threadTransfer learning can be immensely useful when applicable, but many times it's not (e.g. imagine a medical domain where you track a patient through some process, recording procedure choices, measurements and outcomes, etc., where it can be difficult to find relevant data elsewhere).
Some approaches I've found useful:
* Get to know the domain really well
* It's not a lot of data - that's a potential for rich interactive visualizations that allow you to get to know the data quite well, and grok how it relates to the domain knowledge
* Following the advice that ML models in production could/should start with simple heuristics, view your model more as an augmented heuristic than a powerful model to solve everything - that means also figuring out how to catch and handle cases where it's wrong (which is something one ought to do anyway)
* Invest in tailoring priors suitable to the problem, based on your domain knowledge and understanding of the data. This can range from writing your own loss function to training an ad-hoc type of model, not based on DL, e.g. using metaheuristics (genetic algorithms, simulated annealing etc.). The advantage of small data is that evaluation on it can be relatively fast and ad-hoc models using nonstandard techniques can be realistically optimized (sometimes, depending on context of course).
I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.
It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.
There's much more money to be had convincing startups to use big data infrastructure and train huge nets on GPU clusters etc. than convincing them to iterate on small ad-hoc models developed in-house. Not saying that big data or models are necessarily wrong for small startups, just that they don't have to be the default.
Isn't this a problem anyway, even with models based on much larger datasets?
What I meant to say was that retraining your model might not fix that if you baked strong assumptions into it. I edited the blog post to make it clearer.
Out of curiosity, did you apply the constraint as a term in the loss or did you use constrained optimization?
I've only seen (and used) this for linear / convex models and constraints, so that's actually linear / convex programming (highly recommend the cvxpy library [0]). I was curious if you've integrated that into DL models.
Here's a fun example I did for a uni project: say you have a small recipe dataset, where for each recipe you have the ingredient list ("3 tbsp sugar", ".5 kg flour" etc.) and macronutrients (carbs, proteins, fat), and you want to learn to predict the nutritional content given a list of ingredients.
With a bit of text manipulation you can split each ingredient text to "amount", "measurement unit" and "ingredient type". Then you can decompose the nutritional values into nutritional density of ingredients and the conversion of measurement unit -> actual mass for each ingredient type. Then you can introduce constraints in the form of known conversions between units, e.g. 1 cup = 16 x tbsp, known nutritional densities of some simple ingredients, and known unit conversions for specific ingredients.
It worked better than simple regression (that didn't take into account unit conversions) and a simple MLP, though not sure how it would compare if you actually tried to finetune a language model to the task. The overall predictions weren't too accurate, but for common ingredients it actually gave very close nutritional density values (without being constrained), which was cool.
Edit: another constraint I initially forgot about is that each gram of an ingredient cannot contain more than a gram of macronutrients (it can contain less because water), and of course nonnegative nutritional density.
[0] https://www.cvxpy.org/
The breakdown you give above seems to me to be more akin to something DL types tend to call 'feature engineering'. I also have a fun example, in this case it would be identifying land cover from satellite imagery. You can obviously just feed the raw reflectance values (RGB etc.) into a DL model to create a semantic segmentation of classes. However, it's been well established in literature at this point that that is not the most effective way to create a classification. This is similar to my previous comment, where there are lots of solutions that can be found through SGD based on these raw values.
There's a lot of traditional satellite imagery analysis algorithms that are based on very simple 'band-ratios', i.e. NDVI (normalized difference vegetation index) is calculated by Near Infared - Red / Near Infared + Red. This index will visually highlight areas of vegetation that was extremely useful in human sight-based analysis to identify vegetative areas. Now, you'd expect a deep learning model that takes in all the bands to have this information already, it has the NIR band and the Red band. However, explicitly doing the NDVI calculation and using it as an input feature leads to increased accuracies for classification. The exact reasons for this are unknown, but I think you touched on some of this above. With machine learning, and DL even moreso, sometimes it's necessary to hand-hold the optimization to optimize for exactly what you want. It helps 'explainability' and oftentimes helps accuracy, at the cost of some preprocessing.
For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.
I teach Data / AI courses regularly and this topic is a recurring one with students. We the instructors like to make students start without even ML in the first place for them to construct a baseline model, usually using simple heuristics (e.g. conditions, regexes, etc.).
At first, students don't like it and tend to skip it because it is not that impressive, but in the end they learn so much about the data and the problem at hand! Then they are allowed to try simple ML models to see if that's a performance improvement and so on with more complex models. In some cases, improving baseline models with complex ML/DL models is really hard (e.g. time-series). Another benefit of simple models is explainability, for which the industry demand is growing everyday.
Can you elaborate?
A nice keynote by Vincent Warmerdam [1] talks about tips and tricking for advanced feature engineering combined with linear models.
[1] https://www.youtube.com/watch?v=68ABAU_V8qI
Heck, in a large number of domains, simple ratios -- debt to income ratio in finance for example -- will dominate the feature weight for many models and can be used on their own as a pretty good heuristic.
Bias in a lot of contexts is Bad, and large data techniques like DNN are basically about how you can have biases that are so vague the learner has plenty of room to surprise you. But when you add bias that truly aligns with the structure your learner is going to be exposed to, you enable it to reach confidence sooner. Many important techniques for small data are about allowing you to express specific biases. For example:
- PGMs and Probabilistic Programming are about giving you a specific, interpretable structure for how your data are related to each other.
- Picking informed bayesian priors
- Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset)
- feature engineering is about selectively adding and removing biases in terms of what kinds of data transformations are informative
With all that said, the large data ML community has produced an incredible tool for practical work with small data: transfer learning. If your dataset is related to a larger one (e.g. by including natural language), you can borrow informed biases from models that were trained on a much larger corpus.
What does ”permitting your dataset“ mean? Is it a typo? Did you mean ”permuting“?
How much data is considered small? What is the complexity of the dataset itself?
Even in Machine Learning it is possible to learn from small datasets without transfer learning. See meta-learning for instance.
It's very important for anyone studying ML to understand how bias-variance relates to sample size, so I'd encourage finding more resources if this note didn't help clarify! Here's another shot at a summary: for a fixed size of training sample, you must trade off between sensitivity to randomness in the sample (variance) and assumptions that bias the model you train.
It's true that quantifying variance and bias can be hard, and you need systems like PAC learning to go further and actually estimate sufficient sample sizes for a task. But you can still reason usefully about any system that involves using data to select (train) among a class of potential output models!
For example, the statement about meta-learning is incorrect, at least as far as i've seen the term used. Meta learning involves learning hyperparameters (including functions) that are then used to train a model. The extra stage makes these models less biased, but actually require more data. (Of course, in some meta learning systems, hyper parameters are learned with the help of external data - a form of transfer learning.)
One side note, when I speak to other Humanities researchers about this, I always tell them that I have yet to find a technique that will give them novel insights. These techniques almost always tell the researchers things that they already know. I usually follow this up with a note that even formalizing Humanities knowledge in statistical or other computational terms is highly valuable and worth doing. Maybe someone else can take that formalism and build on top of it something truly new.
In statistical learning/forecasting, the researcher typically specifies the statistical model.
In machine learning, the statistical model is approximated by the algorithm.
Since a ML model needs to learn both the model form and the model parameters, it takes more data and also it does not allow for understanding (since it does not output the form of the model it learned).
Also, because you've assumed a structure to the data, you can more easily test if the data has deviated from that structure. This can be data drift or single outliers-- for example, GARCH models (a type of regression) allow the normal distribution of the error to have a varying variance, so you can detect different variance regimes.
In short, they help a human understand and interpret data.
From what little I know, ML is not so good at that. But it has other advantages, and you don't always need or want the understanding. If your want to i.e. detect ground cover in satellite images, then all you care about is valid outputs, not necessarily the importance of near-infrared vs red band.
And ML (can) beat regression models by providing better interpolation, by better handling regions of the data space which violate the assumptions of the regression model, etc.
So it is a tradeoff. Both approaches are highly performant, just at different tasks.
Some of the regression coefficients were negative, which means that increasing performance in, say, long jump, leads to shorter shot-put (I don't remember which throw the model was for) distances.
The model and approach looked understandable and "weird" at the same time. From a purely statistical perspective it makes sense, since that was the results coming from, I assume, maximum likelihood estimation. From a predictive performance, retrospectively it surely worked because it gave good prediction on past data, assuming there were training and test data (most likely, they were not, but let's assume).
But from a future prediction perspective, i.e. the forecasting and thus the manipulation of training to obtain a certain performance, did it make sense? I am very confident it did not, because, among other things, the performances of auxiliary lifts/feats were not independent (you cannot work on a heavier one rep max in the power clean and hope or work toward a shorter long jump performance).
The model by itself might have accurate, but considering that interpretable and thus guiding changes in the training program would have been a quite naive mistake. This kinda mistake is quite common among many who think too much about the machinery of the model and way too little about the domain.
Those are two different questions. For forecasting without manipulation of training it would still make sense. But it wouldn’t make sense for causal analysis.
I believe that (1) the model made little "mechanistic" sense, (1) the forecasting accuracy of the model would have been low, (3) the model had good hind-casting accuracy through overfitting by modeling the "noise" in the data with too many predictors. (4) the model could have not guided any training.
Normality will give you stronger results, but generally linear regression doesn’t require normality. You only need errors to be uncorrelated for OLS to be the best linear unbiased estimator.
Yes, but sometimes in surprising ways.
I build a simple decision-tree model for a medical study, looking at outcomes for acute pneumonia. Went with a single tree over a forest because the model had to be interpretable. Statistically it was almost as good as the forest; I built it using fields with high feature importance values. Thus there is a chance that any 'improvement' by the forest was overfitting. but I digress.
The tree said that blood CO2 levels were the most important factor. The doctors weren't surprised by this (though they had some internal debate if this was more or less important than some other factors). What did surprise them was the cutoff level.
They said they would be concerned if CO2 was above 7. My model had the cutoff at 9.5. Sorry, I forget the units.
Point is, it confirmed what they knew (CO2 levels matter when assessing lung function), but still surprised them (CO2 levels have to be much higher than normal before this becomes discriminant over other factors, such as age).
2. Augmentation
3. Heavy regularization
4. Cross-validation
5. Transfer learning/fine-tuning from related data
6. Sharpness aware optimization
7. Ensembles
8. Any domain knowledge/feature engineering
10. Single-shot / low-shot learning methods
11. Reducing the scope of the problem so you don't even need that much data
12. Iterative model approaches that use submodels focused on different target problems (instead of trying to boil the ocean with a single model)
A lot of these are not what you do, but how you do it, something popularized by CPMAI methodology.
I would also add that I'm not sure the OP was tongue-in-cheek when saying that AGI is coming, so we don't need to worry about low data requirements. But needless to say, AGI is not coming any time soon, and if OpenAI's strategy for AGI is to be believed, it's anything but a low-data method. Unless you're counting on them to build the super-sized model that meets all needs.
Is there a good way for a given problem to estimate how many examples would be required to produce a decent model?
Small data depends on the domain/context. It might mean < 30 observations or it might mean < gigabyte scale.
The advantage of physical systems is that it's straightforward to collect data from natural systems.
For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.
High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.
Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.
Really, I was making a comment about https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... which in my mind is still unresolved (that is: why do the math models we use for physical systems generalize so well, if we wouldn't expect them to do so?)
The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD.
I've made models of physical systems that involve kind of life or death stuff with less than 30 samples.
It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!
Are you sure you can conclude your system is safe?
The real problem that I see is that people don't pick a task that is granular enough.
github.com/aiqc/aiqc
It's a much harder area to hang out in, and pays way less unless you are a world expert consultant. Stakes are often higher. I used to do this stuff, but there's so much hype around ML it's basically impossible to survive in industry even if you do very good work.
For example if you create a model catered to a system and it takes two weeks but is as correct as it can be given what's known about the system, your manager will turn their chin too it while trying to find startups to buy(yes buy the company) that claim they can solve it with deep learning. The startups run away once they learn each sample costs 10k USD or deliver a boosted decision tree model that under performs, but it got so old I changed careers.