100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to collect data. Machine learning people could learn a lot from these old, uncool theories if they want to approach data with rigor. That one can approach data collection and usage in a systematic way is totally lost now, it's just web scraping.
This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.
A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.
Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows.
100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).
Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.
you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data.
It is still taught at university if you do a master degree or PhD at a decent place.
But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field
I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.
Interesting it doesn’t cover technologies for storing data. It does mention analytical operations though.
Anyone have a suggestion for storing large amounts of ML data, training sets or otherwise? I’ve been using FeatureBase and weaviate, and would be interested in learning about other solutions.
We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames.
"ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions.
Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.
Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models).
Note that the term "Critical" here refers to the likes of "Critical Theory", "Critical Social Justice", "Critical Race Theory" etc. It's biased from the offset so I'd take anything from this site with a pinch of salt.
19 comments
[ 4.6 ms ] story [ 57.4 ms ] threadThe design gave me a bit of anxiety, it was so convincing I was worried about clicking into one of the cells!
Anxiety included - I also love the layout.
100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).
[1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...
It is still taught at university if you do a master degree or PhD at a decent place.
But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field
Anyone have a suggestion for storing large amounts of ML data, training sets or otherwise? I’ve been using FeatureBase and weaviate, and would be interested in learning about other solutions.
Would love to get some feedback on it!
https://github.com/Oxen-AI/oxen-release#-oxen
Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.
Milvus (https://milvus.io) and Feast (https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.