At Sumo Logic we do "grep in cloud as a service". We use machine learning to do pattern clustering. Using lines of text to learn printfs they came from.
The primary advantage for customer is easier to use and troubleshoot faster.
This is great. I've been thinking about better ways to search logs for root causes. Splunk is good if you know what you are looking for, but this is exactly what I want to see to show me unexpected things in logs.
just a happy Sumologic user, saying hello and Thanks! Most of your product is great (I am ex splunk user)... The biggest complaint is that I can't cmd+click to open anything in new tabs as everything is so JS crazy front end.
overall the pattern matching stuff is pretty cool. Also, would like a see raw logs around this for when I am trying to debug event grouping errors based on the starting regex.
E.g. With LogReduce you can click on group and see log lines that belongs to it. IS that something that solves your problem, or are you looking for something else.
Feel free to send me an email (it is on my profile).
The entire product I built over the last year can be reduced to basic statistics (e.g. ratios, probabilities) but because of the hype train we build "models" and "predict" certain outcomes over a data set.
One of the products the company I work for sells more or less attempts to find duplicate entries in a large, unclean data set with "machine learning."
The value added isn't in the use of ML techniques itself, it's in the hype train that fills the Valley these days: our customers see "Data Science product" and don't get that it's really basic predictive analytics under the hood. I'm not sure the product would actually sell as well as it does without that labeling.
To clarify: the company I work for actually uses ML. I actually work on the data science team at my company. My opinion is that we don't actually need to do these things, as our products are possible to create without the sophistication of even the basic techniques, but that battle was lost before I joined.
The value added isn't in the use of ML techniques itself, it's in the hype train that fills the Valley these days: our customers see "Data Science product" and don't get that it's really basic predictive analytics under the hood. I'm not sure the product would actually sell as well as it does without that labeling.
So you are misleading your customers through omission? This is the kind of thing that makes people question anyone stating they are using ML. Those of us actually implementing ML techniques (aka training neural nets and automating processes with data) are met with unnecessary skepticism as a result.
edit: OP clarified his position since this post so take that into account when reading.
No, we actually use ML. We just don't need to, in my opinion, because the problems our products solve are more or less solvable without these techniques.
My point was that using ML, even though we don't need to, "adds value" by virtue of the hype train. We need ML to sell products, not to create them.
I do agree that this sort of arrangement lends itself to supporting skepticism around AI and ML. On the other hand I don't think that's a bad thing.
Got it. Thanks for the clarification. It is true that people are using ML where other, simpler options are available, but I wouldn't immediate discount the value of using nets for your problem. I don't know enough about your problem/implementation to speak to it really.
Yes, thanks for highlighting the deficiency in my original post. I can see it is easily interpretated as you did. I added a clarification (or what I hope is one).
It's interesting to me that with all the ML hype, it's still not clear what constitutes ML. A basic k-means or naive Bayes approach will show up in ML textbooks, but those aren't clearly different from "use some statistics to make a prediction".
There's an interesting group of marginal approaches that have existed as-is for years, but have increasingly focused their branding on machine learning as its profile has risen.
> but those aren't clearly different from "use some statistics to make a prediction"
You can reduce 90% of ML to this. Even neural networks are based on statistics.
If I have to draw a line between statistics and ML is that ML learns, it means it can predict things, however statistics only gives you information about the data you have. But for sure statistics and ML overlap a lot.
If you ask me for the most likely new value for a dataset, I won't know. But if I graph a few things and then write a function to spit back the current mean or median, is that machine learning?
I'm not trying to be snarky there, I agree that the bulk of ML tools are fundamentally just statistical tricks with some layer of abstraction. As a result, I have a lot of trouble knowing how much abstraction justifies the ML title. I see some people using "statistics to produce unintuitive solutions" as a standard, but that just begs that we ask unintuitive to who?
I feel like it is foremost a matter of attitude of the practitioner. An applied statistician and a machine learning engineer may deliver exactly the same end product, just the reasoning and assumptions differ. Machine learning uses little to no assumptions, where statisticians do. I also feel that machine learning engineers have a bit less fear of building black boxes.
Caruana showed the cartoon of the difference between a statistician and a machine learning practitioner by showing a cliff. The statistician carefully inches to the edge, stomping her feet to see if the ground is still stable, then 10 meters before the edge she stops and draws her conclusions. The machine learning practitioner dives headfirst from the cliff, with a parachute that reads "cross-validation".
> Norvig teamed up with a Stanford statistician to prove that statisticians, data scientists and mathematicians think the same way. They hypothesized that, if they all received the same dataset, worked on it, and came back together, they’d find they all independently used the same techniques. So, they got a very large dataset and shared it between them.
> Norvig used the whole dataset and built a complex predictive model. The statistician took a 1% sample of the dataset, discarded the rest, and showed that the data met certain assumptions.
> The mathematician, believe it or not, didn’t even look at the dataset. Rather, he proved the characteristics of various formulas that could (in theory) be applied to the data.
> Even that doesn't seem like a clear distinction?
Obviously no, ML uses statistics as statistics uses Maths. But not all ML uses statistics, some algorithms are biological inspired (swarm optimization) other uses theory of information for classification.
The point of ML is you learn something from data, not necessarily with statistics, although it is used in a lot of algorithms. But also function optimization is used in a lot of algorithms. The boundaries are very fuzzy, but for sure not all ML uses statistics and not all statistics are ML.
A lot of it went over the head because I don't know much classical statistics, but I read some articles by stats people that basically boiled down to the distinction not being in the techniques but in common assumptions, rigor, culture, etc.
ML and statistics are a subset of maths. As I said the statistics and ML overlap and also function interpolation. But some ML algorithms are based on biological systems (like swarm optimization), or theory of information.
If you have a problem that you want to classify some vectors, you have different ways to do it. You call all of them ML, but some use statistics, others use interpolation, other uses theory of information, etc. The model doesn't have to be complex or require a lot of data. Instead of saying all the different techniques you sum up saying ML.
Actually there's a very clear definition of what types of problems ML ought be used for, and that category of problem is what defines it. Those familiar with regression (and stats in general) ought to be familiar with it already - it's an issue of relationship of datatype between independent and dependent variable.
In brief, you're going to run up against two types of data - categorical and continuous. (There are facets to this, eg ordinal, but these are really the elemental types of data). The relationship of datatype to independent/dependent variable is what determines what kind of analysis you may conduct.
Categorical Independent vs. Categorical Dependent, for example, is fairly restrictive, as makes logical sense. You may cross-tabulate, you may score likelihood based on previous observation, but obviously, because all of the data involved are non-numeric, there's no chance for regression, ANOVA, etc. Linear Regression is used when both independent and dependent variables are continuous, and cross-category differencing techniques like ANOVA may be used when the independent is categorical and the dependent is continuous.
The part you don't typically learn until grad school is when the independent is continuous and the dependent is categorical, ie, in ML, a classification problem. The standard statistical methods used as foundation for these problems are logistic regression, logit/probit. It's expansion of these methods that lead to ML in the first place.
If I'm reading this correctly, it's just wrong. Whatever the distinction between data analysis and ML might be, it is more than just whether your data and predicted quantities are discrete or continuous.
> Categorical Independent vs. Categorical Dependent, for example, is fairly restrictive, as makes logical sense. You may cross-tabulate, you may score likelihood based on previous observation, but obviously, because all of the data involved are non-numeric, there's no chance for regression, ANOVA, etc
If you are implying that categorical -> categorical predictions are not ML: as a counter example, natural language is a categorical (words) input that could be used to predict any number of categorical variables (parse trees, semantic categories, etc). I think it's safe to say that the field of NLP is doing machine learning.
Thanks for the sanity check. I read that reply, and got bogged down enough that I was worried my initial reaction of "what, that's not relevant!" was born of ignorance. Discrete/continuous is a distinction worth making, but as a hidden 'definition' for ML I really don't understand it.
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E." - Tom Mitchell
> A basic k-means or naive Bayes approach will show up in ML textbooks, but those aren't clearly different from "use some statistics to make a prediction".
Ha, sounds like a classification problem! Let's use ML to find the boundary.
The best distinction I've found between ML and statistics is the following.
Statistics is about modelling the underlying probability distribution that generates your data. A convergence/generalization/etc result will usually be dependent on this underlying distribution.
ML is when you don't care much about the underlying distribution (modulo regularity assumptions), and your model doesn't even come from the same family at all.
I.e. linear regression is usually statistics, because you often believe the underlying data looks like f(x) ~ f(x0) + f'(x0)(x-x0)). Random forests are machine learning because you don't actually think the real world secretly has a random forest flotaing around.
- We use a complex multivariate model to predict customer conversion and prioritize lead response
- We use text analysis to improve content for effectiveness and conversion
Among other things
I think a lot of the real benefits from ML "at work" is more in just cleaning of data and running through the gauntlet of simplest regressions (before jumping onto something more magical whose outputs and decision making process you can't exactly explain to someone).
When people talk about the growth (or sometimes 'excess') of ML solutions these days, I always wonder about this.
A basic linear regression probably isn't ML, a backprop neural net clearly is, but somewhere between the two is a very fuzzy line between "statistics and data cleaning" and "actually machine learning". I think a lot of people have just pushed the ML angle of an already-reasonable approach to tie into that popularity.
ML courses often start with linear regression, and if you build up complicated polynomials to find a nonintuitive model of your problem, I would definitely consider that machine learning.
I've always thought of regressions, even high-order ones, as just a statistical tool. They're present at the start of ML courses, sure, but as a tool used in ML techniques or a good alternative to them.
It looks like that's not the standard view, though.
Neural networks are just functional approximators, so why isn't a linear regressor of k-th order (e.g. Taylor expansion up to k-th order) also considered "ML"? What's the distinction here?
I think these distinctions are akin to discussing musical genre in Youtube comments. Something can be all of: mathematics, advanced IT, computer science, data science, statistics, econometrics, control theory, pattern recognition, machine learning, predictive modeling, data mining, data analysis, data engineering, or even AI if you are feeling fancy.
Off topic: somehow the people not doing data cleaning because it is so boring and ML techniques do not need clean data end up with the worst overfitting I have ever seen.
We use Convolutional Networks for semantic segmentation [1] to identify objects in the users environment to build better recommendation systems and to identify planes (floor, wall, ceiling) to give us better localization of the camera pose for height estimates. All from RGB images.
We've been using "lite" ML for phenotype adjudication in electronic health records with mild success. Random forests and support vector machines will outperform simple linear regression when disease symptoms/progression don't neatly map to hospital billing codes.
At Persyst we use neural networks for EEG interpretation. Our latest version has human-level performance for epileptogenic spike detection. We are now working on bringing the seizure detection algorithm to human-level performance.
Funny you should ask, detecting QRS complexes has been my first project since starting here. I know of a few papers where the authors have applied neural networks to EKGs, but the applications have been purely academic. I'm not aware of any other companies that use NNs in practice. (There may well be some, but they tend to be secretive about how their algorithms work.) At any rate, the false positive rate of our software is now about an order of magnitude lower than anything else on the market.
Based on past experimental data, we use ML to predict how effective a given CRISPR target site will be. This information is very valuable to our clients.
We're a computer vision company, we do a lot of product detection + recognition + search, primarily for retailers, but we've also got revenue in other verticals with large volumes of imagery. My co-founder and I both did our thesis' on computer vision.
In our space, the recent AI / ML advances have made things possible that were simply not realistic before.
That being said, the hype around Deep Learning is getting pretty bad. Several of our competitors have gone out of business (even though they were using the magic of Deep Learning). For example, JustVisual went under a couple of months ago ($20M+ raised) and Slyce ($50M+ raised) is apparently being sold for pennies on the dollar later this month.
Yes, Deep Learning has made some very fundamental advances, but that doesn't mean it's going to make money just as magically!
There's a lot of "DL allows us to do X so we should make a product / service using DL to do X", rather than "We think there's value in something doing Y, what allows us to do Y? <research> DL allows us to do Y better than anything else, lets use DL"
You gave the example of Slyce. Their products are cool, but I can't help but think "is DL the best way to get that end result?" for lots of the things they do.
Can you expand more on "we do a lot of product detection + recognition + search, primarily for retailers" please? Is that something like identifying products in social media images or something?
We have several products, each of which serves different departments within retailers.
The exact things we do depends entirely on which department(s) are licensing it. Basically, anywhere there's a product image (from their own inventory to mobile to social) and we can provide some kind of help, we do. Every department needs totally different things, so it varies quite a bit...but it's all leveraging our core automated detection + recognition + search APIs.
I work in manufacturing. We have an acoustic microscope that scans parts with the goal of identifying internal defects (typically particulate trapped in epoxy bonds). It's pretty hard to define what size/shape/position/number of particles is worthy of failing the device. Our final product test can tell us what product is "good" and "bad" based on electrical measurements, but that test can't be applied at the stage of assembly where we care to identify the defect.
I recently demonstrated a really simple bagged-decision tree model that "predicts" if the scanned part will go on to fail at downstream testing with ~95% certainty. I honestly don't have a whole lot of background in the realm of ML so it's entirely possible that I'm one of those dreaded types that are applying principles without full understanding of them (and yes I do actually feel quite guilty about it).
The results speak for themselves though - $1M/year scrap cost avoided (if the model is approved for production use) in just being able to tell earlier in the line when something has gone wrong. That's on one product, in one factory, in one company that has over 100 factories world-wide.
The experience has prompted me to go back to school to learn this stuff more formally. There is immense value to be found (or rather, waste to be avoided) using ML in complex manufacturing/supply-chain environments.
My only two misgivings about the program thus far: It is 1) pretty expensive and 2) geared towards working professionals rather than academics, but my employer is helping pay for a good chunk of the degree and I'm more interested in acquiring the skills and tools to go solve problems in industry as opposed to doing research.
Otherwise it has been great thus far. The program was attractive to me because it is somewhat marketed towards those that may not have a software background, but have problems in their industry that could benefit from a "proper" data science treatment. I've been referring to my application of the principles as "Six Sigma for the 21st Century" with managers/directors. I think the vast majority of HN would groan at that term, but it helps communicate the potential value to someone who has no technical background with software whatsoever (think old school manufacturing/operations types): Process improvement for environments with many variables that have practically unknowable inter-dependencies (as is the case with the project described in my original comment).
Interesting perspective. I work in Manufacturing and have created similar models in the past and I was in the MIDS program but I dropped out. Like you it was too expensive and had other misgivings as well.
Care to elaborate at all on those additional misgivings? One thing I could see is that the material might not be very mind-blowing to someone who already has a software background.
I don't understand why this comment is unpopular since the GP is phrased in such a fashion that you only notice that they talking hypothetically if you read it carefully.
I don't think there's anything wrong the GP's achievement or post (it's all interesting stuff) but if something has not yet been implemented, it's worth nothing since there is "many a slip 'tween cup and the lip"
Totally fair - I developed the tool for a product that won't be released until early next year so the cost savings are estimates based on expected production volumes. Its performance in a lower volume development environment has been consistent, however.
I didn't downvote the parent, but I also don't consider it to be civil (which is requested in the HN Guidelines). There are a number of ways to point out that this is not yet in production, and many of them don't require the dismissive tone his comment struck.
This sounds like a pretty standard use of ML to me. No need to feel guilty, this stuff just isn't very difficult from a user's perspective, especially if you use the right libraries. It helps if you maintain a good bookkeeping of your experiments, so you have a good picture of what works and what doesn't.
By the way, control engineering for industry used to be very difficult (but is paid very well), and requires knowledge of systems theory, differential equations, and physics. But with the advent of ML, I suspect that might change; things may get a lot easier.
I disagree with the above, but I think I can shed light on what they might mean. Usually, control theory (which is used in most manufacturing processes) requires quite a bit of background knowledge on the processes at hand along with fairly powerful (mathematical/physical) tools to both approximate and model such processes, along with creating systems that use these models to perform the desired task.
I believe that the parent post means that with current simulation-based tools and large amounts of data generated from manufacturing processes, one can work directly with abstract machine learning models instead of creating physical models or approximations thereof---thus being able to dispose of the mathematical baggage of optimization/control theory and work with a black-box, general approach.
I disagree since we have very few guarantees about machine learning algorithms relative to well-known control approximations with good bounds; additionally, I think it's quite dangerous to be toying with such models without extensive testing in industrial processes, which, to my knowledge is rarely done in most settings by experts, much less people only recently coming into the field. Conversely, you're forced to consistently think about these things in control theory, which I believe, makes it harder to screw up since the models are also highly interpretable and can be understood by people
This is definitely not the case in high-dimensional models: what is the 3rd edge connected to the 15th node in your 23rd layer of that 500-layer deep net mean? Is it screwing us over? Do we have optimality guarantees?
Disclaimer: I have no experience implementing any kind of ML.
How easy will it be to update your model if/when the downstream process changes?
At a previous job we had a process that relied heavily on visual inspection from employees. I often considered applying ML to certain inspection steps, but always figured it would be most useful for e.g. final inspection to avoid having to update the models frequently as the processes were updated.
That's an interesting concern that I hadn't considered if I'm understanding you correctly. I'm imagining you could have a situation where a downstream process change helps mitigate the effect of the upstream defect. In that situation your measure of what constitutes good and bad parts will need to change in the ML model.
I think I'm somewhat lucky in that with my product downstream processes are unlikely to change in a significant enough way to warrant "retraining" the model, but I guess that's probably the only way to handle that - retrain in the event of a significant process change. Our product stays fairly stable once it releases to production and the nature of the downstream processes is that they would have very little effect on the perceived severity of the defect at the final electrical test.
First, bagged decision trees are a little hard to interpret; what is the advantage of a bagged model vs the plain trees? Are you using a simple majority vote for combination? What are the variances between the different bootstraps?
Second - what do you mean by 95% ? Do you mean that out of 99999 good parts 4999 are thrown away? and one bad one is picked out as bad ?
Third - what is this telling you about your process? Do you have a theory that has evolved from the stats that tells you why parts are failing? This is the real test for me.. If the ML is telling you where it is going wrong (even if it's unavoidable/too expensive to solve) then you've got something real.
Unfortunately my concern would be that as it stands.. you might find that in production your classifier doesn't perform as well as it did in test... My worry has been generated by the fact that this same thing has happened to me !
It sounds like the OP is scanning for internal defects in bonds via impurities being trapped in there. These occur relatively randomly and there's some balancing point where it's just not worth trying to make the production line cleaner vs binning parts that fail some QA criteria. I do similar things with castings, where you simply just get certain voids and porosity in the steel when cast and either you can spend a tonne of money trying to eliminate them or you can spend less money testing the parts and binning those that aren't up to par.
I'd hazard to guess that the 95% is the reduction in how many parts made it through the first test only to be caught later at the more expensive stage. So instead of binning 100 parts a month at that second stage, they now bin 5 parts a month and catch way more early on.
It sounds like the OP is using ML to identify flaws that simply just occur due to imperfections in the manufacturing process. That's life, it happens. You can know that they will occur without necessarily being able to prevent them because maybe there's some dust or other particulates in the air that deposit into the resin occasionally, or maybe the resin begins to cure and leaves hard spots that form bond flaws. There's heaps of possible reasons. It sounds more like the ML is doing classification of 'this too much of a flaw in a local zone' vs 'this has some flaws but it's still good enough to pass', which is how we operate with casting defects. For example, castings have these things called SCRATA comparitor plates, where you literally look at an 'example' tactile plate, look at your cast item, then mentally decide on a purely qualtative aspect which grade of plate it matches. Here [1] are some bad black and white photos of the plates.
This is pretty spot on. We know why the defects happen and why they cause downstream test failures, but we lack the ability to prevent (all of) them.
To clarify on that 95% value because it is admittedly really vague: That's actually a 95% correct prediction rate. So far we get ~2.5% false-positives and ~2.5% false-negatives. 2.5% of the parts evaluated will be incorrectly allowed to continue and will subsequently fail downstream testing (no big deal). More importantly, 2.5% of parts evaluated will be wrongly identified as scrap by the model and tossed, but this still works out to be a massive cost savings because a lot of expensive material/labor is committed to the device before the downstream test.
D'Angelo Barksdale: Nigga, please. The man who invented them things? Just some sad-ass down at the basement at McDonald's, thinkin' up some shit to make some money for the real players.
Malik 'Poot' Carr: Naw, man, that ain't right.
D'Angelo Barksdale: Fuck "right." It ain't about right, it's about money. Now you think Ronald McDonald gonna go down in that basement and say, "Hey, Mista Nugget, you the bomb. We sellin' chicken faster than you can tear the bone out. So I'm gonna write my clowny-ass name on this fat-ass check for you"?
Wallace: Shit.
D'Angelo Barksdale: Man, the nigga who invented them things still workin' in the basement for regular wage, thinkin' up some shit to make the fries taste better or some shit like that. Believe.
Hi, I work at a manufacturing startup on pretty much the same exact problem (reducing scrape rates and downtime). I'd love to pick your brain if you have a moment to chat :) my email is mykola@oden.io
This is awesome - thank you! I went through a similar exercise described in your link in evaluating the utility of the tool I described above. This is a nice write up of the logic.
In my case the % occurrence of the defect was very high and the False-Positive cost is also very high so my tool could provide value without being too stellar of a model.
We use ML/Deep Learning for customer to product recommendations and product to product recommendations. For years we used only algorithms based on basic statistics but we've found places where the machine learned models out perform the simpler models.
So is this like the Amazon "feature" where I buy a coffee table on Amazon, then I get suggested to buy a coffee table EVERY DAY for 3 months. Literally row after row of coffee table? Because there must be a big pool of people who buy 1 coffee table buying more coffee tables immediately after?
must be the same genius technology that leads Amazon to load up my Prime frontpage with fashion accessories when I've never had any history of searching or buying such, and recommending the same shows "Mozart in the Jungle", "Transparent", "Catastrophe" on Fire TV stick for months even though I've never shown any interest in any of such programming, even after manually "improving recommendations" by clicking "Not Interested".
its amazing that the vaunted Amazon technology is unable to figure out an algorithm that would satisfy a user's deep desire "please stop plastering Jeffrey Tambor's lipstick and mascara covered face on my startup screen, I've gotten tired of looking at it for the past year"
Obviously the goal of ML in this would be that feeding it enough data about users who buy coffee tables would eventually teach it that you probably don't want another coffee table (because who buys two coffee tables in a row?), but might want to buy say... end tables or other living room furniture in a matching style to the coffee table you just bought.
Would the standard models used allow for the fact that humans could, after buying a coffee table, choose to click on the coffee table in anticipation of then getting suggestions for similar furniture. Presumably the machine sees that the end goal of those continually clicking the same item is actually to arrive at similar items .. but wouldn't it be an obvious optimisation for Amazon to set the ML up to already look deeper than the first page reached.
I have a similar thing with Amazon, I don't know how you're supposed to access the bestseller list for a product type. I just know that if you search a product and follow related products that you eventually get a "#5 in ObscureProduct" tag and that tag takes you to the list of the top-10 models of ObscureProduct available. That sort of learnt navigation must play havoc with a suggestion algo (but IMO would be very easy to fix with just a link for any specific enough item to the 'top 10 in this category').
Theory is that the recommendation engine is built for books. So if you buy a specific type of book, it recommends other books in the similar category. I guess they never got a chance to update it to reflect the fact that Amazon sells more than just books.
Haha yes, I remember seeing washing machines on my landing page for months after I bought one from Amazon. I mean, how many of them could a person need?
Seriously though, I don't understand why it's so hard to take this effect into account, as there should be a very strong negative correlation between a purchase in a given category and the probability of buying an article from that category in the near future, so even a simple ML algorithm should be able to pick this up easily. Anyone here who can explain why this is difficult?
The simple algorithm is to build a correlation matrix between of purchases between all items in the store. Then, when given an item to generate recommendations for, you provide the other items with the highest scores, with a "top sellers" correction for the items that are correlated with everything.
I used to work for a company that implemented similar recommendation services. We approached this problem by modelling whether or not a category was likely to have recurring purchases.
It's a hard problem to determine the repeat purchase cadence of a product. At one end of the bell curve you have items re-purchased frequently, e.g. diapers or grocery, and on the other end you have items that are rarely repurchased.
I haven't looked at coffee tables specifically, but I know when I've looked at home products in the past I've been surprised at how frequently people will buy two large items, e.g. TVs or furniture, within a short period. That said, I agree there is room for improvement here. We're constantly running experiments to improve the customer experience, I have faith that in the limit things will improve. Again, we have no shortage of experimental power so if you'd like to join in the experimentation let me know :)
IMO it comes down to the fact that Amazon literally has my last 13 years of purchasing history, yet it seems that all they are doing is "you looked at x, lets show you y variations of that x."
My dream is that I go to Amazon.com and there are a ton of different unrelated products that people who purchase similar things as me buy. So if I only buy "buy it for life" kitchen equipment, it doesn't show me the most popular but crappy version of something, it shows me the one that I'd actually purchase.
Such an easy problem with suuuuuch a difficult solution though. Not to mention the obvious privacy concerns there.
Oh well, I know that they have good people working on the solution, and no chance I could do it better :p
This topic must be extremely interesting (good suggestions could increase sales by a LOT) and smart people must have been working on it for quite a while.
- What is the fundamental reason why this is a hard problem?
- What's up with the coffee tables specifically, could you, for the hell of it, look into that category and tell us what the actual related products are? Let us (fail to) guess how these products are related, but don't let us hanging :-)
I'm late - but that is actually called dynamic remarketing. You look at a certain category of item and then see ads (on amazon or off-site) for other items in that same category. If you actually bought the coffee table on a different device/browser/anywhere else.. then you'll see those ads for a while because they can't recognize that you actually made the purchase already.
Advertising is trained against ROI, not against what will "seem right" to the user.
Maybe in-market* furniture shoppers tend to spend a lot of money. Maybe furniture is a very profitable category. Even if the system is smart enough to assume there's only a 20% chance that you're in the process of significant furniture purchases, furniture ads may still be a better use of the ad slot than a lower value item where you have an 80% chance of being in-market.
Then why show the same damn coffee table over and over? Maybe that's more likely to return your attention to your furniture purchasing? I have no idea. Most likely, they don't know exactly either. Most likely, that's just what the highest-scoring current algorithm decided.
*The duration of "in-market" varies by category. Some product categories have a long consideration phase. For example car shoppers tend to spend 2-3 months considering alternative brands and models before they spend a few weeks narrowing down on a specific car configuration and exact pricing.
It's more like you bought a coffee table and you get coffee beans in the recommendations. Also, you buddy who you are in the same group with gets a coffee table recommendation.
For years we used only algorithms based on basic statistics but we've found places where the machine learned models out perform the simpler models.
This is the right way to approach it. Too many people are looking for "deep" as some sort of silver bullet for an ill defined problem they have. If you can't validate against a simple model trained properly you are already in trouble. Likewise if you don't understand how to evaluate your generalization issues and how/if a simpler model will improve them.
It strikes me that you could do this with an algorithmic approach - is there some additional factor when building PCBs that's specifically hard?
Is this one of those things like the bin packing problem [1] where on first glances you'd expect it to have a definitive solution but it's actually deceptively very hard?
We use ML to model complex interactions in electrical grids in order to make decisions that improve grid efficiency, which has been (at least in the short term) more effective than using an optimizer and trying to iterate on problem specification to get better results.
Generally speaking, I think if you know your data relationships you don't need ML. If you don't, it can be especially useful.
I have a follow on question to this to all the respondents: Can you briefly describe the architecture you are using? Cloud-based offering vs self-hosted, software libraries used, etc...
199 comments
[ 1.8 ms ] story [ 238 ms ] threadyou're all idiots.
The primary advantage for customer is easier to use and troubleshoot faster.
https://www.sumologic.com/resource/featured-videos/demo-sumo...
overall the pattern matching stuff is pretty cool. Also, would like a see raw logs around this for when I am trying to debug event grouping errors based on the starting regex.
E.g. With LogReduce you can click on group and see log lines that belongs to it. IS that something that solves your problem, or are you looking for something else.
Feel free to send me an email (it is on my profile).
One of the products the company I work for sells more or less attempts to find duplicate entries in a large, unclean data set with "machine learning."
The value added isn't in the use of ML techniques itself, it's in the hype train that fills the Valley these days: our customers see "Data Science product" and don't get that it's really basic predictive analytics under the hood. I'm not sure the product would actually sell as well as it does without that labeling.
To clarify: the company I work for actually uses ML. I actually work on the data science team at my company. My opinion is that we don't actually need to do these things, as our products are possible to create without the sophistication of even the basic techniques, but that battle was lost before I joined.
So you are misleading your customers through omission? This is the kind of thing that makes people question anyone stating they are using ML. Those of us actually implementing ML techniques (aka training neural nets and automating processes with data) are met with unnecessary skepticism as a result.
edit: OP clarified his position since this post so take that into account when reading.
My point was that using ML, even though we don't need to, "adds value" by virtue of the hype train. We need ML to sell products, not to create them.
I do agree that this sort of arrangement lends itself to supporting skepticism around AI and ML. On the other hand I don't think that's a bad thing.
There's an interesting group of marginal approaches that have existed as-is for years, but have increasingly focused their branding on machine learning as its profile has risen.
You can reduce 90% of ML to this. Even neural networks are based on statistics.
If I have to draw a line between statistics and ML is that ML learns, it means it can predict things, however statistics only gives you information about the data you have. But for sure statistics and ML overlap a lot.
If you ask me for the most likely new value for a dataset, I won't know. But if I graph a few things and then write a function to spit back the current mean or median, is that machine learning?
I'm not trying to be snarky there, I agree that the bulk of ML tools are fundamentally just statistical tricks with some layer of abstraction. As a result, I have a lot of trouble knowing how much abstraction justifies the ML title. I see some people using "statistics to produce unintuitive solutions" as a standard, but that just begs that we ask unintuitive to who?
Caruana showed the cartoon of the difference between a statistician and a machine learning practitioner by showing a cliff. The statistician carefully inches to the edge, stomping her feet to see if the ground is still stable, then 10 meters before the edge she stops and draws her conclusions. The machine learning practitioner dives headfirst from the cliff, with a parachute that reads "cross-validation".
See also:
http://norvig.com/chomsky.html On Chomsky and the Two Cultures of Statistical Learning.
And http://projecteuclid.org/euclid.ss/1009213726 Statistical Modeling: The Two Cultures by Leo Breiman.
and this joke:
> Norvig teamed up with a Stanford statistician to prove that statisticians, data scientists and mathematicians think the same way. They hypothesized that, if they all received the same dataset, worked on it, and came back together, they’d find they all independently used the same techniques. So, they got a very large dataset and shared it between them.
> Norvig used the whole dataset and built a complex predictive model. The statistician took a 1% sample of the dataset, discarded the rest, and showed that the data met certain assumptions.
> The mathematician, believe it or not, didn’t even look at the dataset. Rather, he proved the characteristics of various formulas that could (in theory) be applied to the data.
Obviously no, ML uses statistics as statistics uses Maths. But not all ML uses statistics, some algorithms are biological inspired (swarm optimization) other uses theory of information for classification.
The point of ML is you learn something from data, not necessarily with statistics, although it is used in a lot of algorithms. But also function optimization is used in a lot of algorithms. The boundaries are very fuzzy, but for sure not all ML uses statistics and not all statistics are ML.
All the other Math areas call that kind of prediction by "interpolation". It's not a magical property that only ML has.
I'd draw the line by the name. An algorithm is ML if it includes the computer deriving a complex model based on data gathered on the field.
A lot of it went over the head because I don't know much classical statistics, but I read some articles by stats people that basically boiled down to the distinction not being in the techniques but in common assumptions, rigor, culture, etc.
If you have a problem that you want to classify some vectors, you have different ways to do it. You call all of them ML, but some use statistics, others use interpolation, other uses theory of information, etc. The model doesn't have to be complex or require a lot of data. Instead of saying all the different techniques you sum up saying ML.
In brief, you're going to run up against two types of data - categorical and continuous. (There are facets to this, eg ordinal, but these are really the elemental types of data). The relationship of datatype to independent/dependent variable is what determines what kind of analysis you may conduct.
Categorical Independent vs. Categorical Dependent, for example, is fairly restrictive, as makes logical sense. You may cross-tabulate, you may score likelihood based on previous observation, but obviously, because all of the data involved are non-numeric, there's no chance for regression, ANOVA, etc. Linear Regression is used when both independent and dependent variables are continuous, and cross-category differencing techniques like ANOVA may be used when the independent is categorical and the dependent is continuous.
The part you don't typically learn until grad school is when the independent is continuous and the dependent is categorical, ie, in ML, a classification problem. The standard statistical methods used as foundation for these problems are logistic regression, logit/probit. It's expansion of these methods that lead to ML in the first place.
> Categorical Independent vs. Categorical Dependent, for example, is fairly restrictive, as makes logical sense. You may cross-tabulate, you may score likelihood based on previous observation, but obviously, because all of the data involved are non-numeric, there's no chance for regression, ANOVA, etc
If you are implying that categorical -> categorical predictions are not ML: as a counter example, natural language is a categorical (words) input that could be used to predict any number of categorical variables (parse trees, semantic categories, etc). I think it's safe to say that the field of NLP is doing machine learning.
Yes, this means ML is "just" statistics - the distinction being that it is automated so you can run it on larger amounts of data quickly.
I thought this was pretty much an accepted definition.
In some ML algorithms you don't learn parameters. For example: some clustering algorithms are based on examples, not on parameters.
> Yes, this means ML is "just" statistics
So, a decision tree based on information theory would you call it statistics? Information theory and statistics are not clearly the same.
> I thought this was pretty much an accepted definition.
Machine Learning: A machine that learns (regardless it uses statistics, information theory, function optimization, biological inspiration or whatever)
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E." - Tom Mitchell
Ha, sounds like a classification problem! Let's use ML to find the boundary.
Statistics is about modelling the underlying probability distribution that generates your data. A convergence/generalization/etc result will usually be dependent on this underlying distribution.
ML is when you don't care much about the underlying distribution (modulo regularity assumptions), and your model doesn't even come from the same family at all.
I.e. linear regression is usually statistics, because you often believe the underlying data looks like f(x) ~ f(x0) + f'(x0)(x-x0)). Random forests are machine learning because you don't actually think the real world secretly has a random forest flotaing around.
Not really a new application though...
I would classify something like this blog post as ML, would you? http://stackoverflow.blog/2016/11/How-Do-Developers-in-New-Y...
A basic linear regression probably isn't ML, a backprop neural net clearly is, but somewhere between the two is a very fuzzy line between "statistics and data cleaning" and "actually machine learning". I think a lot of people have just pushed the ML angle of an already-reasonable approach to tie into that popularity.
It looks like that's not the standard view, though.
Neural networks are just functional approximators, so why isn't a linear regressor of k-th order (e.g. Taylor expansion up to k-th order) also considered "ML"? What's the distinction here?
Off topic: somehow the people not doing data cleaning because it is so boring and ML techniques do not need clean data end up with the worst overfitting I have ever seen.
[1] https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn...
And thanks for the info. I worked years ago on a training program for EKGs and it seemed like a field ripe for application of ML and AI.
We use neural nets to generate descriptors of videos where motion is observed, and classify events as normal/abnormal.
In our space, the recent AI / ML advances have made things possible that were simply not realistic before.
That being said, the hype around Deep Learning is getting pretty bad. Several of our competitors have gone out of business (even though they were using the magic of Deep Learning). For example, JustVisual went under a couple of months ago ($20M+ raised) and Slyce ($50M+ raised) is apparently being sold for pennies on the dollar later this month.
Yes, Deep Learning has made some very fundamental advances, but that doesn't mean it's going to make money just as magically!
There's a lot of "DL allows us to do X so we should make a product / service using DL to do X", rather than "We think there's value in something doing Y, what allows us to do Y? <research> DL allows us to do Y better than anything else, lets use DL"
You gave the example of Slyce. Their products are cool, but I can't help but think "is DL the best way to get that end result?" for lots of the things they do.
The exact things we do depends entirely on which department(s) are licensing it. Basically, anywhere there's a product image (from their own inventory to mobile to social) and we can provide some kind of help, we do. Every department needs totally different things, so it varies quite a bit...but it's all leveraging our core automated detection + recognition + search APIs.
I recently demonstrated a really simple bagged-decision tree model that "predicts" if the scanned part will go on to fail at downstream testing with ~95% certainty. I honestly don't have a whole lot of background in the realm of ML so it's entirely possible that I'm one of those dreaded types that are applying principles without full understanding of them (and yes I do actually feel quite guilty about it).
The results speak for themselves though - $1M/year scrap cost avoided (if the model is approved for production use) in just being able to tell earlier in the line when something has gone wrong. That's on one product, in one factory, in one company that has over 100 factories world-wide.
The experience has prompted me to go back to school to learn this stuff more formally. There is immense value to be found (or rather, waste to be avoided) using ML in complex manufacturing/supply-chain environments.
My only two misgivings about the program thus far: It is 1) pretty expensive and 2) geared towards working professionals rather than academics, but my employer is helping pay for a good chunk of the degree and I'm more interested in acquiring the skills and tools to go solve problems in industry as opposed to doing research.
Otherwise it has been great thus far. The program was attractive to me because it is somewhat marketed towards those that may not have a software background, but have problems in their industry that could benefit from a "proper" data science treatment. I've been referring to my application of the principles as "Six Sigma for the 21st Century" with managers/directors. I think the vast majority of HN would groan at that term, but it helps communicate the potential value to someone who has no technical background with software whatsoever (think old school manufacturing/operations types): Process improvement for environments with many variables that have practically unknowable inter-dependencies (as is the case with the project described in my original comment).
I don't think there's anything wrong the GP's achievement or post (it's all interesting stuff) but if something has not yet been implemented, it's worth nothing since there is "many a slip 'tween cup and the lip"
By the way, control engineering for industry used to be very difficult (but is paid very well), and requires knowledge of systems theory, differential equations, and physics. But with the advent of ML, I suspect that might change; things may get a lot easier.
I believe that the parent post means that with current simulation-based tools and large amounts of data generated from manufacturing processes, one can work directly with abstract machine learning models instead of creating physical models or approximations thereof---thus being able to dispose of the mathematical baggage of optimization/control theory and work with a black-box, general approach.
I disagree since we have very few guarantees about machine learning algorithms relative to well-known control approximations with good bounds; additionally, I think it's quite dangerous to be toying with such models without extensive testing in industrial processes, which, to my knowledge is rarely done in most settings by experts, much less people only recently coming into the field. Conversely, you're forced to consistently think about these things in control theory, which I believe, makes it harder to screw up since the models are also highly interpretable and can be understood by people
This is definitely not the case in high-dimensional models: what is the 3rd edge connected to the 15th node in your 23rd layer of that 500-layer deep net mean? Is it screwing us over? Do we have optimality guarantees?
How easy will it be to update your model if/when the downstream process changes?
At a previous job we had a process that relied heavily on visual inspection from employees. I often considered applying ML to certain inspection steps, but always figured it would be most useful for e.g. final inspection to avoid having to update the models frequently as the processes were updated.
I think I'm somewhat lucky in that with my product downstream processes are unlikely to change in a significant enough way to warrant "retraining" the model, but I guess that's probably the only way to handle that - retrain in the event of a significant process change. Our product stays fairly stable once it releases to production and the nature of the downstream processes is that they would have very little effect on the perceived severity of the defect at the final electrical test.
First, bagged decision trees are a little hard to interpret; what is the advantage of a bagged model vs the plain trees? Are you using a simple majority vote for combination? What are the variances between the different bootstraps?
Second - what do you mean by 95% ? Do you mean that out of 99999 good parts 4999 are thrown away? and one bad one is picked out as bad ?
Third - what is this telling you about your process? Do you have a theory that has evolved from the stats that tells you why parts are failing? This is the real test for me.. If the ML is telling you where it is going wrong (even if it's unavoidable/too expensive to solve) then you've got something real.
Unfortunately my concern would be that as it stands.. you might find that in production your classifier doesn't perform as well as it did in test... My worry has been generated by the fact that this same thing has happened to me !
Several times...
I'd hazard to guess that the 95% is the reduction in how many parts made it through the first test only to be caught later at the more expensive stage. So instead of binning 100 parts a month at that second stage, they now bin 5 parts a month and catch way more early on.
It sounds like the OP is using ML to identify flaws that simply just occur due to imperfections in the manufacturing process. That's life, it happens. You can know that they will occur without necessarily being able to prevent them because maybe there's some dust or other particulates in the air that deposit into the resin occasionally, or maybe the resin begins to cure and leaves hard spots that form bond flaws. There's heaps of possible reasons. It sounds more like the ML is doing classification of 'this too much of a flaw in a local zone' vs 'this has some flaws but it's still good enough to pass', which is how we operate with casting defects. For example, castings have these things called SCRATA comparitor plates, where you literally look at an 'example' tactile plate, look at your cast item, then mentally decide on a purely qualtative aspect which grade of plate it matches. Here [1] are some bad black and white photos of the plates.
[1] http://www.iron-foundry.com/ASTM-A802-steel-castings-surface...
To clarify on that 95% value because it is admittedly really vague: That's actually a 95% correct prediction rate. So far we get ~2.5% false-positives and ~2.5% false-negatives. 2.5% of the parts evaluated will be incorrectly allowed to continue and will subsequently fail downstream testing (no big deal). More importantly, 2.5% of parts evaluated will be wrongly identified as scrap by the model and tossed, but this still works out to be a massive cost savings because a lot of expensive material/labor is committed to the device before the downstream test.
Malik 'Poot' Carr: Naw, man, that ain't right.
D'Angelo Barksdale: Fuck "right." It ain't about right, it's about money. Now you think Ronald McDonald gonna go down in that basement and say, "Hey, Mista Nugget, you the bomb. We sellin' chicken faster than you can tear the bone out. So I'm gonna write my clowny-ass name on this fat-ass check for you"?
Wallace: Shit.
D'Angelo Barksdale: Man, the nigga who invented them things still workin' in the basement for regular wage, thinkin' up some shit to make the fries taste better or some shit like that. Believe.
[pause]
Wallace: Still had the idea, though.
2.5% of what, though? if only 1 in a million parts are actually bad, you're still tossing many more good parts than bad parts.
http://blog.mldb.ai/blog/posts/2016/01/ml-meets-economics/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
In my case the % occurrence of the defect was very high and the False-Positive cost is also very high so my tool could provide value without being too stellar of a model.
We use ML/Deep Learning for customer to product recommendations and product to product recommendations. For years we used only algorithms based on basic statistics but we've found places where the machine learned models out perform the simpler models.
Here is our blog post and related GitHub repo: https://aws.amazon.com/blogs/big-data/generating-recommendat... https://github.com/amznlabs/amazon-dsstne
If you are interested in this space, we're always hiring. Shoot me an email ($my_hn_username@amazon.com) or visit https://www.amazon.jobs/en/teams/personalization-and-recomme...
its amazing that the vaunted Amazon technology is unable to figure out an algorithm that would satisfy a user's deep desire "please stop plastering Jeffrey Tambor's lipstick and mascara covered face on my startup screen, I've gotten tired of looking at it for the past year"
Not saying it works, but that'd be the goal.
Would the standard models used allow for the fact that humans could, after buying a coffee table, choose to click on the coffee table in anticipation of then getting suggestions for similar furniture. Presumably the machine sees that the end goal of those continually clicking the same item is actually to arrive at similar items .. but wouldn't it be an obvious optimisation for Amazon to set the ML up to already look deeper than the first page reached.
I have a similar thing with Amazon, I don't know how you're supposed to access the bestseller list for a product type. I just know that if you search a product and follow related products that you eventually get a "#5 in ObscureProduct" tag and that tag takes you to the list of the top-10 models of ObscureProduct available. That sort of learnt navigation must play havoc with a suggestion algo (but IMO would be very easy to fix with just a link for any specific enough item to the 'top 10 in this category').
I get suggestions for travel guide books for the same country I visited a year ago for which I purchased a guide book.
Seriously though, I don't understand why it's so hard to take this effect into account, as there should be a very strong negative correlation between a purchase in a given category and the probability of buying an article from that category in the near future, so even a simple ML algorithm should be able to pick this up easily. Anyone here who can explain why this is difficult?
(I'm not saying it is a good or likely explanation)
I used to work for a company that implemented similar recommendation services. We approached this problem by modelling whether or not a category was likely to have recurring purchases.
I haven't looked at coffee tables specifically, but I know when I've looked at home products in the past I've been surprised at how frequently people will buy two large items, e.g. TVs or furniture, within a short period. That said, I agree there is room for improvement here. We're constantly running experiments to improve the customer experience, I have faith that in the limit things will improve. Again, we have no shortage of experimental power so if you'd like to join in the experimentation let me know :)
My dream is that I go to Amazon.com and there are a ton of different unrelated products that people who purchase similar things as me buy. So if I only buy "buy it for life" kitchen equipment, it doesn't show me the most popular but crappy version of something, it shows me the one that I'd actually purchase.
Such an easy problem with suuuuuch a difficult solution though. Not to mention the obvious privacy concerns there.
Oh well, I know that they have good people working on the solution, and no chance I could do it better :p
This topic must be extremely interesting (good suggestions could increase sales by a LOT) and smart people must have been working on it for quite a while.
- What is the fundamental reason why this is a hard problem?
- What's up with the coffee tables specifically, could you, for the hell of it, look into that category and tell us what the actual related products are? Let us (fail to) guess how these products are related, but don't let us hanging :-)
I don't think it is.
edit:typo
Maybe in-market* furniture shoppers tend to spend a lot of money. Maybe furniture is a very profitable category. Even if the system is smart enough to assume there's only a 20% chance that you're in the process of significant furniture purchases, furniture ads may still be a better use of the ad slot than a lower value item where you have an 80% chance of being in-market.
Then why show the same damn coffee table over and over? Maybe that's more likely to return your attention to your furniture purchasing? I have no idea. Most likely, they don't know exactly either. Most likely, that's just what the highest-scoring current algorithm decided.
*The duration of "in-market" varies by category. Some product categories have a long consideration phase. For example car shoppers tend to spend 2-3 months considering alternative brands and models before they spend a few weeks narrowing down on a specific car configuration and exact pricing.
I guess this would make more sense.
Is this one of those things like the bin packing problem [1] where on first glances you'd expect it to have a definitive solution but it's actually deceptively very hard?
[1] https://en.wikipedia.org/wiki/Bin_packing_problem
Generally speaking, I think if you know your data relationships you don't need ML. If you don't, it can be especially useful.