I've long been interested in learning about AI and deep learning, but to this day haven't done much that truly excites me within the field. It feels more or less impossible to make anything significant without Google-scale databases and Google-scale computers. AI really does make it easier for the few to jump far ahead, leaving everyone behind.
I also agree that a lot the news around AI is just hype.
Honestly, I'm yet to see anything practical come out of AI.
But hey, if something eventually does, I'm all for it.
Speaking of Google-scale databases, I've always felt the unhyped hero of deep learning was the data sets themselves. I'd be happy to see one less book about deep learning, and one more book on how you can gather and build a quality dataset yourself. Granted, I think gathering and preparing the data is a broader and most difficult task in many ways.
I'm kinda curious now where those giant datasets will come from now that there's a big push for privacy with things like GDPR preventing some random researcher from just buying data off whatever data mining corp is most relevant to their AI's purpose
Many traditional businesses (e.g. banks, insurance) collect customer data internally, and don’t share it with anyone. That’s not likely to change much. For many of these businesses, the customer is other businesses, and privacy rules aren’t even applicable.
You won’t hear about these projects outside industry specific publications, if at all.
I think a lot of GOFAI approaches ought to be revisited to see whether they benefit from the new perceptual and decision capabilities of Deep Learning systems. Alex Graves's papers are particularly good at this.
I can recommend his new book, "The book of Why" very highly. Even though I am very familiar with Bayes nets, I discovered that that a lot of progress has been made in that side of AI.
While I basically agree, really it ought to be called "AI autumn is well on its way", since I'm not sure we're into actual winter (i.e. dramatic reduction in $$ available for research) quite yet. But, probably soon.
It's the same story again like exaggerating the influence of IoT 5 years ago. The whole thing is exaggerated to raise money from investors and attract customers instead of actually buidling superior product
It's 99% marketing and places like HN and reddit eat it up and try to hype it up even more. When you confront these characters about the basis on which they claim AI will solve whatever problem or evolve to whichever point, they only reply "it'll only keep getting better (given time, data, resources, brains, etc)"
It's a buzzword people brainlessly use to fetishize technological progress without understanding the inherent limitations of the technology or the actual practicality and real-life results outside of crafted demos or specific problem domains (for example Alpha Go beating a grandmaster has almost no bearing on a problem like speech cognition).
It's turned me off a lot from reading about advances in the field because I know like a lot of science releases that most of it is empty air that won't really have bearing on the actual software I use (I've watched the past two Google's I/O where pretty much every presentation mentions AI, but the Android experience still remains relatively stale).
IMHO Quantum computing is as well hyped as Cold Fusion and shares some of its properties. Until "quantum supremacy" occurs or something that will show a real speedup we won't hear that much from it.
I think that we will have AI Winter once we see the true limitations that face us having a level 5 fully autonomous self driving car. The other thing we will see happen is the deflation of the AdTech bubble. Once we see both of these events occurring that should start the AI Winter.
I agree. The AdTech sphere is keeping the current hype alive more than anything else. There's some obvious imbalances in AdTech that should lead it to a damning end soon enough.
AI and machine learning is a tool. Like any other tool it's perfect for some problems and doesn't work well for other. Pick the right tools for the problem that you are working on. Don't follow the hype and don't use AI/ML just for sake of using it.
“To build the database, the hospital said it spent nearly two years to study more than 100,000 of its digital medical records spanning 12 years. The hospital also trained the AI tool using data from over 300 million medical records (link in Chinese) dating back to the 1990s from other hospitals in China. The tool has an accuracy rate of over 90% for diagnoses for more than 200 diseases, it said.“
Well first off: letters to investors are among the most biased pieces of writing in existence.
Second: I'm not saying connectionism did not succeed in many areas! I'm a connectionist by heart! I love connectionism! But that being said there is disconnect between the expectations and reality. And it is huge. And it is particularly visible in autonomous driving. And it is not limited to media or CEO's, but it made its way into top researchers. And that is a dangerous sign, which historically preceded a winter event...
I agree that self-driving had/have been overhyped over the previous few years. The problem is harder than many people realize.
The difference between the current AI renaissance and the past pre-winter AI ecosystems is the level of economic gain realized by the technology.
The late 80s-early 90s AI winter, for example, resulted from the limitations of expert systems which were useful but only in niche markets and their development and maintenance costs were quite high relative to alternatives.
The current AI systems do something that alternatives, like Mechanical Turks, can only accomplish with much greater costs and may not even have the scale necessary for global massive services like Google Photos or Youtube autocaptioning.
The spread of computing infrastructure and connectivity into the hands of billions of global population is a key contributing factor.
> The difference between the current AI renaissance and the past pre-winter AI ecosystems is the level of economic gain realized by the technology
I would argue this is well discounted by level of investment made against the future. I don't think the winter depends on the amount that somebody makes today on AI, rather on how much people are expecting to make in the future. If these don't match, there will be a winter. My take is that there is a huge bet against the future. And if DL ends up bringing just as much profit as it does today, interest will die very, very quickly.
Because there is a dearth of experts and a lack of deep technical knowledge among many business people, there are still a great many companies that have not yet started investing in deep learning or AI despite potential profits based on current technology. Non-tech sectors of the economy are probably underinvesting at the moment.
This is analogous to the way electricity took decades to realize productivity gains in the broad economy.
That said, the hype will dial down. I am just not sure the investment will decrease soon.
While I agree there is underinvestment in non-tech sectors, I don't see why that would change and they will use deep learning. There are lots of profitable things in non-tech sectors that can be done with linear regression but not done.
There are lots of things in the non-tech sector that can be automated with simple vanilla software but isn't. To use AI instead, you need to have 1) sophisticated devs in place, 2) a management that gets the value added, 3) lots of data in a usable format, 4) willingness to invest & experiment. Lots of non-tech businesses are lacking one if not all of these.
This. And at the end of the day, deep learning is just a more sophisticated version of linear regression. (To listen to some people talking, you'd think if a machine just curve-fits enough data-points, it'll suddenly wake up and become self-aware or something? The delusion is just unbelievable!)
> I agree that self-driving had/have been overhyped over the previous few years. The problem is harder than many people realize.
The current road infrastructure (markings, signs) has been designed for humans. Once it has been modernized to better aid the self-driving systems, we don't probably need "perfect" AI.
But current signs designed for humans work well. They're machine readable (traffic sign detection is available from basically all manufacturers) and can (usually) be understood without prior knowledge and don't need much change over decades. I think there are few examples where messages were designed for computers but are easy to understand independent of the system, manufacturer. ASCII encoded text files are the only thing that come to mind.
How can something be biased when it's listing facts?
Those are actual features that are available today to anyone, that were made possible by AI. Do you think it would be possible to type "pictures of me at the beach with my dog" without AI in such as short time frame? Or to have cars that drive themselves without a driver? These are concrete benefits of machine learning, I don't understand how that's biased.
How can something be biased when it's listing facts?
If there are 100 facts that indicate a coming AI winter, and Brin just talks up the 15 facts that indicate AI's unalloyed dominance, that's definitely biased.
First, what are said 100 facts? The article looks at fairly mundane metrics such as number of tweets or self-driving accidents...
Second, I'm not quite sure that's how it works. Like in mathematics, if your lemma is X, you can give a 100 examples of X being true, but I only need a single counter-example to break it.
In my opinion a single valid modern use-case of AI is enough to show that we're not in an AI winter. By definition an AI winter means that nothing substantial is coming out of AI for a long period of time, yet Brin listed that Google alone has had a dozen in the past few years.
YouTube captioning in English works surprisingly well, the improvement over the last few years is huge. It still chokes on proper nouns but in general it mostly works.
Mh no it does not. It is just a source of hilarity apart from a few very specific cases (political speeches mostly, because of their slow pace, good english and prononciation I guess).
Every time I activate it I am in for a good laugh more than anything actually useful.
I think it's a bit like self-driving cars in the sense that it's good enough to be impressive but not good enough to be actually usable everywhere. Of course self-driving is worse because people seldom die of bad captions.
Google's captioning works well when people speak clearly and in English. Google translate works well when you translate well written straightforward text into English. It's impressive but it's got a long way to go to reach human grade transcription and translation.
I think when evaluating these things people underestimate how long the tail of these problems is. It's always those pesky diminishing returns. I think it's true for many AI problems today, for instance it looks like current self-driving car tech manages to handle, say, 95% of situations just fine. Thing is, in order to be actually usable you want something that critical to reach something like 99.999% success rate and bridging these last few percent might prove very difficult, maybe even impossible with current tech.
What's important to remember, I think, is that we should not compare YouTube auto captions to human made captions, because auto captions were not created as a substitute for human made captions - if it wasn't for auto captioning, all these videos wouldn't get any captions at all. They may never be perfect, but they're not designed to be, they're creating new value on their own. And IMO they crossed the threshold of being usable, at least for English.
It works for general purpose videos. Transcripts of any kind appear to stop working whenever there's domain knowledge involved. That doesn't matter for most youtube videos but is crucial if you want to have a multi purpose translator/encoder.
A. Cooper had a nice example of this kind: a dancing bear. Sure, the fact that bear dances is very amusing, but let it not distract us from the fact that it dances very very badly.
Have now, ah! Philosophy,
Law and medicine,
And unfortunately also theology
Thoroughly studied, with great effort.
Here I am, I poor gate!
And I'm as smart as before
word for word not completely bad, but then it breaks when we have to translate 'Tor'. Google Translate is clueless because it is unable to derive that here the 'fool' is meant.
It's unable to 'understand' that 'I poor gate' makes no sense at all.
On the other hand Deepl gives translations for news articles that are of such quality that it allows me to read international news as if it were local. Definitely useful.
DeepL is better, but it basically has the same problems. I understand both German and English - and I can easily detect where DeepL also shifts the meaning of sentences, sometimes even to mean the opposite. DeepL like Google Translate has no concept for 'is the meaning preserved?'.
You may think that you can now read German news, but in fact you would not know if the sentence meaning has been preserved in the English translation. The words itself might look as if the sentence makes sense - but the meaning is actually shifted - slight differences, but also possibly the complete opposite.
The translation also does not give you any indication where this might be and where the translation is based on weak training material or where there is some inference needed for a successful translation.
Not sure if I agree. If you have some knowledge of the language is still mostly easier to translate the words you don't understand. Easy texts work, anything more complicated (e.g. science articles) not really.
Hi, why in your analysis you spoke only about the companies that are not doing so well in self driving leaving out waymo success story?
They are already have been hauling passengers without a safety pilot since last October. I guess without the minimum problem otherwise we would have heard plenty in the news like it happened for Tesla and Uber accidents.
Is it not too convenient to leave out the facts that contradict your hypothesis?
The disconnect between the expectations and reality.
Well, to some degree you will always have it, but that is a very interesting facet, and I concur that it is somewhat increasing. It has in the recent past, and it still is..
> And it is not limited to media or CEO's, but it made its way into top researchers.
Interesting. One would assume that top researchers would be more "immune" than anyone else, but if it's true what you say and it already made its way..
For the record, I think you are onto something here.
What would be your favorite pet theory that explains this phenomenon?
Do you believe it has something to do with this culture of "fake it till you make it", as if we have forgotten the value of honesty?
" letters to investors are among the most biased pieces of writing in existence. "
Maybe true but they are words that are about things which are either true or not true. Has nothing to do where the words were shared. Saying they are on an investment letter so not relevant seems very short sighted.
But just looking at the last 12 months it is folly to say we are moving to a AI winter. Things are just flying.
Look at self driving cars without safety drivers or look at something like Google Duplex but there are so many other examples.
Of course Google (or any other company) aren't going to blatently lie in a letter to investors (that kind of thing gets you sued) but it's pretty easy to spin words to sound more impressive than they may actually be.
Using the list provided, one example
"caption over a billion videos in 10 languages on YouTube;" - This doesn't say how accurate the captions acutally are. In my experience youtube captioning even of english dialect isn't exactly great. For one example try turning on the captions on this https://www.youtube.com/watch?v=bQJrBSXSs6o
so it's true I'm sure to say they've captioned the videos AI based techniques, but that doesn't mean they're a perfected option.
Also (purely anecodtally) Google translate also isn't exactly perfect yet either...
... I don't think I understand that video even with my own ears. YouTube captioning has actually significantly improved from it's previous hilarious state
Making cars that drive safely no current, busy roads is a very difficult task. It is not surprising that the current systems do not do that (yet). It is surprising to me how well they still do. The fact though that my phone understands my voice and my handwriting and does on the fly translation of menus and simple requests is a sign of a major progress, too.
AI is overhyped and overfunded at the moment, which is not unusual for a hot technology (synthetic biology; dotcoms). Those things go in cycles, but the down cycles are seldom all out winters. During the slowdowns best technologies still get funding (less lavish, but enough to work on) and one-hit wonders die, both of which is good in the long run. My friends working in biology are doing mostly fine even though there are no longer "this is the century of synthetic biology" posters at every airport and in every toilet.
completely overblown..
'understand' isn't true understanding
'recognize and distinguish' is nothing like how humans do it
'helping diagnose diseases' notice they don't say the AI is doing the diagnoses anymore
This is less useless than you think. Captioning video could allow for video to become searchable as easily as text is now searchable. This could lead to far better search results for video and a leap forward in the way people produce and consume video content.
They mean useless in the end result. Of course having perfect captions could potentially allow indexable videos, but the case is that the captions suck. They're so bad in fact that it's a common meme on Youtube comments for people to say "Go to timestamp and turn on subtitles" so people can laugh at whatever garbled interpretation the speech recognition made.
Have you used/tried them recently? The improvement relative to 5 years ago is major.
At least in English, they are now good enough that I can read without listening to the audio and understand almost everything said. (There are still a few mistakes here and there but they often don’t matter.)
Yes I’ve had to turn them off on permanently. Felt I could follow video better without sound often than with subtitles.
I tried to help a couple channels to subtitle and the starting point was just sooo far from the finished product. I would guess I left 10% intact of the auto-translation. Maybe it would have been 5% five years ago; when things are this bad 100% improvement is hard to notice.
It is super cool how easy it is to edit and improve the subtitles for any channel that allows it.
I'd say the current Youtube autocaptioning system is at an advanced nonnative level (or a drunk native one :)) and it would take years of intensive studying or living in an English-speaking country to reach it.
The vast majority of English learners are not able to caption most Youtube videos as well as the current AI can.
You underestimate the amount of time required to learn another language and the expertise of a native speaker. (Have you tried learning another language to the level you can watch TV in it?)
Almost all native speakers are basically grandmasters of their mother tongue. The training time for a 15-year-old native speaker could be approx. 10 hours * 365 days * 15 years = 54,750 hours, more than the time many professional painists spent on practice.
Not true. The problem with Google captioning and translate is that unlike a weak speaker it makes critical mistakes completely misunderstanding the point.
A weak speaker may use a cognate, idiom borrowed from their native tongue or a similar wrong word more often. The translation app produces completely illegible word salad instead.
I was talking exclusively about auto-captioning, which has >95% accuracy for reasonably clear audio. Automatic translation still has a long way to go, I agree.
To be honest, as the other child comment said, I too have noticed they have gotten way better in the last 5 years. Also, the words of which it isn't 100% sure are in a slightly more transparent gray than the other words, which kind of helps.
You don't need amazing transcription to search a video. A video about X probably repeats X multiple times, and you only really need to detect it properly once.
As for the users, sure the translation may not be perfect, but I'm sure if you were deaf had no other way of watching a video, you would be just fine with the current quality of the transcription.
Often you need exactly that. Because it's the unique words the machine will get wrong. If you look for machine learning tutorials/presentations that mention a certain algorithm, the name of it must be correctly transcribed. At the moment, it appears to me that 95%+ of words work but exactly the ones that define a video often don't. But then again getting those right is hard, there's not much training data to base it on.
Hey, a small advice for the future: never build your belief entirely on a youtube video of a demo. In fact, never build your belief based on a demo, period.
This is notorious with current technology: you can demonstrate anything. A few years ago Tesla demonstrated a driverless car. And what? Nothing. Absolutely nothing.
I'm willing to believe stuff I can test myself at home. If it works there, it likely actually works (though possibly needs more testing). But demo booths and youtube - never.
Almost anything that has to do with image understanding is entirely AI. Good luck writing an algorithm to detect a bicycle in an image. This also includes disease diagnostic as most of those have to do with analyzing images for tumors and so on.
Also, while a lot of these can be seen as "improvements", in many cases, that improvement put it past the threshold of actually being usable or useful. Self-driving cars for example need to be at least a certain level before they can be deployed, and we would've never reached that without machine learning.
I agree, the effects can be very impressive. I meant, that what is achievable is quite clear now and that we need a major innovation/steps for the next leap
This is one of the areas I’m most enthusiastic about but … it’s still nowhere near the performance of untrained humans. Google has poured tons of resources into Photos and yet if I type “cat” into the search box I have to scroll past multiple pages of results to find the first picture which isn’t of my dog.
That raises an interesting question: Google has no way to report failures. Does anyone know why they aren’t collecting that training data?
They collect virtually everything you do on your phone. They probably notice that you scroll a long way after typing cat and so perhaps surmise the quality of search results was low.
Doesn’t that seem like a noisy signal since you’d have to disambiguate cases where someone was looking for a specific time/place and scrolling until they find it?
I’ve assumed that the reason is the same as why none of the voice assistants has an error reporting UI or even acknowledgement of low confidence levels: the marketing image is “the future is now” and this would detract from it.
I would not call it the “AI winter”. If you look at what people have called AI over time, the definition and the approaches have evolved (sometimes drastically) over time.
Instead of being stuck on the fact that deep learning and the current methods seem to have hit a limit I think I am actually excited about the fact that this opens the door for experimenting other approaches that may or may not build on top of what we call AI today.
Yeah, the problem is deep learning sucked a bunch of money - essentially took a loan against the future in the form of VC investments. And if that loan does not get payed, for the next few years you may not afford to explore all that other stuff.
Technically, VCs are not loaning you money. They’re more likely betting on you. It’s true that maybe they’ll be more reluctant to place big bets, but as with everything in life the VC optimism is cyclical.
Sure it is not technically a loan. But it carries the same sentiment change when it blows up. People get extremely cautious, to the point of skipping some really good ideas. And not just those VC's that made the bets but everyone else too. Fear spreads just as effectively as hype.
Perhaps it'd be more correct to call it a "Strong AI Winter". We're no closer to "aware" machines. We've simply gotten very good at automating tasks that were once difficult to automate.
A friend that’s more optimistic about Strong AI once said that the ML that goes on today will probably serve the purpose of driving the peripheral sense organs of a future AI. Although it stretches a bit what’s possible today I could see that. I would call this a win if this ends up happening although I still belive we’re hundreds of years away from Strong AI.
This ability of DL to convert streams of raw noisy data into labeled objects seems like exactly what's needed to solve an intelligent agent's perceptual grounding problem, where an agent that's new to the world must bootstrap its perception systems, converting raw sensory input into meaningful objects with physical dynamics. Only then can the agent reason about objects and better understand them by physical interaction and exploration. This is one of the areas where symbolic AI failed hardest, but DL does best.
With some engineering, it's easy to imagine how active learning could use DL to ground robot senses - much like an infant human explores the world for the first year of life, adding new labels and understanding their dynamics as it goes.
I suspect the potential for DL's many uses will continue to grow and surprise us for at least another decade. If we've learned anything from the past decade of DL, it's that probabilistic AI is surprisingly capable.
Disclaimer: I am a lay technical person and don't know much about AI.
I find this article somewhat condescending. I look at all the current development as stepping stones to progress, not an overnight success that does everything flawlessly. I imagine the future might be some combination of different solutions, and what the author proposes may or may not play a part in it.
I don't see how systematically accurate image classifiers and facial recognition systems built on deep learning is a 'dead end'. Products are products. If deep learning has led to actual profits in actual companies, it's not a dead end. As to whether this leads to AGI is a completely different question.
The point is that the profits may not be as grand as the current level of hype may indicate
Edit: additionally it could be a dead end because the hype tends to narrow the directions we explore with ML. If everyone is obsessing about DL, we could be infuriatingly ignoring other research directions right under our noses.
I agree. Overhype is annoying, but it happens with every technological advance. So does the inevitable backlash jump from "some people claim too much" to "this whole field is a bubble with no real gains". Both are cliched viewpoints that give a false sense of being "in the know" without helping anyone navigate change effectively.
1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!)
2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated.
3. Partially true. But self-driving is a sophisticated area by itself, DL is part of it, it can't really put full claim on its potential future success or ultimate downfall.
4. Gary Marcus isn't an established figure in DL research.
AI winter will ultimately come. But it is because people will become more informed about DL's strengths and limits, thus becoming smarter to tell what is BS what is not. AGI is likely not going to happen just with DL, but that is no way meaning it is a winter. DL has revolutionized the paradigm of Machine Learning itself, the shift has now complete, it will stay for a very very long time, and the successor is likely to build upon it not subvert it completely as well.
Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about.
1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst...
2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere near that ease in scaling.
3) self driving is the poster child of current "AI-revolution". And it is where by far most money is allocated. So if that falls, rest of DL does not matter.
4) Not that this matters, does it?
The scaling argument in the article doesn't make any sense. There are rhetorical queries like "does this model with 1000x as many parameters work 1000x as well?" but what it means to scale or perform are not clearly or consistently defined - let alone defined in a way that would make your point about the utility of the advances.
OpenAI's graph shows new architectures being used with more parameters because people are innovating on architecture and scale at the same time. Arguing that old methods "failed to scale" is like arguing that processor development was a failure because Intel had to develop a 486 instead of making a 386 work with more transistors (or more something).
And what does CFD have to do with anything, except maybe an odd attempt to argue from authority? Can you formalize from CFD a notion of "scaling well" well that anyone else agrees is useful for measuring AI research?
CFD was merely used as an example of something that does scale well. I'm not sure it was the best example, since CFD isn't very common. But basically you have a volume mesh and each cell iterates on the Navier-Stokes equation. So if you have N processor cores, you break the mesh in N pieces, each of which get processed in parallel. Doubling the number of cores allows you process double the amount in the same time, minus communication loses (each section of the mesh needs to communicate the results on its boundary to its neighbors).
I don't fully understand the graph, but it looks like his point is that Alpha Go Zero uses 1e5 times as many resources than AlexNet, but does not produce anywhere near 10,000 times better results. We saw that with CFDt 1e5 more cores resulted in 1e5 better results (= scales). The assertion is that DL's results are much less than 1e5 better, hence it does not scale.
Basically the argument is:
1. CFD produces N times better results given N times more resources [this is implied, requires a knowledge of CFD]. That is, f(ax) = a f(x). Or, f(ax) = 1 a * f(x).
2. Empirically, we see that DL has used 1e5 more resources, but is not producing 1e5 times better results. [No quantitative analysis of how much better the results are is given]
3. Since DL has f(a * x) = b * a * f(x), where b < 1, DL does not scale. [Presumably b << 1 but the article did not give any specific results]
This isn't a very rigorous argument and the article left out half the argument, but it is suggestive.
Thanks for that, that is essentially my point. Agree it is not very rigorous, but it gets the idea across. By scalable we'd typically think "you throw more gpu's at it and it works better by some measure". Deep learning does that only in extremely specific domains, e.g. games and self play as in alpha go. For majority of other applications it is architecture bound or data bound. You can't throw more layers, more basic DL primitives and expect better results. You need more data, and more phd students to tweak the architecture. That is not scalable.
More compute -> more precision is just one field's definition of scalable... Saying that DNNs can't get better just by adding GPUs is like complaining that an apple isn't very orange.
To generalize notions of scaling, you need to look at the economics of consumed resources and generated utility, and you haven't begun to make the argument that data acquisition and PhD student time hasn't created ROI, or that ROI on those activities hasn't grown over time.
Data acquisition and labeling is getting cheaper all the time for many applications. Plus, new architectures give ways to do transfer learning or encode domain bias that let you specialize a model with less new data. There is substantial progress and already good returns on these types of scalability which (unlike returns on more GPUs) influence ML economics.
OK, the definition of scalable is crucial here and it causes lots of trouble (this is also response to several other posts so forgive me if I don't address your points exactly).
Let me try once again: an algorithm is scalable if it can process bigger instances by adding more compute power.
E.g. I take a small perceptron and train it on pentium 100, and then take a perceptron with 10x parameters on Core I7 and get better output by some monotonic function of increase in instance size (it is typically a sub linear function but it is OK as long as it is not logarithmic).
DL does not have that property. It requires modifying the algorithm, modifying the task at hand and so on. And it is not that it requires some tiny tweaking. It requires quite a bit of tweaking. I mean if you need a scientific paper to make a bigger instance of your algorithm this algorithm is not scalable.
What many people here are talking about is whether an instance of the algorithm can be created (by a great human effort) in a very specific domain to saturate a given large compute resource. And yes, in that sense deep learning can show some success in very limited domains. Domains where there happens to be a boatload of data, particularly labeled data.
But you see there is a subtle difference here, similar in some sense to difference between Amdahl's law and Gustafson's law (though not literal).
The way many people (including investors) understand deep learning is that: you build a model A, show it a bunch of pictures and it understands something out of them. Then you buy 10x more GPU's, build model B that is 10x bigger, show it those same pictures and it understands 10x more from them. Look I, and many people here understand this is totally naive. But believe me, I talked to many people with big $ that have exactly that level of understanding.
I appreciate the engagement in making this argument more concrete. I understand that you are talking about returns on compute power.
However, your last paragraph about how investors view deep learning does not describe anyone in the community of academics, practitioners and investors that I know. People understand that the limiting inputs to improved performance are data, followed closely by PhD labor. Compute power is relevant mainly because it shortens the feedback loop on that PhD labor, making it more efficient.
Folks investing in AI believe the returns are worth it due to the potential to scale deployment, not (primarily) training. They may be wrong, but this is a straw man definition of scalability that doesn't contribute to that thesis.
Almost all reasearch domains live on a log curve; a little bit gets you a lot to start with, but eventually you exhaust the easy solutions and a lot of work gets you very little improvement.
You’re arguing we haven’t reached the plateau at the top yet, but you’ve offered no meaningful evidence that is the case.
There are real world indicators that we are reaching diminishing returns for investment in compute and research now.
The ‘winter’ becomes a thing when it becomes apparent to investors that their financial bets are based off nothing more concrete than opinions like yours, when they don’t work out.
Are we there yet? Not sure, myself, I think we can get some more wins from machine generated architectures... but I can’t see any indication that the ‘winter’ isn’t coming sooner or later.
Investment is massively outstripping returns right now... we’ll just have to see if that calms down gradually, or pops suddenly.
History does not have a good story to tell about responsible investors behaving in a reasonable manner and avoiding crashes.
Thanks for taking the time to render the more specific argument! I still don't think this is suggestive in a way that should influence readers. Here are some ways in which a naive "10x resources != 10x improvement" argument can err:
- Improvement is hard to define consistently. Sometimes, improving classification accuracy by 0.5% means reducing error by 20%, and makes economic applications that have 100x the value or frequency of use.
- Resources used in training can be amortized over billions of times the same model is reused (much more cheaply). So even achieving an epsilon improvement in the expected utility of each inference can justify a massive increase in training cost.
- Some other notions of "better results" or "less expensive" include amount of training data required, social fairness of results, memory required or power used during inference, and so on. And there are major advances in current research on each of these better formalized axes!
That last bit is what is so frustrating in reading an article like this. The author is sweeping aside with vague arguments a great deal of work that has been written and justified to a much much higher standard of rigor (not just the VCs we all like to snark about). Readers should beware of trusting a summary like this without engaging directly with the source material.
I certainly encourage everybody to consult the source material! Man, this is a blog, opinion by default not perfect.
But when I hear the keyword "major advances" I'm highly suspicious. I had seen already so many such "major advances" that never went beyond a circle of self citing clique.
As a very concrete "major advance" consider Google Translate's tiny language models [1] that can beam to your phone, live in a few megabytes, and translate photographed text for you with low power usage. This was done with incredibly expensive centralized training, but checks every meaningful box for "scalable" AI.
This paper is amazing, and exactly what I was thinking of posting in response to that part of the article. The amount of research Deepmind is putting out is astonishing, and even if you are paying attention it's hard to keep up with it all. Here are just a few papers I've been looking at from the last few months, maybe none of them are advancements on the level of AlphaGo & Zero but they still show significant progress in a wide variety of areas.
And there are many others besides these, not to mention all the significant research being done by everyone else who isn't at Deepmind. The authors idea that interest and development of these topics is dying down or that Deepmind is running out of meaningful research to do just seems uninformed.
2)Why do you think DL doesn't scale? I am curious. It can easily leverage thousands of GPUs, training on 300 millions of images (https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef...). No other methods is even close to leverage that amount of computational power. I don't really know about CFD, but at least in ML land and dealing with ML problems, DL is very scalable, maybe only next to random forests style algorithm, where they effectively share nothing.
3)It does matter. In fact most valuable startup around DL are CV based startups, they are mainly located in China though.
CFD is good at using big machines "efficiently", but the cost of DNS scales as the cube of the Reynolds number which will never be tractable for most engineering problems. Apart from niche basic research on the edge of tractability, all the effort goes into modeling (RANS, DES, wall, etc.) to deliver statistically calibrated estimates of functionals of interest at feasible cost. Those methods actually don't "scale as well" (though the state of research is ahead of commercial software), but also don't need to because they can solve the problem in less time with less hardware. This situation is actually pretty similar to your DL analogy where more hardware provides diminishing returns for solving the actual problem.
"Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about."
Very weak to appeal to authority. The only true argument I can find against DL/ML/AI atm is the continuing appeal to authority by PhDs who have zero engineering knowledge, zero business sense and zero understanding of risk assessment.
FYI This post is about deep learning. It could be the case that neural networks stop getting so much hype soon, but the biggest driver of the current "AI" (ugh I hate the term) boom is the fact that everything happens on computers now, and that isn't changing any time soon.
We log everything and are even starting to automate decisions. Statistics, machine learning, and econometrics are booming fields. To talk about two topics dear to my heart, we're getting way better at modeling uncertainty (bayesianism is cool now, and resampling-esque procedures aged really well with a few decades of cheaper compute) and we're better at not only talking about what causes what (causal inference), but what causes what when (heterogeneous treatment effect estimation, e.g. giving you aspirin right now does something different from giving me aspirin now). We're learning to learn those things super efficiently (contextual bandits and active learning). The current data science boom goes far far far far beyond deep learning, and most of the field is doing great. Maybe those bits will even get better faster if deep learning stops hogging the glory. More likely, we'll learn to combine these things in cool ways (as is happening now).
Bayesian can be seen as a subset of deep learning or hell a superset.
AI is a superset and Machine learning is a subset of AI and most funding is in deep learning. Once Deep Learning hit the limit I believe there will be an AI winter.
Maybe there will be hype around statistic (cross fingers) which will lead to Bayesian and such.
I guess the point that digitalzombie is trying to make is most of what we call AI or ML or even Deep learning is simply extension of statistics on computers.
Things like the German tank problem or the problem of hardening airplanes during WW2 have that very AI'esque feel to it. Where you use data to build a model, then let that data from the model to change the model as it fits.
Also the whole thing about 'decision making' is either bayesian or frequency based models in nature. Most of these algorithms and math has long existed before the current boom.
Its just that the raw computing power and resources that you have today make it possible for you to deal with large amounts of data to stress test your models.
>Bayesian can be seen as a subset of deep learning or hell a superset.
eh-hem
DIE, HERETIC!
eh-hem
Ok, with that out of my system, no, Bayesian methods are definitely not a subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not involve neural networks with >3 hidden layers. It's just a different paradigm of statistics.
My mentor was very very adamant about Bayesian network and hierarchical as being deep learning.
He sees the latent layer in the hierarchical model as the hidden layer and the Bayesian just have a strict restrictions/assumptions to the network where as the deep learning is more dumb and less assuming. A few of my professor thinks that PGM, probability graphical model is a super set of deep learning/neural network.
But I am not an expert in Neural Network nor know the topic well enough to say such a thing. Other than was deferring to opinions of some one that's better than myself. So I'll keep this in mind and hopefully one day have the time to do more research into this topic.
Honestly as much as it is slightly irritating to see deep learning hogging all the glory, there's a lot of money being sloshed around and quite a bit of it is spilling over to non-deep learning too. Which is great. An AI winter may be coming, though I think it's at minimum several years off, since big enterprises are just getting started with the most hyped things. If the hype doesn't return on its promises enough for sustained investment (that's a rather big if since the low hanging fruit aren't yet all picked) then the companies and funding will eventually recede, maybe even trigger another winter, but just as it takes a while to ramp up, it will also take a while to course correct. In the meantime all the related areas get better funding and attention (and chance to positively contribute to secure further investment) that they'd otherwise not have since we'd still be stuck in the low funding model from the last winter.
I think the problem is the definition of AI. It appears most in the field define it as a superset of ML, encompassing all kinds of statistical methods and data analysis. For the general public, AI is a synonym for deep learning. When large companies speak about AI they always mean deep learning, never just a regression (probably also because many don't see a regression as intelligent). So AI in the public's perception could face a winter but much of the domain of machine learning would be unaffected.
>For the general public, AI is a synonym for deep learning
I'd contend for the general public, AI is a synonym for machines like: HAL; The Terminator; Star Trek's "Data"; the robots in the film "AI"; and so on.
We're nowhere remotely in the vicinity of that, and no-one even has any plausible ideas about how to start.
A random person outside of tech probably doesn't even know what deep learning is. They might have heard of it somewhere in passing.
The argument is that self-driving won't work because Uber and Tesla had well-publicized crashes. But I don't see how this tells us anything about other, apparently more cautious companies like Waymo. There seem to be significant differences in technology.
More generally, machine learning is a broad area and there's no reason to believe that different applications of it will all succeed or all fail for similar reasons. It seems more likely there will be more winners along with many failed attempts.
The argument is that self-driving won't work because Uber and Tesla had well-publicized crashes. But I don't see how this tells us anything about other, apparently more cautious companies like Waymo. There seem to be significant differences in technology.
Yes. I've been saying this for a while. Waymo's approach is about 80% geometry, 20% AI. Profile the terrain, and only drive where it's flat. The AI part is for trying to identify other road users and guess what they will do. When in doubt, assume worst case and stay far away from them.
I was amazed that anyone would try self-driving without profiling the road. Everybody in the DARPA Grand Challenge had to do that, including us, because it was off-road driving and you were not guaranteed a flat road. The Google/Waymo people understood this. Some of the others just tried dumping the raw sensor data into a deep learning system and getting out a steering wheel angle. Not good.
A lot of companies fear the ship's leaving without them. They try to rush ahead without thinking and then crashes. That's what it feels like every time a car company or another tech company say they're going to build the next self-driving car. I don't know why but I always feel like waymo is already 10,000 miles ahead.
Seriously. Frankly, based on Uber's culture I would have been surprised if they didn't kill at least one person with their self-driving efforts. It's a total non-data point. The fact that Uber got as far as they did without killing anyone is strong evidence that the problem is tractable.
As for Tesla - Tesla isn't even trying to make proper self-driving cars. Tesla's goal has always been assisted driving. However you feel about that, it's really not relevant to the success or failure of self-driving cars.
OP can't possibly have been ignorant of the fact that Waymo is the clear leader here with a substantial head start, and a proven record (and an actual fleet of self driving cars now on the road), and yet he chose not to mention it. That really undermines his credibility for me - he seems clearly more interested in making his point than in accurately engaging with reality.
> Nvidia car could not drive literally ten miles without a disengagement.
From the same source as the author cites, that's because their test runs are typically 5 miles and resuming manual control at the end of a test counts as a disengagement.
> Deepmind hasn't shown anything breathtaking since their Alpha Go zero.
Didn't this just happen? Maybe my timescales are off, but I've been thinking about AI and Go since the late 90s, and plenty of real work was happening before then.
Outside a handful of specialists, I'd expect another 8-10 years before the current state of the art is generally understood, much less effectively applied elsewhere.
I had the same response. AlphaZero was published like 5 months ago. Saying they've reached the end of the line because they haven't matched AlphaZero in six months is lame.
Also why does every single result has to be breathtaking? Here's a quick example, at IO they announced that their work on Android improved battery life by up to 30%. That's pretty damn impressive.
> Also why does every single result has to be breathtaking?
If you build the hype like say Andrew Ng it better be. Also if you consume more money per month than all the CS departments of a mid sized country, it better be.
In terms of hype you may be right, but it doesn't mean that if something doesn't live up to the hype of Andrew Ng or Elon Musk it won't still be pretty good.
For instance: even if Elon Musk doesn't colonize Mars but instead just builds the BFR, that would still be amazing; even if BFR is never build but falcon 9 becomes fully reusable that would be great; even if falcon 9 won't be fully reusable, the fact that it cut the launching cost to space is still pretty good.
Even if we don't achieve any great breakthroughs with AGI, the fact that we started to use transfer learning to diagnose human disseases is pretty amazing; the fact that a japanese guy used tensorflow on a raspbery pi to categorize real cucumbers by shape is amazing.
All of this stuff won't go away; people will not say "hey, let's just forget about this deep learning thing and put it in some dusty shelf, it's useless for now". Maybe it will take 20 or 50 more years, maybe it's a slow thaw, but how could this be a winter?
Exactly! Thank you. There’s a delusion that every result needs to be Nobel worthy, But the Nobel prize-worthy discoveries are all founded upon the boring stuff we don’t hear about it.
Honestly I think the raspberry pi indicates the (short-term) future of AI. Most of the “easy” problems have been solved (image classification, game playing), but the hard ones like NLP are orders of magnitude more complex and therefore elusive.
I’m happy to leave the hard problems for the PhDs and the big tech researchers. Go nuts, folks.
In the meantime, the applications for small-scale, pre-trained neural networks seem limitless. Manufacturing, agriculture, retail, pretty much any industry could make use of portable neural networks.
Because it's the only time we see it in action. The speech recognition of my Amazon Echo is still subpar (and it feels like it's getting worse each week) and ad targeting also hasn't really improved. Of all those claims that came with deep learning, Go was the only time where you really saw a result. I'm not sure which version of Android will bring the improved battery life (and which manufacturers) but I wouldn't be surprised if the 30% were a bit optimistic.
I get that a lot of services we use on a daily basis make use of deep learning to accomplish tasks. But I don't really see what has fundamentally changed over the past 5 years in the way I use services. Siri was introduced 7 years ago and while we have clearly made progress in voice recognition, it's nowhere close to what many had hoped.
The OP is obviously not keeping up with the field and has lot to learn about scientific approach. He basically uses the count of tweets from AndrewNg and crashes from risk-taking companies as indicator of "AI winter". He should have tried to look in to metrics such as number of papers, number of people getting in to field, number of dollars in VC money, number of commercial products using DL/RL etc. But you see, that's a lot of work and your conclusion might not align with whatever funky title you had in mind. Being an armchair opinion guy throwing link bait titles is much more easier.
I'll happily read your next post where you will include all of those. In fact amount of VC money spent in that field would only support my claim. And the number of papers is irrelevant. There were thousands of papers about Hopfield network in the 90's and where are all of them now? You see, all the things you point out is the surface. What really matters is that self driving cars crash and kill people, and no one has any idea how to fix it.
Anyway i also don't get what the issue is with the model from radiology. It is already that good?! This is impressive. One model is close to well trained experts.
Just today i had an small idea for a new product based on what google was showing with the capabilities to distinguis two people talking in parallel.
At the last Google IO i was impressed because in comparision to the previous years, ML created better and more impressive products.
I was listing for years at key nodes about big data and was never impressed. I hear now about ML and im getting impressed more and more.
I thought there would be more of a backlash / winter onset when people realize that Alexa is so annoying to deal with (and you basically have to learn a set of commands) because AI isn't that clever yet. Also, when people realize that autocorrect took a dive for making edits when Google started putting a neural net in charge. (No! Stop deleting random words and squishing spaces during edits).
In other words I figured it would be the annoyances at what "should be easy by now" that would get Joe CEO to start thinking "Hm. Maybe this isn't such a good investment." When measurements are made and reliable algorithmic results attract and keep more users than narrowly trained kind of finicky AIs.
I don't want there to be an AI winter, and it won't be as bad as before. There are a lot of applications for limited scope image recognition, and other tasks that we couldn't do before. Unfortunately,I do agree with the post that winter is on its way.
Deep learning maybe not the complete answer to gai, but it’s moving down the right path. Computers though are still years/decades away from approaching human brain power and efficiency, so my take is that current ai hype is 10 years too early - a good time to get in.
Time will tell. I think DL is amazing, but is no the right path towards solving problems such as autonomy. I think if you enter this field today, you should definitely take a look at other methods than DL. I actually spent a few years reading neuroscience. It was painful, and I certainly can't tell I learned how the brain works, but I'm pretty certain it has nothing to do with DL.
A lot of people said the same thing in the '90's about "Neural Networks". Senior colleagues of mine have vivid memories of consultants coming in then saying much the same thing "in 10 years this will completely revolutionize your industry (manufacturing)."
When I first start working in 2004 "data mining" was the big thing and it was going to solve all our problems. Nowadays I'm hearing the same thing again about "Machine Learning".
It's pretty natural to be skeptical people make big promises it ends up being a lot of hot air.
Yeah this is a dumb article. Number of tweets by AndrewNg? Really? All those articles denying the reality of the revolution brought by AI have an emotional basis, but I don't understand what it is. Are they feeling threatened? Or is it an undergrad/early 20s thing, like a complete lack of understanding of the dynamics coupled with abnormally strong opinions?
This is a deep, significant post (pardon pun etc).
The author is clearly informed and takes a strong, historical view of the situation. Looking at what the really smart people who brought us this innovation have said and done lately is a good start imo (just one datum of course, but there are others in this interesting survey).
Deepmind hasn't shown anything breathtaking since their Alpha Go zero.
Another thing to consider about Alpha Go and Alpha Go Zero is the vast, vast amount of computing firepower that this application mobilized. While it was often repeated that ordinary Go program weren't making progress, this wasn't true - the best, amateur programs had gotten to about 2 Dan amateur using Makov Tree Search. Alpha Go added CNNs for it's weighting function and petabytes of power for it's process and got effectiveness up to best in the world, 9 Dan professional, (maybe 11 Dan amateur for pure comparison). [1]
Alpha Go Zero was supposedly even more powerful, learned without human intervention. BUT it cost petabytes and petabytes of flops, expensive enough that they released a total of ten or twenty Alpha Go Zero game to the world, labeled "A great gift".
The author convenniently reproduces the chart of power versus results. Look at it, consider it. Consider the chart in the context of Moore's Law retreating. The problems of Alpha Zero generalizes as described in the article.
The author could also have dived into the troubling question as of "AI as ordinary computer application" (what does testing, debugging, interface design, etc mean when the app is automatically generated in an ad-hoc fashion) or "explainability". But when you can paint a troubling picture without these gnawing problems appearing, you've done well.
I'm sure the same could be said for early computer graphics before the GPU race. You don't need Moore's Law to make machine learning fast, you can also do it with hardware tailored to the task. Look at Google's TPUs for an example of this.
If you want an idea of where machine learning is in the scheme of things, the best thing to do is listen to the experts. _None_ of them have promised wild general intelligence any time soon. All of them have said "this is just the beginning, it's a long process." Science is incremental and machine learning is no different in that regard.
You'll continue to see incremental progress in the field, with occasional demonstrations and applications that make you go "wow". But most of the advances will be of interest to academics, not the general public. That in no way makes them less valuable.
The field of ML/AI produces useful technologies with many real applications. Funding for this basic science isn't going away. The media will eventually tire of the AI hype once the "wow" factor of these new technologies wears off. Maybe the goal posts will move again and suddenly all the current technology won't be called "AI" anymore, but it will still be funded and the science will still advance.
It's not the exciting prediction you were looking for I'm sure, but a boring realistic one.
> Funding for this basic science isn't going away.
What make this 3rd/4th boom in AI different?
The other AI winter, the funding for these science went from well funded to little funding.
I'm skeptical, with respect of course, on your statement because it doesn't have anything to back that up other than it produce useful technologies. Wouldn't this statement imply that the other previous AI which experience AI Winter (expert system, and whatever else) didn't produce useful enough technologies to have funding?
I'm currently on the camp of there is going to be an AI Winter III coming.
> None_ of them have promised wild general intelligence any time soon.
The post talk about Andrew Ng wild expectation on other things such as radiologist tweet. While it's not wild general intelligence. What I think the main article and also I am thinking is the outrageous speculation. Another one is the tesla self driving, it doesn't seem to be there yet and perhaps we're hitting the point of over promise like we did in the past and then AI winter happen because we've found the limit.
The previous AI winters were funded by speculative investments (both public research and industry) with the expectation that this might result in profitable technologies. And this didn't happen - yes, "the other previous AI which experience AI Winter (expert system, and whatever else) didn't produce useful enough technologies to have funding", the technologies developed didn't work sufficiently well to have widespread adoption in the industry; there were some use cases but the conclusion was "useful in theory but not in practice".
The current difference is that the technologies are actually useful right now. It's not about promised or expected technologies of tomorrow, but about what we have already researched, about known capabilities that need implementation, adoption, and lots of development work to apply it in lots and lots of particular use cases. If the core research hits a dead end tomorrow and stops producing any meaningful progress for the next 10 or 20 years, the obvious applications of neural-networks-as-we're-teaching-them-in-2018 work sufficiently well and are useful enough to deploy them in all kinds of industrial applications, and the demand is sufficient to employ every current ML practitioner and student even in absence of basic research funding, so a slump is not plausible.
I've recently had a number of calls from recruiters about new startups in the UK in the AI space, some of them local and some of them extensions of US companies. Some of them were clearly less speculative (tracking shipping and footfall for hedge funds) while others were certainly more speculative sounding. The increase of the latter gives me the impression that there is a bit of speculation going on at the moment.
A lot of this is because there is a somewhat mis-informed (which we will be polite and not call 'gullible') class of investors out there, primarily in the VC world, that thinks that most AI is magic pixie dust and so 'we will use AI/DL' and 'we will do it on the blockchain' has become the most recent version of 'we will do it on the web' in terms of helping get funding. Most of these ventures will flame out in 6-12 months and the consequences of this are going to be the source of the upcoming AI winter OP was talking about.
Strangely enough he didn’t speak at all about waymo self driving cars that are already hauling passengers without a safety driver. Given that he needs to hide the facts that go against his narrative I don’t really think that what he is convinced of will become reality.
In a very confined area. He mentions similar issues with Tesla's coast-to-coast autopilot ride: The software is not general enough yet to handle it. That seems to be the case for Waymo as well.
And how is this a failure of AI?
The most optimistic opinions on where we would see autonomous car were on the 2020s.
Instead we have autonomous car hauling people on the streets without any safety driver since 2017. And if everything goes accordingly their plan they will launch a commercial service by the end of the year in several US cities.
To me it seems a resounding success, not a failure.
> The most optimistic opinions on where we would see autonomous car were on the 2020s.
Sure, keep moving timelines. It's what makes you money in the area. I am sure when around mid-2019 hits, it will suddenly be "most experts agree that the first feasible self-driving cars will arrive circa 2025".
>Deepmind hasn't shown anything breathtaking since their Alpha Go zero
They went on to make AlphaZero, a generalised version that could learn chess, shogi or any similar game. The chess version beat a leading conventional chess program 28 wins, 0 losses, and 72 draws.
That seemed impressive to me.
Also they used loads of compute during the training but not so much during play.(5000 TPUs, 4TPUs).
Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.
Stockfish is not designed to scale to supercomputing clusters or TPUs, Alpha Zero wasn't designed to account for how long it takes to make a move, fair fight was hard to arrange.
There's discussion here https://chess.stackexchange.com/questions/19366/hardware-use...
AlphaZero's hardware was faster and Stockfish had a year old version with non optimum settings. It was still an impressive win but it would be interesting to do it again with a more level playing field.
It's not like humanity really needs another chess playing program 20 years after IBM solved that problem (but now utilizing 1000x more compute power). I just find all these game playing contraptions really uninteresting. There are plenty real world problems to be solved of much higher practicality. Moravec's paradox in full glow.
I think for most people, the research interest in games of various sorts, is not simply a desire for a better and better game contraption, a better mousetrap. But rather the thinking is, "playing games takes intelligence, what can we learn about intelligence by building machines that play games?"
Most games are also closed systems, and conveniently grokkable systems, with enumerable search spaces. Which gives us easily produceable measures of the contraptions' abilities.
Whether this is the most effective path to understanding deeper questions about intelligence is an open question.
But I don't think it's fair to say that deeper questions and problems are being foregone simply to play games.
I think most 'games researchers' are pursuing these paths because they themselves and no one else has put forth any other suggestion that makes them think, "hmm, that's a really good idea, that seems like it might be viable and there is probably something interesting we could learn from it."
This is so true, I can't understand why people miss this. The games are just games. It's intelligence that is the goal.
And comparing Alpha Go Zero against those "other chess programs that existed for 30 years" is exactly missing the point also.
Those programs were not constructed with zero-knowledge. They were carefully crafted by human players to achieve the result. Are we also going to count in all the brain processing power and the time spent by those researchers to learn to play chess? Alpha Go Zero did not need any of that, besides the knowledge about the basic rules of the game. Who compare compute requirements for 2 programs that have fundamentally different goals and achievements? One is carefully crafted by human intervention. The other one learns a new game without prior knowledge...
It shows something about the game, but it's clear that humans don't learn in the way that alpha zero does, do i don't think that alpha zero illuminated any aspect of human intelligence.
I think that fundamentally the goal of research is not necessarily human-like intelligence, just any high-level general intelligence. It's just that the human brain (and the rest of the body) has been a great example of an intelligent entity which we could source of a lot inspiration from. Whether the final result will share a the technical and structural similarity (and how much) to a human, the future will tell.
In principle you are right. In practice we will see. My bet is that attempts that focused on the human model will bear more fruit in the medium term because we have huge capability for observation at scale now which is v. exciting. Obviously ethics permitting!
Not sure if I am reading you correctly but to me you basically are saying "we have no idea but we believe that one day it will make sense".
Sounds more like religion and less like science to me.
I guess we could argue until the end of the world that no intelligence will emerge from more and more clever ways of brute-forcing your way out of problems in a finite space with perfect information. But that's what I think.
But humans could learn in the same way that AlphaZero does. We have the same resources and the same capabilities, just running on million-year-old hardware. Humans might not be able to replicate the performance of AlphaZero, but that does not mean it is useless in the study of intelligence.
I totally agree with you and share your confusion.
On the topic of the different algorithmic approaches, I find it so fascinating how different these two approaches actually end up looking when analyzed by a professional commentator. When you watch the new style with a chess commentator, it feels a lot like listening to the analysis of a human game. The algorithm has very clearly captured strategic concepts in its neural network. Meanwhile, with older chess engines there is a tendency to get to positions where the computer clearly doesn't know what its doing. The game reaches a strategic point and the things its supposed to do are beyond the horizon of moves it can computer by brute force. So it plays stupid. These are the positions that, even now, human players can beat better than human old style chess engines at.
The thing is that you can learn new moves/strategies that were never thought about before in these games but you still doesn't understand anything about intelligence at all.
The problem is that outside perfect information games, most areas where intelligence is required have few obvious routes to allow the computer to learn by perfectly simulating strategies and potential outcomes. Cases where "intelligence" is required typically entail handling human approximations of a lot of unknown and barely known possibilities with an inadequate dataset, and advances in approaches to perfect information games which can be entirely simulated by a machine knowing the ruleset (and possibly actually perturbed by adding inputs of human approaches to the problem) might be at best orthogonal to that particular goal. One of the takeaways from AlphaGo Zero massively outperforming AlphaGo is that even very carefully designed training sets for a problem fairly well understood by humans might actually retard system performance...
I guess there are reasons why researchers build chess programs: it is easy to compare performance between algorithms. When you can solve chess, you can solve a whole class of decision-making problems. Consider it as the perfect lab.
When you can solve chess, you can solve a whole class of decision-making problems
If this were true, there would be a vast demand for grandmasters in commerce, government, the military... and there just isn’t. Poker players suffer from similar delusions about how their game can be generalised to other domains.
I think batmansmk doesn't mean "when X is good at chess, X is automatically good at lots of other things", but "the traits that make you a good chess player (given enough training) also make you good at lots of other things (given enough training)".
I might suspect (but certainly cannot prove) that the traits that make a human good at playing chess are very different to the traits that make a machine good at playing chess, and as such I don't think we can assume that the machine skilled-chess-player will be good at lots of other things in an analagous way to the human skilled-chess-player.
And Gaius point stands before this argument as well, chess is seen as such a weak predictor that playing a game of chess or requesting an official ELO rating isn't used for hiring screening for instance.
I suspect that chess as a metagame is just so far developed that being "good at chess" means your general ability is really overtrained for chess.
Second world chess champion Emanuel Lasker spent a couple years studying Go and by his own report was dejected by his progress. Maybe he would have eventually reached high levels, but I've always found this story fascinating.
True, but I'd phrase it the other way around. The traits that make you (a human) good at general problem solving are also the traits that make you a good chess player. I do suspect, though, that there are some Chess-specific traits which boost your Chess performance but don't help much with general intelligence. (Consider, for example, the fact that Bobby Fischer wasn't considered a genius outside of his chosen field.)
> Poker players suffer from similar delusions about how their game can be generalised to other domains.
Oh that's so true
Poker players in the real life would give up more often than not, whenever they didn't know enough about a situation or they didn't have enough resources for a win with a high probability.
Those traits seem to me like a thing most people desperately need ... Everyone being confident in their assessment of everything seems like one of major problems of today's population.
What is that class of decision-making problems? It's nice to have a machine really good at playing chess, but it's not something I'd pay for. What decision-making problems are there, in the same class, that I'd pay for?
Consider it as the perfect lab.
Seems like a lab so simplified that I'm unconvinced of its general applicability. Perfect knowledge of the situation and a very limited set of valid moves at any one time.
Poker bots actually deal with a (simple) game with imperfect information. It is not the best test because short memory is sufficient to win at it.
The real challenge is to devise a general algorithm that will learn to be a good poker player in thousands of games, strategically, from just a bunch of games played. DeepStack AI required 10 million simulated games. Good human players outperform it at intermediate training stages.
And then the other part is figuring out actual rules of a harder game...
Strongly disagree. There are a lot of approximation algorithms and heuristics in wide use - to the tune of trillions of dollars, in fact, when you consider transportation and logistics, things like asic place & route, etc. These are all intractable perfect info problems that are so widespread and commercially important that they amplify the effect of even modest improvements.
Indeed, there are a few problems where even with perfect information you will be hard pressed to solve them. But that is only a question of computational power or the issue when the algorithm does not allow efficient approximation (not in APX space or co-APX).
The thing is, an algorithm that can work with fewer samples and robustly tolerating mistakes in datasets (also known as imperfect information) will be vastly cheaper and easier to operate. Less tedious sample data collection and labelling.
Working with lacking and erroneous information (without known error value) is necessarily a crucial step towards AGI; as is extracting structure from such data.
This is the difference between an engineering problem and research problem.
Perhaps a unifying way of saying this is: it's a research problem to figure out how to get ML techniques to the point they outperform existing heuristics on "hard" problems. Doing so will result in engineering improvements to the specific systems that need approximate solutions to those problems.
I completely agree about the importance of imperfect information problems. In practice, many techniques handle some label noise, but not optimally. Even MNIST is much easier to solve if you remove the one incorrectly-labeled training example. (one! Which is barely noise. Though as a reassuring example from the classification domain, JFT is noisy and still results in better real world performance than just training on imagenet.)
> Perfect information problem solving is not interesting anymore.
I guess in the same way as lab chemistry isn't interesting anymore ? (Since it often happens in unrealistically clean equipment :-)
I think there is nothing preventing lab research from going on at the same time as industrialization of yesterday's results. Quite on the contrary: in the long run they often depend on each other.
I think chess may actually be the worst lab. Decisions made in chess are done so with perfect knowledge of the current state and future possibilities. Most decisions are made without perfect knowledge.
A human might not be able to, but a computer can. Isn't the explicit reason research shifted to using Go the fact that you can't just number crunch your way through it?
AlphaGo Zero did precisely that. Most of its computations were done on a huge array of GPUs. The problem with Go is that look-ahead is more of a problem than in Chess, as Go has roughly between five and ten times as many possible moves at each point in the game. So Go was more of a challenge, and master-level play was only made possible by advances in computer hardware.
This is not what the terminology "perfect knowledge" means. Perfect knowledge (more often called "perfect information") refers to games in which all parts of the game state are accessible to every other player. In theory, any player in the game has access to all information contained in every game state up to the present and can extrapolate possible forward states. Chess is a very good example of a game of perfect information, because the two players can readily observe the entire board and each other's moves.
A good example of a game of imperfect information is poker, because players have a private hand which is known only to them. Whereas all possible future states of a chess game can be narrowed down according to the current game state, the fundamental uncertainty of poker means there is a combinatorial explosion involved in predicting future states. There's also the element of chance in poker, which further muddies the waters.
Board games are often (but not always) games of perfect and complete information. Card games are typically games of imperfect and complete information. This latter term, "complete information", means that even if not all of the game state is public, the intrinsic rules and structure of the game are public. Both chess and poker are complete, because we know the rules, win conditions and incentives for all players.
This is all to say that games of perfect information are relatively easy for a computer to win, while games of imperfect information are harder. And of course, games of incomplete information can be much more difficult :)
Exactly. To my not-very-well-informed self, even AlphaGo Zero is just a more clever way to brute-force board games.
Side observers are taking joy in the risker plays that it did -- reminded them of certain grand-masters I suppose -- but that still doesn't mean AGZ is close to any form of intelligence at all. Those "riskier moves" are probably just a way to more quickly reduce the problem space anyway.
It seriously reminds me more and more of religion, the AI area these days.
Tell me about it. The brightest minds are working on ads, and we have AI playing social games.
Can AI make the world better? It can, but it won't since we are humans, and humans will weaponize technology every chance it gets. Of course some positive uses will come, but the negative ones will be incredibly destructive.
Just because you haven't seen humongous publicity stunts involving pratical uses of AI doesn't mean they aren't being deployed. My company using similar methods to warn hospitals about patients with high probability of imminent heart attacks and sepsis.
The practical uses of these technologies don't always make national news.
I'm sure you would also have scoffed at the "pointless impractical, wasteful use of our brightest minds" to make the the Flyer hang in the air for 30 yards at Kitty Hawk.
It's not like the research on games is at the expense of other more worthy goals. It is a well constrained problem that lets you understand the limitations of your method. Great for making progress. Alpha zero didn't just play chess well, it learned how to play chess well (and could generalize to other games). I'd forgive it 10000 times the resources for that.
I'd say getting better sample efficiency is a bigger deal. It isn't like POMDP's are a huge step away theoretically from MDP's. But if you attach one of these things to a robot, taking 10^7 samples to learn a policy is a deal breaker. So fine, please keep using games to research with.
This. Learning to play a game is one thing. Learning how to teach computers to learn a game is another thing. Yes chess programs have been good before, but that's missing the point a little bit. The novel bit is not that it can beat another computer, but how it learned how to do so.
It's Deep Blue, not Big Blue. The parameters used by its evaluation function were tuned by the system on games played by human masters.
But it's a mistake to think that a system learning by playing against itself is something new. Arthur Samuel's draughts (chequers) program did that in 1959.
It's not that it's new, it's that they've achieved it. Chess was orders of magnitude harder than draughts. The solution for draughts didn't scale to chess but Alpha Go zero showed that chess was ridiculously easy for it once it had learned Go.
Both Samuel's chequer's program and Deep Blue used alpha-beta pruning for search, and a heuristic function. Deep Blue's heuristic function was necessarily more complex because chess is more complex than draughts. I think the reason master chess games were used in Deep Blue instead of self-play was the existence of a large database of such games, and because so much of its performance was the result of being able to look ahead so far.
The fact that it beat Stockfish9 is not what is impressive with AlphaZero.
What was impressive was the way Stockfish9 was beaten. AlphaZero played like a human player, making sacrifices for position that stockfish thought were detrimental. When it played as white, the fact that is mostly started with the Queen pawn (despite that the King pawn is "best by test") and the way AlphaZero used Stockfish pawnstructure and tempo to basicaly remove a bishop from the game was magical.
Yes, since its a game, it's "useless", but it allowed me (and i'm not the only one) to be a bit better at chess. It's not world hunger, not climate change, it's just a bit of distraction for some people.
PS: I was part of the people thinking that Genetic algorithm+deep learning was not enough to emulate human logical capacities, AlphaZero vs Stockfish games made me admit i was wrong (even if i still think it only works inside well-defined environments)
Playing like a human for me also means making human mistakes. A chess-playing computer playing like a 4000 rated "human" is useless, one that can be configured to play at different ELOs is more interesting, although most can do that and there's no ML needed, nor huge amounts of computing power.
> What was impressive was the way Stockfish9 was beaten.
Without its opening database and without its endgame tablebase?
Frankly, the Stockfish vs AlphaZero match was the beginning of the AI Winter in my mind. The fact that they disabled Stockfish's primary databases was incredibly fishy IMO and is a major detriment to their paper.
Stockfish's engine is designed to only work in the midgame of Chess. Remove the opening database and remove the endgame database, and you're not really playing against Stockfish anymore.
The fact that Stockfish's opening was severely gimped is not a surprise to anybody in the Chess community. Stockfish didn't have its opening database enabled... for some reason.
>Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.
Most humans don't live 2000 years. And realistically don't spend that much of their time or computing power on studying chess. Surely a computer can be more focused at this and the 4h are impressive. But this comparison seems flawed to me.
You're right, though the distinction with the parent poster is that AlphaGo Zero had no input knowledge to learn from, unlike humans (who read books, listen to other players' wisdom, etc). It's a fairly well known phenomenon that e.g. current era chess players are far stronger than previous eras' players, and this probably has to do with the accumulation of knowledge over decades, or even hundreds of years. It's incredibly impressive for software to replicate that knowledge base so quickly.
But software is starting from the same base. To claim it isn't would be to claim that the computers programmed themselves completely (which is simply not true).
Sure, there is some base there, and a fair bit of programming existed in the structure of the implementation. However, the heuristics themselves were not, and this is very significant. The software managed to reproduce and beat the previous best (both human and the previous iteration of itself), completely by playing against itself.
So, in this sense, it's kind of like taking a human, teaching them the exact rules of the game and showing them how to run calculations, and then telling them to sit in a room playing games against themselves. In my experience from chess, you'd be at a huge disadvantage if you started with this zero-knowledge handicap.
> In my experience from chess, you'd be at a huge disadvantage if you started with this zero-knowledge handicap.
One problem is that we can't play millions of games against ourselves in a few hours. We can play a few games, grow tired, and then need to go do something else. Come back the next day, repeat. It's a very slow process, and we have to worry about other things in life. How much of one's time and focus can be used on learning a game? You could spend 12 hours a day, if you had no other responsibilities, I guess. That might be counter productive, though. We just don't have the same capacity.
If you artificially limited AlphaGo to human capacity, then my money would be on the human being a superior player.
All software starts with a base of 4 billion years of evolution and thousands years of social progress and so on. But Alpha Zero doesn't require a knowledge of Go on top of that.
Not so much from the accumulation of knowledge because players can only study so many games. The difference is largely because their are more people today, they have more free time, and they could play vs high level opponents sooner.
Remember people reach peak play in ~15 years, but they don't nessisarily keep up with advances.
PS: You see this across a huge range of fields from running, figure skating, to music people simply spend more time and resources getting better.
> Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.
Few would care. Your examiner doesn't give you extra marks on a given problem for finishing your homework quickly.
Just because alpha zero doesn't solve the problem you want it to doesn't mean that advancements aren't being made that matter to someone else. To ignore that seems disingenuous.
The brain most likely has much more than a petaflop of computing power and it takes at least a decade to train a human brain to achieve the grandmaster level on an advanced board game. In addition, as the other comment says, they learn from hundreds or thousands of years of knowledge that other humans have accumulated and still lose to AlphaZero with mere hours of training.
Current AIs have limitations but, at the tasks they are suited for, they can equal or exceed humans with years of experience. Computing power is not the key limit since it will be made cheaper over time. More importantly, new advances are still being made regularly by DeepMind, OpenAI, and other teams.
Sure, but have you heard about Moravec's paradox? And if so, don't you find it curious that over the 30 years of Moore's law exponential progress in computing almost nothing improved on that side of things, and we kept playing fancier games?
What do you think of recent papers and demos by teams from Google Brain, OpenAI, and Pieter Abbeel's group on using simulations to help train physical robots? Recent advances are quite an improvement over those from the past.
I'm skeptical, and side with Rodney Brooks on this one. First, reinforcement learning is incredibly inefficient. And sure, humans and animals have forms of reinforcement learning, but my hunch it that it works on an already incredibly semantically relevant representation and utilize the forward model. That model is generated by unsupervised learning (which is way more data efficient). Actually I side with Yann Lecun on this one, see some of his recent talks. But Yann is not a robotics guy, so I don't think he fully appreciates the role of a forward model.
Now using models for RL is the obvious choice, since trying to teach a robot a basic behavior with RL is just absurdly impractical. But the problem here, is that when somebody build that model (a 3d simulations) they put in a bunch of stuff they think is relevant to represent the reality. And that is the same trap as labeling a dataset. We only put in the stuff which is symbolically relevant to us, omitting a bunch of low level things we never even perceive.
This is a longer subject, and a HN is not enough to cover it, but there is also something about the complexity. Reality is not just more complicated than simulation, it is complex with all the consequences of that. Every attempt to put a human filtered input between AI and the world will inherently loose that complexity and ultimately the AI will not be able to immunize itself to it.
This is not an easy subject and if you read my entire blog you may get the gist of it, but I have not yet succeeded in verbalizing it concisely to my satisfaction.
Moravec's paradox is the discovery by artificial intelligence and robotics researchers that, contrary to traditional assumptions, high-level reasoning requires very little computation, but low-level sensorimotor skills require enormous computational resources.
I was thinking just that when reading the paragraphs about the uber accident.
There's absolutely nothing indicating that future progress is not possible, precisely because of how absurd it seems right now.
> BUT it cost petabytes and petabytes of flops, expensive enough that they released a total of ten or twenty Alpha Go Zero game to the world
Training is expensive but inference is cheap enough for Alpha Zero inspired bots to beat human professionals while running on consumer hardware. DeepMind could have released thousands of pro-level games if they wanted to and others have: http://zero.sjeng.org/
Moore's law (or at least, the diminishing one) is not relevant here because these are not single threaded programs. Google put 8x on their TPUv2 -> v3 upgrade; parallel matrix multiplies at reduced precision are a long way away from any theoretical limits, as I understand it.
The first generation TPUs used 65536 very simple cores.
In the end you have so many transistors you can fit and there are options on how to arrange and use.
You might support very complex instructions and data types and then four cores. Or you might only support 8 bit ints, very, very simple instructions and use 65536 cores.
In the end what matters is the joules to get something done.
We can clearly see that we have big improvements by using new processor architectures.
Retrospectively it might sound that the Japanese were partially right in pursuing "high performance" computing with their fifth generation projects [1] but the Alpha Zero results are impressive beyond the computing performance achieved. It was a necessary element but not the only one.
I am 100% in agreement with the author on the thesis: deep learning is overhyped and people project too much.
But the content of the post is in itself not enough to advocate for this position. It is guilty of the same sins: projection and following social noises.
The point about increasing compute power however, I found rather strong. New advances came at a high compute cost. Although it could be said that research often advances like that: new methods are found and then made efficient and (more) economical.
A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.
> A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.
Who's to say we won't improve this though? Right now, nets add a bunch of numbers and apply arbitrarily-picked limiting functions and arbitrarily-picked structures. Is it impossible that we find a way to train that is orders of magnitude more effective?
To me, it's a bit like the question "Who's to say we wont find a way to travel faster than the speed of light?", by which I mean that in theory, many things are possible, but in practice, you need evidence to consider things likely.
Currently, people are projecting and saying that we are going to see huge AI advances soon. On which basis are these claims made? Showing fundamental limitations of deep learning is showing we have no idea how to get there. How to get there yet, indeed, just we have no idea how to do time travel yet.
Overhyped? There are cars driving around Arizona without safety drivers as I type this.
The end result of this advancement to our world is earth shattering.
On the high compute cost. There is an aspect of that being true but we have also seen advancement in silicon to support. We look at WaveNet using 16k cycles through a DNN and offering at scale and competitive price kind of proves the point.
> A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.
I'm not even sure how you'd go about doing that. You could use information theory to debunk some of the more ludicrous claims, especially ones that involve creating "missing" information.
One of the things that disappoints me somewhat with the field, which I've arguably only scratched the surface of, is just how much of it is driven by headline results which fail to develop understanding. A lot of the theory seems to be retrofitted to explain the relatively narrow result improvement and seems only to develop the art of technical bullshitting.
There are obvious exceptions to this and they tend to be the papers that do advance the field. With a relatively shallow resnet it's possible to achieve 99.7% on MNIST and 93% on CIFAR10 on a last-gen mid-range GPU with almost no understanding of what is actually happening.
There's also low-hanging fruit that seems to have been left on the tree. Take OpenAI's paper on parametrization of weights, so that you have a normalized direction vector and a scalar. This makes intuitive sense for anybody familiar with high-dimensional spaces since nearly all of the volume of a hypersphere lies around the surface. That this works in practice is great news, but leaves many questions unanswered.
I'm not even sure how many practitioners are thinking in high dimensional spaces or aware of their properties. It feels like we get to the universal approximation theorem and just accept that as evidence that they'll work well anywhere and then just follow whatever the currently recognised state of the art model is and adapt that to our purposes.
We very well might be in a deep-learning 'bubble' and the end of a cycle... but I don't think this time around it's really the end for a long-while, but more likely a pivot point.
The biggest minds everywhere are working on AI solutions, and there's also a lot in medical/science going on to map brains and if we can merge neuroscience with computer science we might have more luck with AI in the future...
So we could have a draught for a year or two, but there will be more research, and more breakthroughs. This won't be like the AI winters of the past where it lay dormant for 10+ years, I don't think.
No, but wait! We're just on the verge of replacing doctors! ;-)
There's still a lot of space for the improvement of "curve-fitting" AI in the workplace. The potential of existing tech is far from being thoroughly exploited right now.
I believe the next big improvements will come more from better integration in the workplace (or road system) than new scientific advances, so that might seem less sexy. But I also believe this will be a sufficient impetus to drive the field forward for the years to come.
I've always understood the claim that deep learning scales to be a claim about deployment and use of trained models, not about training. The whole point is that you can invest (substantial) resources upfront to train a sufficiently good model, but then the results of that initial investment can be used with very small marginal costs.
OP's argument on this front seems disingenous to me.
His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly strange omission. Uber's practices and culture have historically been so toxic that their failures here are truly irrelevant, and Tesla isn't even in the business of making actual self driving cars.
I'm the first to argue that right now AI is overhyped, but this is just sensationalist garbage from the other end of the spectrum.
Hi, it appears that "sensationalist garbage" triggered quite a bit of a discussion. This is typically indicative that the topic is "sensitive". Perhaps because many people feel the winter coming as well. Maybe, maybe not, time will tell.
And FYI, Tesla is in the business of making self driving car. If you read the article, you might learn that Tesla is actually the first company to sell that option to customers. You can go to their website right now and check that out.
Uber, like it or not is one of the big players of this game. I agree they may have somewhat toxic culture, but I guarantee you there are plenty of really smart people there who know exactly the state of the art. And their failure is therefore indicative of that state of the art.
I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes. But I analyze the California DMV disengagement reports in another post if you care to look. And by no means any of these cars is safe for deployment yet.
> Hi, it appears that "sensationalist garbage" triggered quite a bit of a discussion.
Yes. Sensationalist.
> I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes.
So your explicit reason for omitting Waymo, as I understand it, is that it didn't support your argument?
Yes, perhaps. But I'm entitled to my opinion just as you are entitled to yours. And time will tell who was right.
> So your explicit reason for omitting Waymo, as I understand it, is that it didn't support your argument?
You see, when you make any argument, you always omit the infinite number of things that don't support it and focus on the few things that do. The fact that something does not support my argument, does not mean it contradicts it.
You might also note that this is not a scientific paper, but an opinion. Yes, nothing more than an opinion. May I be wrong? Sure. And yet this opinion appears to shared by quite a few people, and makes a bunch of other people feel insecure. Perhaps there is something to it? We will see.
But in the worst case it will make some people think a bit and make an argument either for or against it. I may learn today a good argument against it, that will make me think about it more and perhaps I will change my opinion, or I'll be able to defend it.
So far you have not provided such an argument, but I wholeheartedly encourage you to do so.
> You see, when you make any argument, you always omit the infinite number of things that don't support it and focus on the few things that do.
No. When I make an argument, I try to omit the infinite number of things I think are unlikely to be important, and focus on the few things that I think are most important whether they support my position or not.
Everyone's fallible, and I do my share of focusing too much on points that support my position over more important counter points, but I see that as a failing, not as the reasonable thing to do.
This is a list of your phrases in this comment that I find, in my opinion, condescending.
> And time will tell who was right.
> You see, when you make any argument
> You might also note that this is not a scientific paper, but an opinion. Yes, nothing more than an opinion.
> And yet this opinion appears to shared by quite a few people, and makes a bunch of other people feel insecure. Perhaps there is something to it? We will see.
> So far you have not provided such an argument
I immediately identified this same tone in your paper. In your argumentation, you quite agressively hinted hat people which don't share your views are not very intelligent. You also have a tendency to present your sayings as prophetic, which appeared multiple times both in the paper and in this comment.
These observations put me in alarm towards your arguments, which I found mostly weak, sometimes used in bad faith. I flagged as such the Twitter argument, analysing the frequency of A. Ng's tweets, and denouncing its "outrageous claims", with an example where the AI score is overall only 0.025 less accurate than a practician.
I also thought that you used a different (your own) definition of scaling than most, and used it to make an argument, which was therefore unconvincing (but parent said that already).
Overall, to me, this was not a very pleasant read, and I dislike the fact that you attack the hype on machine learning by enjoying the polarization that comes with anti-hype articles such as yours. I also don't think that making people feel insecure is such a great indicator that what you're saying is relevant or prophetic.
> I've always understood the claim that deep learning scales to be a claim about deployment and use of trained models, not about training. The whole point is that you can invest (substantial) resources upfront to train a sufficiently good model, but then the results of that initial investment can be used with very small marginal costs.OP's argument on this front seems disingenous to me.
Remember that this is about deep "learning". If anything, the first thing that comes to mind is that it's the training (learning) that scales well.
> His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly strange omission.
Omission? That means something that has been left out, so how could it be the focus?
517 comments
[ 3.7 ms ] story [ 285 ms ] threadI've long been interested in learning about AI and deep learning, but to this day haven't done much that truly excites me within the field. It feels more or less impossible to make anything significant without Google-scale databases and Google-scale computers. AI really does make it easier for the few to jump far ahead, leaving everyone behind.
I also agree that a lot the news around AI is just hype.
Honestly, I'm yet to see anything practical come out of AI.
But hey, if something eventually does, I'm all for it.
You won’t hear about these projects outside industry specific publications, if at all.
https://www.theatlantic.com/technology/archive/2018/05/machi...
Things like this reinforcement learner for theorem proving are pretty exciting possibilities. https://arxiv.org/pdf/1805.07563v1.pdf
It's a buzzword people brainlessly use to fetishize technological progress without understanding the inherent limitations of the technology or the actual practicality and real-life results outside of crafted demos or specific problem domains (for example Alpha Go beating a grandmaster has almost no bearing on a problem like speech cognition).
It's turned me off a lot from reading about advances in the field because I know like a lot of science releases that most of it is empty air that won't really have bearing on the actual software I use (I've watched the past two Google's I/O where pretty much every presentation mentions AI, but the Android experience still remains relatively stale).
“The new spring in artificial intelligence is the most significant development in computing in my lifetime.”
He listed many examples below the quote.
“understand images in Google Photos;
enable Waymo cars to recognize and distinguish objects safely;
significantly improve sound and camera quality in our hardware;
understand and produce speech for Google Home;
translate over 100 languages in Google Translate;
caption over a billion videos in 10 languages on YouTube;
improve the efficiency of our data centers;
help doctors diagnose diseases, such as diabetic retinopathy;
discover new planetary systems; ...”
https://abc.xyz/investor/founders-letters/2017/index.html
An example from another continent:
“To build the database, the hospital said it spent nearly two years to study more than 100,000 of its digital medical records spanning 12 years. The hospital also trained the AI tool using data from over 300 million medical records (link in Chinese) dating back to the 1990s from other hospitals in China. The tool has an accuracy rate of over 90% for diagnoses for more than 200 diseases, it said.“
https://qz.com/1244410/faced-with-a-doctor-shortage-a-chines...
Well first off: letters to investors are among the most biased pieces of writing in existence.
Second: I'm not saying connectionism did not succeed in many areas! I'm a connectionist by heart! I love connectionism! But that being said there is disconnect between the expectations and reality. And it is huge. And it is particularly visible in autonomous driving. And it is not limited to media or CEO's, but it made its way into top researchers. And that is a dangerous sign, which historically preceded a winter event...
The difference between the current AI renaissance and the past pre-winter AI ecosystems is the level of economic gain realized by the technology.
The late 80s-early 90s AI winter, for example, resulted from the limitations of expert systems which were useful but only in niche markets and their development and maintenance costs were quite high relative to alternatives.
The current AI systems do something that alternatives, like Mechanical Turks, can only accomplish with much greater costs and may not even have the scale necessary for global massive services like Google Photos or Youtube autocaptioning.
The spread of computing infrastructure and connectivity into the hands of billions of global population is a key contributing factor.
I would argue this is well discounted by level of investment made against the future. I don't think the winter depends on the amount that somebody makes today on AI, rather on how much people are expecting to make in the future. If these don't match, there will be a winter. My take is that there is a huge bet against the future. And if DL ends up bringing just as much profit as it does today, interest will die very, very quickly.
This is analogous to the way electricity took decades to realize productivity gains in the broad economy.
That said, the hype will dial down. I am just not sure the investment will decrease soon.
So I guess we're waiting for something similar to happen with AI and then get AI 2.0?
The current road infrastructure (markings, signs) has been designed for humans. Once it has been modernized to better aid the self-driving systems, we don't probably need "perfect" AI.
Those are actual features that are available today to anyone, that were made possible by AI. Do you think it would be possible to type "pictures of me at the beach with my dog" without AI in such as short time frame? Or to have cars that drive themselves without a driver? These are concrete benefits of machine learning, I don't understand how that's biased.
If there are 100 facts that indicate a coming AI winter, and Brin just talks up the 15 facts that indicate AI's unalloyed dominance, that's definitely biased.
Second, I'm not quite sure that's how it works. Like in mathematics, if your lemma is X, you can give a 100 examples of X being true, but I only need a single counter-example to break it.
In my opinion a single valid modern use-case of AI is enough to show that we're not in an AI winter. By definition an AI winter means that nothing substantial is coming out of AI for a long period of time, yet Brin listed that Google alone has had a dozen in the past few years.
You cannot ask a generic question, then attack the answer based on absence of evidence for a specific example.
>translate over 100 languages in Google Translate;
>caption over a billion videos in 10 languages on YouTube;
barely even work. Yeah, it's a difficult problem but it's not even close to being solved.
Every time I activate it I am in for a good laugh more than anything actually useful.
Google's captioning works well when people speak clearly and in English. Google translate works well when you translate well written straightforward text into English. It's impressive but it's got a long way to go to reach human grade transcription and translation.
I think when evaluating these things people underestimate how long the tail of these problems is. It's always those pesky diminishing returns. I think it's true for many AI problems today, for instance it looks like current self-driving car tech manages to handle, say, 95% of situations just fine. Thing is, in order to be actually usable you want something that critical to reach something like 99.999% success rate and bridging these last few percent might prove very difficult, maybe even impossible with current tech.
It's unable to 'understand' that 'I poor gate' makes no sense at all.
Google Translate is the 'poor gate'.
You may think that you can now read German news, but in fact you would not know if the sentence meaning has been preserved in the English translation. The words itself might look as if the sentence makes sense - but the meaning is actually shifted - slight differences, but also possibly the complete opposite.
The translation also does not give you any indication where this might be and where the translation is based on weak training material or where there is some inference needed for a successful translation.
The disconnect between the expectations and reality. Well, to some degree you will always have it, but that is a very interesting facet, and I concur that it is somewhat increasing. It has in the recent past, and it still is..
> And it is not limited to media or CEO's, but it made its way into top researchers.
Interesting. One would assume that top researchers would be more "immune" than anyone else, but if it's true what you say and it already made its way.. For the record, I think you are onto something here.
What would be your favorite pet theory that explains this phenomenon? Do you believe it has something to do with this culture of "fake it till you make it", as if we have forgotten the value of honesty?
Maybe true but they are words that are about things which are either true or not true. Has nothing to do where the words were shared. Saying they are on an investment letter so not relevant seems very short sighted.
But just looking at the last 12 months it is folly to say we are moving to a AI winter. Things are just flying.
Look at self driving cars without safety drivers or look at something like Google Duplex but there are so many other examples.
Using the list provided, one example
"caption over a billion videos in 10 languages on YouTube;" - This doesn't say how accurate the captions acutally are. In my experience youtube captioning even of english dialect isn't exactly great. For one example try turning on the captions on this https://www.youtube.com/watch?v=bQJrBSXSs6o
so it's true I'm sure to say they've captioned the videos AI based techniques, but that doesn't mean they're a perfected option.
Also (purely anecodtally) Google translate also isn't exactly perfect yet either...
AI is overhyped and overfunded at the moment, which is not unusual for a hot technology (synthetic biology; dotcoms). Those things go in cycles, but the down cycles are seldom all out winters. During the slowdowns best technologies still get funding (less lavish, but enough to work on) and one-hit wonders die, both of which is good in the long run. My friends working in biology are doing mostly fine even though there are no longer "this is the century of synthetic biology" posters at every airport and in every toilet.
Utterly useless. And I don't think it is improving.
At least in English, they are now good enough that I can read without listening to the audio and understand almost everything said. (There are still a few mistakes here and there but they often don’t matter.)
I tried to help a couple channels to subtitle and the starting point was just sooo far from the finished product. I would guess I left 10% intact of the auto-translation. Maybe it would have been 5% five years ago; when things are this bad 100% improvement is hard to notice.
It is super cool how easy it is to edit and improve the subtitles for any channel that allows it.
If by 'almost everything', you mean stuff that a non native English speaker could have understood anyway, then yes.
The vast majority of English learners are not able to caption most Youtube videos as well as the current AI can.
You underestimate the amount of time required to learn another language and the expertise of a native speaker. (Have you tried learning another language to the level you can watch TV in it?)
Almost all native speakers are basically grandmasters of their mother tongue. The training time for a 15-year-old native speaker could be approx. 10 hours * 365 days * 15 years = 54,750 hours, more than the time many professional painists spent on practice.
A weak speaker may use a cognate, idiom borrowed from their native tongue or a similar wrong word more often. The translation app produces completely illegible word salad instead.
As for the users, sure the translation may not be perfect, but I'm sure if you were deaf had no other way of watching a video, you would be just fine with the current quality of the transcription.
Edit: also, a blog post with more examples and a link to the related publication: https://ai.googleblog.com/2018/04/looking-to-listen-audio-vi...
The original google report was discussed here a few weeks ago:
https://news.ycombinator.com/item?id=16813766
This is notorious with current technology: you can demonstrate anything. A few years ago Tesla demonstrated a driverless car. And what? Nothing. Absolutely nothing.
I'm willing to believe stuff I can test myself at home. If it works there, it likely actually works (though possibly needs more testing). But demo booths and youtube - never.
The BICEP2 fiasco is a good example why.
For most things, that people dream of and do marketing about need another leap forward, which we haven’t seen yet (it’ll come for sure)
Also, while a lot of these can be seen as "improvements", in many cases, that improvement put it past the threshold of actually being usable or useful. Self-driving cars for example need to be at least a certain level before they can be deployed, and we would've never reached that without machine learning.
This is one of the areas I’m most enthusiastic about but … it’s still nowhere near the performance of untrained humans. Google has poured tons of resources into Photos and yet if I type “cat” into the search box I have to scroll past multiple pages of results to find the first picture which isn’t of my dog.
That raises an interesting question: Google has no way to report failures. Does anyone know why they aren’t collecting that training data?
I’ve assumed that the reason is the same as why none of the voice assistants has an error reporting UI or even acknowledgement of low confidence levels: the marketing image is “the future is now” and this would detract from it.
what is this 'understand'?
Instead of being stuck on the fact that deep learning and the current methods seem to have hit a limit I think I am actually excited about the fact that this opens the door for experimenting other approaches that may or may not build on top of what we call AI today.
This ability of DL to convert streams of raw noisy data into labeled objects seems like exactly what's needed to solve an intelligent agent's perceptual grounding problem, where an agent that's new to the world must bootstrap its perception systems, converting raw sensory input into meaningful objects with physical dynamics. Only then can the agent reason about objects and better understand them by physical interaction and exploration. This is one of the areas where symbolic AI failed hardest, but DL does best.
With some engineering, it's easy to imagine how active learning could use DL to ground robot senses - much like an infant human explores the world for the first year of life, adding new labels and understanding their dynamics as it goes.
I suspect the potential for DL's many uses will continue to grow and surprise us for at least another decade. If we've learned anything from the past decade of DL, it's that probabilistic AI is surprisingly capable.
I find this article somewhat condescending. I look at all the current development as stepping stones to progress, not an overnight success that does everything flawlessly. I imagine the future might be some combination of different solutions, and what the author proposes may or may not play a part in it.
Edit: additionally it could be a dead end because the hype tends to narrow the directions we explore with ML. If everyone is obsessing about DL, we could be infuriatingly ignoring other research directions right under our noses.
1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!)
2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated.
3. Partially true. But self-driving is a sophisticated area by itself, DL is part of it, it can't really put full claim on its potential future success or ultimate downfall.
4. Gary Marcus isn't an established figure in DL research.
AI winter will ultimately come. But it is because people will become more informed about DL's strengths and limits, thus becoming smarter to tell what is BS what is not. AGI is likely not going to happen just with DL, but that is no way meaning it is a winter. DL has revolutionized the paradigm of Machine Learning itself, the shift has now complete, it will stay for a very very long time, and the successor is likely to build upon it not subvert it completely as well.
1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere near that ease in scaling. 3) self driving is the poster child of current "AI-revolution". And it is where by far most money is allocated. So if that falls, rest of DL does not matter. 4) Not that this matters, does it?
OpenAI's graph shows new architectures being used with more parameters because people are innovating on architecture and scale at the same time. Arguing that old methods "failed to scale" is like arguing that processor development was a failure because Intel had to develop a 486 instead of making a 386 work with more transistors (or more something).
And what does CFD have to do with anything, except maybe an odd attempt to argue from authority? Can you formalize from CFD a notion of "scaling well" well that anyone else agrees is useful for measuring AI research?
I don't fully understand the graph, but it looks like his point is that Alpha Go Zero uses 1e5 times as many resources than AlexNet, but does not produce anywhere near 10,000 times better results. We saw that with CFDt 1e5 more cores resulted in 1e5 better results (= scales). The assertion is that DL's results are much less than 1e5 better, hence it does not scale.
Basically the argument is:
1. CFD produces N times better results given N times more resources [this is implied, requires a knowledge of CFD]. That is, f(ax) = a f(x). Or, f(ax) = 1 a * f(x).
2. Empirically, we see that DL has used 1e5 more resources, but is not producing 1e5 times better results. [No quantitative analysis of how much better the results are is given]
3. Since DL has f(a * x) = b * a * f(x), where b < 1, DL does not scale. [Presumably b << 1 but the article did not give any specific results]
This isn't a very rigorous argument and the article left out half the argument, but it is suggestive.
To generalize notions of scaling, you need to look at the economics of consumed resources and generated utility, and you haven't begun to make the argument that data acquisition and PhD student time hasn't created ROI, or that ROI on those activities hasn't grown over time.
Data acquisition and labeling is getting cheaper all the time for many applications. Plus, new architectures give ways to do transfer learning or encode domain bias that let you specialize a model with less new data. There is substantial progress and already good returns on these types of scalability which (unlike returns on more GPUs) influence ML economics.
Let me try once again: an algorithm is scalable if it can process bigger instances by adding more compute power.
E.g. I take a small perceptron and train it on pentium 100, and then take a perceptron with 10x parameters on Core I7 and get better output by some monotonic function of increase in instance size (it is typically a sub linear function but it is OK as long as it is not logarithmic).
DL does not have that property. It requires modifying the algorithm, modifying the task at hand and so on. And it is not that it requires some tiny tweaking. It requires quite a bit of tweaking. I mean if you need a scientific paper to make a bigger instance of your algorithm this algorithm is not scalable.
What many people here are talking about is whether an instance of the algorithm can be created (by a great human effort) in a very specific domain to saturate a given large compute resource. And yes, in that sense deep learning can show some success in very limited domains. Domains where there happens to be a boatload of data, particularly labeled data.
But you see there is a subtle difference here, similar in some sense to difference between Amdahl's law and Gustafson's law (though not literal).
The way many people (including investors) understand deep learning is that: you build a model A, show it a bunch of pictures and it understands something out of them. Then you buy 10x more GPU's, build model B that is 10x bigger, show it those same pictures and it understands 10x more from them. Look I, and many people here understand this is totally naive. But believe me, I talked to many people with big $ that have exactly that level of understanding.
However, your last paragraph about how investors view deep learning does not describe anyone in the community of academics, practitioners and investors that I know. People understand that the limiting inputs to improved performance are data, followed closely by PhD labor. Compute power is relevant mainly because it shortens the feedback loop on that PhD labor, making it more efficient.
Folks investing in AI believe the returns are worth it due to the potential to scale deployment, not (primarily) training. They may be wrong, but this is a straw man definition of scalability that doesn't contribute to that thesis.
Almost all reasearch domains live on a log curve; a little bit gets you a lot to start with, but eventually you exhaust the easy solutions and a lot of work gets you very little improvement.
You’re arguing we haven’t reached the plateau at the top yet, but you’ve offered no meaningful evidence that is the case.
There are real world indicators that we are reaching diminishing returns for investment in compute and research now.
The ‘winter’ becomes a thing when it becomes apparent to investors that their financial bets are based off nothing more concrete than opinions like yours, when they don’t work out.
Are we there yet? Not sure, myself, I think we can get some more wins from machine generated architectures... but I can’t see any indication that the ‘winter’ isn’t coming sooner or later.
Investment is massively outstripping returns right now... we’ll just have to see if that calms down gradually, or pops suddenly.
History does not have a good story to tell about responsible investors behaving in a reasonable manner and avoiding crashes.
- Improvement is hard to define consistently. Sometimes, improving classification accuracy by 0.5% means reducing error by 20%, and makes economic applications that have 100x the value or frequency of use.
- Resources used in training can be amortized over billions of times the same model is reused (much more cheaply). So even achieving an epsilon improvement in the expected utility of each inference can justify a massive increase in training cost.
- Some other notions of "better results" or "less expensive" include amount of training data required, social fairness of results, memory required or power used during inference, and so on. And there are major advances in current research on each of these better formalized axes!
That last bit is what is so frustrating in reading an article like this. The author is sweeping aside with vague arguments a great deal of work that has been written and justified to a much much higher standard of rigor (not just the VCs we all like to snark about). Readers should beware of trusting a summary like this without engaging directly with the source material.
But when I hear the keyword "major advances" I'm highly suspicious. I had seen already so many such "major advances" that never went beyond a circle of self citing clique.
[1] https://ai.googleblog.com/2015/07/how-google-translate-squee...
Here's a recent example:
Unsupervised Predictive Memory in a Goal-Directed Agent
https://news.ycombinator.com/item?id=17177442
https://arxiv.org/abs/1802.10542
https://arxiv.org/abs/1802.07740
https://www.nature.com/articles/s41586-018-0102-6
https://arxiv.org/abs/1804.09401
https://arxiv.org/abs/1805.06370
https://arxiv.org/abs/1802.03006
https://arxiv.org/abs/1804.08617
https://arxiv.org/abs/1802.01561
And there are many others besides these, not to mention all the significant research being done by everyone else who isn't at Deepmind. The authors idea that interest and development of these topics is dying down or that Deepmind is running out of meaningful research to do just seems uninformed.
3)It does matter. In fact most valuable startup around DL are CV based startups, they are mainly located in China though.
Very weak to appeal to authority. The only true argument I can find against DL/ML/AI atm is the continuing appeal to authority by PhDs who have zero engineering knowledge, zero business sense and zero understanding of risk assessment.
We log everything and are even starting to automate decisions. Statistics, machine learning, and econometrics are booming fields. To talk about two topics dear to my heart, we're getting way better at modeling uncertainty (bayesianism is cool now, and resampling-esque procedures aged really well with a few decades of cheaper compute) and we're better at not only talking about what causes what (causal inference), but what causes what when (heterogeneous treatment effect estimation, e.g. giving you aspirin right now does something different from giving me aspirin now). We're learning to learn those things super efficiently (contextual bandits and active learning). The current data science boom goes far far far far beyond deep learning, and most of the field is doing great. Maybe those bits will even get better faster if deep learning stops hogging the glory. More likely, we'll learn to combine these things in cool ways (as is happening now).
AI is a superset and Machine learning is a subset of AI and most funding is in deep learning. Once Deep Learning hit the limit I believe there will be an AI winter.
Maybe there will be hype around statistic (cross fingers) which will lead to Bayesian and such.
Things like the German tank problem or the problem of hardening airplanes during WW2 have that very AI'esque feel to it. Where you use data to build a model, then let that data from the model to change the model as it fits.
Also the whole thing about 'decision making' is either bayesian or frequency based models in nature. Most of these algorithms and math has long existed before the current boom.
Its just that the raw computing power and resources that you have today make it possible for you to deal with large amounts of data to stress test your models.
eh-hem
DIE, HERETIC!
eh-hem
Ok, with that out of my system, no, Bayesian methods are definitely not a subset of deep learning, in any way. Hierarchical Bayes could be labeled "deep Bayesian methods" if we're marketing jerks, but Bayesian methods mostly do not involve neural networks with >3 hidden layers. It's just a different paradigm of statistics.
He sees the latent layer in the hierarchical model as the hidden layer and the Bayesian just have a strict restrictions/assumptions to the network where as the deep learning is more dumb and less assuming. A few of my professor thinks that PGM, probability graphical model is a super set of deep learning/neural network.
This is where my thinking come from.
IIRC, a paper have shown that gradient descent seems to exhibit MCMCs (blog with paper link inside that led to this conclusion of mine: http://www.inference.vc/everything-that-works-works-because-...).
But I am not an expert in Neural Network nor know the topic well enough to say such a thing. Other than was deferring to opinions of some one that's better than myself. So I'll keep this in mind and hopefully one day have the time to do more research into this topic.
Thank you.
I'd contend for the general public, AI is a synonym for machines like: HAL; The Terminator; Star Trek's "Data"; the robots in the film "AI"; and so on.
We're nowhere remotely in the vicinity of that, and no-one even has any plausible ideas about how to start.
A random person outside of tech probably doesn't even know what deep learning is. They might have heard of it somewhere in passing.
More generally, machine learning is a broad area and there's no reason to believe that different applications of it will all succeed or all fail for similar reasons. It seems more likely there will be more winners along with many failed attempts.
Yes. I've been saying this for a while. Waymo's approach is about 80% geometry, 20% AI. Profile the terrain, and only drive where it's flat. The AI part is for trying to identify other road users and guess what they will do. When in doubt, assume worst case and stay far away from them.
I was amazed that anyone would try self-driving without profiling the road. Everybody in the DARPA Grand Challenge had to do that, including us, because it was off-road driving and you were not guaranteed a flat road. The Google/Waymo people understood this. Some of the others just tried dumping the raw sensor data into a deep learning system and getting out a steering wheel angle. Not good.
As for Tesla - Tesla isn't even trying to make proper self-driving cars. Tesla's goal has always been assisted driving. However you feel about that, it's really not relevant to the success or failure of self-driving cars.
OP can't possibly have been ignorant of the fact that Waymo is the clear leader here with a substantial head start, and a proven record (and an actual fleet of self driving cars now on the road), and yet he chose not to mention it. That really undermines his credibility for me - he seems clearly more interested in making his point than in accurately engaging with reality.
From the same source as the author cites, that's because their test runs are typically 5 miles and resuming manual control at the end of a test counts as a disengagement.
Didn't this just happen? Maybe my timescales are off, but I've been thinking about AI and Go since the late 90s, and plenty of real work was happening before then.
Outside a handful of specialists, I'd expect another 8-10 years before the current state of the art is generally understood, much less effectively applied elsewhere.
If you build the hype like say Andrew Ng it better be. Also if you consume more money per month than all the CS departments of a mid sized country, it better be.
For instance: even if Elon Musk doesn't colonize Mars but instead just builds the BFR, that would still be amazing; even if BFR is never build but falcon 9 becomes fully reusable that would be great; even if falcon 9 won't be fully reusable, the fact that it cut the launching cost to space is still pretty good.
Even if we don't achieve any great breakthroughs with AGI, the fact that we started to use transfer learning to diagnose human disseases is pretty amazing; the fact that a japanese guy used tensorflow on a raspbery pi to categorize real cucumbers by shape is amazing.
All of this stuff won't go away; people will not say "hey, let's just forget about this deep learning thing and put it in some dusty shelf, it's useless for now". Maybe it will take 20 or 50 more years, maybe it's a slow thaw, but how could this be a winter?
I’m happy to leave the hard problems for the PhDs and the big tech researchers. Go nuts, folks.
In the meantime, the applications for small-scale, pre-trained neural networks seem limitless. Manufacturing, agriculture, retail, pretty much any industry could make use of portable neural networks.
I get that a lot of services we use on a daily basis make use of deep learning to accomplish tasks. But I don't really see what has fundamentally changed over the past 5 years in the way I use services. Siri was introduced 7 years ago and while we have clearly made progress in voice recognition, it's nowhere close to what many had hoped.
Anyway i also don't get what the issue is with the model from radiology. It is already that good?! This is impressive. One model is close to well trained experts.
Just today i had an small idea for a new product based on what google was showing with the capabilities to distinguis two people talking in parallel.
At the last Google IO i was impressed because in comparision to the previous years, ML created better and more impressive products.
I was listing for years at key nodes about big data and was never impressed. I hear now about ML and im getting impressed more and more.
In other words I figured it would be the annoyances at what "should be easy by now" that would get Joe CEO to start thinking "Hm. Maybe this isn't such a good investment." When measurements are made and reliable algorithmic results attract and keep more users than narrowly trained kind of finicky AIs.
I don't want there to be an AI winter, and it won't be as bad as before. There are a lot of applications for limited scope image recognition, and other tasks that we couldn't do before. Unfortunately,I do agree with the post that winter is on its way.
Time will tell. I think DL is amazing, but is no the right path towards solving problems such as autonomy. I think if you enter this field today, you should definitely take a look at other methods than DL. I actually spent a few years reading neuroscience. It was painful, and I certainly can't tell I learned how the brain works, but I'm pretty certain it has nothing to do with DL.
When I first start working in 2004 "data mining" was the big thing and it was going to solve all our problems. Nowadays I'm hearing the same thing again about "Machine Learning".
It's pretty natural to be skeptical people make big promises it ends up being a lot of hot air.
The author is clearly informed and takes a strong, historical view of the situation. Looking at what the really smart people who brought us this innovation have said and done lately is a good start imo (just one datum of course, but there are others in this interesting survey).
Deepmind hasn't shown anything breathtaking since their Alpha Go zero.
Another thing to consider about Alpha Go and Alpha Go Zero is the vast, vast amount of computing firepower that this application mobilized. While it was often repeated that ordinary Go program weren't making progress, this wasn't true - the best, amateur programs had gotten to about 2 Dan amateur using Makov Tree Search. Alpha Go added CNNs for it's weighting function and petabytes of power for it's process and got effectiveness up to best in the world, 9 Dan professional, (maybe 11 Dan amateur for pure comparison). [1]
Alpha Go Zero was supposedly even more powerful, learned without human intervention. BUT it cost petabytes and petabytes of flops, expensive enough that they released a total of ten or twenty Alpha Go Zero game to the world, labeled "A great gift".
The author convenniently reproduces the chart of power versus results. Look at it, consider it. Consider the chart in the context of Moore's Law retreating. The problems of Alpha Zero generalizes as described in the article.
The author could also have dived into the troubling question as of "AI as ordinary computer application" (what does testing, debugging, interface design, etc mean when the app is automatically generated in an ad-hoc fashion) or "explainability". But when you can paint a troubling picture without these gnawing problems appearing, you've done well.
[1] https://en.wikipedia.org/wiki/Go_ranks_and_ratings
If you want an idea of where machine learning is in the scheme of things, the best thing to do is listen to the experts. _None_ of them have promised wild general intelligence any time soon. All of them have said "this is just the beginning, it's a long process." Science is incremental and machine learning is no different in that regard.
You'll continue to see incremental progress in the field, with occasional demonstrations and applications that make you go "wow". But most of the advances will be of interest to academics, not the general public. That in no way makes them less valuable.
The field of ML/AI produces useful technologies with many real applications. Funding for this basic science isn't going away. The media will eventually tire of the AI hype once the "wow" factor of these new technologies wears off. Maybe the goal posts will move again and suddenly all the current technology won't be called "AI" anymore, but it will still be funded and the science will still advance.
It's not the exciting prediction you were looking for I'm sure, but a boring realistic one.
What make this 3rd/4th boom in AI different?
The other AI winter, the funding for these science went from well funded to little funding.
I'm skeptical, with respect of course, on your statement because it doesn't have anything to back that up other than it produce useful technologies. Wouldn't this statement imply that the other previous AI which experience AI Winter (expert system, and whatever else) didn't produce useful enough technologies to have funding?
I'm currently on the camp of there is going to be an AI Winter III coming.
> None_ of them have promised wild general intelligence any time soon.
The post talk about Andrew Ng wild expectation on other things such as radiologist tweet. While it's not wild general intelligence. What I think the main article and also I am thinking is the outrageous speculation. Another one is the tesla self driving, it doesn't seem to be there yet and perhaps we're hitting the point of over promise like we did in the past and then AI winter happen because we've found the limit.
The current difference is that the technologies are actually useful right now. It's not about promised or expected technologies of tomorrow, but about what we have already researched, about known capabilities that need implementation, adoption, and lots of development work to apply it in lots and lots of particular use cases. If the core research hits a dead end tomorrow and stops producing any meaningful progress for the next 10 or 20 years, the obvious applications of neural-networks-as-we're-teaching-them-in-2018 work sufficiently well and are useful enough to deploy them in all kinds of industrial applications, and the demand is sufficient to employ every current ML practitioner and student even in absence of basic research funding, so a slump is not plausible.
Sure, keep moving timelines. It's what makes you money in the area. I am sure when around mid-2019 hits, it will suddenly be "most experts agree that the first feasible self-driving cars will arrive circa 2025".
You guys are hilarious.
They went on to make AlphaZero, a generalised version that could learn chess, shogi or any similar game. The chess version beat a leading conventional chess program 28 wins, 0 losses, and 72 draws.
That seemed impressive to me.
Also they used loads of compute during the training but not so much during play.(5000 TPUs, 4TPUs).
Also it got better than humans in those games from scratch in about 4 hours whereas humans have had 2000 years to study them so you can forgive it some resource usage.
In a not equal fight, and the results are still not published. I'm not claiming that AlphaZero wouldn't win, but that test was pure garbage.
I agree AlphaZero had fancier hardware and so it wasn't really a fair fight.
Most games are also closed systems, and conveniently grokkable systems, with enumerable search spaces. Which gives us easily produceable measures of the contraptions' abilities.
Whether this is the most effective path to understanding deeper questions about intelligence is an open question.
But I don't think it's fair to say that deeper questions and problems are being foregone simply to play games.
I think most 'games researchers' are pursuing these paths because they themselves and no one else has put forth any other suggestion that makes them think, "hmm, that's a really good idea, that seems like it might be viable and there is probably something interesting we could learn from it."
Do you have any suggestions?
And comparing Alpha Go Zero against those "other chess programs that existed for 30 years" is exactly missing the point also. Those programs were not constructed with zero-knowledge. They were carefully crafted by human players to achieve the result. Are we also going to count in all the brain processing power and the time spent by those researchers to learn to play chess? Alpha Go Zero did not need any of that, besides the knowledge about the basic rules of the game. Who compare compute requirements for 2 programs that have fundamentally different goals and achievements? One is carefully crafted by human intervention. The other one learns a new game without prior knowledge...
Sounds more like religion and less like science to me.
I guess we could argue until the end of the world that no intelligence will emerge from more and more clever ways of brute-forcing your way out of problems in a finite space with perfect information. But that's what I think.
On the topic of the different algorithmic approaches, I find it so fascinating how different these two approaches actually end up looking when analyzed by a professional commentator. When you watch the new style with a chess commentator, it feels a lot like listening to the analysis of a human game. The algorithm has very clearly captured strategic concepts in its neural network. Meanwhile, with older chess engines there is a tendency to get to positions where the computer clearly doesn't know what its doing. The game reaches a strategic point and the things its supposed to do are beyond the horizon of moves it can computer by brute force. So it plays stupid. These are the positions that, even now, human players can beat better than human old style chess engines at.
https://people.csail.mit.edu/brooks/papers/elephants.pdf
If this were true, there would be a vast demand for grandmasters in commerce, government, the military... and there just isn’t. Poker players suffer from similar delusions about how their game can be generalised to other domains.
I suspect that chess as a metagame is just so far developed that being "good at chess" means your general ability is really overtrained for chess.
Oh that's so true
Poker players in the real life would give up more often than not, whenever they didn't know enough about a situation or they didn't have enough resources for a win with a high probability.
And people can call your bluff even if you fold.
Consider it as the perfect lab.
Seems like a lab so simplified that I'm unconvinced of its general applicability. Perfect knowledge of the situation and a very limited set of valid moves at any one time.
an awful lot of graph and optimization problems. See for instance some examples in https://en.wikipedia.org/wiki/A*_search_algorithm
Did they manage to extend it to games with hidden and imperfect information?
(Say, chess with fog of war also known as Dark Chess. Phantom Go. Pathfinding equivalent would be an incremental search.)
Edit: I see they are working on it, predictive state memory paper (MERLIN) is promising but not there yet.
The real challenge is to devise a general algorithm that will learn to be a good poker player in thousands of games, strategically, from just a bunch of games played. DeepStack AI required 10 million simulated games. Good human players outperform it at intermediate training stages.
And then the other part is figuring out actual rules of a harder game...
(You said problems, not games...)
The thing is, an algorithm that can work with fewer samples and robustly tolerating mistakes in datasets (also known as imperfect information) will be vastly cheaper and easier to operate. Less tedious sample data collection and labelling.
Working with lacking and erroneous information (without known error value) is necessarily a crucial step towards AGI; as is extracting structure from such data.
This is the difference between an engineering problem and research problem.
I completely agree about the importance of imperfect information problems. In practice, many techniques handle some label noise, but not optimally. Even MNIST is much easier to solve if you remove the one incorrectly-labeled training example. (one! Which is barely noise. Though as a reassuring example from the classification domain, JFT is noisy and still results in better real world performance than just training on imagenet.)
I guess in the same way as lab chemistry isn't interesting anymore ? (Since it often happens in unrealistically clean equipment :-)
I think there is nothing preventing lab research from going on at the same time as industrialization of yesterday's results. Quite on the contrary: in the long run they often depend on each other.
A good example of a game of imperfect information is poker, because players have a private hand which is known only to them. Whereas all possible future states of a chess game can be narrowed down according to the current game state, the fundamental uncertainty of poker means there is a combinatorial explosion involved in predicting future states. There's also the element of chance in poker, which further muddies the waters.
Board games are often (but not always) games of perfect and complete information. Card games are typically games of imperfect and complete information. This latter term, "complete information", means that even if not all of the game state is public, the intrinsic rules and structure of the game are public. Both chess and poker are complete, because we know the rules, win conditions and incentives for all players.
This is all to say that games of perfect information are relatively easy for a computer to win, while games of imperfect information are harder. And of course, games of incomplete information can be much more difficult :)
We solved nothing.
IBM Deep Blue doesn't exactly think like humans do.
Most of our algorithms really are 'better brute force'.
https://www.theatlantic.com/magazine/archive/2013/11/the-man...
Side observers are taking joy in the risker plays that it did -- reminded them of certain grand-masters I suppose -- but that still doesn't mean AGZ is close to any form of intelligence at all. Those "riskier moves" are probably just a way to more quickly reduce the problem space anyway.
It seriously reminds me more and more of religion, the AI area these days.
Can AI make the world better? It can, but it won't since we are humans, and humans will weaponize technology every chance it gets. Of course some positive uses will come, but the negative ones will be incredibly destructive.
The practical uses of these technologies don't always make national news.
I'm sure you would also have scoffed at the "pointless impractical, wasteful use of our brightest minds" to make the the Flyer hang in the air for 30 yards at Kitty Hawk.
But attacking not-well-constrained problems is what's needed to show real progress in AI these days, right?
This. Learning to play a game is one thing. Learning how to teach computers to learn a game is another thing. Yes chess programs have been good before, but that's missing the point a little bit. The novel bit is not that it can beat another computer, but how it learned how to do so.
That's a pretty major shift for humanity.
But it's a mistake to think that a system learning by playing against itself is something new. Arthur Samuel's draughts (chequers) program did that in 1959.
Big Blue is fine - it's referring to the company and not the machine. From Wikipedia "Big Blue is a nickname for IBM"
It's not that it's new, it's that they've achieved it. Chess was orders of magnitude harder than draughts. The solution for draughts didn't scale to chess but Alpha Go zero showed that chess was ridiculously easy for it once it had learned Go.
What was impressive was the way Stockfish9 was beaten. AlphaZero played like a human player, making sacrifices for position that stockfish thought were detrimental. When it played as white, the fact that is mostly started with the Queen pawn (despite that the King pawn is "best by test") and the way AlphaZero used Stockfish pawnstructure and tempo to basicaly remove a bishop from the game was magical.
Yes, since its a game, it's "useless", but it allowed me (and i'm not the only one) to be a bit better at chess. It's not world hunger, not climate change, it's just a bit of distraction for some people.
PS: I was part of the people thinking that Genetic algorithm+deep learning was not enough to emulate human logical capacities, AlphaZero vs Stockfish games made me admit i was wrong (even if i still think it only works inside well-defined environments)
Just because Fischer preferred 1. e4, it doesn't make it better than other openings. https://en.chessbase.com/post/1-e4-best-by-test-part-1
Playing like a human for me also means making human mistakes. A chess-playing computer playing like a 4000 rated "human" is useless, one that can be configured to play at different ELOs is more interesting, although most can do that and there's no ML needed, nor huge amounts of computing power.
Without its opening database and without its endgame tablebase?
Frankly, the Stockfish vs AlphaZero match was the beginning of the AI Winter in my mind. The fact that they disabled Stockfish's primary databases was incredibly fishy IMO and is a major detriment to their paper.
Stockfish's engine is designed to only work in the midgame of Chess. Remove the opening database and remove the endgame database, and you're not really playing against Stockfish anymore.
The fact that Stockfish's opening was severely gimped is not a surprise to anybody in the Chess community. Stockfish didn't have its opening database enabled... for some reason.
Most humans don't live 2000 years. And realistically don't spend that much of their time or computing power on studying chess. Surely a computer can be more focused at this and the 4h are impressive. But this comparison seems flawed to me.
So, in this sense, it's kind of like taking a human, teaching them the exact rules of the game and showing them how to run calculations, and then telling them to sit in a room playing games against themselves. In my experience from chess, you'd be at a huge disadvantage if you started with this zero-knowledge handicap.
One problem is that we can't play millions of games against ourselves in a few hours. We can play a few games, grow tired, and then need to go do something else. Come back the next day, repeat. It's a very slow process, and we have to worry about other things in life. How much of one's time and focus can be used on learning a game? You could spend 12 hours a day, if you had no other responsibilities, I guess. That might be counter productive, though. We just don't have the same capacity.
If you artificially limited AlphaGo to human capacity, then my money would be on the human being a superior player.
Remember people reach peak play in ~15 years, but they don't nessisarily keep up with advances.
PS: You see this across a huge range of fields from running, figure skating, to music people simply spend more time and resources getting better.
Few would care. Your examiner doesn't give you extra marks on a given problem for finishing your homework quickly.
https://deepmind.com/blog/deepmind-ai-reduces-google-data-ce...
Just because alpha zero doesn't solve the problem you want it to doesn't mean that advancements aren't being made that matter to someone else. To ignore that seems disingenuous.
Current AIs have limitations but, at the tasks they are suited for, they can equal or exceed humans with years of experience. Computing power is not the key limit since it will be made cheaper over time. More importantly, new advances are still being made regularly by DeepMind, OpenAI, and other teams.
https://www.quora.com/Roughly-what-processing-power-does-the...
Unsupervised Predictive Memory in a Goal-Directed Agent
https://arxiv.org/abs/1803.10760
What do you think of recent papers and demos by teams from Google Brain, OpenAI, and Pieter Abbeel's group on using simulations to help train physical robots? Recent advances are quite an improvement over those from the past.
Now using models for RL is the obvious choice, since trying to teach a robot a basic behavior with RL is just absurdly impractical. But the problem here, is that when somebody build that model (a 3d simulations) they put in a bunch of stuff they think is relevant to represent the reality. And that is the same trap as labeling a dataset. We only put in the stuff which is symbolically relevant to us, omitting a bunch of low level things we never even perceive.
This is a longer subject, and a HN is not enough to cover it, but there is also something about the complexity. Reality is not just more complicated than simulation, it is complex with all the consequences of that. Every attempt to put a human filtered input between AI and the world will inherently loose that complexity and ultimately the AI will not be able to immunize itself to it.
This is not an easy subject and if you read my entire blog you may get the gist of it, but I have not yet succeeded in verbalizing it concisely to my satisfaction.
https://en.wikipedia.org/wiki/Moravec%27s_paradox
Moravec's paradox is the discovery by artificial intelligence and robotics researchers that, contrary to traditional assumptions, high-level reasoning requires very little computation, but low-level sensorimotor skills require enormous computational resources.
Training is expensive but inference is cheap enough for Alpha Zero inspired bots to beat human professionals while running on consumer hardware. DeepMind could have released thousands of pro-level games if they wanted to and others have: http://zero.sjeng.org/
You mean Monte Carlo Tree Search, which is not at all like Ma(r)kov chains. You're probably mixing it up with Markov decision processes though.
Before criticising something it's a good idea to have a solid understanding of it.
The first generation TPUs used 65536 very simple cores.
In the end you have so many transistors you can fit and there are options on how to arrange and use.
You might support very complex instructions and data types and then four cores. Or you might only support 8 bit ints, very, very simple instructions and use 65536 cores.
In the end what matters is the joules to get something done.
We can clearly see that we have big improvements by using new processor architectures.
[1] https://mobile.nytimes.com/1992/06/05/business/fifth-generat...
Why not petaflops of bytes then?
I am 100% in agreement with the author on the thesis: deep learning is overhyped and people project too much.
But the content of the post is in itself not enough to advocate for this position. It is guilty of the same sins: projection and following social noises.
The point about increasing compute power however, I found rather strong. New advances came at a high compute cost. Although it could be said that research often advances like that: new methods are found and then made efficient and (more) economical.
A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.
Who's to say we won't improve this though? Right now, nets add a bunch of numbers and apply arbitrarily-picked limiting functions and arbitrarily-picked structures. Is it impossible that we find a way to train that is orders of magnitude more effective?
Currently, people are projecting and saying that we are going to see huge AI advances soon. On which basis are these claims made? Showing fundamental limitations of deep learning is showing we have no idea how to get there. How to get there yet, indeed, just we have no idea how to do time travel yet.
The end result of this advancement to our world is earth shattering.
On the high compute cost. There is an aspect of that being true but we have also seen advancement in silicon to support. We look at WaveNet using 16k cycles through a DNN and offering at scale and competitive price kind of proves the point.
I'm not even sure how you'd go about doing that. You could use information theory to debunk some of the more ludicrous claims, especially ones that involve creating "missing" information.
One of the things that disappoints me somewhat with the field, which I've arguably only scratched the surface of, is just how much of it is driven by headline results which fail to develop understanding. A lot of the theory seems to be retrofitted to explain the relatively narrow result improvement and seems only to develop the art of technical bullshitting.
There are obvious exceptions to this and they tend to be the papers that do advance the field. With a relatively shallow resnet it's possible to achieve 99.7% on MNIST and 93% on CIFAR10 on a last-gen mid-range GPU with almost no understanding of what is actually happening.
There's also low-hanging fruit that seems to have been left on the tree. Take OpenAI's paper on parametrization of weights, so that you have a normalized direction vector and a scalar. This makes intuitive sense for anybody familiar with high-dimensional spaces since nearly all of the volume of a hypersphere lies around the surface. That this works in practice is great news, but leaves many questions unanswered.
I'm not even sure how many practitioners are thinking in high dimensional spaces or aware of their properties. It feels like we get to the universal approximation theorem and just accept that as evidence that they'll work well anywhere and then just follow whatever the currently recognised state of the art model is and adapt that to our purposes.
The biggest minds everywhere are working on AI solutions, and there's also a lot in medical/science going on to map brains and if we can merge neuroscience with computer science we might have more luck with AI in the future...
So we could have a draught for a year or two, but there will be more research, and more breakthroughs. This won't be like the AI winters of the past where it lay dormant for 10+ years, I don't think.
There's still a lot of space for the improvement of "curve-fitting" AI in the workplace. The potential of existing tech is far from being thoroughly exploited right now. I believe the next big improvements will come more from better integration in the workplace (or road system) than new scientific advances, so that might seem less sexy. But I also believe this will be a sufficient impetus to drive the field forward for the years to come.
OP's argument on this front seems disingenous to me.
His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly strange omission. Uber's practices and culture have historically been so toxic that their failures here are truly irrelevant, and Tesla isn't even in the business of making actual self driving cars.
I'm the first to argue that right now AI is overhyped, but this is just sensationalist garbage from the other end of the spectrum.
And FYI, Tesla is in the business of making self driving car. If you read the article, you might learn that Tesla is actually the first company to sell that option to customers. You can go to their website right now and check that out.
Uber, like it or not is one of the big players of this game. I agree they may have somewhat toxic culture, but I guarantee you there are plenty of really smart people there who know exactly the state of the art. And their failure is therefore indicative of that state of the art.
I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes. But I analyze the California DMV disengagement reports in another post if you care to look. And by no means any of these cars is safe for deployment yet.
Yes. Sensationalist.
> I also omitted Cruise automation and a bunch of other companies, perhaps because they have more responsible backup drivers that so far avoided fatal crashes.
So your explicit reason for omitting Waymo, as I understand it, is that it didn't support your argument?
Yes, perhaps. But I'm entitled to my opinion just as you are entitled to yours. And time will tell who was right.
> So your explicit reason for omitting Waymo, as I understand it, is that it didn't support your argument?
You see, when you make any argument, you always omit the infinite number of things that don't support it and focus on the few things that do. The fact that something does not support my argument, does not mean it contradicts it.
You might also note that this is not a scientific paper, but an opinion. Yes, nothing more than an opinion. May I be wrong? Sure. And yet this opinion appears to shared by quite a few people, and makes a bunch of other people feel insecure. Perhaps there is something to it? We will see.
But in the worst case it will make some people think a bit and make an argument either for or against it. I may learn today a good argument against it, that will make me think about it more and perhaps I will change my opinion, or I'll be able to defend it.
So far you have not provided such an argument, but I wholeheartedly encourage you to do so.
No. When I make an argument, I try to omit the infinite number of things I think are unlikely to be important, and focus on the few things that I think are most important whether they support my position or not.
Everyone's fallible, and I do my share of focusing too much on points that support my position over more important counter points, but I see that as a failing, not as the reasonable thing to do.
> And time will tell who was right.
> You see, when you make any argument
> You might also note that this is not a scientific paper, but an opinion. Yes, nothing more than an opinion.
> And yet this opinion appears to shared by quite a few people, and makes a bunch of other people feel insecure. Perhaps there is something to it? We will see.
> So far you have not provided such an argument
I immediately identified this same tone in your paper. In your argumentation, you quite agressively hinted hat people which don't share your views are not very intelligent. You also have a tendency to present your sayings as prophetic, which appeared multiple times both in the paper and in this comment.
These observations put me in alarm towards your arguments, which I found mostly weak, sometimes used in bad faith. I flagged as such the Twitter argument, analysing the frequency of A. Ng's tweets, and denouncing its "outrageous claims", with an example where the AI score is overall only 0.025 less accurate than a practician.
I also thought that you used a different (your own) definition of scaling than most, and used it to make an argument, which was therefore unconvincing (but parent said that already).
Overall, to me, this was not a very pleasant read, and I dislike the fact that you attack the hype on machine learning by enjoying the polarization that comes with anti-hype articles such as yours. I also don't think that making people feel insecure is such a great indicator that what you're saying is relevant or prophetic.
I hope this helps you prophecies https://www.physics.ohio-state.edu/~kagan/AS1138/Lectures/Go... ;)
Remember that this is about deep "learning". If anything, the first thing that comes to mind is that it's the training (learning) that scales well.
> His focus on Uber and Tesla (while not even mentioning Waymo) is also a truly strange omission.
Omission? That means something that has been left out, so how could it be the focus?
If they are able to combine the 2. A big if though the cost analysis will change for AI quite dramatically.