I'm off the main hubs, SF, London, NYC, but here, in my gamma level city people talk a bit about AI but I have yet to see applications that are not vaporware. Don't get me wrong, I love ML challenges, but the real world seems to have so many other little things to solve first, before we can finally realize our ai dreams.
Things like Alexa or Google Translate already very much exist, use modern AI techniques, and are not vaporware. That's why so many corporate research labs - basically all the FAAMG companies - do AI work.
Agreed. I think this is a well-intentioned effort at marketing and sharing knowledge but it seems a bit fluffy for the "AI" side of things. For ML, I think it's a pretty darn good foundation with the traditional hand-jammed/artisinally-crafted/exceedingly data-driven approaches.
Perhaps I may be mistaken, but this seems to be a very long road for a more shallow understanding of deep learning. I'd venture this was written by someone who has a more traditional machine learning background that wants new people to the industry to have that same foundation; however, I'd venture that that is a rather inefficient way to get to deep learning proficiency.
If I were to give a recommendation, it would be this -- pick a topic/project that interests you, follow the classic knowledge-bootstrapping process where you read through papers and (hopefully) have an expert or trained person walk you through the specifics, then get hands-on instantly. Especially with something like fast.ai that values empirical results over hard theory, something understandably popular in the field.
From there I'd recommend branching out, but I'd use a JIT approach. The field isn't necessarily super well-founded at the moment, and while machine learning fundamentals are useful, ultimately it's a waste of time compared to the long-tail benefit of getting immediate empirical results and feedback hands-on.
Just my 2c, YMMV, and anyone is totally welcome to disagree as they wish! :)
> I'd venture this was written by someone who has a more traditional machine learning background that wants new people to the industry to have that same foundation
This is the vibe I got as well. Which is fair enough (to each his/her own), but I thought I'd mention fast.ai which takes the opposite approach:
> Harvard professor David Perkins, who wrote Making Learning Whole (Jossey-Bass), has much to say about teaching. The basic idea is to teach the whole game. That means that if you're teaching baseball, you first take people to a baseball game or get them to play it. You don't teach them how to wind twine to make a baseball from scratch, the physics of a parabola, or the coefficient of friction of a ball on a bat.
You are right that the "Deep Learning" section is rather shallow up to now. We are currently working on expanding it to offer a more comprehensive view of the field and expect to release this update next week. Stay tuned! :)
I agree with your suggestion, and that is what I did to get started in machine learning as well. Solving some kind of problem and doing practical projects and playing around with models is probably the most efficient way to learn. I started with the Machine Learning course on Coursera by Andrew Ng. And while going through, I started using what I learned on a small project. I did document my journey in case it's useful or interesting for others: https://embracethered.com/blog/posts/2020/machine-learning-b...
"I'd venture this was written by someone who has a more traditional machine learning background that wants new people to the industry to have that same foundation"
Yeah I get the same feeling. I am hesitant to suggest it, but gives a little bit of a gate-keeping vibe. Like only once you've payed sufficient homage to the same education path that I took, shall you be granted permission to be a 'real data scientist'.
That said, I think there is value in something like this more for hiring managers than practitioners... which makes sense considering the intro comments.
As for our opinion (which is just that, an opinion) why we think that the statistical foundations and knowledge about more traditional algorithms is important, it's based on the business needs and our experience. While it might seem less necessary if the goal is to "learn deep learning", it is highly relevant if your task is to "solve this business problem".
Our perspective is the industrial one. And while there are certainly many complex business problems where deep learning is required, there are more cases where a traditional approach is sufficient and actually the better solution (e.g. due to the memory footprint, latency or other reasons). We routinely work on both kinds of problems on a day to day basis, but we would never go straight to deep learning approaches if simpler and faster traditional methods comprise a better solution in a given use case. So, our employees are expected to know both and to be able to judge when to apply which approach.
Yes, this makes sense. I'd suggest retitling the article to be something along the lines of "ML Expert Roadmap", and then continue to flesh out all avenues. As it stands, the roadmap has really nothing to do with AI at all, but your point about not just jumping to the shiny hammer certainly rings true and makes sense. I certainly think that's the right approach algorithmically, especially if you're looking to be a more generalist data shop.
In a world of senseless marketing hype, I think it's a good idea to take the high road on this one. Reputation alone, even if less-buzzy words like ML are used in favor of AI, really carries a long ways. Plus, we're nearing the disenfranchisement hump, and AI's going to start taking a negative connotation with many businesses, I believe. Just shoot straight and I firmly believe it'll carry you for a long ways, there.
This should be renamed to "Intro to AI Roadmap"
What's on this roadmap will get you in the door with some of the basics. In no way will it make you an expert.
The number of highly controversial opinions on this, to me, only shows that it makes sense to try to map a potential path. While the criticism in the comments here may be valid, it is mostly unconstructive and without any proper explanation or reasoning. It may not be perfect, but it's a great overview of what to look into.
Maybe a better way to attack this problem is to watch Harvard's Introduction to AI course (or similar one) and from there pick an area that piqued your interest. This is because AI is such a vast field that nobody can learn everything (on top of data processing etc.). I made some notes from the ^ course + listed relevant Python libraries to have a good place to start: https://stribny.name/blog/2020/10/artificial-intelligence-in...
Yeah that seems like a major flaw in this taxonomy. Like does the author really think there is significant enough difference between 'data' and 'big data' that it warrants an entirely separate track? That's silly.
With all due to respect to the author of the site, mastering all the materials in the machine learning or data scientist path will make you a solid 'applied' machine learning/statistical learning practitioner, but not an expert, and definitely not a research candidate.
We should be careful with how we guard our scale of semantic meaning - if somebody with an undergrad understanding of statistics (frequentist statistics only in this map) can be called an Expert in a statistical/mathematical field, what do we even call somebody with all the same applied software engineering & exploratory analysis, and business experience, but also a PhD in theoretical topics (math, stats, etc)? A 'super expert'? What do we call Francis Chollet, or LeCun, or anybody else? What's the differentiation between an expert from the roadmap and the team of individuals deploying GPT-3 into Google Assistant? Are they the same?
As a hiring manager and team lead for a large fintech firm in London, I would happily see an individual who had really mastered the above path(s) as a strong candidate for an intermediate or upper junior role in applied data science/machine learning. But ... it's not enough to be a senior, and certainly not an expert. Just my two cents.
Expertise is relative. Having compiled a few kernels in my life I wouldn't consider myself an operating systems expert, but to a career plumber, I am. Heck, a friend of mine is a PhD in physics, operates a particle accelerator for work, and to him I'm an operating systems expert despite my protests to the contrary.
Agree 100% with you but would add that from the perspective of a lot of business people, the bar for expert is actually pretty low and however far over it you are, they dont actually get any more out or you because they don't know what to do with you. What a lot of businesses want is someone they can call an expert but that still operates within the realm and understanding of a non specialist manager.
Using a PhD as a gateway into applied ML is so horrifically misguided I hardly know where to even begin debunking it.
PhDs are one especially crappy way to prove you have the intellectual chops to engage with ML. There are far more direct, practical, and expedient alternative paths to get there.
Importantly, the number of people who were perfectly capable of doing a PhD but chose not to (because, frankly, it's a very bad deal) vastly dwarfs the number of people who stuck around in academia and obtained one. Additionally, my observation at several major tech companies is that PhDs have a bent of mind that is roughly orthogonal to the pursuit of real business value.
Yikes! This doesn't really come across as a nice comment (bent minds???). There are good concrete reasons to pursue a PhD (ignoring soft reasons like pure interest): wanting a research career is one - it's pretty difficult to get hired as a scientist without a PhD. Also, historical evidence doesn't really support your claim that R&D is orthogonal to business value. Sure, pure science is often independent from $$$ (despite plenty of examples of producing real value), but applied R&D is oftentimes parallel to value generation. if R&D in general is useless, why do top tech companies spend big money on research groups?
Getting a PhD is a fine deal if you have good reasons.
I explicitly caveated my statement with "Using a PhD as a gateway into applied ML" for a reason. The vast majority of people going into applied ML are not pioneering new methods. They're using ML as a tool to support business objectives. This is the group I'm talking about.
The phrase "bent of mind" roughly implies "the way someone thinks". Its usage is declining I suppose but there's nothing connotatively nefarious there.
Nothing wrong with this roadmap but I'd suggest that this and similar ones are squarely in the "trade" category of formation, where the focus is on a large number of practical topics instead of a more solid grounding in the fundamentals, that mostly dont even concern themselves with practical dat science - what I would call the "university" approach, but maybe not in the sense of a modern university.
For work as a tradesperson, it is worth only focusing on the practical application- and I'm saying this genuinely. But I think it should be clearly distinguished from the different kinds of lasting benefits that a more solid fundamental education provides, including the flexibility to adapt.
I'm very biased here, having studies Electrical Eng and CS before modern ML was remotely mainstream, and comparing my conceptual understanding and what I learned in school about math and linear algebra with the way things are understood by tradespeople with far more knowledge of modern tools than me. So crotchety old person- maybe, but I'm happy I went to university.
I would not advise anyone to go down the "data science" career path at this point, unless one of these topics is a true passion of yours and you can't imagine doing anything else (and even then I would recommend alternatives if possible).
The explosion of AI/ML/Data Science teams in places that they really don't belong is going to have major backlash soon. The market is currently flooded with desperate PhDs who have reduced years studying their field into a flurry of buzzwords on a resume that they don't understand.
The vast majority of AI/ML/DS teams don't know what they're doing. Management and "leadership" in this industry would be laughable if it wasn't so horrific. The vision of AI/ML/DS has turned into a nightmare of people shoving data they don't understand into models they don't understand and making sure someone else is using that so they can justify their work. You could likely cut entire 100+ DS teams at most companies today and only see a positive impact on the business.
There are of course really interesting niches that require these skills and skilled practitioners. But if you're reading a guide on how to be an "AI Expert", they you likely aren't one of these people. And because of the signal noise problem in this space it's hard for even really excellent people to get matched to the hard problems that genuinely need their help.
I wonder what the impact of easy-to-use cloud ML tools is going to be on the industry. BigQuery already lets users train models using a simple SQL query. AutoML makes simple ML tasks as easy as run and verify accuracy and run again. Of course, a basic amount of knowledge is still need to avoid common pitfalls - unclean data, overfitting, etc. Perhaps this will just mean data scientists can produce better work with the same amount of time. It doesn't seem like a good use of everyone's time to have to know so much cutting-edge research to solve run-of-the-mill business problems. Hardly any other area of CS requires that.
For readers who read that and think they still want to do it:
> The vision of AI/ML/DS has turned into a nightmare of people shoving data they don't understand into models they don't understand...
Don't be this ^. Please.
If you do want to come into the field, please first take the time to understand the tools. Learn enough to re-derive and re-implement the tools you're using. Learn the assumptions that need to hold. Don't just learn to point the tools at data and hit go. It's not like a lot of programming where it either succeeds or fails. Failures are often silent in this field. If you can't reason through how and when a given tool will succeed and, probably more importantly, reason through how and when a given tool will fail, you haven't learned enough to know what to use and when.
I totally agree with you, but let's be honest, in this publish-or-perish world in academy, and the cut-throat environment of VC-funded start-ups, people will, if not outright lie, exaggerate the nature and importance of what they are doing. Too much money and "prestige" is at stake.
The "right" way is slow, lonely and unglamorous. And most importantly without that fast money, acquisitions and promotions.
I would say, however, there is ample opportunity for infrastructure designers/engineers in the space, especially in a world where everyone is going "cloud" but cloud doesn't cut it in the most demanding applications, so those of us who love data center design are sort of loving the resurgence of onprem/Colo usage and the opportunities for pushing the envelop on data transfer/processing rates, etc. The skills and knowledge gained are very useful in other industries, with a side bonus of, if you pay attention, knowing a bit more about the "big picture" of AI than many of the phd's!
> The explosion of AI/ML/Data Science teams in places that they really don't belong is going to have major backlash soon
I'm not disagreeing with you, just curious, did that happen with Big Data teams? If not, what happened with all the people working around that concept a few years ago? Might be an indicator of what could happen to DS teams.
There’s a lot of truth in this, but I find the people that pivot toward practical software engineering skills early in their PhD are still highly valuable to tech companies.
I think the issue here is you can’t become highly skilled in software engineering overnight, and compensating by flexing softer “problem solving skills” isn’t enough to compensate. If you want to be effective in data science, you need to commit to being a rockstar programmer AND statistician AND be well versed in a number of different algorithms.
Unsurprisingly, this is difficult, and so the market of people who do it well is small. That’s not, however, a referendum on the value of predictive modeling and software engineering. I don’t see advanced math and software engineering skills going out of fashion anytime soon, simply because they are hard to acquire and immensely powerful in the right hands.
If you are looking to work in an enterprise as a data engineer, submission of research papers with code is certainly not a pre-requisite. Learning something useful like CPMAI methodology is more practical.
This is great in some aspects but really shallow in others.
In fundamentals, I don't see a reference to geometry or calculus. Good luck understanding methods like UMap or more modern clustering techniques, optimal transport methods in AI, and more recent gradient techniques without those.
Some might say this is more for the AI researcher, but people need to implement bleeding edge stuff all the time.
I really appreciate the taxonomy and links given there, but whenever I see someone claiming "you can't do Z before you've mastered X and Y" it tells me that someone's afraid of others succeeding with simply skipping the masterhood of X and Y and going straight to Z, picking up just enough X and Y in the process.
In fact, since the direct path is more rocky than going via X and Y, it naturally selects the most motivated people, so those succeeding in it will generally outperform those who went X-Y-Z.
At this point I feel like some crazy person talking to themselves when I say this, but "AI" is a research field that includes many more sub-fields than machine learning, let alone deep learning, and it's impossible for someone to be an "AI Expert" while ignoring this, even if they think they are because they ignore it. I appreciate that synecdoche is a thing, but to recommend a "roadmap" to becoming an "AI Expert in 2020" while ignoring most of AI is...
... well I don't know what it is, anymore. Sign of the times? Perhaps the field (AI that is) should abandon its name just to be sure that if a backlash ever comes against deep learning it won't take everyone else's reseach with it?
On the other hand, I feel a bit like the horses and the cows in Animal Farm, when the pigs took over the Revolution. Not that there was any revolution, not really, but the way that the research trends have shifted lately, from what would make good science to what will make you hired by Google, is a little bit of a shock to me. And I stared my PhD just three years ago. It's come to the point where I don't want to associate my research with "machine learning" and I don't want to use the term in my thesis, for fear of the negative connotations (of sleazy practices and shoddy research) that I am concerned might be attached to it in the time to come.
And it's such a shame because the people who really advanced the field, people like Joshua Bengio, Jurgen Schmidhuber, Geoff Hinton, Yan Le Cunn etc are formidable scientists, dedicated to their preferred approaches and with the patience to nurture their ideas against all opposition. The field they helped progress so much deserved better.
GOFAI didn't work. ML is the only AI approach that ever did. Sure, your tree searches and your planners and your grammars and your knowledge graphs and your expert systems still have their places... but they're not AI, and they never will be.
Amicably, can I ask you how you know what you say above to be true? Where does your knowledge of AI come from?
Edit: I'm asking because there is evident in your comment a confusion that is all too common on the internets today, of thinking of "GOFAI" (symbolic AI) and "ML" (machine learning) as two somehow incompatible and perhaps exclusive sub-fields of AI. This is as far from the truth as it could be, for instance machine learning really took off as a subject of research in the 1980's with the work of Ryszard Michalski and others, who considered machine learning approaches for the purpose of overcoming the "knowledge acquisition bottleneck", i.e. the difficulty of hand-crafting production rules for expert systems. Indeed, most of the early machine learning systems were propositional-logic based, i.e. symbolic. And of course, one of the most widely used and well-known type of machine learning approach used to this day, decision trees, hail from that time and also learn propositional logic, symbolic models.
Of course, most people today know "ML" as a byword for deep learning, or at best statistical pattern recognition (if that). It's just another aberration brought on by the sudden explosion of interest in a once small field.
I refer you to Michalski's textbook, "Machine Learning: An Artificial Intelligence Approach" and Tom Mitchell's "Machine Learning" for more information on the early days of the field, i.e. until ca. 2000.
It doesn't much matter what people wrongly thought in the past, any more than one has to study the four fundamental elements of water, air, earth and fire to become an expert in chemistry. GOFAI didn't work, hence the winter. If it did I'd think otherwise.
It's true that you can, in some places, merge the two. The significant majority of the time, this just makes your ML system worse in the long run, per The Bitter Lesson. Occasionally—very occasionally—your problem is fundamentally simple, so something brittle like AlphaZero works, even though we'd still rather shave the fixed-function parts off with MuZero and the like. But it's no coincidence this is reserved for simple, thoughtless problems (like brute-force move search) and kept isolated from the thinking, as intelligence needs generalization and abstraction, and GOFAI doesn't generalize or abstract.
> And of course, one of the most widely used and well-known type of machine learning approach used to this day, decision trees
Though decision trees live a healthy life in data analysis, alongside things like k-means clustering, they're obviously not AI.
> Of course, most people today know "ML" as a byword for deep learning, or at best statistical pattern recognition (if that).
‘Statistical pattern recognition’ is just name calling devoid of real criticisms. I can ask GPT-3,
> a = ["fitness", "health", "heart"], b = ["lifting", "curls", "squats"], c = ["running", "jogging"], so what is b.append("pushups")?
or I can give it the Loebner Prize questions which it can almost ace, or I can test it on 10-digit addition and it gets 60% right (with thousands separators, but still suffering BPEs), or I can ask it to do quirky tasks like shuffling letters, or ask it to write sentences with novel words and find that it understands nuanced differences between the definitions...
You can scream ‘but it's just statistical pattern recognition’ all you want, but the criticism doesn't mean anything if ‘statistical pattern recognition’ includes this level of generalization, reasoning and algorithmic sophistication in natural language with scaling curves that assure us the best is yet to come.
Before I continue this conversation, I'd like to ask, do you think this is an appropriate response to my comment? I responded to your original comment respectfully and politely.
Note that I did not use "pattern recognition" in a derogatory manner, neither did I say that anything is "just" pattern recognition. I think you may be mistaking my comment for a different opinion that you disagree with.
While waiting for Veedrac's reply to my comment, I thought I'd clarify here what I meant by "statistical pattern recognition". "Pattern recognition" is the name of the sub-field of AI research from which statistical machine learning grew into what it is today, for example many problems in machine vision are typically considered as pattern recognition problems, etc. "Statistical" refers to the methods used to achieve the task, e.g. neural networks are normally filed under "statistical AI" (for historical reasons).
Like I say in my previous comment, modern machine learning research started as a discipline that was separate from pattern recognition (and Pattern Recognition was sometimes considered distinct to AI, as a research subject). I quote below from Tom Mitchell's wildly influential paper, "Generalisation as Search" (AI 18, 1982):
"5.2. Statistical pattern recognition
The field of statistical pattern recognition deals with one important subclass of generalization problems. In this subclass, the instances are represented by points in n-space, and the generalizations are represented by decision surfaces in n-space (e.g. hyperplanes, polynomials of specified degree). The matching predicate corresponds to determining whether a given point (instance) lies on one side or another of a given decision surface (generalization). The field of Statistical Pattern Recognition has developed very good generalization methods for particular classes of decision surfaces. Many of these methods are relatively insensitive to errors in the data and some have well understood statistical convergence properties, under certain assumptions about the probability distribution of input instances."
"In contrast to work in Statistical Pattern Recognition, work on the generalization problem within Artificial Intelligence has focused on problems involving a different class of instance and generalization languages. These languages are incompatible with numerically oriented representations that describe objects as feature vectors in n-space. For example, Winston's program [21] for learning descriptions of simple block structures such as arches and towers, represents instance block structures in terms of their component blocks and relationships among these. In this domain the natural representation for instances is a generalized graph rather than a feature vector. (...)
Pattern recognition these days is dominated by neural methods, e.g. CNNs for object classification etc. and so is machine learning so it kind of makes sense that the three terms are used interchangeably, but unfortunately many people are not aware of the historical context of the terms, hence misunderstandings as the one by Veedrac above, regarding my comment that machine learning has become a byword for statistical pattern recognition: it's not because I'm dismissive of statistical pattern recognition, or of the ability of deep learning systems to perform it; it's because the terms have really become interchangeable and I think even among researchers.
In any case, before commenting on a complex subject of research with a long history, my recommendation remains to first become well acquainted with the subject and its history. Otherwise, one runs the risk of appearing confused.
Edit: Note that Mitchell's use of "generalisation" in the excerpt above does not refer to the ability of a model to generalise to test, or unseen, data. Rather, "generalisation" in Mitchell's paper refers to the ability of a system "to take into account a large number of specific observations, then to extract and retain the important common features that characterize classes of these observations."...
> Note that I did not use "pattern recognition" in a derogatory manner, neither did I say that anything is "just" pattern recognition. I think you may be mistaking my comment for a different opinion that you disagree with.
> [...]
> In any case, before commenting on a complex subject of research with a long history, my recommendation remains to first become well acquainted with the subject and its history. Otherwise, one runs the risk of appearing confused.
Sorry, I did misunderstand what you were saying.
I do find it somewhat baffling how you're talking as if 1980s terminology is today's terminology, or that everyone should just recognize it as the default way of things. As much as you argue I should adopt the subject's history, I'm saying you should adopt its present.
The point you're making would have held fine two decades ago, but as a description of modern ML, it's like calling a sedan a cart. Even if you don't mean it as a criticism (‘just a cart without a horse’), the world has moved on.
>> Sorry, I did misunderstand what you were saying.
Thank you for your honesty and there's no need to apologise.
Let me explain my point above, if I can. You said that search, planning,
expert systems, etc, are "not AI, and they never will be". I understand that
as saying that such systems are not artificial intelligences, in the sense of
an intelligent machine created by humans out of whole cloth (without trying to
define what "intelligent" means).
That is certainly true, but it is also uncontroversial that the above are
sub-fields of the field of research that is known as "AI". That is, there is a
field that researches approaches that could lead to the creation of
intelligent machines and that is called "AI" and then there's the ultimate
target of that field which is to create "AI".
My original comment bemoans the fact that in some sectors, "AI", as a field of
research, has become synonymous with only one sub-sub-field of AI research,
that is, deep learning.
Contrary to what you state, these "GOFAI" fields (symbolic AI, if you will)
are still active and far from having "failed" in any way, they are "SOTA" in
their respective tasks. For example, the field of automated theorem proving is
dominated by systems that employ the resolution rule, a logic-based approach
and while recent efforts have been made to make inroads into theorem proving
with deep neural nets (e.g. a recent attempt to use transformers) results are
still very far from being comparable to the traditional approaches. I know
more about automated theorem proving that I know e.g. about planning or search
(because my PhD is in a subject close to the former) but my understanding is
that in those fields too, traditional techniques dominate- which is why
research in them is still active.
If I am permitted to tout my own horn, my broader subject area can be
described as "program learning", i.e. machine learning of arbitrary programs
from examples of their inputs and outputs. In this area also, deep learning
systems are hopelessly outclassed by symbolic approaches, not least because
these approaches learn precise representations of target programs (rather than
approximations) from a mere handful of examples (four or five, etc).
And so on. These are just some examples of AI research that is ongoing, that
is not deep learning, and that is state of the art for its respective tasks.
In view of that, I consider that using "AI" to mean "deep learning" (as the
article above does) is not only misinformed but also misinforming and actively
harmful. In the sense of harming people's understanding, that is.
As to your comment about how GOFAI "failed", I'm afraid this opinion, which is
common, is also misinformed. Here, too, a knowledge of the history of the
field helps, but to summarise, the last winter happened because of strictly
political reasons and for no reason that had anything to do with the
scientific merits, or practical successes of the relevant approaches. In fact,
expert systems, now widely considered a failure, were one of the first big
success stories of AI; a success story that was cut short only because, again,
of political reasons.
I could talk about the AI winter subject for hours (it's a favourit subject of
mine) but a good starting point is this article by the editor of the IEEE
journal Intelligent Systems: (Avoiding Another Winter)
https://www.computer.org/csdl/magazine/ex/2008/02/mex2008020....
The wikipedia page on https://en.wikipedia.org/wiki/AI_winter also has a
decent summary and some sources. Finally, see the wikipedia article on the
Lighthill Report Veedrac↗
I don't want to dismiss automated provers, as they are often quality, useful tools (SAT solvers in particular), but if you're interested in learning AI, traditional approaches are no longer more than briefly and tangentially relevant.
That's my point, that you don't need to learn woodworking to build a car, even if wooden carts still have occasional uses, and some cars have wood trim or wooden trailers.
> If I am permitted to tout my own horn, my broader subject area can be described as "program learning", i.e. machine learning of arbitrary programs from examples of their inputs and outputs. In this area also, deep learning systems are hopelessly outclassed by symbolic approaches, not least because these approaches learn precise representations of target programs (rather than approximations) from a mere handful of examples (four or five, etc).
I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three
strings n1, n2, and n3, return true if length(n1) <
length(n2) < length(n3), and false otherwise”, and that's with 100 examples. So, probably less practically useful than GPT-3, which wasn't even trained on the task.
> the last winter happened because of strictly political reasons and for no reason that had anything to do with the scientific merits, or practical successes of the relevant approaches
I disagree, but you've not given me much more than a list of vague references that don't all exactly support your argument, so I don't have much clue where you diverge.
If GOFAI worked, we'd see some indication of it working (again, in an AI context), but we don't.
> But if you think about it, in the era where the trend is to use deep learning, the most interesting results can only come from not using deep learning. In a gold rush, if everyone is digging up Widow's Creek, then Widow's Creek is the last place to dig for gold.
This analogy doesn't work. Neural networks are giving unparalleled results by the bucket. That's why people are digging there. A gold mine might have plenty of competing miners, but it's sure going to be a lot more likely to give you chunks of gold than a random patch of grass in your backyard.
I'm happy to see we're still in healthy disagreement. However, I have to
apologise for confusing you by describing my field as "program learning" which
is admittedly vague, but I didn't want to go into the particulars. My field is
not program synthesis, which is constructing programs from complete, formal
specifications. Rather, it's Inductive Programming and more specifically Inductive
Logic Programming (ILP), which is learning programs from examples, i.e.
"incomplete specifications". I'm not familiar with the General Program
Synthesis Benchmark Suite, but the problem you list (test three strings are
ordered by length) is trivial for ILP approaches. Again, I don't want to point
to my own research, directly (I'm going through a bit of a modesty phase) (oh,
alright, it's just that the documentation of my project is crap). However, I
have to say that even so, if something is a difficult problem for program
synthesis approaches, then it's very unlikely that neural networks will do any
better at it. For instance, do you know how well deep neural nets perform on
this benchmark? I can't find any results with a quick search.
You make the point that one does not need to learn these "obsolete" AI
approaches because they are not relevant anymore. I don't understand why you
say that. These approaches are still state of the art for their respective
tasks and there is no other approach that has been shown to do any better,
including deep neural networks. In what sense are they "no longer more than
briefly and tangentially relevant" as you say?
Regarding the gold rush, the point of the analogy is that in a gold rush only
a very few people will ever strike gold. This is exactly the state of research
into deep learning currently. After a few initial big breakthroughs, like CNNs
and LSTMs, progress has stalled and the vast, vast majority of published
papers (or papers put on arxiv permanently) present incremental results, if
that. Literally thousands of deep learning papers are published each month and
the chance to have an impact is miniscule. From my point of view, as a
researcher, going into deep learning right now would be career suicide. Not to
mention that, while the first few successes were achieved by small academic
teams, who had typical academic motives (er, glory), the game has now passed
to the hands of big corporate teams that have quite different incentives, so
it's almost impossible for small teams or individual researchers to make a
dent.
As to the winter and whether GOFAI works, perhaps I haven't convinced you with
my sources, but in that case, I have to go back to my earlier question and ask
where your knowledge comes from. You clearly have a strong opinion on GOFAI
and the AI winter of the '80s, but what knowledge does this opinion come from?
Can you say? And if this sounds like a challenge, well, that's because it is.
I'm challenging you to re-examine the basis of your convictions, if you like.
Because to me, they sound like they are not well-founded and that you should
put some water in your wine. The things you say "don't work", work and the
things you say work, don't work as well as you say.
For my part, I certainly agree that GPT-3 or the next iteration of a large
transformer-built language model can be a useful tool, but such a tool will
always be limited by the fact that it's, well, a language model, and it can
only do what language models do, which does not include e.g. the ability for
reasoning (despite big claims to the contrary) or arithmetic (ditto) or
generation of novel programs. For instance, the append() example you
show above is clearly memorised: you haven't given the model any examples of
append(), so it can't possibly learn its definition from examples. It only
returns a correct result because it's seen the results of append() before. Not
the same result, but close enough. Like I say, this ability can definitely
be useful- ...
> Rather, it's Inductive Programming and more specifically Inductive Logic Programming (ILP), which is learning programs from examples, i.e. "incomplete specifications". I'm not familiar with the General Program Synthesis Benchmark Suite, but the problem you list (test three strings are ordered by length) is trivial for ILP approaches.
The General Program Synthesis Benchmark Suite works from input-output examples, not “complete, formal specifications”.
How would you tackle this with ILP?
> However, I have to say that even so, if something is a difficult problem for program synthesis approaches, then it's very unlikely that neural networks will do any better at it. For instance, do you know how well deep neural nets perform on this benchmark?
I'm not aware of any serious at-scale attempts. Your option is basically to try few-shot with GPT-3.
OTOH, learning these trivial programs from 100 examples is a largely artificial framing used to support a field which hadn't worked its way up to meaningful problems, and in the more general sense, large networks are promising; eg. the GitHub-trained GPT:
> These approaches are still state of the art for their respective tasks and there is no other approach that has been shown to do any better, including deep neural networks. In what sense are they "no longer more than briefly and tangentially relevant" as you say?
“if you're interested in learning AI”
These techniques were invented from the field of AI, but that does not mean they remain in the field of AI.
> You clearly have a strong opinion on GOFAI and the AI winter of the '80s, but what knowledge does this opinion come from? Can you say?
I can argue why ML approaches are good and promising and point at that. I can argue why ML approaches make conceptual sense whereas GOFAI does not, though I don't see us resolving that short-term so I'd rather not. But what I can't so easily do is point to the non-existence of GOFAI AI successes. It's just not there.
You do have tools Watson and WolframAlpha which use GOFAI techniques for fact search over a large set of human-built knowledge repositories (trivia q's / math tools), but Watson is mostly considered a stunt, and I'm not aware of anyone calling WolframAlpha AI.
> the ability for reasoning (despite big claims to the contrary)
The nebulousness of the term ‘reasoning’ is pulling a lot of weight here. It's clearly doing sophisticated computations of some sort, beyond brute memorization.
There are more examples too, this is just addressing the one point people get wrong most often. BPEs are an interim performance hack, not an indictment on the approach in general.
> or generation of novel programs
Is clearly false.
> For instance, the append() example you show above is clearly memorised: you haven't given the model any examples of append(), so it can't possibly learn its def...
In the interest of pruning this conversation a bit I will not continue the
discussion about GPT-3. Apologies, but this thread is growing too fast and I
don't have the time to give your comments the attention they deserve. I am
happy for you to have the last word in that matter.
>> These techniques were invented from the field of AI, but that does not mean
they remain in the field of AI.
Like I say above, it is pretty uncontroversial that these approaches are part
of the field of AI research. You can consult wikipedia or e.g. the calls for
papers from major AI conferences, AAAI and IJCAI, if in doubt.
So I have to ask again, why do you say these approaches are are not in the
field of AI research? According to whom? And based on what?
I would please like an answer to the above question.
Further, I can certainly point you to successes of symbolic AI, which you say
don't exist. For one thing, the entire fields of automated theorem proving,
planning, search, game playing, knowledge representation and reasoning, etc.
that you say are "not AI", but are like I say still active and still state of
the art in their respective tasks. These are certainly successful- they have
produced systems and techniques that still work best than any alternative and
actually quite well.
For examples of specific systems that were successful in their time, see Logic
Theorist [1] that proved 38 of the first 52 theorems in Principia Mathematica;
Chinook [2], the first computer program to win a world championship against
humans (in checkers/draughts); Deep Blue [3], the first AI system to defeat a
human grandmaster (Garry Kasparov) in chess; MYCIN [4] the first AI system to
outperform human experts in disease diagnosis (specifically, diagnosis of
infections); and so on.
Of course these systems have been superseded - but they were successes
nonetheless. Another reason to learn the history of AI is to become aware of
those systems- they, indeed, were "there".
Again I have to ask you- where does your knowledge of AI come from? When
you make such strong statements about what works and what doesn't, what failed
and what succeeded, are you sure you are well informed? Do you draw your
knowledge from primary sources, or are you trusting the opinions of others who
claim to be experts- but may not be (like in the article above)?
>> How would you tackle this with ILP?
Below I've defined the problem in the format expected by Louise [5]:
To explain, shorter/2 is a predicate defined as background knowledge by me.
triadic_chain is a metarule, a second-order clause that provides inductive
bias. length/2 is an ISO Prolog predicate.
Like I say, this is a trivial problem, not least because its solution is easy
to figure out and the background knowledge and metarules are trivial to define
by hand. Louise can also perform predicate invention to define new
background knowledge (kind of like inventing new features) and also new
metarules. That is to say, Louise can learn the shorter/2 and length/2
programs, also from very few examples- and then reuse them as background
knowledge. But showing how to do that would make for a larger example. I'm
happy to oblige if you are curious.
I should point out that there exists no neural net approach that can learn the
same (or a similar) program from a single positive example- not least because
neural nets cannot make use of background knowledge (i....
> You can consult wikipedia or e.g. the calls for papers from major AI conferences, AAAI and IJCAI, if in doubt.
As to Wikipedia, see the second paragraph. The sections where it mentions, eg., symbolic or sub-symbolic approaches are prefixed with “Researchers in the 1960s and the 1970s” or “By the 1980s”. Kind'a telling.
Like, my point is not about whether you can find the odd person trying to solve intelligence with grammars, or what were GOFAI conferences still harbour GOFAI research in the corners, my point is that a) these approaches don't work as a way to actually tackle AI, the problem, b) the vast majority of the field does not take them as seriously as a method of doing so, regardless of other uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise without having much care for those parts of the field.
> Further, I can certainly point you to successes of symbolic AI, which you say don't exist. For one thing, the entire fields of automated theorem proving, planning, search, game playing, knowledge representation and reasoning, etc. that you say are "not AI", but are like I say still active and still state of the art in their respective tasks.
Yes, but there's a reason I suffixed that comment with “(again, in an AI context)”. GOFAI is great if you ignore the last two letters of the name, and how it failed almost all its major promises.
These used to be considered AI because it was thought that you could build a useful reasoning agent out of a combination of these techniques, given appropriate developments. Now (almost) nobody does that; Google Map's pathfinding is just a pathfinder, not a general reasoner.
> Below I've defined the problem in the format expected by Louise
Right, OK, I figured it'd be something like this, when you said it was trivial, but this is just another perspective on my original criticism. You wrote the program you wanted it to generate as the background knowledge.
It must be so, because your examples don't specify, even roughly approximately, the program you wanted generated. Another valid solution would be (among many)
ordered(A,B,C):-shorter(A,B),shorter(A,C).
and the only reason it didn't choose this is because you gave it the program you wanted it to generate (obfuscated a little, yet still, there was only one application available). It didn't ‘learn’ anything.
> I should point out that there exists no neural net approach that can learn the same (or a similar) program from a single positive example- not least because neural nets cannot make use of background knowledge (i.e. a library of programs from which to build other programs).
This is not true, but as you wanted to prune, I'll leave it there.
>> Like, my point is not about whether you can find the odd person trying to
solve intelligence with grammars, or what were GOFAI conferences still harbour
GOFAI research in the corners, my point is that a) these approaches don't work
as a way to actually tackle AI, the problem, b) the vast majority of the field
does not take them as seriously as a method of doing so, regardless of other
uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise
without having much care for those parts of the field.
>> Yes, but there's a reason I suffixed that comment with “(again, in an AI
context)”. GOFAI is great if you ignore the last two letters of the name, and
how it failed almost all its major promises.
>> These used to be considered AI because it was thought that you could build a
useful reasoning agent out of a combination of these techniques, given
appropriate developments. Now (almost) nobody does that; Google Map's
pathfinding is just a pathfinder, not a general reasoner.
I keep asking- how do you know these things to be true? Are they just your
opinion?
Because GOFAI just observably doesn't work. The ideas are brittle, can't generalize and abstract the way is needed, has made very little progress recently (if any) an AI context, and you just _don't see_ anything that would argue otherwise.
In contrast, ML methods do work, observably and clearly, and they work in a ridiculously general way, to a degree larger than almost anyone thought (or even thinks) is reasonable for them to.
And it's not just my opinion; there's a reason AI conference attendance has shot up a factor of 10 or so in the last few years, why NeurIPS is the leading one (and even historically GOFAI conferences are majority NNs), why the big AI labs with big AI cash are all doing NNs, and why all of a sudden AI is a popular topic outside academia.
If this doesn't answer your question, perhaps answer the opposite; how do you know that it's wrong?
>> If this doesn't answer your question, perhaps answer the opposite; how do you
know that it's wrong?
I know the literature. It's my job.
>> And it's not just my opinion; there's a reason AI conference attendance has
shot up a factor of 10 or so in the last few years, why NeurIPS is the leading
one (and even historically GOFAI conferences are majority NNs), why the big AI
labs with big AI cash are all doing NNs, and why all of a sudden AI is a
popular topic outside academia.
That's still an opinion- "it's not just my opinion, everyone says so". A.k.a.
"It is known", in Dothraki. And of course it is of no consequence who's
spending money on what and who's submitting papers where. The volume of
research was never a criterion for its quality. Heed thee well the legend of
our Lord Geoff Hinton's years in the academic wilderness and how he emerged
victorious with the laws of deep learning in his hands.
I think what you've said so far has convinced me you're expressing a personal
opinion that is strongly held without a good reason to do so. You make
sweeping statements with great certainty, but you don't really seem to know
how you know the things you know, so you end up "knowing" some things that you
don't really know. For instance, you claimed that "GOFAI" successes just
"aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't
seem to have heard of these before (I'm more surprised about not knowning of
Deep Blue than MYCIN).
You also claim that "these approaches are not AI". That's a "No True
Scottsman" right there. Except there really is no True Scottsman (i.e. "AI" in
the sense you use it)- ask Yoshua Bengio:
Bengio: In terms of how much progress we’ve made in this work over the last
two decades: I don’t think we’re anywhere close today to the level of
intelligence of a two-year-old child. But maybe we have algorithms that are
equivalent to lower animals, for perception. And we’re gradually climbing this
ladder in terms of tools that allow an entity to explore its environment.
Spectrum: Will any of these ideas be used in the real world anytime soon?
Bengio: No. This is all very basic research using toy problems. That’s fine,
that’s where we’re at. We can debug these ideas, move on to new hypotheses.
This is not ready for industry tomorrow morning.
> For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN).
At this point I think we're just hopelessly talking past each other. Of course I know about Deep Blue. I didn't know about MYCIN, but, like, “MYCIN was never actually used in practice”, so I don't feel particularly bad about missing that one.
But neither of those challenge my point. If you want to go back in time 30 years, then sure, if you want to be an AI expert, then you have to know GOFAI. That's what the ‘OF’ stands for.
> I know the literature. It's my job.
Yah I read the literature too. (Albeit it seems a very different subset.) That's not an argument though.
But none of those sources says that e.g. search or planning are not AI fields. That was your original claim, if I'm not mistaken? Anyway it doesn't matter. It's a very strange thing to say and I was just trying to understand what made you say it- strictly out of curiousity.
I too can quote Hinton -from memory and without a link. I remember him saying that the next big thing in AI will come from a grad student who distrusts everything he (Hinton) has ever said. Unfortunately, I won't be that grad student- I haven't heard everything that Hinton has ever said.
I best summarized my claim when I said the following. Whether or not it's an ‘AI field’ is not very interesting to me, as long as the following holds.
---
Like, my point is not about whether you can find the odd person trying to solve intelligence with grammars, or what were GOFAI conferences still harbour GOFAI research in the corners, my point is that a) these approaches don't work as a way to actually tackle AI, the problem, b) the vast majority of the field does not take them as seriously as a method of doing so, regardless of other uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise without having much care for those parts of the field.
Apologies for splitting the thread, but I thought it'd be easier to read this
way.
This comment addresses your concerns about me writing the program I wanted
Louise to generate. I like to see background knowledge ("BK", e.g. shorter/2)
as a library of sub-programs from which the learner can select the ones
necessary to compose a target program. The example above is trivial because
I've defined a BK predicate that is necessary and sufficient to learn, so the
learner was indeed served the solution "on a plate".
However, as I said in my previous comment, Louise can learn its own background
knowledge. This can be done by predicate invention, or more simply, by
incrementally learning necessary sub-programs.
Below is a problem definition and learning session that first learns length/2
(renamed llength/2 to avoid name clashes with the built-in) and shorter/2 from
list and numeric function primitives, before using the learned predicates as
BK for ordered/2. Like I say in my previous comment, it's a little larger
than the previous one:
The BK for this problem consists of tail/2, similar to "car" in Lisp (i.e.
matches the head of a list) and the pair of p/2 and s/2, that act as
"dereferencers" to Peano number functions. These are bog-standard Prolog
programs and useful whenever a target program must manipulate a list, or
perform numerical reasoning. In other words, they're pretty much generic, like
a standard library of sorts.
I've added the full source of the experiment file for the learning task on
pastebin. It includes a few more detailed comments and a set of constraints to
clean up the learned hypothesis, mostly for aesthetic reasons:
Of course this is still a toy problem and we know the solution. But I hope it
demonstrates the principle. On the other hand, you'd still not be able to
solve this with alternative approaches, e.g. I see that the benchmark suite
you pointed to is used for genetic programming. I'm also not aware of neural
approaches that build programs incrementally, from a couple of examples of
each sub-program.
That is to say, this is a toy problem for ILP. For other approaches it's
unsolvable.
Heck, I don't think it even got `shorter` right; it gave
shorter(A,B):-llength(A,C),llength(B,D),s(C,D).
which means len(A) + 1 == len(B), not len(A) < len(B), and AFAICT it can't learn len(A) < len(B), not because the program isn't expressible with the primitives you gave, but because it just doesn't reason that far.
So again, it's only trivial because it isn't learning the program, it's learning to put fit the puzzle pieces of the program together, after you wrote the program and then chopped it up.
>> Heck, I don't think it even got `shorter` right; it gave
shorter(A,B):-llength(A,C),llength(B,D),s(C,D).
>> which means len(A) + 1 == len(B), not len(A) < len(B), and AFAICT it can't
learn len(A) < len(B), not because the program isn't expressible with the
primitives you gave, but because it just doesn't reason that far.
Regarding reasoning "that far" Louise can learn the complete successor /
predecessor relation (</2 and >/2) on its own and only from the primitives
s(N,s(N)) and p(s(N), N):
However, in the ordered/3 problem I define p/2 and s/2 by hand so that I can
put in ground_peano/1 to avoid infinite recursion when Louise tries to pass
two lists to s/2 or p/2 (at that point, their termination conditions never
obtain, so they keep recursing).
You can chalk the potential for infinite recursion up as a limitation, you're
very welcome- but there are techniques to avoid this and guarantee termination
(Knuth-Bendix ordering of the Herbrand base, see ref [1]) which I haven't come
round to implementing yet (because they are not necessary given a bit of
common sense in defining BK, as above). On the other hand that's actually a
feature, in the sense that earlier systems required more specific language
bias than the metarules, that would avoid this kind of type-unsafety, but also
demanded more expert knowledge from the user. In any case, there's outs.
>> You've just kicked the can down the road; what you've given there cannot
solve, for instance, the same problem but with <= instead of <.
I don't think you've understood my point. At every step of the way you've put in more effort than writing the program and specifying the examples. Clearly this is not trivial.
If you're actually inferring programs and not just doing a sort of guided line-by-line generalization of a program you had already written, the only things you would need would be
Aren't you shifting the goalposts a bit? First you ask me how I'd solve this
problem with ILP:
“Given three strings n1, n2, and n3, return true if length(n1) < length(n2) <
length(n3), and false otherwise” (a)
I say this is a trivial problem to solve with ILP and I show you a, well,
trivial solution and you complain that it's - trivial.
Then I show you a more elaborate version that learns sub-programs on the way
to the full solution and you say that a) it doesn't solve a different problem,
with ≤ instead of < and b) that it's not trivial anymore.
And now you're saying you want a solution that learns from examples only. You
would have saved us both a lot of time had you clarified your expectations up
front.
No matter. There isn't anything that can do what you ask. Or rather, there are
many approaches that could learn (a) just from examples, with a brute-force
search. But there is no approach that could learn arbitrary programs only from
examples. The reason is that the space of all programs that can be computed by
a Universal Turing Machine ("arbitrary") is infinite and any learner trying to
find one of them blindly, without some kind of hint to guide it, would be lost
for ever inside it.
Most machine learning approaches that learn programs from examples adopt some
sort of inductive bias to guide a search for a program that satisfies some set
of goodness criteria, including neural approaches [1]. In ILP, inductive bias
consists primarily of BK and language bias (like the metarules in Louise). ILP
has a certain advantage in this, in that the languages of examples, bias and
hypotheses are the same (some first order logic language, like Prolog or ASP)
and so ILP systems can learn their own bias, like Louise can learn its own BK
and metarules. By way of comparison, neural nets, with their hand-crafted
architectures, minutely fine-tuned to specific domains or even particular
datasets, cannot do that (e.g. a trained model can't be used as a feature to
another neural net, in the way that ILP hypotheses can be used as BK). Of
course you need to start somewhere, from obvious primitives like head/2,
tail/2, s/2 and p/2 that I used above.
But I digress. The bottom line is that learning arbitrary programs from
examples is a hard problem for any machine learning approach [2].
Classification is a piece of cake, by comparison. And that is why there has
been so little progress in this problem even after decades of research [3].
The take home message of course is that neural nets are not the end of the
line in AI research and it would be disastrous for the progress of the field
to allow research into neural nets to eclipse every other approach. If this
happens it will all have to be discovered again, from scratch. And in another
70 years.
[3] This is where I'd normally say that there have been recent breakthroughs
that promise to overturn years of slow progress, but that's a story for
another time (and another venue most like).
“I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) < length(n2) < length(n3), and false otherwise”, and that's with 100 examples. So, probably less practically useful than GPT-3, which wasn't even trained on the task.”
I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting.
> The reason is that the space of all programs that can be computed by a Universal Turing Machine ("arbitrary") is infinite and any learner trying to find one of them blindly, without some kind of hint to guide it, would be lost for ever inside it.
I'm not looking to solve arbitrary programs. I'm looking, at least at this first hurdle, to solve a few largely trivial ones.
If you saw 100 examples, you would be able to solve this problem, or at least get very close. Therefore this sort of theoretical argument cannot hold.
> By way of comparison, neural nets, with their hand-crafted architectures, minutely fine-tuned to specific domains or even particular datasets, cannot do that (e.g. a trained model can't be used as a feature to another neural net, in the way that ILP hypotheses can be used as BK).
Section 4.7.1 shows this is entirely feasible for neural networks, though the technique is very different.
Though I prefer even more the elegance (and general absurdity) of learning the bias by pretraining on WebMath (GitHub, arXiv Math, Math StackExchange).
> The bottom line is that learning arbitrary programs from examples is a hard problem
I agree it's hard, but in my mind this sentence completes to “and therefore our only hope is to use most powerful tool we have available, neural networks.”
>> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting.
I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem where you talking about, if not that?
The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples.
What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.
Ah, I see what you mean. You mean that I explicitly told Louise how to compose
each program from its sub-programs. In truth, I did not. In the incremental
learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning
targets and a few primitives from which to construct them. I specifically and
very deliberately did not tell it to use each target to define another. It
figured it out on its own.
For instance, I did not direct Louise to define shorter/2 by means of
length/2. In order to do this I would have to specify length/2 as BK for
shorter/2, but I didn't. Note the BK declarations in the experiment file I put
on pastebin:
background_knowledge(llength/2, [tail/2,p/2]).
background_knowledge(shorter/2, [s/2]).
% ground_peano/1 is added here so it's reported by list_mil_problem/1
background_knowledge(ordered/3, [ground_peano/1]).
background_knowledge(ordered_leq/3, [leq/2,ground_peano/1]).
Remember how in an erarlier comment I pointed out that the definition of
shorter/2 changed to use leq/2, instead of s/2, when the examples of
ordered_leq/3 where in the BK? Again, Louise figured that on its own.
In fact, this ability of Louise (actually, its learning procedure, Top Program
Construction, or TPC) is kind of native, in the sense that TPC was
originally conceived as an algorithm to select relevant background knowledge
for a different learning system ("Thelma" for "Theory Learning Machine"; hence
"Louise"). So it can figure out what BK it needs on its own. Automatic
selection of relevant background knowledge in ILP was the original subject of
my PhD research. Interestingly, it turns out that once we had a system that
could perform this task, it could also learn its own programs.
> You mean that I explicitly told Louise how to compose each program from its sub-programs. In truth, I did not. In the incremental learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning targets and a few primitives from which to construct them. I specifically and very deliberately did not tell it to use each target to define another. It figured it out on its own.
No, I get this, it's just not really more than a surface level pretense of choice. The hard thing about generating programs is that there are many possible programs; something like m^n, where ’m’ is the number of functions you have available to use (say, ~1000) and `n` is the number of steps the program needs to take (say, ~5 in this case), and there's another factor for where to put the parameters which here is low enough to be mostly negligible. It turns out even 1000^5 is really big, so this problem is hard if you don't do it smart.
The ‘choice’ you are offering Louise is something like, idk, m=5, n=2. 5^2 is not big. You argue about the terminology, but whatever you call it, it's still exponentially less interesting than the problem first posed. All the interesting work has been done for it, by you.
With the way you first laid out the question, there's a good chance (>1%) I could have gotten the answer mostly right (up to parameter order) without looking at the examples, just the background knowledge and the target type.
Many apologies for the delay in replying - I missed the "more" link at the
bottom of the thread. And here I was, refreshing the page disappointed that no
more criticism was forthcoming.
>> The hard thing about generating programs is that there are many
possible programs; something like m^n, where ’m’ is the number of functions
you have available to use (say, ~1000) and `n` is the number of steps the
program needs to take (say, ~5 in this case), and there's another factor for
where to put the parameters which here is low enough to be mostly negligible.
It turns out even 1000^5 is really big, so this problem is hard if you don't
do it smart.
Indeed, the complexity of the raw, combinatorial problem is the greatest
hurdle in solving it in the general sense, however this time complexity is
calculated somewhat differently than in your comment. Let me show you.
First, in terms of ILP, the "number of functions you have available to use" is
the number of predicate symbols defined in the BK, which I'll notate as p.
"Where to put the parameters" is the number of body literals (similar to
function calls) in each metarule, which I'll notate as k. I'll notate the
number of metarules as m.
"The number of steps the program needs to take" is not relevant to the
calculation: we are trying to calculate the complexity of constructing the
program by blindly combining a set of building blocks (BK predicates and
metarules)- not the complexity of executing the program. What is relevant is
the size of the target theory, i.e. its number of clauses (program lines),
because of course a larger program means a larger number of combinations of
our building blocks. I'll notate the size of the target theory as n.
Putting it all together, the time complexity of constructing a program of n
clauses from p predicate symbols with m metarules with at most k body literals
(of any arity) is O(pmᵏ⁺¹)ⁿ [1]. This is an exponential time complexity that
corresponds to the size of the search space for programs that can be
constructed from these components, i.e. that's the number of constructible
programs. The time complexity of the problem is such that even n = 5 is
sufficient to completely bog down a powerful modern computer.
Louise can manage it because it doesn't conduct a search of that space,
instead it only constructs a unique object in that space, the Top program,
that can be constructed in polynomial time O(pmᵏ⁺¹) [2], i.e. the number of
constructible clauses. Indeed, Louise is capable of learning large programs,
of a few thousand clauses in a few minutes. In other words, the problem is
manageable because of the advances encapsulated by Louise's learning
procedure, Top Program Construction, not because the problem is trivial, as
you portray it - and not because I'm leading Louise by the hand, as you
suggest. Even if I was leading Louise by the hand, the combinatorial space
of constructible programs would still grow exponentially.
Regarding learning "only from examples" as I understand you to mean it, there
is some literature on that, of the kind you say is not "AI" (i.e. it predates
2012's deep learning boom). To my knowledge, this was first discussed in the
following:
1. Introduction
This paper addresses a deep difficulty with the generalization problem as
defined above: If consistency with the training instances is taken as the sole
determiner of appropriate generalizations, then a program can never make the
inductive leap necessary to classify instances beyond those it has observed.
Only if the program has other sources of information, or biases for choosing
one generalization over the other, can it non-arbitrarily classify instances
beyond those in the training set. In this paper, we use the term bias to
refer to any basis for choosing one generalization over another, othe...
I can't edit my post but I forgot to calculate the actual size of the
hypothesis space for the multi-predicate problem for ordered/3. That is: p =
5 (tail/2, p/2, s/2 and llength/2 and shorter/2), m = 4, k = 3, n = 3 (the
target theory is a clause for each target). The size of the hypothesis search
space, i.e. the total number of programs of size n = 3 is (pm^(k+1))^n =
2,097,152,000.
As I say above, Louise's TPC procedure avoids searching this space and so
effectively ignores the exponential term, reducing the complexity of the
problem to that of, in the worst case, enumerating (pm^(k+1)) = 1,280 clauses.
In other words, the problem is easy for Louise, not because the problem itself
is trivial, but because Louise's learning procedure, TPC, is efficient.
For a further example, in the upcoming publication I mention above, Louise is
shown to learn a 2,567 clause theory in under 220 seconds with perfect
accuracy after training on 20% of all training examples. The hypothesis space
for this problem (grid world navigation) is in the order of 2*10^4944 but
Louise shrinks it to the problem of enumerating, at worst, a little over 81
million clauses.
> "The number of steps the program needs to take" is not relevant to the calculation: we are trying to calculate the complexity of constructing the program by blindly combining a set of building blocks (BK predicates and metarules)- not the complexity of executing the program.
I meant in the sense of denotational semantics, so looping (or rather, finding a fixed point of a loop) is one ‘step’.
> Have you tried doing that? I suggest you do- if only to get a feel for the true difficulty of the problem.
How exactly would you suggest? By the nature of my criticism, I can't construct the Louise BK blind.
If anything you agree here; you say there are only 1,280 clauses it could construct, so I only need discriminatory power of 1-in-13 to have a >1% chance of getting the answer right.
> Regarding learning "only from examples" as I understand you to mean it, there is some literature on that, of the kind you say is not "AI" (i.e. it predates 2012's deep learning boom).
Yes, and as usual for GOFAI it doesn't solve the problem.
>> I meant in the sense of denotational semantics, so looping (or rather, finding a fixed point of a loop) is one ‘step’.
I understand, but we don't need to take that term into account. We're only interested in the cost of a blind combinatorial search. Even if we added that term in, we'd just get a higher complexity.
>> How exactly would you suggest? By the nature of my criticism, I can't construct the Louise BK blind.
I mean, try to define some BK and generate combinations of it until you solve the problem. You say that I made the problem easy because I defined some BK by hand. I suggested you try doing that to see whether the problem is as easy as you think.
>> If anything you agree here; you say there are only 1,280 clauses it could construct, so I only need discriminatory power of 1-in-13 to have a >1% chance of getting the answer right.
There are only 1280 clauses, but there are about 2 billion programs of size n = 3 that can be constructed with those clauses, one of which is the target program. The challenge is to find the target program in that 2 billion.
The advantage of Louise is that it only needs to look at the 1280 clauses, not the 2 billion programs. The trick is to find how to do that. It's like the difference between sorting a list with bubblesort and quicksort. Quicksort has to do a lot less work, but the trick is figuring out quicksort.
>> Yes, and as usual for GOFAI it doesn't solve the problem.
Deep learning doesn't "solve" that problem either- the point is that you can't learn only from examples.
> I mean, try to define some BK and generate combinations of it until you solve the problem. You say that I made the problem easy because I defined some BK by hand. I suggested you try doing that to see whether the problem is as easy as you think.
I genuinely don't understand what you expect me to do here. How can I possibly define BK for a problem I have zero information about beyond the type signature?
> There are only 1280 clauses, but there are about 2 billion programs
Doesn't matter, since the programmer gives that information. As you admit, Louise cannot check for arbitrary programs.
> Deep learning doesn't "solve" that problem either- the point is that you can't learn only from examples.
Deep neural nets do not learn only from examples! They encode strong inductive
biases in their carefully hand-engineered and hand-tuned architectures, hence
for example CNNs are used for image recognition and LSTMs for sequence
learning etc. Without these biases deep neural nets would not be able to
generalise as well as they do (in the sense of local generalisation but not
global generalisation as meant by François Chollet [1]).
The biggest advances in deep neural nets have come from the discovery and use
of good inductive biases: training with gradient descent, backpropagation,
more hidden layers, the "constant error carousel", convolutional layers, ReLu
over sigmoid, attention, etc, etc. One could say that deep neural nets are all
about good inductive bias.
It's interesting that you bring up the ARC dataset. The paper that introduced
it (also from Chollet) [2] makes a strong claim about the necessity of
"knowledge priors" for a system to be considered intelligent. These are
described at length in section III.1.2 "Core knowledge priors" and are exactly
a set of strong inductive biases that the author of the paper considers
necessary for a machine learning system to solve the ARC tasks and that
consist of such problem-specific biases as object cohesion, object
persistence, object influence via contact, etc. It is exactly such "knowledge
priors" that are encoded as background knowledge in ILP systems.
Indeed, in the ARC challenge on Kaggle, the best-performing systems (i.e. the
ones that solved the most tasks) were crude approximations of the ILP
approach: a library of hand-crafted functions and a brute-force search
procedure to combine them. I note also that attempts to use deep learning to
solve the challenge didn't go anywhere.
Humans also have strong inductive biases that help us solve such problems. But
I'm not the best placed to discuss all this - I'm not a cognitive scientist.
In the end, what you are asking for is magick: a learner that learns only from
examples, without any preconceived notions about how to learn from those
examples, or what to learn from them. There is no such machine learning
system.
>> Doesn't matter, since the programmer gives that information. As you admit,
Louise cannot check for arbitrary programs.
I don't understand what you mean "check for arbitrary programs". I can give
Louise zero BK and metarules and ask it to generate all Prolog programs, say.
Prolog is a Turing complete language so that would give me the set of all
programs computable by a Universal Turing Machine (it would take a while). But
what would that achieve?
At this point I'm not sure I understand what your remaining objections are
against the approach I showed you. For the purpose of learning arbitrary
programs it works better than anything else. Of course it's not magick.
Perhaps you should take my suggestion to think about the problem a bit more
carefully, if you're really intersted in it. Or are you? I mean, if you
consider AI solved, e.g. by GPT-3, then I can see how you wouldn't be
interested in thinking any further about the issue.
P.S. To clarify, I'm keeping this discussion up for your sake, albeit eagerly.
You have expressed some strongly held, but incorrect opinions that it seems to
me you have acquired by consulting inexpert sources, probably because you have
a day job that has nothing to do with AI and doesn't leave you enough time to
study the matter properly. My day job is to study AI and I feel that such a
privilege is only justified if I spend time and effort to help others improve
their knowledge on the subject. I&#x...
My initial response was a fairly kneejerk reaction to the snark. The following is a rewrite. Please don't; if you really think so little of me, rather don't reply than reply unpleasantly.
> Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures
“Solomonoff Induction does not learn only from evidence! It encodes strong inductive biases in its construction and choice of Turing machine...”
but it doesn't matter. Our universe is not a random soup of maximal entropy.
The tasks I am talking about solving are overtly not impossible.
You talk about ML methods like the success of, say, image recognition comes from image-recognition-specific architectures. You mention ‘hand-engineered’ or ‘hand-tuned’. And yet, to throw your snark back at you, if you were up to date with the literature, you would know this is not true.
Consider ViT as an example. The same Transformer, the same minimal inductive biases, work as well for language modelling as for image segmentation as for proof search—the only difference perhaps that ViT works on patches for efficiency, though the paper shows that probably hurts performance in the limit. All it takes is an appropriate quantity of data to learn the appropriate task-specific adaptations the network needs. Heck, even cross-domain works; it's all one architecture, so it's all one inductive bias.
To my mind, this is what it means to learn from examples. There is no way that an architecture designed for language translation could also encode task-specific priors for these different tasks.
For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example.
ML people call that data leakage.
> I don't understand what you mean. Yes, Louise can check for arbitrary programs. I can give it zero BK and metarules and ask it to generate all Prolog programs, say.
Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs.
I see I made you feel bad with my advice to read up. I'm sorry, because that
was not my intention. However, you really do need to take my advise seriously.
You've insisted throughout our conversation that you don't need to read older
machine learning or AI papers because they're not relevant anymore. And yet,
they are. And you do need to read them because without them you will not be
able to understand the recent developments you seem to be intersted in.
Take for instance your example of ViT. This is a transformer, so it's clearly
not an unbiased generaliser that learns only from examples. You say so
yourself: "it's all one inductive bias". Yes, that's how machine learning
works and deep neural nets don't do anything different, neither do they learn
only from examples, as you seemed to suggest in your previous comment (you
replied "That's literally what DL is" to my comment that "you can't learn only
from examples").
But I think you misunderstood my comment about how the biggest advances in
deep neural nets have come from purpose-built architectures. That is not to
say that the same architectures cannot be applied to different domains- but
the state of the art systems are always fine-tuned for specific tasks or
datasets. This hasn't changed recently and it hasn't changed in the last 30
years.
>> For sure, one might call this ‘strong inductive biases’, in that the
program is not random bytes (as a truly bias-free algorithm must be), but
please at least admit that this is a complete different conceptual plane to
the sort of biases you give Louise. Louise's biases aren't merely task
specific, they're problem-specific. It would be one thing if Louise's biases
were a handwritten web of a million BK rules: fine, whatever, as long as it
solves the task that is obviously possible to solve. But they're not, they're
tuned per example.
A truly bias-free algorithm is not "random bytes". It's a learner that
memorises its training examples and can only recognise its training examples.
Hence why it can't generalise. This is in Mitchell's paper which I suggested
you read.
Louise's biases are not problem-specific in the short example I showed
you. I defined BK predicates with wide applicability in programs processing
lists and numbers. There is no such limitation, theoretical or practical, in
the general sense, either. You can give Louise a million irrelevant BK
predicates, if you like, and it will still find the ones it needs to complete
the learning task assuming they're in there somewhere. In fact, it will find
all of the relevant ones - and return the superset of all programs that
solve the task (so you can use it for example to identify interesting
relations in your dataset). Like I say in a previous comment, Louise's
learning algorithm was originally designed to select relevant BK.
Additionally, like I said in an earlier comment, Louise can learn its own
bias, both the BK and the metarules, so it is not only not limited to
task-specific bias, it is not even limited to user-provided bias. Under some
circumstances it can even invent new examples. And then use them to learn a
hypothesis that generalises better to unseen examples.
*
>> Louise can perhaps generate all Prolog programs. Louise cannot search the
space of Prolog programs.
> Louise's biases are not problem-specific in the short example I showed you.
This is clearly untrue.
You were customizing the BK to each specific task. You were also customizing the stepping stones for each specific task.
Justifications can come later. At least admit that you customized the BK for each problem instance and prior to doing so the solver did not solve the problem asked.
Not responding to the rest since you've missed my entire point and I don't feel like rephrasing it.
I did not "customize the BK to each specific task". You can go back and see
what I did. I provided some generic BK predicates that manipulate lists and
numbers, I defined some metarules and I gave a few examples of each program's
inputs and outputs.
I don't understand your criticism and I don't understand what you want me to
"at least admit".
>> This is clearly untrue.
Can you show me which biases in the example I showed are problem-specific?
>> Not responding to the rest since you've missed my entire point and I don't
feel like rephrasing it.
I don't think I missed your point. I think you, yourself, are horribly
confused about what point you are trying to make. And the reason of course is
that you want to be able to express strong opinions about AI and machine
learning, but you don't want to have to do the hard work to understand the
subject. So you keep saying "five impossible things before breakfast", like
asking for a learner that learns only from examples, or saying that's what
deep learning is, etc.
I'm sorry but despite what the article above suggests, there is't an easy way
to being an expert- not even in machine learning. If you want to know what
you're talking about, then you'll have to do your homework.
I don't know why you need to reply unpleasantly. It should be possible to give and receive criticism, even strong criticism, without having it turn into a flamewar just because we're on the internet.
Indeed, you yourself have criticised my work and my field mercilessly in this thread and I did not once reply with unpleasantness. In fact, what you keep dismissing as irrelevant and basically cheating (Louise) is my PhD research. I would be well within reason to be defensive about it. Instead, I believe I have remained polite and respectful towards you throughout and strove to answer all your questions about it.
Although you did take my criticism as snark, so this is perhaps something that is not entirely objective - you might perceive my criticism as a personal attack, say. Again, this should not be the case. In my field of work, criticism is what makes your work better and without criticism one never improves. So I do mean it when I say that my contribution to this thread was for your sake and to help you improve your knowledge of a subject you seem to be interested in.
In any case, I'm sorry this conversation turned sour. I didn't want to make you upset and I apologise for having done so.
> I believe I have remained polite and respectful towards you throughout
I disagree. I do not mind in the slightest being told I am wrong, or having my ideas criticized. But calling me too stupid to understand my own point, or too intellectually lazy to want to understand a subject, or to talk down to me like a child—that is not kosher. This conversation is not worth being attacked, or my day being made unpleasant because you choose not to avoid the impulse to throw insults.
To the other side of things, it might help calm you to know I never much considered what I was saying a criticism of Louise. Louise, from what I can tell, is fine, and an interesting take on the task. What I was objecting to was only the way you used it in the argument. A bike is cheating if you bring it to a 100 metre sprint, but that doesn't mean they serve no purpose. Eg. I do not consider SAT solvers particularly relevant to AI progress, but one can hardly deny they are quality tools.
As far as I can tell, I did not talk down to you as to a child, and I
certainly did not call you intellectually lazy or stupid. I criticised the
fact that you don't want to put in the hard work to understand the subject you
are discussing, which is what you have stated from the start of the
conversation, claiming you don't need to read up on the history of AI because
it is not relevant (I'm paraphrasing your point but correct me if I
misunderstood it).
It seems to me I am right to think that you took my criticism as an insult to
your faculties. If I say something wrong, I expect to be corrected and
criticised if I insist on it, but I don't take that as an insult.
>> To the other side of things, it might help calm you to know I never much
considered what I was saying a criticism of Louise.
And still you persist with the same style of commenting. "Calm" me? And you
complain that I talk down to you? You have replied to my original comment with
arrogance to tell me that my entire field of study is "not AI" and irrelevant
- and then continued to insist you don't need to know anything about the ~70
years of work you dismiss even when it became clear that this only causes you
to make elementary errors. You speak of things you know nothing about with
great conviction and then you get upset with me for pointing out this can only
result in errors and confusion. Given all that, I have shown great patience
and courtesy. Others would have just ignored you as ignorant and unwilling to
learn. I gave you the benefit of the doubt. Was that a mistake?
A poor choice of words, sorry. I meant, I understood you to be saying you found the criticism of Louise unpleasant, and I thought it would lessen that to know that I didn't and don't think Louise was bad.
There's a split happening right now in the market between Data Scientist and Machine Learning Engineer roles. The former is generally a more soft-skills heavy role with results that are meant to drive human decisions. The later is about building systems that automatically make decisions (ie: recommendation systems, etc.). Data Science is over saturated while ML Engineering is less so.
A quick plug and reminder that a learning roadmap for data engineering that is detailed with resources is available here: https://awesomedataengineering.com
I chuckle a bit when they put PCA as being the very last thing you learn, way after dimensionality reduction (PCA is the classic and most "simple" dimensionality reduction technique)
Also, the reality is that very few full on AI experts (and yes, this means Ph.Ds with a lot of publications at top conferences who are also FAANG applied ML engineers) will have more than 70% of these skills. Luckily, this is a field which highly rewards specialization. They don't need the breadth if they have depth in their particular areas.
My experience has been that in the vast majority of applied roles involving AI, there is not enough emphasis on the software engineering skills necessary for productizing the models, leading to situations like baron_harkonnens critique.
Are people really planning their carrier like this? Isn't it more that you either work for some company nearby, or a company you know by some luck or explore the AI/ML topics you or your professor likes and you got stuck? (or you start disliking it and try hard to find something else)?
98 comments
[ 2.0 ms ] story [ 213 ms ] threadIf I were to give a recommendation, it would be this -- pick a topic/project that interests you, follow the classic knowledge-bootstrapping process where you read through papers and (hopefully) have an expert or trained person walk you through the specifics, then get hands-on instantly. Especially with something like fast.ai that values empirical results over hard theory, something understandably popular in the field.
From there I'd recommend branching out, but I'd use a JIT approach. The field isn't necessarily super well-founded at the moment, and while machine learning fundamentals are useful, ultimately it's a waste of time compared to the long-tail benefit of getting immediate empirical results and feedback hands-on.
Just my 2c, YMMV, and anyone is totally welcome to disagree as they wish! :)
Best of luck,
T
This is the vibe I got as well. Which is fair enough (to each his/her own), but I thought I'd mention fast.ai which takes the opposite approach:
> Harvard professor David Perkins, who wrote Making Learning Whole (Jossey-Bass), has much to say about teaching. The basic idea is to teach the whole game. That means that if you're teaching baseball, you first take people to a baseball game or get them to play it. You don't teach them how to wind twine to make a baseball from scratch, the physics of a parabola, or the coefficient of friction of a ball on a bat.
Yeah I get the same feeling. I am hesitant to suggest it, but gives a little bit of a gate-keeping vibe. Like only once you've payed sufficient homage to the same education path that I took, shall you be granted permission to be a 'real data scientist'.
That said, I think there is value in something like this more for hiring managers than practitioners... which makes sense considering the intro comments.
Our perspective is the industrial one. And while there are certainly many complex business problems where deep learning is required, there are more cases where a traditional approach is sufficient and actually the better solution (e.g. due to the memory footprint, latency or other reasons). We routinely work on both kinds of problems on a day to day basis, but we would never go straight to deep learning approaches if simpler and faster traditional methods comprise a better solution in a given use case. So, our employees are expected to know both and to be able to judge when to apply which approach.
In a world of senseless marketing hype, I think it's a good idea to take the high road on this one. Reputation alone, even if less-buzzy words like ML are used in favor of AI, really carries a long ways. Plus, we're nearing the disenfranchisement hump, and AI's going to start taking a negative connotation with many businesses, I believe. Just shoot straight and I firmly believe it'll carry you for a long ways, there.
We should be careful with how we guard our scale of semantic meaning - if somebody with an undergrad understanding of statistics (frequentist statistics only in this map) can be called an Expert in a statistical/mathematical field, what do we even call somebody with all the same applied software engineering & exploratory analysis, and business experience, but also a PhD in theoretical topics (math, stats, etc)? A 'super expert'? What do we call Francis Chollet, or LeCun, or anybody else? What's the differentiation between an expert from the roadmap and the team of individuals deploying GPT-3 into Google Assistant? Are they the same?
As a hiring manager and team lead for a large fintech firm in London, I would happily see an individual who had really mastered the above path(s) as a strong candidate for an intermediate or upper junior role in applied data science/machine learning. But ... it's not enough to be a senior, and certainly not an expert. Just my two cents.
Just because the man who can see is king among blind men doesn't mean they're qualified to call him visionary.
PhDs are one especially crappy way to prove you have the intellectual chops to engage with ML. There are far more direct, practical, and expedient alternative paths to get there.
Importantly, the number of people who were perfectly capable of doing a PhD but chose not to (because, frankly, it's a very bad deal) vastly dwarfs the number of people who stuck around in academia and obtained one. Additionally, my observation at several major tech companies is that PhDs have a bent of mind that is roughly orthogonal to the pursuit of real business value.
Getting a PhD is a fine deal if you have good reasons.
The phrase "bent of mind" roughly implies "the way someone thinks". Its usage is declining I suppose but there's nothing connotatively nefarious there.
Ehhh, French? I like Chollet(many people dont) but he is not in the same league of Yan LeCun at all.
It seems both the author and you have a naive view of expertise in academia.
For work as a tradesperson, it is worth only focusing on the practical application- and I'm saying this genuinely. But I think it should be clearly distinguished from the different kinds of lasting benefits that a more solid fundamental education provides, including the flexibility to adapt.
I'm very biased here, having studies Electrical Eng and CS before modern ML was remotely mainstream, and comparing my conceptual understanding and what I learned in school about math and linear algebra with the way things are understood by tradespeople with far more knowledge of modern tools than me. So crotchety old person- maybe, but I'm happy I went to university.
The explosion of AI/ML/Data Science teams in places that they really don't belong is going to have major backlash soon. The market is currently flooded with desperate PhDs who have reduced years studying their field into a flurry of buzzwords on a resume that they don't understand.
The vast majority of AI/ML/DS teams don't know what they're doing. Management and "leadership" in this industry would be laughable if it wasn't so horrific. The vision of AI/ML/DS has turned into a nightmare of people shoving data they don't understand into models they don't understand and making sure someone else is using that so they can justify their work. You could likely cut entire 100+ DS teams at most companies today and only see a positive impact on the business.
There are of course really interesting niches that require these skills and skilled practitioners. But if you're reading a guide on how to be an "AI Expert", they you likely aren't one of these people. And because of the signal noise problem in this space it's hard for even really excellent people to get matched to the hard problems that genuinely need their help.
> The vision of AI/ML/DS has turned into a nightmare of people shoving data they don't understand into models they don't understand...
Don't be this ^. Please.
If you do want to come into the field, please first take the time to understand the tools. Learn enough to re-derive and re-implement the tools you're using. Learn the assumptions that need to hold. Don't just learn to point the tools at data and hit go. It's not like a lot of programming where it either succeeds or fails. Failures are often silent in this field. If you can't reason through how and when a given tool will succeed and, probably more importantly, reason through how and when a given tool will fail, you haven't learned enough to know what to use and when.
The "right" way is slow, lonely and unglamorous. And most importantly without that fast money, acquisitions and promotions.
I'm not disagreeing with you, just curious, did that happen with Big Data teams? If not, what happened with all the people working around that concept a few years ago? Might be an indicator of what could happen to DS teams.
I think the issue here is you can’t become highly skilled in software engineering overnight, and compensating by flexing softer “problem solving skills” isn’t enough to compensate. If you want to be effective in data science, you need to commit to being a rockstar programmer AND statistician AND be well versed in a number of different algorithms.
Unsurprisingly, this is difficult, and so the market of people who do it well is small. That’s not, however, a referendum on the value of predictive modeling and software engineering. I don’t see advanced math and software engineering skills going out of fashion anytime soon, simply because they are hard to acquire and immensely powerful in the right hands.
In fundamentals, I don't see a reference to geometry or calculus. Good luck understanding methods like UMap or more modern clustering techniques, optimal transport methods in AI, and more recent gradient techniques without those.
Some might say this is more for the AI researcher, but people need to implement bleeding edge stuff all the time.
In fact, since the direct path is more rocky than going via X and Y, it naturally selects the most motivated people, so those succeeding in it will generally outperform those who went X-Y-Z.
... well I don't know what it is, anymore. Sign of the times? Perhaps the field (AI that is) should abandon its name just to be sure that if a backlash ever comes against deep learning it won't take everyone else's reseach with it?
On the other hand, I feel a bit like the horses and the cows in Animal Farm, when the pigs took over the Revolution. Not that there was any revolution, not really, but the way that the research trends have shifted lately, from what would make good science to what will make you hired by Google, is a little bit of a shock to me. And I stared my PhD just three years ago. It's come to the point where I don't want to associate my research with "machine learning" and I don't want to use the term in my thesis, for fear of the negative connotations (of sleazy practices and shoddy research) that I am concerned might be attached to it in the time to come.
And it's such a shame because the people who really advanced the field, people like Joshua Bengio, Jurgen Schmidhuber, Geoff Hinton, Yan Le Cunn etc are formidable scientists, dedicated to their preferred approaches and with the patience to nurture their ideas against all opposition. The field they helped progress so much deserved better.
Edit: I'm asking because there is evident in your comment a confusion that is all too common on the internets today, of thinking of "GOFAI" (symbolic AI) and "ML" (machine learning) as two somehow incompatible and perhaps exclusive sub-fields of AI. This is as far from the truth as it could be, for instance machine learning really took off as a subject of research in the 1980's with the work of Ryszard Michalski and others, who considered machine learning approaches for the purpose of overcoming the "knowledge acquisition bottleneck", i.e. the difficulty of hand-crafting production rules for expert systems. Indeed, most of the early machine learning systems were propositional-logic based, i.e. symbolic. And of course, one of the most widely used and well-known type of machine learning approach used to this day, decision trees, hail from that time and also learn propositional logic, symbolic models.
Of course, most people today know "ML" as a byword for deep learning, or at best statistical pattern recognition (if that). It's just another aberration brought on by the sudden explosion of interest in a once small field.
I refer you to Michalski's textbook, "Machine Learning: An Artificial Intelligence Approach" and Tom Mitchell's "Machine Learning" for more information on the early days of the field, i.e. until ca. 2000.
It's true that you can, in some places, merge the two. The significant majority of the time, this just makes your ML system worse in the long run, per The Bitter Lesson. Occasionally—very occasionally—your problem is fundamentally simple, so something brittle like AlphaZero works, even though we'd still rather shave the fixed-function parts off with MuZero and the like. But it's no coincidence this is reserved for simple, thoughtless problems (like brute-force move search) and kept isolated from the thinking, as intelligence needs generalization and abstraction, and GOFAI doesn't generalize or abstract.
> And of course, one of the most widely used and well-known type of machine learning approach used to this day, decision trees
Though decision trees live a healthy life in data analysis, alongside things like k-means clustering, they're obviously not AI.
> Of course, most people today know "ML" as a byword for deep learning, or at best statistical pattern recognition (if that).
‘Statistical pattern recognition’ is just name calling devoid of real criticisms. I can ask GPT-3,
> a = ["fitness", "health", "heart"], b = ["lifting", "curls", "squats"], c = ["running", "jogging"], so what is b.append("pushups")?
and it'll happily answer
> b.append("pushups") returns ["lifting", "curls", "squats", "pushups"]
or I can give it the Loebner Prize questions which it can almost ace, or I can test it on 10-digit addition and it gets 60% right (with thousands separators, but still suffering BPEs), or I can ask it to do quirky tasks like shuffling letters, or ask it to write sentences with novel words and find that it understands nuanced differences between the definitions...
You can scream ‘but it's just statistical pattern recognition’ all you want, but the criticism doesn't mean anything if ‘statistical pattern recognition’ includes this level of generalization, reasoning and algorithmic sophistication in natural language with scaling curves that assure us the best is yet to come.
Before I continue this conversation, I'd like to ask, do you think this is an appropriate response to my comment? I responded to your original comment respectfully and politely.
Note that I did not use "pattern recognition" in a derogatory manner, neither did I say that anything is "just" pattern recognition. I think you may be mistaking my comment for a different opinion that you disagree with.
Like I say in my previous comment, modern machine learning research started as a discipline that was separate from pattern recognition (and Pattern Recognition was sometimes considered distinct to AI, as a research subject). I quote below from Tom Mitchell's wildly influential paper, "Generalisation as Search" (AI 18, 1982):
"5.2. Statistical pattern recognition
The field of statistical pattern recognition deals with one important subclass of generalization problems. In this subclass, the instances are represented by points in n-space, and the generalizations are represented by decision surfaces in n-space (e.g. hyperplanes, polynomials of specified degree). The matching predicate corresponds to determining whether a given point (instance) lies on one side or another of a given decision surface (generalization). The field of Statistical Pattern Recognition has developed very good generalization methods for particular classes of decision surfaces. Many of these methods are relatively insensitive to errors in the data and some have well understood statistical convergence properties, under certain assumptions about the probability distribution of input instances."
"In contrast to work in Statistical Pattern Recognition, work on the generalization problem within Artificial Intelligence has focused on problems involving a different class of instance and generalization languages. These languages are incompatible with numerically oriented representations that describe objects as feature vectors in n-space. For example, Winston's program [21] for learning descriptions of simple block structures such as arches and towers, represents instance block structures in terms of their component blocks and relationships among these. In this domain the natural representation for instances is a generalized graph rather than a feature vector. (...)
https://www.sciencedirect.com/science/article/abs/pii/000437...
Pattern recognition these days is dominated by neural methods, e.g. CNNs for object classification etc. and so is machine learning so it kind of makes sense that the three terms are used interchangeably, but unfortunately many people are not aware of the historical context of the terms, hence misunderstandings as the one by Veedrac above, regarding my comment that machine learning has become a byword for statistical pattern recognition: it's not because I'm dismissive of statistical pattern recognition, or of the ability of deep learning systems to perform it; it's because the terms have really become interchangeable and I think even among researchers.
In any case, before commenting on a complex subject of research with a long history, my recommendation remains to first become well acquainted with the subject and its history. Otherwise, one runs the risk of appearing confused.
Edit: Note that Mitchell's use of "generalisation" in the excerpt above does not refer to the ability of a model to generalise to test, or unseen, data. Rather, "generalisation" in Mitchell's paper refers to the ability of a system "to take into account a large number of specific observations, then to extract and retain the important common features that characterize classes of these observations."...
> [...]
> In any case, before commenting on a complex subject of research with a long history, my recommendation remains to first become well acquainted with the subject and its history. Otherwise, one runs the risk of appearing confused.
Sorry, I did misunderstand what you were saying.
I do find it somewhat baffling how you're talking as if 1980s terminology is today's terminology, or that everyone should just recognize it as the default way of things. As much as you argue I should adopt the subject's history, I'm saying you should adopt its present.
The point you're making would have held fine two decades ago, but as a description of modern ML, it's like calling a sedan a cart. Even if you don't mean it as a criticism (‘just a cart without a horse’), the world has moved on.
Thank you for your honesty and there's no need to apologise.
Let me explain my point above, if I can. You said that search, planning, expert systems, etc, are "not AI, and they never will be". I understand that as saying that such systems are not artificial intelligences, in the sense of an intelligent machine created by humans out of whole cloth (without trying to define what "intelligent" means).
That is certainly true, but it is also uncontroversial that the above are sub-fields of the field of research that is known as "AI". That is, there is a field that researches approaches that could lead to the creation of intelligent machines and that is called "AI" and then there's the ultimate target of that field which is to create "AI".
My original comment bemoans the fact that in some sectors, "AI", as a field of research, has become synonymous with only one sub-sub-field of AI research, that is, deep learning.
Contrary to what you state, these "GOFAI" fields (symbolic AI, if you will) are still active and far from having "failed" in any way, they are "SOTA" in their respective tasks. For example, the field of automated theorem proving is dominated by systems that employ the resolution rule, a logic-based approach and while recent efforts have been made to make inroads into theorem proving with deep neural nets (e.g. a recent attempt to use transformers) results are still very far from being comparable to the traditional approaches. I know more about automated theorem proving that I know e.g. about planning or search (because my PhD is in a subject close to the former) but my understanding is that in those fields too, traditional techniques dominate- which is why research in them is still active.
If I am permitted to tout my own horn, my broader subject area can be described as "program learning", i.e. machine learning of arbitrary programs from examples of their inputs and outputs. In this area also, deep learning systems are hopelessly outclassed by symbolic approaches, not least because these approaches learn precise representations of target programs (rather than approximations) from a mere handful of examples (four or five, etc).
And so on. These are just some examples of AI research that is ongoing, that is not deep learning, and that is state of the art for its respective tasks. In view of that, I consider that using "AI" to mean "deep learning" (as the article above does) is not only misinformed but also misinforming and actively harmful. In the sense of harming people's understanding, that is.
As to your comment about how GOFAI "failed", I'm afraid this opinion, which is common, is also misinformed. Here, too, a knowledge of the history of the field helps, but to summarise, the last winter happened because of strictly political reasons and for no reason that had anything to do with the scientific merits, or practical successes of the relevant approaches. In fact, expert systems, now widely considered a failure, were one of the first big success stories of AI; a success story that was cut short only because, again, of political reasons.
I could talk about the AI winter subject for hours (it's a favourit subject of mine) but a good starting point is this article by the editor of the IEEE journal Intelligent Systems: (Avoiding Another Winter) https://www.computer.org/csdl/magazine/ex/2008/02/mex2008020.... The wikipedia page on https://en.wikipedia.org/wiki/AI_winter also has a decent summary and some sources. Finally, see the wikipedia article on the Lighthill Report Veedrac ↗ I don't want to dismiss automated provers, as they are often quality, useful tools (SAT solvers in particular), but if you're interested in learning AI, traditional approaches are no longer more than briefly and tangentially relevant. YeGoblynQueenne ↗ I'm happy to see we're still in healthy disagreement. However, I have to
apologise for confusing you by describing my field as "program learning" which
is admittedly vague, but I didn't want to go into the particulars. My field is
not program synthesis, which is constructing programs from complete, formal
specifications. Rather, it's Inductive Programming and more specifically Inductive
Logic Programming (ILP), which is learning programs from examples, i.e.
"incomplete specifications". I'm not familiar with the General Program
Synthesis Benchmark Suite, but the problem you list (test three strings are
ordered by length) is trivial for ILP approaches. Again, I don't want to point
to my own research, directly (I'm going through a bit of a modesty phase) (oh,
alright, it's just that the documentation of my project is crap). However, I
have to say that even so, if something is a difficult problem for program
synthesis approaches, then it's very unlikely that neural networks will do any
better at it. For instance, do you know how well deep neural nets perform on
this benchmark? I can't find any results with a quick search. Veedrac ↗ > Rather, it's Inductive Programming and more specifically Inductive Logic Programming (ILP), which is learning programs from examples, i.e. "incomplete specifications". I'm not familiar with the General Program Synthesis Benchmark Suite, but the problem you list (test three strings are ordered by length) is trivial for ILP approaches. YeGoblynQueenne ↗ In the interest of pruning this conversation a bit I will not continue the
discussion about GPT-3. Apologies, but this thread is growing too fast and I
don't have the time to give your comments the attention they deserve. I am
happy for you to have the last word in that matter. Veedrac ↗ > You can consult wikipedia or e.g. the calls for papers from major AI conferences, AAAI and IJCAI, if in doubt. YeGoblynQueenne ↗ >> Like, my point is not about whether you can find the odd person trying to
solve intelligence with grammars, or what were GOFAI conferences still harbour
GOFAI research in the corners, my point is that a) these approaches don't work
as a way to actually tackle AI, the problem, b) the vast majority of the field
does not take them as seriously as a method of doing so, regardless of other
uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise
without having much care for those parts of the field. Veedrac ↗ Because GOFAI just observably doesn't work. The ideas are brittle, can't generalize and abstract the way is needed, has made very little progress recently (if any) an AI context, and you just _don't see_ anything that would argue otherwise. YeGoblynQueenne ↗ >> If this doesn't answer your question, perhaps answer the opposite; how do you
know that it's wrong? Veedrac ↗ > For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN). YeGoblynQueenne ↗ But none of those sources says that e.g. search or planning are not AI fields. That was your original claim, if I'm not mistaken? Anyway it doesn't matter. It's a very strange thing to say and I was just trying to understand what made you say it- strictly out of curiousity. Veedrac ↗ I best summarized my claim when I said the following. Whether or not it's an ‘AI field’ is not very interesting to me, as long as the following holds. YeGoblynQueenne ↗ Well, a) is not known and b) and c) are not correct. YeGoblynQueenne ↗ Apologies for splitting the thread, but I thought it'd be easier to read this
way. Veedrac ↗ You've just kicked the can down the road; what you've given there cannot solve, for instance, the same problem but with <= instead of <. YeGoblynQueenne ↗ >> Heck, I don't think it even got `shorter` right; it gave
>> which means len(A) + 1 == len(B), not len(A) < len(B), and AFAICT it can't
learn len(A) < len(B), not because the program isn't expressible with the
primitives you gave, but because it just doesn't reason that far. Veedrac ↗ I don't think you've understood my point. At every step of the way you've put in more effort than writing the program and specifying the examples. Clearly this is not trivial. YeGoblynQueenne ↗ Aren't you shifting the goalposts a bit? First you ask me how I'd solve this
problem with ILP: Veedrac ↗ > Aren't you shifting the goalposts a bit? YeGoblynQueenne ↗ >> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting. Veedrac ↗ The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples. [deleted] ↗ (comment deleted) YeGoblynQueenne ↗ Ah, I see what you mean. You mean that I explicitly told Louise how to compose
each program from its sub-programs. In truth, I did not. In the incremental
learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning
targets and a few primitives from which to construct them. I specifically and
very deliberately did not tell it to use each target to define another. It
figured it out on its own. Veedrac ↗ > You mean that I explicitly told Louise how to compose each program from its sub-programs. In truth, I did not. In the incremental learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning targets and a few primitives from which to construct them. I specifically and very deliberately did not tell it to use each target to define another. It figured it out on its own. YeGoblynQueenne ↗ Many apologies for the delay in replying - I missed the "more" link at the
bottom of the thread. And here I was, refreshing the page disappointed that no
more criticism was forthcoming. [deleted] ↗ (comment deleted) YeGoblynQueenne ↗ I can't edit my post but I forgot to calculate the actual size of the
hypothesis space for the multi-predicate problem for ordered/3. That is: p =
5 (tail/2, p/2, s/2 and llength/2 and shorter/2), m = 4, k = 3, n = 3 (the
target theory is a clause for each target). The size of the hypothesis search
space, i.e. the total number of programs of size n = 3 is (pm^(k+1))^n =
2,097,152,000. Veedrac ↗ > "The number of steps the program needs to take" is not relevant to the calculation: we are trying to calculate the complexity of constructing the program by blindly combining a set of building blocks (BK predicates and metarules)- not the complexity of executing the program. YeGoblynQueenne ↗ >> I meant in the sense of denotational semantics, so looping (or rather, finding a fixed point of a loop) is one ‘step’. Veedrac ↗ > I mean, try to define some BK and generate combinations of it until you solve the problem. You say that I made the problem easy because I defined some BK by hand. I suggested you try doing that to see whether the problem is as easy as you think. YeGoblynQueenne ↗ Deep neural nets do not learn only from examples! They encode strong inductive
biases in their carefully hand-engineered and hand-tuned architectures, hence
for example CNNs are used for image recognition and LSTMs for sequence
learning etc. Without these biases deep neural nets would not be able to
generalise as well as they do (in the sense of local generalisation but not
global generalisation as meant by François Chollet [1]). Veedrac ↗ My initial response was a fairly kneejerk reaction to the snark. The following is a rewrite. Please don't; if you really think so little of me, rather don't reply than reply unpleasantly. [deleted] ↗ (comment deleted) YeGoblynQueenne ↗ I see I made you feel bad with my advice to read up. I'm sorry, because that
was not my intention. However, you really do need to take my advise seriously.
You've insisted throughout our conversation that you don't need to read older
machine learning or AI papers because they're not relevant anymore. And yet,
they are. And you do need to read them because without them you will not be
able to understand the recent developments you seem to be intersted in. Veedrac ↗ > Louise's biases are not problem-specific in the short example I showed you. YeGoblynQueenne ↗ I did not "customize the BK to each specific task". You can go back and see
what I did. I provided some generic BK predicates that manipulate lists and
numbers, I defined some metarules and I gave a few examples of each program's
inputs and outputs. Veedrac ↗ As I said, rather don't reply than reply unpleasantly. I'm cutting this here. YeGoblynQueenne ↗ I don't know why you need to reply unpleasantly. It should be possible to give and receive criticism, even strong criticism, without having it turn into a flamewar just because we're on the internet. Veedrac ↗ > I believe I have remained polite and respectful towards you throughout YeGoblynQueenne ↗ As far as I can tell, I did not talk down to you as to a child, and I
certainly did not call you intellectually lazy or stupid. I criticised the
fact that you don't want to put in the hard work to understand the subject you
are discussing, which is what you have stated from the start of the
conversation, claiming you don't need to read up on the history of AI because
it is not relevant (I'm paraphrasing your point but correct me if I
misunderstood it). Veedrac ↗ > "Calm" me?
That's my point, that you don't need to learn woodworking to build a car, even if wooden carts still have occasional uses, and some cars have wood trim or wooden trailers.
> If I am permitted to tout my own horn, my broader subject area can be described as "program learning", i.e. machine learning of arbitrary programs from examples of their inputs and outputs. In this area also, deep learning systems are hopelessly outclassed by symbolic approaches, not least because these approaches learn precise representations of target programs (rather than approximations) from a mere handful of examples (four or five, etc).
I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) < length(n2) < length(n3), and false otherwise”, and that's with 100 examples. So, probably less practically useful than GPT-3, which wasn't even trained on the task.
> the last winter happened because of strictly political reasons and for no reason that had anything to do with the scientific merits, or practical successes of the relevant approaches
I disagree, but you've not given me much more than a list of vague references that don't all exactly support your argument, so I don't have much clue where you diverge.
If GOFAI worked, we'd see some indication of it working (again, in an AI context), but we don't.
> But if you think about it, in the era where the trend is to use deep learning, the most interesting results can only come from not using deep learning. In a gold rush, if everyone is digging up Widow's Creek, then Widow's Creek is the last place to dig for gold.
This analogy doesn't work. Neural networks are giving unparalleled results by the bucket. That's why people are digging there. A gold mine might have plenty of competing miners, but it's sure going to be a lot more likely to give you chunks of gold than a random patch of grass in your backyard.
You make the point that one does not need to learn these "obsolete" AI approaches because they are not relevant anymore. I don't understand why you say that. These approaches are still state of the art for their respective tasks and there is no other approach that has been shown to do any better, including deep neural networks. In what sense are they "no longer more than briefly and tangentially relevant" as you say?
Regarding the gold rush, the point of the analogy is that in a gold rush only a very few people will ever strike gold. This is exactly the state of research into deep learning currently. After a few initial big breakthroughs, like CNNs and LSTMs, progress has stalled and the vast, vast majority of published papers (or papers put on arxiv permanently) present incremental results, if that. Literally thousands of deep learning papers are published each month and the chance to have an impact is miniscule. From my point of view, as a researcher, going into deep learning right now would be career suicide. Not to mention that, while the first few successes were achieved by small academic teams, who had typical academic motives (er, glory), the game has now passed to the hands of big corporate teams that have quite different incentives, so it's almost impossible for small teams or individual researchers to make a dent.
As to the winter and whether GOFAI works, perhaps I haven't convinced you with my sources, but in that case, I have to go back to my earlier question and ask where your knowledge comes from. You clearly have a strong opinion on GOFAI and the AI winter of the '80s, but what knowledge does this opinion come from? Can you say? And if this sounds like a challenge, well, that's because it is. I'm challenging you to re-examine the basis of your convictions, if you like. Because to me, they sound like they are not well-founded and that you should put some water in your wine. The things you say "don't work", work and the things you say work, don't work as well as you say.
For my part, I certainly agree that GPT-3 or the next iteration of a large transformer-built language model can be a useful tool, but such a tool will always be limited by the fact that it's, well, a language model, and it can only do what language models do, which does not include e.g. the ability for reasoning (despite big claims to the contrary) or arithmetic (ditto) or generation of novel programs. For instance, the append() example you show above is clearly memorised: you haven't given the model any examples of append(), so it can't possibly learn its definition from examples. It only returns a correct result because it's seen the results of append() before. Not the same result, but close enough. Like I say, this ability can definitely be useful- ...
The General Program Synthesis Benchmark Suite works from input-output examples, not “complete, formal specifications”.
How would you tackle this with ILP?
> However, I have to say that even so, if something is a difficult problem for program synthesis approaches, then it's very unlikely that neural networks will do any better at it. For instance, do you know how well deep neural nets perform on this benchmark?
I'm not aware of any serious at-scale attempts. Your option is basically to try few-shot with GPT-3.
OTOH, learning these trivial programs from 100 examples is a largely artificial framing used to support a field which hadn't worked its way up to meaningful problems, and in the more general sense, large networks are promising; eg. the GitHub-trained GPT:
https://www.youtube.com/watch?v=y5-wzgIySb4
or any of the GPT-3 programming demos:
https://twitter.com/sharifshameem/status/1284103765218299904 https://twitter.com/sharifshameem/status/1284815412949991425 https://www.reddit.com/r/commandline/comments/jl8jyr/the_nlc...
> These approaches are still state of the art for their respective tasks and there is no other approach that has been shown to do any better, including deep neural networks. In what sense are they "no longer more than briefly and tangentially relevant" as you say?
“if you're interested in learning AI”
These techniques were invented from the field of AI, but that does not mean they remain in the field of AI.
> You clearly have a strong opinion on GOFAI and the AI winter of the '80s, but what knowledge does this opinion come from? Can you say?
I can argue why ML approaches are good and promising and point at that. I can argue why ML approaches make conceptual sense whereas GOFAI does not, though I don't see us resolving that short-term so I'd rather not. But what I can't so easily do is point to the non-existence of GOFAI AI successes. It's just not there.
You do have tools Watson and WolframAlpha which use GOFAI techniques for fact search over a large set of human-built knowledge repositories (trivia q's / math tools), but Watson is mostly considered a stunt, and I'm not aware of anyone calling WolframAlpha AI.
> the ability for reasoning (despite big claims to the contrary)
The nebulousness of the term ‘reasoning’ is pulling a lot of weight here. It's clearly doing sophisticated computations of some sort, beyond brute memorization.
> or arithmetic (ditto)
http://gptprompts.wikidot.com/logic:math#toc6
There are more examples too, this is just addressing the one point people get wrong most often. BPEs are an interim performance hack, not an indictment on the approach in general.
> or generation of novel programs
Is clearly false.
> For instance, the append() example you show above is clearly memorised: you haven't given the model any examples of append(), so it can't possibly learn its def...
>> These techniques were invented from the field of AI, but that does not mean they remain in the field of AI.
Like I say above, it is pretty uncontroversial that these approaches are part of the field of AI research. You can consult wikipedia or e.g. the calls for papers from major AI conferences, AAAI and IJCAI, if in doubt.
So I have to ask again, why do you say these approaches are are not in the field of AI research? According to whom? And based on what?
I would please like an answer to the above question.
Further, I can certainly point you to successes of symbolic AI, which you say don't exist. For one thing, the entire fields of automated theorem proving, planning, search, game playing, knowledge representation and reasoning, etc. that you say are "not AI", but are like I say still active and still state of the art in their respective tasks. These are certainly successful- they have produced systems and techniques that still work best than any alternative and actually quite well.
For examples of specific systems that were successful in their time, see Logic Theorist [1] that proved 38 of the first 52 theorems in Principia Mathematica; Chinook [2], the first computer program to win a world championship against humans (in checkers/draughts); Deep Blue [3], the first AI system to defeat a human grandmaster (Garry Kasparov) in chess; MYCIN [4] the first AI system to outperform human experts in disease diagnosis (specifically, diagnosis of infections); and so on.
Of course these systems have been superseded - but they were successes nonetheless. Another reason to learn the history of AI is to become aware of those systems- they, indeed, were "there".
Again I have to ask you- where does your knowledge of AI come from? When you make such strong statements about what works and what doesn't, what failed and what succeeded, are you sure you are well informed? Do you draw your knowledge from primary sources, or are you trusting the opinions of others who claim to be experts- but may not be (like in the article above)?
>> How would you tackle this with ILP?
Below I've defined the problem in the format expected by Louise [5]:
Given this problem definition, Louise can learn the following (Prolog) program: To explain, shorter/2 is a predicate defined as background knowledge by me. triadic_chain is a metarule, a second-order clause that provides inductive bias. length/2 is an ISO Prolog predicate.Like I say, this is a trivial problem, not least because its solution is easy to figure out and the background knowledge and metarules are trivial to define by hand. Louise can also perform predicate invention to define new background knowledge (kind of like inventing new features) and also new metarules. That is to say, Louise can learn the shorter/2 and length/2 programs, also from very few examples- and then reuse them as background knowledge. But showing how to do that would make for a larger example. I'm happy to oblige if you are curious.
I should point out that there exists no neural net approach that can learn the same (or a similar) program from a single positive example- not least because neural nets cannot make use of background knowledge (i....
As to Wikipedia, see the second paragraph. The sections where it mentions, eg., symbolic or sub-symbolic approaches are prefixed with “Researchers in the 1960s and the 1970s” or “By the 1980s”. Kind'a telling.
Like, my point is not about whether you can find the odd person trying to solve intelligence with grammars, or what were GOFAI conferences still harbour GOFAI research in the corners, my point is that a) these approaches don't work as a way to actually tackle AI, the problem, b) the vast majority of the field does not take them as seriously as a method of doing so, regardless of other uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise without having much care for those parts of the field.
> Further, I can certainly point you to successes of symbolic AI, which you say don't exist. For one thing, the entire fields of automated theorem proving, planning, search, game playing, knowledge representation and reasoning, etc. that you say are "not AI", but are like I say still active and still state of the art in their respective tasks.
Yes, but there's a reason I suffixed that comment with “(again, in an AI context)”. GOFAI is great if you ignore the last two letters of the name, and how it failed almost all its major promises.
These used to be considered AI because it was thought that you could build a useful reasoning agent out of a combination of these techniques, given appropriate developments. Now (almost) nobody does that; Google Map's pathfinding is just a pathfinder, not a general reasoner.
> Below I've defined the problem in the format expected by Louise
Right, OK, I figured it'd be something like this, when you said it was trivial, but this is just another perspective on my original criticism. You wrote the program you wanted it to generate as the background knowledge.
It must be so, because your examples don't specify, even roughly approximately, the program you wanted generated. Another valid solution would be (among many)
and the only reason it didn't choose this is because you gave it the program you wanted it to generate (obfuscated a little, yet still, there was only one application available). It didn't ‘learn’ anything.> I should point out that there exists no neural net approach that can learn the same (or a similar) program from a single positive example- not least because neural nets cannot make use of background knowledge (i.e. a library of programs from which to build other programs).
This is not true, but as you wanted to prune, I'll leave it there.
>> Yes, but there's a reason I suffixed that comment with “(again, in an AI context)”. GOFAI is great if you ignore the last two letters of the name, and how it failed almost all its major promises.
>> These used to be considered AI because it was thought that you could build a useful reasoning agent out of a combination of these techniques, given appropriate developments. Now (almost) nobody does that; Google Map's pathfinding is just a pathfinder, not a general reasoner.
I keep asking- how do you know these things to be true? Are they just your opinion?
Can you please satisfy my curiousity on this?
In contrast, ML methods do work, observably and clearly, and they work in a ridiculously general way, to a degree larger than almost anyone thought (or even thinks) is reasonable for them to.
And it's not just my opinion; there's a reason AI conference attendance has shot up a factor of 10 or so in the last few years, why NeurIPS is the leading one (and even historically GOFAI conferences are majority NNs), why the big AI labs with big AI cash are all doing NNs, and why all of a sudden AI is a popular topic outside academia.
If this doesn't answer your question, perhaps answer the opposite; how do you know that it's wrong?
I know the literature. It's my job.
>> And it's not just my opinion; there's a reason AI conference attendance has shot up a factor of 10 or so in the last few years, why NeurIPS is the leading one (and even historically GOFAI conferences are majority NNs), why the big AI labs with big AI cash are all doing NNs, and why all of a sudden AI is a popular topic outside academia.
That's still an opinion- "it's not just my opinion, everyone says so". A.k.a. "It is known", in Dothraki. And of course it is of no consequence who's spending money on what and who's submitting papers where. The volume of research was never a criterion for its quality. Heed thee well the legend of our Lord Geoff Hinton's years in the academic wilderness and how he emerged victorious with the laws of deep learning in his hands.
I think what you've said so far has convinced me you're expressing a personal opinion that is strongly held without a good reason to do so. You make sweeping statements with great certainty, but you don't really seem to know how you know the things you know, so you end up "knowing" some things that you don't really know. For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN).
You also claim that "these approaches are not AI". That's a "No True Scottsman" right there. Except there really is no True Scottsman (i.e. "AI" in the sense you use it)- ask Yoshua Bengio:
Bengio: In terms of how much progress we’ve made in this work over the last two decades: I don’t think we’re anywhere close today to the level of intelligence of a two-year-old child. But maybe we have algorithms that are equivalent to lower animals, for perception. And we’re gradually climbing this ladder in terms of tools that allow an entity to explore its environment.
Spectrum: Will any of these ideas be used in the real world anytime soon?
Bengio: No. This is all very basic research using toy problems. That’s fine, that’s where we’re at. We can debug these ideas, move on to new hypotheses. This is not ready for industry tomorrow morning.
https://spectrum.ieee.org/tech-talk/artificial-intelligence/...
Or, you know, ask any AI researcher :)
Edit: Which "GOFAI" conferences are majority NNs? What period are we talking about?
At this point I think we're just hopelessly talking past each other. Of course I know about Deep Blue. I didn't know about MYCIN, but, like, “MYCIN was never actually used in practice”, so I don't feel particularly bad about missing that one.
But neither of those challenge my point. If you want to go back in time 30 years, then sure, if you want to be an AI expert, then you have to know GOFAI. That's what the ‘OF’ stands for.
> I know the literature. It's my job.
Yah I read the literature too. (Albeit it seems a very different subset.) That's not an argument though.
> Or, you know, ask any AI researcher :)
OpenAI is explicitly about the path to AGI, https://openai.com/about/.
DeepMind was also founded to tackle AGI (no source, sorry).
Geoffrey Hinton thinks NNs will get to AGI https://www.technologyreview.com/2020/11/03/1011616/ai-godfa....
Even in your own link, Yoshua Bengio is saying that this is a path to AGI, it's just not there yet.
> Which "GOFAI" conferences are majority NNs?
I said “historically GOFAI conferences”, so eg. AAAI.
I too can quote Hinton -from memory and without a link. I remember him saying that the next big thing in AI will come from a grad student who distrusts everything he (Hinton) has ever said. Unfortunately, I won't be that grad student- I haven't heard everything that Hinton has ever said.
---
Like, my point is not about whether you can find the odd person trying to solve intelligence with grammars, or what were GOFAI conferences still harbour GOFAI research in the corners, my point is that a) these approaches don't work as a way to actually tackle AI, the problem, b) the vast majority of the field does not take them as seriously as a method of doing so, regardless of other uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise without having much care for those parts of the field.
This comment addresses your concerns about me writing the program I wanted Louise to generate. I like to see background knowledge ("BK", e.g. shorter/2) as a library of sub-programs from which the learner can select the ones necessary to compose a target program. The example above is trivial because I've defined a BK predicate that is necessary and sufficient to learn, so the learner was indeed served the solution "on a plate".
However, as I said in my previous comment, Louise can learn its own background knowledge. This can be done by predicate invention, or more simply, by incrementally learning necessary sub-programs.
Below is a problem definition and learning session that first learns length/2 (renamed llength/2 to avoid name clashes with the built-in) and shorter/2 from list and numeric function primitives, before using the learned predicates as BK for ordered/2. Like I say in my previous comment, it's a little larger than the previous one:
The BK for this problem consists of tail/2, similar to "car" in Lisp (i.e. matches the head of a list) and the pair of p/2 and s/2, that act as "dereferencers" to Peano number functions. These are bog-standard Prolog programs and useful whenever a target program must manipulate a list, or perform numerical reasoning. In other words, they're pretty much generic, like a standard library of sorts.I've added the full source of the experiment file for the learning task on pastebin. It includes a few more detailed comments and a set of constraints to clean up the learned hypothesis, mostly for aesthetic reasons:
https://pastebin.com/nPYFKKpx
Of course this is still a toy problem and we know the solution. But I hope it demonstrates the principle. On the other hand, you'd still not be able to solve this with alternative approaches, e.g. I see that the benchmark suite you pointed to is used for genetic programming. I'm also not aware of neural approaches that build programs incrementally, from a couple of examples of each sub-program.
That is to say, this is a toy problem for ILP. For other approaches it's unsolvable.
I checked by running with
and just got Heck, I don't think it even got `shorter` right; it gave which means len(A) + 1 == len(B), not len(A) < len(B), and AFAICT it can't learn len(A) < len(B), not because the program isn't expressible with the primitives you gave, but because it just doesn't reason that far.So again, it's only trivial because it isn't learning the program, it's learning to put fit the puzzle pieces of the program together, after you wrote the program and then chopped it up.
Oops. Haha well spotted @^_^
This is correct for </2:
Paste that in a file and consult it to test it: Regarding reasoning "that far" Louise can learn the complete successor / predecessor relation (</2 and >/2) on its own and only from the primitives s(N,s(N)) and p(s(N), N): However, in the ordered/3 problem I define p/2 and s/2 by hand so that I can put in ground_peano/1 to avoid infinite recursion when Louise tries to pass two lists to s/2 or p/2 (at that point, their termination conditions never obtain, so they keep recursing).You can chalk the potential for infinite recursion up as a limitation, you're very welcome- but there are techniques to avoid this and guarantee termination (Knuth-Bendix ordering of the Herbrand base, see ref [1]) which I haven't come round to implementing yet (because they are not necessary given a bit of common sense in defining BK, as above). On the other hand that's actually a feature, in the sense that earlier systems required more specific language bias than the metarules, that would avoid this kind of type-unsafety, but also demanded more expert knowledge from the user. In any case, there's outs.
>> You've just kicked the can down the road; what you've given there cannot solve, for instance, the same problem but with <= instead of <.
That's a different problem. Off we go:
If you're actually inferring programs and not just doing a sort of guided line-by-line generalization of a program you had already written, the only things you would need would be
I was never in doubt that it was possible to write a different program which would ‘solve’ <=. My criticism was that “it's a different problem”.“Given three strings n1, n2, and n3, return true if length(n1) < length(n2) < length(n3), and false otherwise” (a)
I say this is a trivial problem to solve with ILP and I show you a, well, trivial solution and you complain that it's - trivial.
Then I show you a more elaborate version that learns sub-programs on the way to the full solution and you say that a) it doesn't solve a different problem, with ≤ instead of < and b) that it's not trivial anymore.
And now you're saying you want a solution that learns from examples only. You would have saved us both a lot of time had you clarified your expectations up front.
No matter. There isn't anything that can do what you ask. Or rather, there are many approaches that could learn (a) just from examples, with a brute-force search. But there is no approach that could learn arbitrary programs only from examples. The reason is that the space of all programs that can be computed by a Universal Turing Machine ("arbitrary") is infinite and any learner trying to find one of them blindly, without some kind of hint to guide it, would be lost for ever inside it.
Most machine learning approaches that learn programs from examples adopt some sort of inductive bias to guide a search for a program that satisfies some set of goodness criteria, including neural approaches [1]. In ILP, inductive bias consists primarily of BK and language bias (like the metarules in Louise). ILP has a certain advantage in this, in that the languages of examples, bias and hypotheses are the same (some first order logic language, like Prolog or ASP) and so ILP systems can learn their own bias, like Louise can learn its own BK and metarules. By way of comparison, neural nets, with their hand-crafted architectures, minutely fine-tuned to specific domains or even particular datasets, cannot do that (e.g. a trained model can't be used as a feature to another neural net, in the way that ILP hypotheses can be used as BK). Of course you need to start somewhere, from obvious primitives like head/2, tail/2, s/2 and p/2 that I used above.
But I digress. The bottom line is that learning arbitrary programs from examples is a hard problem for any machine learning approach [2]. Classification is a piece of cake, by comparison. And that is why there has been so little progress in this problem even after decades of research [3].
The take home message of course is that neural nets are not the end of the line in AI research and it would be disastrous for the progress of the field to allow research into neural nets to eclipse every other approach. If this happens it will all have to be discovered again, from scratch. And in another 70 years.
____________________
[1] e.g. see Learning explanatory rules from noisy data https://arxiv.org/abs/1711.04574 by DeepMind, which also uses metarules.
[2] See for example:
Deep Learning for Program Synthesis
Synthesizing a program from a specification has been a long-standing challenge.
(...)
This problem is extremely challenging, and the complexity of the synthesized programs by existing approaches is still limited.
https://sunblaze-ucb.github.io/program-synthesis/index.html
[3] This is where I'd normally say that there have been recent breakthroughs that promise to overturn years of slow progress, but that's a story for another time (and another venue most like).
No, I don't think so. My original comment was
“I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) < length(n2) < length(n3), and false otherwise”, and that's with 100 examples. So, probably less practically useful than GPT-3, which wasn't even trained on the task.”
I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting.
> The reason is that the space of all programs that can be computed by a Universal Turing Machine ("arbitrary") is infinite and any learner trying to find one of them blindly, without some kind of hint to guide it, would be lost for ever inside it.
I'm not looking to solve arbitrary programs. I'm looking, at least at this first hurdle, to solve a few largely trivial ones.
If you saw 100 examples, you would be able to solve this problem, or at least get very close. Therefore this sort of theoretical argument cannot hold.
> By way of comparison, neural nets, with their hand-crafted architectures, minutely fine-tuned to specific domains or even particular datasets, cannot do that (e.g. a trained model can't be used as a feature to another neural net, in the way that ILP hypotheses can be used as BK).
I assume you've seen GPT-f? (https://arxiv.org/abs/2009.03393)
Section 4.7.1 shows this is entirely feasible for neural networks, though the technique is very different.
Though I prefer even more the elegance (and general absurdity) of learning the bias by pretraining on WebMath (GitHub, arXiv Math, Math StackExchange).
> The bottom line is that learning arbitrary programs from examples is a hard problem
I agree it's hard, but in my mind this sentence completes to “and therefore our only hope is to use most powerful tool we have available, neural networks.”
I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem where you talking about, if not that?
What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.
For instance, I did not direct Louise to define shorter/2 by means of length/2. In order to do this I would have to specify length/2 as BK for shorter/2, but I didn't. Note the BK declarations in the experiment file I put on pastebin:
Remember how in an erarlier comment I pointed out that the definition of shorter/2 changed to use leq/2, instead of s/2, when the examples of ordered_leq/3 where in the BK? Again, Louise figured that on its own.In fact, this ability of Louise (actually, its learning procedure, Top Program Construction, or TPC) is kind of native, in the sense that TPC was originally conceived as an algorithm to select relevant background knowledge for a different learning system ("Thelma" for "Theory Learning Machine"; hence "Louise"). So it can figure out what BK it needs on its own. Automatic selection of relevant background knowledge in ILP was the original subject of my PhD research. Interestingly, it turns out that once we had a system that could perform this task, it could also learn its own programs.
No, I get this, it's just not really more than a surface level pretense of choice. The hard thing about generating programs is that there are many possible programs; something like m^n, where ’m’ is the number of functions you have available to use (say, ~1000) and `n` is the number of steps the program needs to take (say, ~5 in this case), and there's another factor for where to put the parameters which here is low enough to be mostly negligible. It turns out even 1000^5 is really big, so this problem is hard if you don't do it smart.
The ‘choice’ you are offering Louise is something like, idk, m=5, n=2. 5^2 is not big. You argue about the terminology, but whatever you call it, it's still exponentially less interesting than the problem first posed. All the interesting work has been done for it, by you.
With the way you first laid out the question, there's a good chance (>1%) I could have gotten the answer mostly right (up to parameter order) without looking at the examples, just the background knowledge and the target type.
>> The hard thing about generating programs is that there are many possible programs; something like m^n, where ’m’ is the number of functions you have available to use (say, ~1000) and `n` is the number of steps the program needs to take (say, ~5 in this case), and there's another factor for where to put the parameters which here is low enough to be mostly negligible. It turns out even 1000^5 is really big, so this problem is hard if you don't do it smart.
Indeed, the complexity of the raw, combinatorial problem is the greatest hurdle in solving it in the general sense, however this time complexity is calculated somewhat differently than in your comment. Let me show you.
First, in terms of ILP, the "number of functions you have available to use" is the number of predicate symbols defined in the BK, which I'll notate as p. "Where to put the parameters" is the number of body literals (similar to function calls) in each metarule, which I'll notate as k. I'll notate the number of metarules as m.
"The number of steps the program needs to take" is not relevant to the calculation: we are trying to calculate the complexity of constructing the program by blindly combining a set of building blocks (BK predicates and metarules)- not the complexity of executing the program. What is relevant is the size of the target theory, i.e. its number of clauses (program lines), because of course a larger program means a larger number of combinations of our building blocks. I'll notate the size of the target theory as n.
Putting it all together, the time complexity of constructing a program of n clauses from p predicate symbols with m metarules with at most k body literals (of any arity) is O(pmᵏ⁺¹)ⁿ [1]. This is an exponential time complexity that corresponds to the size of the search space for programs that can be constructed from these components, i.e. that's the number of constructible programs. The time complexity of the problem is such that even n = 5 is sufficient to completely bog down a powerful modern computer.
Louise can manage it because it doesn't conduct a search of that space, instead it only constructs a unique object in that space, the Top program, that can be constructed in polynomial time O(pmᵏ⁺¹) [2], i.e. the number of constructible clauses. Indeed, Louise is capable of learning large programs, of a few thousand clauses in a few minutes. In other words, the problem is manageable because of the advances encapsulated by Louise's learning procedure, Top Program Construction, not because the problem is trivial, as you portray it - and not because I'm leading Louise by the hand, as you suggest. Even if I was leading Louise by the hand, the combinatorial space of constructible programs would still grow exponentially.
Regarding learning "only from examples" as I understand you to mean it, there is some literature on that, of the kind you say is not "AI" (i.e. it predates 2012's deep learning boom). To my knowledge, this was first discussed in the following:
As I say above, Louise's TPC procedure avoids searching this space and so effectively ignores the exponential term, reducing the complexity of the problem to that of, in the worst case, enumerating (pm^(k+1)) = 1,280 clauses.
In other words, the problem is easy for Louise, not because the problem itself is trivial, but because Louise's learning procedure, TPC, is efficient.
For a further example, in the upcoming publication I mention above, Louise is shown to learn a 2,567 clause theory in under 220 seconds with perfect accuracy after training on 20% of all training examples. The hypothesis space for this problem (grid world navigation) is in the order of 2*10^4944 but Louise shrinks it to the problem of enumerating, at worst, a little over 81 million clauses.
I meant in the sense of denotational semantics, so looping (or rather, finding a fixed point of a loop) is one ‘step’.
> Have you tried doing that? I suggest you do- if only to get a feel for the true difficulty of the problem.
How exactly would you suggest? By the nature of my criticism, I can't construct the Louise BK blind.
If anything you agree here; you say there are only 1,280 clauses it could construct, so I only need discriminatory power of 1-in-13 to have a >1% chance of getting the answer right.
> Regarding learning "only from examples" as I understand you to mean it, there is some literature on that, of the kind you say is not "AI" (i.e. it predates 2012's deep learning boom).
Yes, and as usual for GOFAI it doesn't solve the problem.
I understand, but we don't need to take that term into account. We're only interested in the cost of a blind combinatorial search. Even if we added that term in, we'd just get a higher complexity.
>> How exactly would you suggest? By the nature of my criticism, I can't construct the Louise BK blind.
I mean, try to define some BK and generate combinations of it until you solve the problem. You say that I made the problem easy because I defined some BK by hand. I suggested you try doing that to see whether the problem is as easy as you think.
>> If anything you agree here; you say there are only 1,280 clauses it could construct, so I only need discriminatory power of 1-in-13 to have a >1% chance of getting the answer right.
There are only 1280 clauses, but there are about 2 billion programs of size n = 3 that can be constructed with those clauses, one of which is the target program. The challenge is to find the target program in that 2 billion.
The advantage of Louise is that it only needs to look at the 1280 clauses, not the 2 billion programs. The trick is to find how to do that. It's like the difference between sorting a list with bubblesort and quicksort. Quicksort has to do a lot less work, but the trick is figuring out quicksort.
>> Yes, and as usual for GOFAI it doesn't solve the problem.
Deep learning doesn't "solve" that problem either- the point is that you can't learn only from examples.
I genuinely don't understand what you expect me to do here. How can I possibly define BK for a problem I have zero information about beyond the type signature?
> There are only 1280 clauses, but there are about 2 billion programs
Doesn't matter, since the programmer gives that information. As you admit, Louise cannot check for arbitrary programs.
> Deep learning doesn't "solve" that problem either- the point is that you can't learn only from examples.
That's literally what DL is.
You can also learn programs from examples. People do it all the time. What else would you call the Abstraction and Reasoning Challenge, https://www.kaggle.com/boliu0/visualizing-all-task-pairs-wit...?
The biggest advances in deep neural nets have come from the discovery and use of good inductive biases: training with gradient descent, backpropagation, more hidden layers, the "constant error carousel", convolutional layers, ReLu over sigmoid, attention, etc, etc. One could say that deep neural nets are all about good inductive bias.
It's interesting that you bring up the ARC dataset. The paper that introduced it (also from Chollet) [2] makes a strong claim about the necessity of "knowledge priors" for a system to be considered intelligent. These are described at length in section III.1.2 "Core knowledge priors" and are exactly a set of strong inductive biases that the author of the paper considers necessary for a machine learning system to solve the ARC tasks and that consist of such problem-specific biases as object cohesion, object persistence, object influence via contact, etc. It is exactly such "knowledge priors" that are encoded as background knowledge in ILP systems.
Indeed, in the ARC challenge on Kaggle, the best-performing systems (i.e. the ones that solved the most tasks) were crude approximations of the ILP approach: a library of hand-crafted functions and a brute-force search procedure to combine them. I note also that attempts to use deep learning to solve the challenge didn't go anywhere.
Humans also have strong inductive biases that help us solve such problems. But I'm not the best placed to discuss all this - I'm not a cognitive scientist.
In the end, what you are asking for is magick: a learner that learns only from examples, without any preconceived notions about how to learn from those examples, or what to learn from them. There is no such machine learning system.
>> Doesn't matter, since the programmer gives that information. As you admit, Louise cannot check for arbitrary programs.
I don't understand what you mean "check for arbitrary programs". I can give Louise zero BK and metarules and ask it to generate all Prolog programs, say. Prolog is a Turing complete language so that would give me the set of all programs computable by a Universal Turing Machine (it would take a while). But what would that achieve?
At this point I'm not sure I understand what your remaining objections are against the approach I showed you. For the purpose of learning arbitrary programs it works better than anything else. Of course it's not magick. Perhaps you should take my suggestion to think about the problem a bit more carefully, if you're really intersted in it. Or are you? I mean, if you consider AI solved, e.g. by GPT-3, then I can see how you wouldn't be interested in thinking any further about the issue.
_________________
[1] https://blog.keras.io/the-limitations-of-deep-learning.html
[2] https://arxiv.org/abs/1911.01547
P.S. To clarify, I'm keeping this discussion up for your sake, albeit eagerly. You have expressed some strongly held, but incorrect opinions that it seems to me you have acquired by consulting inexpert sources, probably because you have a day job that has nothing to do with AI and doesn't leave you enough time to study the matter properly. My day job is to study AI and I feel that such a privilege is only justified if I spend time and effort to help others improve their knowledge on the subject. I&#x...
> Deep neural nets do not learn only from examples! They encode strong inductive biases in their carefully hand-engineered and hand-tuned architectures
“Solomonoff Induction does not learn only from evidence! It encodes strong inductive biases in its construction and choice of Turing machine...”
but it doesn't matter. Our universe is not a random soup of maximal entropy.
The tasks I am talking about solving are overtly not impossible.
You talk about ML methods like the success of, say, image recognition comes from image-recognition-specific architectures. You mention ‘hand-engineered’ or ‘hand-tuned’. And yet, to throw your snark back at you, if you were up to date with the literature, you would know this is not true.
Consider ViT as an example. The same Transformer, the same minimal inductive biases, work as well for language modelling as for image segmentation as for proof search—the only difference perhaps that ViT works on patches for efficiency, though the paper shows that probably hurts performance in the limit. All it takes is an appropriate quantity of data to learn the appropriate task-specific adaptations the network needs. Heck, even cross-domain works; it's all one architecture, so it's all one inductive bias.
To my mind, this is what it means to learn from examples. There is no way that an architecture designed for language translation could also encode task-specific priors for these different tasks.
For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example.
ML people call that data leakage.
> I don't understand what you mean. Yes, Louise can check for arbitrary programs. I can give it zero BK and metarules and ask it to generate all Prolog programs, say.
Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs.
Take for instance your example of ViT. This is a transformer, so it's clearly not an unbiased generaliser that learns only from examples. You say so yourself: "it's all one inductive bias". Yes, that's how machine learning works and deep neural nets don't do anything different, neither do they learn only from examples, as you seemed to suggest in your previous comment (you replied "That's literally what DL is" to my comment that "you can't learn only from examples").
But I think you misunderstood my comment about how the biggest advances in deep neural nets have come from purpose-built architectures. That is not to say that the same architectures cannot be applied to different domains- but the state of the art systems are always fine-tuned for specific tasks or datasets. This hasn't changed recently and it hasn't changed in the last 30 years.
>> For sure, one might call this ‘strong inductive biases’, in that the program is not random bytes (as a truly bias-free algorithm must be), but please at least admit that this is a complete different conceptual plane to the sort of biases you give Louise. Louise's biases aren't merely task specific, they're problem-specific. It would be one thing if Louise's biases were a handwritten web of a million BK rules: fine, whatever, as long as it solves the task that is obviously possible to solve. But they're not, they're tuned per example.
A truly bias-free algorithm is not "random bytes". It's a learner that memorises its training examples and can only recognise its training examples. Hence why it can't generalise. This is in Mitchell's paper which I suggested you read.
Louise's biases are not problem-specific in the short example I showed you. I defined BK predicates with wide applicability in programs processing lists and numbers. There is no such limitation, theoretical or practical, in the general sense, either. You can give Louise a million irrelevant BK predicates, if you like, and it will still find the ones it needs to complete the learning task assuming they're in there somewhere. In fact, it will find all of the relevant ones - and return the superset of all programs that solve the task (so you can use it for example to identify interesting relations in your dataset). Like I say in a previous comment, Louise's learning algorithm was originally designed to select relevant BK. Additionally, like I said in an earlier comment, Louise can learn its own bias, both the BK and the metarules, so it is not only not limited to task-specific bias, it is not even limited to user-provided bias. Under some circumstances it can even invent new examples. And then use them to learn a hypothesis that generalises better to unseen examples. *
>> Louise can perhaps generate all Prolog programs. Louise cannot search the space of Prolog programs.
I don't follow. What do you mean?
This is clearly untrue.
You were customizing the BK to each specific task. You were also customizing the stepping stones for each specific task.
Justifications can come later. At least admit that you customized the BK for each problem instance and prior to doing so the solver did not solve the problem asked.
Not responding to the rest since you've missed my entire point and I don't feel like rephrasing it.
I don't understand your criticism and I don't understand what you want me to "at least admit".
>> This is clearly untrue.
Can you show me which biases in the example I showed are problem-specific?
>> Not responding to the rest since you've missed my entire point and I don't feel like rephrasing it.
I don't think I missed your point. I think you, yourself, are horribly confused about what point you are trying to make. And the reason of course is that you want to be able to express strong opinions about AI and machine learning, but you don't want to have to do the hard work to understand the subject. So you keep saying "five impossible things before breakfast", like asking for a learner that learns only from examples, or saying that's what deep learning is, etc.
I'm sorry but despite what the article above suggests, there is't an easy way to being an expert- not even in machine learning. If you want to know what you're talking about, then you'll have to do your homework.
Indeed, you yourself have criticised my work and my field mercilessly in this thread and I did not once reply with unpleasantness. In fact, what you keep dismissing as irrelevant and basically cheating (Louise) is my PhD research. I would be well within reason to be defensive about it. Instead, I believe I have remained polite and respectful towards you throughout and strove to answer all your questions about it.
Although you did take my criticism as snark, so this is perhaps something that is not entirely objective - you might perceive my criticism as a personal attack, say. Again, this should not be the case. In my field of work, criticism is what makes your work better and without criticism one never improves. So I do mean it when I say that my contribution to this thread was for your sake and to help you improve your knowledge of a subject you seem to be interested in.
In any case, I'm sorry this conversation turned sour. I didn't want to make you upset and I apologise for having done so.
I disagree. I do not mind in the slightest being told I am wrong, or having my ideas criticized. But calling me too stupid to understand my own point, or too intellectually lazy to want to understand a subject, or to talk down to me like a child—that is not kosher. This conversation is not worth being attacked, or my day being made unpleasant because you choose not to avoid the impulse to throw insults.
To the other side of things, it might help calm you to know I never much considered what I was saying a criticism of Louise. Louise, from what I can tell, is fine, and an interesting take on the task. What I was objecting to was only the way you used it in the argument. A bike is cheating if you bring it to a 100 metre sprint, but that doesn't mean they serve no purpose. Eg. I do not consider SAT solvers particularly relevant to AI progress, but one can hardly deny they are quality tools.
It seems to me I am right to think that you took my criticism as an insult to your faculties. If I say something wrong, I expect to be corrected and criticised if I insist on it, but I don't take that as an insult.
>> To the other side of things, it might help calm you to know I never much considered what I was saying a criticism of Louise.
And still you persist with the same style of commenting. "Calm" me? And you complain that I talk down to you? You have replied to my original comment with arrogance to tell me that my entire field of study is "not AI" and irrelevant - and then continued to insist you don't need to know anything about the ~70 years of work you dismiss even when it became clear that this only causes you to make elementary errors. You speak of things you know nothing about with great conviction and then you get upset with me for pointing out this can only result in errors and confusion. Given all that, I have shown great patience and courtesy. Others would have just ignored you as ignorant and unwilling to learn. I gave you the benefit of the doubt. Was that a mistake?
A poor choice of words, sorry. I meant, I understood you to be saying you found the criticism of Louise unpleasant, and I thought it would lessen that to know that I didn't and don't think Louise was bad.
A quick plug and reminder that a learning roadmap for data engineering that is detailed with resources is available here: https://awesomedataengineering.com
For those who wish to focus on data engineering.
Also, the reality is that very few full on AI experts (and yes, this means Ph.Ds with a lot of publications at top conferences who are also FAANG applied ML engineers) will have more than 70% of these skills. Luckily, this is a field which highly rewards specialization. They don't need the breadth if they have depth in their particular areas.
My experience has been that in the vast majority of applied roles involving AI, there is not enough emphasis on the software engineering skills necessary for productizing the models, leading to situations like baron_harkonnens critique.
Are people really planning their carrier like this? Isn't it more that you either work for some company nearby, or a company you know by some luck or explore the AI/ML topics you or your professor likes and you got stuck? (or you start disliking it and try hard to find something else)?