Backward chaining is goal-driven, it's aimed at answering specific questions where you can prune your search space effectively. This is opposed to data-driven forward chaining where you basically keep looking at what things can be derived without having specific queries in mind. I wouldn't say one is better than the other, it depends on circumstances and use cases. In dynamic systems where there is a lot of data changes/data is fed in dynamically, backward chaining gives you a completeness guarantee whereas FC is eventually consistent (specific answer you are looking for might be not triggered yet). On the other hand, FC is useful when we want to use triggering behaviours - once a specific information is added to the system, a trigger can be executed/specific action can be taken.
Caveat first off: it's 7+ years since I looked at this, so take "modern" in that context. I used SWI Prolog specifically [1].
At that point, CLIPS and Prolog occupied similar spaces - but not the same. Prolog is backward chaining. So it's really good at answering questions that can be derived from a base set of facts. For example. Let's say:
There's a road from City A to City B, and a road from City B to City C.
It's then very easy in Prolog to say "there's a route from "Start" to "Finish" if either:
1. there's a road from Start to Finish, or
2. There's a road from Start to City X, and a route from X to Finish.
The beauty of prolog is that you don't need to explicitly code iteration over all the available roads; that's what the backward chaining algorithm does for you.
In arriving at the answer, it'll exhaustively test all permutations. Importantly though, it won't add to the list of facts you started with (unless you specifically tell it to). Once the question is answered, the intermediate results are discarded. In that sense, Prolog - at least in purest form - is somewhat stateless. Assert some facts, then ask questions about inferences that can be drawn from them.
CLIPS is different. It's a forward chaining engine, from a class of related products known as Production Systems. It's designed to be used where new facts are being asserted; when that happens, it figures out the consequences as they ripple through. As new facts are asserted - and their consequences computed - the underlying state evolves. So intermediate results are retained (in fact, that's key to operation).
CLIPS (and Production Systems in general) were used, for example, in medical diagnosis applications. As new symptoms were asserted, the rules rippled through, suggesting the most likely diagnosis.
The key to making this efficient is the forward chaining algorithm, known as Rete [2] (pronounced "Rait").
Again: these are examples, and the reality is likely much more nuanced now. But at least in their purest original form, CLIPS and Prolog are more complementary than overlapping.
I only spent a few minutes with it, but highly recommend the user's guide. I thought it was clearly and well-written, and also pretty funny.
> A fact such as (duck) or (quack) is said to consist of a single field [...] As a simple analogy, you can think of a field as a picture frame. The frame can hold a picture, perhaps a picture of your pet duck (For those of you who are curious what a picture of a “quack” looks like, it could be (1) a photo of an oscilloscope trace of a duck saying “quack”, where the signal input comes from a microphone, or (2) for those of you who are more scientifically inclined, a Fast Fourier Transform of the “quack” signal, or (3) a TV-huckster selling a miracle cure for wrinkles, losing weight, etc.).
There are a few moments in my life as a young programmer that I remember as mind-expanding. CLIPS was one of those moments (along with arrays, files and recursion).
It's forward-chaining (not backward, as Prolog is) so you can see the database of facts build up step by step. A nice bridging point between procedural and declarative programming.
And a foot in the door for parens-based syntax (it looks a bit like LISP) without the s-expression model.
Rule based systems can be extremely useful for certain types of problems. I'm working on incorporating a rule based system into our business code to allow easier changes to that logic by the people it affects.
As for expert systems in terms of ai, I do think that's gone by the wayside, but probably not entirely gone (ditto for decision trees).
Would you mind sharing which rule-based system you're working with? IMO Answer Set Programming is very useful for things like this. The language is small and maps straightforwardly onto intuitive ways of thinking about rules. It's closer to Prolog than things like CLIPS but really neither forward- nor backwards-chaining. It's hiding all operational specifics from you, which makes it very scalable as a representation. I think it's something many teams would do well to reach for when they take a first stab at implementing business logic in a rule-based format.
For the moment it's just a tiny language and interpreter I've been building, taking inspiration from other rule-based systems. We wanted something that'd run in our application and we're running PHP (:-/) so I didn't have too many options. (I'd really love to open source it once it's complete, but we'll see.)
I'll have to look into Answer Set Programming more as I havn't seen it before. Thanks!
In Germany they actually managed to get grasp on major parts of a government funding aimed to advance ai research here. I figure old professors riding the current hype wave can be quite convincing to incompetent officials despite not having anything to do with recent achievements...
The challenge is scale, if you aren't using a very flat and controlled (database like) rule system - so more like a knowledge store than a reasoning system - then you run into big problems of complexity real fast. But even worse is the brittleness. Everything is fine, until it suddenly isn't and then you discover you can't touch anything without everything blowing up.
On the other hand, if you have a system of case statements or if thens that is a bit opaque (or very opaque) refactoring them to use an expert system is a very good idea.
And, if you have a homebrew knowledge store then using a rule system is also a good idea - but now-a-days you might want to try building a knowledge graph instead, mostly because that's v.trendy.
Not really, no. This whole industry is very fad driven, and there is a tendency to always aggressively jump on the "new shiny"... and once that happens, anything deemed "old" gets treated - by some members of the community - as though it is completely dead, obsolete, verboten, whatever. But rarely is that actually the case. Quite often "old" tools and techniques are quite useful for specific things.
As an example, the box-end wrench didn't become useless because the air-ratchet was invented. As another commentor said, "use the right tool for the job".
All of that said, sometimes the older techniques do legitimately have major shortcomings, or limitations, or may simply be superseded by better tools. As somebody else said, one of the biggest limitations to some older approaches to "expert systems" was making them scale. Some of that we got for free thanks to Moore's Law. In other cases, we may still be waiting for some new algorithmic breakthrough or whatever.
Anyway, things like CLIPS, OPS5, etc. still have uses, even if they didn't turn out to be the be-all end-all of AI as they may once have been perceived to be.
Clips was inspired by Charles Forgy’s OPS5 that was implemented in Common Lisp and introduced the Rete algorithm that is also used in Clips.
I didn’t do much with Clips but OPS5 was a large part of my working life for years (supported commercial versions on Macintosh and Xerox Lisp Machines, extended the Rete algorithm to support multiple worlds for an internal project).
Clips was directly designed based on ART (Automated Reasoning Tool), an expensive early Expert System Shell written in Lisp. Unfortunately ART seems to be lost...
I thought the developer had access to the ART software system, software written in ART and ART documentation - since that was what NASA was using already to develop rule-based expert systems?
Given that ART was written in Common Lisp and exposed a full Common Lisp inside, it would also be trivial to get large amounts of information about its implementation (data structures, functions, arglists, documentation, traces, macro expansions, disassemblies, ...). I'm pretty sure ART itself had extensive debugging&tracing tools for the rules engine.
Not having source to some Common Lisp software, says very little, when the full Common Lisp environment is available for introspection.
I wouldn't expect that ART was shipped in source to its customers. But I would expect that both ART and the underlying Lisp had excellent introspection capabilities.
> The prototype version of CLIPS was developed in the spring of 1985 in a little over two months. Particular attention was given to making the tool compatible with expert systems under development at that time by the Artificial Intelligence Section. Thus, the syntax of CLIPS was made to very closely resemble the syntax of a subset of the ART expert system tool developed by Inference Corporation.
Since ART was what they were using.
> Although originally modelled from ART, CLIPS was developed entirely without assistance from Inference or access to the ART source code.
Sure, it wouldn't say anything else in their position.
I would point out that the vast majority of conspiracy theories fall off the stupid tree and hit every branch on the way down, but that's exactly what someone who's involved in a conspiracy would say. If you believe we had access to the ART source code or were smart enough to glean large amounts of information about its implementation, you also have to believe that we were too stupid or inept to make use of that knowledge. The crown jewels of the ART source code would have been their specific implementation of the rete algorithm and there's no way you could look at CLIPS performance at that point in time and compare it to ART and come to the conclusion that we knew anything specific about their implementation.
The points that were mentioned on the CLIPS history page were 'compatibility' and 'syntax'.
I would guess to be compatible one would need to know both syntax and semantics (what the constructs are doing and how a rule system executes the rules) of the constructs of ART. Having access to documentation and actual implementations helps with that.
I never did say anything about source code or performance. I would also not think that people were stupid - especially since they were smart enough to develop expert systems.
Back in 2003 I worked for a startup that used CLIPS. While it's quite powerful and expressive, as the number of rules grows it becomes very difficult to reason about and debug them. The application logic becomes spaghetti-like and very brittle. So it's an interesting paradigm to learn about, but not very useful for real-world applications IMO.
Sounds like game logic. There was a board-game prototype a friend was working on, and I suggested using a rules-based approach for coding the game logic.
More recently, I was tempted to make a tool for the designers on Storm Casters to specify game and meta-game logic like this, but had to abandon that due to higher priorities.
Love Drools. The consultancy I worked for made so many happy customers by just simplifying a lot of complex optimization use cases with Drools. Hard to do better than Drools for rule-based anything if you’re on the JVM.
As someone who is only recently getting into rules system development, outside of the manual, are there any other great references you would recommend for understanding rules systems like modern Drools?
You might find the CLIPS docs useful [0]. iirc, DROOLS was at least in part inspired by CLIPS. Plus the CLIPS docs are very well written as noted elsewhere.
The book "Expert Systems: Principles and Programming" [1] is also good though now out of print.
Pyke [1], a Python based rules engine, supports both forward and backwards chaining. The docs are good. We used it in an AI class I was a TA for; the previous years used Prolog and everyone (most everyone) consistently hated Prolog.... For students used to the video games that are IDEs these days, SWI Prolog and its UI is awful. Pyke and PyKnow were well received, however. As it is Python based, it is a lot easier to deal with than Prolog.
Ok… I know how this sounds, but… have you considered changing the name? I, too, like Drools a lot, but seriously, the looks when I tell a bunch of stuffy MBAs that we’re going to manage the complex rules using a tool called “Drools”…
I’ve been using this in our Python project via clipspy (https://github.com/noxdafox/clipspy) and it’s been pretty nice. Clips is really REPL-centric and provides a lot of good debugging tools. It’s powerful and fast. The maintainer (Gary Riley) also does a great job answering questions on SO. Pretty happy with it.
Gary was also one of the first and primary authors of CLIPS.
If I recall correctly, a summer intern in the AI section under the Mission Planning Directorate, whose name I have forgotten, developed a prototype or a proof-of-concept of CLIPS. At the time, 1984-ish, the AI group at JSC had many expensive Symbolics and Lisp Machine computers running ART. The intern's project showed that a pattern matching rules based engine could be developed in a conventional programing language for distribution to traditional cost effective hardware.
> a conventional programing language for distribution to traditional cost effective hardware.
In 1984, C was considered a new programming language. The VAX first came out in 1977, and the Sun 1 in 1982. Traditional hardware in 1984 would be an IBM mainframe.
They gave a lot of IAAI talks! WOW! I gave only one, from our joint work with GM.
It looks like they were able to make expert systems useful.
Our work on KnowledgeTool might be equally useful!
After all, it appears that their work was stuck with C, and our work on KnowledgeTool was based on, was a pre-processor to -- sorry to mention this -- PL/I. But, gee, they based on C, so mentioning PL/I should not be so bad!
Our use of KnowledgeTool was to monitor server farms and networks. There in particular we were trying to do problem detection. So, in expert systems,
When I see A and B and
5 or more cases of C
without a D, then it
looks really bad --
raise an alarm.
and, thus, have staff investigate.
And this statement of a symptom, test, for something wrong, was supposed to come from operator experience. So, broadly the idea of KnowledgeTool for that work was not to find better ways to monitor but just to encode what operators had already learned, mostly just from experience.
Hmm .... In such monitoring, we have two ways to be wrong, (1) a false alarm where we say that the system is sick when it is well and (2) a missed detection where we say that a system is well when it is sick.
So, we want the false alarm rate, to be called the probability of Type I error, and the missed detection rate, to be called the probability of Type II error.
Quite broadly there is a trade-off between these two rates: If we accept a higher false alarm rate, then we stand to get a lower missed detection rate.
Not all means of detection are equivalent: Some poor detectors have really high rates for both false alarms and missed detections, and some good detectors have really low rates on both of the errors.
A problem with expert systems was that we had no idea what rates we were getting or if our detectors were poor or good.
Commonly at least in principle we can adjust the rate of false alarms. That is good because (1) a false alarm sends the bridge staff on wild goose chasing and (2) false alarm rate too high is a standard complaint.
Looking at the problem in a little more detail, we easily had wide, deep rapidly flowing oceans of data. So, we could easily have data on 1000 variables with the data for each variable arriving at 100 times a second. So, write some expert system rules? No thanks!
So, I dreamed up a solution, basically, right, as guess from Type I/II, a statistical hypothesis test but one that is both distribution free and multi-dimensional, maybe the first such. False alarm rate can be set in advance of collecting data, and is in a useful sense for each selected false alarm rate has the detection rate the highest possible (for that false alarm rate).
I did this work to improve on expert systems for monitoring. Yes, I published the work, in Information Sciences.
Ah, beating expert systems! What's in the paper is some applied math based on some advanced pure math prerequisites, especially an idea from ergodic theory. What's in the paper really SHOULD be deployed. I suspect that more could be done.
Well, the OP starts with
> Developed at NASA’s Johnson Space Center from 1985 to 1996, the C Language Integrated Production System (CLIPS) is a rule-based programming language useful for creating expert systems and other programs where a heuristic solution is easier to implement and maintain than an algorithmic solution.
Gee, I thought that my "algorithmeic", really original mathematical statistics, was easier than doing as well with heuristics.
Okay, I'll give in: Nearly any good academic research library should have and be able to find the paper:
"A Real-Time System-Adapted Anomaly
Detector", Information Sciences, volume
115, pages 221-259.
The core math is some measure preserving transformations of the data -- get to move data around without changing probability distribution -- for any distribution. The usual place see measure preserving is in ergodic theory. Recall, my work is distribution free which means need make no assumptions at all about probability distributions. In the context of computer and network performance data, distribution-free is nearly essential. After moving the data around, get to do some counting that leads to calculating false alarm rate.
So the paper is some applied probability, that is, does some probability calculations in the service of some, call it, mathematical statistics.
The simplest view is that it's all just nearest neighbors detection -- if get a point too far from the old points, then raise an alarm. Since I wrote the paper, that idea has become common. But apparently what has not become common is how to adjust and calculate false alarm rate, and in practice that is really important. Indeed, for any detection, my work can report the lowest false alarm rate for which the (real time) observation would still be a detection -- so get to have not just a detection but, intuitively, a <i>seriousness</i> measure.
"Joe, this is a detection, and it will remain a detection with a false alarm rate of one in a billion. Put down your pizza, and let's investigate that." So, right, could display a strip chart with such data.
For calculating false alarm rate there is a simple approach that gives the right answer, but the simple approach makes an impossible assumption of independence. Well, the right answer is still true; it's just that an actual proof of false alarm rate, where don't assume independence, is a bit tricky mathematically.
This math is a close cousin of not independence but a weaker assumption exchangeability where often can get the same results as if had independence.
My math may be something of a rationalization of part of resampling as pursued by P. Diaconis and B. Efron at Stanford.
In an intuitive sense, we almost have independence. Hmm -- approximate independence is not a big field. There is a paper by M. Talagrand, A New Look at Independence that may be able to shed some light on my paper, find another way to do the calculations and show that the same calculations hold approximately, maybe with some bounds, in other situations. Talagrand is one heck of a good mathematician, a student of Choquet, a member of Bourbaki.
It would be nice to have the most powerful statistical hypothesis tests as from the classic result of Neyman-Pearson; alas, in the context we don't have enough data to use Neyman-Pearson. But I did come up with a weaker but still useful sense in which my techniques do yield the most powerful test. In practice, the test will likely be seen as nicely powerful, that is, even when false alarm rate has been selected to something small, the detection rate is still, likely in practice, about the highest can hope for for that selected false alarm rate.
There is an issue of how to make the computations fast -- so, need something out of computational geometry. I worked up a technique that should be reasonably fast. The solid state drives of today should do wonders for this technique.
The computational geometry and the most powerful test derivation are not in the paper.
There's more that can be done.
Might also use the work for other anomaly detection problems. One nice point is, although might be using lots of variables, don't encounter "over fitting".
What we were doing with expert systems was mostly just thresholds on one variable at a time. Well, in some cases, we know so much about the variable that that is ...
I used K-means clustering in various domains (access control, fraud control, network data plane quality controls), so wanted to check your work.
One problematic area for K-means, at least from my point of view, is that it is not really possible to easily trace what features/variables are mostly responsible for the anomaly.
One way to get a copy of the paper would be to make a photocopy at a library or pay a service to do that.
Explanation is a problem, also for my work.
Broadly, for a first cut, pick a target system want to monitor. Then use variables that are for the target, the whole target, nothing but the target or some such. Then if get a detection, start diagnosis, looking for cause, with the target.
For two targets that interact, have detectors for each and then one more for variables from both of them.
Broadly, but crudely, have a hierarchy of detectors and then when start getting detections chase down the tree of the hierarchy, like the patient is sick so, look at the major measures and the major organs, etc. When find a suspicious organ, zoom in, drill down, etc. Finally discover the problem is a USB cable that is loose from vibrations from a cooling fan or some such??
In this tree, might want to have somewhat coordinated false alarm rates.
One motivation for my work was a cluster for transactions. One day one computer in the cluster got a little sick in the head and was throwing away all its incoming transactions. So, to the load leveling it looked not very busy and was getting nearly all the transactions and, thus, essentially ruined the work of the whole cluster. And that's not the only case of a cluster getting totally sick due to just one computer in the cluster getting sick. So, I was hoping that getting data from all the computers in the cluster would raise an alarm from the cluster and data from each of the computers in the cluster would essentially do the diagnosis of which computer in the cluster was sick.
Likely more could be done. E.g., might want to do some data scaling.
I don't yet have a big server farm, and I don't know anyone who does who cares enough about anomaly detection to use my work. Expert systems? At one time, yup. My work? Nope. If my startup does well, then, sure, I'll deploy my detector, lots of instances.
My startup isn't anomaly detection.
At one time, I wrote lots of VCs about my anomaly detection work, that it was from IBM's Watson lab, that it was published in Information Sciences, that I had work on fast algorithms, that I had good results on real data, etc. I got back just nothing.
I guessed that the target customers would be high end shops so that I would have to come in with a highly polished product, with lots of good data handling utility tools, some first class hand holding, etc., all of which would be expensive before the first sale. So, I gave up on the idea that I could do a startup from my anomaly work. There might be a way; if I ran a large, really serious server farm, I have at least a little project pursuing my work and/or related work. But, apparently mostly that's not the way server farm management works.
So, I picked another problem for a startup, one where I could get a good solution and bring it to good revenue with just my own efforts as sole, solo founder. I did that and am about to go for an alpha test.
Somehow anomaly detection just doesn't get people very interested.
Once I gave a talk, and some in the audience mentioned that could use my work for fraud detection in, say, credit cards. Well, maybe, but that audience had nothing to do with credit cards.
At one time there were some Soviets in England watching some government offices and keeping track of lights, comings, goings, etc. They guessed that if a war was on the way, this data would show anomalies and early warning. Yup, the Soviets saw the broad issue. I don't know if they used anything like my work or not.
So, maybe the secret is not just some good work in applied probability with some useful results but publicity, hype, fads, group think, a movement, etc., even if what are selling is just total nonsense.
My work is on the shelves of the research libraries. I did my part. If people want anomaly detection, there's some good work there. I've told the Sand H...
The new hotness is combining both so that your NN doesn't think an upside down school bus in a field is a rectangular flower, or a lamb in a child's arms is actually a cat.
So, I didn't really like Prolog when I had to study it for a class, but I'm generally fond of rule-based systems. Would love to try them as a way to specify game-logic.
Hmmm. I think I just figured out what I'm going to do for my next game jam.
It's a very good solution for complex problems where a large amount of knowledge or expertise is required in order to reach a solution. Its documentation and REPL are excellent for those who want to approach fundamentals of AI by solving concrete problems.
It also comes with several language bindings allowing to integrate it with other AI/ML platforms such as tensorflow, numpy etc.. In the past I helped few students which were trying to bring together machine learning models and expert systems to benefit from both worlds.
Surprised to see lack of any comments on experts systems domain from the semantic technologies perspective. Is it because both domains are disconnected (concepts-wise, tools-wise or otherwise) or because people from the experts systems domain don't see any value in semantic technologies? Or something else? A larger related question is what is the optimal strategy for integrating semantic technologies with / into AI systems (except for real-time AI, e.g., autonomous driving, where semantic approach is most likely not feasible performance-wise).
Expert systems are AI. (and some of us are salty about statistics creeping overreach on terminology).
It seems to me that HN might not be too frequented hy people who work with symbolic AI, but expert systems and semantic tools are very, very deeply connected. In fact, semantic technologies are derived from work on expert systems, and using JESS with RDF store was part of my coursework at university.
63 comments
[ 5.6 ms ] story [ 140 ms ] thread[1] https://www.metalevel.at/prolog
[2] https://github.com/klaussinani/awesome-prolog
Why is it inherently better to use a bakeards chaining system under all circumstances?
Caveat first off: it's 7+ years since I looked at this, so take "modern" in that context. I used SWI Prolog specifically [1].
At that point, CLIPS and Prolog occupied similar spaces - but not the same. Prolog is backward chaining. So it's really good at answering questions that can be derived from a base set of facts. For example. Let's say:
There's a road from City A to City B, and a road from City B to City C.
It's then very easy in Prolog to say "there's a route from "Start" to "Finish" if either:
1. there's a road from Start to Finish, or
2. There's a road from Start to City X, and a route from X to Finish.
The beauty of prolog is that you don't need to explicitly code iteration over all the available roads; that's what the backward chaining algorithm does for you.
In arriving at the answer, it'll exhaustively test all permutations. Importantly though, it won't add to the list of facts you started with (unless you specifically tell it to). Once the question is answered, the intermediate results are discarded. In that sense, Prolog - at least in purest form - is somewhat stateless. Assert some facts, then ask questions about inferences that can be drawn from them.
CLIPS is different. It's a forward chaining engine, from a class of related products known as Production Systems. It's designed to be used where new facts are being asserted; when that happens, it figures out the consequences as they ripple through. As new facts are asserted - and their consequences computed - the underlying state evolves. So intermediate results are retained (in fact, that's key to operation).
CLIPS (and Production Systems in general) were used, for example, in medical diagnosis applications. As new symptoms were asserted, the rules rippled through, suggesting the most likely diagnosis.
The key to making this efficient is the forward chaining algorithm, known as Rete [2] (pronounced "Rait").
Again: these are examples, and the reality is likely much more nuanced now. But at least in their purest original form, CLIPS and Prolog are more complementary than overlapping.
[1] http://www.swi-prolog.org/ [2] https://en.wikipedia.org/wiki/Rete_algorithm
EDIT: minor grammar changes + clarified naming in Prolog example.
> A fact such as (duck) or (quack) is said to consist of a single field [...] As a simple analogy, you can think of a field as a picture frame. The frame can hold a picture, perhaps a picture of your pet duck (For those of you who are curious what a picture of a “quack” looks like, it could be (1) a photo of an oscilloscope trace of a duck saying “quack”, where the signal input comes from a microphone, or (2) for those of you who are more scientifically inclined, a Fast Fourier Transform of the “quack” signal, or (3) a TV-huckster selling a miracle cure for wrinkles, losing weight, etc.).
There are a few moments in my life as a young programmer that I remember as mind-expanding. CLIPS was one of those moments (along with arrays, files and recursion).
It's forward-chaining (not backward, as Prolog is) so you can see the database of facts build up step by step. A nice bridging point between procedural and declarative programming.
And a foot in the door for parens-based syntax (it looks a bit like LISP) without the s-expression model.
Is this not the case?
As for expert systems in terms of ai, I do think that's gone by the wayside, but probably not entirely gone (ditto for decision trees).
I'll have to look into Answer Set Programming more as I havn't seen it before. Thanks!
On the other hand, if you have a system of case statements or if thens that is a bit opaque (or very opaque) refactoring them to use an expert system is a very good idea.
And, if you have a homebrew knowledge store then using a rule system is also a good idea - but now-a-days you might want to try building a knowledge graph instead, mostly because that's v.trendy.
Not really, no. This whole industry is very fad driven, and there is a tendency to always aggressively jump on the "new shiny"... and once that happens, anything deemed "old" gets treated - by some members of the community - as though it is completely dead, obsolete, verboten, whatever. But rarely is that actually the case. Quite often "old" tools and techniques are quite useful for specific things.
As an example, the box-end wrench didn't become useless because the air-ratchet was invented. As another commentor said, "use the right tool for the job".
All of that said, sometimes the older techniques do legitimately have major shortcomings, or limitations, or may simply be superseded by better tools. As somebody else said, one of the biggest limitations to some older approaches to "expert systems" was making them scale. Some of that we got for free thanks to Moore's Law. In other cases, we may still be waiting for some new algorithmic breakthrough or whatever.
Anyway, things like CLIPS, OPS5, etc. still have uses, even if they didn't turn out to be the be-all end-all of AI as they may once have been perceived to be.
I didn’t do much with Clips but OPS5 was a large part of my working life for years (supported commercial versions on Macintosh and Xerox Lisp Machines, extended the Rete algorithm to support multiple worlds for an internal project).
Inference Corp was then Brightware, which then was bought by Firepond, then ???
Given that ART was written in Common Lisp and exposed a full Common Lisp inside, it would also be trivial to get large amounts of information about its implementation (data structures, functions, arglists, documentation, traces, macro expansions, disassemblies, ...). I'm pretty sure ART itself had extensive debugging&tracing tools for the rules engine.
Not having source to some Common Lisp software, says very little, when the full Common Lisp environment is available for introspection.
I wouldn't expect that ART was shipped in source to its customers. But I would expect that both ART and the underlying Lisp had excellent introspection capabilities.
http://clipsrules.sourceforge.net/WhatIsCLIPS.html
> The prototype version of CLIPS was developed in the spring of 1985 in a little over two months. Particular attention was given to making the tool compatible with expert systems under development at that time by the Artificial Intelligence Section. Thus, the syntax of CLIPS was made to very closely resemble the syntax of a subset of the ART expert system tool developed by Inference Corporation.
Since ART was what they were using.
> Although originally modelled from ART, CLIPS was developed entirely without assistance from Inference or access to the ART source code.
Sure, it wouldn't say anything else in their position.
I would guess to be compatible one would need to know both syntax and semantics (what the constructs are doing and how a rule system executes the rules) of the constructs of ART. Having access to documentation and actual implementations helps with that.
I never did say anything about source code or performance. I would also not think that people were stupid - especially since they were smart enough to develop expert systems.
More recently, I was tempted to make a tool for the designers on Storm Casters to specify game and meta-game logic like this, but had to abandon that due to higher priorities.
The book "Expert Systems: Principles and Programming" [1] is also good though now out of print.
[0] http://www.clipsrules.net/Documentation.html [1] http://www.amazon.com/exec/obidos/tg/detail/-/0534384471/ref...
[1] http://pyke.sourceforge.net/index.html
If I recall correctly, a summer intern in the AI section under the Mission Planning Directorate, whose name I have forgotten, developed a prototype or a proof-of-concept of CLIPS. At the time, 1984-ish, the AI group at JSC had many expensive Symbolics and Lisp Machine computers running ART. The intern's project showed that a pattern matching rules based engine could be developed in a conventional programing language for distribution to traditional cost effective hardware.
In 1984, C was considered a new programming language. The VAX first came out in 1977, and the Sun 1 in 1982. Traditional hardware in 1984 would be an IBM mainframe.
It looks like they were able to make expert systems useful.
Our work on KnowledgeTool might be equally useful!
After all, it appears that their work was stuck with C, and our work on KnowledgeTool was based on, was a pre-processor to -- sorry to mention this -- PL/I. But, gee, they based on C, so mentioning PL/I should not be so bad!
Our use of KnowledgeTool was to monitor server farms and networks. There in particular we were trying to do problem detection. So, in expert systems,
and, thus, have staff investigate.And this statement of a symptom, test, for something wrong, was supposed to come from operator experience. So, broadly the idea of KnowledgeTool for that work was not to find better ways to monitor but just to encode what operators had already learned, mostly just from experience.
Hmm .... In such monitoring, we have two ways to be wrong, (1) a false alarm where we say that the system is sick when it is well and (2) a missed detection where we say that a system is well when it is sick.
So, we want the false alarm rate, to be called the probability of Type I error, and the missed detection rate, to be called the probability of Type II error.
Quite broadly there is a trade-off between these two rates: If we accept a higher false alarm rate, then we stand to get a lower missed detection rate.
Not all means of detection are equivalent: Some poor detectors have really high rates for both false alarms and missed detections, and some good detectors have really low rates on both of the errors.
A problem with expert systems was that we had no idea what rates we were getting or if our detectors were poor or good.
Commonly at least in principle we can adjust the rate of false alarms. That is good because (1) a false alarm sends the bridge staff on wild goose chasing and (2) false alarm rate too high is a standard complaint.
Looking at the problem in a little more detail, we easily had wide, deep rapidly flowing oceans of data. So, we could easily have data on 1000 variables with the data for each variable arriving at 100 times a second. So, write some expert system rules? No thanks!
So, I dreamed up a solution, basically, right, as guess from Type I/II, a statistical hypothesis test but one that is both distribution free and multi-dimensional, maybe the first such. False alarm rate can be set in advance of collecting data, and is in a useful sense for each selected false alarm rate has the detection rate the highest possible (for that false alarm rate).
I did this work to improve on expert systems for monitoring. Yes, I published the work, in Information Sciences.
Ah, beating expert systems! What's in the paper is some applied math based on some advanced pure math prerequisites, especially an idea from ergodic theory. What's in the paper really SHOULD be deployed. I suspect that more could be done.
Well, the OP starts with
> Developed at NASA’s Johnson Space Center from 1985 to 1996, the C Language Integrated Production System (CLIPS) is a rule-based programming language useful for creating expert systems and other programs where a heuristic solution is easier to implement and maintain than an algorithmic solution.
Gee, I thought that my "algorithmeic", really original mathematical statistics, was easier than doing as well with heuristics.
From you, too, we want to hear what utility you see in the paper!
"A Real-Time System-Adapted Anomaly Detector", Information Sciences, volume 115, pages 221-259.
The core math is some measure preserving transformations of the data -- get to move data around without changing probability distribution -- for any distribution. The usual place see measure preserving is in ergodic theory. Recall, my work is distribution free which means need make no assumptions at all about probability distributions. In the context of computer and network performance data, distribution-free is nearly essential. After moving the data around, get to do some counting that leads to calculating false alarm rate.
So the paper is some applied probability, that is, does some probability calculations in the service of some, call it, mathematical statistics.
The simplest view is that it's all just nearest neighbors detection -- if get a point too far from the old points, then raise an alarm. Since I wrote the paper, that idea has become common. But apparently what has not become common is how to adjust and calculate false alarm rate, and in practice that is really important. Indeed, for any detection, my work can report the lowest false alarm rate for which the (real time) observation would still be a detection -- so get to have not just a detection but, intuitively, a <i>seriousness</i> measure.
"Joe, this is a detection, and it will remain a detection with a false alarm rate of one in a billion. Put down your pizza, and let's investigate that." So, right, could display a strip chart with such data.
For calculating false alarm rate there is a simple approach that gives the right answer, but the simple approach makes an impossible assumption of independence. Well, the right answer is still true; it's just that an actual proof of false alarm rate, where don't assume independence, is a bit tricky mathematically.
This math is a close cousin of not independence but a weaker assumption exchangeability where often can get the same results as if had independence.
My math may be something of a rationalization of part of resampling as pursued by P. Diaconis and B. Efron at Stanford.
In an intuitive sense, we almost have independence. Hmm -- approximate independence is not a big field. There is a paper by M. Talagrand, A New Look at Independence that may be able to shed some light on my paper, find another way to do the calculations and show that the same calculations hold approximately, maybe with some bounds, in other situations. Talagrand is one heck of a good mathematician, a student of Choquet, a member of Bourbaki.
It would be nice to have the most powerful statistical hypothesis tests as from the classic result of Neyman-Pearson; alas, in the context we don't have enough data to use Neyman-Pearson. But I did come up with a weaker but still useful sense in which my techniques do yield the most powerful test. In practice, the test will likely be seen as nicely powerful, that is, even when false alarm rate has been selected to something small, the detection rate is still, likely in practice, about the highest can hope for for that selected false alarm rate.
There is an issue of how to make the computations fast -- so, need something out of computational geometry. I worked up a technique that should be reasonably fast. The solid state drives of today should do wonders for this technique.
The computational geometry and the most powerful test derivation are not in the paper.
There's more that can be done.
Might also use the work for other anomaly detection problems. One nice point is, although might be using lots of variables, don't encounter "over fitting".
What we were doing with expert systems was mostly just thresholds on one variable at a time. Well, in some cases, we know so much about the variable that that is ...
but there are no PDFs that I could download without payin 37.95 $USD at ( https://www.sciencedirect.com/science/article/pii/S002002559... )
I used K-means clustering in various domains (access control, fraud control, network data plane quality controls), so wanted to check your work.
One problematic area for K-means, at least from my point of view, is that it is not really possible to easily trace what features/variables are mostly responsible for the anomaly.
Explanation is a problem, also for my work.
Broadly, for a first cut, pick a target system want to monitor. Then use variables that are for the target, the whole target, nothing but the target or some such. Then if get a detection, start diagnosis, looking for cause, with the target.
For two targets that interact, have detectors for each and then one more for variables from both of them.
Broadly, but crudely, have a hierarchy of detectors and then when start getting detections chase down the tree of the hierarchy, like the patient is sick so, look at the major measures and the major organs, etc. When find a suspicious organ, zoom in, drill down, etc. Finally discover the problem is a USB cable that is loose from vibrations from a cooling fan or some such??
In this tree, might want to have somewhat coordinated false alarm rates.
One motivation for my work was a cluster for transactions. One day one computer in the cluster got a little sick in the head and was throwing away all its incoming transactions. So, to the load leveling it looked not very busy and was getting nearly all the transactions and, thus, essentially ruined the work of the whole cluster. And that's not the only case of a cluster getting totally sick due to just one computer in the cluster getting sick. So, I was hoping that getting data from all the computers in the cluster would raise an alarm from the cluster and data from each of the computers in the cluster would essentially do the diagnosis of which computer in the cluster was sick.
Likely more could be done. E.g., might want to do some data scaling.
I don't yet have a big server farm, and I don't know anyone who does who cares enough about anomaly detection to use my work. Expert systems? At one time, yup. My work? Nope. If my startup does well, then, sure, I'll deploy my detector, lots of instances.
My startup isn't anomaly detection.
At one time, I wrote lots of VCs about my anomaly detection work, that it was from IBM's Watson lab, that it was published in Information Sciences, that I had work on fast algorithms, that I had good results on real data, etc. I got back just nothing.
I guessed that the target customers would be high end shops so that I would have to come in with a highly polished product, with lots of good data handling utility tools, some first class hand holding, etc., all of which would be expensive before the first sale. So, I gave up on the idea that I could do a startup from my anomaly work. There might be a way; if I ran a large, really serious server farm, I have at least a little project pursuing my work and/or related work. But, apparently mostly that's not the way server farm management works.
So, I picked another problem for a startup, one where I could get a good solution and bring it to good revenue with just my own efforts as sole, solo founder. I did that and am about to go for an alpha test.
Somehow anomaly detection just doesn't get people very interested.
Once I gave a talk, and some in the audience mentioned that could use my work for fraud detection in, say, credit cards. Well, maybe, but that audience had nothing to do with credit cards.
At one time there were some Soviets in England watching some government offices and keeping track of lights, comings, goings, etc. They guessed that if a war was on the way, this data would show anomalies and early warning. Yup, the Soviets saw the broad issue. I don't know if they used anything like my work or not.
So, maybe the secret is not just some good work in applied probability with some useful results but publicity, hype, fads, group think, a movement, etc., even if what are selling is just total nonsense.
My work is on the shelves of the research libraries. I did my part. If people want anomaly detection, there's some good work there. I've told the Sand H...
https://github.com/nilqed/jsXSI
Is the resurgence of expert systems finally a thing after statistics-all-the-things fatigue?!
Can't wait till marketing folks run with "3rd wave AI".
http://pyke.sourceforge.net/index.html
Amazing!
Hmmm. I think I just figured out what I'm going to do for my next game jam.
It is used in medicine as a diagnosis tool: https://pdfs.semanticscholar.org/64c0/672703724bd32e04b94c10... http://www.jatit.org/volumes/research-papers/Vol4No10/5Vol4N...
In software security for malware detection and categorization: https://ieeexplore.ieee.org/abstract/document/4724664
In gaming for driving AI: https://www.youtube.com/watch?v=-S1CkfzEHSU&t=489s
And plenty of other fields.
It's a very good solution for complex problems where a large amount of knowledge or expertise is required in order to reach a solution. Its documentation and REPL are excellent for those who want to approach fundamentals of AI by solving concrete problems.
It also comes with several language bindings allowing to integrate it with other AI/ML platforms such as tensorflow, numpy etc.. In the past I helped few students which were trying to bring together machine learning models and expert systems to benefit from both worlds.
It seems to me that HN might not be too frequented hy people who work with symbolic AI, but expert systems and semantic tools are very, very deeply connected. In fact, semantic technologies are derived from work on expert systems, and using JESS with RDF store was part of my coursework at university.