I like this, but can I make one recommendation? Ditch the classes and just operate on primitive data! You're very close to thinking functionally with the pipelining of commands operating on a message, but you've got a bit of overhead with the explicit data types.
Of course, this is just an opinion from someone who has been doing a lot of functional and declarative stuff these days with Javascript using underscore, d3, and angular, so your mileage may vary. :)
Though I would have to disagree with you about explicit data types. In static typed languages key to productivity is using data types and resolve most of the bugs at compile time as opposed to run time. It would be a bad call in this case to borrow advices that better apply to other environments.
Hey, he could have been banging out payroll programs in COBOL for most of his career and just been confronted with his first Java or Smalltalk program.
Indeed. See C.A.R. Hoare's CSP and the Occam language derived (somewhat horribly) from it and implemented with a lightning fast hardware scheduler in the Transputer a wonderful little device that just couldn't gain traction. Messages could be fielded by processes within a given instance of the Transputer or, via serial hardware links, transparently by processes elsewhere. The programmer didn't need care where the heavy lifting was actually carried out.
Being C programmer at the time and rather repulsed by all that was missing in Occam yet enamored with the Transputer I designed a rather straightforward and easily understandable extension of C that fully incorporated CSP and could be compiled to either directly use the Transputer's hardware or to architectures that didn't have it. Alas the design died on a corner of my desk because I had "real" work to do.
I still think Hoare had a good idea and would love to see this author's work fully fleshed out and implemented.
I'm not sure I see the benefit of creating a generic pipeline vs just having a "manager" class or function that calls each step in succession. The process of "logging in" is conceptually a single step, thus it makes sense to have one unit that abstracts over the entire process. Factoring out the logic of successive calls into a pipeline doesn't seem very useful here. I could see a benefit if it were the case that each subprocess existed and is executed independently, but in the login case and probably most cases in general web development, this usually isn't true.
To me, the benefit of a pipeline is when each filter can register itself (e.g. an event queue), thus the entity generating it has no knowledge of where it is going or who is processing it. A key indicator of this is when order of execution doesn't matter. In the case of logging in, order is critical so something somewhere must control the order each filter is ran. At this point you might as well have a unit that orchestrates this process itself.
What about the case where you would have multiple managers? For example, today you wrote a FooManager that handles all the Foos, next month you'll write a BarManager, and after that a BazManager and so on. Each of these will probably have a slightly different way of handling things...
Creating pipelines for all of these scenarios seems like it would mean you're doing things consistently across your code base, which will hopefully make it easier to maintain. It also might make it easier to reuse code (you can share filters between different services).
The problem I see with this is that the pipeline concept also constrains your code in other ways. For example, using the pipeline there is no clean way to allow communication between each successive subprocess. Return values in this case aren't clean (would involve casting), and mutating the data passed around seems like a cure worse than the disease. I know of no way to construct a generic pipeline that allows for all the flexibility of simply successive function calls. I suspect most problems won't cleanly fit the constraints imposed by a generic pipeline.
People are mentioning the Lisp-y nature of this writing, even if is not about Lisp. Is this not specifically something from Erlang, with message-based communication? Granted, this article is short on where that shines in Erlang, cross-process/cross-thread communication.
Or, more generally, any "actor" concurrency system. It's built-in in Erlang, you can use Akka in the JVM, there's just a ton of options for message-passing concurrency in Clojure, Go channels offer something similar, and pretty much every language you can think of has some library implementing it.
I see a parallel to aspect oriented programming / advice. The "filters" are equivalent to aspects and "registering" them is equivalent to advising a method. The neat thing about an AOP system is that aspects are cross-cutting and can be applied to any method. It seems like this approach requires special setup (the Pipeline object).
Any research in concurrency is a good thing. The massively parallel program model that I want to know more about is IBM's TrueNorth.[1]
None of the articles I've found had any substance, but the it was fitting that hardware diagram[2] looks very similar to Bret's "vision from the 70s" :) [3]
Odd that Web API was chosen over servicestack.net given 'Messaging-based web services' has been the fundamental concept behind servicestack.net for several years which has always promoted a message-based design as a first-class concept where all its features are built around message-based DTOs to take advantage of the benefits of messaging: https://github.com/ServiceStack/ServiceStack/wiki/Advantages...
The pipeline described in the article is very similar to ServiceStack's request pipeline (https://github.com/ServiceStack/ServiceStack/wiki/Order-of-O...) where Global, Service and Action Request and Response Filters allows composition of custom logic to be selectively applied at different levels of granularity and priorities.
ServiceStack's message-based design is what allows it to provide .NET's most succinct, end-to-end typed API without any code-gen or post-build steps. You're able to re-use the code-first DTO's services were defined with, as-is, inside clients. This has many benefits, message-based DTO designs are more resilient, and support the natural evolution of services. It's inherently simpler with less friction as client and server call-sites have symmetrical parity as the same DTO the client sends is transparently hydrated into services on the server.
Message-based design's are also inherently more re-usable as they encapsulate services in their most re-usable form, which is what allows the same service in ServiceStack to be called via HTML/Web, REST, SOAP, MQ endpoints in JSON, XML, SOAP 1.1/2, HTML, CSV, JSV, MessagePack and Protocol Buffers formats with no effort.
Every so often somebody rediscovers messaging as a cool programming model. This has been going on since the 80s at least. Somebody wrote a paper on the duality of procedural vs message based coding.
Messaging has some cool properties. Static deadlock analysis. Potentially idem-potent design allowing response-less interfaces (e.g. send a state message instead of an incremental update; eventually state will settle). Transparent distributed processing. Queuing and fault tolerance, not for free but pretty easy.
It takes a mental wrench to move to this model, but I think its worth it.
23 comments
[ 2.5 ms ] story [ 58.8 ms ] threadOf course, this is just an opinion from someone who has been doing a lot of functional and declarative stuff these days with Javascript using underscore, d3, and angular, so your mileage may vary. :)
Though I would have to disagree with you about explicit data types. In static typed languages key to productivity is using data types and resolve most of the bugs at compile time as opposed to run time. It would be a bad call in this case to borrow advices that better apply to other environments.
Arcitechture and code seperated and only requests and representations gets passed around by a microkernel.
Being C programmer at the time and rather repulsed by all that was missing in Occam yet enamored with the Transputer I designed a rather straightforward and easily understandable extension of C that fully incorporated CSP and could be compiled to either directly use the Transputer's hardware or to architectures that didn't have it. Alas the design died on a corner of my desk because I had "real" work to do.
I still think Hoare had a good idea and would love to see this author's work fully fleshed out and implemented.
To me, the benefit of a pipeline is when each filter can register itself (e.g. an event queue), thus the entity generating it has no knowledge of where it is going or who is processing it. A key indicator of this is when order of execution doesn't matter. In the case of logging in, order is critical so something somewhere must control the order each filter is ran. At this point you might as well have a unit that orchestrates this process itself.
Creating pipelines for all of these scenarios seems like it would mean you're doing things consistently across your code base, which will hopefully make it easier to maintain. It also might make it easier to reuse code (you can share filters between different services).
I think this is much of the point behind this pattern, actually.
http://en.wikipedia.org/wiki/Actor_model#Actor_libraries_and...
None of the articles I've found had any substance, but the it was fitting that hardware diagram[2] looks very similar to Bret's "vision from the 70s" :) [3]
[1] http://www.technologyreview.com/news/517876/ibm-scientists-s...
[2] http://www.networkworld.com/news/2013/080813-ibm-devises-sof...
[3] http://worrydream.com/dbx/slides/slide.032.png
The pipeline described in the article is very similar to ServiceStack's request pipeline (https://github.com/ServiceStack/ServiceStack/wiki/Order-of-O...) where Global, Service and Action Request and Response Filters allows composition of custom logic to be selectively applied at different levels of granularity and priorities.
ServiceStack's message-based design is what allows it to provide .NET's most succinct, end-to-end typed API without any code-gen or post-build steps. You're able to re-use the code-first DTO's services were defined with, as-is, inside clients. This has many benefits, message-based DTO designs are more resilient, and support the natural evolution of services. It's inherently simpler with less friction as client and server call-sites have symmetrical parity as the same DTO the client sends is transparently hydrated into services on the server.
Message-based design's are also inherently more re-usable as they encapsulate services in their most re-usable form, which is what allows the same service in ServiceStack to be called via HTML/Web, REST, SOAP, MQ endpoints in JSON, XML, SOAP 1.1/2, HTML, CSV, JSV, MessagePack and Protocol Buffers formats with no effort.
Messaging has some cool properties. Static deadlock analysis. Potentially idem-potent design allowing response-less interfaces (e.g. send a state message instead of an incremental update; eventually state will settle). Transparent distributed processing. Queuing and fault tolerance, not for free but pretty easy.
It takes a mental wrench to move to this model, but I think its worth it.