With parametric polymorphism and generalized algebraic data types, you could:
(0) Give every process in the pipeline could its own input and output type, rather than forcing all processes to use strings.
(1) Enforce, statically, that only "compatible" processes can be composed. For instance, if `Foo` is a process that outputs integers, and `Bar` is another process that takes strings as input, then `Foo` can't feed its output to `Bar`.
However, this is involves some rather tricky type hackery (so-called "type-aligned sequences"). Hardly a good fit for a language with Go's design goals.
Well, I think unfortunately my sketchy code examples maybe did not cover the idea in enough detail.
What I'm doing in practice is to create different struct types for each type of process I use, which all are of a "base interface" called "process" (mainly for the Run() method, so that the pipeline component can call it).
The typing of channels for each process struct type should now enforce what components can be connected together (you will not be able to assign a "string chan" to a "[]byte chan" field, for example).
Did this answer the question, or do you see further problems here?
Ah, okay. I misunderstood the intention of your original example, then. My bad.
In order to prevent further embarrassing myself, I'm going to tell you how I (currently) understand your design, and, if I'm wrong, please tell me: Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping" (connecting processes to each other) is handled elsewhere. In fact, using your "pipeline" abstraction, you could perfectly well run a bunch of processes that aren't connected to each other at all.
> Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
Exactly!
> This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping"
Ah, yes, didn't think about that, but yeah, but that's a good point! I should consider a better name for this ... Thanks for pointing it out!
Catnaroek said it already: without parametric polymorphism you are forced to either: 1) commit to a single type upfront which limit the utility (string in your example I think), 2) use interface{} and suffer the boxing in every circonstances while not having any static type checking.
I actually like your pattern. I think it makes sense to have a process as a pair of input and output streams.
For those that are interested in libraries like these, OCaml has an implementation that written by Jane Street called Pipes[0]. It's been around for years, helps trade billions of dollars a day, is written in a language that actually has parametric polymorphism, and a configurable notion of "pushback". I've written about it before[1].
What do you mean with "pushback"? Is it the same as "back-pressure"?
(If so, Go has "implicit backpressure" in its bounded buffers on the channels ... although I don't think it is changeable at runtime, unless you replace the channel ... but not 100% sure ...)
A lot of cool concepts come from languages like OCaml and Haskell. But for better or worse, we live in a world where simple languages rule the day (C++ being one notable exception), and if we get lucky, we get some of cool OCaml/Forth/Haskell/LISP/Erlang features to play with, in our simple little worlds with our simple little toy languages like Java and C# and Go and Javascript... it's sad in a way, but it's reality. I'd much rather write F# (basically .NET-hosted OCaml) than C#, but no luck finding gainful employment with that...
It takes an interesting, slightly different approach, in my view especially for slightly lower-level operations compared to the "scipipe" pattern above. For example, it looks like something I'd use for implementing the internals of components in my pattern.
20 comments
[ 2.1 ms ] story [ 12.9 ms ] threadIn the meanwhile, please see this working example on the Go playground: http://play.golang.org/p/voUfPGQulf
I think it should read: https://play.golang.org/p/vrwEkHy-DR
In the meanwhile, please see this working example on the Go playground: http://play.golang.org/p/voUfPGQulf
At best these are copy-pastable patterns.
(0) Give every process in the pipeline could its own input and output type, rather than forcing all processes to use strings.
(1) Enforce, statically, that only "compatible" processes can be composed. For instance, if `Foo` is a process that outputs integers, and `Bar` is another process that takes strings as input, then `Foo` can't feed its output to `Bar`.
However, this is involves some rather tricky type hackery (so-called "type-aligned sequences"). Hardly a good fit for a language with Go's design goals.
Well, I think unfortunately my sketchy code examples maybe did not cover the idea in enough detail.
What I'm doing in practice is to create different struct types for each type of process I use, which all are of a "base interface" called "process" (mainly for the Run() method, so that the pipeline component can call it).
An example of this in action is shown in this code example on the Go playground: http://play.golang.org/p/voUfPGQulf
The typing of channels for each process struct type should now enforce what components can be connected together (you will not be able to assign a "string chan" to a "[]byte chan" field, for example).
Did this answer the question, or do you see further problems here?
In order to prevent further embarrassing myself, I'm going to tell you how I (currently) understand your design, and, if I'm wrong, please tell me: Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping" (connecting processes to each other) is handled elsewhere. In fact, using your "pipeline" abstraction, you could perfectly well run a bunch of processes that aren't connected to each other at all.
Is my understanding correct?
Exactly!
> This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping"
Ah, yes, didn't think about that, but yeah, but that's a good point! I should consider a better name for this ... Thanks for pointing it out!
Catnaroek said it already: without parametric polymorphism you are forced to either: 1) commit to a single type upfront which limit the utility (string in your example I think), 2) use interface{} and suffer the boxing in every circonstances while not having any static type checking.
I actually like your pattern. I think it makes sense to have a process as a pair of input and output streams.
See also answer to Catnaroek above!
[0]: https://ocaml.janestreet.com/ocaml-core/111.28.00/doc/async_...
[1]: https://news.ycombinator.com/threads?id=seliopou&next=980930...
What do you mean with "pushback"? Is it the same as "back-pressure"?
(If so, Go has "implicit backpressure" in its bounded buffers on the channels ... although I don't think it is changeable at runtime, unless you replace the channel ... but not 100% sure ...)
It takes an interesting, slightly different approach, in my view especially for slightly lower-level operations compared to the "scipipe" pattern above. For example, it looks like something I'd use for implementing the internals of components in my pattern.