Hi folks, I'm one of the creators of Upstream. We'll try to be around all day to answer any questions you might have. We're feeling pretty nervous, but excited to get your feedback!
Being a sysadmin and being interested in monitoring systems, I know quite well
where and how to apply stream processing as a general paradigm. Having this
"stream-processing-as-a-service", I don't have even the slightiest idea where
it could be useful. The fact that the service only provides webUI for defining
things, and thus the processing capabilities are very limited, doesn't help.
I'm curious, when you say a limited set of capabilities, what libraries or systems would you juxtapose us with?
Our large goal was to make the sharing and collaboration piece much easier than other systems allow for. So let's say that one person might be responsible for instrumenting a system, but others want to be able to create their own notifications or visualizations. One user can publish a stream of events, and then allow others to subscribe and create their own transformations/visualizations. There are other pub/sub systems that achieve a similar result, but generally access control doesn't exist and usually lacks any real system of discover-ability. In the same way that github/bitbucket/etc make repositories easier, we wanted to enable a similar effect with streams of data. And in the same way, we built in a number of tools that make manipulating or visualizing a stream of data easy without leaving the system (sure, you shouldn't probably do the bulk of your code editing out of github's text editor, but it's very handy for quick changes).
I'm not sure if that helps frame our system better. We're still working a bit on how to position the system, so we might even try and revise our landing page to emphasize this more.
So your service basically provides (a) place where one can send data for
further processing, (b) place where one can process data that was collected,
(c) tools for plotting and charting available data, and (d) tools for sending
alerts on available data. And registry with available data streams.
Mind you, access control is most probably not a feature for your customers,
but for your company. Limiting data visibility for employees is usually
unnecessary for smallish businesses, and big businesses rarely take part in
developing somebody else's product like this.
With all this, I would say that you are like AMQP broker, except you have
a stream registry, can do a little of trivial data processing (handful of set
functions on a window), have a charting UI, and no way of extracting data
back, from what I've seen in the documentation.
And you asked what I compare your service with. These would be Splunk and
Riemann from finished software, and in terms of processing capabilities,
plenty of stream processing systems invented in academia, like Aurora.
5 comments
[ 2.2 ms ] story [ 23.6 ms ] threadOur large goal was to make the sharing and collaboration piece much easier than other systems allow for. So let's say that one person might be responsible for instrumenting a system, but others want to be able to create their own notifications or visualizations. One user can publish a stream of events, and then allow others to subscribe and create their own transformations/visualizations. There are other pub/sub systems that achieve a similar result, but generally access control doesn't exist and usually lacks any real system of discover-ability. In the same way that github/bitbucket/etc make repositories easier, we wanted to enable a similar effect with streams of data. And in the same way, we built in a number of tools that make manipulating or visualizing a stream of data easy without leaving the system (sure, you shouldn't probably do the bulk of your code editing out of github's text editor, but it's very handy for quick changes).
I'm not sure if that helps frame our system better. We're still working a bit on how to position the system, so we might even try and revise our landing page to emphasize this more.
Mind you, access control is most probably not a feature for your customers, but for your company. Limiting data visibility for employees is usually unnecessary for smallish businesses, and big businesses rarely take part in developing somebody else's product like this.
With all this, I would say that you are like AMQP broker, except you have a stream registry, can do a little of trivial data processing (handful of set functions on a window), have a charting UI, and no way of extracting data back, from what I've seen in the documentation.
And you asked what I compare your service with. These would be Splunk and Riemann from finished software, and in terms of processing capabilities, plenty of stream processing systems invented in academia, like Aurora.