If you only need one hour then indeed it's great price but if you need it 24/7 then it's 260$ a month which is really expensive if you compare alternative (you can get dedicated server capable of same throughput easily for half the price).
true, but Amazon's own messaging / pub sub services, e.g. SNS or SQS go for 50 cents/million or 2000 for four billion - not really a fair comparison of course as they do something quite different.
What you mean exactly with the compared alternative?
Capable of the same throughput? In theory or with a real reproducible benchmark?
And which type of throughput? What kind of messages?
I did understand. Like I said it was a style choice. I could've just as well have said "What kind of project is that successful and can't scrap together $230/mo?" and avoided the "your project" reference altogether. Choice of style. Didn't think you'd get hung up over it.
"AWS is overpriced service for people that do not have time or resources to do things on their own and pay massive prices for that."
Isn't that the point of AWS?
The time / resources cost to build (monitor and maintain..) your own infrastructure isn't zero.
Did these companies have spare teams lying around the place, such that no new hires were needed to make the transition? If not, then it's not so much of a saving at the current cost of dev / ops personnel, just now on a different balance sheet.
Yep. Aws is funny. You want to at do it first, then it doesnt make sense at e.g the 10k per month price point. So you do something else.
Then you come back when you have due diligence and compliance and availability requirements. And you see how much that is really costing you with the arrangements you have.
Finally if you're super successful, you could self host or otherwise. But "rack costs" (naively) trade off against people and auditors.
Summary: AWS is expensive, especially for bandwidth. Practically if your business requirements are not bandwidth intensive, it ends up being paradoxically cheaper than the alternatives if your business involves legal agreements and things like availability commitments to customers.
It's not really rocket science you know, to install elasticsearch/redis/cassandra/0MQ/puppet etc and scale it to hundreds of servers. You do not need to build your infrastructure from ground up, just go with dedicated servers or with colocation if you want your own hardware and you will save tons of money comparing to AWS.
Although I generally disagree with the parent comment, I have to say that AWS actually IS overpriced. Just look at Cloud plattforms like OVH.com or even digitalocean.com, they charge almost nothing for the traffic, whereas it comes at a whopping 0.09$/GB at AWS.
If you aren't dependant on specific AWS products like DynamoDB you will definitely find a much better deal elsewhere.
I think the most obvious comparison is Firebase, but as a self-hosted solution deepstream is much cheaper (5-10x, dependant on the usecase). The creators of deepstream.io have actually written a blog post that covers this exact topic: https://deepstream.io/blog/realtime-framework-overview/
That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes.
Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).
A lot of it is log data from requests passing through CloudFlare. We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers. We have about 4 million sites on CloudFlare and each customer has access to analytics about their site which are stored in a CitusDB database.
Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).
Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]
EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!
Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you.
These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.
Deepstream relies on garbage collection to free up dereferenced
memory. If a machine's CPU is overutilized above 100% for a
consecutive time, garbage collection will be delayed and memory
can add up. If this continuous for a prolonged period, the
server will run out of memory and eventually crash - so be
generous enough when it comes to resource allocation to make
sure that your processors get some breathing space every once
in a while.
"Be generous when it comes to resource allocation".
--
> The costs of running a six-instance cluster for an hour on AWS are 36 cents (6 x t2.medium @ 0.052$/h + 1 x cache.t2.medium @ 0.068$/h)
AFAIK, pricing in AWS world depends upon the region. Bandwidth, hard disks and so on also contribute to the price.
deepstream solves similar problems as meteor, but is conceptually quite different. It's a standalone server that you install the same way you would install e.g. Nginx or a database. Clients connect to it using small SDKs that come in different languages.
It similarly provides data-sync, pub-sub and request response with no opinion about your frontend framework or technology stack and has an open ecosystem of connectors that make it work with all sorts of databases, caches and message buses.
It's also significantly faster than meteor, making it possible to also use it for multiplayer gaming, realtime trading etc...
Isn't Meteor a stand-alone server? It sits on top of node.js, and has a number of extra packages included to make it more of a complete framework, but those packages can easily be removed or replaced.
I'm interested to know what happens when there is a failure or a deployment? Having static servers handling load is only part of the problem in our experience. The true complexity and scalability of a system comes when you consider how it copes under load with unexpected failures (network, hardware), but more importantly expected maintenance such as regular deploys, scaling up and scaling down events. Do you have any metrics for that? Those are the problems we've been focussing most of our energy on at Ably (https://www.ably.io), not just message per second rates which is often not really the problem.
yup, you can cluster deepstream across multiple regions. The only thing to take into account is that its datalayer (e.g. redis cache / some combination of cache and db) needs to be clustered as well
47 comments
[ 2.6 ms ] story [ 98.3 ms ] threadIt's not about 260$ it's about tens of thousands of dollars saved by companies i work with after migrating from AWS.
AWS is overpriced service for people that do not have time or resources to do things on their own and pay massive prices for that.
Isn't that the point of AWS?
The time / resources cost to build (monitor and maintain..) your own infrastructure isn't zero.
Did these companies have spare teams lying around the place, such that no new hires were needed to make the transition? If not, then it's not so much of a saving at the current cost of dev / ops personnel, just now on a different balance sheet.
Then you come back when you have due diligence and compliance and availability requirements. And you see how much that is really costing you with the arrangements you have.
Finally if you're super successful, you could self host or otherwise. But "rack costs" (naively) trade off against people and auditors.
Summary: AWS is expensive, especially for bandwidth. Practically if your business requirements are not bandwidth intensive, it ends up being paradoxically cheaper than the alternatives if your business involves legal agreements and things like availability commitments to customers.
We're handling 4m to 6m 1.5k log lines per second using Apache Kafka on a cluster of around 100 nodes.
Each log line (in JSON) is roughly 1.5kB.
Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).
More details: https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]
EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!
These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.
--
> The costs of running a six-instance cluster for an hour on AWS are 36 cents (6 x t2.medium @ 0.052$/h + 1 x cache.t2.medium @ 0.068$/h)
AFAIK, pricing in AWS world depends upon the region. Bandwidth, hard disks and so on also contribute to the price.
I'm not sure what to make of such conclusions.
It similarly provides data-sync, pub-sub and request response with no opinion about your frontend framework or technology stack and has an open ecosystem of connectors that make it work with all sorts of databases, caches and message buses.
It's also significantly faster than meteor, making it possible to also use it for multiplayer gaming, realtime trading etc...
That's ~25MB of data per second for an in-memory workload over 6 machines. I think they missing a zero on the size of the messages?
I see a lot of advantages on using a more standard protocol such as MQTT over deepstream
It's closer to a self hosted version of Firebase or Parse than MQTT