What a phenomenal landing page. Thanks to the animations I have a good idea of what this does and how it does it. You barely need to know what message queuing is to understand it.
The animations are very slick, but I can still grumpily pick at them: the "Clustering and Quorum-based Replication" one shows 4 total servers (suboptimal number because quorum requires 3/4 where I'd prefer to require 2/3 or 3/5) and all the followers acking before the primary proceeds (the animation could show one lagging instead).
4 might be useless compared to 3 from a "required quorum" PoV but if you want to stay up with 1 failure and the load can't be handled on 3 servers but 4 is OK, it is optimal isn't it?
I.e. selection of ideal server count is a multi-dimensional problem.
Worse than useless from a quorum perspective, because you've increased the number of servers that have to participate in quorum (quickly) without increasing the number that don't. Looking at it the other way, you go from having problems when 2/3 are slow/broken to having problems when 2/4 are slow/broken. (also note slow doesn't even mean the machine is having problems; in geographically distributed setups it may just be due to physical distance from the leader.)
Yes, 4 servers might handle more load than 3 servers [1], but so would 5 servers. I'd guess that if you've got enough message queue traffic that 3 servers can't handle the load, those servers aren't a significant fraction of your total cost. You might as well go to 5. Likewise, if 5 aren't enough, I'd skip over 6 and go to 7. If 7 aren't enough...I'd probably rethink my design entirely...but I'd prefer 9 to 8.
Alternatively, quorum systems can support "zero-weight replicas" which effectively don't vote but still can be useful for eventually consistent reads. Then you have this server that helps handle the load but doesn't increase the number of servers that have to participate in a round.
[1] if the load is overwhelmingly eventually consistent read operations and their multi-hop topology with proxies doesn't sufficiently reduce the load on the replicas. It's not obvious to me if this scenario is plausible or not.
Does anyone recommend any particular piece of software for creating these types of visualizations with relative ease? I've always been a fan of animations for helping to understand some technical topics.
As erwinkle mentioned, these animations use anime.js
If you want to build something but don't really know how to do it, I would suggest the following:
1. Build the visual content in an SVG with Inkscape. It should be easy for you to build the layout with assets/item to animate. Don't forget to group items and set names so you can reference them later.
2. Once exported, insert the SVG in an HTML page and add anime.js. This library will let you animate the content you just created (you should be able to reference your named items from Inkscape and animate them). The learning curve might be tough depending on your experience regarding animation (or CSS in general).
I've heard Framer can make some impressive animations, but I don't know much about how well they export to web. What's on display here is really cool, it's all animated SVG.
Okay, this is going to sound weird, but I've used Keynote in the past. I'm sure PowerPoint can do this as well, but Keynote has Magic Move, and you can export this animation. It's very easy to work with, and do simple animations. I'm sure there are much better tools for this, but as someone who knew Keynote, this was always effective for me.
Keynote is surprisingly good for this, and I’ve used it for animated data flow visualizations in the past.
While I like the idea of something like anime.js and think it raises the ceiling on what’s possible, I’d still probably start with something like keynote just to get the idea out of my head and into a format I can validate before going into code.
Lottie.js is a library for animations in JavaScript, and after effects can export to it! You're obviously limited in what you can export, but if you do a basic after effects animation with shapes and text, it works fine.
Pretty sure it was made by Airbnb, and used in tons of apps like Google home
All the replies mention technologies to help you build one, but this chart looks like a modified, animated version of a Petri Net to me, which is a very useful chart for visually describing the flow of data between systems. I've built similar charts using PowerPoint and exporting the PowerPoint as a GIF... same effectiveness for internal use, but not something you're going to want to brag to your friends about.
It has a gorgeous landing page but I'm not really sure why this is better than any other MQ. Can someone more knowledgeable provide any insight into its comparative advantages?
Edit: I did see the comparisons page, but.. well, there's more to life than Rabbit and Kafka!
> Can someone more knowledgeable provide any insight into its comparative advantages?
Well, you know, there is the "Comparison With Alternative Systems" page[1]. :-)
Sadly it doesn't include NATS in the comparison. I would have been interested to see that. But the usual suspects (Rabbit & Kafka) are given a mention.
I'd also be interested in seeing how it compares to AMQP implementations (Apache Qpid for example) and any of the proprietary message queue systems (IBM and Microsoft) they refer to but don't name-check.
I mean one unsubstantiated armchair take: it's newer than Rabbit / Kafka (RabbitMQ was built in 2007, Kafka in 2011), since then the learnings from distributed systems, message queues and programming languages has improved by leaps and bounds, and the existing ones will have backwards compatibility / legacy issues.
That said, there are indeed plenty of commercial and noncommercial alternatives, so now that it's been published I'm sure there will be some thorough people making comparisons soon!
My guess is that at the time when Bloomberg's infrastructure was considering supporting a message queue, they surveyed existing solutions (mind you that this was a long time ago) and found them lacking for one reason or another, rightfully or not.
RabbitMQ was probably avoided because nobody wanted to learn Erlang.
Also, R&D had a lot of experience building message oriented middleware "from scratch" in a low overhead high availability way, so first instinct was probably to start hacking in C++.
Nowadays it might be the case that some teams within Bloomberg need the performance or would rather have a bespoke solution instead of spending on migrating to something else off the shelf.
Keep in mind that this is a company that has its own implementation of most of the C++ standard library.
Queueing things can result in better throughput because you don't have to wait for the other side to process your last message before you can begin work on the next one.
That's just buffering. A lot of people use Kafka this way, as an impedance adapter for a producer with an internal architecture that can't tolerate blocking on production. Of course this requires you to unrealistically assume that Kafka is a zero-impedance message sink.
But what I think the other post is alluding to is the fact that in-order delivery is often not a requirement, and can be an anti-feature. I know that in every use of Kafka I have personally encountered the relative order of messages was quite irrelevant but the in-order implementation of Kafka converts the inability to process a single record into an emergency.
> I don't understand how you can make queueing high-performance. Queueing things is literally the opposite of making things fast.
You likely need to read about message queues, why they are used, and why they have performance constraints. Message queues are a system, and the system to maintain those queues and content has an overhead. The project is building a solution that is higher performing than comparable systems.
Queuing things often makes things faster (!) as you are usually dealing with limited resources (ex: web and job workers) and need to operate on other systems (ex: payments) that have their own distinct capacity.
If you have 1000 web workers, and your upstream payment provider supports 10 payments per minute, you'll quickly have timeouts and other issues with 1000 web workers trying to process payments. Your site won't process anything else while waiting for the payment provider if all web workers are doing this task.
A queue will let you process 1000 web requests, and trickle 10 requests to your payment provider. At some point that queue will fill, but that's a separate issue.
Meanwhile, your 1000 web workers are free to process the next requests (ex: your homepage news feed).
It's a bit difficult to cover as it is highly dependent on the queue system you use.
You'd usually want your queue system to fail the enqueue if it is full, and you'd want monitoring to ensure the queue isn't growing at an unsustainable rate.
It also forces you to think a bit about your message payload (rich data or foreign keys the worker loads).
RabbitMQ, Redis-based queues (ex: ActiveJob or sidekiq), Gearman, and others will all offer different mechanisms to tackle full queues.
A queue is a thing to be managed carefully when designing systems:
- you need to ensure that your producers don't produce at a faster rate than your consumers can keep up
- if the consumer is unable to handle the request now, queueing it and handling it later is often not the right thing to do. Most likely by the time the resource is available the nature of the request will change.
TCP itself is already a queue. All these message queue systems also make the silly decision of layering their own queueing on top of TCP, leading to all sorts of problems.
Basically those things only sort of work if you have few messages and are happy with millisecond latency.
It really depends what you design your system to handle. A sudden burst of traffic that can be spread for minutes is fine (ex: Elasticsearch indexing requests can usually be delayed through background jobs).
> Basically those things only sort of work if you have few messages and are happy with millisecond latency.
Not really... queues are great to defer processing or have longer running job than your HTTP and TCP timeouts will allow. Building large data export won't happen within a single HTTP request.
i'm not an expert here but each message is pretty important right? What happens if you put a UDP datagram on the wire and it never shows up? Wouldn't you have to come up with some kind of signal to indicate the message made it to the receiver? After that you'll probably start signaling delivery of groups of messages instead of individually and then ordering them on the receiver by when they were sent and then maybe some validation of integrity and sender backoff mechanism for network congestion... and then you've made TCP.
TCP means that all bytes must be copied to a local buffer in order to be re-sent in the case the other end didn't receive them (even though by that time they're probably old and irrelevant). Once the buffer is full you cannot send more data and must wait for the other end to ack them (potentially indefinitely, though most implementations will eventually time out after long enough). This means that if you wanted to send 10Gbps unthrottled to another continent, you'd need a buffer of 500MB per TCP session, assuming no data loss, and consumer keeping up at line rate.
TCP is just not a good fit for anything that needs high-performance or bounded sending times. What it's good for is giving you a simple high-level abstraction for streaming bytes when you'd rather not design your networked application around networking.
Building the right logic for your application is not difficult (most often, just let the requester resend, which you needs to do anyway in the age of fault-tolerant REST services, or just wait for more recent data to be pushed) and easier than tweaking TCP to have the right performance characteristics.
UDP doesn't provide reliable delivery, so you would need to implement some kind of delivery assurance in your application.
UDP also has very small message (datagram) size limits, so you would also need to implement some way to fragment and re-combine messages in your application.
At this point you've built an ad-hoc re-implementation of 80% of TCP.
You’re misunderstanding the discussion. If you think of queuing as “waiting to process” then yes it’s slower than not waiting. But that’s not at all the necessary implication of these components. In fact “synchronous” requests are buffered (queued?) internally and can be delayed before processing. So the distinction is irrelevant to performance. And the implementation is what matters.
That fan-out functionality is particularly interesting for anyone needing to implement the upstream part of a firehose API, which of course Bloomberg do.
> This section explains the leader election algorithm at a high level. It is by no means exhaustive and deliberately avoids any formal specification or proof. Readers looking for an exhaustive explanation should refer to the Raft paper, which acts as a strong inspiration for BlazingMQ’s leader election algorithm.
So their own homegrown leader election algorithm?
> BlazingMQ’s leader election and state machine replication differs from that of Raft in one way: in Rafts leader election, only the node having the most up-to-date log can become the leader. If a follower receives an election proposal from a node with stale view of the log, it will not support it. This ensures that the elected leader has up-to-date messages in the replicated stream, and simply needs to sync up any followers which are not up to date. A good thing about this choice is that messages always flow from leader to follower nodes.
> BlazingMQ’s elector implementation relaxes this requirement. Any node in the cluster can become a leader, irrespective of its position in the log. This adds additional complexity in that a new leader needs to synchronize its state with the followers and that a follower node may need to send messages to the new leader if the latter is not up to date. However, this deviation from Raft and the custom synchronization protocol comes in handy because it allows BlazingMQ to avoid flushing (fsync) every message to disk. Readers familiar with Apache Kafka internals will see similarities between the two systems here.
"a new leader needs to synchronize its state with the followers and that a follower node may need to send messages to the new leader if the latter is not up to date". I thought a hallmark of HA systems was fast failover? If I come to your house and knock on the door, but it takes you 10mins to get off the couch to open the door, it's perfectly acceptable for me to claim you were "unavailable". Pedants will argue the opposite.
"This section explains the leader election algorithm at a high level. It is by no means exhaustive and deliberately avoids any formal specification or proof."
FWIW they mention this at the bottom of their document
> Just like BlazingMQ’s other subsystems, its leader election implementation (and general replicated state machinery) is tested with unit and integration tests. In addition, we periodically run chaos testing on BlazingMQ using our Jepsen chaos testing suite, which we will be publishing soon as open source. We have also tested our implementation with a TLA+ specification for BlazingMQ’s elector state machine.
> Carefully architected and written in C++ from the ground up with no dependency on any external framework, BlazingMQ provides…
What a weird selling point.
There are pros and cons to having no dependencies. Not long ago, it was a common decision for C++ projects because dependency management was a mess. But that hasn’t been the case for a long time (arguably we have the opposite problem - there are so many ways of doing it: Conan, vcpkg, bazel, spack, raw cmake, nix, etc)
So what would the pros and cons be today and why is it a selling point?
For instance, a pro is that everything is bespoke to the task at hand, no hammering square pegs into round holes designed for a slightly different use case.
A con is that the entire attack surface is managed by one team. CVEs identified and solved on another project is a pretty good thing when you depend on that project.
The main reason I’m surprised, though, is that there are some no-brainer dependencies these days. Fmt, catch2/gtest, metaprogramming libraries, etc.
Though my approach is to fork and maintain the fork until my PR is accepted, merged, and in a tagged release.
Another is to apply a diff to the dependency within the build system, which is what I do a lot in Nix (mainly to solve cmake issues on hermetic builds).
Either way, it isn't unsurmountable. It doesn't really matter so much if it's used as an executable rather than as a library, though in the case of the latter it's really handy to be able to e.g. pass a custom spdlog sink for logging.
Hi, one of the authors here. BlazingMQ depends on two other open source C++ libraries: https://github.com/bloomberg/bde and https://github.com/bloomberg/ntf-core. I believe documentation writer wanted to highlight that BlazingMQ does not depend on frameworks like ZooKeeper, etc.
Ah, yes, that makes sense! I'm thinking about framework from the wrong angle.
I can definitely see why that's a bonus. So many data pipeline tools, but so many written in Java. It's for that reason that I've always reinvented the wheel. Perhaps I'll check this out...
> So what would the pros and cons be today and why is it a selling point?
I'm not saying they pulled it off because I would need to evaluate it, but having your own dependency stack can result in a much more streamlined solution for extra performance since you don't have to compromise to someone else's data model. As this is a high speed message bus, it's entirely appropriate.
These are also basically application-level network routers which really don't need huge levels of dependencies so to be honest I'd be running away from anything that claims it's written using 1000s of the newest, latest and greatest whatevers. Churning out a faster string grinder is what I'm looking for.
It doesn't use UDP. So my tongue-in-cheek reply is, it doesn't qualify to have "blazing" in the name. Hopefully that's not just funny inside my own head. :)
If it gains wide adoption, then they've got a global community of people providing free software engineering for their messaging platform. Its a win-win.
Why wouldn’t they? This technology is not a competitive advantage for their company, wouldn’t make sense for them to sell as a standalone product, and it helps their engineering department attract good talent.
Because they're not making money off of their queueing system, but what they build with it; to them it's a cog in their machine, and open sourcing it means others may adopt it and help them maintain and improve it.
For the 200 repos that they have made open source, they had over 6000 people contribute feedback and/or code. So in addition to attracting talent, they also get more testers and feedback.
> Supports a unique multi-hop network topology leading to a distribution tree for each queue, thereby leading to network bandwidth savings for certain use cases.
This is very interesting. One of the benefits of Kafka is partition routing so I'm curious to learn how this may compare.
No idea, but I am interested too. One of the reasons this was probably created was because of a network constrained environment where data is expensive in more than one way.
Specifically, egress from Cloud Providers (transmitting data from inside the cloud to a computer outside the cloud) is the usual culprit for being "very expensive". It's part of a pricing strategy by cloud providers that encourages folks to put all their infrastructure inside of the cloud and to have none of it outside the cloud. As one example of the ink spilled over this topic, see this discussion from 2 years ago about AWS's very high egress fees:
Ingress can also be costly especially if there is a steady state of high load traffic sent from on-prem machines to machines inside cloud (not sure about aws but have experienced this with azure where we had to resort to buy their expressroute which was very costly and ultimately unsustainable for us)
Something else I was caught out on recently is cross availability zone traffic, especially for multi AZ kubernetes where the nodes are very chatty. It's possible to get the services to consider network topology to limit cross az traffic, but it's not the default.
Bloomberg (who built this) operate a system that makes money by transmitting the same data to a huge amount of clients. Having one server transmit the same piece of information to a million client is approximately 500 times slower than having one server transmit it to 1000 proxies who each pass it on to 1000 end users.
The actual architecture of most of these systems in finance relies heavily on UDP multicast, which is a technology that big tech has basically forgotten because it can be tough to administer at large scale.
Multicast puts the entire load of distributing the data in its natural place: the network. The pub/sub queues that the rest of the world uses are more complicated and a lot worse.
This appears to be the beginning of a "cloud-native" bloomberg that can't lean on multicast.
UDP Multicast relies on constantly throwing out the entire state of your system in order to be reliable, whereas with TCP you only need to throw out changes. This is fine in finance because the state is changing constantly anyway so you might as well just continually throw out the multicast traffic.
Message queuing is a more complicated system, but it allows you to push the bulk of your work out to the distributed network instead of concentrating it on the producer.
That is definitely not true about state management - you can build very TCP-like systems on top of UDP (see QUIC), and the final adaptation you need is to figure out how to make that work with multicast (a non-trivial adaptation, but possible). Once you do, you have the foundations of a pub/sub system in the form of a one-to-many (or many-to-many if you are brave enough) network protocol, and now you basically only have to figure out how to do state recovery of new clients and clients who are behind, and that can basically just be a database. The system described in the OP is kind of a combination of a database and load balancer in order to do this over unicast connections.
You have to do this state management with a TCP-based system anyway - the use of TCP doesn't magically come with the full state of past messages.
This is not right in finance at least. Messages updates are typically incremental just like in TCP. The reliability problem is dealt with with careful network stack tuning, core affinity and out of band recovery.
Check out 0MQ (ZeroMQ), Informatica Ultra Messaging (formerly 29 West), and aeron.io
They create reliable uni/multicast protocols over UDP, including strategies for NAK storms etc.
Yes, UDP on top of IP multicast - UDP is a pretty thin layer on top of IP in general, and the IP protocol is deceptively powerful. You just can't run protocols like TCP over IP multicast.
maybe if you're hosting replicas or proxies outside of a highcost bandwith area?
low cost end replicas at the end. if the bandwidth usage is half the cost in their multihop topology, 14XXXX compared to 16XXXX? depends on the number of proxies or replication and pricing.
I’ll admit, I went into their repo expecting to find Rust, or at the very least go. Pleasantly surprised to see C++ code though I would need to look further to see if it’s “good” C++ code. The naming conventions of the source tree lends me to suspect it’s not.
i too determine if code is good based off its naming conventions and other things like
my linter is set to enforce MiXeD_SpOnGeBoB-KeBaB_CaSe as the default casing
It's a joke, as some Rust projects used to repetitively claim to be "memory safe & blazing fast", thus becoming a tongue-in-cheek phrase. I also anticipated language to be Rust, but it would be way too comical.
it never ceases to amaze me how the software engineering profession has very smart folks reimplement the same conceptual primitives over and over again. I'm curious if hospitals (if that's even analogous to a company) do similar things by having different processes to solve the same problem.
I imagine this is used for Bloomberg (the terminal) and not Bloomberg (the website)?
going back to the article - fantastic animations. I'm just as curious to how that was made as the queue itself.
In the US, all hospitals are required to use electronic health records (EHRs) and most are implemented in Epic, the leading EHR system. But they are almost always electronic implementations of their old paper record systems, dating back to who-knows-when.
Yeah, pretty much any medium to large company is going to have a bunch of processes for doing things that any company that size needs to do and some more that are a bit more industry/domain specific but still basically the same as those at every other company in the space.
Part of this evolution is from funded development to opensource development.
Then there is language variances like flavors so what ends up happening is a million different conceptual primitives that fundamentally the same.
It isn't the same because of a few things. At the end of the day too it is also about how good the support is over the rest. If a new MQ can demonstrate better tooling nothing stops a company from adopting it.
mainframe emulation is a thing, aka code change (Ala swap mq libraries) at enterprise scale, and supporting production without a migration path can very much stop adoption.
That really depends on how the MQ integration is designed and whether or not the in queue data is persistent or doesn't get stale. In most cases an application doesn't like stale data so a long lived queue is rare. People who adopt something typically PoC before adoption.
What you say is true.
But...this is Bloomberg so I think we can give them a pass and take a deeper look.
I can't think of a more relevant company within which to develop messaging middleware.
Exactly. Handling message queuing and distribution is likely a core capability for Bloomberg, so it makes a lot of sense for them to develop something tailored for their needs. LinkedIn did the same with the original Kafka development, where distributed eventual state consistency was a core capability.
If anything, we should be thankful that they've decided to share it with the world.
I feel like on the contrary there's an enormous cost attached to using prefab components that only kinda do what you need them to do. Shops that have embrace this philosophy have been some of the most inefficient ones I've ever encountered. Screens of yaml code was written to implement tasks that could have been performed with lines of programming code; and many man-years were spent implementing functions that copied fields from one object to another nearly but not quite identical object.
It's a big driver of both complex and slow software. Often the adaptation layer necessary to get it to actually satisfy your needs rivals the size and complexity of the piece of existing solution code you're trying to leverage.
Software that is built from the ground-up to your exact needs is often both faster and slimmer and more maintainable and suffers from far less dependency churn.
Yeah. This works both ways. At small scale or early stage you should try like hell to work backwards from pre-existing components so that everything does exactly what you need them to do by definition. Eventually you outgrow that and it makes sense to build things that do exactly what you want because the incremental value is worth it.
There is a balance to Not In House syndrome where you build everything yourself, and the opposite: everything should be vendored in. I am inclined to think that a healthy NIH syndrome is the better place to be. Companies which vendor everything move along at a snails pace, and often getting your outside product to work with internal needs is a huge years long endeavor, to the point just writing a decent version yourself that is exactly what you need is really useful.
I think I just got it wrong. Not invented here sounds better and more right. I just typically think of only the acronym, and decided to write it out in case someone hadn’t heard it (and thus made a mistake). Good catch.
In my experience, the problem with vendor: What if a feature is missing? I see this often when I am forced to use a vendor product. And, on day one, no one (hands on) usually knows anything about the vendor product, so there is a huge learning curve. If you follow NIH, then you organically build knowledge in the org. And features can be added with time+money. (Sure, some vendor products will add features, but usually for an insane price tag.) You cannot win either way: They hardest part is knowing which one to choose.
>it never ceases to amaze me how the software engineering provision has very smart folks reimplement the same conceptual primitives over and over again.
Well, no one cares, but I'm not saying that to be mean to dismissive, it's just that software production as an industry (esp. at a company like Bloomberg) can basically do whatever it wants because of low capital costs to produce something. Things like writing another queue system will have zero impact on the industry unless it's adopted widely (and cargo culted) and later made worse.
Maybe I'm about to eat my shoe, but if a capable of team at Bloomberg wanted to reinvent something considered unnecessary in terms of industry need, they probably could if for no other reason that management allowing some indulgence in exchange for keeping talented people around is Worth It™.
Bloomberg had a history of inventing their own stuff going back to the first terminals. They had their own networking protocols and everything, when I was last there you could still see remnants in how machines were named.
They do adopt a lot of industry stuff, but it is always heavily modified. And of course they love “Invented Here” as a corporate tradition going all the way back to the beginning.
It’s actually nice from a developer perspective, the R&D group is highly valued by management.
I don't understand that phrase. Probably a large team of very well paid engineers developed this product. Bloomberg is well-known in the industry for paying well. "[L]ow capital costs"? This software probably cost millions in salaries to build, and millions more to maintain over the next 10+ years.
>You wrote: low capital costs to produce something
>I don't understand that phrase.
You don't need a big factory with large capital costs to make or distribute software. You're only paying people, and salaries are not capital investments.
1. It is easier to implement from scratch than understand and modify: we see this everyday at a way smaller scale with libraries/classes/functions. Not saying this is optimal but this is a driver for sure.
2. Recruiting: some devs love doing this. Fancy OSS project to put on your resume, some actually believe there's unfulfilled need, you get people like us discussing it (otherwise we wouldn't be thinking about Bloomberg).
It's a somewhat difficult argument to say that it's easier to build from scratch, but I hear what you're saying. At any rate, it is absolutely more fun to build from scratch.
I've polled developers about what part of software development they hate, and in general, it's deployment and maintenance (ie: devops). Using a off-the-shelf-component means that the developer only gets to do the un-fun part: installing, configuring, productionizing. If they code their own solution, they get to spend a few weeks coding!
Smart people solving "solved problems" over and over again is exactly how the tech industry progresses. Meanwhile other industries are stuck with processes from fifty years ago because "it has always been done this way", and anyone even suggesting the slightest bit of change is a heretic.
When I left Bloomberg around 7 years ago Kafka was being aggressively rolled out and there was a lot of interest in data streaming tech in general.
I am wondering if they ran into performance or resource usage issues with Kafka at scale, and decided to roll their own for use cases that better fit their workloads.
Bloomberg does have vast backend infrastructure for moving around / computing data which the Terminal is a consumer of.
I've had to do this mostly because of batshit crazy rules many large companies have about how all software has to be supported. The issue there being the definition of supported in the eyes of clueless auditor types.
If software is written 100% externally, but the vendor only wrote 10% of it and is transparently taking 90% from an open source project so they can sell fraudulent support (assuming they don't have people contributing to the open source project here), that counts as supported.
If it is written 100% internally, even if it is built on top of some ancient internal codebase that nobody can figure out and has been in life support mode for 20 years, then it counts as supported.
But if its written 50% internally and 50% of it came from an external open source library with a BSD license, and you are a major contributor to that project, then it is considered unsupported and you get in trouble.
This is how you end up with security issues because some sysadmin decided they had to roll-their-own crypto library or authentication system, because its better to have an unknown implementation show up on a scanner than a known one that has a list of CVEs that can be checked.
Nats is simple because it doesn't bother addressing some of the scalability problems that Kafka was designed to address, like clustering with partitions:
You linked to NATS Streaming, which has been deprecated for years. It’s not relevant. It can still be used for legacy code that still needs it, as the URL even shows, but it is not recommended for new applications.
The current standard is NATS JetStream, which absolutely addresses scalability, arguably to an even larger degree than Kafka, since it can be used to create a globally interconnected supercluster of clusters where messages can be configured to flow between regions as needed. I doubt anyone would run a Kafka cluster that spans multiple regions.
I doubt anyone would run a Kafka cluster that spans multiple regions.
Stretch clusters and clusters with brokers in different AZs are absolutely a thing in Kafka. I make no comment on the ease relative to Nats, which I don't know at all.
I was talking about different regions altogether, not just different AZs. My understanding is that it’s generally a bad idea in Kafka due to the latency, but I could be wrong.
EDIT:
Definitely sounds like there are some tight latency requirements[0]:
> A stretched 3-data center cluster architecture involves three data centers that are connected by a low latency (sub-100ms) and stable (very tight p99s) network, usually a “dark fiber” network that is owned or leased privately by the company.
100ms would theoretically be enough to span the contiguous United States, but the references to "very tight p99s" and "dark fiber" make me wonder if 100ms is actually acceptable, or just a theoretical maximum that can only be allowed under absolutely perfect conditions. Either way, suggesting the user should have access to dark fiber between their regions does not fill me with confidence about the robustness of this solution. I'm sure it would be fine for geographically close datacenters, as AZs are designed to be.
Same here! I don’t get it. It’s much more lightweight, fully FOSS and their custodian company (Synadia) is very friendly and open to self-hosting. Basically, they run a global super-cluster of Nats which is pretty much vanilla afaik.
My main gripe is lack of “best practices”. You’re pretty much on your own when it comes to important decisions like subject namespacing and some operational stuff can be a bit tricky, like changing options and migrating. They had “nats by example” but that seems to be quite out of date and not super helpful. They have started doing more videos which are great but not that much content overall.
Their auth story is very sophisticated but perhaps a bit over-engineered. Some use cases like having anonymous “self-signed” client keys are tricky to work with.
Other than that, it’s an amazing piece of tech. One that is scalable in a way that just makes sense and gets out of the way when multiple machines are involved.
Garden variety NATS (i.e. the non-"Jetstream" case, hereafter) is "at most once" (messages will be lost.) BlazingMQ is "at least once" (>1 delivery of a message will occur.)
That difference is down persistence. NATS doesn't persist messages to stable storage. BlazingMQ must use persistent storage. From that all sorts of application design, performance and other implications are derived. And they all matter. A lot.
Very interesting. Thank you for benchmarks. Thank you for sharing. In a zero-replication no-leader-election (or all-in-one-node) scenario what is min latency observed? It appears predominant delay is in ACKing to ensure durability, right?
C++ and Java client examples are very simple which is cool.
Would appreciate if there is hook to enable `SO_TIMESTAMPNS` on socket and retrieve `cmsg` to get accurate timestamping and permit client code benchmarking.
The python client library is planned to be published. They say they need to remove some dependencies and update the workflow to something more suitable for OSS.
There are a number of ways to wrap native code libraries as a managed library. I did it once (packaged an Obj-C library as a .NET DLL for a Xamarin app) back in 2018 but had lots of help from a teammate who knew that dance far better than me and I forgot all of the details. I googled and found at least four different blogs posts showing how to do it.
I know that's not an answer but if you're feeling adventurous you might be able to help the project.
It's not on the roadmap for the next 6 months. But priorities can change depending upon the demand. One thing to note is that BlazingMQ client libraries are very stateful, and implementing a batteries included library is a non-trivial effort.
I’ve been getting pretty deep in C++ recently and the name Bloomberg shows up all across the C++ standards community. They seem to be a strong C++ shop.
Is this implemented with any sort of thought to security?
I don't see anything that implies as such, all FAQ and comparisons to other message queues say nothing of security (I skimmed and ctrl+f so I could have missed it). This doesn't bode well for a network-based application written in a memory unsafe language.
edit: I'm surprised so many people are interpeting this as a weird cargo cult thing. Application security includes a lot of mundane and critical things like:
Authentication, authorization, message signing / authentication (e.g. HMAC), encryption, secret/key management, how the system handles updates, etc.
I care more about methodology than specific methods. I would like applications that handle I/O, especially network I/O, to be designed and built with security in mind. I asked my question because that doesn't appear to be the case here, and I think that is dissapointing and concerning from a security perspective.
That is an obviously false dichotomy. The author of the comment never mentioned Rust, and wasn’t asking about Rust. Such a false dichotomy feels like flamebait, which is against HN guidelines.
Your false dichotomy also implies that “unsafe” blocks are as unsafe as C++, which is not true. “Unsafe” in Rust turns off very few checks[0], most of them are still active. No one would write a serious Rust program entirely in unsafe anyways.
Regardless, asking about security considerations is a valid thing to do, even if it were written in Rust. Security is not just about memory safety.
Furthermore, choice of language can have an effect as to the actual security of the networking layer.
Having parsers (and serializers) proved absent of runtime errors (e.g. with something like SPARK) is a form of guarantee I wish I'd see as the default in any 'serious' network-facing library or component. It's not even that hard to achieve, compared to the learning curve of the borrow checker.
Once rust gets plugged into Why3 and gets some industrial-grade proof capacity, the question of 'is it written in rust?' will be automatic (as in 'why would you do it any other way?').
Rust's guarantees are all about what's happening inside your own process with memory ownership. "Security" in the context of a middleware like this is almost certainly more about external validation/auth— what prevents some random node from injecting messages into the queue, or adding itself as a subscriber to the output?
Also, it's a super bad-faith argument to talk about an entire program being in an unsafe block. Rust's unsafe is about specifically and intentionally opting out of certain guardrails in a limited scope of code where it should be fairly easy for a human reviewer to understand and validate the more complex invariants that aren't enforceable by the compiler itself.
This is absolutely not what Rust unsafe block is about. It is about getting around the limitations of the compiler to gain efficiency. And it happens a lot.
> Unsafe Rust exists because, by nature, static analysis is conservative. When the compiler tries to determine whether or not code upholds the guarantees, it’s better for it to reject some valid programs than to accept some invalid programs. Although the code might be okay, if the Rust compiler doesn’t have enough information to be confident, it will reject the code. In these cases, you can use unsafe code to tell the compiler, “Trust me, I know what I’m doing.”
Security purely in terms of the language used, or security in terms of cluster access, authentication and so on?
I mean, I don't often see documentation for networked services that goes in-depth on how exactly they write memory safe code in C or C++, as if that would mean the application is therefore secure; it's always much higher level.
> Security purely in terms of the language used, or security in terms of cluster access, authentication and so on?
I'm personally interested in both, but from a docs perspective I'd mostly expect the latter, more "security for users of this system" than anything else. I'm a little embarassed I didn't remember it until now, but I think the general term is "application security."
I'm concerned that I don't see any hints of either in the project. I'm not at a computer with access to my Github account so I can't easily search the source code for hints of obvious signs of care or neglect with respect to security.
Not sure where they are on the roadmap but authentication in particular is 'mid-term' but lacks detail.
I expect that if you want to deploy this then you're doing it on an internal network; its deployment examples are based on Docker so I expect it's relying on what you can do with k8s.
Thanks! That is the sort of stuff I was asking about.
It looks like it's an after thought but at least on their mind now, which is very fair with respect to Bloomberg's wants/needs. It'd be nice if they had a bit of a warning about using this until it has some basic auth(n) and TLS since they're releasing it to the public. I think it is, relativley speaking, rude to release insecure networked software without giving users a notice as to what sorts of insecurity is at least known/expected.
Adding a veneer of security isn't necessarily superior to leaving it out altogether. Systems of this sort are best secured at the network level, i.e. only trusted hosts should be able to connect to it. Redis is a good example of where this has been tried: it does support password based log in, but the password is stored and transmitted in plaintext, and a redis server will happily accept thousands of auth attempts per second making brute forcing a viable attack. Rather than improve the auth system Redis has instead doubled down on encouraging appropriate network level security by defaulting to only being accessible to the local host, so admins have to go through an explicit step (with warnings) before they can just expose it to the internet.
> Is this implemented with any sort of thought to security?
There's a hefty load of cargo cult mentality revolving the topic of security. Security is not a product, but a process. This sort of talk is getting tiresome.
I perused the code and it was written in a C++03 style. Pointer chasing and manual memory management in the few files I poked through. (IE, no unique_ptr, just new/delete).
This may be perfectly safe... however, I'd give it a few years of battle testing before touching it.
Lol. I'm sorry but my work has me directly interfacing with Bloomberg and other financial institutions. If anything, I trust the code LESS because of who wrote it.
Just because software is heavily used and deals with finance, doesn't mean it's secure or well written. Consider the Equifax breach of 2017, everyone's credit info leaked because Equifax was using a crusty old version of struts with known critical CVEs.
268 comments
[ 3.1 ms ] story [ 281 ms ] threadBMQ license is apache 2.0 for anyone curious
I.e. selection of ideal server count is a multi-dimensional problem.
Yes, 4 servers might handle more load than 3 servers [1], but so would 5 servers. I'd guess that if you've got enough message queue traffic that 3 servers can't handle the load, those servers aren't a significant fraction of your total cost. You might as well go to 5. Likewise, if 5 aren't enough, I'd skip over 6 and go to 7. If 7 aren't enough...I'd probably rethink my design entirely...but I'd prefer 9 to 8.
Alternatively, quorum systems can support "zero-weight replicas" which effectively don't vote but still can be useful for eventually consistent reads. Then you have this server that helps handle the load but doesn't increase the number of servers that have to participate in a round.
[1] if the load is overwhelmingly eventually consistent read operations and their multi-hop topology with proxies doesn't sufficiently reduce the load on the replicas. It's not obvious to me if this scenario is plausible or not.
As seen in their source code here: view-source:https://bloomberg.github.io/blazingmq/assets/animations/anim...
1. Build the visual content in an SVG with Inkscape. It should be easy for you to build the layout with assets/item to animate. Don't forget to group items and set names so you can reference them later.
2. Once exported, insert the SVG in an HTML page and add anime.js. This library will let you animate the content you just created (you should be able to reference your named items from Inkscape and animate them). The learning curve might be tough depending on your experience regarding animation (or CSS in general).
Good luck :)
While I like the idea of something like anime.js and think it raises the ceiling on what’s possible, I’d still probably start with something like keynote just to get the idea out of my head and into a format I can validate before going into code.
Pretty sure it was made by Airbnb, and used in tons of apps like Google home
Works really well
Edit: I did see the comparisons page, but.. well, there's more to life than Rabbit and Kafka!
Well, you know, there is the "Comparison With Alternative Systems" page[1]. :-)
Sadly it doesn't include NATS in the comparison. I would have been interested to see that. But the usual suspects (Rabbit & Kafka) are given a mention.
[1] https://bloomberg.github.io/blazingmq/docs/introduction/comp...
That said, there are indeed plenty of commercial and noncommercial alternatives, so now that it's been published I'm sure there will be some thorough people making comparisons soon!
RabbitMQ was probably avoided because nobody wanted to learn Erlang.
Also, R&D had a lot of experience building message oriented middleware "from scratch" in a low overhead high availability way, so first instinct was probably to start hacking in C++.
Nowadays it might be the case that some teams within Bloomberg need the performance or would rather have a bespoke solution instead of spending on migrating to something else off the shelf.
Keep in mind that this is a company that has its own implementation of most of the C++ standard library.
But what I think the other post is alluding to is the fact that in-order delivery is often not a requirement, and can be an anti-feature. I know that in every use of Kafka I have personally encountered the relative order of messages was quite irrelevant but the in-order implementation of Kafka converts the inability to process a single record into an emergency.
That is decent but still below the speed of the network, which should be the main blocker.
You likely need to read about message queues, why they are used, and why they have performance constraints. Message queues are a system, and the system to maintain those queues and content has an overhead. The project is building a solution that is higher performing than comparable systems.
Queuing things often makes things faster (!) as you are usually dealing with limited resources (ex: web and job workers) and need to operate on other systems (ex: payments) that have their own distinct capacity.
If you have 1000 web workers, and your upstream payment provider supports 10 payments per minute, you'll quickly have timeouts and other issues with 1000 web workers trying to process payments. Your site won't process anything else while waiting for the payment provider if all web workers are doing this task.
A queue will let you process 1000 web requests, and trickle 10 requests to your payment provider. At some point that queue will fill, but that's a separate issue.
Meanwhile, your 1000 web workers are free to process the next requests (ex: your homepage news feed).
Would you recommend something to read about that?
You'd usually want your queue system to fail the enqueue if it is full, and you'd want monitoring to ensure the queue isn't growing at an unsustainable rate.
It also forces you to think a bit about your message payload (rich data or foreign keys the worker loads).
RabbitMQ, Redis-based queues (ex: ActiveJob or sidekiq), Gearman, and others will all offer different mechanisms to tackle full queues.
- you need to ensure that your producers don't produce at a faster rate than your consumers can keep up
- if the consumer is unable to handle the request now, queueing it and handling it later is often not the right thing to do. Most likely by the time the resource is available the nature of the request will change.
TCP itself is already a queue. All these message queue systems also make the silly decision of layering their own queueing on top of TCP, leading to all sorts of problems.
Basically those things only sort of work if you have few messages and are happy with millisecond latency.
> Basically those things only sort of work if you have few messages and are happy with millisecond latency.
Not really... queues are great to defer processing or have longer running job than your HTTP and TCP timeouts will allow. Building large data export won't happen within a single HTTP request.
ex: https://shopify.engineering/high-availability-background-job...
1. They're referring to the queueing itself. They're not saying "this will make some other thing fast" they're saying "this is fast at queueing".
2. Queuing is not the opposite of making things fast.
TCP is just not a good fit for anything that needs high-performance or bounded sending times. What it's good for is giving you a simple high-level abstraction for streaming bytes when you'd rather not design your networked application around networking.
Building the right logic for your application is not difficult (most often, just let the requester resend, which you needs to do anyway in the age of fault-tolerant REST services, or just wait for more recent data to be pushed) and easier than tweaking TCP to have the right performance characteristics.
UDP also has very small message (datagram) size limits, so you would also need to implement some way to fragment and re-combine messages in your application.
At this point you've built an ad-hoc re-implementation of 80% of TCP.
So their own homegrown leader election algorithm?
> BlazingMQ’s leader election and state machine replication differs from that of Raft in one way: in Rafts leader election, only the node having the most up-to-date log can become the leader. If a follower receives an election proposal from a node with stale view of the log, it will not support it. This ensures that the elected leader has up-to-date messages in the replicated stream, and simply needs to sync up any followers which are not up to date. A good thing about this choice is that messages always flow from leader to follower nodes.
> BlazingMQ’s elector implementation relaxes this requirement. Any node in the cluster can become a leader, irrespective of its position in the log. This adds additional complexity in that a new leader needs to synchronize its state with the followers and that a follower node may need to send messages to the new leader if the latter is not up to date. However, this deviation from Raft and the custom synchronization protocol comes in handy because it allows BlazingMQ to avoid flushing (fsync) every message to disk. Readers familiar with Apache Kafka internals will see similarities between the two systems here.
"a new leader needs to synchronize its state with the followers and that a follower node may need to send messages to the new leader if the latter is not up to date". I thought a hallmark of HA systems was fast failover? If I come to your house and knock on the door, but it takes you 10mins to get off the couch to open the door, it's perfectly acceptable for me to claim you were "unavailable". Pedants will argue the opposite.
how is this possibly a selling point?
> how is this possibly a selling point?
Context.
In an overview doc? The informal version is accessible to more audiences.
As the canonical design doc for the system? It's certainly not.
FWIW the phrase seems to come from here: https://bloomberg.github.io/blazingmq/docs/architecture/elec...
The full sentence reads:
"This section explains the leader election algorithm at a high level. It is by no means exhaustive and deliberately avoids any formal specification or proof."
Which makes a lot more sense.
And as noted in a sibling comment there is actually a TLA+ spec for the leader election: https://github.com/bloomberg/blazingmq/tree/main/etc/tlaplus
> Just like BlazingMQ’s other subsystems, its leader election implementation (and general replicated state machinery) is tested with unit and integration tests. In addition, we periodically run chaos testing on BlazingMQ using our Jepsen chaos testing suite, which we will be publishing soon as open source. We have also tested our implementation with a TLA+ specification for BlazingMQ’s elector state machine.
> Carefully architected and written in C++ from the ground up with no dependency on any external framework, BlazingMQ provides…
What a weird selling point.
There are pros and cons to having no dependencies. Not long ago, it was a common decision for C++ projects because dependency management was a mess. But that hasn’t been the case for a long time (arguably we have the opposite problem - there are so many ways of doing it: Conan, vcpkg, bazel, spack, raw cmake, nix, etc)
So what would the pros and cons be today and why is it a selling point?
For instance, a pro is that everything is bespoke to the task at hand, no hammering square pegs into round holes designed for a slightly different use case.
A con is that the entire attack surface is managed by one team. CVEs identified and solved on another project is a pretty good thing when you depend on that project.
The main reason I’m surprised, though, is that there are some no-brainer dependencies these days. Fmt, catch2/gtest, metaprogramming libraries, etc.
This alone is an important point to avoid dependencies, specially if you're anything like Bloomberg.
Though my approach is to fork and maintain the fork until my PR is accepted, merged, and in a tagged release.
Another is to apply a diff to the dependency within the build system, which is what I do a lot in Nix (mainly to solve cmake issues on hermetic builds).
Either way, it isn't unsurmountable. It doesn't really matter so much if it's used as an executable rather than as a library, though in the case of the latter it's really handy to be able to e.g. pass a custom spdlog sink for logging.
I can definitely see why that's a bonus. So many data pipeline tools, but so many written in Java. It's for that reason that I've always reinvented the wheel. Perhaps I'll check this out...
I'm not saying they pulled it off because I would need to evaluate it, but having your own dependency stack can result in a much more streamlined solution for extra performance since you don't have to compromise to someone else's data model. As this is a high speed message bus, it's entirely appropriate.
These are also basically application-level network routers which really don't need huge levels of dependencies so to be honest I'd be running away from anything that claims it's written using 1000s of the newest, latest and greatest whatevers. Churning out a faster string grinder is what I'm looking for.
https://bloomberg.github.io/blazingmq/docs/architecture/netw...
Arguably, if you start with UDP but you also need to ensure retransmission and reordering, you eventually end up reimplementing TCP but poorly.
https://github.com/USNavalResearchLaboratory/norm
That guy really knows his stuff regarding performance.
https://devboard.gitsense.com/bloomberg
For the 200 repos that they have made open source, they had over 6000 people contribute feedback and/or code. So in addition to attracting talent, they also get more testers and feedback.
Full Disclosure: This is my tool
This is very interesting. One of the benefits of Kafka is partition routing so I'm curious to learn how this may compare.
Cloud Providers can be expensive.
Do you have a story around this? If so, would love to hear more about it.
Article "AWS’s Egregious Egress" (2021) https://blog.cloudflare.com/aws-egregious-egress/
HN discussion: https://news.ycombinator.com/item?id=27930151
Multicast puts the entire load of distributing the data in its natural place: the network. The pub/sub queues that the rest of the world uses are more complicated and a lot worse.
This appears to be the beginning of a "cloud-native" bloomberg that can't lean on multicast.
Message queuing is a more complicated system, but it allows you to push the bulk of your work out to the distributed network instead of concentrating it on the producer.
You have to do this state management with a TCP-based system anyway - the use of TCP doesn't magically come with the full state of past messages.
By having a proxy/repeater/site server, you are reducing the amount of WAN traffic you need to send.
low cost end replicas at the end. if the bandwidth usage is half the cost in their multihop topology, 14XXXX compared to 16XXXX? depends on the number of proxies or replication and pricing.
https://github.com/mTvare6/hello-world.rs
I imagine this is used for Bloomberg (the terminal) and not Bloomberg (the website)?
going back to the article - fantastic animations. I'm just as curious to how that was made as the queue itself.
Of course they do, unless the processes are regulated by a country (such as handling highly addictive opiates in the US).
Then there is language variances like flavors so what ends up happening is a million different conceptual primitives that fundamentally the same.
It isn't the same because of a few things. At the end of the day too it is also about how good the support is over the rest. If a new MQ can demonstrate better tooling nothing stops a company from adopting it.
If anything, we should be thankful that they've decided to share it with the world.
It's a big driver of both complex and slow software. Often the adaptation layer necessary to get it to actually satisfy your needs rivals the size and complexity of the piece of existing solution code you're trying to leverage.
Software that is built from the ground-up to your exact needs is often both faster and slimmer and more maintainable and suffers from far less dependency churn.
In the original sense, the NIH approach would be to invent a rivalling technique to the message broker. That is indeed a questionable idea.
Seen Not Implemented Here too.
Well, no one cares, but I'm not saying that to be mean to dismissive, it's just that software production as an industry (esp. at a company like Bloomberg) can basically do whatever it wants because of low capital costs to produce something. Things like writing another queue system will have zero impact on the industry unless it's adopted widely (and cargo culted) and later made worse.
Maybe I'm about to eat my shoe, but if a capable of team at Bloomberg wanted to reinvent something considered unnecessary in terms of industry need, they probably could if for no other reason that management allowing some indulgence in exchange for keeping talented people around is Worth It™.
They do adopt a lot of industry stuff, but it is always heavily modified. And of course they love “Invented Here” as a corporate tradition going all the way back to the beginning.
It’s actually nice from a developer perspective, the R&D group is highly valued by management.
I don't understand that phrase. Probably a large team of very well paid engineers developed this product. Bloomberg is well-known in the industry for paying well. "[L]ow capital costs"? This software probably cost millions in salaries to build, and millions more to maintain over the next 10+ years.
>I don't understand that phrase.
You don't need a big factory with large capital costs to make or distribute software. You're only paying people, and salaries are not capital investments.
1. It is easier to implement from scratch than understand and modify: we see this everyday at a way smaller scale with libraries/classes/functions. Not saying this is optimal but this is a driver for sure.
2. Recruiting: some devs love doing this. Fancy OSS project to put on your resume, some actually believe there's unfulfilled need, you get people like us discussing it (otherwise we wouldn't be thinking about Bloomberg).
I've polled developers about what part of software development they hate, and in general, it's deployment and maintenance (ie: devops). Using a off-the-shelf-component means that the developer only gets to do the un-fun part: installing, configuring, productionizing. If they code their own solution, they get to spend a few weeks coding!
I am wondering if they ran into performance or resource usage issues with Kafka at scale, and decided to roll their own for use cases that better fit their workloads.
Bloomberg does have vast backend infrastructure for moving around / computing data which the Terminal is a consumer of.
If software is written 100% externally, but the vendor only wrote 10% of it and is transparently taking 90% from an open source project so they can sell fraudulent support (assuming they don't have people contributing to the open source project here), that counts as supported.
If it is written 100% internally, even if it is built on top of some ancient internal codebase that nobody can figure out and has been in life support mode for 20 years, then it counts as supported.
But if its written 50% internally and 50% of it came from an external open source library with a BSD license, and you are a major contributor to that project, then it is considered unsupported and you get in trouble.
This is how you end up with security issues because some sysadmin decided they had to roll-their-own crypto library or authentication system, because its better to have an unknown implementation show up on a scanner than a known one that has a list of CVEs that can be checked.
https://docs.nats.io/legacy/stan/intro/partitioning
The current standard is NATS JetStream, which absolutely addresses scalability, arguably to an even larger degree than Kafka, since it can be used to create a globally interconnected supercluster of clusters where messages can be configured to flow between regions as needed. I doubt anyone would run a Kafka cluster that spans multiple regions.
Stretch clusters and clusters with brokers in different AZs are absolutely a thing in Kafka. I make no comment on the ease relative to Nats, which I don't know at all.
EDIT:
Definitely sounds like there are some tight latency requirements[0]:
> A stretched 3-data center cluster architecture involves three data centers that are connected by a low latency (sub-100ms) and stable (very tight p99s) network, usually a “dark fiber” network that is owned or leased privately by the company.
100ms would theoretically be enough to span the contiguous United States, but the references to "very tight p99s" and "dark fiber" make me wonder if 100ms is actually acceptable, or just a theoretical maximum that can only be allowed under absolutely perfect conditions. Either way, suggesting the user should have access to dark fiber between their regions does not fill me with confidence about the robustness of this solution. I'm sure it would be fine for geographically close datacenters, as AZs are designed to be.
[0]: https://docs.confluent.io/platform/current/multi-dc-deployme...
Mostly because of the speed of light, not because Kafka is for some reason unable to form a cluster across regions but Nats can.
My main gripe is lack of “best practices”. You’re pretty much on your own when it comes to important decisions like subject namespacing and some operational stuff can be a bit tricky, like changing options and migrating. They had “nats by example” but that seems to be quite out of date and not super helpful. They have started doing more videos which are great but not that much content overall.
Their auth story is very sophisticated but perhaps a bit over-engineered. Some use cases like having anonymous “self-signed” client keys are tricky to work with.
Other than that, it’s an amazing piece of tech. One that is scalable in a way that just makes sense and gets out of the way when multiple machines are involved.
They're fundamentally different designs.
Garden variety NATS (i.e. the non-"Jetstream" case, hereafter) is "at most once" (messages will be lost.) BlazingMQ is "at least once" (>1 delivery of a message will occur.)
That difference is down persistence. NATS doesn't persist messages to stable storage. BlazingMQ must use persistent storage. From that all sorts of application design, performance and other implications are derived. And they all matter. A lot.
Seems situation dependent.
the github shows its codebase is in c++ with gcc-7.3, that means it might be using c++14 in the code, or even c++17, which is nice.
C++ and Java client examples are very simple which is cool.
Would appreciate if there is hook to enable `SO_TIMESTAMPNS` on socket and retrieve `cmsg` to get accurate timestamping and permit client code benchmarking.
> In a zero-replication no-leader-election (or all-in-one-node) scenario what is min latency observed?
We saw sub-millisecond latency in this case (typically double-digit microseconds).
Also, it's worth identifying and setting aside use-cases for broker-less messaging where 0mq, nanomsg, etc. could be more appropriate.
Edit: There's only published performance data of itself, which has little value: https://bloomberg.github.io/blazingmq/docs/performance/bench...
I know that's not an answer but if you're feeling adventurous you might be able to help the project.
I'd tolerate wrappers for certain things, but not for this thing.
Amazing to see a non-Rust product posted to HN.
[1] https://vittorioromeo.info/emcpps.html
> BlazingMQ is an actively developed project and has been battle-tested in production at Bloomberg for 8+ years.
https://github.com/bloomberg/blazingmq
I don't see anything that implies as such, all FAQ and comparisons to other message queues say nothing of security (I skimmed and ctrl+f so I could have missed it). This doesn't bode well for a network-based application written in a memory unsafe language.
edit: I'm surprised so many people are interpeting this as a weird cargo cult thing. Application security includes a lot of mundane and critical things like:
Authentication, authorization, message signing / authentication (e.g. HMAC), encryption, secret/key management, how the system handles updates, etc.
You can read a bit more about it here: https://en.m.wikipedia.org/wiki/Application_security
Your false dichotomy also implies that “unsafe” blocks are as unsafe as C++, which is not true. “Unsafe” in Rust turns off very few checks[0], most of them are still active. No one would write a serious Rust program entirely in unsafe anyways.
Regardless, asking about security considerations is a valid thing to do, even if it were written in Rust. Security is not just about memory safety.
[0]: https://doc.rust-lang.org/std/keyword.unsafe.html#unsafe-abi...
Having parsers (and serializers) proved absent of runtime errors (e.g. with something like SPARK) is a form of guarantee I wish I'd see as the default in any 'serious' network-facing library or component. It's not even that hard to achieve, compared to the learning curve of the borrow checker.
Once rust gets plugged into Why3 and gets some industrial-grade proof capacity, the question of 'is it written in rust?' will be automatic (as in 'why would you do it any other way?').
Also, it's a super bad-faith argument to talk about an entire program being in an unsafe block. Rust's unsafe is about specifically and intentionally opting out of certain guardrails in a limited scope of code where it should be fairly easy for a human reviewer to understand and validate the more complex invariants that aren't enforceable by the compiler itself.
https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html
"you can give up guaranteed safety in exchange for greater performance "
I mean, I don't often see documentation for networked services that goes in-depth on how exactly they write memory safe code in C or C++, as if that would mean the application is therefore secure; it's always much higher level.
I'm personally interested in both, but from a docs perspective I'd mostly expect the latter, more "security for users of this system" than anything else. I'm a little embarassed I didn't remember it until now, but I think the general term is "application security."
I'm concerned that I don't see any hints of either in the project. I'm not at a computer with access to my Github account so I can't easily search the source code for hints of obvious signs of care or neglect with respect to security.
Not sure where they are on the roadmap but authentication in particular is 'mid-term' but lacks detail.
I expect that if you want to deploy this then you're doing it on an internal network; its deployment examples are based on Docker so I expect it's relying on what you can do with k8s.
It looks like it's an after thought but at least on their mind now, which is very fair with respect to Bloomberg's wants/needs. It'd be nice if they had a bit of a warning about using this until it has some basic auth(n) and TLS since they're releasing it to the public. I think it is, relativley speaking, rude to release insecure networked software without giving users a notice as to what sorts of insecurity is at least known/expected.
And if you believe that, you're wrong
There's a hefty load of cargo cult mentality revolving the topic of security. Security is not a product, but a process. This sort of talk is getting tiresome.
This may be perfectly safe... however, I'd give it a few years of battle testing before touching it.
Here's an example:
https://github.com/bloomberg/blazingmq/blob/main/src/groups/...
Also they do make heavy use of managed pointers in the codebase.
Just because software is heavily used and deals with finance, doesn't mean it's secure or well written. Consider the Equifax breach of 2017, everyone's credit info leaked because Equifax was using a crusty old version of struts with known critical CVEs.
"Every byte counts" was the motto of the team that designed it.