Honestly, I've only skimmed the article, but this comment rubbed me the wrong way right off the bat. The article (discussing use of Kafka) doesn't seem to have any direct relation to WeWork's failure.
There is a lot that you can learn especially from businesses that failed.
Just because the company wasn't successful does not mean you should discount everything related to that company, especially when the content and lessons are technical, and the business failed for non-technical reasons (shady business practices and self-dealing).
Just getting in here to say someone should edit the HN title to have a date. Hearing about success at WeWork now is akin to being in accounting and finance in the late 90s doling out praise for Arthur Anderson.
The general trend in these posts (and tweet storms) is the same - pick any technology and then preach the wisdom of not using it because "You're not Google", or "You won't need it". It's a great tactic as it puts the speaker in a position of judgement, and they can freely share their opinions on some architecture, without getting any pushback from the creators of said architecture. Quite a one-sided affair that makes you look like you know everything. It's, frankly, elitist.
This entire article can be surmised by "You don't need Kafka because the throughput seems to be low". To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka.
But of course, "You don't need Kafka" because <reasons>.
> To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka.
WeWork has very predictable scale because its data ingestion needs are tied to expensive investments in physical capital. They don't have to worry about getting slashdotted. Same thing at my company, where data needs are tied to expensive enterprise contracts. In fact, we're considering Kafka for something we do, but it didn't sneak up on us--we knew for years that a particular use case involved a lot of data and wasn't ideal for a relational database.
Why not see the coin from the other side. Developers giving their experience when they are working sane ?
Many of these articles, I have seen authors quoting from their experience building large systems.
Moreover, as a system architect, you often have to analyze the requirements and choose a good stack for implementation. A young dev in a startup is biased against the architecture in one way, because they might be seeing Kafka on their resume as a shining star to greater riches.
The interesting part is that indeed, the people who write these articles are usually building large scale systems, writing books, newsletters and building companies for them, and then telling everyone else how they don't need it.
Of course engineers have a tendency to over-engineer solutions. I don't think anyone is surprised by that. But this "oh you don't need it, I've used it at <bigcorp>, and you're not that" thinking just rubs me wrong. Especially when it's backed up by hollow arguments like the ones in this post.
Since this is going to get some attention and might attract some welcome naysayers, it might be a good time for me to ask for resources if I do want to use Kafka, specifically edge cases that I won't see as a newcomer until I actually try to use it in production.
I've already done my due dilligence and I consider Kafka to be one of the potential solutions I could use for a service, and while I've gotten started using it fairly easily in a dev env, there's obviously things I could miss. Any pointers from experienced users would be appreciated.
I just learned Kafka this year to augment an existing system at work. One thing I have found is, there's a good amount of smarts in the Kafka clients. Look carefully at the Kafka clients available for the runtimes you work with, and evaluate them for robustness, features, and development activity (sadly, for example, kafka-node seems to have lost all momentum).
Basically, if you have an unbounded data problem, and you might have something that is _practically_ unbounded, that book really helped me gain insight to describe what we used Kinesis for (which is really similar to Kafka in a lot of ways).
Thinking about event vs processing time, ordering and windowing requirements, etc, were really helpful in thinking about my own data problems.
In the end, the author of this article claimed that WeWork just simply didn't have much of an unbounded data problem. Which I generally agree with. But simply saying "you can do this in PostgreSQL" isn't really a great takeaway, and, it really feels that was the message being reinforced here.
Kind of a lengthy ad hominem against WeWork, no?--and in an alternate reality where they did in fact have an IPO.
I love Postgres but "just use Postgres for your small data" is IMHO a facile argument to an ambitious, naturally distributed IoT project. The fact that WeWork's IoT strategy is/was perhaps useless is rather beside the point.
When you have a monolith database that everybody wants to access Kafka with CDC is a very good pattern for teams to adopt.
The alternative is modifying your app for arbitrary data access and dealing with auth or giving direct data access.
Giving direct access to your database is fast but leads to very strong data coupling.
Exposing the data through the app layer is slower than cdc kafka integration but comes with a long term commitment to support that access. I have seen these type of data access queries easily hit performance edge cases.
There are other pros and cons but those are the ones that end up being the most troubling.
Often a single apps data can be incredibly valuable and many teams want to access it. So coming up with patterns that
1. Protect the app
2. Protect the data store
3. Decrease the maintenance cost of the app
4. Decrease maintenance cost of the datatype
5. Add sight complexities from kafka.
> Giving direct access to your database is fast but leads to very strong data coupling.
Or you could just give "direct" access to (a DB role that only has been granted SELECT on) a single schema within your database, entirely consisting of views. Then you've got a stable data-access API that you can iterate your data model behind, with nothing more than your DB itself.
(And you don't need to put much thought into setting it up, if you don't want to; your stable schema-API-layer can just start off as a bunch of views with 1:1 relationships to your real tables, which each just "SELECT * FROM their_underlying_table".)
Maintenance costs of any software should be taken into consideration.
I can say that we've had our troubles with kafka sure. But app teams 100% love working with it instead of having to have familiarity with all the various data sources we have.
Working with all sorts of different data but all of its kafka is much easier than all sorts of data from
Many data sources.
The app teams i've seen loving it were app teams that siloed off ops but hey maybe your use cases were not mine.
I'm seeing too many places using it in some form of RPC style system as part of some "workflow" orchestration magic system with unicorns and ponies for everyone
I once looked into authen/authorization with Kafka topics and whether one can only be allowed to subscribe to specific ones and came up short with any solution to how to do it with Kafka. From my POV, points: 1 to 4 listed above now have been moved to one single point (Kafka) and its capabilities are a lot limited than those other tools/entrypoints (of data access) mentioned. Have there been new development to Kafka's IAM?
I mostly disagree, but in a native implementation you probably do have strong data coupling. You can have cdc feed to another system that performs enrichment or transformation.
Cdc also enables all sorts of non obvious changes that are infra focused. It's much easier to update your version of sql server, db2, mongo, etc. If you don't have to worry about every other app and how it connects to your shared service.
You may think that you're being smooth by updating a A record or Cname when doing a migration but very often there's an app out there accessing using an ip.
Kafka is (IME) pleasant to use for people writing application code. It can handle high message volumes. It has good documentation. Its core abstraction is easy to understand and use.
OTOH, the author is correct that it's more work to set up and maintain than non-distributed software. Don't introduce it unless you have carefully considered alternatives.
I became de facto responsible for running a small Kafka cluster at a small company that could have gotten away with something less shiny. That was not fun. (Especially since the company started with Kafka 0.8 and encountered many changes along the way to modern releases). Eventually the company ponied up for Confluent to take over running Kafka for us as a service. Now I'm back to enjoying the nice Kafka APIs without having to deal with the annoying Kafka servers.
I like the Kafka API/capabilities but dislike running actual Kafka servers. If I were to start a one-man venture that needed event streaming I'd probably try Redpanda [1]. Its developer hangs out here, which is how I first learned of it. It's supposed to implement Kafka service but with no Zookeeper or JVM knowledge needed to run it. I wouldn't even consider running the original Apache Kafka for a solo/small team.
the tl;dr is that we wrote a pretty fast fragmented buffer parser (due to our thread-per-core design), we consume some json files and codegen it.... which basically means, if you connect an existing app that uses the kafka driver and it doesn't work it's a bug, and it's a matter of teaching the codegen to fix it most of the time. Transactions should land in the next month or so, which is the only thing really missing today.
we have a ton of solo devs typically that come from the Go, Rust, JS & Python seem to like it quite a bit because of what you said.
this. I'm working on a project which will need to be deployed in a local network without internet access. deploying kafka is a nightmare, more so in docker or k8s. I ended up switching to other tech rather than undergo this
I always find these posts ironic, because I work at a company where we are often times dealing with the opposite problem; trying to redesign systems that worked fine when we were a small startup but have hit scaling issues now that we are many times the size.
People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Now people are relying on your systems, and you can’t have any down time. All redesigns have to be able to be implemented in a way that can transition from the previous system without breaking anything. As we often say, “we are changing the tires while driving 90mph down the road”.
I wish we had spent more time on thinking about scale when we were smaller. Our new mantra is that everything we make now has to be designed to horizontally scale, even if it is something small right now.
I have had the same issue, and wishing we thought about that up the road rather than down the road is exactly the feeling when we huddle up for team meetings on optimisation. We could not have any down time for most days of the year.
Everybody at a company experiencing scaling issues feels the way you do.
The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.
Yeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing.
It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time.
Still wish we had made some different ones, though!
I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price.
(If you're reading this and you know who you are, I'm sorry!)
> It's a rule of thumb that hasn't led me astray yet.
How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
Meh. I read the article, and I agree with it. But that is not what you are saying.
Don’t use it for relational data. Don’t use it as your primary data store.
At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc
> What’s missing from MongoDB is a SQL-style join operation, which is the ability to write one query that mashes together the activity stream and all the users that the stream references. Because MongoDB doesn’t have this ability, you end up manually doing that mashup in your application code, instead.
Even with a normalized relational database one wouldn't make a single query to return rows from a join of tables T1, T2, T3, etc when there are repeating relations such as the users in the social graph. This might be done sometimes paying for the duplication to save a network round trip, that's all and no different.
The deep relations (7-table join) example in the link is something I wouldn't do (including the repeated 'user' entities) with an RDBMS, given that I expect some elements of some tables to appear multiple times, that's a waste of I/O and hard to cache.
> Users have friends, friends have posts, posts have comments and likes, each comment has one commenter and each like has one liker.
I would likely gather all the instances of User id in each context (user, friend, commenter, and liker) and query it separately from the JOIN query. Depends on your data which way works out better on average/worst-cases.
> I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road.
Yeah, that can't end well. It's a bit ironic from parent poster's viewpoint but "boring", well-understood technology is generally also the easiest to migrate away from. That's one key reason to choose it at the outset even over more "innovative" and perhaps more "ergonomic" choices like mongo, that in reality aren't nearly as simple to deal with.
The key part you highlight is that of hiring. If you want to be scale ready from day 1, you may not find the scale out talent because they either are working on large scale problems or would want to work on large scale problems. In your early days, you are mostly hunting for a product market fit.So your problem statement will not even be relevant to the hires.
This is the best case, the worst case is that the scale out thinking hampers your product market fit progress. That's the risk you run.
So, while it seems to have gone scale out on day 1 in hindsight, it's always always better to start small and scale out as needs arise, even if that means re-architecting the whole stack. It's painful, but doable. It's also a reasonable challenge to entice new hires. You have scaled out a system from degree 1 to degree m. Now scale that out from m to r++. Something like that.
On balance it's probably more important to create (a) stable public APIs backed by tests and (b) an automated deployment process than to get the internal implementation right from the start.
These two things help any system at any stage. They also make it easier to handle migration if that's necessary.
So, Let's Encrypt just uses a very big piece of iron for their database server, they don't scale horizontally.
Stack Overflow also famously buys the biggest hardware they can get and upgrade their database server accordingly.
I always wonder if horizontal scaling is really always the right answer, when you can just buy a bigger box (of bigger VM) and spare yourself a lot of issues.
I am not a proponent of scaling out horizontally as the best solution to all things, but being able to do so comes as a result of design that helps to minimize coupling of services and single points of failure, in general, which are other benefits.
But I also tend to think "you can always get a bigger box" can take you pretty far, and much further than most people that only think in terms of AWS realize.
Up to a certain point, sure, you can vertically scale.
I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that.
There is a limit to how much you can vertically scale.
Even for the things we CAN vertically scale, we need to be able to handle hardware failure and OS upgrades without taking any downtime, so you are going to need to be able to horizontally scale enough to handle that.
I agree with Jorams, this is not a common service. Most apps are things with an app server and db and most of all: they have a load that is easily handled by modest gear.
I know, and I get that we aren’t the average tech company.
I don’t think we are unique, though, and there are many engineers on here who work at all sorts of interesting places with non-standard workloads. I was just pointing out the reasons we can’t vertically scale.
StackOverflow seems to be doing well keeping the number of servers small, and having one dedicated hot failover for each machine.
Today you can without issues have 128 cores and 1TB RAM in one server (or 48 cores and 1.5TB memory if you prefer Intel). Of course not everything scales perfectly, but you can get pretty far before you are really forced to scale horizontally.
I don’t think the size where you need to horizontally scale your database is as ‘mindboggling’ly large as you make it out to be. There are a lot of engineers working at a lot of companies who are at and above that scale.
Most websites (probably 99.9% of websites) never reach that scale. And I suspect most working engineers work on products that aren't at that scale. I mean, how many websites, anywhere in the world are bigger than stackoverflow? 100?
There's interesting tech out there, but I think adding "interesting" technology to your stack without a very good reason is usually a mistake.
True. And likewise, not everyone needs an ETL pipeline.
"Keep your architecture as simple as possible, but no simpler". There are important problems kafka solves. But its worth checking if you have one of those problems before using it.
I think many engineers underestimate a well-tuned postgres with a good memcache layer. Most of your traffic likely won’t even hit the database. And a good ORM supports this out of the box.
Scaling the writeable DB horizontally is the hard part. You can get pretty far with tuning, bare metal (if load isn't too lumpy), many read only copies, caches, and other boring solutions.
That's a bit of survivor bias: Most startups don't go from small to large, or even really medium. Most startups are strapped for resources. Paying an extra engineer, or taking valuable time away from an existing one to implement an extremely scalable infrastructure that might not be used. They should focus on shipping a good product that people want, and getting people to buy it. I can't think of startups that failed because their stack needed to be upgraded for scalability. It's generally not why startups fail. I can find a whole lot who burned through their cash before they had a chance to need something orders of magnitude larger in scale than currently necessary. If it's a minimal cost in time & money? Sure, go for high scalability. But it it's going to reduce your runway significantly then worry about the bigger problem of becoming successful enough to need it in the first place.
If a startup is really concerned about avoiding the problems you're encountering at your company they should just take an extra week when planning their infrastructure to map out details of how they would transition to a more scalable infrastructure. Then save that file somewhere and dust it off if they ever need it. And if they need it, they'll have ample investment money coming their way to pay for the transition.
My last 3 consulting gigs have all had the same general issue - they went with the "quick to start up", scalable solutions and immediately smacked into a brick wall of complexity. These are 1 to 3 person eng teams that started with serverless functions, NoSQL databases, and event-based architecture and not even 6 months down the line had to hire an expensive consultant to shake out their architecture and point them at something less scalable but more practical to build on at their size.
If they didn't have VC, they'd likely die on the spot, drowned before they even had a shot at real product-market fit under the weight of their own accidental complexity.
Now, is there a set of common patterns that works well enough for startups to iterate quickly with, that _also_ scales? Probably. Can you be successful with NoSQL et al out of the gate? Almost definitely. Does it make it _much harder_? I have three anecdotal data points that point to "yes".
From my work over the last year, I'd suggest "choose boring technology" over "worry about scaling early" 10 out of 10 times. You need something that scales _conceptually_ before it can scale in the cloud, if you don't know what your core business value is or how your product actually works it's really damn hard to build a cloud-native app that anyone can actually iterate on. You're fighting incidental complexity every step of the way.
Maybe in 5 or 10 years, when the average developer has a wealth of experience in scaling technologies, when serverless functions have fewer dramatic downsides, when we start seeing out-of-the-box event driven full stack frameworks, when the bleeding sharp edges are sanded off NoSQL and we've figured out a good story around schemas and validation, then it starts to look more attractive for young companies. Until then, though? Stick with the boring stuff.
EDIT: Though, re-reading this, I suppose I should acknowledge my own survivorship bias - the companies that figured out their scalable, cloud-based tech stack early have no need for my consulting services...
Great comment. This mirrors the ridiculous 'big data' fad of a few years ago, where startups would deploy Hadoop clusters to crunch through Terabytes (or even Gigabytes!!) of data ... a task that could have been done much better by a well resourced monolithic system that was easy to implement, and easy to maintain and understand and just worked.
There is no startup in the scaling phase that doesn’t have stuff that needs to be rewritten.
If your startup’s product is constantly iterating to find product market fit, it’s a fool’s errand to keep making sure everything is perfectly horizontally scalable in the first place. It’s very rare that the things you expect to be important early on actually are in the product that gets popular.
You’re absolutely right to change your mantra now that you’re scaling, but you were right to not do it early on. The difference between life an death in an early startup is quick iteration.
I wonder what the ratio of failed startups that didn't engineer for scaling early vs those that did is. Having to rewrite parts of your application to match scaling issues seems like a great problem to have.
There's two very different ways to interpret the "don't worry about scaling" stance and I think that's where things go sideways.
On the one, people take it as literally "don't worry about it, just make it work" and don't think through simple decisions.
On the other, with some basic analysis you can identify where your likely bottlenecks are going to be and make some preparations to dramatically simplify scaling when you need to do it. Sometimes minor architectural decisions, that don't take much time, can lead to a happier long term life.
But I 100% agree with designing for horizontally scaling.
It's one of the reasons I always try to steer people in the direction of Heroku for new projects, because while it reduces your ops burden and makes your life easier...it also forces you to design for horizontal scalability from day 1 via 12 Factor.
That’s how I take this, with a related point of thinking about how you’d replace components when the time comes. Maybe you have a monolith-leaning design but you have tests, a clean API with some idea of ownership, etc. rather than a bunch of code directly making SQL calls, copying the same logic instead of using a function, or touching the shared files so when you do split things out and redesign them the process is safe.
(Having flashbacks to the same code in 3 PHP repos, maintained by one person, no two copies identical)
At my work we’re one step past where it sounds you’re at. I’ve been around since the very early days so I’m both part of the original problem and the solution. Now our biggest problem is we don’t like to work with the things we built to handle scale. I’m not sure how common that is but I don’t believe we’re alone. A lot of the problem from my point of view was trying to chase trends.
We have Kafka in places the article’s author would suggest are inappropriate. We have a lot of “one true systems” that hid horizontally scaling instead of making it easy to build clusters of micro-services. Now we’re riding micro-services pretty hard and that isn’t going well since we have too many mixed architectures in use.
I think it’s correct to push out worrying about scale while you seek product market fit. I think that once you do need to scale you really need to put thought into the process. This is difficult if everything is on fire and failing. If it’s not, burn some time to plan. It’ll pay off. Be sure you don’t reinvent things you can cheaply buy. Make sure developers still like their day jobs after the changes are made before making more changes with the same underlying design theories.
As companies grow there will be "pain points" that come up due to scaling (performance, architecture, development, etc). When you get to that point it's normal to wish the pain didn't exist by "doing it right" the first time, but I think it's still better to wait until you have the pain. Because that pain tells you unequivocally what you need to value in the scaling.
Yea, I don't think it's always a waste of time to think about scale at the start. You don't have to do all the work up front, but you can definitely put thought into doing the early work in a way that will make scaling easier later. There is no sense in committing to an architecture early that limits your options later. Also, avoiding sloppiness and ensuring simplicity up front usually pays dividends later.
Under-scaling and over-scaling are both suboptimal for all the obvious reasons.
Middle-scaling is also suboptimal - you’re paying too much upfront and you don’t get enough later on anyway.
Stage-appropriate scaling is the smart way. Scale as you can see the need. This requires willpower - you will be pressured to kick things down the road just one more time, every single time.
The path of the wise is generational design. Build every component such that it can be easily destroyed and replaced. Anything that mustn’t be destroyed should be scrutinized and planned for in advance. Store your data in most mundane formats so it’s easier read. Keep interactions simple and need-to-know. These are platitudes, you will need to fill the planned obsolesce roadmap for yourself. This unites your hands make the quickest crappiest implementation at the moment secure in knowledge you can fix it or scale it as needed later.
>People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later.
Well ... you're going to have to. You don't know what bottlenecks you're going to hit as you scale up, so why spend precious development time trying to guess which systems to optimize? And to be clear, 'cloud-scale' architectures are complicated and bring their own set of issues - why subject yourself to that?
Plus a well designed (monolithic) system should get you very far with vertical (resource) scaling and/or data sharding.
Like suggested by many people that the number of companies will not reach that scale, so starting small and simple is the most efficient. There are few things in mind that may help :
1. Define the domain segregation and be discipline on the implementation. It will not only help ease the process when the system need to be segregated into multiple service, but also help on organizing the team.
2. After segregating the domain, even if all of them live in the same database dont use `JOIN` as much as possible. Define the API for each domain and use it as the main communication.
3. As boring as it sounds, use boring technologies. Pick framework that has been used by many like Jango, Rails and Laravel. They offer not only tools and libraries but also programming pattern that will help us to implement domain segregation. The framework I mentioned have good established (way to work with if not integrated) ORM, event driven method, front end development and means to scale.
4. Usually postgreSQL is enough and can bring long way, just remember point 1 when designing the DB.
I think those points are quite straightforward and doesn't make the development more difficult, while trying to be future proof.
There's horizontal scaling, and then there's horizontal scaling.
A lot of software assumes a single server is fine and as a consequence doesn't even bother trying to abstract that away; the singleton nature of the software then percolates virally through it all.
On the flip side, some software actually isolates their scaling limitations and pushes the bottleneck into a single part of the system.
With the former, you're screwed if you hit your vertical scaling limits (to say nothing of what your system availability probably looks like, if scaling is actually a concern). With the latter, you can play games with your scaling, perhaps by shoving a hierarchical or replicated cache in front of your scaling bottleneck, and effectively buy time to build a solution with better scaling. (Or perhaps never bother with that, and just be acutely aware of your system's scaling dimensions, like StackOverflow.)
There's no such thing as a system that truly scales infinitely, just systems whose scale limits exceed our imagination.
(Admittedly, most bugs of the form "INT64_MAX overflow" are generally a leak and not the result of growth :).)
Spending 5x the effort to redesign when you have 10x the money/resources isn't an unreasonable tradeoff. Since the vast majority of early stage startups fail spending money on things which don't matter until they succeed is not a good business decision.
Of course, WeWork at this point had 1500 engineers so calling it a small startup or using it as a case study (either way) for small startups seems a bit silly.
People always assume either there would be high-velocity development or high-scalability architecture. In theory, this would be correct at the end of the day. But in fact, we're far from that equilibrium. For example, here we have these two major players:
Ruby: High-velocity development; Decisions have been made so you don't need to make them; Suffer from scalability issues both performance-wise and architecturally in the endgame.
Java: Much slower development; Could be fast if correctly architected, but doomed as well if done wrong; Require a hell of infrastructures to work well.
But you can also find some superstars among the minor players without many trade-offs, for example:
Elixir: Development velocity and decision-wise are very close to Ruby; Performance is on-par or faster than Java without decisions; Messaging and cache already baked-in the language and tech stack though you don't need them in the beginning; Hire Ruby programmers and just start rollin'
Amen. I've been through some interviews where I'm the oldest at the table and they tell me their stack and I try to keep a straight face.
It's hard. Hard to keep a straight face because I'm a genuine honest person with a lot of experience.
"We plan to hire 15 people within 3 months and double year after year".
Well, yeah, with every acronyms AWS could come up with, using them all for no reasons at all (except S-C-A-L-I-N-G), you could build it all rock solid on a mid-size linode and do it with 5 dudes
Expected: world domination, runway blast-off to the moooooon
Actual: as many employees as concurrent users, grind for the next 2 years and run out of money
> I've been through some interviews where I'm the oldest at the table and they tell me their stack and I try to keep a straight face.
> Well, yeah, with every acronyms AWS could come up with, using them all for no reasons at all (except S-C-A-L-I-N-G), you could build it all rock solid on a mid-size linode and do it with 5 dudes
I've been through some interviews where I was the youngest person at the table, but it otherwise sounds like I had a very similar experience to you (they had ~1000 total users - their "scaling issues" were entirely due to poorly architected code). In my case, I politely questioned it during the interview and they actually ended up letting me take over and simplify things.
It wasn't about the cost effectiveness, it was about the complexity added of 20 different products names cobbled together in a chart representing a tech stack. Sorry that wasn't clear.
Yep. WhatsApp scaled their software with over 10m users per engineer on their team.
I read an article a few years ago about how StackOverflow ran (still?) their entire site off just 6 physical computers or something. And before you claim its because they have hand written assembly everywhere - nope. Their code is just clean C#.
I adore Kafka - I think its a lovely way to build scalable software. But almost every website out there will never need to scale past what a simple postgres + rails/nodejs/whatever + nginx setup can provide, assuming each of those pieces is set up correctly. And you personally probably aren't working on software at that scale.
I personally also suspect we can make the experience of building on top of event sourcing better than the experience of building on top of a traditional database, because decoupling reads from writes gives you more architecture options. But I don't think we're there yet.
WeWork had about 1500 engineers at its height (the rationality of that is a separate discussion). At that scale the limitation isn't having to maintain some technology but rather decoupling interfaces so communication and bureaucracy doesn't block engineers from getting work done. You can use Postgres but then getting access, schemas, resource management, coupling, and so on become the limiting factors on usage. Paying for 5 more engineers to maintain Kafka and unblocking 50 in the process is a net gain.
I last used Kafka in 2015 and I really miss it. Used it as a message broker for loads of cellular IOT devices. It was critical for helping us extract services out from a Rails 2.3/PostGIS 1x monolith, especially its ability to store messages for a period of time while a consumer was offline.
It used to be that distributed systems were a big trade off. They were operationally complex, they had limited apis (NoSQL), but they scaled. The best solution used to be to build things using a non-scalable but easy to use and run system, and then re-write it later if it needed to scale (often in a big hurry).
This is just not the case any more, though. Why? Two reasons:
1. We’ve gotten much better at distributed systems so the apis aren’t nearly as limited. It’s no longer that you either choose hard-to-use things like Hadoop/NoSQL or elegant but unscalable single-server databases. You can have both good abstractions and scale.
2. The cloud makes it possible to get systems as a service so there should be way less ops than running a single node system yourself
In the case of Kafka, I’m super biased as I’m one of the original authors, but I think the abstractions Kafka gives, stream processing capabilities, connectors, etc are just way better than a lot of the traditional solutions. Using something worse until you “need” Kafka might make sense on premise, but not in the cloud.
Confluent offers a Kafka service which is fully managed so you don’t do any of the upgrades, security patches, midnight pages, etc you just use the APIs. This is super affordable for the kind of simple apps the article describes. The price varies by cloud, but e.g. on GCP it starts at $0.11/GB for reads and $0.10/GB stored. That is a lot cheaper than using a single node system and then rebuilding everything if you need to scale, but not only that, it is also lower operational overhead (effective none) and a better interface/abstraction.
I think this isn’t unique to Kafka, either. There are great managed systems that are built to scale for most of the kinds of data systems you would use—-CockroachDB, Spanner, Aurora, Snowflake, Elasticsearch, Bigquery, etc.
Basically, you can have nice things now, just like the big tech companies.
The problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing:
- Auto-scaling: Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling).
- Multi-tenancy: Planning for a multi-tenant kafka cluster is not for the faint of heart. Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands. This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated.
- Decoupled brokers and storage: Any broker scaling or failure can lead to downtime while event storage is redistributed.
Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc.
Kafka is an industry standard, it's benefits go well beyond its own qualities and extend into the ecosystem around it.
Don't attempt to replace industry standard components with any home rolled systems unless you were already capable of building said system yourself and you understand why you are a special snowflake and why you need something special.
Can you make your point without the appeal to a (missing) authority? There is no standards document I've ever seen which names kafka explicitly.
Kafka has some benefits (lovely API) and some tradeoffs (hard to maintain, zookeeper is a bit of a mess). But there are obviously a lot of other ways to build successful, scalable software. WhatsApp was implemented on top of Erlang, and scaled to over 500m users when they had only 35 employees.
I want to disagree with your claim that kafka should be considered the default option - but you haven't dignified your (controversial) claim with any actual arguments. There's nothing here to even disagree with.
It should be considered the default option because everything works with it out of the box and for pretty much all Big Data folk it's considered table stakes.
All major streaming event processors are designed around it or work seamlessly with it (Spark, Beam, Flink, Storm, etc).
There are pre-existing integrations for all long term storage options (S3, GCS, Swift, etc).
Maybe it could have been elaborated in the original comment but it shouldn't need to be - all of this is about 10 minutes research away from anyone that wants to understand why Kafka is considered the core of modern data architectures.
WhatsApp isn't really a relevant counter-example and if anything reveals the misconception yourself and many others hold about Kafka which is that it's a messaging solution. It's not. You can bastardize one on top of it but really, don't. WhatsApp was built around RabbitMQ which -is- a messaging solution (one of the more complex ones, crazy complicated compared to Kafka at least).
Kafka -doesn't- compete with RabbitMQ any more than MongoDB competes with PostgreSQL. Apples and oranges if you are trying to do anything but the most basic stuff.
Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?
> Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?
Very few people roll out of bed and think "my application needs a high performance distributed event stream". For every problem that can be solved with kafka, there are other solutions that don't involve kafka. Many of the alternate solutions don't involve event streams at all. Are they better? I don't know. That question is unknowable without looking into the problems in particular, and looking at what other technology is out there. And I say this with a lot of fondness for kafka. I really like it. I just think there are always technical tradeoffs to make.
> any more than MongoDB competes with PostgreSQL
MongoDB and Postgres do compete with each other though! They work differently - so part of the competition isn't around features; its around how your people think about your data. "Think of your data in a way that makes sense for mongodb!" say mongodb sales reps. "Think of your problem as a message queue" say fans of rabbitmq. They aren't wrong in the general case. But there are lots of problems where I could use kafka, or I could use redis, or I could use a naked postgres instance. You need a much more complex heuristic to decide what technology to adopt than "Should I use kafka?" -> "Yes."[1]
And I say all that as a big fan of kafka. I think its API is delightful and just today I was plagerizing its API design to sort out backpressure in a system I'm building.
> What could you reasonably advocate for in its place?
I think we are agreeing on a different point than I was making.
Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Though you can shoehorn it into the same problem space, this is what LINE Messenger did which is of equal scale to WhatsApp and runs entirely on Kafka. So I completely agree Kafka isn't the solution to all A needs to talk to B problems, even when Kafka might be able to do the job.
I think what was perhaps lost is that I am not talking about adding Kafka to an arbitrary application but rather operating a Big Data pipeline - i.e data from applications, devices, sensors, whatever and transporting it to data lake, warehouse and online stores like Druid.
While you can use Kafka in other contexts this is it's original purpose and main use case and where I don't think any other piece of software replaces it right now. Pulsar is on the way there but not quite ready yet. Redpanda is just Kafka without being the real thing - maybe you save some CPU, maybe you hit an incompatibility, really it's not worth the effort because Kafka in 99% of use cases is network bound as it's already efficient enough. (Not always true, there are some CPU bound cases with tons of producers but they are rare)
If you chose anything else you are signing up for writing all of these integrations yourself and that just doesn't make sense. If you pick Kafka you get all the hookups for free and you only need to write custom code for your existing custom applications.
If say you were to want something more "lightweight" sure you could use Redis PUBSUB but now you are writing code that dumps from Redis PUBSUB into S3 for archival instead of just pulling something like Secor or Kafka Connect and being done with it.
Perhaps I should have been more clear that I think it's only in this specific domain/use case that I think it's completely dominant.
redpanda is not just a performance (latency) gain. The main thing people pay us money for is not performance, it is operational simplicity - no zk, no jvm, nothing to tune (https://vectorized.io/docs/autotune) . one binary you can `scp` around a few servers is really quite easy to work with.
the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.
I get that but often times you need or want ZK anyway. If you are running Druid, Pinot, Hadoop or anything else Big Data they generally have a ZK dependency. If you need service discovery, consensus or config storage ZK is generally great for all of those usecases too.
I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)
Aside from pissing all over WeWork, which given their immense losses they probably deserve, This article could have been summed up in a couple of short sentences.
There's a lot of words and effort going on here to make Kafka sound immensely complex (and dare I say Kafka-esque?)
The article continually chides Kafka as overweight, hard to setup, complex, blah blah blah. None of these things are true in modern day deployments.
Three instances of Kafka, Two ZK instances, and you're off and running. All of that will run in docker and the docker images will stand themselves up if you let them. You need even less for developemnt.
Postgres is a bit more complex given that you've got to manage machine sizes and database schemas.
But even with that being said, the article gets one thing right. For average data use, most any system will do and some startups are over-engineering their stacks.
You don't need Kafka if you don't know (and I mean really know, because it's hurting) where you need it.
But if you do know where you need it, you wish you had it yesterday.
This is the way with scaling in startups. Product-market fit and agility and lean operations is what you want early on, and scaling should come later. Big architectural decisions are hard to reverse and committing to some pattern that needs something like Kafka, like event sourcing, is probably too much risk for most products. Most event-sourced CQRS implementations fail.
I say choose Kafka somewhere in the middle, when scaling needs are somewhat predictable and product market fit is good, but try and use Confluent's hosted service or something similarly light on ops. Do non-critical things with it first and ramp up. Don't build big from the get go - that rarely ends well.
As someone who is currently working at enterprise where a migration of Kafka from old to a newer version is taking place. IMO this whole Kafka thing seems quite unstable compared to RDBMSs where migrations are relatively painless on application code.
Kafka is causing huge churn in applications codebases. I know that "best practices" experts will tell our company that we are doing it wrong. However to me Kafka seems a clever algorithms circlejerking and answer to operational complexity is either go to managed cloud or go to hell.
So you set up your Postgres instance, great. Then what, you hope it doesn't break?
Master-master Postgres is significantly fiddlier and less well-tested than Kafka. And what do you gain? Fiddlier APIs and a query language that makes it easy to deadlock yourself with no warning or write queries that will run for months.
Just use Kafka unless you've got a good reason not to. It gets a lot more of the basics right.
To really appreciate the ludicrousness of what WeWork is doing, you have to take that example even further and assume that WeWork needs to track every single working adult on the globe clocking in and out.
That's about 100k events per second.
You can handle that using TimescaleDB running on a single Azure DS4 Standard node.[1]
Maybe this already exists, but it would be nice if there was a library that provided a uniform streaming interface that allowed you to use Postgres as your stream processor (as the author proposes) and then as you scale seamlessly migrate to Kafka (or another stream processor).
This could be achieved by good architecture and system design.
You should have various aspects of your application configurable and independent enough to enable you to develop an alternative system and make a small downtime migration.
Many of scaling issues I see in startups, and stories told by devs in startups, seem to be because corners are cut to make the MVP and push it to the market as quickly as possible, and then suffer scaling woes due to poor design and architecture choices.
Are VCs really so incompetent to understand when a company is being hyper dramatic when it talks about "Kafka for IoT", when it is essentially a real-estate leasing entity ?
128 comments
[ 2.9 ms ] story [ 189 ms ] threadThere is a lot that you can learn especially from businesses that failed.
Just because the company wasn't successful does not mean you should discount everything related to that company, especially when the content and lessons are technical, and the business failed for non-technical reasons (shady business practices and self-dealing).
This entire article can be surmised by "You don't need Kafka because the throughput seems to be low". To which the retort can be many: they might be building for the future - to integrate more systems, they might have fan-out use cases, they might want to have a log to replay in case of disasters - so on and so forth. All well documented and very valid use cases of Kafka.
But of course, "You don't need Kafka" because <reasons>.
WeWork has very predictable scale because its data ingestion needs are tied to expensive investments in physical capital. They don't have to worry about getting slashdotted. Same thing at my company, where data needs are tied to expensive enterprise contracts. In fact, we're considering Kafka for something we do, but it didn't sneak up on us--we knew for years that a particular use case involved a lot of data and wasn't ideal for a relational database.
Why not see the coin from the other side. Developers giving their experience when they are working sane ?
Many of these articles, I have seen authors quoting from their experience building large systems.
Moreover, as a system architect, you often have to analyze the requirements and choose a good stack for implementation. A young dev in a startup is biased against the architecture in one way, because they might be seeing Kafka on their resume as a shining star to greater riches.
Of course engineers have a tendency to over-engineer solutions. I don't think anyone is surprised by that. But this "oh you don't need it, I've used it at <bigcorp>, and you're not that" thinking just rubs me wrong. Especially when it's backed up by hollow arguments like the ones in this post.
I've already done my due dilligence and I consider Kafka to be one of the potential solutions I could use for a service, and while I've gotten started using it fairly easily in a dev env, there's obviously things I could miss. Any pointers from experienced users would be appreciated.
I just learned Kafka this year to augment an existing system at work. One thing I have found is, there's a good amount of smarts in the Kafka clients. Look carefully at the Kafka clients available for the runtimes you work with, and evaluate them for robustness, features, and development activity (sadly, for example, kafka-node seems to have lost all momentum).
My recommendation for anyone new to the area, is the streaming systems book: http://streamingbook.net/, which came after two excellent blog posts: https://www.oreilly.com/radar/the-world-beyond-batch-streami..., and https://www.oreilly.com/radar/the-world-beyond-batch-streami...
Basically, if you have an unbounded data problem, and you might have something that is _practically_ unbounded, that book really helped me gain insight to describe what we used Kinesis for (which is really similar to Kafka in a lot of ways).
Thinking about event vs processing time, ordering and windowing requirements, etc, were really helpful in thinking about my own data problems.
In the end, the author of this article claimed that WeWork just simply didn't have much of an unbounded data problem. Which I generally agree with. But simply saying "you can do this in PostgreSQL" isn't really a great takeaway, and, it really feels that was the message being reinforced here.
I love Postgres but "just use Postgres for your small data" is IMHO a facile argument to an ambitious, naturally distributed IoT project. The fact that WeWork's IoT strategy is/was perhaps useless is rather beside the point.
The alternative is modifying your app for arbitrary data access and dealing with auth or giving direct data access.
Giving direct access to your database is fast but leads to very strong data coupling.
Exposing the data through the app layer is slower than cdc kafka integration but comes with a long term commitment to support that access. I have seen these type of data access queries easily hit performance edge cases.
There are other pros and cons but those are the ones that end up being the most troubling.
Often a single apps data can be incredibly valuable and many teams want to access it. So coming up with patterns that
1. Protect the app 2. Protect the data store 3. Decrease the maintenance cost of the app 4. Decrease maintenance cost of the datatype 5. Add sight complexities from kafka.
Can be very valuable.
Or you could just give "direct" access to (a DB role that only has been granted SELECT on) a single schema within your database, entirely consisting of views. Then you've got a stable data-access API that you can iterate your data model behind, with nothing more than your DB itself.
(And you don't need to put much thought into setting it up, if you don't want to; your stable schema-API-layer can just start off as a bunch of views with 1:1 relationships to your real tables, which each just "SELECT * FROM their_underlying_table".)
I can say that we've had our troubles with kafka sure. But app teams 100% love working with it instead of having to have familiarity with all the various data sources we have.
Working with all sorts of different data but all of its kafka is much easier than all sorts of data from Many data sources.
I'm seeing too many places using it in some form of RPC style system as part of some "workflow" orchestration magic system with unicorns and ponies for everyone
CDC -> Change Data Capture
Cdc also enables all sorts of non obvious changes that are infra focused. It's much easier to update your version of sql server, db2, mongo, etc. If you don't have to worry about every other app and how it connects to your shared service.
You may think that you're being smooth by updating a A record or Cname when doing a migration but very often there's an app out there accessing using an ip.
OTOH, the author is correct that it's more work to set up and maintain than non-distributed software. Don't introduce it unless you have carefully considered alternatives.
I became de facto responsible for running a small Kafka cluster at a small company that could have gotten away with something less shiny. That was not fun. (Especially since the company started with Kafka 0.8 and encountered many changes along the way to modern releases). Eventually the company ponied up for Confluent to take over running Kafka for us as a service. Now I'm back to enjoying the nice Kafka APIs without having to deal with the annoying Kafka servers.
I like the Kafka API/capabilities but dislike running actual Kafka servers. If I were to start a one-man venture that needed event streaming I'd probably try Redpanda [1]. Its developer hangs out here, which is how I first learned of it. It's supposed to implement Kafka service but with no Zookeeper or JVM knowledge needed to run it. I wouldn't even consider running the original Apache Kafka for a solo/small team.
[1] https://vectorized.io/
the tl;dr is that we wrote a pretty fast fragmented buffer parser (due to our thread-per-core design), we consume some json files and codegen it.... which basically means, if you connect an existing app that uses the kafka driver and it doesn't work it's a bug, and it's a matter of teaching the codegen to fix it most of the time. Transactions should land in the next month or so, which is the only thing really missing today.
we have a ton of solo devs typically that come from the Go, Rust, JS & Python seem to like it quite a bit because of what you said.
People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Now people are relying on your systems, and you can’t have any down time. All redesigns have to be able to be implemented in a way that can transition from the previous system without breaking anything. As we often say, “we are changing the tires while driving 90mph down the road”.
I wish we had spent more time on thinking about scale when we were smaller. Our new mantra is that everything we make now has to be designed to horizontally scale, even if it is something small right now.
The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.
It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time.
Still wish we had made some different ones, though!
I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price.
(If you're reading this and you know who you are, I'm sorry!)
http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
Don’t use it for relational data. Don’t use it as your primary data store.
At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc
But “Never use it” is certainly not good advice.
> What’s missing from MongoDB is a SQL-style join operation, which is the ability to write one query that mashes together the activity stream and all the users that the stream references. Because MongoDB doesn’t have this ability, you end up manually doing that mashup in your application code, instead.
Even with a normalized relational database one wouldn't make a single query to return rows from a join of tables T1, T2, T3, etc when there are repeating relations such as the users in the social graph. This might be done sometimes paying for the duplication to save a network round trip, that's all and no different.
> Users have friends, friends have posts, posts have comments and likes, each comment has one commenter and each like has one liker.
I would likely gather all the instances of User id in each context (user, friend, commenter, and liker) and query it separately from the JOIN query. Depends on your data which way works out better on average/worst-cases.
Yeah, that can't end well. It's a bit ironic from parent poster's viewpoint but "boring", well-understood technology is generally also the easiest to migrate away from. That's one key reason to choose it at the outset even over more "innovative" and perhaps more "ergonomic" choices like mongo, that in reality aren't nearly as simple to deal with.
This is the best case, the worst case is that the scale out thinking hampers your product market fit progress. That's the risk you run.
So, while it seems to have gone scale out on day 1 in hindsight, it's always always better to start small and scale out as needs arise, even if that means re-architecting the whole stack. It's painful, but doable. It's also a reasonable challenge to entice new hires. You have scaled out a system from degree 1 to degree m. Now scale that out from m to r++. Something like that.
These two things help any system at any stage. They also make it easier to handle migration if that's necessary.
Stack Overflow also famously buys the biggest hardware they can get and upgrade their database server accordingly.
I always wonder if horizontal scaling is really always the right answer, when you can just buy a bigger box (of bigger VM) and spare yourself a lot of issues.
But I also tend to think "you can always get a bigger box" can take you pretty far, and much further than most people that only think in terms of AWS realize.
I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that.
There is a limit to how much you can vertically scale.
Even for the things we CAN vertically scale, we need to be able to handle hardware failure and OS upgrades without taking any downtime, so you are going to need to be able to horizontally scale enough to handle that.
A CDN is essentially "horizontal scale as a service". The vast majority of products are very different.
I don’t think we are unique, though, and there are many engineers on here who work at all sorts of interesting places with non-standard workloads. I was just pointing out the reasons we can’t vertically scale.
Today you can without issues have 128 cores and 1TB RAM in one server (or 48 cores and 1.5TB memory if you prefer Intel). Of course not everything scales perfectly, but you can get pretty far before you are really forced to scale horizontally.
https://yourdatafitsinram.net/
Horizontally scaling your database is a last resort used at truly mindboggling scales.
Especially with today’s hardware or simple cloud vms you’ll come a long way.
There's interesting tech out there, but I think adding "interesting" technology to your stack without a very good reason is usually a mistake.
"Keep your architecture as simple as possible, but no simpler". There are important problems kafka solves. But its worth checking if you have one of those problems before using it.
If a startup is really concerned about avoiding the problems you're encountering at your company they should just take an extra week when planning their infrastructure to map out details of how they would transition to a more scalable infrastructure. Then save that file somewhere and dust it off if they ever need it. And if they need it, they'll have ample investment money coming their way to pay for the transition.
If they didn't have VC, they'd likely die on the spot, drowned before they even had a shot at real product-market fit under the weight of their own accidental complexity.
Now, is there a set of common patterns that works well enough for startups to iterate quickly with, that _also_ scales? Probably. Can you be successful with NoSQL et al out of the gate? Almost definitely. Does it make it _much harder_? I have three anecdotal data points that point to "yes".
From my work over the last year, I'd suggest "choose boring technology" over "worry about scaling early" 10 out of 10 times. You need something that scales _conceptually_ before it can scale in the cloud, if you don't know what your core business value is or how your product actually works it's really damn hard to build a cloud-native app that anyone can actually iterate on. You're fighting incidental complexity every step of the way.
Maybe in 5 or 10 years, when the average developer has a wealth of experience in scaling technologies, when serverless functions have fewer dramatic downsides, when we start seeing out-of-the-box event driven full stack frameworks, when the bleeding sharp edges are sanded off NoSQL and we've figured out a good story around schemas and validation, then it starts to look more attractive for young companies. Until then, though? Stick with the boring stuff.
EDIT: Though, re-reading this, I suppose I should acknowledge my own survivorship bias - the companies that figured out their scalable, cloud-based tech stack early have no need for my consulting services...
and it is pretty good. So even, in a one person situation, i decided to split services sometimes into two.
Main reason for having separate services is - a different privacy/security context or deployment time:
So one service had so many native libs that it needed that i didnt want to wait 3 minutes to build to deploy when independent code had changed.
So far I would say the setup totally practical for our use-case. A monolithic setup would have been slower. Mainly, with regard to quickly iterating.
If your startup’s product is constantly iterating to find product market fit, it’s a fool’s errand to keep making sure everything is perfectly horizontally scalable in the first place. It’s very rare that the things you expect to be important early on actually are in the product that gets popular.
You’re absolutely right to change your mantra now that you’re scaling, but you were right to not do it early on. The difference between life an death in an early startup is quick iteration.
On the one, people take it as literally "don't worry about it, just make it work" and don't think through simple decisions.
On the other, with some basic analysis you can identify where your likely bottlenecks are going to be and make some preparations to dramatically simplify scaling when you need to do it. Sometimes minor architectural decisions, that don't take much time, can lead to a happier long term life.
But I 100% agree with designing for horizontally scaling.
It's one of the reasons I always try to steer people in the direction of Heroku for new projects, because while it reduces your ops burden and makes your life easier...it also forces you to design for horizontal scalability from day 1 via 12 Factor.
(Having flashbacks to the same code in 3 PHP repos, maintained by one person, no two copies identical)
We have Kafka in places the article’s author would suggest are inappropriate. We have a lot of “one true systems” that hid horizontally scaling instead of making it easy to build clusters of micro-services. Now we’re riding micro-services pretty hard and that isn’t going well since we have too many mixed architectures in use.
I think it’s correct to push out worrying about scale while you seek product market fit. I think that once you do need to scale you really need to put thought into the process. This is difficult if everything is on fire and failing. If it’s not, burn some time to plan. It’ll pay off. Be sure you don’t reinvent things you can cheaply buy. Make sure developers still like their day jobs after the changes are made before making more changes with the same underlying design theories.
Middle-scaling is also suboptimal - you’re paying too much upfront and you don’t get enough later on anyway.
Stage-appropriate scaling is the smart way. Scale as you can see the need. This requires willpower - you will be pressured to kick things down the road just one more time, every single time.
The path of the wise is generational design. Build every component such that it can be easily destroyed and replaced. Anything that mustn’t be destroyed should be scrutinized and planned for in advance. Store your data in most mundane formats so it’s easier read. Keep interactions simple and need-to-know. These are platitudes, you will need to fill the planned obsolesce roadmap for yourself. This unites your hands make the quickest crappiest implementation at the moment secure in knowledge you can fix it or scale it as needed later.
Well ... you're going to have to. You don't know what bottlenecks you're going to hit as you scale up, so why spend precious development time trying to guess which systems to optimize? And to be clear, 'cloud-scale' architectures are complicated and bring their own set of issues - why subject yourself to that?
Plus a well designed (monolithic) system should get you very far with vertical (resource) scaling and/or data sharding.
1. Define the domain segregation and be discipline on the implementation. It will not only help ease the process when the system need to be segregated into multiple service, but also help on organizing the team.
2. After segregating the domain, even if all of them live in the same database dont use `JOIN` as much as possible. Define the API for each domain and use it as the main communication.
3. As boring as it sounds, use boring technologies. Pick framework that has been used by many like Jango, Rails and Laravel. They offer not only tools and libraries but also programming pattern that will help us to implement domain segregation. The framework I mentioned have good established (way to work with if not integrated) ORM, event driven method, front end development and means to scale.
4. Usually postgreSQL is enough and can bring long way, just remember point 1 when designing the DB.
I think those points are quite straightforward and doesn't make the development more difficult, while trying to be future proof.
A lot of software assumes a single server is fine and as a consequence doesn't even bother trying to abstract that away; the singleton nature of the software then percolates virally through it all.
On the flip side, some software actually isolates their scaling limitations and pushes the bottleneck into a single part of the system.
With the former, you're screwed if you hit your vertical scaling limits (to say nothing of what your system availability probably looks like, if scaling is actually a concern). With the latter, you can play games with your scaling, perhaps by shoving a hierarchical or replicated cache in front of your scaling bottleneck, and effectively buy time to build a solution with better scaling. (Or perhaps never bother with that, and just be acutely aware of your system's scaling dimensions, like StackOverflow.)
There's no such thing as a system that truly scales infinitely, just systems whose scale limits exceed our imagination.
(Admittedly, most bugs of the form "INT64_MAX overflow" are generally a leak and not the result of growth :).)
Of course, WeWork at this point had 1500 engineers so calling it a small startup or using it as a case study (either way) for small startups seems a bit silly.
Ruby: High-velocity development; Decisions have been made so you don't need to make them; Suffer from scalability issues both performance-wise and architecturally in the endgame.
Java: Much slower development; Could be fast if correctly architected, but doomed as well if done wrong; Require a hell of infrastructures to work well.
But you can also find some superstars among the minor players without many trade-offs, for example:
Elixir: Development velocity and decision-wise are very close to Ruby; Performance is on-par or faster than Java without decisions; Messaging and cache already baked-in the language and tech stack though you don't need them in the beginning; Hire Ruby programmers and just start rollin'
Amen. I've been through some interviews where I'm the oldest at the table and they tell me their stack and I try to keep a straight face.
It's hard. Hard to keep a straight face because I'm a genuine honest person with a lot of experience.
"We plan to hire 15 people within 3 months and double year after year".
Well, yeah, with every acronyms AWS could come up with, using them all for no reasons at all (except S-C-A-L-I-N-G), you could build it all rock solid on a mid-size linode and do it with 5 dudes
Expected: world domination, runway blast-off to the moooooon
Actual: as many employees as concurrent users, grind for the next 2 years and run out of money
Reality test: failed.
I've been through some interviews where I was the youngest person at the table, but it otherwise sounds like I had a very similar experience to you (they had ~1000 total users - their "scaling issues" were entirely due to poorly architected code). In my case, I politely questioned it during the interview and they actually ended up letting me take over and simplify things.
I read an article a few years ago about how StackOverflow ran (still?) their entire site off just 6 physical computers or something. And before you claim its because they have hand written assembly everywhere - nope. Their code is just clean C#.
I adore Kafka - I think its a lovely way to build scalable software. But almost every website out there will never need to scale past what a simple postgres + rails/nodejs/whatever + nginx setup can provide, assuming each of those pieces is set up correctly. And you personally probably aren't working on software at that scale.
I personally also suspect we can make the experience of building on top of event sourcing better than the experience of building on top of a traditional database, because decoupling reads from writes gives you more architecture options. But I don't think we're there yet.
It used to be that distributed systems were a big trade off. They were operationally complex, they had limited apis (NoSQL), but they scaled. The best solution used to be to build things using a non-scalable but easy to use and run system, and then re-write it later if it needed to scale (often in a big hurry).
This is just not the case any more, though. Why? Two reasons:
1. We’ve gotten much better at distributed systems so the apis aren’t nearly as limited. It’s no longer that you either choose hard-to-use things like Hadoop/NoSQL or elegant but unscalable single-server databases. You can have both good abstractions and scale.
2. The cloud makes it possible to get systems as a service so there should be way less ops than running a single node system yourself
In the case of Kafka, I’m super biased as I’m one of the original authors, but I think the abstractions Kafka gives, stream processing capabilities, connectors, etc are just way better than a lot of the traditional solutions. Using something worse until you “need” Kafka might make sense on premise, but not in the cloud.
Confluent offers a Kafka service which is fully managed so you don’t do any of the upgrades, security patches, midnight pages, etc you just use the APIs. This is super affordable for the kind of simple apps the article describes. The price varies by cloud, but e.g. on GCP it starts at $0.11/GB for reads and $0.10/GB stored. That is a lot cheaper than using a single node system and then rebuilding everything if you need to scale, but not only that, it is also lower operational overhead (effective none) and a better interface/abstraction.
I think this isn’t unique to Kafka, either. There are great managed systems that are built to scale for most of the kinds of data systems you would use—-CockroachDB, Spanner, Aurora, Snowflake, Elasticsearch, Bigquery, etc.
Basically, you can have nice things now, just like the big tech companies.
Kafka is an industry standard, it's benefits go well beyond its own qualities and extend into the ecosystem around it.
Don't attempt to replace industry standard components with any home rolled systems unless you were already capable of building said system yourself and you understand why you are a special snowflake and why you need something special.
Kafka has some benefits (lovely API) and some tradeoffs (hard to maintain, zookeeper is a bit of a mess). But there are obviously a lot of other ways to build successful, scalable software. WhatsApp was implemented on top of Erlang, and scaled to over 500m users when they had only 35 employees.
I want to disagree with your claim that kafka should be considered the default option - but you haven't dignified your (controversial) claim with any actual arguments. There's nothing here to even disagree with.
https://www.confluent.io/blog/removing-zookeeper-dependency-...
All major streaming event processors are designed around it or work seamlessly with it (Spark, Beam, Flink, Storm, etc).
There are pre-existing integrations for all long term storage options (S3, GCS, Swift, etc).
Maybe it could have been elaborated in the original comment but it shouldn't need to be - all of this is about 10 minutes research away from anyone that wants to understand why Kafka is considered the core of modern data architectures.
WhatsApp isn't really a relevant counter-example and if anything reveals the misconception yourself and many others hold about Kafka which is that it's a messaging solution. It's not. You can bastardize one on top of it but really, don't. WhatsApp was built around RabbitMQ which -is- a messaging solution (one of the more complex ones, crazy complicated compared to Kafka at least). Kafka -doesn't- compete with RabbitMQ any more than MongoDB competes with PostgreSQL. Apples and oranges if you are trying to do anything but the most basic stuff.
Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?
Ah, yes, I like synergyc technobullshit, too.
If Erlang itself could manage a damn load of >500M concurrent users (if was made for that), then everything else is sucks hard. A lot.
Seriously, half of the "devops" today know shit about the underlying systems.
Very few people roll out of bed and think "my application needs a high performance distributed event stream". For every problem that can be solved with kafka, there are other solutions that don't involve kafka. Many of the alternate solutions don't involve event streams at all. Are they better? I don't know. That question is unknowable without looking into the problems in particular, and looking at what other technology is out there. And I say this with a lot of fondness for kafka. I really like it. I just think there are always technical tradeoffs to make.
> any more than MongoDB competes with PostgreSQL
MongoDB and Postgres do compete with each other though! They work differently - so part of the competition isn't around features; its around how your people think about your data. "Think of your data in a way that makes sense for mongodb!" say mongodb sales reps. "Think of your problem as a message queue" say fans of rabbitmq. They aren't wrong in the general case. But there are lots of problems where I could use kafka, or I could use redis, or I could use a naked postgres instance. You need a much more complex heuristic to decide what technology to adopt than "Should I use kafka?" -> "Yes."[1]
And I say all that as a big fan of kafka. I think its API is delightful and just today I was plagerizing its API design to sort out backpressure in a system I'm building.
> What could you reasonably advocate for in its place?
I haven't used it but by all reports RedPanda is lovely: https://vectorized.io/
[1] Though I'm convinced the answer to "Should I use mongodb" is always "No."
Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Though you can shoehorn it into the same problem space, this is what LINE Messenger did which is of equal scale to WhatsApp and runs entirely on Kafka. So I completely agree Kafka isn't the solution to all A needs to talk to B problems, even when Kafka might be able to do the job.
I think what was perhaps lost is that I am not talking about adding Kafka to an arbitrary application but rather operating a Big Data pipeline - i.e data from applications, devices, sensors, whatever and transporting it to data lake, warehouse and online stores like Druid.
While you can use Kafka in other contexts this is it's original purpose and main use case and where I don't think any other piece of software replaces it right now. Pulsar is on the way there but not quite ready yet. Redpanda is just Kafka without being the real thing - maybe you save some CPU, maybe you hit an incompatibility, really it's not worth the effort because Kafka in 99% of use cases is network bound as it's already efficient enough. (Not always true, there are some CPU bound cases with tons of producers but they are rare)
If you chose anything else you are signing up for writing all of these integrations yourself and that just doesn't make sense. If you pick Kafka you get all the hookups for free and you only need to write custom code for your existing custom applications.
If say you were to want something more "lightweight" sure you could use Redis PUBSUB but now you are writing code that dumps from Redis PUBSUB into S3 for archival instead of just pulling something like Secor or Kafka Connect and being done with it.
Perhaps I should have been more clear that I think it's only in this specific domain/use case that I think it's completely dominant.
A different example of a system that can start small is Joe Hellertein's research (https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf)
the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.
I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)
There's a lot of words and effort going on here to make Kafka sound immensely complex (and dare I say Kafka-esque?)
The article continually chides Kafka as overweight, hard to setup, complex, blah blah blah. None of these things are true in modern day deployments.
Three instances of Kafka, Two ZK instances, and you're off and running. All of that will run in docker and the docker images will stand themselves up if you let them. You need even less for developemnt.
Postgres is a bit more complex given that you've got to manage machine sizes and database schemas.
But even with that being said, the article gets one thing right. For average data use, most any system will do and some startups are over-engineering their stacks.
But if you do know where you need it, you wish you had it yesterday.
This is the way with scaling in startups. Product-market fit and agility and lean operations is what you want early on, and scaling should come later. Big architectural decisions are hard to reverse and committing to some pattern that needs something like Kafka, like event sourcing, is probably too much risk for most products. Most event-sourced CQRS implementations fail.
I say choose Kafka somewhere in the middle, when scaling needs are somewhat predictable and product market fit is good, but try and use Confluent's hosted service or something similarly light on ops. Do non-critical things with it first and ramp up. Don't build big from the get go - that rarely ends well.
Kafka is causing huge churn in applications codebases. I know that "best practices" experts will tell our company that we are doing it wrong. However to me Kafka seems a clever algorithms circlejerking and answer to operational complexity is either go to managed cloud or go to hell.
Master-master Postgres is significantly fiddlier and less well-tested than Kafka. And what do you gain? Fiddlier APIs and a query language that makes it easy to deadlock yourself with no warning or write queries that will run for months.
Just use Kafka unless you've got a good reason not to. It gets a lot more of the basics right.
That's about 100k events per second.
You can handle that using TimescaleDB running on a single Azure DS4 Standard node.[1]
[1]: https://blog.timescale.com/blog/timescaledb-vs-6a696248104e/
Edit: Fixed my maths and language, and linking to hard data instead.
You should have various aspects of your application configurable and independent enough to enable you to develop an alternative system and make a small downtime migration.
Many of scaling issues I see in startups, and stories told by devs in startups, seem to be because corners are cut to make the MVP and push it to the market as quickly as possible, and then suffer scaling woes due to poor design and architecture choices.
Or do they think buzzwords will increase ROI?