If I see premature scaling it's usually at the database side of things. Inexperienced devs can be quick to dismiss a SQL database and go with something based on key value pair caching.
So I agree with the article keep it simple and easy to implement, worry about scaling later.
Similar experiences. Worked on a project, and a 'lead' guy left with .. a few weeks of a node/koa codebase in place that only he really knew. Between the rest of us we had loads of C#, PHP and Java experience (multiples years each over 3 people).
"Let's rebuild something based on what the bulk of us already know," I suggested. (Keep in mind what we had was a few weeks of prototype code that was not in production.)
"No, I really want to learn more node," was the only other input, and that 'won' the decision. 18 months later... "yeah, koa isn't really a goof fit for our use cases".
Whilst the JVM may be established, ie. boring technology Kotlin and Clojure certainly aren't. It's now 8 years since Clojure 1.0 and there is a grand total of 15 Clojure jobs listed on Indeed.co.uk, 12 of which are in London. Kotlin boasts 2.
edit: ack - should be "good" fit, not "goof" fit, obviously.
And yes, per another point in the replies, someone else should have stepped in to decide differently when there was a tie. I could possibly have pushed harder, but the 'bozo bit' basically was flipped on that project, and much of what I lobbied for was ignored, at least in part because I was the one saying it.
You'd think that, right? That was another part of my argument. And yes, it's ... not quite grown in to a monster, but few on the team really know how to deal with it. That actually came up a couple months ago - I heard a comment along the lines of "well, no one's really fully understood how to deal with that koa code since the beginning". Even the koa stuff, I think was done by someone who had used it a couple of times previously and liked it, but... no long term track record of using it in production.
So... about 2 years later, there's some traction from folks to rebuild it in something else. The latest shiny object is AWS lambda.
I've asked many devs why they chose Mongo over a RDBMS. They can never give me a technical answer. It's always 'it's cool' or because they just read something about it on some non-technical blog.
It usually only takes me a minute or two to ask them a future function question that's not reasonably attainable (and performant) with a NoSQL solution... so they wind up going back to whatever RDBMS is their standard.
I don't think I ever heard a good technical reason to use a traditional RDBMS. And as a default choice it is not a very good one, when something distributed can deliver an easy to maintain resilient solution pretty much out of the box.
I'm sure many can be produced, but the first off of the top of my head is accessibility for quasi-developers. If you ever want to hire some analyst roles you are going to find that it's a lot easier to get people that have at least some familiarity with SQL instead of having to train them on whatever custom NoSQL flavor you are using.
Of course you could solve that by having an ETL process dump your stuff into SQL (we keep our stuff in Cassandra and dump it into SQL for analysis), but that's another ETL you need to keep track of and in our case we use a quite complex data model and the ETL only supports a subset of it.
Another one just off the top of my head is joins. Some data is just inherently better suited in normal form.
> It usually only takes me a minute or two to ask them a future function question that's not reasonably attainable (and performant) with a NoSQL solution.
And suddenly I have flashbacks to dev meetings where we were deciding if we should do joins in Python or denormalize our data across a dozen tables because we needed Cassandra, obviously.
The only reason I can think of is not wanting to deal with migrations when you are changing schemas all the time. As long as you can keep all of the previous schemas in your head and account for that in your code, you don't have to migrate anything.
It's not really a very nice way to go long term, but can be convenient when prototyping.
I think CouchDB had some real potential, but Mongo is like Couch without the interesting parts. Couch was like indexes on crack. You can make an index that does arbitrary transforms. Sad part was, it didn't have any smarts to pre-cache those "views" in an intelligent way, which to me would be the whole point of that endeavor. I think something like it will come back eventually, but it's still a very research-y idea.
Huh. I was starting to prototype some stuff with CouchDB when the transition was happening and when I looked at CouchBase it seemed like something important was being abandoned but now I can't recall what I thought that was. I'll look at it again.
As an aside, what I really wanted from CouchDB was that I could create a view and as I inserted new documents into the DB, CouchDB would automatically generate the new view documents so when I query the view, everything is already precached.
At the time I needed to implement a fire hose listener that would would sip inserts and decide which views with which parameters needed to get called, and then just hit them with REST to do the precaching. It seemed cumbersome, but it sort of made sense why it would be hard for the DB to know what queries on the view were implicated by a new document.
Has CouchBase improved the story there at all? To me that was always the killer promise of CouchDB... Storing documents in a format that made sense at creation time and having arbitrarily complex views on that data... without a cache hiccup. But that "without a cache hiccup" never seemed to quite materialize as a first class feature.
Yup! MySQL + Redis (for query/object caching and queues) gets you the best of both worlds in a lot of scenarios, and is reasonably easy to set up and manage. And it will probably scale better than you expect at first!
I am using postgres and redis - they are both trivial to set up and cost about a few lattes a month for a fairly scalable solution. If I had huge traffic, a few hundred bucks a month would be totally adequate - and this is running on the too-expensive heroku. So, unless Riak costs pennies and takes literal minutes to set up, the 10x claim won't hold.
Riak looks like it would be overpowered by my needs - but then again, I learn something new every day.
Can you explain why you think that? I would imagine using two rather popular systems in a somewhat common combination would be a boon to maintainability and set up, if at least for the large pool of developers and online resources already available for it.
But then again, I could be super mistaken. I'm interested in seeing another side of it.
This was also my thought when I made the comment...
I am sure that for certain use cases, there are more tailored solutions, but "SQL + keystore" seems like a fairly universal, affordable and versatile stack.
They are different databases. Riak is a key value store, not unlike the core of redis. The SQL database is obviously relational.
Ignoring the difference for a minute. If you can fit data in Redis, chances are that they can fit in riak and a RDBMS is not needed at all.
Why Riak only then?
It is one of the very few databases that can be clustered. It is sharded, multi-master, highly available, to talk in buzzwords. It is very reliable and very robust, you can add/remove nodes as you wish, replication is highly configurable, etc...
Postgre/Mysql/MariaDB + Redis have no HA. They are both the disaster in disaster recovery.
I always see this discussion cast as one between scalability (KV) and dev efficiency (SQL), but other reasons to adopt columnar databases include:
1. Difficulty in transitioning a system with SQL-Gone-Wild to a system that can be implemented with other dbs without complexity overdrive
2. Escalating costs to get any additional availability guarantees out of SQL (availability & cost are key business issues)
In AWS & Google Cloud, you're going to need to pay outright for capacity as well as capacity devoted db recovery (a replica / set of replicas in another availability zone). On the other side with something like dynamo / cloud datastore, your payment model is based on requests / sec and the costs for replicated data. I'm just saying that there's nothing simple to deciding where price / availability / dev productivity balances will lead you to, and you can be an inexperienced developer who chooses either side of these technologies without really understanding the full story.
A potential client told me he was planning on having three SQL databases. I asked him why, what each would be for, and he didn't really know. Just a vague sense that his app was going to be too much for a single db.
Microservices do require more overhead. Additionally, they require you to be familiar enough with you business logic to know where you boundaries lie - changing abstraction boundaries is much harder with microservices than with a monolith.
I think that while they are great for scaling, they are a lot harder to use to successfully get off the ground (though I'm betting that there are places that have successfully done so)
You also have to know how to "do" microservices. Whether it's incorporating message queues like RabbitMQ, protocols like gRPC, etc., it's very hard to get right the first time by winging it without previous experience. Redesigning microservices after is a PITA.
That really depends on the startup, and sometimes starting with micro-services helps with hiring - In our case thanks to starting with micro-services we hired developers knowing different programming languages and they can work together on the project by exposing end-points/consumers. Of course I wouldn't recommend doing that for everyone but in our use-case we had to start with distributed system anyway due to quite complicated domain.
Even with microservices, I wouldn't recommend changing languages between the bunch without a very good reason. You -will- build up infrastructure and packages that are language and stack specific: monitoring, logging, helpers, etc. that take a lot of time to reinvent.
If you're switching languages/stacks just for funsies, you're going to make services more difficult to maintain over time (and just generally slow down development)
As far as languages go, pick one. When you hit a problem that absolutely cannot be reasonably solved under the specifications you care about, consider a different one for that service. Apart from that, there's little value, and lots of overhead, in moving. Getting to write that one Haskell microservice is sure a romantic idea, until that employee leaves and nobody knows Haskell anymore, and suddenly the service is dysfunctional because dependencies have changed. (For non-mission-critical funsies, sure, use whatever, though non-mission-critical apps tend to have a knack for becoming very important for workflows, hah)
That's a good comment and of course there is a list of approved stacks (not random list of languages) and we try to think about future. Hiring people wasn't our only reason and we had technical reasons to go with micro-services and multiple languages, but it was nice perk.
I totally agree that one should start with a monolith and when he feels confident that all bases were covered in the codebase, he can begin to worry about breaking up the monolith into microservices.
This growth journey makes more sense than starting out the door working on your microservices before business logic as this is like putting the cart before the horse.
Even then, I have a feeling that people jump too quickly from a "messy monolith" to microservices.
Instead, people should start splitting up the monolith into sensible modules and libraries. Instead of a "user accounts microservice", start with a "user accounts library" with a well-documented API that stays in the main codebase and is deployed as part of the rest of the app.
write the MVP in Rails or Django and scale things that need scaling later.
you can scale the base app by giving it more hardware, but when you have to batch process lots of stuff then think of services (most of the time you can just split that to another server).
>>write the MVP in Rails or Django and scale things that need scaling later.
There's a new alternative: write it in Elixir/Phoenix. You'll get the same productivity benefits with none of the performance trade-offs. Elixir apps are reliable and concurrent by default, and the Phoenix framework has none of the "magic" that Rails has, making it very developer-friendly. :)
This isn't to say that Elixir/Phoenix are perfect. The ecosystem isn't as mature as that of Ruby/Rails, so finding answers to questions can take a bit longer. But that is changing quite quickly.
This is one where opinions are widespread and mostly the choice doesn't matter so much, hah.
Fwiw, I've been pretty intrigued by the idea of dotnet on linux lately (integrates with postgresql and everything). dotnet isn't exactly hip, but C# is fast and it's mature (and has some pretty dope functionalish features. You can run F# if you want to, too). Planning to try how well this works in practice soon.
There are things I like a lot about Elixir, but also plenty that I'm not quite a fan of.
What's really important in this context is building something quickly. That's why GP mentioned Rails and Django; they come with many things either built in or built by community, so you can focus on building your first version quickly. To the extent that is true with other tools or frameworks, sure, go for it.
I've been wanting to get into Elixir / Phoenix. The biggest tradeoff there seems to be the developer productivity tradeoff you mentioned — the package ecosystem is still pretty immature.
The application layer is rarely the piece that needs to scale immediately. Usually the first step is to separate mysql or mongo and add more resources or further cluster as visits increase.
There is a case to made to switch ruby/rail for php/laravel. Just as rapid for development and easier scaling plus bigger ecosystem. There is also a case to be made for switching Django for vue or even simple jquery in some cases.
The "how" really matters when deciding between microservices and a monolith. For instance, pumping out a bunch of Lambda/API Gateway web endpoints that back a static site served out of S3 or some other static hosting service would qualify as a microservices approach and yet is a huge time saver during the experimentation phase. No worrying about ops, monitoring or anything else off the list of overhead issues that isn't building your product. But if Microservices means configuring servers/clusters, service discovery and such, it actually adds to your overhead list and you're right that it will slow you down.
To my mind, services like Lambda are perfect for startups. In addition to scaling down, costing nothing when unused and allowing rapid iteration with zero operational concerns, they largely scale up and, in the unlikely event that the startup is wildly successful, they'll handle almost any load you throw at them until you've had time to re-engineer the backend to be more economical for the increased load.
I'm currently in the experimentation phase and I find FaaS to be a blessing. I'm currently using Google Cloud Functions since I hooked the initial app (which has a real-time messaging component) to Firebase and it's saved so much time wrt overhead.
Serverless microservices, some call these nanoservices / functions as a service / microservices as a service, like AWS Lambda are definitely a distinct lighter weight category because you're not managing apps or servers to support them.
Microservices sound ideal at first because it's easy to not see that the brunt of the effort is not writing the first version but monitoring, maintaining, and handling when some go down. By using microservices one introduces additional edge cases that don't exist in a "monolith". Quotations around monolith because I believe startups use it with some exaggeration. For example, a Django project of ~10k LoC might be composed of 15 separate Django apps behind one API and I don't see this as monolithic.
Yes, but also understand explicitly what the scaling limits of your initial solution are, communicate them to the CEO, and explain how long it will take to build a scalable solution for the next level and how much lead time will be necessary.
Needing scalability means you should have customers which means you have revenue.
I tell every startup that I advise that "revenue solves many problems" because if you have cash flow, everything else becomes easier.. you can buy spot capacity in the short term. You can hire contractors to help refactor. You can rent coworking space for an office.
There are many "good enough" solutions for now. Take advantage of them until you know what your next steps should be.. and then commit accordingly.
This approach really lacks the product vision. Although I agree with it to some extent, my experience is with semi-mature product, the code will stick around for ~2 years on average. You need to be experienced to make implementations that will be easy to replace when needed.
E.g. one of your assumptions there are contractors who can help you. That can be true to a certain level, but even product vendors often lack the scale to support their own thing above the basic levels (this is true for post-IPO companies), now imagine this when you can't afford to incubate your product at an expensive tech hub location.
Also, when you have cashflow, your business is under pressure to provide better customer service and marketing, and this is where the things getting tricky: you need to decide whether you want to grow your business or pay back the technical debts. Because the investors priority is always business over tech (rightly), you need to look 1-2 years ahead and make your key technical decisions accordingly.
I was hoping for a little more substance or specific examples, but I think it's long enough to make the point that microservices have been cargo culted and don't make sense at every stage, combined with early startups can be prone to over-engineering beyond their current stage.
There's a myth around scaling [1]. Actually, it's almost like a cognitive bias. We assume that we can usually use the same structure as we scale. Because of this bias, we haven't thought enough about how to make bigger structural transitions normal.
The GP points at the transition between the quick & dirty architecture and the architecture that handles Google-scale, but those aren't the only good stable points for a system.
While it's true you don't need to design for scaling right off the bat, if you do become successful and keep pushing out the scaling pieces , the cost of scaling later increases nonlinearly. At some point it becomes so important that you start losing customers and/or new business opportunities.
A CTO should be able to foresee this and act before scaling becomes a behemoth project and no amount of money can buy time-to-market.
> the cost of scaling later increases nonlinearly At some point it becomes so important that you start losing customers and/or new business opportunities.
That simply is your opinion. You are more likely to lose customers for not focusing on the core product first.
I've found that most startups fail to scale when it comes to making fast/responsive product decisions.
This may not be obvious, since those startups are typically "agile" in the buzzword sense.
The cost comes when the universe of possible change is constrained to the point where the right questions aren't being asked because the answer would be "no we can't do that". So the startup enters a dark age in which scientific thought and rational problem solving isn't really relevant. Often at this point the team starts using Scrum to force the business to plan better, when the core problem is that the existing infrastructure is not able to support the needed experimentation fast enough, making it useless.
This can happen at various stages of funding, since sometimes a series A or B can occur without solid product market fit, or the company may be really good at growth in spite of having a promising but mediocre product.
So I view the CTO's job as preserving the ability of the business to behave scientifically and boldly until product market fit is found.
I'd submit it's important to always be able to behave scientifically and boldly. Product-Market fit isn't a one time solution. Markets move and change, Facebook, Apple and Google have been successful because they've changed their products as the markets have changed.
Exactly. I totally agree, but typically with a startup achieving it at least once (the first time) is of particular importance. After that the role is more general as "CTO" than as "Startup CTO". But I'm just splitting hairs. You are totally right.
The problem is ego. "Scalability" is a sexy problem to have, when in fact, "simplicity" and low-risk "trailing edge" technologies is the prescription for success. Yep, i'm talking to you. You don't need an angular app backed by MongoDB to display static content. Static HTML will work just fine.
As a CTO I see it as part of my job to find the right tool for the right job.
Part of that was when I told the team I wanted our splash page and everything on it to be just static HTML files that would be hand edited if and when it was needed. There was no need for fancy CMS tools for a landing page that gets updated maybe once a quarter.
So, that's part of why you need a CTO - to know when to use static HTML!
> As a CTO I see it as part of my job to find the right tool for the right job.
If that's a minuscule part of the job, then I agree with you here.
My point is, you are not a CTO if the technology behind the product is a static web site. What do your VP and Directors do all day? What about the teams under them?
There is a huge difference between a CTO and a front-end engineering.
I am biased because in my work, I have indeed seen many projects that are struggling or outright failing because of performance problems. The underlying sentiment of this article may be (mis)interpreted by some as "don't worry about performance." I feel that is not the right guidance, at least in many cases.
That said, there is an important premise presented by the article. If I can reframe it slightly: don't over-complicate your system architecture, especially not needlessly. If, by selecting some fancy and trendy new technology, you will complicate your architecture or development process for unknown or modest performance gains, it's probably wise to postpone such adoption until the need is more palpable.
My advise, again showing my bias, is quite simple: just select foundation technologies that are mature and high-performance. Using high-performance foundation software gives your application a high performance ceiling, which in turn provides breathing room to build your application in a more "brute-force" fashion. With this approach, you can defer performance tuning and optimization for some time, and perhaps indefinitely.
I have seen plenty of small and small-medium businesses operate well with very modest technology platforms. But importantly, they are modest in architecture (few servers, few processes, low system complexity, simple deployments) because they used high-performance platforms (e.g., JVM, Go, so on). On the flip-side, I've seen many small businesses struggling and prematurely adopting complex system architecture because their software platform (e.g., Ruby, PHP) presents a bottleneck early in the business' life, which creates a challenging scale event before the company has even found their market sweet spot.
When I've written about this in the past, I've pointed out that many people call this situation a "good problem to have," which is of course valid in important ways. But it's also crude and demeaning to your development team. You're basically saying, "Fantastic, our business is scaling. Yes, our servers are on fire and my developers are panicking, but that's great!"
It's better, in my opinion, to have used a high-performance foundation and (potentially) avoid that unnecessary pain. All else being equal (e.g., importantly, your development team is comfortable with a JVM language, for example), I recommend keeping your application simple by using platforms that give you a high ceiling to grow.
All fair points. I think the thrust of what I was saying is simply: resist the pressures to provide huge scale before there is reasonable evidence that the product has some traction. Some huge percentage of companies will have to do a major pivot before they get that traction, so the balance should be heavily on simplicity, speed and agility, and if that's at the expense of scale, so be it. Obviously, performance does matter, and so does good architecture, but staying in business matter most :-)
You and I completely agree on this: huge scale is a irrational target out of the gate for most projects. The potential for huge scale preoccupies many an architect whose aspirations exceed their budget, to say nothing of their market reality. A premature preoccupation with huge scale tends to yield over-engineered systems that are needlessly complex and work against all of the goals you cite: simplicity and the ability to react quickly to changing business needs.
It seems my addition here is reduced to this: All else being equal—if your team is equally comfortable with two technology options—selecting the platform with a higher performance ceiling will further defer the architectural complexity typically associated with scale. Architectural complexity (e.g., multiple languages, lots of orchestrated or inter-dependent processes, many abstraction layers, complex deployments) tends to be the friction force against the simplicity, speed, and agility we both seek.
The issue I keep encountering, again and again, is that you really have no idea what to scale until you need to.
For instance, I had a client where a very large online retailer wanted a RESTful API into my client's data that they could call directly from their customer's browser. Before the Black Friday Freeze. This was a completely unforeseen requirement and the solution was NOT to scale the client's overall website to handle the 5-10K requests per second but create a micro service on a new stack that could.
Your startup will pivot, sign weird deals and change your target customer. Optimizing for scale early doesn't help, getting to the point where you know you need to pivot does.
While i absolutely agree with the author, how does this line of thinking change when you start using AWS or something similar? I'm building a SAAS app at the moment and a lot of these scaling decisions are incredibly easy to implement without imposing any significant costs in terms of maintenance (and money, a lot of times). This would've been a different story had i been using my own infrastructure.
I think there's a key difference between building your architecture with plenty of headroom to scale (leaving plenty of low-hanging fruit), and actually implementing high-scale function before it's required.
In a cloud environment, it's quite easy to be comfortable that you _could_ scale out to 10-100x of your current capacity, if you have to. But it's probably not worth proving that fact until you have some customers.
Or put another way, being in a very flexible environment makes it even more of a no-brainer to delay building for scale, since you have so many free or nearly-free options there.
Now of course, if you really do hit scale you'll have the fun problem of needing to optimize your spend, but you can probably delay thinking about that until well after you've proven product-market fit.
Yes, if and only if you know exactly what you need to build and how big the scale will be from day 1.
What usually happens is the first thing you build is wrong, and you need to iterate aggressively on not just the software but the entire business concept. Once you have product-market fit, then you are in a position to design an optimal architecture. If you go microservices before that, you are adding unjustifiable overhead, which will inarguably slow you down compared to a monolith when you are in the 1-3 engineer phase, and don't fool yourself about that.
I frankly have no idea what you are talking about. You are making wild assumptions that monolith are better just because...
It will all come down the the experience of the team, esp. the leadership - the managers, the leads.
A strong, experienced, mature and effective leadeship will enable a team to perform "magic" and set the right technology foundation that will prove its worth as the business grows over time
The point he's making is the technology foundation can be built for the wrong product or the wrong market. After 2 full product reconfigurations or pivots requiring a TON of code being dumped you will wish you didn't have the overhead of building infrastructure with the first product in mind. In the end, the base reasoning for the foundations being built the way they didn't hold up in production.
Building something quickly that can be broken up over time is the best way to an MVP, and you can deal with scalability issues when you HAVE scalability issues. I consider scalability to be a good problem to have, because by that time you have actual traction.
If you are a bigco with a big marketing budget in a field in which you already have experience and business then by all means scale to millions of users at day one. Don't take on the cost of it in a startup, because the problem is TRACTION not technology with regards to scaling.
You are making assumptions about both the premise and the conclusion :)
Designing and building for scalability, flexibility, availability, etc does not mean going all out to boil the ocean and then find that you attempted to boil the wrong ocean.
I am not making "wild assumptions", I'm putting a stake in the ground that a monolith is lower overhead than microservices, and thus better when you need to iterate quickly, which is always the case in early stage startups.
You're making fantastical, hand-wavy claims that belie the hard tradeoffs that need to be made to get a startup off the ground. Microservices absolutely add complexity and require more effort to set up correctly and orchestrate, pretending that experienced leadership makes that problem go away is dangerously idealistic. What experienced leadership gives you is good judgement on whether the overhead is worth it given all the factors at play.
The way you speak about the team, and different roles makes me thing you've maybe worked at "startups" that are actually established businesses with traction which like to retain the mythology and allure of being a startup (perhaps to justify lack of profitability). Things are very different in a true startup where the tiny handful of employees all must wear many different hats.
Are people, who are just starting, still building complex chef and puppet systems when Heroku and Elastic Beanstalk exist? .i.e scalability is easy now. It's a click away or even automatic which these types of services. It's also affordable.
I didn't say they were replaceable for everyone and for every scenario. I'm just making the point that small startups no longer need to use them in the beginning if all they want is a way to auto-scale. Yes they made sys admin stuff easier but now it's even easier with scaling services that are already preconfigured. I just think it's strange that people are still discouraging founders to account for scaling when today it's cheap from both a time and money perspective to do from the start.
Dumb, often limited scalability is easy now. Elastic Beanstalk approach is only as good as it's the weakest link, which you have no control over.
EC2 is having issues with recovering from standby? Good luck getting anywhere. You can sit on the chat with the useless AWS 'technicians' that are paid to waste your time until the issue is corrected.
I'm not sure if I'm making myself clear enough. I'm not arguing that this is great for everything. In the context of the post, that's good enough for a lot of apps just starting out WITHOUT the work of setting up something custom and durable ala Netflix's infrastructure.
My answer to most scaling question is: by the time we have that problem, we'll have a lot of customers and money so we'll hire smarter engineers with experience in scaling to figure that out. I still make sure we don't do anything stupid right now so the future engineers have more options on how to scale. Also it helps that I'm not working on a zero-revenue consumer app.
I think I can give one concrete example where a NoSQL choice (often) makes sense. Common modern day scenario in mobile is to build an app and iterate. Migration in RDBMS rapidly becomes a real pain. Worse, you have no good sync solution in case you need it. Contrast with something like Couchbase Mobile. NoSQL embedded db is much easier to get going with. (Just check the Android docs on using SQLite and see how it compares to "here's my JSON, done".) Sync doesn't get in the way, but if/when you need it, you're going to be really happy it's there. (Disclosure: I work for Couchbase.)
102 comments
[ 3.7 ms ] story [ 190 ms ] threadSo I agree with the article keep it simple and easy to implement, worry about scaling later.
'Because it's cool'
Oy.
"Let's rebuild something based on what the bulk of us already know," I suggested. (Keep in mind what we had was a few weeks of prototype code that was not in production.)
"No, I really want to learn more node," was the only other input, and that 'won' the decision. 18 months later... "yeah, koa isn't really a goof fit for our use cases".
C# is a great language, JVM is great w/ Kotlin/Clojure, both of those have tons of web servers.
And yes, per another point in the replies, someone else should have stepped in to decide differently when there was a tie. I could possibly have pushed harder, but the 'bozo bit' basically was flipped on that project, and much of what I lobbied for was ignored, at least in part because I was the one saying it.
Otherwise, it will grow into a monster that nobody want to deal with.
So... about 2 years later, there's some traction from folks to rebuild it in something else. The latest shiny object is AWS lambda.
It usually only takes me a minute or two to ask them a future function question that's not reasonably attainable (and performant) with a NoSQL solution... so they wind up going back to whatever RDBMS is their standard.
Of course you could solve that by having an ETL process dump your stuff into SQL (we keep our stuff in Cassandra and dump it into SQL for analysis), but that's another ETL you need to keep track of and in our case we use a quite complex data model and the ETL only supports a subset of it.
Another one just off the top of my head is joins. Some data is just inherently better suited in normal form.
https://news.ycombinator.com/newsguidelines.html
Curious if you have an example of this.
It's not really a very nice way to go long term, but can be convenient when prototyping.
I think CouchDB had some real potential, but Mongo is like Couch without the interesting parts. Couch was like indexes on crack. You can make an index that does arbitrary transforms. Sad part was, it didn't have any smarts to pre-cache those "views" in an intelligent way, which to me would be the whole point of that endeavor. I think something like it will come back eventually, but it's still a very research-y idea.
As an aside, what I really wanted from CouchDB was that I could create a view and as I inserted new documents into the DB, CouchDB would automatically generate the new view documents so when I query the view, everything is already precached.
At the time I needed to implement a fire hose listener that would would sip inserts and decide which views with which parameters needed to get called, and then just hit them with REST to do the precaching. It seemed cumbersome, but it sort of made sense why it would be hard for the DB to know what queries on the view were implicated by a new document.
Has CouchBase improved the story there at all? To me that was always the killer promise of CouchDB... Storing documents in a format that made sense at creation time and having arbitrarily complex views on that data... without a cache hiccup. But that "without a cache hiccup" never seemed to quite materialize as a first class feature.
I think this would be 10x more expensive to set up, manage, maintain and use than something like Riak.
Riak looks like it would be overpowered by my needs - but then again, I learn something new every day.
But then again, I could be super mistaken. I'm interested in seeing another side of it.
I am sure that for certain use cases, there are more tailored solutions, but "SQL + keystore" seems like a fairly universal, affordable and versatile stack.
Ignoring the difference for a minute. If you can fit data in Redis, chances are that they can fit in riak and a RDBMS is not needed at all.
Why Riak only then?
It is one of the very few databases that can be clustered. It is sharded, multi-master, highly available, to talk in buzzwords. It is very reliable and very robust, you can add/remove nodes as you wish, replication is highly configurable, etc...
Postgre/Mysql/MariaDB + Redis have no HA. They are both the disaster in disaster recovery.
1. Difficulty in transitioning a system with SQL-Gone-Wild to a system that can be implemented with other dbs without complexity overdrive 2. Escalating costs to get any additional availability guarantees out of SQL (availability & cost are key business issues)
In AWS & Google Cloud, you're going to need to pay outright for capacity as well as capacity devoted db recovery (a replica / set of replicas in another availability zone). On the other side with something like dynamo / cloud datastore, your payment model is based on requests / sec and the costs for replicated data. I'm just saying that there's nothing simple to deciding where price / availability / dev productivity balances will lead you to, and you can be an inexperienced developer who chooses either side of these technologies without really understanding the full story.
If you're inexperienced with relational databases, perhaps a Key-value store with a simple API is much simpler and easier to implement.
Microservices do require more overhead. Additionally, they require you to be familiar enough with you business logic to know where you boundaries lie - changing abstraction boundaries is much harder with microservices than with a monolith.
I think that while they are great for scaling, they are a lot harder to use to successfully get off the ground (though I'm betting that there are places that have successfully done so)
If you're switching languages/stacks just for funsies, you're going to make services more difficult to maintain over time (and just generally slow down development)
As far as languages go, pick one. When you hit a problem that absolutely cannot be reasonably solved under the specifications you care about, consider a different one for that service. Apart from that, there's little value, and lots of overhead, in moving. Getting to write that one Haskell microservice is sure a romantic idea, until that employee leaves and nobody knows Haskell anymore, and suddenly the service is dysfunctional because dependencies have changed. (For non-mission-critical funsies, sure, use whatever, though non-mission-critical apps tend to have a knack for becoming very important for workflows, hah)
This growth journey makes more sense than starting out the door working on your microservices before business logic as this is like putting the cart before the horse.
Instead, people should start splitting up the monolith into sensible modules and libraries. Instead of a "user accounts microservice", start with a "user accounts library" with a well-documented API that stays in the main codebase and is deployed as part of the rest of the app.
Having nice hard boundaries that you can't just penetrate can aid and abet you in debugging.
It also probably helps you with respect to managing security.
As in most things, good judgement is key.
you can scale the base app by giving it more hardware, but when you have to batch process lots of stuff then think of services (most of the time you can just split that to another server).
Yeah. A starting monolith, some cronnish way to run batches, and some way to listen to queues goes a long way.
There's a new alternative: write it in Elixir/Phoenix. You'll get the same productivity benefits with none of the performance trade-offs. Elixir apps are reliable and concurrent by default, and the Phoenix framework has none of the "magic" that Rails has, making it very developer-friendly. :)
This isn't to say that Elixir/Phoenix are perfect. The ecosystem isn't as mature as that of Ruby/Rails, so finding answers to questions can take a bit longer. But that is changing quite quickly.
Fwiw, I've been pretty intrigued by the idea of dotnet on linux lately (integrates with postgresql and everything). dotnet isn't exactly hip, but C# is fast and it's mature (and has some pretty dope functionalish features. You can run F# if you want to, too). Planning to try how well this works in practice soon.
There are things I like a lot about Elixir, but also plenty that I'm not quite a fan of.
There is a case to made to switch ruby/rail for php/laravel. Just as rapid for development and easier scaling plus bigger ecosystem. There is also a case to be made for switching Django for vue or even simple jquery in some cases.
To my mind, services like Lambda are perfect for startups. In addition to scaling down, costing nothing when unused and allowing rapid iteration with zero operational concerns, they largely scale up and, in the unlikely event that the startup is wildly successful, they'll handle almost any load you throw at them until you've had time to re-engineer the backend to be more economical for the increased load.
Microservices sound ideal at first because it's easy to not see that the brunt of the effort is not writing the first version but monitoring, maintaining, and handling when some go down. By using microservices one introduces additional edge cases that don't exist in a "monolith". Quotations around monolith because I believe startups use it with some exaggeration. For example, a Django project of ~10k LoC might be composed of 15 separate Django apps behind one API and I don't see this as monolithic.
I tell every startup that I advise that "revenue solves many problems" because if you have cash flow, everything else becomes easier.. you can buy spot capacity in the short term. You can hire contractors to help refactor. You can rent coworking space for an office.
There are many "good enough" solutions for now. Take advantage of them until you know what your next steps should be.. and then commit accordingly.
E.g. one of your assumptions there are contractors who can help you. That can be true to a certain level, but even product vendors often lack the scale to support their own thing above the basic levels (this is true for post-IPO companies), now imagine this when you can't afford to incubate your product at an expensive tech hub location.
Also, when you have cashflow, your business is under pressure to provide better customer service and marketing, and this is where the things getting tricky: you need to decide whether you want to grow your business or pay back the technical debts. Because the investors priority is always business over tech (rightly), you need to look 1-2 years ahead and make your key technical decisions accordingly.
It's not just the stack, the architecture matters more.
Build a snappy prototype that doesn't scale then sell it to some clueless guy.
I highly recommend Eric's Schmidt take on this: https://youtu.be/hcRxFRgNpns?t=838
Spending your own money provides good discipline in pragmatism, I've found.
The GP points at the transition between the quick & dirty architecture and the architecture that handles Google-scale, but those aren't the only good stable points for a system.
[1] https://michaelfeathers.silvrback.com/the-myth-of-scaling
A CTO should be able to foresee this and act before scaling becomes a behemoth project and no amount of money can buy time-to-market.
That simply is your opinion. You are more likely to lose customers for not focusing on the core product first.
This may not be obvious, since those startups are typically "agile" in the buzzword sense.
The cost comes when the universe of possible change is constrained to the point where the right questions aren't being asked because the answer would be "no we can't do that". So the startup enters a dark age in which scientific thought and rational problem solving isn't really relevant. Often at this point the team starts using Scrum to force the business to plan better, when the core problem is that the existing infrastructure is not able to support the needed experimentation fast enough, making it useless.
This can happen at various stages of funding, since sometimes a series A or B can occur without solid product market fit, or the company may be really good at growth in spite of having a promising but mediocre product.
So I view the CTO's job as preserving the ability of the business to behave scientifically and boldly until product market fit is found.
Sometimes you need to put a twist on the problem to make it a bit more interesting if you want to attract smart people.
Great engineers will not stick around if you call it a day and become a feature factory.
Unless your goal is to retain sub-par seat fillers, who show up for a paycheck. Then you can do whatever you want.
Part of that was when I told the team I wanted our splash page and everything on it to be just static HTML files that would be hand edited if and when it was needed. There was no need for fancy CMS tools for a landing page that gets updated maybe once a quarter.
So, that's part of why you need a CTO - to know when to use static HTML!
If that's a minuscule part of the job, then I agree with you here.
My point is, you are not a CTO if the technology behind the product is a static web site. What do your VP and Directors do all day? What about the teams under them?
There is a huge difference between a CTO and a front-end engineering.
That said, there is an important premise presented by the article. If I can reframe it slightly: don't over-complicate your system architecture, especially not needlessly. If, by selecting some fancy and trendy new technology, you will complicate your architecture or development process for unknown or modest performance gains, it's probably wise to postpone such adoption until the need is more palpable.
My advise, again showing my bias, is quite simple: just select foundation technologies that are mature and high-performance. Using high-performance foundation software gives your application a high performance ceiling, which in turn provides breathing room to build your application in a more "brute-force" fashion. With this approach, you can defer performance tuning and optimization for some time, and perhaps indefinitely.
I have seen plenty of small and small-medium businesses operate well with very modest technology platforms. But importantly, they are modest in architecture (few servers, few processes, low system complexity, simple deployments) because they used high-performance platforms (e.g., JVM, Go, so on). On the flip-side, I've seen many small businesses struggling and prematurely adopting complex system architecture because their software platform (e.g., Ruby, PHP) presents a bottleneck early in the business' life, which creates a challenging scale event before the company has even found their market sweet spot.
When I've written about this in the past, I've pointed out that many people call this situation a "good problem to have," which is of course valid in important ways. But it's also crude and demeaning to your development team. You're basically saying, "Fantastic, our business is scaling. Yes, our servers are on fire and my developers are panicking, but that's great!"
It's better, in my opinion, to have used a high-performance foundation and (potentially) avoid that unnecessary pain. All else being equal (e.g., importantly, your development team is comfortable with a JVM language, for example), I recommend keeping your application simple by using platforms that give you a high ceiling to grow.
You and I completely agree on this: huge scale is a irrational target out of the gate for most projects. The potential for huge scale preoccupies many an architect whose aspirations exceed their budget, to say nothing of their market reality. A premature preoccupation with huge scale tends to yield over-engineered systems that are needlessly complex and work against all of the goals you cite: simplicity and the ability to react quickly to changing business needs.
It seems my addition here is reduced to this: All else being equal—if your team is equally comfortable with two technology options—selecting the platform with a higher performance ceiling will further defer the architectural complexity typically associated with scale. Architectural complexity (e.g., multiple languages, lots of orchestrated or inter-dependent processes, many abstraction layers, complex deployments) tends to be the friction force against the simplicity, speed, and agility we both seek.
except when you're selling an idea to a VC and scaling is all they think about.
For instance, I had a client where a very large online retailer wanted a RESTful API into my client's data that they could call directly from their customer's browser. Before the Black Friday Freeze. This was a completely unforeseen requirement and the solution was NOT to scale the client's overall website to handle the 5-10K requests per second but create a micro service on a new stack that could.
Your startup will pivot, sign weird deals and change your target customer. Optimizing for scale early doesn't help, getting to the point where you know you need to pivot does.
In a cloud environment, it's quite easy to be comfortable that you _could_ scale out to 10-100x of your current capacity, if you have to. But it's probably not worth proving that fact until you have some customers.
Or put another way, being in a very flexible environment makes it even more of a no-brainer to delay building for scale, since you have so many free or nearly-free options there.
Now of course, if you really do hit scale you'll have the fun problem of needing to optimize your spend, but you can probably delay thinking about that until well after you've proven product-market fit.
It does not take more time to design properly from the get go.
That, of course, assumes you have a CTO and a team capable of their job :)
What usually happens is the first thing you build is wrong, and you need to iterate aggressively on not just the software but the entire business concept. Once you have product-market fit, then you are in a position to design an optimal architecture. If you go microservices before that, you are adding unjustifiable overhead, which will inarguably slow you down compared to a monolith when you are in the 1-3 engineer phase, and don't fool yourself about that.
It will all come down the the experience of the team, esp. the leadership - the managers, the leads.
A strong, experienced, mature and effective leadeship will enable a team to perform "magic" and set the right technology foundation that will prove its worth as the business grows over time
Building something quickly that can be broken up over time is the best way to an MVP, and you can deal with scalability issues when you HAVE scalability issues. I consider scalability to be a good problem to have, because by that time you have actual traction.
If you are a bigco with a big marketing budget in a field in which you already have experience and business then by all means scale to millions of users at day one. Don't take on the cost of it in a startup, because the problem is TRACTION not technology with regards to scaling.
Designing and building for scalability, flexibility, availability, etc does not mean going all out to boil the ocean and then find that you attempted to boil the wrong ocean.
If that did happen, you got the wrong team :)
You're making fantastical, hand-wavy claims that belie the hard tradeoffs that need to be made to get a startup off the ground. Microservices absolutely add complexity and require more effort to set up correctly and orchestrate, pretending that experienced leadership makes that problem go away is dangerously idealistic. What experienced leadership gives you is good judgement on whether the overhead is worth it given all the factors at play.
The way you speak about the team, and different roles makes me thing you've maybe worked at "startups" that are actually established businesses with traction which like to retain the mythology and allure of being a startup (perhaps to justify lack of profitability). Things are very different in a true startup where the tiny handful of employees all must wear many different hats.
Honestly curious, I've never seen anyone scale just by "using puppet/chef".
Dumb, often limited scalability is easy now. Elastic Beanstalk approach is only as good as it's the weakest link, which you have no control over.
EC2 is having issues with recovering from standby? Good luck getting anywhere. You can sit on the chat with the useless AWS 'technicians' that are paid to waste your time until the issue is corrected.
Put priority on paid users vs free, to cover server costs