If you are in decision making position, I struggle to see why would you want to spend so much energy installing Prometheus + Grafana stack instead of just using DataDog.
Even worse if your company attempts to build Prometheus/Grafana from scratch. Just why? What a huge waste of money.
If it contributes to your core business then ok, I can see why you may want to build custom solutions.
i think there is a huge difference between using prometheus and paying for datadog. the article argues for not building prometheus which makes sense in 99.95% of cases.
btw, the build vs buy should be a topic of discussion whenever people want to build something. at an absolute minimum they need to understand (and use) what’s already out there before they embark on the journey of building themselves
> so much energy installing Prometheus + Grafana stack instead of just using DataDog.
Because it could be done in few commands the second time. And learning is only possible when wasting time on this kind of things, rather than trying hard to custom fit your usecase using some buyable tool.
That being said, I kind of partially agree with you and the author though. Don't maintain service just to feel that you are in control, but with the exception that if you are doing it for the learning purpose.
On premise Prometheus/Grafana could be lower than 5% TCO compared to DataDog, at least at places I have worked, and I am including employee time. This isn't particulary a discussion about cap ex. vs. op ex. either.
Interesting you should bring this up as we are currently in the process of moving from Datadog to Prometheus/Grafana. What we've found is all the time you save in not setting up Prometheus we lose in support tickets to Datadog to get them to explain undocumented functionality and hidden "magic" they put in place. With Prometheus/Grafana the answers are always out there because of the sheer number of people out there. We've even resorted to sending known metrics on known intervals just to understand what Datadog does differently for monitors vs graphs. Also the majority of the time for a monitoring system is not spent on setting it up but rather on generating the dashboards and alerts.
Also with all buys systems there is a hidden costs in managing the legion of user accounts to access these new services. It's rarely as easy as just setting up SAML or single sign on.
The final thing that people never talk about is cost. Yes there is the cost of an engineer to set it up but often that costs is cheaper than the bought service. This is especially true of pay per api call type services.
I'm totally on board with the overall message that building (i.e. engineering) your own internal tools come with lots of overhead, but take issue with this one point:
> My counter-argument to that is there is also lock-in with internal systems. The most common version of this is the keeper of the spreadsheet.
The author then disparages spreadsheets as becoming the exclusive domain of one employee who wouldn't want processes to change.
In reality and my experience, though, spreadsheets are one of the most versatile and accessible systems, and their close cousins (Airtable, Notion) great as well! You can customize it to your own processes and they're pretty universally understood, so the barrier to change is pretty low.
Author here I actually think spreadsheets are great because they are so accessible. What isn’t great though is when there is a business processs that is a spreadsheet and knowledge that exists in one persons head.
The issue with excel is that there's no standard construction that anyone can query; most likely it's extremely hacked together since it's not a database, but made to look like one. Spreadsheets are great for doing stuff related to exports or before importing, but so, so often they lead to data islands built to either actively or passively protect someone's job.
> their close cousins (Airtable, Notion) great as well!
Their close cousins are locked, proprietary, slow, bloated, exceedingly complex, poorly designed (ui/ux), emojized, vc-backed feature extravaganza and have a subscription fee.
Experts of Excel use keyboard exclusively, their keystrokes are a melody of efficiency, expressivity and productivity that is continued to be mocked in similar fashion as the 2007-era Mac vs. PC advertisements.
I found Notion to be fairly simple to get started. Whereas when I opened Airtable my head exploded and I closed it immediately and haven't revisited it since.
I agree. Notion is a lot easier to get started on. I also should mention that Notion/Airtable vs. Excel are somewhat different types of swissarmy knives to be used in different environments. One is used in the battlefield and space exploration, the other one in a children's playground. Jokes aside, Notion has some nice features like Wiki and it wears too many hats like a joker in a circus. I love the fact that I can write a recipe with tables and markdown -> publish it to the internet to be consumed. Can't do that with Excel.
And it's also important to keep in mind that if your team is using various external tools, there is a high chance that only a couple of people will actually know how those tools work and even less of them will be able or happy to use those external tools.
All of this is true, but sometimes people forget that buying a solution doesn't mean there will be no work. Sometimes, the integration and maintenance of that integration ends up being more than building and maintaining your own solution, especially if you have to do any large changes to how your systems work in order to integrate.
there may be no options. For example, I (profitably) ran a small cloud. My supervisor kept questioning me "why don't we just use openstack instead of your software"? The answer is even in this article!
> Most enterprise systems require an engineering team to keep them running.
If we wanted a team of 3+ to run the cloud then we could buy openstack, or cloudstack, probably 2+, but that's also without pushing features. And suddenly we wouldn't be profitable anymore. I left, so I guess they will find out.
Not always a poor job, but sometimes a bet that turns out wrong. I have seen great products and companies disappear in unforeseen mergers or API prices skyrocketing for instance.
Risk is low for commodities, but high for anything "disruptive".
The people that have to perform the technical integration between two independently developed products (and could warn that the integration work is more involved than starting from scratch) are a different group from who decides the acquisition is happening.
Nowadays you're often buying SaaS so not only do you need to ensure the product is a fit currently but you must predict the fit going into the future.
It's not an easy thing to do. I've found that open source libraries are often more stable, require less ongoing maintenance due to API changes and have better support lifetimes compared to the SaaS equivalent.
That's very true. Especially for tooling/products/services that sells themselves as shrinkwrapped products that only allow for a particular flow - sometimes reimplementing your own subset from scratch that's geared towards your usecase is less effort in the long run than hacking around a blackbox solution.
I'd much rather occasionally work on feature for a well-engineered internal reimplementation than constantly fight with shellscripts wrapping around a solution with a huge impedance mismatch.
Exactly. I've even had this happen with things like CSS, where a master stylesheet changes somewhere upstream on another server, and hundreds of hours of my own custom, vendor-supported modifications now make my web application or website look like doo-doo.
Plus maybe I'm weird but I think it's enough to love the energy you can get from building something. It's fun. And considerable portions of work should be fun. Especially if you work for yourself, where building stuff for money, for other people, often isn't enough without a strong connection between the work, customer, and the subject's own interests and values system.
(IMO a lot of those kinds of projects also build on the edge of one's core expertise, extending it outwards, almost like a recon mission, so it's less of a binary yes/no core expertise condition.)
> I think it's enough to love the energy you can get from building something
I think this is good if you can get there. The IKEA of software is pure dopamine. If had to grow and saw those trees less so. If you are pouring concrete and get it delivered, pure dopamine. If you have to crush the aggregate by hand, less so.
My job involves a lot of integration work so I'm probably biased, but I find successfully cracking open a poorly documented API and extracting all the data in a usable way to be deeply satisfying, especially if you can be confident enough in your edge cases that you think it'll run until the next unannounced and undocumented change comes along.
True. At that, one thing about the buying vs building argument that's missing is picking and choosing your battles. Competitive advantage comes from that. Build what you are capable of producing effectively based on your environment. Obviously you can run into roadblocks, but part of being an adult is planning and adjusting based on your circumstances and needs, not along someone else's. Too many of these arguments fall into the realm of, "Everyone, listen up! I have the single solution for everyone's problem for all of eternity!"
That is why likes of Office 365 are popular, each component individually may not be best, but you get dozens of apps that are active in one go, for most parts are integrated well and have a simple billing.
In unreal and unity, most of the third party code I’ve tried to use ended up unusable because of either bugs, performance issues, or limitations. There are great exceptions but it’s been frustrating overall. The ones that do work, well, the documentation is often terrible and it takes longer to learn and integrate than it would to make your own. Finally, unity has a history of retroactively changing their terms of service to squeeze more money out of you for asset store buys, so it’s scary to trust them. I write a lot more of my own systems now.
unity (and maybe unreal) have a real problem in that the majority of users are hobbyists, students, and new programmers. None of them are really willing to pay much for 3rd party code and they need $$$$$$$$$$$$$ of support because they are new.
For example make a JSON serialization library and you'll be asked "how do I spawn enemies from JSON" which is arguably entirely orthogonal to your library but don't answer and get downvoted for bad support. Do answer and you'll just be asked more questions about how to adapt your example to their personal project and you'll have to teach them programming in the process.
These incentives make it almost impossible to make money selling 3rd party code (plugins/add-ons). If you charge what it should really cost given the amount of work put in the market won't bare it. If you charge what the market will bare you'll go broke.
Maybe Unity should split the market into "Pro" and "Hobbyist" and the Pro market would have prices more inline with what it actually costs. Check out the prices of libraries like Radgametools.com (you'll have to google the prices) for comparison.
Unity also changes their APIs around with every tiny version increment, meaning that any app store package has to be constantly on top of installing new versions of unity and updating all the little parts that break every week.
+1. And I'd add to that that sometimes there's significant work in the pre-buy phase as well:
- Researching what your options are / what's already out there.
- Comparing different alternatives.
- "Hopping on a call" with a sales rep to get a product demo (there's this super annoying trend where many SaaS companies' landing pages don't explain what they do and the only option they give is to "schedule a demo").
For CRUD-like internal tools or simple 3rd party integrations, my experience has been that it's often much faster (typically < 1 hour) to build a production-ready app on Retool (https://retool.com) than it is to even get started with SaaS vendors.
We recently purchased an enterprise scheduling software that I will not name (you may have heard of it, but it's unlikely). Told numerous times throughout the sales cycle by their architects and engineers that their API could do all the things we needed "easily" and that they could finish off the last few features we needed within 2-3 minor releases.
Fast forward almost a year, at least 7 releases, and probably $200k of payroll on our end and their product barely runs, and doesn't handle all of our standard workflow, let alone edge cases. Best we can assume is they were writing the API from scratch because we needed it and didn't have the skill to do so. Documentation was offensively bad when it was correct, which most of the time it was not, to the point where there were several instances of the endpoints in the docs being wrong, causing us to call support to ask what the real endpoints were we were supposed to be calling.
All this to say yeah, buying is great if the company is professional and has a well-documented product, and the sales and technical pre-sales folks know what they're talking about. But that isn't necessarily the case, even for six-figure purchases. And when it's not the case you can easily burn a non-trivial amount of money before you ever realize it.
I have been in the boat of the company that ripped you off before. Sales will promise the world, product/engineering caves “just one last time” to help win the deal with the easiest shittiest thing. Years go by and things inexplicably break all the time.
It’s such an easy trap to fall into, because I hear similar stories all the time.
I guess the moral of the story is to assume sales is lying to you until proven otherwise? Haha not sure, just do as much research as you can but you’ll probably get burned anyway from time to time
Why don't more companies mandate that their engineers talk to the company's engineers, alone, in an unrecorded room, before agreeing to buy?
I mean... talking to people with every incentive in the world to stretch the truth and/or little technical acumen doesn't sound like a recipe for success.
You get in a room with Intel, you have to sign a waiver that gives Intel total property over everything discussed in that room. Even your own products, if you happen to mention something about them. No sensible company will let their Engineers in an Intel conference room, if they have any sense.
Isn't that a win for everyone except the sales rep though?
If the deal closes, Z% of the company ends up being tied up in trying to kludge Y into doing what was sold, spiking engineer burnout and lowering morale, and furthering a negative relationship with sales.
Plus you've now pissed off your new customer, by lying to them.
I worked for a FAANG where this was part of my job. I would often come into rooms and question why the client was spending money on a product that didn't appear to suit their needs.
This was incredibly effective, as it meant that the client actually trusted us when we said something would work.
That being said, I'd normally avoid calling our products crap (even when they were) and just push the client to use something that wasn't crap.
This only worked though, because we had a separate reporting line to sales, so any VP pressure had to go through our VP (which happened, but not as often as you'd think).
My understanding is that they changed this after I left, with predictable consequences.
That’s the problem, isn’t it: Sales is a short-term metric (quarterly?), whereas client retention is long-term (multi-year, depending on contract length).
I had the pleasure of carpooling with some random owner of a small tech firm, and he flat out said it was difficult to keep on top of salespersons.
Meaning they were pretty shifty by nature and hard to trust.
One of the worst times I had as a developer was when my direct supervisor was a salesperson, who put me on a project that she had sold, and she was also the PM on the project. I thought I was going to die.
I’ve been on calls almost like this. What you get are solutions/sales engineers accompanied by sales. They’re all generally helpful, including the engineer to engineer conversations. I’ve even been in a position where a representative of a vendor worked directly with us for months, yet the same issues kept coming up ultimately leading to the project being scrapped. It’s a weird world where everyone is trying to sell something to everyone, I really don’t like it
There's negative incentive for the selling company to do this. Too much of a risk that your prospective customer's engineers are just there for inspiration.
> assume sales is lying to you until proven otherwise
Having dealt with enterprise sales countless times, yes they do and I yes I do presume so.
They lie in many ways, the most innocent one is by demoing unrealistic perfect scenarios without any corner cases, promising integrations that do not exist ("it's in the roadmap!") etc.
This is so standard it doesn't even deserve mentioning.
Ah yes, this is how SAP and Oracle make their money: you "buy" a "COTS" solution, and then spend ten times as much having them build something for you .. which in the end you don't own.
Precisely. There is a cost to integrating off the shelf software which people often forget. Less so in small startups, but in large enterprises this is non-trivial.
I've heard that large companies don't mind SaaS because they used to budget 30% or more of a software purchase's sticker price on annual support and maintenance.
Good article, I am definitely filing away to pull out in some future team discussions.
This paragraph is confusing me, either it's missing a 'not' or something, or I'm just confused. Is anyone following?
> The question you should be asking is what else could be done instead of tuning your own stuff or building a new internal system. The answer is usually spending more time coming up with the correct architecture instead of fighting fires or developing actual customer-facing features.
Instead of building internal systems you would be coming up with the correct architecture instead of fighting fires or developing actual customer-facing features? Wait, what? there are too many 'instead of's in there and i'm not sure they are all meant how they are said.
I think developing actual customer-facing features is the goal, but here it sounds like the thing you would like to avoid... and I'm not really sure about fighting fires (ideally you want a system where you do less of that, right?) or coming up with the correct architecture (?).
I think he's right, if you're at a small/medium sized place, all the reasons he lists there are solid reasons to buy and not build things for yourself. That being said, there's no reason to make that an absolute rule, I think it's ok to build some things and buy some things. Just think of all the things we've ended up with because someplace built some thing and it ended up being some big thing that we all know. Isn't that how Slack and Docker both got started?
I own a Screenshot API service (shameless plug: https://getscreenshot.rasterwise.com/) and this exactly the argument that I make for small utilitarian services like mine. You shouldn't build them because buying them is several orders of magnitude cheaper.
I have a small paragraph from a blog post where I explain this with some math that is probably crappy but captures the idea:
"When you spend time in areas that are not the core of your product, you're actually being financially inefficient. Taking screenshots is likely not a core task of your business so it doesn't makes much sense to waste development resources in this area.
Here is a brief example of how inefficient it gets:
A mid-level developer in a small market earns $90K USD per year, working 40 hours per week. His/her effective hourly rate is $47 USD per hour.
Writing a basic implementation of a screenshot utility will take at least 5 hours. But this is just a simple prototype. Many use cases will need to be addressed, and there's likely going to be a large amount of time spent in optimizing, securing, provisioning, testing, etc.
A realistic utility that can be used for a development workflow is going to take at the very least 30 hours of development time, but likely more.
Other un-accounted areas of development time are: documentation, training and maintenance.
Given this scenario is very likely that writing a semi-good solution will take more than a week and maintaining it, will take at least an hour every month.
This means that writing this solution will cost almost $2000 USD of development time plus another $500 USD or so, just to support it every year. And of course, this doesn't include the cost of the infrastructure you're paying to run it."
My argument is that the more utilitarian the service, the more cost-effective that is to buy it instead of writing it. Writing it could be simple but there's a drain in engineering resources when you try to maintain it in the long run.
I think the screenshot service is a perfect example of a service suited for buy don't build - it's something complex to implement but with a really simple API to integrate with.
Yah it’s a pretty ideal example. Another might be CI. It seems from reading other comments that bad examples in buying external products are from complex “business apps”.
One of the few things I appreciate from my abusive first boss was making me acutely aware of the value of my own time.
He was incredibly stingy with money and always wanted to know how much it would cost him to have his engineering team implement something vs how much it would cost to buy it and integrate it.
Admittedly he was pathologic about this to the point of having us create a timer that counted in pounds sterling that he'd start running at the start of every meeting, but I feel like it made me way more efficient with my time (and hate long meetings)
I had a similar client early in my career. Working for him was a love/hate relantionship. He tripled my hourly rate telling me my time was highly valuable but then had very high standards and aggressive timelines.
I think you had great boss (assuming he did not start timer each time you stopped typing on keyboard). Those endless meetings become unproductive pretty fast. I suspect that in big part they become a self sustained cancer used to justify time and salaries of unproductive company members.
As for the subject itself: frankly I do not see any advantage of hosting on say Azure over hosting myself. Either requires a good deal of maintenance. And no you can not really rely on Azure doing it for you.
Uptime for me is probably just as good on my servers as each one I have in my office also has ready to use up to date shadow copy located elsewhere. Azure is way more expensive of course.
The amount of hardware/processing power I have on my own server would cost me a fortune to have on Azure. Scalability does not matter much either because:
I am not Google and do not have to serve the rest of the world.
On top of that my servers are usually high performance native C++ applications that can handle thousands of requests per second sustainably and without breaking a sweat.
Vertical scalability with modern CPUs and multilevel storage is absolutely insane.
> frankly I do not see any advantage of hosting on say Azure over hosting myself.
In my opinion, cloud hosting only makes sense if your systems are architectured around being hosted on the cloud.
If you can build your application in such a way that it runs in the free tier of serverless, then it's going to be considerably more cost effective than renting a dedicated server - but if you can't, then it absolutely won't be.
>"If you can build your application in such a way that it runs in the free tier of serverless"
Serverless - that would be vendor/architectural trap. Besides free tier does not come anywhere close to be able to serve my applications. They serve real medium/big size businesses.
You mentioned serverless and architecting for it. I consider it vendor and technological lock in with no benefits. You might have a different opinion but to each their own.
You also mentioned free tier. To me it is irrelevant as the amount of resources it gives is useless for me.
You seem to want me to defend Azure, but you're the one that brought it up. I have no prescriptive views on how to host things.
The only thing I can really offer you is that in the spirit of this article, when I did some work that would run on AWS lambda it was very handy not to have to think about any of the infrastructure that was around the business logic I needed to code.
I fully expect that code will spend its entire useful lifetime running in AWS Lambda with no need for it to escape the vendor or technology behind it. It's even possible no one will think about it until it breaks and stops sending events.
If it costs the company £50/hour for me to look into something, me being able to complete a task quickly and then never look into it again is almost certainly going to save more money than writing a dedicated process and running it on a physical server that I then need to maintain.
Some engineers love to say "I can build this over a weekend". But the main cost is almost always on the maintenance side.
I forgot where I saw this number - in the entire lifecycle of a piece of software, (on average) maintenance cost is at least 8x of the initial dev cost.
Yes, buying a solution also costs time (and $$$) to integrate and maintain, e.g., using a 3rd party API - time to write code, time to set up monitoring / alerting, error handling... There are always exceptions, edge cases, special situations...
But in general there are fewer and fewer things that a web company needs to build from scratch. It takes less engineering time to launch a web product than before, thanks to all those out of box solutions, e.g., SaaS, apis...
I like building a software/web business today than 10 years ago :) One-person (or tiny team) web businesses will be very common (I'm running one [1]), and running an API business is not bad because it actually saves customers time & money(vs building one in-house) [2] thus they are willing to pay a small fee (compared with hiring one or more full-time engineers).
I think "maintenance" somewhat mischaracterizes it. In my experience, it's often that an engineer can build the typical use case over a weekend, but will have to continue pouring in more dev time while it's in production to handle a huge number of edge cases that weren't properly handled originally. That work is only "maintenance" because you already claimed the work was completed, and finance has it marked as having been put into service, so you can't capitalize further costs against it.
Yea, in software development, it’s always work in progress. There’s not a clear finish line.
The word “maintenance” can’t perfectly capture the meaning of all the continuous development, small incremental improvements, bug fixes, operational tasks...
These sorts of discussion always seem somewhat confused to me because "buying" vs "building" is not a binary choice. It is more often a choice between:
1. Buy some platform/framework which you will then need to hire a small army of costly consultants to integrate and customize for your particular business need. Or
2. "Build" your own solution by orchestrating a bunch of open source technologies to solve your problem.
Moreover, it is not quite clear up front whether 1 or 2 will be costlier in terms of development effort and overhead.
It would seem to me there are still two choices to make, meaning it’s still a binary choice-merely with more caveats and words involved to describe the options?
I think it's a matter of unknown future needs that's the problem. If you want to do feature flags, you can basically build a service that stores key value pairs segemented however you please, throw it behind redis cache, and call it a day. This is faster and probably cheaper than buying a SaaS!
However, if you want a bunch of neat features, custom rollouts, and all that stuff, you're simply not going to be able to get the same value (unless you have HUGE scale) building it, and should just buy it.
Most organizations aren't comfortable saying "this is all we'll ever need" and are worried about both building, buying, and then migrating, which can be significantly more expensive than either of those options in a vacuum.
In my experience, integrating a thing like feature flags is a major pain if you do not have control over the implementation. It ends up being kludges and suboptimal choices everywhere.
Agree with @Aeolun's comment. Feature Flagging is one exception in buy vs. build where there just isn't any good SaaS out there that makes it "just buy it" worth it. More so if you need customizations beyond what is offered.
Yeah, hosting feature flags in some service is pretty easy (just use ParameterStore!). The hard part is actually using the flags in your application code, which I don't think is something that is easily solved in general
1 There is some other expense you can save significantly on by DIY. An example would be DIY k8s on bare metal via bare metal providers for a massively bandwidth intensive service where paying AWS or GCP outbound data rates would be thousands and thousands a month.
2 You are at sufficient scale that there is an economy of scale that can be accessed. This overlaps a bit with the first point but can also happen if you are just big enough. The first point is more about some special need.
This is where free & opensource has really changed the world. Its not buy vs build any more its download vs buy vs build and its best to use a free package that is out there. Its incredible what is available now, in the future I wonder if anyone will need to build anything.
Sometimes "building" means writing 6 shell scripts (each <10 LoC) and "buying" means getting CKA/CKAD certificates for all people involved and spending half a year on training.
Yeah, doing work to automate some idiosyncratic process, either for clients or internal users, may be different. You do have to be careful you're not slowly implementing a CMS, CRM, email marketing tool, etc, feature by feature, though.
I have to disagree.. Not everyone has enough time to learn "ready" systems either, they can get pretty overwhelming. When I decide to build instead of buy, it's mostly because I don't believe in the design of the product that is favored by thousand people with reason being "it just sounds cool".
This works great until you hire a second person who now has to figure out how your system works from the documentation you wrote, rather than the documentation and blogposts on the internet written by the thousands of people who thought the other thing was cool.
From an engineer perspective there is a learning opportunity that can’t be underestimated. If you have free time to build it’s always interesting how much you can grow by doing it yourself. However, if you are working for a startup and time to market is essential, than buy it for sure.
It's all trade-offs. At work a customer could only send some data via mail, our integration could only accept via SFTP.
I had just had a look at Azure and had seen their Logic Apps, and quickly whipped up one which downloaded the mail and uploaded the files to SFTP. Nice.
Until a few months ago when it suddenly stopped working because for some reason Azure doesn't like the key exchange algorithms Bitvise SSH server provides. Or at least, that's as far as I've been able to determine. Zero logging on the Azure side so no clue what's wrong.
After spending hours trying to figure out what went wrong, I made my own program that does the same. It took a bit longer than the Logic App, but at least I can fix it when it breaks...
The most egregious cases of violating this involve trying to create your own CMSs, search engines, and God forbid - databases. There are of course, exceptions. AirTable wrote their OWN database engine, but they knew what they were getting into, and they did it right.
I can't speak to CMS, but are search engines and databases really such a big deal? If postgres or influx or something kinda fits your workload then it's probably not worth the time, but on the other hand I've seen both custom databases and the cludges people have written to work around off-the-shelf DBs, and I vastly prefer the former both in terms of performance and maintainability in a problem space (similarly with search engines).
The problem with search engines is that people think it's just tokenizing the search query and then doing an AND query for the words, but then the fun starts. First of all, there is specific search theory that one has to dive into and have the Knuth tomes nearby. Then, different languages, thesaurus, taxonomies, "did you mean". And we are not even at indexing speed, replication, performance, etc. There is absolutely no chance you would be able to create and maintain a reasonably good search engine part time.
I think all four points in the article are deeply wrong.
1. It is easy to run your own services. You should be investing in internal developer tooling that makes this easy, and in fact that developer tooling should sit in front of any vendor solution you ever buy so that the way it integrates with your alerting, monitoring, data exporting, etc., is completely standardized to be uniform with service delivery in any other system in the org.
2. and 3. You do need complete control over what the application does because it absolutely always is unique and special on a per-use-case basis every time. A good example is search. Anyone who thinks search is a commodity service you can just throw ElasticSearch or Algolia in front of is sorely mistaken and dangerously naive. Every different search use case is going to have different success criteria, different data privacy concerns, different timeliness and freshness concerns, etc. and you need business software to control these elements in ways that fit into standard internal product management and QA procedures.
4. Vendor lock-in is a critical problem. If you choose GCP vs AWS, you are defining culture and you are defining experimentation and exploration that you cannot do. You’re essentially cleaving away many future possibilities from even being testable. It’s much worse than just having a crufty old system to maintain, it’s about brittleness and lack of ability to appropriately empower engineers to consider whatever part of the solution space they decide is needed. Companies that “get it” will prioritize “ease of swapping” so that you can constantly improve and leverage autonomy without needless parochial constraints on what can be considered. Thinking, “yeah but just buying it solved our problem today” is such a death knell of weak leadership who cannot fathom strategy or how to leverage real solution ideation from their staff.
Thank you for your answer. I was desperately scrolling in search of one like this to support.
I'm especially angry at the dismissal of the vendor lock-in problem in the main article. I've seen quite a few start-ups being trapped by a vendors which were very cheap at first (trial period, starter plans, etc.) and ended up eating much of the profits. Not to mention the innovation cost...
> The desire to build custom versions of everything seems to come from a few places: 1 [...] 2 [...] 3 [...] 4
There are personal reasons as well:
5. Innate curiosity and excitement about technology
6a. Get experience
6b. Resume talking point
Many of us picked a career in IT because just like building things out of Legos is fun, building a streamlined full CI/CD pipeline is fun, building a full stack application is fun, etc. Speaking of career, one needs to acquire experience in the new technologies to pad their resume and it's convenient to do it on company time.
I'm not justifying putting your interests ahead of your company's, but understand that's what some of us do. I'd say #5 and #6 are often stronger drivers for decisions than #1-4.
227 comments
[ 6.4 ms ] story [ 242 ms ] threadIf you are in decision making position, I struggle to see why would you want to spend so much energy installing Prometheus + Grafana stack instead of just using DataDog.
Even worse if your company attempts to build Prometheus/Grafana from scratch. Just why? What a huge waste of money.
If it contributes to your core business then ok, I can see why you may want to build custom solutions.
btw, the build vs buy should be a topic of discussion whenever people want to build something. at an absolute minimum they need to understand (and use) what’s already out there before they embark on the journey of building themselves
Because it could be done in few commands the second time. And learning is only possible when wasting time on this kind of things, rather than trying hard to custom fit your usecase using some buyable tool.
That being said, I kind of partially agree with you and the author though. Don't maintain service just to feel that you are in control, but with the exception that if you are doing it for the learning purpose.
Also with all buys systems there is a hidden costs in managing the legion of user accounts to access these new services. It's rarely as easy as just setting up SAML or single sign on.
The final thing that people never talk about is cost. Yes there is the cost of an engineer to set it up but often that costs is cheaper than the bought service. This is especially true of pay per api call type services.
> My counter-argument to that is there is also lock-in with internal systems. The most common version of this is the keeper of the spreadsheet.
The author then disparages spreadsheets as becoming the exclusive domain of one employee who wouldn't want processes to change.
In reality and my experience, though, spreadsheets are one of the most versatile and accessible systems, and their close cousins (Airtable, Notion) great as well! You can customize it to your own processes and they're pretty universally understood, so the barrier to change is pretty low.
Their close cousins are locked, proprietary, slow, bloated, exceedingly complex, poorly designed (ui/ux), emojized, vc-backed feature extravaganza and have a subscription fee.
Experts of Excel use keyboard exclusively, their keystrokes are a melody of efficiency, expressivity and productivity that is continued to be mocked in similar fashion as the 2007-era Mac vs. PC advertisements.
https://www.microsoft.com/en-us/microsoft-365/p/office-home-...
The alternative could instead be a system with open standards that many vendors implement, and only relying on standardized behavior.
This works to some extent with for example SQL or C, where you can migrate from one DB or compiler to another with limited effort.
> Most enterprise systems require an engineering team to keep them running.
If we wanted a team of 3+ to run the cloud then we could buy openstack, or cloudstack, probably 2+, but that's also without pushing features. And suddenly we wouldn't be profitable anymore. I left, so I guess they will find out.
Risk is low for commodities, but high for anything "disruptive".
It's not an easy thing to do. I've found that open source libraries are often more stable, require less ongoing maintenance due to API changes and have better support lifetimes compared to the SaaS equivalent.
I'd much rather occasionally work on feature for a well-engineered internal reimplementation than constantly fight with shellscripts wrapping around a solution with a huge impedance mismatch.
Plus maybe I'm weird but I think it's enough to love the energy you can get from building something. It's fun. And considerable portions of work should be fun. Especially if you work for yourself, where building stuff for money, for other people, often isn't enough without a strong connection between the work, customer, and the subject's own interests and values system.
(IMO a lot of those kinds of projects also build on the edge of one's core expertise, extending it outwards, almost like a recon mission, so it's less of a binary yes/no core expertise condition.)
I think this is good if you can get there. The IKEA of software is pure dopamine. If had to grow and saw those trees less so. If you are pouring concrete and get it delivered, pure dopamine. If you have to crush the aggregate by hand, less so.
For example make a JSON serialization library and you'll be asked "how do I spawn enemies from JSON" which is arguably entirely orthogonal to your library but don't answer and get downvoted for bad support. Do answer and you'll just be asked more questions about how to adapt your example to their personal project and you'll have to teach them programming in the process.
These incentives make it almost impossible to make money selling 3rd party code (plugins/add-ons). If you charge what it should really cost given the amount of work put in the market won't bare it. If you charge what the market will bare you'll go broke.
Maybe Unity should split the market into "Pro" and "Hobbyist" and the Pro market would have prices more inline with what it actually costs. Check out the prices of libraries like Radgametools.com (you'll have to google the prices) for comparison.
- Researching what your options are / what's already out there.
- Comparing different alternatives.
- "Hopping on a call" with a sales rep to get a product demo (there's this super annoying trend where many SaaS companies' landing pages don't explain what they do and the only option they give is to "schedule a demo").
For CRUD-like internal tools or simple 3rd party integrations, my experience has been that it's often much faster (typically < 1 hour) to build a production-ready app on Retool (https://retool.com) than it is to even get started with SaaS vendors.
- https://www.forestadmin.com // Fastest way to build self-hosted admin panels on top of SQL (Postgres, MySQL...) and Mongo databases
- https://www.appsmith.com // Open source alternative to retool
- https://www.internal.io // A no-code alternative to all the above
- https://www.basedash.com // Very spreadsheet like XP, YC20 startup
Fast forward almost a year, at least 7 releases, and probably $200k of payroll on our end and their product barely runs, and doesn't handle all of our standard workflow, let alone edge cases. Best we can assume is they were writing the API from scratch because we needed it and didn't have the skill to do so. Documentation was offensively bad when it was correct, which most of the time it was not, to the point where there were several instances of the endpoints in the docs being wrong, causing us to call support to ask what the real endpoints were we were supposed to be calling.
All this to say yeah, buying is great if the company is professional and has a well-documented product, and the sales and technical pre-sales folks know what they're talking about. But that isn't necessarily the case, even for six-figure purchases. And when it's not the case you can easily burn a non-trivial amount of money before you ever realize it.
It’s such an easy trap to fall into, because I hear similar stories all the time.
I guess the moral of the story is to assume sales is lying to you until proven otherwise? Haha not sure, just do as much research as you can but you’ll probably get burned anyway from time to time
I mean... talking to people with every incentive in the world to stretch the truth and/or little technical acumen doesn't sound like a recipe for success.
I'm assuming Salesforce (e.g.) isn't going to suddenly pivot into Lyft-for-dogs, or whatever the product is.
I'm pretty sure my company wouldn't put me on the front lines during sales for reasons like this
Customer "How are you at X and Y"
Me "Oh, our X is really good, on the other hand our Y is utter shite"
Sales rep next to me: evaporates
If the deal closes, Z% of the company ends up being tied up in trying to kludge Y into doing what was sold, spiking engineer burnout and lowering morale, and furthering a negative relationship with sales.
Plus you've now pissed off your new customer, by lying to them.
This was incredibly effective, as it meant that the client actually trusted us when we said something would work.
That being said, I'd normally avoid calling our products crap (even when they were) and just push the client to use something that wasn't crap.
This only worked though, because we had a separate reporting line to sales, so any VP pressure had to go through our VP (which happened, but not as often as you'd think).
My understanding is that they changed this after I left, with predictable consequences.
I had the pleasure of carpooling with some random owner of a small tech firm, and he flat out said it was difficult to keep on top of salespersons.
Meaning they were pretty shifty by nature and hard to trust.
This is why pilots exist.
Having dealt with enterprise sales countless times, yes they do and I yes I do presume so.
They lie in many ways, the most innocent one is by demoing unrealistic perfect scenarios without any corner cases, promising integrations that do not exist ("it's in the roadmap!") etc.
This is so standard it doesn't even deserve mentioning.
This paragraph is confusing me, either it's missing a 'not' or something, or I'm just confused. Is anyone following?
> The question you should be asking is what else could be done instead of tuning your own stuff or building a new internal system. The answer is usually spending more time coming up with the correct architecture instead of fighting fires or developing actual customer-facing features.
Instead of building internal systems you would be coming up with the correct architecture instead of fighting fires or developing actual customer-facing features? Wait, what? there are too many 'instead of's in there and i'm not sure they are all meant how they are said.
I think developing actual customer-facing features is the goal, but here it sounds like the thing you would like to avoid... and I'm not really sure about fighting fires (ideally you want a system where you do less of that, right?) or coming up with the correct architecture (?).
I have a small paragraph from a blog post where I explain this with some math that is probably crappy but captures the idea:
"When you spend time in areas that are not the core of your product, you're actually being financially inefficient. Taking screenshots is likely not a core task of your business so it doesn't makes much sense to waste development resources in this area.
Here is a brief example of how inefficient it gets:
A mid-level developer in a small market earns $90K USD per year, working 40 hours per week. His/her effective hourly rate is $47 USD per hour.
Writing a basic implementation of a screenshot utility will take at least 5 hours. But this is just a simple prototype. Many use cases will need to be addressed, and there's likely going to be a large amount of time spent in optimizing, securing, provisioning, testing, etc.
A realistic utility that can be used for a development workflow is going to take at the very least 30 hours of development time, but likely more.
Other un-accounted areas of development time are: documentation, training and maintenance.
Given this scenario is very likely that writing a semi-good solution will take more than a week and maintaining it, will take at least an hour every month.
This means that writing this solution will cost almost $2000 USD of development time plus another $500 USD or so, just to support it every year. And of course, this doesn't include the cost of the infrastructure you're paying to run it."
My argument is that the more utilitarian the service, the more cost-effective that is to buy it instead of writing it. Writing it could be simple but there's a drain in engineering resources when you try to maintain it in the long run.
He was incredibly stingy with money and always wanted to know how much it would cost him to have his engineering team implement something vs how much it would cost to buy it and integrate it.
Admittedly he was pathologic about this to the point of having us create a timer that counted in pounds sterling that he'd start running at the start of every meeting, but I feel like it made me way more efficient with my time (and hate long meetings)
As for the subject itself: frankly I do not see any advantage of hosting on say Azure over hosting myself. Either requires a good deal of maintenance. And no you can not really rely on Azure doing it for you.
Uptime for me is probably just as good on my servers as each one I have in my office also has ready to use up to date shadow copy located elsewhere. Azure is way more expensive of course.
The amount of hardware/processing power I have on my own server would cost me a fortune to have on Azure. Scalability does not matter much either because:
I am not Google and do not have to serve the rest of the world.
On top of that my servers are usually high performance native C++ applications that can handle thousands of requests per second sustainably and without breaking a sweat.
Vertical scalability with modern CPUs and multilevel storage is absolutely insane.
In my opinion, cloud hosting only makes sense if your systems are architectured around being hosted on the cloud.
If you can build your application in such a way that it runs in the free tier of serverless, then it's going to be considerably more cost effective than renting a dedicated server - but if you can't, then it absolutely won't be.
Serverless - that would be vendor/architectural trap. Besides free tier does not come anywhere close to be able to serve my applications. They serve real medium/big size businesses.
You also mentioned free tier. To me it is irrelevant as the amount of resources it gives is useless for me.
The only thing I can really offer you is that in the spirit of this article, when I did some work that would run on AWS lambda it was very handy not to have to think about any of the infrastructure that was around the business logic I needed to code.
I fully expect that code will spend its entire useful lifetime running in AWS Lambda with no need for it to escape the vendor or technology behind it. It's even possible no one will think about it until it breaks and stops sending events.
If it costs the company £50/hour for me to look into something, me being able to complete a task quickly and then never look into it again is almost certainly going to save more money than writing a dedicated process and running it on a physical server that I then need to maintain.
Some engineers love to say "I can build this over a weekend". But the main cost is almost always on the maintenance side.
I forgot where I saw this number - in the entire lifecycle of a piece of software, (on average) maintenance cost is at least 8x of the initial dev cost.
Yes, buying a solution also costs time (and $$$) to integrate and maintain, e.g., using a 3rd party API - time to write code, time to set up monitoring / alerting, error handling... There are always exceptions, edge cases, special situations...
But in general there are fewer and fewer things that a web company needs to build from scratch. It takes less engineering time to launch a web product than before, thanks to all those out of box solutions, e.g., SaaS, apis...
I like building a software/web business today than 10 years ago :) One-person (or tiny team) web businesses will be very common (I'm running one [1]), and running an API business is not bad because it actually saves customers time & money(vs building one in-house) [2] thus they are willing to pay a small fee (compared with hiring one or more full-time engineers).
[1] https://www.listennotes.com/blog/the-boring-technology-behin...
[2] https://www.listennotes.com/blog/how-i-accidentally-built-a-...
The word “maintenance” can’t perfectly capture the meaning of all the continuous development, small incremental improvements, bug fixes, operational tasks...
1. Buy some platform/framework which you will then need to hire a small army of costly consultants to integrate and customize for your particular business need. Or
2. "Build" your own solution by orchestrating a bunch of open source technologies to solve your problem.
Moreover, it is not quite clear up front whether 1 or 2 will be costlier in terms of development effort and overhead.
1. "Buying" a solution that solves your problem exactly and doesn't require engineering resources to implement and maintain
2. "Building" a solution where you have to solve every problem from scratch where you don't have any particular expertise.
> is not a binary choice
Then, offering two options. :)
However, if you want a bunch of neat features, custom rollouts, and all that stuff, you're simply not going to be able to get the same value (unless you have HUGE scale) building it, and should just buy it.
Most organizations aren't comfortable saying "this is all we'll ever need" and are worried about both building, buying, and then migrating, which can be significantly more expensive than either of those options in a vacuum.
this is where experience matters.
1 There is some other expense you can save significantly on by DIY. An example would be DIY k8s on bare metal via bare metal providers for a massively bandwidth intensive service where paying AWS or GCP outbound data rates would be thousands and thousands a month.
2 You are at sufficient scale that there is an economy of scale that can be accessed. This overlaps a bit with the first point but can also happen if you are just big enough. The first point is more about some special need.
Sometimes "building" means writing 6 shell scripts (each <10 LoC) and "buying" means getting CKA/CKAD certificates for all people involved and spending half a year on training.
What am I supposed to buy, except some PDF/Excel/whatever library every now and then?
But maybe I'm just not the target audience for this article.
If the answer is no, lean towards buy. If the answer is yes, lean towards build.
For ops, netflix would say "yes" whereas most companies would say no.
I had just had a look at Azure and had seen their Logic Apps, and quickly whipped up one which downloaded the mail and uploaded the files to SFTP. Nice.
Until a few months ago when it suddenly stopped working because for some reason Azure doesn't like the key exchange algorithms Bitvise SSH server provides. Or at least, that's as far as I've been able to determine. Zero logging on the Azure side so no clue what's wrong.
After spending hours trying to figure out what went wrong, I made my own program that does the same. It took a bit longer than the Logic App, but at least I can fix it when it breaks...
It’s cost dependent and you should carefully study both options.
It’s part of the engineering: study fixed and variable costs.
Outsourcing is cost effective when the market is mature enough.
You should maybe not rebuild cloudflare’s core services to operate a website.
Depending on your scale, you certainly should build your own ML stack for instance.
1. It is easy to run your own services. You should be investing in internal developer tooling that makes this easy, and in fact that developer tooling should sit in front of any vendor solution you ever buy so that the way it integrates with your alerting, monitoring, data exporting, etc., is completely standardized to be uniform with service delivery in any other system in the org.
2. and 3. You do need complete control over what the application does because it absolutely always is unique and special on a per-use-case basis every time. A good example is search. Anyone who thinks search is a commodity service you can just throw ElasticSearch or Algolia in front of is sorely mistaken and dangerously naive. Every different search use case is going to have different success criteria, different data privacy concerns, different timeliness and freshness concerns, etc. and you need business software to control these elements in ways that fit into standard internal product management and QA procedures.
4. Vendor lock-in is a critical problem. If you choose GCP vs AWS, you are defining culture and you are defining experimentation and exploration that you cannot do. You’re essentially cleaving away many future possibilities from even being testable. It’s much worse than just having a crufty old system to maintain, it’s about brittleness and lack of ability to appropriately empower engineers to consider whatever part of the solution space they decide is needed. Companies that “get it” will prioritize “ease of swapping” so that you can constantly improve and leverage autonomy without needless parochial constraints on what can be considered. Thinking, “yeah but just buying it solved our problem today” is such a death knell of weak leadership who cannot fathom strategy or how to leverage real solution ideation from their staff.
I'm especially angry at the dismissal of the vendor lock-in problem in the main article. I've seen quite a few start-ups being trapped by a vendors which were very cheap at first (trial period, starter plans, etc.) and ended up eating much of the profits. Not to mention the innovation cost...
There are personal reasons as well:
5. Innate curiosity and excitement about technology
6a. Get experience
6b. Resume talking point
Many of us picked a career in IT because just like building things out of Legos is fun, building a streamlined full CI/CD pipeline is fun, building a full stack application is fun, etc. Speaking of career, one needs to acquire experience in the new technologies to pad their resume and it's convenient to do it on company time.
I'm not justifying putting your interests ahead of your company's, but understand that's what some of us do. I'd say #5 and #6 are often stronger drivers for decisions than #1-4.