Ask HN: What's a build vs. buy decision that you got wrong?
What are services/products that you built and wished you had bought?
Have you bought something that you had to scrap and build yourself anyways?
Have you bought something that you had to scrap and build yourself anyways?
278 comments
[ 4.7 ms ] story [ 269 ms ] threadThe institutional knowledge around your data never develops because the engineer whose job it is to manage datadog and observability never gets the opportunity to learn or demonstrate technical depth, and so jumps ship to another team or company literally as soon as possible.
SRE recruiters make this worse, making point after point of gaslighting new hires.
I don't believe the same is true today but in the past we did exactly this because we were spending a bunch of time banging our collective heads against some RDS idiosyncrasies and it turned out running our own Postgres was a breeze.
Plenty of things can and should just simply be bought. It's not always so cut and dry, though.
Do we spend two engineers to build a system or buy a system (and spend two engineers integrating and maintaining it)? I've seen this happen more times than I can count.
Those factors make running them well both harder and more expensive than running any ol system.
I wish this wasn't the case, but all the time I spent on Honeycomb and various other OpenTelemetry packages simply can't get it done. Automated APM platforms are simply that much better.
Currently in the later stages of a Datadog project. Which of course means we are working to get the cost down a little. Modern apps product a lot of logs, traces and metrics.
SaaS solutions for monitoring are usually great except for the cost. You can build (I'd advise going with the Grafana stack) but I'd only go down that route if SaaS was way too expensive.
Monitoring always takes a long time to setup whatever you do. Manually setting up every app, team, server just takes ages.
Anything I've seen built that was not contributing business value ended up being a distraction.
I like the control and I like tinkering. But as I've become busier, I realize how much revenue-producing/move-the-ball-forward time I've lost and am still losing by doing all of this myself.
What are you running?
- Tiny Tiny RSS
- Nextcloud
- Docspell
- Gitea
- Youtrack
- Prometheus
- Minio
- Set of wireguard mesh networks for management, monitoring, service access, backups
Almost everything is managed through a single Nix flake with deploy-rs and agenix
All of my stuff is public-facing and needs more reliability than I can provide via home internet, so I have a rented 1U in a datacenter. Proxmoxing my way through about 20 Wordpress sites, mail infrastructure to handle 100k messages a month for a couple of organizations, Gitlab (would have preferred Youtrack!), Nextcloud, FreshRSS, Mautic, Bookstack, Nextcloud, etc. Really want to dive into Minio on another machine for backup storage.
EDIT: See what I did in that last sentence? More tinkering! Argh!
I do recommend managing it via terraform from the get go as it will save you quite some time later and lower the wtfs/minute metric.
In which case, nobody will care how they got their skills, whether it was on the job or not. Just that they are really good at hammering and can prove it.
It was really disheartening to have 2.5 years of hard work dropped like that, but it was absolutely the right choice. They had designed it themselves and it was barely functional. They had a lot of upgrades planned to make it do what they really wanted.
I now think they were desperately trying to pump up their valuation so they could sell, since they did that less than a year after I left.
It's absurdly complex and had a reputation for being annoyingly slow for any user interaction (though reportedly that's improved in the recent half-decade). If your needs are unusual (say, for a B2B2C multi-sided marketplace where dynamics are counterintuitive and your schema constantly changes) it may be difficult to iterate fast enough with Salesforce. But if you're doing any kind of B2B sales, and you want something that everyone knows, that will grow (however painfully) with your organization, it's the default choice.
Also worth noting that they acquired Slack last year for $27B, and overall have a $128B market cap.
[0] What's Salesforce. https://retool.com/blog/salesforce-for-engineers/
https://news.ycombinator.com/item?id=33804293 by Eric Lippert last month
I had the same thing happen at a previous job. My team built and executed and hit every object and was ready to ship and that's when they decided to inform us that they don't want to ship what we built and wants us to integrate our features into a product they had acquired. Like that decision could have been made 6 months or even a year earlier. Instead they procrastinated until the last moment. I decided to leave the company after that. Seeing 2 years of effort and good execution flushed down the toilet was very disheartening.
Use Temporal, StepFunctions, something and try to avoid this urge.
Tears follow when the team either doesn’t account for all the edge cases or doesn’t have the resources to address them.
We built one for a medium sized business without any of that. It's essentially a form system where the users can select the next destination for the form's approval on their own. Users understand the business process and are responsible for implementing it in any other context, turns out, they are capable of managing it in an online forms system as well.
Then all you really need is an auditing system that tracks all the states the document has moved through and displays that to users who are making decisions based on the form and that state. Add "final approval" and "return for revision" and "recall" states and you're pretty much set.
No business specific logic. No need to keep the system configuration in sync with the organizational chart. No need to build "vacation delegation" or "user impersonation" features. You just need to keep the forms up to date with the business use cases, the users will manage everything else on their own.
Our system has been in place for around 6 years now. We do maybe two form updates a year. We have not changed the backend code or system logic since it was deployed. The only other support issues we have to deal with are when the LDAP integration configuration needs to be updated.
That's also a valid criticism. Especially if you're expecting a lot of automated processes to be kicked off once an appropriate approval chain exists. This really is only well suited to businesses where there are limited opportunities for automation. In this particular case, once they recognized that all of the terminal business process steps are mostly manual anyways, they understood the utility of something so simple.. and cheap.
In reality you just end up making a shitty, nerfed version of a programming language, that business users can't understand, because you still have to understand conditional logic to model workflows, oh and your documentation is terrible because devs don't bother with the boring stuff. Most of the time the devs end up implementing the workflows anyway because they don't actually work properly.
If you really need a workflow engine definitely use something off-the-shelf, but I would go so far as to say that in the 95% case, you don't even need a workflow engine: you need a developer who is capable of writing some python scripts. Even if you pay a developer a full salary to do nothing but sit around and make changes to Python scripts on-demand, that's still going to be way cheaper than the complicated workflow engine solution, which will probably require a team (or multiple teams) to maintain.
You can get these "as a service" which might scratch the itch for some.
(Disclaimer: I work for a company that sells Airflow-as-a-service and adjacent consulting)
In reality you just end up [with] a shitty, nerfed version of a programming language, that [...] users can't understand, because you still have to understand [CMake's] logic to [utilize its magic behavior], oh and [the] documentation is terrible because devs don't bother with the boring stuff. Most of the time [you] end up implementing [workarounds] anyway because [CMake doesn't] actually work properly.
Especially gradle.
Workflows on the other hand are more conditional and imperative, mapping better to normal programming languages, with the exceptions of transient error handling, long timers and distributing workload. Here, writing a custom DSL would be a much bigger mismatch.
Best for build configuration would be a hybrid, where the declaration of dependencies is done with normal programming language, which at the end calls a build(dep_tree)-function where all the magic happens. With the risk of developers abusing this setup-step and run half the build in their own reinvented imperative flow here instead... trust me, I've seen this happen even in makefiles that run shell commands outside of targets. This is what Scons tries to be, it seems however not to be very popular compared to cmake.
Conan's recent development is promising and it gives you the full power of Python. It can also be used declaratively but with limitations. If I remember correctly, there are discussions in the C++ world about a declarative exchange format for dependencies and build information, but it's in the early stages. It's not trivial because there are also C++ modules now.
It was a disaster of course. The business people weren't programmers, had no understanding of programming, didn't want to learn programming even greatly simplified with docs and examples, and I ended up translating their queries into code anyway. So essentially I had written a DSL for myself.
Temporal is a workflow engine that doesn't use a DSL or nerfed version of a programming language. It runs your arbitrary code with any of the supported runtimes (currently Go/Java/Node/Python).
I'm able to write, deploy, and maintain workflow code by myself and use their cloud service for persistence and admin UI.
Programmers love to work on it, so there's never a shortage of good implementations, companies need it, so some of them get proper funding and teams, and yet it's really hard so there's no way your DIY solution will have the features of the big ones, and it probably won't have the reliability.
And worse, you'll probably need to learn and support the existing thing anyway, why not just skip to it?
I actually had quite opposite about this particular area. At the time we were to develop particular product for a client company. The product would've greatly benefited from using workflow engine. I did some shopping around, talked to sales reps and have discovered that we would have to shell out at least $350K for our particular case. So I've proposed to the boss that I would quickly build one that would cover basics. The boss has agreed and I built it in about one month. It worked fine for what it was intended for.
After a while we have approached a vendor (the one we would have paid those 350K for their wares). We showed them what we have built, how it was used. They were impressed enough and we became sales and implementation partners. They have routed a gobbles of jobs, training and installations for us to do.
As for original home built engine - over the time we have replaced it with the one from our partner without much troubles.
Win win for everyone involved
The days of Airflow and similar seem like a stone age in comparison.
One of the great things a Temporal workflow can do is can send or wait around for signals from external processes, indefinitely if needed. It’s much easier to start orchestrating things you already have. You don’t really need to buy into it as much as you do with Airflow. If it exceeds retries or timeouts, it can send a signal or launch a process to notify a human that something needs fixed, then someone can intervene, then notify the workflow that it can keep going now. Airflow is much more all-or-nothing success or failure in my experience. Very hard to re-enter the workflow after something got twisted.
Certainly Airflow has more ETL integrations at this point in part due to how much longer it’s been around and the use cases it’s been evangelized for.
I had never worked in place that had much investment in the ETL integrations, we used dockerized processes and just the docker operator, as it was easier to develop and test independant of an airflow instance.
But it seemed it was bad at more flexible ones, like, load data, then process each entry in a certain way, then trigger a new workflow based on each entry (like send an email to every entry on the data fetched based on some condition)
Does Temporal does this?
The other useful pattern is always running workflows that can be used to model lifecycle of various entities. For example you can have an always running workflow per customer which would manage its service subscription and other customer related features.
At a previous job, we had a fair amount of celery tasks and logic around starting them based on user input or on a schedule, retrying on failures and marking progress or cleaning up state in various databases.
Is that a workflow engine?
Open source analogue would be Apache Airflow.
Abstractly, it's some directed acrylic graph (DAG) that is asynchronously computed, sometimes on a schedule.
Unfortunately, most things fall under DAG. But the framework / engine exists to manage the complexity of the ever-extending pipelines declared by the engineers
Event/push-based workflows also fall under this taxonomy.
DAG may too specific. It's really a dependency graph, that likely has a DAG topology.
Yes, but that hasn't meant it was a mistake -- 'buy-then-build' can be a great strategy. Often the 'then-build' never happens, but going into a decision with the mindset readiness for 'then-build', you can learn from existing products, hit their limits and understand what is the custom version of it you'll need in your context. Recent examples are on smaller scale, though - using a library that speeds work up early, hitting its limits, and replacing or extending with DIY that does less things but goes deeper for my use case.
> What are services/products that you built and wished you had bought?
The most annoying recurring version of this has been being just a little too early - building something, then discovering a few months or years later a public product that does the same, but better. At that stage, rebuilding to use has low ROI, and one ends up maintaining a legacy monster. There was a period when public offering of supporting backend infra was maturing, ie things like secrets/configuration management, logging, observability, monitoring, a/b tests, a bit earlier even basic web frameworks (ie building anything on PHP before Laravel came out meant you built your framework first; iirc worse than the frontend framework landscape in 2022).
I wonder at what point companies would finally bite the bullet and swap to buying? Maybe cataloging the feature set of the internal software, seeing what is a necessity, and seeing which vendors can cover those requirements?
When it comes to build versus buy, there are only right answers with hind-sight.
>> Maybe cataloging the feature set of the internal software, seeing what is a necessity, and seeing which vendors can cover those requirements?
Feature set is part of it. Longevity is another. (been relyinb on Google Reader lately?)
Cost us part of it (internal is usually more expensive, but not always.)
Flexibility is another - building up the knowledge and expertise helps morph the system to your needs, not the other way around.
I'd say it's mostly better to buy, you'll likely get much more for much less, but it comes at a cost beyond just money. Beware of things on your critical path, to which you are beholden to outside "partners".
All that said, I think developers err on the build side way more than they should.
Really? I was a DevOps engineer a few years ago and can't really imagine a useful tool in that category so I'd appreciate a few names I could check out. While I've switched to frontend development, I'm still interested in CI/CD and just can't imagine any tooling being able to make it easier then what we had back then too (ephemeral envs using docker-swarm/k8s and pipeline triggers, basically)
So we went the "build-then-buy" route with Netlify. AFAICT in hindsight, what we built was more stable than Netlify, and had the same interface for devs, basically. (… and might have been operationally cheaper.)
(It's all moved to an -and-buy-another-different-thing-for-no-real-reason-other-than-different-people, now.)
It's even possible with good old Terraform, used correctly.
https://www.quali.com/
https://www.withcoherence.com/
https://webapp.io/
https://www.uffizzi.com/
https://www.qovery.com/
https://www.devzero.io/
These exist in the B2B space (and isn't an exhaustive list), there are a ton more in the B2C space
When I first arrived, and knew nothing, they told me we were going to build a replacement. That sounded great to me at the time. Obviously the right thing to do is replace something old and busted.
Well, I and the company learned over time that actually that was not really the best business strategy. It may have been possible, but not with the resources we had available. I ended up doing a whole lot of work that went unused, through no fault of my own.
When both can be used simultaneously you can now gradually move from one to the other, or revert back as needed.
This is what I followed.
My assumptions around how much integration our custom product customiser/editor needed with the rest of the e-commerce platform ware wrong. I thought I need a user system and "saved designs" for the customers, but that's somewhat rarely used, and could have been bolted onto a standard system.
Maintaining and updating it is extra work over what the core business is, there is now a lot of custom code to fix old assumptions and implement features that we didn't previously expect. All of which come as standard with Shopify.
We also believe that customers are increasingly used to seeing the Shopify checkout, it is a reassuringly familiar experience. I suspect it has a measurable effect on dropouts.
If I was to start again now I would 100% just use Shopify, no question. We are considering a large project to move to it. It would be quite satisfying to delete all that code. But it would probably bring new problems, and thing we are used to being able to customise that we will be unable to.
Do I regret doing it? No not really, hindsight is 20:20. A lot of lessons were learnt, but that enabled us to build a successful business.
Additionally, the admin interface is slow and buggy, lacks the necessary information to effectively pick and pack orders at scale, and is unable to handle multiple kinds of inventories (local vs supplier inventory) and lead times.
Furthermore, it takes 47 clicks to ship a product (together with Australia Post), whereas our system can do it with one click. This saves us just under 2K hours per year. Our system also integrates with a robot[0], which can automate the shipping process further.
Our system removes the siloing of data which is a huge problem when it comes to effective ecommerce customer support, and followup service. An example of this, our system tracks deliveries to customers, so we can touch base with them when an order is delivered, or reach out when an order is awaiting collection at Australia Post.
[0] https://raspberry.piaustralia.com.au/pages/the-raspberry-pi-...
Nothing bad ever happened. They're still rocking the same system and the only notable change that went through was adapting to gdpr deletion requests. And they all avoided that okta hack from some time ago.
What's expensive mistake are you talking about?
I'm thinking of firebase specifically. I'm using it for a website I'm building and I've spent very little time on integrating and using it. Quite less than it would take me to write the stuff myself.
> Yes authentication is important to get right, but it's not _that_ complex...
Though I agree in some cases, I think that authentication complexity is rapidly changing and providing a _good_ authentication flow is not straightforward.
Consumer adoption of Passkeys, biometrics on mobile, OIDC/OAuth etc is really starting to take off and that really complicates your login flow quickly.
The eng time to get auth done right (and importantly securely) is not trivial, nor is maintenance. Even companies who's core competency is security get hacked (LastPass just this last week), it is that much harder to worry about when that isn't your core business.
> SSO in large scale business is a case where buy is the right option...
100% agree; any team that I've talked to wants solid, off the shelf SSO to add into their product within a sprint and doesn't want to embark on untangling the SAML/OIDC knot.
> ...teams have to work hard to unentangle themselves from Okta etc.
Agreed, a huge complaint that I hear all the time. Okta/Auth0 have decided to take the interesting road of increasing cost per user as you scale rather than offering volume discounts.
Whenever you're considering SaaS, it is critical that you look at cost per user over time and make sure your contract scales with you instead of explodes when you cross a threshold.
1) Better abstractions for disentangling authorization
2) Better technical literature on the subject [1][2]
3) Increasing comfort with third-party infra services (RDS, LaunchDarkly, etc.)
Note: I'm cofounder of an authorization-as-a-service company (Oso) [3]
[1] https://www.osohq.com/academy
[2] https://research.google/pubs/pub48190/
[3] https://www.osohq.com/
Our company (https://aembit.io/) solves auth problems (specifically identity and authentication between workloads).
I have been doing security and auth for the last 20 years in different shape and form. It's a minefield. Grabbing and using some SDK for auth is simple. Making sure that you account for the whole lifecycle (identity, authentication, authorization, secrets management, secrets rotation, addressing vulnerabilities as they pop up) is incredibly complex.
Soon afterwards keycloak came on the scene and negated a lot what we had done.
Installing and configuring it was relatively easy. Keeping it up to date and secure is a different thing
Otherwise, if you are going to be selling into the enterprise and the majority of your users are paid, this is an area where using a SaaS tool is a no-brainer. Your sales team is going to run into customers that need XYZ-compliant auth and it's solved out of the box with the SaaS and cost isn't an issue since it'll just be baked into the per-user pricing.
Auth is pretty easy to implement, but difficult to get and keep right. Then there are the nooks and crannies that crop up and appear and get discovered that you have to be aware of and keep up with. I am of course biased, but it seems to me that paying a company to keep up with the rapidly changing environment is much more efficient than trying to do it yourself.
And with WebAuthn and Passkeys -- you can implement that yourself without too much trouble. It's not trivial but not impossible, but the same argument applies -- nooks, crannies, corner cases, risks, etc.
“We can only be great at one thing. The rest we can only be good at.”
This doesn’t quite answer the question, but I think it’s related.
A few years later, our internal version of this software was crappy and our data was still not something that we could monetize. In the future, I'd refrain from being too cautious at the beginning when you are facing bigger risks that may not be sexy, but are very dangerous.
I remember having a hard time to convince our CEO to use BitBucket in 2011 because our precious code (that no one would actually want or could actually understand.....) was going to be "in the cloud".
I bet there are many firms that gave into the fear and regret it now.
Just like with coding, it's best to start small so you get better at the core skills. Building a chess board as a first project is like learning Javascript then trying to build a github clone, you're just setting yourself up for failure.
TLDR: I wanted to travel the US and explore national parks in a camper van. I'm handy as a builder and knew that I could, technically, build everything. Turns out that the challenge wasn't in the technical aspect but rather the sheer volume of work involved.
A little more detail: Having grown up around 4x4 vehicles, I wanted something four wheel drive. No full-sized American made vans come from the factory as four wheel drive. Even if you find a new 4x4 van on the lot, it will be an after market conversion. This means that they're harder to find and, thus, cost a bit more than your typical 4x4 truck. But I was determined. I found a 1987 Ford Econoline in rough shape for around a grand. I bought it, named it "Polar Bear," and set to work on it.
One of the biggest setbacks for the project was the ongoing expense and hassle of repairing an old van with a custom conversion. I learned more about automotive mechanics with this vehicle than anything else I've owned. Still, a lot of repairs were well beyond my scope and I ended up spending tens of thousands of dollars rebuilding various components. Repairs were a near constant problem and this drastically slowed my build process.
Another hitch in my plan was fuel economy. Polarbear would take me a mere 9.5 miles on a gallon of gas. With a 3-speed transmission, the engine was always running at a high RPM. The alternative factory transmission with an overdrive wasn't as strong as the 3-speed and the gas mileage was only nominally better. After extensive research, I learned that a manual 5-speed swap could be done and would increase my MPG to over 20. However, with all of the expense and hassle of installing a clutch, I never took on this endeavor. Ultimately, I never did that national parks tour. I did, however, go a whole heckuvalotta cool places.
The camper conversion, which was supposed to be the focus, took me a solid ten years to finally complete. There are a lot of rather sloppy looking camper vans in the world and I really wanted mine to look good. This meant taking time on the details. And oh my, where there details. Unlike a box truck, a cargo van's walls are all continuously curved. Cutting wood to smoothly fit the curves of the walls is tricky and takes time to figure out. I also learned a ton about automotive and RV electrical systems in the process. When I began, I imagined a very complex system. I quickly learned that the more simple the design, the better the design. Everything has the potential to break and the more complex it is, the more likely it is to break. In the end, I decided to entirely skip water pumps and simply went with a gravity based system. And despite having a propane tank mounted to the van, I opted to use a portable camping stove instead of running more propane lines. In my opinion, these were good decisions.
Over the course of ten years, I spent enough money to have bought a nice completed rig right from the start. At the very least, I could have purchased a completed 2wd camper and had it converted to 4x4 for far less money. This would have also given me a more modern 4x4 drive train and suspension.
Still, I have no regrets. What I learned was priceless and the adventures I had along the way are some of my best memories in life. I finally sold the van last year for almost 10x what I paid for it - but far far less than I had invested in it. The following link is the build thread I posted to the Sportsmobile Forums for anyone who may be interested in seeing what she looked like:
https://www.sportsmobileforum.com/forums/f24/polar-bear-1-a-... (the last page has the final pictures)
Vehicle builds rarely, if ever do. Unless you absolutely, positively, must have an electrical motor in a Javelin or something.
If you're renovating because you want to continue to live in and enjoy the house, it's more a personal question of whether spending the money will increase your happiness.
I really want to try the van thing but there are a few hard factors preventing it for me. So, I live vicariously through such accounts from time to time.
[1] https://gocampcampervans.com/
But, I think it's highly possible your suggestion could help somebody else reading this and in any event, I definitely appreciated your effort. Thank you.
It certainly cost me a lot more than just doing it myself, but by doing it this way, I achieved two goals: immediate use for camping and the ability to figure out what I would do differently by living in it first.
There are a few things I learned pretty quickly:
1. Get rid of the "fancy" toilet. Just use a bucket with eco bags that can be buried or tossed into a trash can. No cleanup mess, lighter, and frees up a lot of valuable space.
2. Moved the sink from the door side and put it on the other side of the van next to the stove (there was an uncomfortable seat there before). Again, freed up a lot of space and made better use of it.
3. Removed a lot of hidden unnecessary support wood. Like in the backs of cabinets against the walls. People don't realize how heavy that stuff is and every extra pound affects van handling and gas mileage.
The list goes on and on, but one thing I did learn after taking my van into cold climates... insulation is critical and very hard to add after the fact.
I'd build one or two, test them for several months while making adjustments and fixing various weaknesses, then I'd build ten, twenty, however many were needed.
One particular client decided they wanted to skip all that and just spend the money on Dell. They didn't have many servers, certainly not enough to justify a separate server room. Their offices weren't air conditioned at night, and they had an entire summer where one server or another would become unresponsive over the course of a weekend, and sometimes at night between weekdays. Accessing iDRAC was beyond what they wanted to do, so of course I had to do that.
They had Dell support, but Dell had no "fix" for unstable machines other than to tell them to build a server room. Mind you - the ambient temperatures were always below 100º - any reasonable person would say that while that's not ideal for servers, "premium" servers should still be able to handle warm rooms, particularly when they're idle, and not crash or lock up.
After that fiasco, they gradually replaced the Dells, one at a time, with machines I built. I wish they had tried harder to return the Dells, but they just wrote them off.
I've learned that any savings in time and money (mostly - the Dells cost more than the machines I build, even accounting for extra billable time) aren't worth the loss in time, productivity and reliability in the long run. Of course, the opposite would be true if I just wanted more billable work, but I can't do that, unlike many others in the field.
BTW - when I build a new generation of servers, I do months of testing, but the same general platform can last a good five years, like Ryzen 1000 through 5000 systems with ECC have lasted since 2017.
Do you have a favored brand for server motherboards?
Somewhat related:
I don't like server motherboards like those from, say, Supermicro, because they don't care very much about security. For example, they've had motherboards for years that let you share the primary ethernet with IPMI. Is there a way to disable this? No. Is there a way to force IPMI to ONLY use the dedicated ethernet? No - the settings are stored in the battery backed memory. Wouldn't a jumper make sense? Sure, but they don't think so.
What this means is that if your motherboard battery dies and your dedicated IPMI port is somehow disconnected, your server can be completely, 100% taken over via the public interface. While it's not a likely attack scenario, the point is that if you have a need to ship servers to various datacenters, you genuinely can't be sure that your server won't be utterly vulnerable on first boot unless you're there, confirming it yourself.
Does Supermicro care? No. They refuse to consider the loss of all security due to a dead battery to be a security issue. Having a security issue is one thing - ignoring it when told about it (or worse - pretending it's not an issue) is something else entirely.
Then one day these products eventually go EOL and the company is often stuck maintaining a zombie product. Or the products undergo a major refactor that breaks their customizations and integrations, and they end up stranded forever on version X.
I hear ERP systems often go like this too.
The only thing worse to me is entire huge products built in stored procedures. I know one product that was written in about 2 million lines of PL/SQL. It did some amazing things, but we were locked into PL/SQL for all time, and the Oracle scaling bills and HW were astronomical…
This happens with any product you buy. Dealing with it has to be included in the value proposition.
> we were locked into PL/SQL for all time, and the Oracle scaling bills and HW were astronomical
Yeah Oracle is expensive. Scaling costs should have been part of the calculation to commit to it. For most orgs that use Oracle, the cost is not really an issue in the grand scheme of what they spend on IT and servivces.
Open Verse Media
When I first started, in 2016, at Open Verse Media, an ebook publisher, they asked me to look at their content management system (CMS/CRM). The staff had to rely on it, but it was very slow. The COO, whom I'll call Robin, had overseen the creation of this app. The actual work of creating the software was outsourced to one firm, but after two years Robin felt they were too expensive. She fired that first firm and then hired a firm in Ohio, which I'll call MegaStars.
The app had been built using a popular software framework called Ruby On Rails. Whenever Robin felt that a new feature was needed, she would ask MegaStars to add the feature. MegaStars billed $500 an hour, and over the course of seven years, a total of $3 million had been spent on the creation of this app.
The staff hated the app. When the head of marketing wanted to bring up the top 100 best-selling books, they would click on a link, and it would take a full 60 seconds for the page to come up. The staff had gotten used to the fact that they always needed to be engaged in two tasks, that is, something to keep them busy while they waited for the pages in the CMS to render. An advanced search, with multiple filters, could take up to five minutes to render a report. Many of the lower-level staff would simply go into Slack and engage in gossip with their peers while waiting for each page to slowly appear.
So on my first day I logged into our main web server, and right away I could see that the app was generating several thousand errors each hour, all of which were being written to a log file. Since this app was single-threaded, the work of writing the errors to the log file had to happen while each page was rendering. This was one reason why it was so slow.
This arms-length relationship needed to be closer.
Why did this app have so many problems? Well, when Robin requested a new feature, MegaStars would tell her exactly how much time was needed to get that feature done. If they felt a new feature needed 30 hours to build, they would simply quote $15,000 as the price tag. Sometimes the new work conflicted with old work and generated new problems, but that wasn't in the estimate and therefore the new problems needed to be ignored as much as possible. This tactic of ignoring new problems had been going on for many years. Additionally, much of the code base was now out of date and suffered version conflicts whenever some parts of the system needed to use newer libraries of code (which in Ruby On Rails are called “gems”).
MegaStars could have said, "Pay us $100,000 and we will clean up all of these problems." But then Robin might ask, "Why did you allow these problems to exist? What are we paying you for?" It might seem like a scam, if MegaStars asked for more money to fix the problems that they themselves had created.
Here was the central dynamic of the situation: Robin felt she held power because she could terminate the relationship at any time. In fact, all of the problems in the relationship were because she could end the relationship at any time and was leaning on that fact as her main way of getting compliance. MegaStars was unwilling to commit to the long-term health of the software while Robin was constantly threatening to fire them.
When you work with an outside agency, they typically can't or won't go back and clean up the code, because the customer is not willing to pay $500 an hour for that work. Some of the better agencies try to include the clean-up work in the overall price, but then those agencies seem expensive — and they get undercut by other agencies that are willing to do the absolute minimum, even if ...
(I don't think it was answering the question, which I took to be about build-it-yourself vs. buy-something-prebuilt, not outsource-to-someone-else, but I enjoyed it nonetheless.)
I actually wrote about this experience and the ultimate refactor. [0]
[0] https://koptional.com/article/nuanced-strategy-build-vs-buy
Hooking in staff rosters, asset availability, appointment reminders, holidays and the multitude of other quirks is daunting.
IMO your product is the only thing you build, as much as possible should buy everything else.
There is eventually an inflection point where your product is so mature that the opportunity cost of improving it operationally vs marginal next feature you will eventually save money, but that scale has been growing YoY for a while (AWS mostly gets cheaper every generation so far) .
Spent hours messing around with the physical rig and software settings, but the balance was never quite right.
I'd imagine that the kits today are better all-around. At the time it was pretty cutting edge and the idea of being able to do steadicam shots with my DSLR but on a budget I could afford was too good to be true.