Ask HN: When does it make sense to roll your own X?
Suppose you find a library that kinda does what you need, but not really. Do you adapt it to your needs or do you build a version from scratch that really suits your needs, but might take a bit of time to develop?
151 comments
[ 5.7 ms ] story [ 148 ms ] threadOther things I take into account are how important the X is to the business. My company's product parses a very strange industry-specific file format. There are some commercial parsers and some open-source ones which do most of what I need but I opted to roll my own as working with this file format is very important and I can't risk license concerns.
Also, it might not be very nice but I don't want to help my competitors.
By expending resources to build "X", your company is now in the business of building "X". Is building "X" a business the company wants to be in?
The story behind Amazon building AWS for internal needs, and then selling it externally, is an exemplar of this thought process. They were already in the AWS business... They just decided to package and sell the product to others.
Outside of licensing. It is usually based upon the needs of the project, if I can find a library that does 80% of what I need and I can add the remaining 20% and I am happy to live with that code then I'll definitely use a 3rd party. In general I favor those libraries, but just so many times the licensing trips the flag that it isn't worth the risk and adding a few extra days or even weeks to a project is well worth the time.
One other point, I also am really big on logging/metric collection, if I need to go in and instrument an entire library then it adds another level of work so I start thinking about just creating our own. But not all libraries do need detailed logging and metrics (although IMO most do).
"This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file." from https://www.apache.org/licenses/LICENSE-2.0
I'm not a lawyer and not your lawyer.
Over the years I have sought council and been advised by multiple different attorney's that for a closed source commercial product to avoid certain licenses, like GPL and others. Apache in fairness is not one of them, my bad. MIT and Apache are the two we have used and included in projects in recent memory.
In the end I am not a lawyer either and my core point still is the same, a major question for whether to use a 3rd party library still is around what license the library is released under. And it is usually best to get advice from an attorney for any of the more complex license models.
The entire point of the GPL is to spread the adoption of free as in freedom software. If you aren't willing to share your project with the community you are not allowed to use the community's work.
An example I dealt with a few years ago, I was working on an embedded project and the associated desktop software that the original developer utilized some GPL licensed open source libraries, compiled into the binaries (both embedded and desktop). When I did a license review and discussed it with the attorney it was deemed that GPL licensed components could not be included due to the licensing terms and the way that product was distributed along with the closed nature.
In general I follow the rule, and have been told multiple times, that GPL (any version) with rare exception is not to be used in commercial software (including API's and server based products) unless you are strictly dynamically linking to the library. But if you are compiling it into your commercial product and especially if you are making alterations to it and compiling it in you are in violation of the license terms (and definitely the spirit of the license). Of course, if your product is open source, then that is a totally different story.
Even more open licenses like Apache and MIT still have copyright notices you need to comply with at minimum, so all that has to be taken into consideration as it needs to be added to product manuals, about pages etc otherwise you are in technical violation of the terms. Hence, the license terms of any library should be a key consideration on whether you include it into your project.
People tend to understand open source software as software that is free to use however you want. This is not what GPL software is though. It's entended to be an ecosystem of software that is open and free to use as long as you are also willing to share the software you create using the ecosystem.
I still think that ties into the original question, when does it make sense to roll your own? If it's not part of your core business and doesn't give you an edge, then it never makes sense, unless you're doing it for the fun of it.
You have millions in the project and a dozen PhDs working with you, and you've already authored public papers demonstrating the weaknesses of things previously believed to be strong.
If you are not at least one of those things, it's malpractice for you to even consider letting your employer pay for your crypto "invention"s. That's like a nurse building her own OR and insisting the hospital let her cut people open in it. No. Don't. Stop.
1) Encryption 2) Date/time math 3) Distributed locking/concurrency primitives
All of these are of sufficient complexity to virtually guarantee that you’re going to miss a score of subtle edge cases, and the consequences thereof are likely simultaneously pervasive, silent and catastrophic.
I mean, obviously, someone has to code crypto, or else it wouldn't exist. But, whenever you try talking about the topic online -- whether you're doing it professionally or for self-education -- a probably well-meaning chorus of spam comments warn against doing anything.
I suspect that some crypto-folks have given up rebutting the don't-roll-your-own stuff and just down-vote reflexively.
If you decide to open source it then kudos to you. If you don’t, I won’t think less of you.
There are lots of very good reasons to prefer building something. First, the design of the code will fit your use case more or less perfectly. Unless you get it wrong, in which case you can change it relatively easily. If you use an off the shelf library/framework, you may need to jump through hoops to use it. Those hoops may result in other hoops, which result in other hoops, etc, etc. In the end, you may introduce nearly as much code complexity adapting your code to someone else's design than you save by using their library/framework. (NB: You could just use "convention over configuration" ala Rails, which is another way of saying, "Do everything my way and you'll never have any conflicts" ;-) ).
Second you have control over the code. If you use a library/framework, it may change over time in ways that don't fit your project. You are stuck making the choice of maintaining all of that change or eventually suffering from bitrot. A good example is if your library has another dependency. It replaces that dependency with a new one which isn't supported on your system.
Related to that, you may decide that you need new functionality that the library/framework does not give you. If you were working with your own code, it's easy to add that functionality. If you don't, you need to either maintain a series of patches, or try to get your changes merged upstream. Some projects are easier than others to work with.
When I've decided that I'd rather not roll my own, if possible, I look at the alternatives. The first thing I do is read the code. Can I understand it? Will I be able to fix bugs if I need to? How much of the code is related to what I'm doing and how much is unrelated? How will the structure of the code affect what I'm doing? Are there any controversial dependencies?
Then I look at the community. I look at open issues. I look at how discussions progress. I look at closed issues. Do people get yelled at for asking questions? Are suggestions for improvement valued? I look at open and closed PRs. How easy is it for an outsider to make contributions?
After that, I make an estimate in my head of the maintenance cost of code I wrote myself vs the maintenance code of using one of the alternatives that I've researched. Usually if we are talking about 1 week of work or less rolling your own wins out (there are exceptions, though). If I only need a handful of lines of code, I'll frequently make my own derived library with those lines of code (For example, I do a fair amount of Ruby code and there is useful code in Rails... But I'm not going to grab ActiveSupport just because I want stringify_keys or something similar).
Finally, if a library is important to my code, I'll usually make an adaptor for it. Instead of calling the functions of the library directly, I'll make another library that wraps it. That way if I run into problems with the dependency I can swap it out without much difficulty. Of course, for frameworks that doesn't make sense because part of what you are buying with the framework is the design.
On some teams it is easier to say "the wise developers of leftPad think these are the hoops we need to jump through" than it is to present your own design that has strengths and weaknesses. On others they will walk miles through flaming coals just to avoid the downsides of someone else's designs.
I honestly have no idea which group is right but devs seem to fall into one or the other. I myself started in the first camp but am wandering toward the second. Sounds like you went the other direction.
Libraries also tend to be general purpose so up to half of the code in a library may not be pertinent to your usage.
I wrote an app that used a lot of Oauth and ended up writing my own instead of using the excellent Passport library. It would have been easier to use that, but I gained an understanding of Oauth that I would not otherwise have, and my code is small, understandable and easy for me to maintain and modify. Just my 3 cents worth.
Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do, job scheduling systems that could be replaced by a postgres table, most Kafka applications.
I once worked with a guy who would pattern match whatever problem you had to the trendy Apache or Google library and then spend weeks/months setting it all up. He was easily the least productive person I've ever worked with.
> job scheduling systems that could be replaced by a postgres table
I've never used Postgres in my life ever.
Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#djang...
I have used nowait before to batch scheduled jobs together for a slow API call.
Moving everything to rq, and so far so good.
I wish you the best, but I cannot help but think you will regret that decision.
1. https://youtu.be/SUQYuA9AJT0?t=7546
I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.
Either Mongo, MySQL or List in Redis.
Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)
API is pretty small and comprehensible.
Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.
There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)
However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines
These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road.
I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.
C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.
Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.
#pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required.
> And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.
I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.
You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work.
Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.
GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.
Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself.
But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.
What about null?
Shameless plug: https://github.com/RantyDave/cppbor
JSON Schema – a whole topic unto itself – gives you further control over locking down how this can be represented (e.g. field MUST be present but can be string matching this regex or null).
I agree with your comment. I would add that there are cases where your problem looks really easy right now, but will get increasingly complicated over time in ways you don't expect. In this case, "overkill" 3rd party solutions are actually better. For example, you could write an HTTP client using sockets pretty easily, and it will work in simple cases. But you will, over time, run into edge cases and problems that a mature library has already solved for you.
Also, this problem reminds me of having to decide whether to solve the integral, or look it up, back in the day. Solving from scratch was always preferable if you could do it, but sometimes it was just too hard.
It’s entirely possible that the new problems you encounter will be different than the ones the mature library was designed to address, and then you end up hacking around solutions to problems you don’t have in order to fix the ones you do.
We became the third company I know about to ever solve this problem, but I had built what my colleague called “the hardest to use HTTP library ever”. I was happy to move back to AFNetworking once we could do so without a regression in performance or hitting that bug.
Person tries to find a library that does what one line of code would do (cough left pad cough and others) and then guess what, the library fails at that thing that maybe not 90% of people need but 50% need
And that even applies to some corners of famous apps or libraries big sigh
Invented here syndrome: https://dev.to/mortoray/invented-here-syndrome-4mg8
- Do you need the get results now, or long-term maintenance is more important?
- Are you going to roll your own internal tool or publish it as open source?
- How responsive are the developers of the original library? Did you attempt to ask them for help/consulting/etc?
- Can you wrap the library so that you get 80-90% of the work out of the way? Can you fork it so that you get 50-70% of the work already done? As everyone mentions here, licensing is important for this question.
- Do you have people who have created and maintained libraries before? Do you know what it takes to publish your own library?
For example, I've written hundreds of small libraries and projects in Javascript, so I know the tools, common issues and details of creating a new library. So for me, creating a new one is fairly easy. For someone who has written the same amount of code in their career, but mainly focused on a single codebase, the cost (both short and long-term) of creating a new library is a lot higher.
E.g., you need a web-framework to program your e-commerce site, The web-framework is not a core business need (you are just selling shit online), but incidental. Therefore, don't roll your own.
You need a web-framework to implement a SaaS app, and this SaaS app is your main business (think Canva, or lucid charts etc). Therefore, you should roll your own to suite your SaaS app, and make it fit intimately with your business needs.
in that case, their business need is to have fun! And i would say writing it all from scratch is most fun of all.
- How stable are your needs? Will your ideal setup be the same 6 months from now?
- Is this a core piece of the business or a peripheral concern?
- Which is more valuable: developers that understand the underlying theory, or developers that are familiar with the preexisting library?
- Will the act of rolling your own library make your team more valuable? Will it make them too expensive?
- How will this choice affect developer retention? Your ability to hire new developers?
Are you saying that developers should be kept ignorant and incompetent so that they'll have no choice but to stay at your company writing poor-quality software?
On the other hand, withholding opportunities for growth is also a way to frustrate employees and cause them to leave. Know your people, and take their changing needs into consideration when planning business strategy. There’s lots of strategies that can work here, but not all of them are right for every company. A few off the top of my head:
- Build a training pipeline so you can regularly hire promising young developers for cheap and help them grow into a more senior position elsewhere.
- As your developers improve, be prepared to give raises and promotions to reflect their increasing value.
- Hire experienced developers with the skills you need at a fair price and offer stable employment instead of fast-track growth.
You're not getting that with django, or ror, or ... And no, caching will not get you there, especially not if you take every framework's approach to caching: always set cache headers to never ever cache because it might be dynamic.
The thing is, if you expect to really grow, you're going to have to roll your own, as that's the only way you'll ever get it really fast, really doing what you want.
The frameworks you mention are great prototyping tools but as a community, we’re missing the knowledge of how to take a proven prototype and continue improving the quality rather than bolting on questionable new features.
No, we aren't. We know how to do that, and can do it when we want to. In software dev we rarely want to, because (loaded use of the qualifier “questionable” aside) product teams tend to perceive (not entirely inaccurately, though sometimes the particulars are in error) market demand for additional features. (Also, what physical products are very often optimizing is unit production cost for equivalent products, not quality. But software already has an essentially zero unit cost, so there's essentially no gains to be had optimizing that.)
My gut says always: most of the times turn-key stuff has weaknesses and good libs with perfect apis, well maintained and with a great community are rare but they are there of course.
Hard question since what the gut says is more fun while the alternative is just about gluing libs together. But using libs and once they don't fulfill your needs build yourself is the right but more boring way to go.
I'm saying it's dangerous to have this line of thought that the "correct" way to build a system is with a mish mash of third party libraries and that doing any non-glue coding yourself is only for fun. You're insinuating that the decision OP is making is between efficiency and fun, whereas in reality it's an optimisation problem for efficiency that a lot of people mess up because they don't understand one side of the equation. (and if OP wants to have fun and roll his own that's great but it's another case entirely, I'm assuming he's not asking HN for permission to do that).
I use NPM as an example because for JS devs in the places I visit it's becoming a cultural thing, which IME is not the case in other communities. I don't see as many python devs entertain the thought of maintaining something for years with "crappy libraries" in it for the sake of saving "days" of work. But in JS land that attitude is all over the place and the phrasing of the first bit of your post reminded me of it, so...
For example, I lead a project where we wrote a driver instead of using an open source one. Writing our own manged risk, because the open source ones were unstable or had data integrity issues. Overall, the "cost" of "writing a driver" was about 5-10% of the total effort, once we consider integration, QE, installers, signing...
The thing with 3rd party frameworks is that they aren't built for your requirements. So, if you don't "roll your own X," you need to make sure that the X you choose will meet your business critical requirements early in a proof of concept.
Also, if you are going to "roll your own X," it's worth it to do a few POCs with other Xs. This way you can learn how to make a better design, and what to encapsulate in your X.
(In my case, the 3rd party drivers encapsulated poorly. Most of my effort spent writing our driver was things I'd need to figure out anyway, because the other drivers didn't encapsulate basic details that I didn't care about.)
---
Other times, it makes sense to "roll your own X" to keep things simple. ORMs are a great example. Their leaning curve can be higher than simple queries and boring code. If you only have a few queries, why bother adding something that's just going to get in the way?
Which gets to: Do not use frameworks in place of design patterns.
For work projects, I like to look for the best available FOSS option and then try to help out on that project so I can feel a bit invested in it. This helps calm down my hacker urges to build my own thing from scratch :)
An example might be analytics. It's good to know that you can drop in a third-party library and just go, but you need to make sure to look at how you can move your data off of that platform and use it yourself. You also need to consider how much of a time investment switching off of Y would be. How much effort will rolling your own be? How much effort will rolling your own after using Y? How hard will it be to swap Y out for Z?
Another classic case here is web hosting. I use GCP. I've used AWS. I really don't want to, but if Google or Amazon kicked me off/out/raised the prices too much, I'll host from my own hardware, and go to Fry's if I need to get more hardware than I have. I really don't want to, but I will.
If there is no Z, don't try to find the needle in the npm haystack of poorly maintained libraries with tons of needless sub-dependencies. Unless the problem domain is very beyond your capabilities, just roll your own.
Please don't make another build system. Please.
But in general, if X is not part of your core business, only as a last resort.
99% of the time, even if there is a suboptimal library, as long as its actively maintainted, you're better off extending the functionality, than re-implementing.
Then if it's something that doesn't really matter and the library code is pedestrian (a logging library), I'll use the library.
If it's something that matters a lot and it's expert knowledge and is a large body of work (a complex math/compression/crypto library), I'll use the library.
- you are on your own and don't care about the future maintenance cost of your code
- you want to learn
- the available solutions are objectively crap
Invalid reasons:
- job security
- you're a control freak
- you think all code you write is immediately better than other people's battle tested solutions, aka inflated ego.
- you want to be seen as the maintainer of some prestigious project rather than as the contributor to someone else's prestigious project.
From scratch only if I'm doing something truly unique. New algorithm/protocol. Unpopular language I'm forced to use is missing something I need. Existing solutions too slow or buggy. Maybe a couple times a year I get to make wheels.
NIH is a huge problem at most (all?) software companies. At my current place I spend about 1/2 of my time dicking around with crap internal tools and libraries.
It's rare I write anything that's not glue code. Makes everything boring but I don't get paged often and we have relatively few bugs.