Indeed, one could even automate the process, some sort of tool that automatically copies the dependency’s code, kinda like an install. I think someone already wrote such tool.
Copying someone else's code and putting it in your software repository is an excellent way to find yourself violating a wide variety of open source licenses.
This is actually one of the most common causes for GPL violations. And companies have gotten into serious trouble for that.
Either rewrite the code, or maintain it as a dependency and follow the licensing rules. Don't simply copy code from it into your project unless you have explicit reason to believe that that is OK to do.
I'm not sure if I agree - commonly-used packages will have known and documented faults, while validating something you (your organization) created can be challenging.
I recall using an unusual library a while ago where the author had embedded their own log4j-style implementation. I couldn't configure that the same way I configured other logging.
Hardly a problem today since I just write to `stderr` and have the container log aggregator push elsewhere but boy was it annoying.
Avoiding dependencies is a noble goal, and something to be valued, but this simple rule is too simplistic.
The problem lies in the fact that there are a great many things I can hack together in an afternoon to "replace" some kind of external dependency, but the quality discrepancy of these hacks is highly variant. My understanding of what can or should be done in an afternoon might differ with my colleagues'.
Unfortunately, like all things in engineering, you have to carefully reason about the pros/cons, requirements, and costs. After that analysis, you can make a judgment on depend vs. build (also, buy vs. build).
Agreed. For libs that are "afternoon-y" in their scope (so, not an HTTP server or crypto), if you need to get off the fence you can use some cheap heuristics to assess the quality of a library without auditing its code. For instance, you can look at its popularity (in downloads or Github stars), its release/version history, its number of open issues, and its development activity. If I see high issue counts and many major releases with breaking changes, I'm going to avoid it. If I see 2+ years of stability with mostly minor releases, low issue counts, and high use rates, I figure it's going to probably be better than whatever half-baked solution I could scribble in an afternoon.
I wouldn't consider a high number of open issues a problem on its own. All big popular projects with a history have a high number of open issues. There are some exceptions, who may be closing isses aggressvely, but it is more about a style of managing of those issues, not about project health.
Over time an issue tracker inevitably becomes a collection of hard-to-reproduce bugs, incomplete patches, underspecified feature requests, random tracebacks, etc. Maintainers can choose to just close everything which is not actionable immediately, or be in comfort with such issues, and let them live in the bug tracker. I personally like a style when an issue is closed only if it is fixed, or if it doesn't contain useful information, or if it is a duplicate.
A better indicator is activity and responsiveness of the maintainers in the issue tracker.
I don't really worry about something I could write in an afternoon.
I can look at the code, get a good grasp of it (hopefully), judge the quality, docs, prospects of getting updates/needing updates/being able to update it myself, pretty comfortably. In other words, the risk evaluation is incredibly straight forward.
Additionally, the risk itself is fairly low. If it goes out of date or stops working or just turns out to suck, the most I risked is an afternoon of work. Leftpad was a debacle due to it's scale, but fixing Leftpad was pretty easy (I'm not recommending importing one liners as dependencies mind you)
-
But when it comes to stuff that isn't small, it's usually also the kind of stuff that holds the most insane amounts of risk for a project and is the hardest to evaluate.
Stuff like application frameworks, threading frameworks, massive networking libraries, etc.
The interface is _huge_. To the point that even when you try and wrap their complexity in nice packages with separation of concern and encapsulation they leak out into the rest of your code and end up being a nightmare to ever change.
Instead of spending an afternoon writing dependencies like this, spend that time investigating your "too-big-to-fail" dependencies. Try and keep a finger on their pulse, because they're the ones that will really come back to bite you if things go south.
> Additionally, the risk itself is fairly low. If it goes out of date or stops working or just turns out to suck, the most I risked is an afternoon of work.
Sometimes, the opportunity cost (time spent) is the largest term in the risk equation, but often there are other terms that might be orders of magnitude larger. For example, the risk of depending on the wrong abstraction, or becoming coupled to a hack.
What you're saying makes sense. My only point is that there's a lot more subtle judgment required in these decisions than often meets the eye.
A simple example would be an HTTP client. It’s easy to write a naive thing that makes GET requests with no request body, TLS, connection pooling, etc. Why should I use a dependency when I can write it in an afternoon? Well, I used to think that before I tried writing one :) The first draft was easy. Adding features got messy.
I had the opposite experience. All I needed was a way to do a simple GET. That's it (and that's all it still is, by the way). Instead of spending half an hour writing the code, I decided to use libcurl---that's what it's for, right?
Until I found it wasn't installed on some of our test machines (it was needed for testing, not for production and for reasons beyond my pay grade, libcurl was not available on the machines). Then I thought, well, I could include the libcurl into our vendor repo. It worked, but it was a nightmare to use. It took way too long to figure out the proper "configure" options to use for what systems, it nearly tripled the time to build it on the build servers, and even then, it was hit-or-miss.
After several years of this misery, I removed libcurl, and wrote what I should have years earlier. Using libcurl as a dependency did NOT save us any time.
> The problem lies in the fact that there are a great many things I can hack together in an afternoon to "replace" some kind of external dependency, but the quality discrepancy of these hacks is highly variant.
Perhaps it's a domain-specific thing, but when someone uses the words "hack together" I imagine it means using dependencies without really understanding what's going on in them, precisely to avoid figuring out how to code a solution properly.
Writing it yourself obviously needs to also imply doing it correctly. Even if that means you must learn a bit about what is the right way to do it (a side benefit, though usually viewed as a downside).
This is horrible advice. There's a reason that you don't write your own hashtable implementations.
Yes, I can write a hashtable implementation in an afternoon, but it's going to have bugs that I'll spend the next year fixing, and still not achieve the performance of the pre-built version.
All that work of finding existing solutions and learning how to use them? That's part of the job.
Find a bug in the dependency? Submit a patch.
Worried about the dependency changing? Lock the version.
Too many external repos to retrieve those dependencies? Use a local cache.
> I can write a hashtable implementation in an afternoon, but it's going to have bugs
If it has any bugs that would surface in a year of production (while the dependency version wouldn't) then you didn't write an equivalent in an afternoon.
The advice, if it's to be useful at all, must be things that you could completely replace in the same quality, in an afternoon.
I'd extend "afternoon" to "half a week", but in general I agree with OP.
> Yes, I can write a hashtable implementation in an afternoon, but it's going to have bugs that I'll spend the next year fixing, and still not achieve the performance of the pre-built version.
The meaning of "afternoon work" should be considered "of good enough quality". Tests, structure, reasonable docs, all that. It shouldn't be a fastest written something, it should be a normal code.
> and still not achieve the performance of the pre-built version.
Some losses in performance are acceptable for greater visibility and better fit for the project. If you need non-trivial performance gains - well, those are also achievable by code, are you sure you can actually write such code in a few days?
> Find a bug in the dependency? Submit a patch.
That's the point. To submit a good patch, you have to internalize the system. It's easier to do if the system is yours - doesn't do much except what you need.
> Worried about the dependency changing? Lock the version.
Now you've locked yourself out of upstream bug fixes.
We do reinvent the wheel whenever we need to have an actual wheel for a device, not an abstract concept. Similarly, we write for loops, "reinventing" them for our specific purpose. Those are all different wheels, loops and needs. Don't mistake the "idea" of a hashtable with an implementation.
> Worried about the dependency changing? Lock the version.
And get p0wned a year later when some security researcher finds a vulnerability in code that you don't even use, but pulled in as part of that dependency.
> This is horrible advice. There's a reason that you don't write your own hashtable implementations.
Of course you do and release(d) them as open source (public domain). Take Java - it has decent a HashMap but it's node based. It's memory inefficient to a point its nodes and arrays are top 3 of memory consumption. An array based hashtable takes around 3.6 times less memory for larger ones (on 4bytes compressed pointers) and over 10 times less for smaller ones. Perf. wise it's on par or better as well (nowadays architecture is heavily driven by locality and access patterns)
Also you make your code so it can switch between both on the fly, if need be.
> Of course you do and release(d) them as open source (public domain).
How ironic though. Of course it did work a few times, but if the advice is to not use dependencies, then the better advice would be to not use dependencies that were written in an afternoon to avoid using some other dependency :)
Indeed! Although I spent like a weekend to do it (the inital release was 512 loc). It passed all standard jdk/jsr-166 Map tests[0] and then some more, incl. perf., memory consumption, garbage collection harness. Tests are also public domain.
Also the release is not available as dependency, so the interested user would have to clone the repository on their right own.
The part with afternoon deps would be that all their code can be read and cloned, if need be. Free to pick the few functions needed - I'd assume around 200-400 loc top.
Agree-ish - buttt makes me think: How about... dont keep a dependency you could replace with an afternoon of programming?
Factor, re-factor, and (most especially) DELETE should be tools in the toolbox -- but see if you need it/keep it (e.g. protoype it in, etc first) before you re-write.
Lines of code is a passable metric for the quality of work if they are considered "spent", not "created". That is, the programmer's work is better, all else being equal, if it takes fewer lines of code. Who said so? Knuth, Wirth, Dijkstra, Perlis?..
A hundred afternoons later my small application is finally completed; now I must maintain update and document it all for ever rather than relying on third party components.
What I really wish is people would look closer to home; for example use some of the thousands of functions that ship with your operating system before downloading a package.
This reminds me of the Node left-pad module problem in a way. I think if something is so trivial to write, you should write it rather than using a dependency.
If it is non-trivial, I prefer the official standard libraries for a programming language. That is if a solution exists in the standard library.
I think the Go standard library with its batteries included mantra and the level of support it gets is good example of a library that should be used when a solution exists within it or by utilizing it.
If you copy the code into your project you at least need to keep track of the original authors and licensing or you're in violation of the copyright 99% of the time.
The single best way to avoid dependencies is to use a language with a large standard library.
Given the variance in standard library coverage, it’s rarely productive to argue about this topic in a language agnostic way. Using only stdlib in Go is very different from using only stdlib in JavaScript.
The best way to avoid dependencies is to use a language that is built in a way such that dependencies are worthless. Like APL, J, or kdb+/q. All of these languages are incredibly small, have almost non-existent standard libraries, and yet are designed in such a way that a large standard library becomes superfluous.
Having a large standard library speaks poorly of the composability and orthogonality of language primitives.
Because kdb+/q has such few datatypes and you can't define anymore, it's very easy:
q) .j.k"[2.2, 3.5]"
2.2 3.5
Serialization is also trivial:
q) .j.j 0 1 2 3
"[0,1,2,3]"
In general, sending stuff across memory boundaries (files, network, ram, etc) is exceedingly trivial in kdb+/q. To execute a function on a remote server, simply connect to a server and send across the call to the handle. For example, to send synchronously compute 1+1:
q) h:hopen `::6666
q) h(+;1;1)
2
You can send over anything you want, even the entire source code of program to be executed! This is a really flexible environment, where you can create really powerful app-engines. All members of a cluster can send code, data, and messages to any other node, async or sync.
Much like Common Lisp and SmallTalk, you can easily connect to production nodes and modify code while the service is running.
It's rare to find such a dynamic, flexible, and interpreted language that also has world class performance, even often beating hand written C. Combined with an integrated database, you get a distributed system that can't be beat, at least performance-wise. And the craziest thing is that all of this sits in at 650kb executable with libc has the only dependency. And all probably in less lines of code than a simple javascript webapp!
i'm so used to python's stdlib that whenever i go to javascript i get legit angry.
i really like writing typescript code, but holy shit, when you have to pull libraries to even the smallest thing (lol left-pad), it get super infuriating.
JavaScript standard library was maybe bad in the past but nowadays they added a lot of features (yes, even left-pad).
Sure, if you know that you have to support old and broken browsers you have to use these dependencies to ensure correct support, but if you know that your code will run on a specific interpreter (for example a modern version of nodejs, or modern browsers) you don't have to worry about it too much.
Also most JavaScript programmers tends to abuse dependencies, I mean even for things that are really 10 lines of code that you write in 1 minute.
Is that really 5 minutes? (For when left-pad was relevant)
var cache = [
'',
' ',
' ',
' ',
' ',
' ',
' ',
' ',
' ',
' '
];
function leftPad (str, len, ch) {
// convert `str` to a `string`
str = str + '';
// `len` is the `pad`'s length now
len = len - str.length;
// doesn't need to pad
if (len <= 0) return str;
// `ch` defaults to `' '`
if (!ch && ch !== 0) ch = ' ';
// convert `ch` to a `string` cuz it could be a number
ch = ch + '';
// cache common use cases
if (ch === ' ' && len < 10) return cache[len] + str;
// `pad` starts with an empty string
var pad = '';
// loop
while (true) {
// add `ch` to `pad` if `len` is odd
if (len & 1) pad += ch;
// divide `len` by 2, ditch the remainder
len >>= 1;
// "double" the `ch` so this operation count grows logarithmically on `len`
// each time `ch` is "doubled", the `len` would need to be "doubled" too
// similar to finding a value in binary search tree, hence O(log(n))
if (len) ch += ch;
// `len` is 0, exit the loop
else break;
}
// pad `str`!
return pad + str;
}
One of the points of the article is that when you code a dependency for your purpose, you get smaller code. For example, I know I'll always pass it a string, so I don't need to type-check.
I started with the problem of "I need a function that pads a string with a character out to some length". Coding it up took under 1 minute, easily under 5 minutes.
function leftPad (str, len, ch) {
const neededPadding = len - str.length;
if (neededPadding <= 0) {
return str;
}
return ch.repeat(neededPadding) + str;
}
It took me longer to write this comment.
The fact that the left-pad code has optimizations (which don't matter for the place I'm using it) and type-checks (which don't matter; my higher level unit tests would catch that mistake) is beside the point.
More generic is better. If I go into a codebase that uses standard libraries. Even if I don’t know how to do something about it someone on the Internet does. Your custom framework - not so much.
I don’t have to care about how the underlying libraries work. I can treat them as a black box.
> I don’t have to care about how the underlying libraries work. I can treat them as a black box.
When I wrote haskell, the majority of the libraries did just work, and I didn't have to dig into their code to find bugs often.
When I wrote javascript, hundreds of the libraries I used did not just work. I usually had to care very much about their details because they were poorly implemented, full of bugs and incorrect abstractions, and often abandoned soon after.
I agree that there's benefits in reusing some well-socialized and well-implemented generic frameworks and abstractions.
It's not worth using generic abstractions that are not well understood, buggy, and don't match your needs closely. In that case, write your own.
More generic is not always better. Above, I'm arguing that it's important for code to be easier to reason about. If a generic abstraction helps with that, cool, but it's not always going to be the case.
That’s also why I stay away from the clusterf%%% of front end development and JS if possible except for simple AWS Lambda scripts that have one dependency - AWS SDK.
Any other scripting I do with Python. Any more complicated development it’s using a language with an ecosystem with adults - C# or Go.
Almost everything I work with has bugs, so chances are I'm going to run into one. It's a lot easier for me to fix bugs when there are fewer layers and more of them are written by me. Of course, I can't write all the layers, but if they run on my service, I have to be prepared to fix them, or suffer from them being broken until a benevolent force fixes them for me. (Sometimes that happens, but usually not for the harder problems)
As long as it works and is well documented, yes. But the moment there’s a bug somewhere or the docs aren’t adequate for your needs, you are in much worse trouble.
Again my general rule - you (generic you) are no special snowflake. What are the chances that you come across a bug in a library that has had 2,176,677 (in the case of Dapper) or Entity Framework (supported by MS) that no one else has come across, found a workaround, and posted the answer somewhere on the Internet compared to your code where you didn’t think about a corner case?
I can count on one hand[0] the number of JS dependencies I've used in enterprise/large projects for extended periods of time where I've never needed to manually debug the library or read the source code. For enterprise software, sometimes the easiest solution is to directly hotpatch a vendored version of the dependency.
This is especially true with dependencies where bugs are fixed in major versions, but where upgrading and dealing with breaking changes would require significant code refactoring.
To drive the point home, I've been bitten by bugs in NPM itself.[1] Fixing that required reading through the source and manually swapping out one NPM's internal dependencies to a newer version.
And it doesn't matter if someone somewhere has had the same problem and posted it on the Internet unless I can find their answer online faster than I can fix the problem myself in my own library. Often this is not the case, filtering through issue trackers and trying to find the one blog post or comment that tells me how to solve the problem can be a big time sink.
[0]: Okay, maybe 2. But the point stands, it's not a rare or exceptional occurance.
I've had some serious train wrecks because of shit NPM libraries over the years.
Probably the worst was with a decently popular library someone had brought in, that tried to do a refactor from callbacks to async/await, without understanding at all how async/await worked. They'd leaked an async operation in the library code, so 'await'ing a specific function call in their API that returned a promise, didn't actually await everything the call was doing, ending up in a debugging nightmare. Of course their perfectly manicured suite of 8000 tests with 110% test coverage didn't catch it either, because the number of people who can write good quality tests is shockingly low, and library-writers aren't somehow magically ahead of the pack in that regard.
JS really feels like PHP did back when I was a newbie learning that shit. In other ecosystems, the 95th percentile devs seem to write all the libraries, so everyone comes here and posts repeatedly about how great dependencies are. In JS, it's the average dev writing all the libraries, and the average dev's code is enough to make my brain bleed.
I'm a big proponent of different advice for different ecosystems. If you're doing front-end JS, the pendulum has swung so far to one side that 'NIH syndrome' is treated like it's going to lead to the fourth reich, which makes 'chill out a bit on dependencies' pretty good advice if you're looking to get a leg up in the industry. But I'm sure there's other ecosystems where the same advice will just leave you with a tangled mess while your competitors leapfrog you in productivity with a good 3rd party dependency.
I'd say take any advice in threads like these with a grain of salt unless it's given in a bit of a narrower context. Taking some one liner about software engineering in general and applying it to your specific project is probably just a coin flip as to whether it's going to improve your code or not.
This is actually a great example of why you SHOULDN'T use left pad.
I've not profiled it, but I'm going to guess that now-a-days this will be faster than the current implementation on npm.
function leftPad (str, len, ch) {
str += '';
len = len - str.length;
if (len <= 0) return str;
if (!ch && ch !== 0) ch = ' ';
ch += '';
return ch.repeat(len) + str;
}
Why? because the VM is (very likely) going to do exactly what the cache would have done. It can replace `ch.repeat(len) + str;` with a presized string allocation and a memcpy of ch + str characters.
> This code checks for ch !== 0, yet the Number type in JS has no repeat method.
> Static typing would actually fix this kind of coding error.
That's not a coding error, that's addressing an oddity of JS coercion rules that a less experienced developer could easily have missed.
> if (!ch && ch !== 0) ch = ' '
That code says that if `ch` is falsey and not equal to 0, then set it to a space. The only arguable falsey value that should be excluded here is a literal `false`, but that's not a single character and is fairly ambiguous either way. I'd certainly fall on the side that a literal false should not be converted to `'false'` here.
> ch += '';
The next line converts to to a string by adding it to the empty string.
> return ch.repeat(len) + str;
So by the time it gets to this line we know ch is a string.
Static typing is great, but the bug you claim is there is not actually there.
What magic would enable that? I don't think the VM is likely to cache a string of 6 spaces, it has no way of knowing that would be a common parameter, and no heuristic to determine that it's likely.
It may specialize or inline, but that's a separate matter.
JIT is the magic sauce and this a pretty regular optimization.
Step 1, inline repeat.
Step 2, remove the intermediate array allocation
Step 3, allocated a string array sized for the pad + str
Step 4, Use one of the many CPU instructions to repeatably copy the padding character and then the `str` into the same array of characters.
None of these optimization would be out of the question for the Jit (and I'd expect them). You don't need the cache at all, it's just a waste. The only thing it saves it creating the intermediate string which is HIGHLY likely to be optimized away with the simple code.
Yeah benchmarks can be much more difficult to get right than it might seem at first glance. Good thing they didn't try to write their own benchmarking code, otherwise they might have fallen into those traps you just mentioned.
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency[1]. The author of said library works on V8, and already considered all those problems and much, much, more[2].
Okay, you win. I'm not going to read the entire source of that package just to make a point. Though I do find it strange that the author of that library would write an entire blog post on the topic and then not take his own advice in the implementation of the library he wrote.
Yes, particularly when you know the types that will be passed in. True that typechecking in JS can be a little timeconsuming, so it might take 10-15 for me to write a general purpose left pad.
Also I seem to remember that someone benchmarked the cached version and found it to be slower than the naive approach anyway. I could be mistaken there.
This is so hilariously overengineered: not failing when passed the wrong types, arbitrarily caching padding of len < 10, etc
The most egregious is the bad big O analysis for a pointless binary search. The loop does indeed run O(log(n)) times but `ch += ch` still takes O(ch.length) which is growing exponentially. It ends up being a complicated way of still taking O(n) time while creating a lot of intermediate strings.
It isn't any faster than just creating the padding with a loop or `new Array(len).fill(ch).join('')` or `ch.repeat(len)`
It's not overengineered if thousands of downstream projects are relying on it, some of which might see significant benefits from those performance optimizations.
Too bad that's a flawed benchmarking methodolgy. JITs are notoriously hard to correctly profile and the benchmark lib isn't even sort of doing the right thing.
For example, it's missing warmup. The results aren't being consumed in a way that wouldn't optimize them away. The framework itself imposes a pretty large amount of overhead (moreso than I'd expect from leftpad).
It is somewhat likely that what they are measuring isn't leftpad performance, but rather how fast the JIT ends up optimizing the benchmark code.
Yeah benchmarks can be much more difficult to get right than it might seem at first glance. Good thing they didn't try to write their own benchmarking code, otherwise they might have fallen into those traps you just mentioned.
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1].
There's no portion of the code that does warmups. There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. There is a lot of code though... so that's... good?
You make the assumption that just because a lib is popular or widely used it is "correct" or "the best". When it comes to microbenchmarks, that's usually flawed. Very VERY few people actually get them right, benchmark.js is no exception.
That, of course, doesn't mean that benchmark.js can't be useful. For macrobenchmarks it will be roughly right. However, for something as small as leftpad, it's almost certainly not the right way to measure performance.
Okay, you win. I'm not going to read the entire source of that package just to make a point. Though I do find it strange that the author of that library would write an entire blog post on the topic and then not take his own advice in the implementation of the library he wrote.
Tell me, where in that blog post does he mention doing warm up cycles or avoiding having the JIT optimize away the method? (Hint, he doesn't mention that... so, no, he didn't actually miss his own advice.)
The article is completely consumed with getting the timing of benchmarking right. Which, to be fair, is a place where microbenchmarks often go wrong. It, however, isn't the ONLY place they go wrong.
> There's no portion of the code that does warmups.
This isn't true - Benchmark.js will repeatedly rerun benchmarks until it gathers statistically meaningful results.
> There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark.
True, and there's actually nothing benchmark.js can do to ensure that doesn't happen in the general case but when this does happen the results are usually pretty obvious - we'd see billions of ops/sec. Incidentally the left-pad benchmarks do not suffer from this issue.
> Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1].
This is the "benchmark lib" that was mentioned in the very first sentece of the comment you replied to.
That would depend on the underlying implementation of the string object, no? (If the string is implemented as a linked list it's O(n))
Anyway, it makes no practical difference in this case, since the one they labeled "O(n)" is the naive implementation that most people would write if they implemented left-pad themselves.
I highly doubt js engines would compile a string down to a linked list. But you're right they might compile it to a circular buffer or deque which can have O(1) prepends.
Well, my point was more that if a programmer thinks '5 minutes work' it's often 10+ minutes; so when a programmer thinks 'an afternoon', you can possibly lose a week. And then the article really doesn't work.
And yeah, I would and do write leftpad myself it it's not in the stdlib. But if there is a large library full of similar (string) functions that I might need, I would include that library. Not a singular dependency for this type of function.
The issue I have with this is a lack of specification. Left pad _what_?
Numbers or ASCII-only-printing? OK that's a reasonable. Is there a desired overflow behavior?
Past that it becomes more an issue of where and why. The suddenly not-trivial example includes questions about fonts, layout, and multi-byte characters. Emoji, etc.
Incidentally, in pseudoscope:
Create a valid full-space pad string (termination / etc), then decrement back from the end of the source string and over-write the pad characters from the end to the start of the string, exiting either on no more pad characters or no more input.
A second algorithm might combine those two steps as one pass, fill the output buffer from back to front. Only for C style strings would this be an issue given the dynamic end point for the data structure.
That was my first thought: I've seen these projects before — they're where you find 5 slightly different implementations of similar logic, no logging or tests, failures as soon as someone uses Unicode, etc. and I get an order of magnitude performance improvement by replacing that code with an external module which has had the other 19 afternoons' worth of work it actually takes.
> and I get an order of magnitude performance improvement
Have you heard the adage about premature optimization being the root of all evil? Yes, even with the second part. What is the premature optimization here, in your opinion?
Most of cases developers create something new - that's the state of industry now, not too good but it's how it is. If you'd be refactoring the existing code - sure, find the problem, design the solution, have reasons going from A to B. If, however, you're writing new functionality, you don't know if you'll have problems of this kind with this code - so optimize for developmentality. You can remove those excessive crutches later - if and when you need them. In my experience, having them trumps looking into code and spending time figuring what it does mere months later - your own code, that is.
The point was that when something is large enough to be “an afternoon”, it's probably more work than you're expecting and you haven't yet discovered important details. If there's something which does what you need, it's far more likely that _other people_ have invested time sanding off the rough edges which you have yet to discover.
If it's not hard to use that library you're probably better off unless it's a problem you understand very well and will see a real advantage to tackling differently. For example, if you use a library and don't like it that experience will still be useful for having clarified what exactly it is that you want to do and the rough size of what you're taking on.
Well, when the programmer was burned too much by incorrect scoping before?
Don't buy generalized statements like "programmers are always underestimate efforts needed", or even, for that matter, "a task always requires all the possible time it might take" (Parkinson's law). There are exceptions from them :) which sometimes, in a good team, look more than laws themselves.
"This'll take an afternoon" - three weeks later......
Programmers are notorious for this.
BUT even apart from this problem ... you absolutely should use every dependency you can that will save you time.
Try to write less code not more. When you write code you write bugs, add complexity, add scope increase need for testing, increase the cognitive load required to comprehend the software, introduce the need for documentation..... there's a vast array of reason to use existing code even if you truly could estimate it and build it in an afternoon.
You also assume that you understand all the edge cases and fickle aspects of the dependency, all the weird ins and outs that the dependency author probably spent much resources understanding, fixing and bug hunting.
There's a hard fact that proves the above poster to be wrong..... how many dependencies took only an afternoon of time in total to write? Hard to say (maybe look at the github commit history) but I'd guess almost none. It didn't take the dependency author an afternoon, so why will it take you an afternoon?
Even worse .... you just lost an afternoon coding features for your core application.
Multiply this by every dependency that "you could build in an afternoon" and you'll be in Duke Nukem Forever territory.
I'd advise doing the opposite of this articles suggestion.
Find a dependency that will save you an afternoon? Grab it.
Somehow here we assume a good programmer routinely makes errors in time estimation by an order of magnitude, yet conveniently forget cases when, say, a non-trivial GPL library is embedded as a dependency into a project, and customer is asking for code, and legal team runs with hairs on fire because company didn't plan to release the code...
But that is a completely different topic. Licensing is an issue, yes, but that is part of the upfront decision process.
In this day an age, this many years into open source licensing, if your team is not on top of that from day one, they have failed as a team.
I worked at Bell Labs from the mid 1980s to 2000, and by early 1990s (1992? 1993?) they already had a full internal team dedicated to open source licensing issues, including training and consulting. That was 27 or 28 years ago. Before some of the developers on this thread were even born.
Exactly. I'm not reinventing the wheel. I may write some convenience wrapping around Spring Security, for instance, but why would I rewrite auth-z when it's a solved problem?
> you absolutely should use every dependency you can that will save you time
> Find a dependency that will save you an afternoon? Grab it.
Agree. The point of the article, though, is that dependencies are often saving much less time than they promise - so much less that it's better to avoid them.
And when you run into a bug or design problem in a dependency of a dependency of a dependency?
It often takes less time to write some code than to understand someone else's code.
Most programmers I've worked with get lost easily when jumping through layers of other people's code. I certainly do.
Solid, well tested dependencies that solve hard problems are worthwhile. But dependencies have a cost in debuggability and maintenance, so it's worth using them with care. And often, they aren't worth the time, when compared to writing a dozen lines of code.
I get easily lost jumping through layers of code written by corporate developers in an afternoon. I generally don't have problems jumping through layers of popular, well documented and single-purpose third-party libraries.
While I agree that if you think it'll just take an afternoon, for the sake of this article it had better!
But conceding that charitable assumption to the article, I agree with its basic premise: dependencies cost a lot of time in diffuse, non-codey ways.
There are AAA dependencies you pull into every project, but most other dependencies require a good degree of due diligence, evaluation, risk, and their own long-term maintainance.
Its not that it always tips the scales all the way to 'roll your own', but
I think the cost of new dependencies is underrated.
But that analysis is part of the design process on the front end. You don't just 'take' libraries or utilities without evaluating them. And you don't just write bespoke libraries without thinking about the APIs.
So do your upfront work, by all means. It isn't an all or nothing decision.
- Dependencies break over time. They have a nonzero maintenance cost.
- They impose API boundaries on you that may not fit your existing data structures
- It's harder to change underlying bugs
- They might introduce security issues
Sure, use dependencies. But there's a reasonable position between "never write any code" and "never take on dependencies". Of which NPM is one of the only ecosystems being at one extreme.
I could say all of the same things about the in house tools that the “architect” wrote three jobs ago - including the bespoke ORM, object mapper, and logging framework.
Or two jobs ago where two developers who had worked at the company for 10 and 15 years respectively were maintaining a bespoke 15 year old EHR written in PowerBuilder and depended on SQL Server 2003 - in 2016.
Every company thinks they are their own special snowflake where cross cutting concerns can’t be handled by a third party.
Funny you should bring up these as examples. I've made (1) and (3) using an existing object mapper. They were made out of limitations with JPA/Hibernate and for higher-level functionality in logging. The ORM was never sent to prod. The logging events were gold. Specific events were 'major' and filtering on a user ID in a narrow timespan could show the expected/unexpected events for the traces as a sequence diagram through all the microservice layers. Clicking on an event then searched Loggly for all logs for trace-id at time. It got to the point that non-techs were answering customer issues with it and we hardly had to check/wait for Loggly to answer.
This is pretty much spot on. Except you also missed the main cost, which is the insane amount of time it takes to learn the 85 dependencies on your project to an extent that you actually understand what your code is doing.
Every single project I go into seems to have a smorgasboard of dependencies, then when I take the time to investigate one of them I find out it's being used incorrectly by at least 50% of the team because they don't even understand how they work at the most basic level. Which is pretty understandable because by the time anyone gets through understanding 10 of the 85, they've probably been kicked off the team for not actually building anything.
People love to say rubbish like "write less code!", as if LoC is the only metric that matters (weren't we past that thought process by the 90s?). Which goes a long way to explaining all the fucking terrible codebases I have to work with where it's impossible to accomplish anything without reading documentation for 8 hours, when it would take 20 minutes to just read even a semi-readable piece of code that implements whatever requirements you need from the dependency.
On a C code project for a large Fortune 100 company a half dozen years ago, I encountered a pesky header include that made no sense. And that header was part of a patch that I really did not want to pick up, so I started digging into it.
Turns out that they had some constant in the code, and the developer just did a grep for that value in the source tree, and that constant already existed in an existing header file, so they just included it.
And that CONSTANT_VALUE_STRING had nothing to do with the technology that the C source was addressing. So some lazy slacker pulled in a random header file that contained the proper constant value for an unrelated technology.
The dependency on that was pure lunacy on so many levels.
And that was an internal dependency, not an external library.
So the lesson here? Not all dangerous dependencies are external.
Everyone who's used NPM in production for a not-insignificant amount of time has realized just how bad nodejs dependency hell can be. Unfortunately, webdev-du-jour has decided pulling in a hundred npm packages is better than writing a few hundred lines of code.
I keep hoping things like [1] are a joke but I'm starting to suspect they're not.
I'm sorry that my framework and bundler are using so many packages. Lemme just quickly install Android Studio and download a few gigabyte to develop and build my application. Ah yikes I'm on a different version, need to redownload now.
At least Android Studio doesn't break when you try to deploy it a few months down the line (with package lock), with the exact same version, because a dependency of a dependency of a dependency made an unreviewed and untested "security fix" that caused a regression.
> "This'll take an afternoon" - three weeks later......
> Programmers are notorious for this.
From my experience with these personal failings, the problem usually comes from the question being phrased in the context like, "before you begin working on this, how long do you think this will this take you to complete?". If there's no opportunity to scope, with requires not insignificant work towards the solution, the estimates will always be wrong. If I understand the actual scope of the problem, which means have the architecture mostly worked out, and have a bit of experience (and luck), my estimates can be pretty close, usually eaten up by that oh-so-seductive feature creep that ruins my work file balance.
I recently read through the (free online) 'book' on Basecamp's 'shape up' methodology; I thought the 'hill chart' describes this really well - the work needs to be in progress going up the hill discovering what it's all about, before you get to the top and can accurately assess how much 'real work' (!) there is to do, and then it's all downhill from there.
> you absolutely should use every dependency you can that will save you time.
Absolutely. As long as it does save you that time over the foreseeable lifetime of the project. Or you are deliberately incurring a technical debt because of some deadline.
On the other hand, saving an afternoon (or even a week), over the next two weeks means very little.
Essentially, it'll take you an afternoon to write and then weeks of work properly fixing the bugs and handling the edge cases. Potentially and probably, while you're trying to do something else.
We all too often forget the scope: requirements, developing, testing, to say the least.
My favorite example is NPM. While the author has a point, I tend to rely on the wisdom of the crowd. Sometimes there is a reason why a couple of million developers - in the case of NPM packages - seem to be lazy.
In my experience, we ended up copy/pasting and modifying some code and syncing it with the "superfluous" package. Good intentions, badly executed.
Leftpad was the right itch at the right time and people found better ways to deal with NPM. NPM got better after that, as well as native implementations.
This goes both ways. When was the last time someone properly scoped the maintenance effort of an external library? This goes double for external systems, like kafka or mysql. I've never seen anyone so far even get within two orders of magnitude of the real cost of operating kafka, much less an organization that accurately compared that to the cost of DIY.
The "don't reinvent the wheel" argument often acts as though using a 3rd party lib is "free", and building it yourself is costly with no benefit.
This is sometimes true, but often not. From SFTP libraries to SVG rendering libraries, there have probably been about 3-5 major dependencies of my company's project that I have had to learn and extend or fix bugs in to make them work just in the last year.
And sometimes this means using our own fork that we have to keep maintained.
I'm not saying I would have rather written these particular dependencies from scratch, but they were definitely not cost free. Nor are they all of better quality than what I would have produced had I written them from scratch.
That's the other common refrain - to "defer to the expertise of the crowd".
Don't get me wrong, many 3rd party libraries are of great quality by amazing men and women who I am very thankful for. But certainly not all of them.
There's no magic that says "every third party library is made by an expert with the highest standards".
It really depends (haha) on what it is. I needed to copy a file in npm scripts. can't use `cp` because that fails on windows. I looked on npm to copy a file. First hit 197 dependencies, 1170 files, 47000 lines of JavaScript.
Taking 197 dependencies means 197 things that need updates several times a year at a minimum. Any of those updates could break my code, introduce a bug, add a vulnerability on top of the ones already in the packages. So it's not like adding more dependencies is magically free.
I’ve been working on the same medium-size (fewer than 1M LoC) codebase for about 7 years now. I feel like over the years, my estimates of how long something will take have gotten better for one reason: I’ve found the scaling factor I have to apply to my intuitive estimate that brings into the realm of reason.
So, if I think something looks like about a day’s work, I’ll actually estimate it at about 3.5 or 4 days. Thus, for a project to qualify as “just an afternoon,” I’d have to naively estimate it at under an hour.
I rarely have time to spare, but I also rarely go over by more than maybe a third.
Your multiplier may vary depending on how horrifying your codebase is. On a side project with good test coverage, my multiplier is only about 2.
I do this all the time. My head tells me "five lines, tops" -- corresponding to about 10 minutes of "programming." Add in testing, bugs, another 10-20 lines of comments and docs, we're looking at an afternoon.
Never do I give that raw 10-minute estimate to anybody, because it can be wrong by a factor of 10.
Ironically, these days with front end development I'm finding it hard to accurately scope how long it will take to incorporate 3rd-party dependencies. The docs make it seem straightforward enough, but they don't cover how to use it correctly under TypeScript instead of ES, or how to use it with Angular instead of React, or how to build it with Rollup instead of webpack, and I often spend an entire day googling obscure blog posts on how to get a dependency working in my own ecosystem.
(1) Most programs don't have that many direct dependencies, especially smaller dependencies. It's often dependencies of dependencies. In npm, adding just `jest` will make your `node_modules` directory explode.
(2) if Linus's Law is "given enough eyeballs, all bugs are shallow", then having fewer eyes on a library means more bugs.
(3) Oftentimes you think you can replace a dependency with an afternoon of programming, but it turns out, it's not quite as simple as you think.
(4) Sometimes you only use a small piece of a library to start with, but over time use more and more. If it's your own code, then you're going to be continuously refactoring, updating, rewriting it. If it's a library, then you can just start using the additional pieces as you need.
Even if this is true, the problem is you don't know what's going to take an afternoon of programming. How many times have we all said "should only take an hour" just for it to take four days?
If someone spent the time and effort building something you need, I don't see a problem with using it. It all depends on what kind of system you're building, what the security and stability guarantees are, and in general what trade-offs you want to make.
I knew which video it would be before clicking... But to your point they're often not really "store-bought" bricks though. More like bricks someone was giving away on the side of the road, you're free to use them to build your house but with no guarantees they work and the instructions are missing or incomplete. Oh and they're the wrong shaped brick but you only figure that out later.
Ugh please just don't use popups at all. I block every single one I see with custom CSS rules. GDPR cookie warnings, "please subscribe", stupid "can i help you" chat popups, everything.
When coding, be prepared to put it in the garbage bin! If you're prepared for this, you can code things more quickly, and not worry about the code slowing you down later. This works when you're unsure exactly what to build and need to iterate (agile). First build is an iteration, not arbitrary "sprints" or "increments"!
The cost of rebuilding MUST be budgeted though. If you don't have this freedom, things are bound to suck one way or another. Then, next best thing, build it as simple as you can, and put effort into making it composable and pluggable. So you retain freedom to swap out components. This is also an investment, and takes some more time and effort.
If you even can't even have that, results are bounded by those restrictions.
My rule is, don't use a dependency to implement your core business. Is JSON parsing our core business? No, so why would we ever write -- and thereby commit to supporting for its entire lifetime -- JSON parsing code? All the code you write and support should be directly tied to what you as a business decide are your fundamental value propositions. Everything else you write is just fat waiting to be cut by someone who knows how to write a business case.
To be clear, this is about the lifetime support of code. It's very, very rare that code can be written once and never touched. But that long tail of support eats up time and money, and is almost always discounted in these conversations. I don't even care that Jackson JSON parsing has years of work behind it, when I can hack together a JSON parser in a day. I care that Jackson will continue to improve their offering without any further input, while that's not true of my version.
Well, one special edge-case would be where you only need to parse some extremely tiny subset of JSON (for example: you only need to parse dictionaries whose keys and values are positive integers, like {1:2,3:4}). Then, depending how expensive the full json parser is, it might be worth your while just writing the limited parser yourself.
Of course, you might say, inevitably feature-creep will expand the list of things your parser needs to parse, but that's not a law of physics. Sometimes in certain limited, well-defined projects, it really is true that YAGNI.
No, I won't beat them. But if it a limited subset that I can implement with twenty lines straightforward code, that will often be cheaper.
I've been on projects where they imported xml-parsers many times bigger than the rest of the whole codebase just to send a well formatted order number.
With all the technical debt associated with it, which is the problem basing your project on a dependency that would allow you to easily scale and add features is a huge benefit.
This is like saying you should roll your own crypto because you only need to do a very limited sub set of crypto operations so why use something like NaCl or Tink.
Encryption is a terrible edge case. If you are forced to half-ass encryption, you should seriously question the project requirements. Bad encryption can be worse than none at all. Things won't end well if data security is treated as a detail.
YANGI is nice but when the PM asks you why it would take two months to accept a new JSON format from a client and you’ll answer well because we didn’t want to use an industry standard fully functional and vetted JSON parser so we essentially wrote our own edge case parser we both know how that conversation will end.
And YANGI doesn’t have anything against dependencies.
When there are new requirements you do a quick estimate if you should add four new lines to the existing 20 or if it is worth to switch to an external library. 4 new lines to the 20, just add them to the core. But if this is regularly occurring that you have to add things, or requirements affecting this particular little parser that was supposed to be simple and static isn’t, then you should probably change your decision and use the library.
But you do that only then. Because chances are that with your approach you are going to drag along a large generic library that you only use a tiny fraction of. And that also has costs. In particular if your immediate impulse always is to add another library instead of writing things yourself.
By using a third party library you are writing twenty lines less code, so it's cheaper in that aspect.
There are probably libraries that are faster than your twenty lines of un-optimized code, so it's cheaper as far as computing resources are considered too.
The only time it could matter is when you ship the code to the client through the wire (such as a Javascript bundle).
You can’t use a library with zero lines of code. On top of this library’s always have development overhead outside of the code you write. Ex: What version number should you use? Did the latest version break something? Did the old version break something on the latest compiler? Etc etc.
It’s cheaper in the sense that it is faster to write and maintain those 20 lines of code. Because someone has to evaluate the library, understand it well enough to actually call it and then make sure it stays up to date. And often there are a few lines of code to translate your data into a form that the library requires etc.
Plus for every developer to come, one call to an external library usually also means 30 pages of documentation to trawl through if they ever want to change anything, 29.9 of which is completely irrelevant to whatever your narrow use case is.
That's the real cost. The size of the code means absolutely nothing.
And it's going to take more than an afternoon to evaluate these parsers. You have to look at the options, evaluate the API, evaluate if they're stable and supported, evaluate if they integrate well with you're project, evaluate any dependencies they might have, etc. Then you need a plan to manage these dependencies long term.
If you're needs can be solved adequately by strtok then that's a far simpler and more maintainable solution that can be knocked out in an afternoon.
Your example is more apt than intended: That's not valid json, which only allows string keys. If you use a library it'll either barf now or later when they fix it, so if you're forced to work with an API like that and can't change it, a custom parser is really the only way to go.
That's not JSON, though. It's absolutely something else. Maybe a JS snippet. Maybe YAML. Definitely not JSON, though.
(Some JSON libraries do have option flags, but usually it's about whether, during deserializing into a known type, unknown fields are an error or silently ignored. Or whether C-style comments are an error or considered as whitespace.)
While acceptable, also misleading: JS only does string keys, but unlike JSON it'll convert whatever it's given into strings. Not a problem most of the time, since it'll do the same conversion for both accessing and setting, but good to be aware of if you're doing something like iterating Object.keys()
> Of course, you might say, inevitably feature-creep will expand the list of things your parser needs to parse
If you've done your parser correctly, you'll be able to replace its implementation with the new dependency, with little to no need for extra refactoring in the rest of the codebase.
You can also apply YAGNI to 'do we need our own custom parser'?
You don't know what your requirements are. The customers haven't told you yet.
If you pick a library with a straightforward interface, especially one that isn't too opinionated, you can always drop in a custom implementation later on. Frameworks, not so much (but that cuts both ways; the people who will write libraries often love writing frameworks too)
Great rule. I was wondering, how do you manage updating the Jackson JSON parsing package. What if you have 100 such packages and they get updated weekly with breaking changes ?
If you're in a larger corporater environment this can also be used to create some predictable labour needs - create a seasonal updating taskforce so that the business get a more transparent view of how much labour is being sunk into maintaining these, break it down into specific dependencies if you've got one or two that you think are particularly expensive- showing after the fact labour numbers from one season may motivate sane inhousing for next season.
Only update dependencies when your code requires the new version, depends on a bug fix or it fixes a security vulnerability. Otherwise, continue using the same version.
Have good test coverage to catch bugs that may originate in dependencies and subscribe to a third-party service to track vulnerabilities in your dependencies.
Then you get 5 year out of date packages, which eventually have a security vulnerability, and now you have the task of upgrading and working through 5 years of (potentially) breaking changes and deprecations.
It's generally easier in the long run to keep your dependencies up to date. If a package has a new breaking change each week, that's a sign you probably shouldn't be using it for production code.
If you have a hundred direct dependencies and they all break the API on a weekly basis then: you are either at a scale where you can handle that, or you are using wrong dependencies, or you are doing something wrong.
I can understand max 10 dependencies iterating so quick. But only when they are your own internal dependencies and these should definitely not break the API weekly.
There's lots of opinions on this, all with good justification. My current team leaves most dependencies unlocked and depends on good automated tests to sniff out broken dependencies. If necessary we lock dependencies to a particular version or range (e.g. <2.0.0). Once tested, we freeze for distribution.
Some people just never upgrade until they need to. That's workable, though when you do need to upgrade a package you may be spending the rest of the week working out a cascade of breaking changes.
If you only upgrade when you need to, but not necessarily to the latest versions, odds are that whatever breakage is caused by the latest nodejs/npm/etc incompatibility has already been documented in issue trackers or stackoverflow
For what reason are you updating your packages? Is there a severe security issue in that package or, if it works today, could you pin it to that version and wait until there is a compelling reason to update it.
Here's some reasoning - if this project was inhoused would we detect and patch it any quicker? Would we have a dev constantly assigned to it that would be pushing out patches to the rest of the team... or is it the sort of software we'd write once and then wait until a compelling reason to invest more into. Whether software is inhouse or outsourced you still retain decision making about how much time to invest in its maintenance.
> if this project was inhoused would we detect and patch it any quicker?
If it's a bespoke library, no one but you and hackers directly targeting you will test for security vulnerabilities. (Good thing you have a red team... right?) For widely-used libraries, the number of vulnerabilities isn't going to be much different from your own library, but the likelihood that they're found and exploited in your system is quite lower.
So no, in most cases, you would not detect and patch vulnerabilities quicker, because you probably don't see them until it's too late.
> if it works today, could you pin it to that version and wait until there is a compelling reason to update it.
If you pin versions for a long time, eventually there comes a point where you have to update something because of a critical bug or security advisory, and of course since it's a critical bug or advisory, you have to update "right now", "priority 1", "all hands on deck", "the board is involved" and everything. The fix is in version 5.1.2 of the library, but you're stuck at 2.6.5, so now you have to do three major version upgrades (with all the changes to your codebase that entails) before you can even think about upgrading to the version containing the security fix. And that's still an easy case. If the library in question is a framework like Rails or React, version upgrades of that size may be a major undertaking that takes weeks or months to prepare, execute and validate. That's very much not fun when management is pressuring you to close that vulnerability.
I think it's never a good idea to sit on ancient libraries. Put a recurring task in your team backlog to update dependencies on a schedule. It's not going to result in less work spent upgrading, in all likelihood it's more work in terms of raw hours worked compared to the update-on-security-advisory strategy, but it's much more plannable and less stressful. That doesn't mean you have to upgrade to latest-greatest immediately (you always have the freedom to hold off a particular upgrade until the new major version has had some time to mature etc.), but there should be some time reserved on your schedule for doing your updates.
For instance, I have my update-all-lib-deps reminder in my calendar on the 1st of every even month. When it comes up, I put a task in my backlog with a checklist containing every application I have to check, upgrade and deploy. Go 1.15 just came out today, so that's going to be on my desk come October. Great timing, actually, we're going to be one or two point releases into the 1.15 branch at that point, so it's going to be a safe and easy upgrade.
> don't use a dependency to implement your core business
In logic language, you're saying "If X is your core business, don't outsource X".
> Is JSON parsing our core business? No, so why would we ever write -- and thereby commit to supporting for its entire lifetime -- JSON parsing code? All the code you write and support should be directly tied to what you as a business decide are your fundamental value propositions. Everything else you write is just fat waiting to be cut by someone who knows how to write a business case.
The rest of your argument is interpreted as "If X is not your core business, don't in-house X".
These two logical implication statements are not equivalents of each other, but are converses. Casual language often conflates If, Only-If, and If-And-Only-If.
You should spend time implementing your core business implies that you shouldn’t spend time implementing things that aren’t in your core business, otherwise the first statement is pretty useless.
If I wanted to learn more about rigorous, non-elementary logic, do you have a recommended resource? I've taken a course in intro level probability theory which covered it generally and another course that built on it lightly but nothing rigorous and I am wooed by how concise things become in a logical form.
A Tour Through Mathematical Logic. You don't have to do any proofs. If you learn Propositional Logic and First Order Logic you'll already have most of the tools to invent the rest.
I think the problem is that the individual contributor has decided to make that chunk of logic their business. This will probably not benefit the team or the organization.
Quite agree, every single line of code written requires lifetime support. Code adds up and reduces productivity gradually, so only write code in core business logics.
Beside _lifetime support_, working on that core business feature make us _understand_ deeply about the that feature.
I've seen people integrate dependency for their core business. It helped to get started fast, but will create a blockage that required understanding deeper to overcome
I think a JSON parser is not a good example though — takes longer than a few hours / an afternoon, to write a JSON parser, add tests, fix bugs, corner cases. More like a week, or weeks, ...
I suppose a JSON parser was just an example. Made the whole answer sound weird to me though :- ) when the blog is about afternoon-projects and then a reply is about a week(s), could be month(s), long project.
There's a fair middle ground when the dependency in itself doesn't have dependencies, and is small enough with a permissive license such that the entirety of its code can be dropped in to your project. Especially for very specific functionalities. I have used such tiny xml parsers, and I'm not affected by the fact that my copy is no longer the latest version. Its not so far from copying and pasting snippets of existing code.
> I think a thing is not a good example though — takes longer than a few hours / an afternoon, to write a thing, add tests, fix bugs, corner cases. More like a week, or weeks, ...
You're making my point for me. This is exactly what I meant by the lifetime of support you're signing up for by writing lines of code. Once you write that code, you're now in the business of supporting that code. Was that a good decision for your business?
Same with CSV. It looks easy, but it isn't. I've never seen anyone who writes their own CSV parser actually implement features necessary to conform to the standard like quoting and escape sequences. The end result is software that breaks when delimiters or quotes appear in user input. Honestly, I prefer xlsx spreadsheets because of that. Nobody fools themselves into implementing the parser or serializer for the format themselves. The only tiny pitfall with them is when people create spreadsheets manually in excel and write numbers as text, but parsing strings to numbers is absolutely trivial. You have to do that with CSV anyway.
So you're saying that I should implement my own ormapper just because my product is using a database?
And even this is not thee case, writing everything yourself will end up in your own hands. No Bugfixes, no patches or improvements without spending man work.
I've worked in such a company and it was a mess accompanied by dev leaders who's to proud of their code to allow any change.
I'm confused by your response. Is your core business mapping objects to databases? As in, that's what you get paid for? If not, my heuristic is that you should not be writing an ORM tool.
Shared dependencies can also reduce, code though. Do you really want 10 slightly different implementations of the same thing after you've brought in a few large dependencies?
It's a matter of judgement, but here's a few observations:
- With a little experience, you know what gets fiddly and what doesn't. Today for instance, I needed a way to remove tags in an SVG document, which looks a lot like HTML tags. I quickly ended up finding that Regex is not the solution (a well known guy on SO wrote an answer that looks like a huge warning sign). I also couldn't enumerate all the corner cases. So I found a lib that does it, along with an SO answer that turns it into a two-liner.
- Dependencies vary in quality. Some are basically like another standard lib. Boost for instance is very well used. The tough ones are where the lib seems to be "finished", where there seem to be few commits recently, but the project was once lively and functional. IIRC libev comes to mind here. And then there are the totally dead projects, where there's a load of issues open and nobody saying anything.
- Try to lock down versions. If you get a thing working with a certain version, there's no reason you need the newest new as soon as it's pushed. You can probably live with doing a scan for updates now and again.
- Your afternoon of programming needs to have a clear end. That hashmap you wrote will very likely spew out issues over the next few days. CSV parser, maybe. Bessel function, that'll work.
The old classic. I still end up using regexes when I need to clean up HTML because even though it's not the right way, it's the way that is sufficient 95% of the time.
Your dependency that you could code "in an afternoon" may handle far more corner cases than you suspect. (That may be what you meant by "fiddly".) Sure, you don't care about covering all those corner cases... but you might care about some, even some that you haven't thought about yet. And you might care about some more next month. That can make that "afternoon" take a lot longer than you expect.
>Try to lock down versions. If you get a thing working with a certain version, there's no reason you need the newest new as soon as it's pushed. You can probably live with doing a scan for updates now and again.
Agree ! it irks me a lot that I often see update bots tracking new releases.. it is just begging to be exposed to regressions.
We need to find a happy medium though. Otherwise whenever you actually need to update something (e.g. you need add a new dependency which only handles one of your other dependency if it jumps 20 releases ), you have a huge version gap to cover.
Specifically on the SVG filtering example, which I think is a good illustration of when to use or not use a dependency:
Writing an SVG (or at least XML) parser is a necessary task for writing a filter that doesn’t get stuck due to weirdo issues. That is way more than an afternoon of work! But once you have a parser, dropping tags you don’t want or transforming them somehow is totally an afternoonable task size. So, do use a dependency for SVG parsing, but don’t look for a special “SVG filter all” package. Just do the filtering yourself.
Totally agreeing with this but the issue is maintenance. Sure, I can find a package and copy and paste a few files and make it my own — or do it from scratch even!
But then, that's not really my core domain, so I'll most likely never touch that piece again anytime soon. A quality package tends to have that covered (with a small risk, too if it's an untrusted source).
So true. Oftentimes I will look at the source of a simple ruby gem or node package and see that it is actually really just one single file, with 100 various .yml/spec/lint/test/cloud/coverage/cov/tox/blah landmines in the repo to confuse you. In those cases, I'll copy-paste the code with attribution back at the top of my source file.
The headache is not worth all of the pomp and circumstance for some of these tiny little tools.
- Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. -> It's gonna take longer than an afternoon. Always
- Opportunity cost: Your app needs to DO something, and spending time building dependencies steals quality from your core business proposition, even if it's a small (Haha: see previous point) dependency.
- While there are a ton of crap libraries out there, there are also a lot of good ones, which include hard lessons learned, about even simple tasks. You _could_ rewrite a very simple http client in an afternoon (telnet hostname 80\n GET / HTTP/1.0\n\n), but you'll probably have a ton of glaring flaws just waiting to bite you _badly_.
332 comments
[ 4.5 ms ] story [ 285 ms ] threadIt usually just boils down to keeping a copy of the version of a dependency you used.
This is actually one of the most common causes for GPL violations. And companies have gotten into serious trouble for that.
Either rewrite the code, or maintain it as a dependency and follow the licensing rules. Don't simply copy code from it into your project unless you have explicit reason to believe that that is OK to do.
Dance with the devil you know and all that.
Hardly a problem today since I just write to `stderr` and have the container log aggregator push elsewhere but boy was it annoying.
The problem lies in the fact that there are a great many things I can hack together in an afternoon to "replace" some kind of external dependency, but the quality discrepancy of these hacks is highly variant. My understanding of what can or should be done in an afternoon might differ with my colleagues'.
Unfortunately, like all things in engineering, you have to carefully reason about the pros/cons, requirements, and costs. After that analysis, you can make a judgment on depend vs. build (also, buy vs. build).
Over time an issue tracker inevitably becomes a collection of hard-to-reproduce bugs, incomplete patches, underspecified feature requests, random tracebacks, etc. Maintainers can choose to just close everything which is not actionable immediately, or be in comfort with such issues, and let them live in the bug tracker. I personally like a style when an issue is closed only if it is fixed, or if it doesn't contain useful information, or if it is a duplicate.
A better indicator is activity and responsiveness of the maintainers in the issue tracker.
I can look at the code, get a good grasp of it (hopefully), judge the quality, docs, prospects of getting updates/needing updates/being able to update it myself, pretty comfortably. In other words, the risk evaluation is incredibly straight forward.
Additionally, the risk itself is fairly low. If it goes out of date or stops working or just turns out to suck, the most I risked is an afternoon of work. Leftpad was a debacle due to it's scale, but fixing Leftpad was pretty easy (I'm not recommending importing one liners as dependencies mind you)
-
But when it comes to stuff that isn't small, it's usually also the kind of stuff that holds the most insane amounts of risk for a project and is the hardest to evaluate.
Stuff like application frameworks, threading frameworks, massive networking libraries, etc.
The interface is _huge_. To the point that even when you try and wrap their complexity in nice packages with separation of concern and encapsulation they leak out into the rest of your code and end up being a nightmare to ever change.
Instead of spending an afternoon writing dependencies like this, spend that time investigating your "too-big-to-fail" dependencies. Try and keep a finger on their pulse, because they're the ones that will really come back to bite you if things go south.
Sometimes, the opportunity cost (time spent) is the largest term in the risk equation, but often there are other terms that might be orders of magnitude larger. For example, the risk of depending on the wrong abstraction, or becoming coupled to a hack.
What you're saying makes sense. My only point is that there's a lot more subtle judgment required in these decisions than often meets the eye.
Until I found it wasn't installed on some of our test machines (it was needed for testing, not for production and for reasons beyond my pay grade, libcurl was not available on the machines). Then I thought, well, I could include the libcurl into our vendor repo. It worked, but it was a nightmare to use. It took way too long to figure out the proper "configure" options to use for what systems, it nearly tripled the time to build it on the build servers, and even then, it was hit-or-miss.
After several years of this misery, I removed libcurl, and wrote what I should have years earlier. Using libcurl as a dependency did NOT save us any time.
The support aspect of internal libraries, especially in the age of Stack Overflow, is widely overlooked by the very people who Must Be Stopped.
Perhaps it's a domain-specific thing, but when someone uses the words "hack together" I imagine it means using dependencies without really understanding what's going on in them, precisely to avoid figuring out how to code a solution properly.
Writing it yourself obviously needs to also imply doing it correctly. Even if that means you must learn a bit about what is the right way to do it (a side benefit, though usually viewed as a downside).
Yes, I can write a hashtable implementation in an afternoon, but it's going to have bugs that I'll spend the next year fixing, and still not achieve the performance of the pre-built version.
All that work of finding existing solutions and learning how to use them? That's part of the job.
Find a bug in the dependency? Submit a patch.
Worried about the dependency changing? Lock the version.
Too many external repos to retrieve those dependencies? Use a local cache.
Don't reinvent the wheel.
If it has any bugs that would surface in a year of production (while the dependency version wouldn't) then you didn't write an equivalent in an afternoon.
The advice, if it's to be useful at all, must be things that you could completely replace in the same quality, in an afternoon.
It's the left-pads and is-odds, to begin with.
And the quality of the original might be questionable for many cases.
> Yes, I can write a hashtable implementation in an afternoon, but it's going to have bugs that I'll spend the next year fixing, and still not achieve the performance of the pre-built version.
The meaning of "afternoon work" should be considered "of good enough quality". Tests, structure, reasonable docs, all that. It shouldn't be a fastest written something, it should be a normal code.
> and still not achieve the performance of the pre-built version.
Some losses in performance are acceptable for greater visibility and better fit for the project. If you need non-trivial performance gains - well, those are also achievable by code, are you sure you can actually write such code in a few days?
> Find a bug in the dependency? Submit a patch.
That's the point. To submit a good patch, you have to internalize the system. It's easier to do if the system is yours - doesn't do much except what you need.
> Worried about the dependency changing? Lock the version.
Now you've locked yourself out of upstream bug fixes.
> Don't reinvent the wheel.
Here is a wheel patented in 1972 in US, with noticeable benefits over the traditional idea: https://en.wikipedia.org/wiki/Mecanum_wheel :) .
We do reinvent the wheel whenever we need to have an actual wheel for a device, not an abstract concept. Similarly, we write for loops, "reinventing" them for our specific purpose. Those are all different wheels, loops and needs. Don't mistake the "idea" of a hashtable with an implementation.
And get p0wned a year later when some security researcher finds a vulnerability in code that you don't even use, but pulled in as part of that dependency.
Of course you do and release(d) them as open source (public domain). Take Java - it has decent a HashMap but it's node based. It's memory inefficient to a point its nodes and arrays are top 3 of memory consumption. An array based hashtable takes around 3.6 times less memory for larger ones (on 4bytes compressed pointers) and over 10 times less for smaller ones. Perf. wise it's on par or better as well (nowadays architecture is heavily driven by locality and access patterns)
Also you make your code so it can switch between both on the fly, if need be.
How ironic though. Of course it did work a few times, but if the advice is to not use dependencies, then the better advice would be to not use dependencies that were written in an afternoon to avoid using some other dependency :)
Indeed! Although I spent like a weekend to do it (the inital release was 512 loc). It passed all standard jdk/jsr-166 Map tests[0] and then some more, incl. perf., memory consumption, garbage collection harness. Tests are also public domain. Also the release is not available as dependency, so the interested user would have to clone the repository on their right own.
The part with afternoon deps would be that all their code can be read and cloned, if need be. Free to pick the few functions needed - I'd assume around 200-400 loc top.
[0] http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/test...
Factor, re-factor, and (most especially) DELETE should be tools in the toolbox -- but see if you need it/keep it (e.g. protoype it in, etc first) before you re-write.
Lines of code is a passable metric for the quality of work if they are considered "spent", not "created". That is, the programmer's work is better, all else being equal, if it takes fewer lines of code. Who said so? Knuth, Wirth, Dijkstra, Perlis?..
With things like reduce, map, the like their is less need to rely on lodash or underscore for them.
That being said, lodash has worked great for utility stuff. It encapsulates like 99% of utility functions I need in my apps.
If it is non-trivial, I prefer the official standard libraries for a programming language. That is if a solution exists in the standard library.
I think the Go standard library with its batteries included mantra and the level of support it gets is good example of a library that should be used when a solution exists within it or by utilizing it.
Removing a dependency doesn’t have to mean writing from scratch.
Given the variance in standard library coverage, it’s rarely productive to argue about this topic in a language agnostic way. Using only stdlib in Go is very different from using only stdlib in JavaScript.
Having a large standard library speaks poorly of the composability and orthogonality of language primitives.
Much like Common Lisp and SmallTalk, you can easily connect to production nodes and modify code while the service is running.
It's rare to find such a dynamic, flexible, and interpreted language that also has world class performance, even often beating hand written C. Combined with an integrated database, you get a distributed system that can't be beat, at least performance-wise. And the craziest thing is that all of this sits in at 650kb executable with libc has the only dependency. And all probably in less lines of code than a simple javascript webapp!
i really like writing typescript code, but holy shit, when you have to pull libraries to even the smallest thing (lol left-pad), it get super infuriating.
Sure, if you know that you have to support old and broken browsers you have to use these dependencies to ensure correct support, but if you know that your code will run on a specific interpreter (for example a modern version of nodejs, or modern browsers) you don't have to worry about it too much.
Also most JavaScript programmers tends to abuse dependencies, I mean even for things that are really 10 lines of code that you write in 1 minute.
I started with the problem of "I need a function that pads a string with a character out to some length". Coding it up took under 1 minute, easily under 5 minutes.
It took me longer to write this comment.The fact that the left-pad code has optimizations (which don't matter for the place I'm using it) and type-checks (which don't matter; my higher level unit tests would catch that mistake) is beside the point.
All of those properties make the code easier to reason about and test.
Being easier to reason about, and being a more exact abstraction for my needs, are both incredibly valuable properties.
I don’t have to care about how the underlying libraries work. I can treat them as a black box.
When I wrote haskell, the majority of the libraries did just work, and I didn't have to dig into their code to find bugs often.
When I wrote javascript, hundreds of the libraries I used did not just work. I usually had to care very much about their details because they were poorly implemented, full of bugs and incorrect abstractions, and often abandoned soon after.
I agree that there's benefits in reusing some well-socialized and well-implemented generic frameworks and abstractions. It's not worth using generic abstractions that are not well understood, buggy, and don't match your needs closely. In that case, write your own.
More generic is not always better. Above, I'm arguing that it's important for code to be easier to reason about. If a generic abstraction helps with that, cool, but it's not always going to be the case.
Any other scripting I do with Python. Any more complicated development it’s using a language with an ecosystem with adults - C# or Go.
Rob pike espoused "A little copying is better than a little dependency", and go refuses to add suitable abstractions to build generic reusable pieces.
Go has a strong culture of doing exactly the sort of thing I was talking about, and you were arguing against.
Almost everything I work with has bugs, so chances are I'm going to run into one. It's a lot easier for me to fix bugs when there are fewer layers and more of them are written by me. Of course, I can't write all the layers, but if they run on my service, I have to be prepared to fix them, or suffer from them being broken until a benevolent force fixes them for me. (Sometimes that happens, but usually not for the harder problems)
But I can maintain code that uses Entity Framework (or Dapper) much easier than I can maintain code based on a custom ORM that the “architect“ wrote.
I can count on one hand[0] the number of JS dependencies I've used in enterprise/large projects for extended periods of time where I've never needed to manually debug the library or read the source code. For enterprise software, sometimes the easiest solution is to directly hotpatch a vendored version of the dependency.
This is especially true with dependencies where bugs are fixed in major versions, but where upgrading and dealing with breaking changes would require significant code refactoring.
To drive the point home, I've been bitten by bugs in NPM itself.[1] Fixing that required reading through the source and manually swapping out one NPM's internal dependencies to a newer version.
And it doesn't matter if someone somewhere has had the same problem and posted it on the Internet unless I can find their answer online faster than I can fix the problem myself in my own library. Often this is not the case, filtering through issue trackers and trying to find the one blog post or comment that tells me how to solve the problem can be a big time sink.
[0]: Okay, maybe 2. But the point stands, it's not a rare or exceptional occurance.
[1]: https://github.com/npm/npm/issues/18942
Probably the worst was with a decently popular library someone had brought in, that tried to do a refactor from callbacks to async/await, without understanding at all how async/await worked. They'd leaked an async operation in the library code, so 'await'ing a specific function call in their API that returned a promise, didn't actually await everything the call was doing, ending up in a debugging nightmare. Of course their perfectly manicured suite of 8000 tests with 110% test coverage didn't catch it either, because the number of people who can write good quality tests is shockingly low, and library-writers aren't somehow magically ahead of the pack in that regard.
JS really feels like PHP did back when I was a newbie learning that shit. In other ecosystems, the 95th percentile devs seem to write all the libraries, so everyone comes here and posts repeatedly about how great dependencies are. In JS, it's the average dev writing all the libraries, and the average dev's code is enough to make my brain bleed.
I'm a big proponent of different advice for different ecosystems. If you're doing front-end JS, the pendulum has swung so far to one side that 'NIH syndrome' is treated like it's going to lead to the fourth reich, which makes 'chill out a bit on dependencies' pretty good advice if you're looking to get a leg up in the industry. But I'm sure there's other ecosystems where the same advice will just leave you with a tangled mess while your competitors leapfrog you in productivity with a good 3rd party dependency.
I'd say take any advice in threads like these with a grain of salt unless it's given in a bit of a narrower context. Taking some one liner about software engineering in general and applying it to your specific project is probably just a coin flip as to whether it's going to improve your code or not.
I've not profiled it, but I'm going to guess that now-a-days this will be faster than the current implementation on npm.
Why? because the VM is (very likely) going to do exactly what the cache would have done. It can replace `ch.repeat(len) + str;` with a presized string allocation and a memcpy of ch + str characters.Static typing would actually fix this kind of coding error.
> Static typing would actually fix this kind of coding error.
That's not a coding error, that's addressing an oddity of JS coercion rules that a less experienced developer could easily have missed.
> if (!ch && ch !== 0) ch = ' '
That code says that if `ch` is falsey and not equal to 0, then set it to a space. The only arguable falsey value that should be excluded here is a literal `false`, but that's not a single character and is fairly ambiguous either way. I'd certainly fall on the side that a literal false should not be converted to `'false'` here.
> ch += '';
The next line converts to to a string by adding it to the empty string.
> return ch.repeat(len) + str;
So by the time it gets to this line we know ch is a string.
Static typing is great, but the bug you claim is there is not actually there.
It may specialize or inline, but that's a separate matter.
P.S. I think there's a bug here.
Nowadays there's a native padStart function and the left-pad package is deprecated as a result.
JITs only optimize hot code which means any benchmark without a warmup is going to measure cold + hot code time.
Further, JITs will optimize away unused results. They aren't using the leftpad results in any meaningful way.
What they are likely measuring is how long it takes for the JIT to optimize the benchmarking framework.
watch: https://vimeo.com/78900556 for how to microbenchmark a JITed language.
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency[1]. The author of said library works on V8, and already considered all those problems and much, much, more[2].
[1]: https://github.com/left-pad/left-pad/blob/master/perf/perf.j...
[2]: https://mathiasbynens.be/notes/javascript-benchmarking
Sure, It's popular. It's also wrong.
Also I seem to remember that someone benchmarked the cached version and found it to be slower than the naive approach anyway. I could be mistaken there.
The most egregious is the bad big O analysis for a pointless binary search. The loop does indeed run O(log(n)) times but `ch += ch` still takes O(ch.length) which is growing exponentially. It ends up being a complicated way of still taking O(n) time while creating a lot of intermediate strings.
It isn't any faster than just creating the padding with a loop or `new Array(len).fill(ch).join('')` or `ch.repeat(len)`
It's not overengineered if thousands of downstream projects are relying on it, some of which might see significant benefits from those performance optimizations.
For example, it's missing warmup. The results aren't being consumed in a way that wouldn't optimize them away. The framework itself imposes a pretty large amount of overhead (moreso than I'd expect from leftpad).
It is somewhat likely that what they are measuring isn't leftpad performance, but rather how fast the JIT ends up optimizing the benchmark code.
I'd suggest watching this video https://vimeo.com/78900556
It's about microprofiling the JVM, but the principles are the same for other JIT based languages (such as javascript).
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1].
[1]: https://mathiasbynens.be/notes/javascript-benchmarking
There's no portion of the code that does warmups. There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. There is a lot of code though... so that's... good?
You make the assumption that just because a lib is popular or widely used it is "correct" or "the best". When it comes to microbenchmarks, that's usually flawed. Very VERY few people actually get them right, benchmark.js is no exception.
That, of course, doesn't mean that benchmark.js can't be useful. For macrobenchmarks it will be roughly right. However, for something as small as leftpad, it's almost certainly not the right way to measure performance.
The article is completely consumed with getting the timing of benchmarking right. Which, to be fair, is a place where microbenchmarks often go wrong. It, however, isn't the ONLY place they go wrong.
This isn't true - Benchmark.js will repeatedly rerun benchmarks until it gathers statistically meaningful results.
> There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark.
True, and there's actually nothing benchmark.js can do to ensure that doesn't happen in the general case but when this does happen the results are usually pretty obvious - we'd see billions of ops/sec. Incidentally the left-pad benchmarks do not suffer from this issue.
This is the "benchmark lib" that was mentioned in the very first sentece of the comment you replied to.
https://github.com/left-pad/left-pad/blob/master/perf/O(n).j...
They are repeatedly appending a single character to the front of a string. This is actually O(n^2) so of course they are winning against it.
Anyway, it makes no practical difference in this case, since the one they labeled "O(n)" is the naive implementation that most people would write if they implemented left-pad themselves.
Quick googling shows that this optimization might exist but only for firefox and only if you use "unshift": https://lannonbr.com/blog/2020-01-27-shift-optimizations https://jandemooij.nl/blog/2017/12/06/some-spidermonkey-opti...
But it's very unlikely that the jit can optimize `str = ch + str;`
You’re a webdev aren’t you?
https://stackoverflow.com/questions/13865420/why-is-haskells...
And yeah, I would and do write leftpad myself it it's not in the stdlib. But if there is a large library full of similar (string) functions that I might need, I would include that library. Not a singular dependency for this type of function.
Numbers or ASCII-only-printing? OK that's a reasonable. Is there a desired overflow behavior?
Past that it becomes more an issue of where and why. The suddenly not-trivial example includes questions about fonts, layout, and multi-byte characters. Emoji, etc.
Incidentally, in pseudoscope:
Create a valid full-space pad string (termination / etc), then decrement back from the end of the source string and over-write the pad characters from the end to the start of the string, exiting either on no more pad characters or no more input.
A second algorithm might combine those two steps as one pass, fill the output buffer from back to front. Only for C style strings would this be an issue given the dynamic end point for the data structure.
Have you heard the adage about premature optimization being the root of all evil? Yes, even with the second part. What is the premature optimization here, in your opinion?
Most of cases developers create something new - that's the state of industry now, not too good but it's how it is. If you'd be refactoring the existing code - sure, find the problem, design the solution, have reasons going from A to B. If, however, you're writing new functionality, you don't know if you'll have problems of this kind with this code - so optimize for developmentality. You can remove those excessive crutches later - if and when you need them. In my experience, having them trumps looking into code and spending time figuring what it does mere months later - your own code, that is.
If it's not hard to use that library you're probably better off unless it's a problem you understand very well and will see a real advantage to tackling differently. For example, if you use a library and don't like it that experience will still be useful for having clarified what exactly it is that you want to do and the rough size of what you're taking on.
Don't buy generalized statements like "programmers are always underestimate efforts needed", or even, for that matter, "a task always requires all the possible time it might take" (Parkinson's law). There are exceptions from them :) which sometimes, in a good team, look more than laws themselves.
"This'll take an afternoon" - three weeks later......
Programmers are notorious for this.
BUT even apart from this problem ... you absolutely should use every dependency you can that will save you time.
Try to write less code not more. When you write code you write bugs, add complexity, add scope increase need for testing, increase the cognitive load required to comprehend the software, introduce the need for documentation..... there's a vast array of reason to use existing code even if you truly could estimate it and build it in an afternoon.
You also assume that you understand all the edge cases and fickle aspects of the dependency, all the weird ins and outs that the dependency author probably spent much resources understanding, fixing and bug hunting.
There's a hard fact that proves the above poster to be wrong..... how many dependencies took only an afternoon of time in total to write? Hard to say (maybe look at the github commit history) but I'd guess almost none. It didn't take the dependency author an afternoon, so why will it take you an afternoon?
Even worse .... you just lost an afternoon coding features for your core application.
Multiply this by every dependency that "you could build in an afternoon" and you'll be in Duke Nukem Forever territory.
I'd advise doing the opposite of this articles suggestion.
Find a dependency that will save you an afternoon? Grab it.
In this day an age, this many years into open source licensing, if your team is not on top of that from day one, they have failed as a team.
I worked at Bell Labs from the mid 1980s to 2000, and by early 1990s (1992? 1993?) they already had a full internal team dedicated to open source licensing issues, including training and consulting. That was 27 or 28 years ago. Before some of the developers on this thread were even born.
> Find a dependency that will save you an afternoon? Grab it.
Agree. The point of the article, though, is that dependencies are often saving much less time than they promise - so much less that it's better to avoid them.
It often takes less time to write some code than to understand someone else's code.
Most programmers I've worked with get lost easily when jumping through layers of other people's code. I certainly do.
Solid, well tested dependencies that solve hard problems are worthwhile. But dependencies have a cost in debuggability and maintenance, so it's worth using them with care. And often, they aren't worth the time, when compared to writing a dozen lines of code.
But conceding that charitable assumption to the article, I agree with its basic premise: dependencies cost a lot of time in diffuse, non-codey ways.
There are AAA dependencies you pull into every project, but most other dependencies require a good degree of due diligence, evaluation, risk, and their own long-term maintainance.
Its not that it always tips the scales all the way to 'roll your own', but I think the cost of new dependencies is underrated.
So do your upfront work, by all means. It isn't an all or nothing decision.
Dependencies have costs:
- Dependencies break over time. They have a nonzero maintenance cost.
- They impose API boundaries on you that may not fit your existing data structures
- It's harder to change underlying bugs
- They might introduce security issues
Sure, use dependencies. But there's a reasonable position between "never write any code" and "never take on dependencies". Of which NPM is one of the only ecosystems being at one extreme.
Ya got me ... pypi, crates, NPM, gems, cpan I'll use em all! Shhhh.... if my employer finds out they'll fire me.
:-)
Or two jobs ago where two developers who had worked at the company for 10 and 15 years respectively were maintaining a bespoke 15 year old EHR written in PowerBuilder and depended on SQL Server 2003 - in 2016.
Every company thinks they are their own special snowflake where cross cutting concerns can’t be handled by a third party.
I suppose because it was easier than working in the actual problem domain, which they knew little about and didn’t care to learn.
Every single project I go into seems to have a smorgasboard of dependencies, then when I take the time to investigate one of them I find out it's being used incorrectly by at least 50% of the team because they don't even understand how they work at the most basic level. Which is pretty understandable because by the time anyone gets through understanding 10 of the 85, they've probably been kicked off the team for not actually building anything.
People love to say rubbish like "write less code!", as if LoC is the only metric that matters (weren't we past that thought process by the 90s?). Which goes a long way to explaining all the fucking terrible codebases I have to work with where it's impossible to accomplish anything without reading documentation for 8 hours, when it would take 20 minutes to just read even a semi-readable piece of code that implements whatever requirements you need from the dependency.
On a C code project for a large Fortune 100 company a half dozen years ago, I encountered a pesky header include that made no sense. And that header was part of a patch that I really did not want to pick up, so I started digging into it.
Turns out that they had some constant in the code, and the developer just did a grep for that value in the source tree, and that constant already existed in an existing header file, so they just included it.
And that CONSTANT_VALUE_STRING had nothing to do with the technology that the C source was addressing. So some lazy slacker pulled in a random header file that contained the proper constant value for an unrelated technology.
The dependency on that was pure lunacy on so many levels.
And that was an internal dependency, not an external library.
So the lesson here? Not all dangerous dependencies are external.
I keep hoping things like [1] are a joke but I'm starting to suspect they're not.
[1]https://www.npmjs.com/package/is-odd
From my experience with these personal failings, the problem usually comes from the question being phrased in the context like, "before you begin working on this, how long do you think this will this take you to complete?". If there's no opportunity to scope, with requires not insignificant work towards the solution, the estimates will always be wrong. If I understand the actual scope of the problem, which means have the architecture mostly worked out, and have a bit of experience (and luck), my estimates can be pretty close, usually eaten up by that oh-so-seductive feature creep that ruins my work file balance.
FTI: https://basecamp.com/shapeup
Absolutely. As long as it does save you that time over the foreseeable lifetime of the project. Or you are deliberately incurring a technical debt because of some deadline.
On the other hand, saving an afternoon (or even a week), over the next two weeks means very little.
We all too often forget the scope: requirements, developing, testing, to say the least.
My favorite example is NPM. While the author has a point, I tend to rely on the wisdom of the crowd. Sometimes there is a reason why a couple of million developers - in the case of NPM packages - seem to be lazy.
In my experience, we ended up copy/pasting and modifying some code and syncing it with the "superfluous" package. Good intentions, badly executed.
Leftpad was the right itch at the right time and people found better ways to deal with NPM. NPM got better after that, as well as native implementations.
Better cope with NPM than fight it, my 2 cents.
Besides, in reality pulling in and using the dependency takes time as well. There's no real guarantee it's cheaper in terms of developer time.
This is why you time box things. Spend XX hours trying to get a thing working and if you aren't close, you grab a library and move on.
This is sometimes true, but often not. From SFTP libraries to SVG rendering libraries, there have probably been about 3-5 major dependencies of my company's project that I have had to learn and extend or fix bugs in to make them work just in the last year.
And sometimes this means using our own fork that we have to keep maintained.
I'm not saying I would have rather written these particular dependencies from scratch, but they were definitely not cost free. Nor are they all of better quality than what I would have produced had I written them from scratch.
That's the other common refrain - to "defer to the expertise of the crowd".
Don't get me wrong, many 3rd party libraries are of great quality by amazing men and women who I am very thankful for. But certainly not all of them.
There's no magic that says "every third party library is made by an expert with the highest standards".
all I needed was
Taking 197 dependencies means 197 things that need updates several times a year at a minimum. Any of those updates could break my code, introduce a bug, add a vulnerability on top of the ones already in the packages. So it's not like adding more dependencies is magically free.- You should absolutely use community-supported tools to solve your problems.
- You should substitute idiomatic code for libraries.
You have made an argument for the latter that does not detract from the former.
Lots of things can go wrong when writing a file: https://danluu.com/deconstruct-files/
So, if I think something looks like about a day’s work, I’ll actually estimate it at about 3.5 or 4 days. Thus, for a project to qualify as “just an afternoon,” I’d have to naively estimate it at under an hour.
I rarely have time to spare, but I also rarely go over by more than maybe a third.
Your multiplier may vary depending on how horrifying your codebase is. On a side project with good test coverage, my multiplier is only about 2.
Never do I give that raw 10-minute estimate to anybody, because it can be wrong by a factor of 10.
They just won’t have unit tests, and they’ll probably have lots of defects and other technical debt.
(2) if Linus's Law is "given enough eyeballs, all bugs are shallow", then having fewer eyes on a library means more bugs.
(3) Oftentimes you think you can replace a dependency with an afternoon of programming, but it turns out, it's not quite as simple as you think.
(4) Sometimes you only use a small piece of a library to start with, but over time use more and more. If it's your own code, then you're going to be continuously refactoring, updating, rewriting it. If it's a library, then you can just start using the additional pieces as you need.
If someone spent the time and effort building something you need, I don't see a problem with using it. It all depends on what kind of system you're building, what the security and stability guarantees are, and in general what trade-offs you want to make.
In other words, "never" is usually bad advice.
https://youtu.be/D59v74k5flU
Fuck making popup positioning in browsers. That shit is hard to get just right.
The cost of rebuilding MUST be budgeted though. If you don't have this freedom, things are bound to suck one way or another. Then, next best thing, build it as simple as you can, and put effort into making it composable and pluggable. So you retain freedom to swap out components. This is also an investment, and takes some more time and effort.
If you even can't even have that, results are bounded by those restrictions.
To be clear, this is about the lifetime support of code. It's very, very rare that code can be written once and never touched. But that long tail of support eats up time and money, and is almost always discounted in these conversations. I don't even care that Jackson JSON parsing has years of work behind it, when I can hack together a JSON parser in a day. I care that Jackson will continue to improve their offering without any further input, while that's not true of my version.
Of course, you might say, inevitably feature-creep will expand the list of things your parser needs to parse, but that's not a law of physics. Sometimes in certain limited, well-defined projects, it really is true that YAGNI.
[1]: https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of...
I've been on projects where they imported xml-parsers many times bigger than the rest of the whole codebase just to send a well formatted order number.
This is like saying you should roll your own crypto because you only need to do a very limited sub set of crypto operations so why use something like NaCl or Tink.
And YANGI doesn’t have anything against dependencies.
When there are new requirements you do a quick estimate if you should add four new lines to the existing 20 or if it is worth to switch to an external library. 4 new lines to the 20, just add them to the core. But if this is regularly occurring that you have to add things, or requirements affecting this particular little parser that was supposed to be simple and static isn’t, then you should probably change your decision and use the library.
But you do that only then. Because chances are that with your approach you are going to drag along a large generic library that you only use a tiny fraction of. And that also has costs. In particular if your immediate impulse always is to add another library instead of writing things yourself.
By using a third party library you are writing twenty lines less code, so it's cheaper in that aspect.
There are probably libraries that are faster than your twenty lines of un-optimized code, so it's cheaper as far as computing resources are considered too.
The only time it could matter is when you ship the code to the client through the wire (such as a Javascript bundle).
That's the real cost. The size of the code means absolutely nothing.
A json parser can probably be implemented in an afternoon. But a conformant xml parser can take months.
There are some weird things in xml, for example, this is correct xml:
And for external entities, you need an http client in the xml parser, although it is probably better to not support that part of the standard.If you're needs can be solved adequately by strtok then that's a far simpler and more maintainable solution that can be knocked out in an afternoon.
(Some JSON libraries do have option flags, but usually it's about whether, during deserializing into a known type, unknown fields are an error or silently ignored. Or whether C-style comments are an error or considered as whitespace.)
Let’s say I’m scraping a website. I can:
1) complain to the owner that “it’s not JSON”.
2) write a parser for a syntax that has no spec, (it’s not JSON, but it sure looks an awful lot like JSON with unquoted keys.)
or
3) Set ALLOW_UNQUOTED_FIELD_NAMES to true in the Jackson library.
While acceptable, also misleading: JS only does string keys, but unlike JSON it'll convert whatever it's given into strings. Not a problem most of the time, since it'll do the same conversion for both accessing and setting, but good to be aware of if you're doing something like iterating Object.keys()
> Of course, you might say, inevitably feature-creep will expand the list of things your parser needs to parse
If you've done your parser correctly, you'll be able to replace its implementation with the new dependency, with little to no need for extra refactoring in the rest of the codebase.
You don't know what your requirements are. The customers haven't told you yet.
If you pick a library with a straightforward interface, especially one that isn't too opinionated, you can always drop in a custom implementation later on. Frameworks, not so much (but that cuts both ways; the people who will write libraries often love writing frameworks too)
Have good test coverage to catch bugs that may originate in dependencies and subscribe to a third-party service to track vulnerabilities in your dependencies.
It's generally easier in the long run to keep your dependencies up to date. If a package has a new breaking change each week, that's a sign you probably shouldn't be using it for production code.
I can understand max 10 dependencies iterating so quick. But only when they are your own internal dependencies and these should definitely not break the API weekly.
* corrected spelling
Some people just never upgrade until they need to. That's workable, though when you do need to upgrade a package you may be spending the rest of the week working out a cascade of breaking changes.
The solution to that is simple, stop using node.js ;)
Here's some reasoning - if this project was inhoused would we detect and patch it any quicker? Would we have a dev constantly assigned to it that would be pushing out patches to the rest of the team... or is it the sort of software we'd write once and then wait until a compelling reason to invest more into. Whether software is inhouse or outsourced you still retain decision making about how much time to invest in its maintenance.
If it's a bespoke library, no one but you and hackers directly targeting you will test for security vulnerabilities. (Good thing you have a red team... right?) For widely-used libraries, the number of vulnerabilities isn't going to be much different from your own library, but the likelihood that they're found and exploited in your system is quite lower.
So no, in most cases, you would not detect and patch vulnerabilities quicker, because you probably don't see them until it's too late.
> if it works today, could you pin it to that version and wait until there is a compelling reason to update it.
If you pin versions for a long time, eventually there comes a point where you have to update something because of a critical bug or security advisory, and of course since it's a critical bug or advisory, you have to update "right now", "priority 1", "all hands on deck", "the board is involved" and everything. The fix is in version 5.1.2 of the library, but you're stuck at 2.6.5, so now you have to do three major version upgrades (with all the changes to your codebase that entails) before you can even think about upgrading to the version containing the security fix. And that's still an easy case. If the library in question is a framework like Rails or React, version upgrades of that size may be a major undertaking that takes weeks or months to prepare, execute and validate. That's very much not fun when management is pressuring you to close that vulnerability.
I think it's never a good idea to sit on ancient libraries. Put a recurring task in your team backlog to update dependencies on a schedule. It's not going to result in less work spent upgrading, in all likelihood it's more work in terms of raw hours worked compared to the update-on-security-advisory strategy, but it's much more plannable and less stressful. That doesn't mean you have to upgrade to latest-greatest immediately (you always have the freedom to hold off a particular upgrade until the new major version has had some time to mature etc.), but there should be some time reserved on your schedule for doing your updates.
For instance, I have my update-all-lib-deps reminder in my calendar on the 1st of every even month. When it comes up, I put a task in my backlog with a checklist containing every application I have to check, upgrade and deploy. Go 1.15 just came out today, so that's going to be on my desk come October. Great timing, actually, we're going to be one or two point releases into the 1.15 branch at that point, so it's going to be a safe and easy upgrade.
In logic language, you're saying "If X is your core business, don't outsource X".
> Is JSON parsing our core business? No, so why would we ever write -- and thereby commit to supporting for its entire lifetime -- JSON parsing code? All the code you write and support should be directly tied to what you as a business decide are your fundamental value propositions. Everything else you write is just fat waiting to be cut by someone who knows how to write a business case.
The rest of your argument is interpreted as "If X is not your core business, don't in-house X".
These two logical implication statements are not equivalents of each other, but are converses. Casual language often conflates If, Only-If, and If-And-Only-If.
You should spend time implementing your core business implies that you shouldn’t spend time implementing things that aren’t in your core business, otherwise the first statement is pretty useless.
outsource = O
Object = x
x ∉ C ↔ O(x)
If the symbols don't show up:
if-and-only-if x is not in C, O(x)
https://thedailywtf.com/articles/What_Is_Truth_0x3f_
Beside _lifetime support_, working on that core business feature make us _understand_ deeply about the that feature.
I've seen people integrate dependency for their core business. It helped to get started fast, but will create a blockage that required understanding deeper to overcome
... Look, a tiny json parser — Not an afternoon project: https://github.com/rafagafe/tiny-json/blob/master/tiny-json....
And a question about small JSON parsers — didn't see any afternoon projects among the answers:
https://stackoverflow.com/questions/6061172/smallest-less-in...
I suppose a JSON parser was just an example. Made the whole answer sound weird to me though :- ) when the blog is about afternoon-projects and then a reply is about a week(s), could be month(s), long project.
You're making my point for me. This is exactly what I meant by the lifetime of support you're signing up for by writing lines of code. Once you write that code, you're now in the business of supporting that code. Was that a good decision for your business?
$csv = str_getcsv( $input );
But PHP makes everything easy :trollface:
- With a little experience, you know what gets fiddly and what doesn't. Today for instance, I needed a way to remove tags in an SVG document, which looks a lot like HTML tags. I quickly ended up finding that Regex is not the solution (a well known guy on SO wrote an answer that looks like a huge warning sign). I also couldn't enumerate all the corner cases. So I found a lib that does it, along with an SO answer that turns it into a two-liner.
- Dependencies vary in quality. Some are basically like another standard lib. Boost for instance is very well used. The tough ones are where the lib seems to be "finished", where there seem to be few commits recently, but the project was once lively and functional. IIRC libev comes to mind here. And then there are the totally dead projects, where there's a load of issues open and nobody saying anything.
- Try to lock down versions. If you get a thing working with a certain version, there's no reason you need the newest new as soon as it's pushed. You can probably live with doing a scan for updates now and again.
- Your afternoon of programming needs to have a clear end. That hashmap you wrote will very likely spew out issues over the next few days. CSV parser, maybe. Bessel function, that'll work.
For those who are in today's 10000, you might mean this piece of art: https://stackoverflow.com/questions/1732348/regex-match-open...
https://blog.codinghorror.com/regular-expressions-now-you-ha...
Agree ! it irks me a lot that I often see update bots tracking new releases.. it is just begging to be exposed to regressions.
We need to find a happy medium though. Otherwise whenever you actually need to update something (e.g. you need add a new dependency which only handles one of your other dependency if it jumps 20 releases ), you have a huge version gap to cover.
Writing an SVG (or at least XML) parser is a necessary task for writing a filter that doesn’t get stuck due to weirdo issues. That is way more than an afternoon of work! But once you have a parser, dropping tags you don’t want or transforming them somehow is totally an afternoonable task size. So, do use a dependency for SVG parsing, but don’t look for a special “SVG filter all” package. Just do the filtering yourself.
But then, that's not really my core domain, so I'll most likely never touch that piece again anytime soon. A quality package tends to have that covered (with a small risk, too if it's an untrusted source).
The headache is not worth all of the pomp and circumstance for some of these tiny little tools.
- Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. -> It's gonna take longer than an afternoon. Always
- Opportunity cost: Your app needs to DO something, and spending time building dependencies steals quality from your core business proposition, even if it's a small (Haha: see previous point) dependency.
- While there are a ton of crap libraries out there, there are also a lot of good ones, which include hard lessons learned, about even simple tasks. You _could_ rewrite a very simple http client in an afternoon (telnet hostname 80\n GET / HTTP/1.0\n\n), but you'll probably have a ton of glaring flaws just waiting to bite you _badly_.