Sending data between two machines should not be difficult even in 2006, and this post shows that it isn't. However the author seems to want to imply that this would hold for any task, which is of course totally false.
if any given task is trivial (eg. can be done within minutes manually) author's method is the way I go too.
Instead of downloading node.js application with numerous dependencies, I also prefer implementing it in few lines of code.
Some can claim running a 'proper' application (like nodejs/express with JWT authentication) is better. But if your business not solely depend on your solution, trivial stuff wins on the long run IMHO.
I am working on a pet project to build a slot machine. I'm looking at it from a regulatory angle as well. As such, I want to understand the code as much as possible.
I did some initial tests in Node. When I installed express, for routing, it added tens of dependencies and 2.5MB of code files. They call it "minimal".
No thanks. I'll skip the router, for now or write my own.
I don’t think anyone is suggesting downloading a library is better than writing a single short function. It’s a false dilemma.
People who are salty about this subject have dealt with NIH situations where people think they can do better, and are either wrong immediately, or become so when their attention shifts to some other problem.
I have written and maintained internal libraries. Many times it’s the wrong choice, sometimes it’s the only choice. But I’m also more responsive to feedback than the NIH bozos my coworkers typically complain about, which may in part be why I get to hear about their grievances.
There’s an old half-joke, half-theory that we should elect a President who doesn’t want the job, instead of people who do.
You should have the individual contributor who doesn’t want snowflake code write the snowflake code. They’ll keep it no-nonsense.
Too many devs don't understand that dependencies add fragility to a system.
The larger and more complex a system you design, the more likely it is that you end up with something breaking because two dependencies are incompatible or even your code and a dependant break due to updating the dependant.
Further, security is rarely considered. The more code you have, the more likely you are to have a security vulnerability. You have to worry not only about code you maintain, but also the code of your dependencies.
If you can write the functionality you crave rapidly, do it. You'll save yourself (and others) headaches in the future.
Except that happens far more often when you roll your own code internally. For anything of even moderate complexity, only the author will know how it works.
> For anything of even moderate complexity, only the author will know how it works.
I think that's as much true of external dependencies as it is of ones that you wrote for yourself. With the exception that your own solution is probably smaller, much smaller, because it only ever needed to solve your immediate needs and not everything else a third party had to deal with.
So it's quite possible that the internal solution is more approachable when the time comes to debug it.
If this person is bad at knowledge transfer, they end up being the only expert and thereby gaining status for having done so, when objectively this should cost them status instead.
Internal people do this too, but often don’t have the stones to admit they got bored. Instead they victim blame and try to prop up their status as the smartest person in the room.
Personally I’d rather be mad at someone I’ll never meet. It makes meetings a lot more pleasant.
I am sure some developers are very good at their job but the average developer like me write more bugs and security issues than a popular and maintained dependency can have.
Moreover, it's getting quite easy to maintain dependencies today. I can think about packages lock files, Docker containers, semantic versionning, or even Github sending me alerts and pull requests when a security vulnerability happens.
I prefer to add a good dependency and having to deal with it rather than writing broken code and fixing it later. Of course I'm not talking about a 5 lines dependency, but real features.
Dependencies are not always visible, especially transitive ones.
Illustrative anecdote: this Friday I copy-pasted a little snippet of HTML/CSS code from an existing (and shelved) project. I've noticed for some time an icon in it wasn't displaying anymore, but this time I really needed to fix it. Turned out that the glyph icon where from a font used by Bootstrap, which was not available freely anymore. Bootstrap (my direct dependency) had itself that third party dependency that I thought was part of it. When they pulled the plug, my code broke.
I'm more careful about dependencies than I was a few years before. Often, if a peripheral thing can be done in an "ugly" way but allows me to avoid an external dependency, I'll wrote code myself.
Adding anything to a system increases the fragility of the system.
“My point today is that, if we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." - Djikstra
In an age of public bug databases and Stack Overflow, it’s pure ego trip to think that your handwritten solution is going to be better than an OSS solution or stay that way. If you reached for hand-written once, you’ll probably reach for it again and again and the older bits you wrote will be eclipsed by progress on those libraries. I’ve seen it many times before, I’m living it right now.
It’s vendor lock-in with you as the beneficiary. A sports coach would call you a Ball Hog. Let other people play.
You can run a perfectly respectable project with about 10% “special sauce”. That includes the bits that simply nobody has, and the exceptions to my advice above. You can sustain a small number of hand written replacements on par with existing libraries. And the team can absorb the Bus Number effects of that much code.
And “about” means “about”. If you are less, a competitor might steal your lunch. If you’re higher, you should be reflecting on why and asking if you really do still need all this stuff. Maybe that library you looked at before finally got the feature you needed.
> If you can write the functionality you crave rapidly, do it.
Let's take the example of someone who wrote their own Perl ORM and Object Model then.
> You'll save yourself (and others) headaches in the future.
Or it'll hamper development on the system when you leave because nothing works how people expect, it's too complex for people to pick up and modify, and trying to do anything with it causes the entire structure to creak because it's EVERYWHERE.
That's a nice chart to keep in mind, but kinda tangential. Joe's point is that with the right tools, it is sometimes faster to write your own solution than it is to pick one from all the available canned solutions and learn to build & install & configure & use it (and figure out what to do when it doesn't work like it's supposed to).
I think his point is more it's faster to solve the problem you have ("I want to get files from machine X") rather than the problem you imagine you have ("I need an FTP server") - it's just wrapped up in "I solved it myself using Erlang".
Well both Joe's program and an FTP server & client are possible solutions to the problem "I want to get files from machine X."
But I very much agree that the speed of the custom solution can in large part be attributed to the fact that you can reject the cost and complexity of existing solutions like FTP (which set out to solve more than just Joe's immediate problem).
Similarly, I often transfer files by piping them to netcat.
EDIT: the point about tools still stands. For example:
This is very easy to do given that unix is my toolbox. It'd be much more involved to write from scratch in (say) C.
On the other hand, Joe's tool is much more powerful; it doesn't require him to manually interact with the shell on two different hosts, and his program can list files on the remote host and automatically download them under the correct name whereas I'm stuck manually redirecting nc's output to the right file (and cursing myself when I accidentally forget to change the name and overwrite a file I wasn't supposed to fetch). Those features were very easy to add to his program because Erlang is a powerful tool. On unix shell, it's much harder.
Consider adding 'tar' on each side to avoid naming errors. If you squint just right, 'tar' is "the pipe fitting that remembers what things used to be called." :)
Writing from scratch costs more, but provides full scope for creativity.
Reusing increases productivity, at the costs of reduced scope for creativity, along with other hidden costs..
There's a clear-cut "scissor statement" that you can use to divide programmers into two opposite sides. You ask the following question:
"--If you need to invert a 2x2 matrix in your program, do you write down the algebraic formula for the solution by hand or do you call a linear algebra library? (that you would not need otherwise)"
The answer to this question is obvious to most programmers. Also, it is obvious to them that the opposite answer is clearly wrong and absurd. Unfortunately, not everybody gives the same answer!
Okay, you got me. I am in the camp that says calling the library is (in the general case) obviously the better choice.
My (camp's) reasoning:
1. Your handrolled solution is likely worse than the library anyway
2. You're wasting effort reinventing the wheel
3. You risk adding unnecessary opacity by littering the program with dense math
4. Other programmers will assume there is a pertinent reason you handrolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!).
The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded).
For the people who read OP's comment and believe I am (obviously) wrong, what is your camp's way of looking at this?
> I am in the camp that says calling the library is (in the general case)
You misunderstood the question. The question is not about the general case. It is about solving a linear system of two equations and two variables. The solution is not found by "an algorithm", it is a simple formula that fits in one line.
I was tending to go in your direction, but then the formula contains a division and we could be faced with a division by zero error. Which makes me a bit uncertain whether it is so simple. If we are guaranteed the matrix is invertible then doing it by hand is something I would prefer. And likely if it is not then as well since...
The invertibility issue also plagues any library and while it may not blow up by a division by zero we likely still would need to handle that case. This requires to understand the libraries handling of corner cases and how it affects the results.
Sure, if you're solving a 2x2 in ONE place in a piece of software, it makes sense to just perform the computation directly without using a library.
Realistically speaking, though, if you're doing one 2x2 computation, you're probably going to end up doing a bunch of other linear algebra as well. If you don't do linear algebra in your everyday work, the threshold from rolling-your-own solution to grabbing a library is going to come up very quickly.
The Armstrong example is perhaps a bit mis-titled. He's talking about "implementing from scratch" only insofar as he can use the facilities of his platform to perform exactly the task he needs to do rather than bringing in a huge dependency, in this case, an FTP server that does a lot more than the one task at hand.
If you don't know how to do linear algebra, a library is unlikely to help you. Rote stuff like matrix inversion is the easy part, figuring that that is what you need is where things tend to get hairy.
I think also that people can "know" linear algebra but not be skilled it's practical implementation and usage in software. That's partly why computation libraries are among the longest-lived pieces of software in existence.
The question asked about inverting a 2x2 matrix, nothing about solving a linear system of equations. The latter is certainnly one common use of inverting a matrix, sure, but from the problem as given, the use-case could just as well be "report to the end user what the inverse is".
And, now, we know why we have an explosion of libraries that do the job badly OR end up with bloated binaries by including LINPACK when a one-liner would have done the trick.
You triggered a conversation thread that made me understand the ecosystem dynamics a bit better. Thank you.
> Your handrolled solution is likely worse than the library anyway
This assumption led to a lot of O(n^2) left-padding happening over the years, via left-pad from node.js.
Ideally libraries would be well-tested and of high quality, but at this point there may be too many of them to vet.
My own 2 cents is that if it's easier to implement the code from scratch than to evaluate and choose a library, perhaps the invent-it-here choice works best.
My point is not that I'm better at it than some specific library, it's that choosing a library would take more effort than writing some straightforward code.
EDITED TO ADD: In fact getting LAPACK into the build if it's not already there is more of a maintenance imposition than just writing a couple of lines of code for this.
LAPACK is the underlying library behind most linear algebra packages in other languages and furthermore it was just an example. My guess is that you probably don’t have just one line of matrix manipulation ever, in which case using a well established library is a good idea. If not, then sure. Do it yourself. But that’s still more code that you have to maintain now.
That's arguable, because 10 lines of code almost never just stays 10 lines of code, and my point in the first place was more contesting that the dependency is hard to choose, or that there aren't solid proven libraries for this.
The difference between those two things is often blurry from a distance. Business logic is algorithmic too, it's just a question of how specific and complex the algorithm is.
Business logic changes and grows more complex over time, how to invert a matrix doesn't. Trust me, I implement algorithms for a large library for a living, they very rarely change or grow.
I’m aware that you have a definition of “algorithm” that only includes things that don’t change, and I think I know roughly what you mean, but also algorithms are pretty much unavoidable in line of business development. Whether or not they’re well defined, sufficiently general, and have a good name is another thing. In this case you’re right but in a codebase with no documentation where people are doing things like re-implementing JSON parsers from scratch, as I was stuck with at my last job, I’m very wary of “this algorithm is simple” because I don’t trust my coworkers in that case to actually know what they’re talking about. I’m completely sure you do, for what it’s worth.
You're the only one talking about business logic. By definition business logic is not the same as fundamental algorithms or mathematics. Inverting a matrix doesn't change.
A business wants to be able to change its logic to evolve how they do things. You keep trying to conflate these things but they are polar opposites.
What definition are you going on exactly? Nobody has said the phrase “fundamental algorithms” before you in the thread. Yes, I would agree that those are distinct from business logic. Nonetheless the actual definition of the word “algorithm” is not formally or universally agreed upon at all. I’m curious what definition you’re using and where it came from, if only because I now feel confused.
When something doesn't need to change, that's at least an established algorithm. Algorithms like sorting, partitioning, hashing, reduce, map, swap etc. are common and don't change. Mathematics like linear algebra don't change. They are fundamental elements of most nontrivial programs.
Sure, that part I agree with. But the stuff around it, i.e. how those fundamentals are used and composed, also qualify as algorithmic to me. It’s probably a lower-level definition than you’re using, because to me all programs are an algorithm of some kind.
I don't know what you are trying to say at this point. You were arguing that every part of a program changes and expands. It should be pretty clear that that is not true.
1 large, complex dependency (plus the 20 lines of wrapper code to transform your program's internal data to initialise and call the library) almost never just stays as 1 large, complex dependency (plus 20 lines of ...).
If you're truly just doing a 2x2 matrix manipulation once in the codebase, then I concede. I guess I'm blanking on situations where that would be the case though, or where more generalized capabilities wouldn't be useful.
But I was really just wanting to show, via irony, that the structure of the original argument is self-annihilating.
That is, "X almost never stays as just X therefore X is bad" applies to both branches reasonably, and is therefore invalid as an argument in support of one branch.
In fact it was not presented with that level of precision, it merely asked if one had to solve that particular problem and gave no indication about what other algorithms and manipulations the codebase might be using. And furthermore, I maintain that the original example is contrived until proven otherwise.
This is why it's important to add a comment to those 10 lines describing the choice, and urging future contributors to reevaluate the situation.
(Of course, one should always reevaluate the situation when making changes to code, but we all need a nudge sometimes when we're afraid of doing the wrong thing.)
Parent's comment is getting downvoted to all heck, but it is completely, obviously true in the Javascript ecosystem. If you're a competent-to-expert Javascript programmer, you will regularly run into libraries that are not coded to your standards.
And honestly, even throwing those libraries out, it's sometimes not even a question of doing a "better" job than the library. The library is designed for general-use, but my needs aren't always general-use. Sometimes libraries are good, but they're wasting time on stuff that I don't need.
This is the reason why every Unity game on the web takes 4-5 seconds to load, and my custom engine takes on the order of milliseconds to load. It's not because I'm a better programmer, it's not because Unity devs are crap, it's purely because I know exactly what my engine needs to do, and I don't waste time on anything that it doesn't need to do.
I can think of tons of examples where I've started out using an existing library and then realized that getting rid of abstraction made my code faster and easier to debug. Heck, I can think of tons of times where I've privately forked libraries and deleted codepaths or rewritten algorithms to make them more efficient for my use-cases. There are very few JS libraries that I regularly use where I have not at some point needed to care about their internals.
And I am definitely not a crazy, insane, masterful programmer. If I'm occasionally circumventing d3 internals or doing stuff manually, it's not because I'm special. People have this assumption that if something has a bunch of stars on Github, there's no way they could possibly code something more appropriate or efficient, and for a lot of people, that just isn't true.
"Surely this core cryptography library with 21 contributors and led by a superstar Node.JS dev with 9+ million weekly downloads and 1000+ dependants will be a great example to read and learn from..."
Opens source code.
Closes source code.
"Well, looks like I'm going to be a Rust developer from now on."
I had the same experience, but I tried to help. Now my name is in the contributors list, but I still don't trust it. My changes have all been lost to rewrites, and I assume new bugs have replaced the old ones I fixed.
... but I think the majority are in the “I just need shit to work and develop fast” camp, and don’t care if it’s not optimal, so they can finish their current project and move on.
That's because Node and JS are geared towards software engineers new to the field. The ecosystem is geared towards delivering things quickly and cheaply. Like most things, that will work 80% of the time. Most projects are shacks, not skyscrapers.
The problem is, 20% of projects are skyscrapers, and you can't build those out of pine. Sometimes, you can't build them out of standard steel and rear. You actually have to pick the correct raw materials, create a stable and maintainable architecture, adhere to strict regulatory standards, and use advanced tools to create a working edifice.
The software industry's current practices have encouraged a one-size-fits-all mentality, and that just means that it's almost time for another SLDC paradigm shift to swing the pendulum back the other way to achieve balance. It will happen, be patient.
1. If you were Joe Armstrong or at his level, probably not.
2. If you're Joe Armstrong the amount of time you're wasting is negligible, and you don't need to worry about working with an unneeded dependency. I mean I would be wasting a significant amount of time because I'm not Joe Armstrong but there are things I get asked to do that other people would use a library for and I think what, why?
3. Opacity is in the eye of the beholder. If you're working on a project only you and a few people you know can handle it will ever look at, then you are that beholder. If you're working for a big organization as a consultant and your contract runs down in a month, think of the worst guy you have ever worked with that was still a programmer - that guy is the beholder.
4. If you write a comment then they won't make the assumption.
Later: Why the downvote?
Even Later: Upvoted, so I guess someone didn't like or maybe just clicked wrong.
I'm not sure if the people who come after Joe Armstrong to maintain his code are not at his level. Or at least close enough that this particular decision wouldn't be a problem
1. The computation is a handful of simple arithmetic statements, there is almost nothing one can do to optimize those. The library is likely written to handle inversion of any size matrix, that means it adds a bunch of control code to be more general, but also slower for this case.
2. You are wasting effort evaluating libraries. And I wonder what you lose in compile time?
3. You are calling into a black box, and you talk about opacity, a simple comment: /* this inverses the matrix */ will deal with your perceived issue.
4. Other programmers will assume there is a pertinent reason you included yet another dependency. Wonder what it does that 5 lines of code couldn't have done (surprise: nothing!).
>The computation is a handful of simple arithmetic statements, there is almost nothing one can do to optimize those.
I'm not sure that's right in this case. For inverting a matrix, it makes a huge difference whether the determinant is exactly zero or just very close, and (IIRC) floating point math may give the wrong answer. A library will have worked through this issue.
The history of various bugs in various very commonly used libraries, with them being extremely obvious after being discovered, strongly suggests this:
If there is some situation that is somewhat likely to come up and needs special code, but it is not immediately obvious (and, going by OP's evident lack of realizing it, that's some proof to say it is not immediately obvious)? It may not be there. And even if it is the common path, it can still be buggy.
Is is less likely that a library with some actual use out there has these bugs vs. handrolling it, but, if you handroll it, you're writing it for _your_ case, specifically with what you have in mind, whereas the library is more general than that. That's usually upside, but there's also downside, in what may be obvious to your usecase may not be obvious to the library.
The central point of the top comment here, that opinions differ and that there isn't actually an obvious answer even if you think there is one, holds true here.
You have already got floating point inaccuracy on the input, even if the library somehow avoid inaccuracy when computing the determinant, it can't tell the reason for a zero or almost-zero arising. What is the library supposed to do? You will have to deal with the issue of either a very big numbers matrix, or no matrix at all, in any case.
This is a great example of the fallacy of believing that a library automatically solves arbitrary issues.
>You have already got floating point inaccuracy on the input,
Not if they come in as integers that have to be coerced at some point.
>What is the library supposed to do?
Compare ad to bc instead of ad - bc to zero, for one.
>This is a great example of the fallacy of believing that a library automatically solves arbitrary issues.
I didn’t say that and don’t believe it. My point only depends on he library having avoided more domain-specific rookie mistakes than I would have caught in five minutes, when it comes to computational linear algebra. Like the sibling commenter, I don’t think libraries are a panacea, and the answer depends on the specifics of the case.
>Not if they come in as integers that have to be coerced at some point.
In which case you can calculate the determinant exactly with your own roll, but a library will likely force you to convert everything to doubles.
>Compare ad to bc instead of ad - bc to zero, for one.
That is the same thing, a compare instruction is basically just a subtract that doesn't write the output, so you get zero/equality in exactly the same cases. Most compilers will probably turn your code into doing just one subtract and then check the flags, no matter which version you write.
> 1. Your handrolled solution is likely worse than the library anyway
I quite doubt that, though I used to feel the same.
The one feature I need in the library is probably implemented for a specific use case which is undocumented or misrepresented; and use cases outside of that are likely to be egregiously unsupported. Fixing the issues is almost always more work than just implementing X directly and looking up gotchas.
(aside: to understand this first-hand, go pick any JavaScript[0] animation library, and then go add a library that integrates it the core framework you're using (or not)... et voila, enjoy the descent into hell.)
A simple implementation of X that doesn't lock you into patterns you don't need or will one day harm you is worth its wait in gold. Seriously, I'd rather sacrifice DRY code than fuck myself eight months down the road by using a shitty over-engineered pattern that doesn't actually fit my needs.
> 2. You're wasting effort reinventing the wheel
Probably not; it turns out most of the wheel was invented and given to you in the language and runtime. For the article, it's 100% true, BEAM is pretty much an operating system and a great one at that.
We often abstract for the sake of abstraction; I honestly think Elixir/Erlang has helped break me of quite a bit of that. Rust is also nice because it'll make things so very painful for you when you abstract for no reason.
> 3. You risk adding unnecessary opacity by littering the program with dense math
Opaque libraries are the exact same thing but are even harder to understand since the source code doesn't comply with your same coding standards, methodologies, or test suite.
> 4. Other programmers will assume there is a pertinent reason you hand-rolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!).
Documentation is the answer here, it doesn't have much to do with library or DIY.
Why go through the trouble to properly evaluate a solution, decide the best solution was to implement your specific use case (or otherwise), and then not document it? You're just asking to repeat yourself or forget.
On the flip-side, its really easy to assume that library authors did something brilliant when in fact they've done the opposite. Or someone will assume you've chosen this library because you took the time to properly evaluate it and never question it...
Ex.
After eight months of having the library run slow and leak memory some junior engineer pulls you aside and softly says "I have a minor in mathematics, I noticed that library Y is only doing simple matrix multiplication... I can do the same thing with this [shows you four lines of code] and its a lot faster... I just assumed we had to use that library since its in our code base."
Adding libraries before you need them causes more headaches than not because now you've littered a dependency throughout your code which may not be sufficient for future use cases. This goes back to I'd rather have less DRY code than be bound to a shitty inflexible pattern.
> The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded).
I don't totally agree with either of those statements fully but I don't strongly disagree with them either. I'd like to add for me these would be of much high significance:
1. Your future use case are unknown.
2. You do not know if the library will be maintained; so, by using it, you're going to own it anyway. There are no free lunches.
3. You're using only a small, defined subset of some standard (i.e. RFC or specification). The existing solutions provide too much cruft for ...
I used to be in the "implement from scratch" side, but these days my position is a lot more "it depends".
My approach to deciding if using an external library or write the code goes more or less as follows:
a) Limit yourself to a few big libraries that are widely used and well supported.
Especially if you work with a language where it is trivially easy to publish a new library, let's say javascript, you will find that even if you are just an average programmer half of the libraries you have access to have worse code that you would have written (so your point 1. is not necessarily correct)
Also using a library has a number of costs with it: increase in code size, time wasted updating it when needed, learning how to use it, understanding its documentation (if there is any at all), understanding why it doesn't behave how it is supposed to, checking github for bugs not yet fixed, etc. (so the effort spent in your 2. point is not necessarily bigger)
This cost is reduced is the library is a quality one, and also if you use a good amount of it, so my second point is
b) try to avoid a library if you just you need to use little of its functionality
Not only it may be cheaper to re-implement the part that you need, but adding a whole library means that you give charte blanc to other developers to use all the other parts of that library. This may be something that you want to avoid, expecially if the library has varying quality and not all parts are well-written. On the other side, if you know that you will probably need other parts in the future, it may be better to use the library instead.
My third rule is
c) keep in mind your team
If most people in your team already know a library, the cost of using it is greatly reduced. On the other side, if it new for most of them, it may be easier for them to maintain custom code (relates to your point 3.).
Every time I npm install my projects and get the "you have 10 new critical vulnerabilities!" message it makes me a little more cautious about relying on third party code.
I think there's a balance between rolling your own and using a lib, and it boils down to the domain of the problem being solved. If you're really just flipping a 2x2 matrix, the linear algebra lib will have way too much junk you don't need, and learning to use it would be more work than just solving a math problem. On the other hand, if you need a lot of common functionality (like multitouch events on the frontend), then why reinvent the wheel when there are a handful of great libraries for it.
3. Add a comment at the top of the function to the effect of "if this ever ends up doing much more than what the name says, consider replacing it with a library."
You first need to divide the programmers into two bins based on whether or not they even understand the question and could reason out the answer on their own. What world do you live in where programmers know linear algebra and understand what inverting a matrix is?
Sure in your individual case that may be but the scissor statement was presented generally as if one could walk up to any programmer. It only being immediately understood by a smaller subset makes it not very helpful.
I can't think of a similar scissor the majority of programmers both already have a solid understanding of and would be likely to be divided on - but I'd love to be out thought!
I didn’t see it as trying I be helpful in the way you’re thinking, just illustrative.
IMO, the best way to write the code (pulling in a lib vs doing it yourself) is to consider the context in which it’s going to be read, maintained, and used. If that’s by a bunch of applied scientists or linalg experts who can read a formula more easily than debugging and maintaining integrated libraries, then roll it yourself.
In the actual case of just needing to invert a single 2x2 matrix in one spot of the program it takes more to understand what the library is being asked to do, how to format the ask, and what it should be giving back than it does to understand the 3 commented steps to do it. Maybe put a comment that if the needs change in the future it might be worth pulling in <linear algebra library of choice>. On the flipside if you expect the needs to increase in the future might as well do it now.
In the general case look at complexity of the options and then look at context of who has to maintain that complexity don't just assume 3rd party means it'll be easier for others to grok while bug hunting.
The "scissor statement" ignores the very real fact that every individual system must be constructed with different tradeoffs and in a different context each time. You will find that engineers in one environment might include LAPACK, but then the same engineer given a different context would roll their own algorithm.
Unless they're not really an engineer. Engineering is pragmatic, not dogmatic. Your job is to do the good thing for your system, not to apply first principles to everything. Approaching it from a position of dogma is not engineering.
I have a M.Sc in maths. I can derive the algorithm to solve a 2x2 linear system in the reals or the complex numbers, but I’m not sure I know how to do that properly in IEEE floats or doubles.
Having said that, I’m not convinced I could easily judge the quality of a third party solution, find a library that favors accuracy over speed, if I needed one, or whether existing libraries properly document how much overhead calling a routine taking nxn matrices for 2x2 matrices takes, if I needed a fast implementation.
And for how many years after high school do you confidently remember how to invert a matrix? For me personally, that number is smaller than the number of years I’ve been out of high school.
I get a thrill when I get to do a math-class math thing in programming. It’s rare. It’s happened with linear algebra once. That one time was about ten years ago, was very simple, and occupied perhaps an hour total of my time, at most. At this point I’d have to google to be 100% sure I even know what it means to invert a matrix.
... but I remember it being super simple and I’d probably be in the “just write it” camp.
Not even one year, if it is about niche knowledge that require accuracy.
Schools don't teach formulas. That's the misunderstanding of a student that only sees the next exam. Schools teach you where to look when you don't have the answer.
Yeah, but at least I heard about it, a quick refresher should be enough, it’s not an alien concept.
And a good programmer should be able to learn a new domain if they want to excel in the craft, otherwise we’re back to software factories and code monkeys.
It still applies even if you don't know what matrices are.
One kind of programmer will look up what it is and how the algorithm to invert it works, then implement that.
The other kind of programmer will search for libraries implementing these features.
The end result is that the first kind of programmer will know how a lot of things work, while the second kind of programmer will know a lot of libraries.
Dealing with matrices is often taught in university (although there is probably a very high correlation between self-taught programmers and library users).
But this is really another version of the same question: When faced with a task you don't understand, do you Google "How to do X", or do you search "library to do X in Y language".
It's a debt decision, the difficulty of creating the code is relatively small compared to the everlasting cost of maintaining it afterwards.
Also, managing the dependencies whims also is debt.
On the other hand, it's not necessarily about LAPACK specifically or the 2x2 matrix. On embedded systems image size actually matters because you have to distribute the image to the embedded system and in many cases your systems have like 4k of flash. So if LAPACK costs you 2k of flash just to invert a 2x2 matrix and you can come up with a solution for inverting 2x2 matrices easily with 200 bytes of code because you have your Linear Algebra textbook handy, why would you include LAPACK?
In fact even if LAPACK does all linear algebra operations correctly 100% of the time, it also introduces a new paradigm, meaning I have to learn how to do things the LAPACK way which may not be intuitive for the class of problem I have. Or LAPACK may do things correctly for the general case at the cost of CPU time whereas my problem needs to be calculated in real-time, in which case including LAPACK is a net-negative for me.
I think the two schools of programmer divide pretty cleanly into people who assume infinite resources and no CPU time constraints and people who have to make trade-offs between resources and time constraints. Serious embedded programmers will almost never want to just include LAPACK and be done with it unless their problem domain involves a ton of linear algebra. Whereas I've yet to meet a web developer who wouldn't just pull in 400 megabytes of libraries doing god knows what to invert a 2x2 matrix with a function call. There are web developers out there who have to optimize pretty close to what the system is capable of, but most web developers don't really care that much about that because they have infinite computing resources(just spin up more instances!) but they need to release their website "like, yesterday."
So it's not about camps per se. As is everything in engineering, it's about trading a resource you have in abundance for a resource that is scarce. For embedded programmers, the real-time requirements and storage capabilities of the MCU are king and they'll gladly spend all the time in the world rolling their own 2x2 matrix inverter. And for web developers, their development time is king so it's better to just cobble together something working using LAPACK and move on because they can just reserve a bigger, faster server.
What’s the old line about experts answering “it depends” for everything?
I’d give good odds that the person doing a 2x2 matrix didn’t even notice it was linear algebra. I wrote a trivial image rotation, zoom and crop feature a number of years ago and I bet if you looked at that code you’d tell me I fucked up by not using linear algebra. Instead I had a series of geometry diagrams on my whiteboard trying to figure out unit tests for my hand written code. Which the next guy promptly broke but that’s another story for a different day.
If I know it’s linear algebra, it will probably come up again. Now my decision is somewhat informed and also more difficult.
Ultimately the kind which imports a library becomes react developers or django developers or other framework developers, while the kind who learns how to implement things on their own become generic software engineers. So you can see which kind a company hires based on their job ads, if it lists a lot of frameworks and libraries then they expect you to have spent most of your time importing and using libraries. While if the ad just lists domain specific knowledge and some generic coding bits they will mostly have people who likes doing things on their own.
My answer: it depends (how long the project might live? (a one of script vs. 10 years project that has to be maintained) What is the cost of adding a dependency? (deployment, security risks, onboarding curve) Do ai need to care about numeric stability and such? etc.
For example, for a one of script deployed in Pythonista on iOS, I would probably use a library if It is available/installable via pip with a single command, rolling out my own otherwise for 2x2 cases with a single google search unless I need to handle special cases.
I don't want to sound mean, but the simple fact that you forgot to divide by the matrix determinant for me is a great illustration of why using a library might be a good option.
I think for most of us can make this mistake, and reasoning in higher order (i.e. using a function from a tested library) helps avoiding this kind of error.
>This doesn't do any input checking including if the determinant is 0
Which seems to me like it assumes away a big part of the difficulty -- you'd need to follow up with: for this problem domain, what am I doing with the answer? How do I need to handle a singular matrix? Should do I throw an exception, and how should the caller handle it?
Avoid np.linalg.inv if you can do it in another way.. It slows the already slow python even more. Recently replaced np.linalg.inv by an affine transformation inverse (what was really needed) in one of our codebases to get a speedup of 100x.
What about the group that would do one or the other based on their mood? If I felt curious that day I'd definitely do it the first way as it has been forever since I've needed to do any sort of matrix math and it would be interesting to re-visit it. On the other hand if I was tired or in a more down mood I'd probably just use the library as I wouldn't care.
I love this story! The thing is that these statements are easier to come by than the story assumes. You do not need deep learning for natural language processing. It is actually very easy: You just think about something which is clearly obvious, start saying it around, and find that many people think you are completely nuts or you are trolling them. When the percentage of people who think you are nuts approaches 50%, it means that you found a good scissor statement.
> "--If you need to invert a 2x2 matrix in your program, do you write down the algebraic formula for the solution by hand or do you call a linear algebra library? (that you would not need otherwise)"
If you wanted to use equations in your code and I was reviewing it, you're going to have a LONG session explaining the finer points of floating point and how you avoided all the problems that can cause.
I can quote sections of IEEE-754 from memory, and I would use a library. I simply have zero confidence that my algebraic equations would handle massive cancellation and roundoff correctly in all cases.
And these are the WORST class of bugs to introduce into your program--a function that works right 99.9% of the time but fails catastrophically at random times.
I think the show Silicon Valley made two perfect archetypes of this divide:
Gilfoyle: the proudly independent coder who prefers to build everything himself rather than be subject to trusting other's code. Documentation is minimal. Code is intense, efficient, and requires a (too) high level of intelligence and effort just to make sense of. Gravitates towards back-end bare-metal systems code and hardware. Background is self-taught pragmatic hacker. Refuses management roles, preferring to do a department's work himself rather than trust underlings.
"Why would I use a library to perform elliptic curve cryptographic functions I am fully capable of writing myself? Do you think I'm a fucking moron?"
Dinesh: the extremely needy but clever programmer whose code is a massive pile of dependencies and abstractions. Unhindered by pride and happy to trust any workaround or outside source if it means him doing less "work" (he does not view learning about new libraries/codebases as work). Documentation is conversational and verbose, with huge doc blocks above the simplest functions. Code is highly abstracted and spread across many files but with an overall working logic to it - often following various (obscure) engineering standards rather than using personal pragmatism. Loves to communicate, and so has a flair for front-end-facing code and making things pretty and understandable rather than completely efficient. Background is more academic and group project based, with a deep understanding of standards and love of new technologies. In a management role, tries to befriend, integrate, and understand all his employees programming styles - and sends many memos with emoticons.
"Oooh, look, I found this neat library that animates my system font whenever I press the 'r' key! Arrrrrr! Hah. Pirates..."
I think, reading this again, most people would want to be Gilfoyle. However, undoubtedly the one who would most happily and realistically fit into real software teams is Dinesh. Real life Gilfoyle is basically confined to reinventing the wheel in small-scale obscure projects that don't interconnect well with the rest of the app-o-sphere, incapable of scaling beyond a single programmer. Worse: he's incapable of delivering sub-par work in a timely manner using heaps of thrown-together trash dependencies, which is pretty much essential to modern programming. The ability to happily navigate the garbage dump, jump through all the standardization hoops, and communicate well is arguably far more of the modern programming job than real deep understanding, efficiency, and sole dedication to the craft ever will be.
I like to implement everything from scratch (compilers, TCP/IP/arp/rarp stack, databases). But I like to use recommended libraries, not my own code. By implementing everything myself, I have a ready list of questions about how certain problems were handled, and a firmer understanding of where to look when the software misbehaves.
I wouldn't call the article biased, but I don't know many people who can write erlang while I do know many who can write Java, c#, c++
The code and steps needed to compile/bundle/start/maintain for many other languages would be far more than in erlang or other beauties.
Where are the tests? Even in erlang, unit tests are useful. Why would I trust my code?
What about security? You might not want this server to allow nasty things to happen if unintended use takes place.
So even in this particular case, I think it is worth evaluating opensource solutions, their maintenance, their license, and have a proper server in place that will be far easier to maintain and trust its robustness (security and availability)
I think it is fair to assume that Joe, the original author of Erlang, was working on a project that already was using Erlang. Sadly, he is not around anymore so it is not possible to bring clarity into the particulars.
While the author's main point has some merit and is worth discussing, the example chosen is wrong, and the reason it's wrong is the reason you should take a beat before you consider ever rolling your own solution.
So now we've got some simple code to copy files between machines, because we hired someone who was too academic to figure out how to set up a simple FTP server. Here's what comes next:
1. We need to open firewall ports so these two machines can communicate. What are they? Who knows. If it were FTP, the sysadmin would know.
2. We need authentication; we can't just allow any old machines to connect. If we were using FTP, we'd have that built-in. And it would have been audited, so we can have at least baseline confidence in its safety and accuracy.
3. We can't go having the disk overflow. We need something to manage quotas, because even though we trust our users, sometimes software goes rogue. Maybe the default FTP server can't handle this, but some other product can.
4. We need logging so there's an audit trail, in case something goes wrong and we need to track it down.
And on and on and hey look you've reinvented FTP and now no one else on your team knows how it works, it's full of bugs, you've hardcoded a bunch of inefficiencies, there's no documentation, we have to explain what it is and how it works to the operations team, and hey here comes the boss asking us to make sure it works over SSL.
If you think you can do it better, make sure you think about what you're doing.
You came up with a bunch of things to complicate the scenario that do not apply to the author's problem, and those things are precisely the kind of things that make a general solution more complex.
Here, by writing their own tool, Joe could avoid all the complexity that does not apply to his scenario.
I don't think the point of the example was to demonstrate a complete and general solution to the problem of transferring files across computers, but a specific solution that solves the immediate problem in the specific environment Joe was operating in, faster than you'd get it done by trying to apply a more complex (and more general) solution.
Another point I'd like to make (that the original post doesn't) is that they could easily extend this custom solution with additional logic that doesn't exist in any FTP server off the shelf. In fact I've had to reject libraries and programs on similar grounds; they almostkinda do what we need, but then a vital piece of the puzzle is missing and forking the project & implementing our special sauce (and then maintaining it) is going to be much more painful than rolling a custom solution from scratch.
Maybe the two most painful experiences in my career are the customer who changes his mind every six hours (and once a week that happens twice in one day) and when the smartest person in the room is also the biggest fool. They can write the most arcane code that nobody else will want or be able to try to untangle, and then act surprised when people are frustrated with them.
He didn't need an ftp server, he just needed to manually copy some files. Is it really wrong to do that without setting up an FTP server with authentication, disk quotas, logging etc?
right. the other side of that learning curve is that if you implement some X instead of integrating a third party library..even if you didn't before, you now know a lot more about X. enough that you might be able to debug issues with X, or have an informed opinion as to the quality of other implementations of X
Once upon a time there was a systems programmer who wanted to use a readymade 3rd party solution to an specific problem.
He evaluated 3 options, then downloaded, compiled, configured and integrated the best choice into the project over the course of a full day. Its limited database integration configuration necessitated some odd quirks like duplicating a db column, but oh well.
When finally done, integration tests revealed that it didn't actually solve the entire problem. It only solved 90% of the problem. Not only that, it had a significant but elusive bug that only occurred in 5% of situations. He filed a bug report, but the project hadn't received any updates in over a year. Damn.
When it became clear neither of the other two readymade libraries would solve 100% of the problem either, the programmer wrote his own solution. It took 3 days of work but he learned a LOT about exactly how it works, and the result solved 100% of the problem with no major bugs.
All of the quirks disappeared. The solution fit the problem exactly, and remained flexible and maintainable. It eventually served as a solid foundation for future developments which had no readymade solution.
This parable is heavily dependent on the quality of a given ecosystem.
When I bring in a third-party solution for something that might be feasible for me to implement myself, it must 1) be actively maintained, 2) be decoupled from my code at the function-interface boundary, and 3) fully cover my use-case, at least as it's known at the time of introduction.
In the ecosystems I work with, finding all of these qualities is not uncommon.
There’s a questionnaire out there for anyone writing a library and the first couple of questions are what ones did you try, why don’t they work, and why can’t they be fixed. Sounds like your coworker wrote a library the “right” way.
We had our own snowflake wrapper around the saucelabs client. I eventually figured out how to file a PR to do the same. Now I get to use that solution on every other project I ever work on. I have similarly used my own Stack Overflow answer to solve a problem I also had three years prior. And I recently found my SO answer to another question verbatim in our own codebase. Ironically, from a guy who often ignores my advice.
Get as many things out of your head and into the group memory as you can. Things get forgotten otherwise. Speaking of, did you guys ever consider open sourcing his solution?
Imagine you have a project where you need to draw a fairly complex object. You have found a file in a standard format that you can use.
However, there are 10 dependencies, some of which are non-trivial. You spend 4 hours changing versions of libraries, downloading a newer compiler than the one you are using, and never get the thing running.
You could have slapped some code together to read the file, and then draw it yourself. It would have taken a half hour.
However, later on you are tasked with with representing many more objects, which have corner cases you didn't account for. This eats up hours and hours of development. Also, you're then tasked with drawing those objects in many different ways, more and more development has been added.
The two take aways are choose based on what you have to do, and make that choice based on knowing 100% what you have to do. If a customer can't make up their mind, they're hurting development whether they know it or not.
> If you have the right tools it's often quicker to implement something from scratch than going to all the trouble of downloading compiling and installing something that somebody else has written.
Heavily agree. Shameless self plug, this is the philosophy behind Distilled[0]: provide a good-enough, low-abstraction tool so you can build on top of it without it getting in your way or forcing you to memorize configuration options.
Adding unnecessary black-boxes to your code can occasionally come back to bite you. I try not to be religious about this, but I am a little biased towards avoiding the abstractions of the 3rd-party libraries unless they come with a lot of benefits. A big "aha" moment for me was trying to teach interns how to use Grunt/Gulp, and eventually realizing that just teaching them Bash was easier. Then I started noticing other stuff -- for example, that the documentation for our inline documentation library was longer than some of the documentation pages we were generating with it.
There's a principle here that's true for abstractions in general[1], but I find it is especially true for some 3rd-party dependencies. I regularly need to read the source code of 3rd-party dependencies that I use, so I no longer treat them as free.
Obviously this stuff isn't a hard rule. I still use 3rd-party dependencies. But there's a balance here. You should be at least a tiny bit cautious about ready-made solutions.
This is extremely context dependent. For instance, a dev has to consider:
- guess whether or not this is an isolated case, or whether or not this will become core functionality
- a self-assessment of the true difficulty of the problem
- a self-assessment of their own skills and knowledge in the area
- security reasoning
- API access/readabilty for other developers to use this code
- maintainability of new code
I have personally seen personal implementations that lead to bug, after bug, that have already been reasoned about in equivalent libraries.
Often for the simple fact that other devs who have to work on this code, its likely that the abstraction and readability of a third party library is probably greater than the 'quick-and-dirty' implementation.
That’s a good list. I’d throw “facility at evaluating existing solutions” on there, too.
The person who reaches for the “New File” button is often not used to asking if the thing they need already exists. They can end up duplicating or triplicating business logic which results in other engineers confidently declaring that a problem has been fixed when they only fixed the obvious occurrence.
An important reason to try it from scratch first is that it helps your understanding of the problem. That way, you can either confirm that it's better not to take on a dependency, or leverage the experience to find, evaluate, and appreciate the right dependency when one turns out to be desirable. Just like people build their own web frameworks before they settle for an established one. That's a good thing, in my opinion.
146 comments
[ 2.8 ms ] story [ 161 ms ] threadInstead of downloading node.js application with numerous dependencies, I also prefer implementing it in few lines of code.
Some can claim running a 'proper' application (like nodejs/express with JWT authentication) is better. But if your business not solely depend on your solution, trivial stuff wins on the long run IMHO.
Of course security/access-control is debatable...
I did some initial tests in Node. When I installed express, for routing, it added tens of dependencies and 2.5MB of code files. They call it "minimal".
No thanks. I'll skip the router, for now or write my own.
People who are salty about this subject have dealt with NIH situations where people think they can do better, and are either wrong immediately, or become so when their attention shifts to some other problem.
I have written and maintained internal libraries. Many times it’s the wrong choice, sometimes it’s the only choice. But I’m also more responsive to feedback than the NIH bozos my coworkers typically complain about, which may in part be why I get to hear about their grievances.
There’s an old half-joke, half-theory that we should elect a President who doesn’t want the job, instead of people who do.
You should have the individual contributor who doesn’t want snowflake code write the snowflake code. They’ll keep it no-nonsense.
The larger and more complex a system you design, the more likely it is that you end up with something breaking because two dependencies are incompatible or even your code and a dependant break due to updating the dependant.
Further, security is rarely considered. The more code you have, the more likely you are to have a security vulnerability. You have to worry not only about code you maintain, but also the code of your dependencies.
If you can write the functionality you crave rapidly, do it. You'll save yourself (and others) headaches in the future.
I think that's as much true of external dependencies as it is of ones that you wrote for yourself. With the exception that your own solution is probably smaller, much smaller, because it only ever needed to solve your immediate needs and not everything else a third party had to deal with.
So it's quite possible that the internal solution is more approachable when the time comes to debug it.
Personally I’d rather be mad at someone I’ll never meet. It makes meetings a lot more pleasant.
Moreover, it's getting quite easy to maintain dependencies today. I can think about packages lock files, Docker containers, semantic versionning, or even Github sending me alerts and pull requests when a security vulnerability happens.
I prefer to add a good dependency and having to deal with it rather than writing broken code and fixing it later. Of course I'm not talking about a 5 lines dependency, but real features.
Illustrative anecdote: this Friday I copy-pasted a little snippet of HTML/CSS code from an existing (and shelved) project. I've noticed for some time an icon in it wasn't displaying anymore, but this time I really needed to fix it. Turned out that the glyph icon where from a font used by Bootstrap, which was not available freely anymore. Bootstrap (my direct dependency) had itself that third party dependency that I thought was part of it. When they pulled the plug, my code broke.
I'm more careful about dependencies than I was a few years before. Often, if a peripheral thing can be done in an "ugly" way but allows me to avoid an external dependency, I'll wrote code myself.
“My point today is that, if we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger." - Djikstra
In an age of public bug databases and Stack Overflow, it’s pure ego trip to think that your handwritten solution is going to be better than an OSS solution or stay that way. If you reached for hand-written once, you’ll probably reach for it again and again and the older bits you wrote will be eclipsed by progress on those libraries. I’ve seen it many times before, I’m living it right now.
It’s vendor lock-in with you as the beneficiary. A sports coach would call you a Ball Hog. Let other people play.
You can run a perfectly respectable project with about 10% “special sauce”. That includes the bits that simply nobody has, and the exceptions to my advice above. You can sustain a small number of hand written replacements on par with existing libraries. And the team can absorb the Bus Number effects of that much code.
And “about” means “about”. If you are less, a competitor might steal your lunch. If you’re higher, you should be reflecting on why and asking if you really do still need all this stuff. Maybe that library you looked at before finally got the feature you needed.
Let's take the example of someone who wrote their own Perl ORM and Object Model then.
> You'll save yourself (and others) headaches in the future.
Or it'll hamper development on the system when you leave because nothing works how people expect, it's too complex for people to pick up and modify, and trying to do anything with it causes the entire structure to creak because it's EVERYWHERE.
I think his point is more it's faster to solve the problem you have ("I want to get files from machine X") rather than the problem you imagine you have ("I need an FTP server") - it's just wrapped up in "I solved it myself using Erlang".
But I very much agree that the speed of the custom solution can in large part be attributed to the fact that you can reject the cost and complexity of existing solutions like FTP (which set out to solve more than just Joe's immediate problem).
Similarly, I often transfer files by piping them to netcat.
EDIT: the point about tools still stands. For example:
This is very easy to do given that unix is my toolbox. It'd be much more involved to write from scratch in (say) C.On the other hand, Joe's tool is much more powerful; it doesn't require him to manually interact with the shell on two different hosts, and his program can list files on the remote host and automatically download them under the correct name whereas I'm stuck manually redirecting nc's output to the right file (and cursing myself when I accidentally forget to change the name and overwrite a file I wasn't supposed to fetch). Those features were very easy to add to his program because Erlang is a powerful tool. On unix shell, it's much harder.
The bad ones get us stuck with that solution for a very long time. These are the experiences that prompt people to post to threads like this.
"--If you need to invert a 2x2 matrix in your program, do you write down the algebraic formula for the solution by hand or do you call a linear algebra library? (that you would not need otherwise)"
The answer to this question is obvious to most programmers. Also, it is obvious to them that the opposite answer is clearly wrong and absurd. Unfortunately, not everybody gives the same answer!
My (camp's) reasoning:
1. Your handrolled solution is likely worse than the library anyway
2. You're wasting effort reinventing the wheel
3. You risk adding unnecessary opacity by littering the program with dense math
4. Other programmers will assume there is a pertinent reason you handrolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!).
The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded).
For the people who read OP's comment and believe I am (obviously) wrong, what is your camp's way of looking at this?
You misunderstood the question. The question is not about the general case. It is about solving a linear system of two equations and two variables. The solution is not found by "an algorithm", it is a simple formula that fits in one line.
The invertibility issue also plagues any library and while it may not blow up by a division by zero we likely still would need to handle that case. This requires to understand the libraries handling of corner cases and how it affects the results.
Realistically speaking, though, if you're doing one 2x2 computation, you're probably going to end up doing a bunch of other linear algebra as well. If you don't do linear algebra in your everyday work, the threshold from rolling-your-own solution to grabbing a library is going to come up very quickly.
The Armstrong example is perhaps a bit mis-titled. He's talking about "implementing from scratch" only insofar as he can use the facilities of his platform to perform exactly the task he needs to do rather than bringing in a huge dependency, in this case, an FTP server that does a lot more than the one task at hand.
The question asked about inverting a 2x2 matrix, nothing about solving a linear system of equations. The latter is certainnly one common use of inverting a matrix, sure, but from the problem as given, the use-case could just as well be "report to the end user what the inverse is".
You triggered a conversation thread that made me understand the ecosystem dynamics a bit better. Thank you.
This assumption led to a lot of O(n^2) left-padding happening over the years, via left-pad from node.js.
Ideally libraries would be well-tested and of high quality, but at this point there may be too many of them to vet.
My own 2 cents is that if it's easier to implement the code from scratch than to evaluate and choose a library, perhaps the invent-it-here choice works best.
EDITED TO ADD: In fact getting LAPACK into the build if it's not already there is more of a maintenance imposition than just writing a couple of lines of code for this.
10 lines of code is easier to maintain than a whole dependency.
This is only true when you implement business logic, not when you implement algorithms.
A business wants to be able to change its logic to evolve how they do things. You keep trying to conflate these things but they are polar opposites.
1 large, complex dependency (plus the 20 lines of wrapper code to transform your program's internal data to initialise and call the library) almost never just stays as 1 large, complex dependency (plus 20 lines of ...).
But I was really just wanting to show, via irony, that the structure of the original argument is self-annihilating.
That is, "X almost never stays as just X therefore X is bad" applies to both branches reasonably, and is therefore invalid as an argument in support of one branch.
That is the precise premise of the hypothetical.
(Of course, one should always reevaluate the situation when making changes to code, but we all need a nudge sometimes when we're afraid of doing the wrong thing.)
And honestly, even throwing those libraries out, it's sometimes not even a question of doing a "better" job than the library. The library is designed for general-use, but my needs aren't always general-use. Sometimes libraries are good, but they're wasting time on stuff that I don't need.
This is the reason why every Unity game on the web takes 4-5 seconds to load, and my custom engine takes on the order of milliseconds to load. It's not because I'm a better programmer, it's not because Unity devs are crap, it's purely because I know exactly what my engine needs to do, and I don't waste time on anything that it doesn't need to do.
I can think of tons of examples where I've started out using an existing library and then realized that getting rid of abstraction made my code faster and easier to debug. Heck, I can think of tons of times where I've privately forked libraries and deleted codepaths or rewritten algorithms to make them more efficient for my use-cases. There are very few JS libraries that I regularly use where I have not at some point needed to care about their internals.
And I am definitely not a crazy, insane, masterful programmer. If I'm occasionally circumventing d3 internals or doing stuff manually, it's not because I'm special. People have this assumption that if something has a bunch of stars on Github, there's no way they could possibly code something more appropriate or efficient, and for a lot of people, that just isn't true.
Opens source code.
Closes source code.
"Well, looks like I'm going to be a Rust developer from now on."
... but I think the majority are in the “I just need shit to work and develop fast” camp, and don’t care if it’s not optimal, so they can finish their current project and move on.
The problem is, 20% of projects are skyscrapers, and you can't build those out of pine. Sometimes, you can't build them out of standard steel and rear. You actually have to pick the correct raw materials, create a stable and maintainable architecture, adhere to strict regulatory standards, and use advanced tools to create a working edifice.
The software industry's current practices have encouraged a one-size-fits-all mentality, and that just means that it's almost time for another SLDC paradigm shift to swing the pendulum back the other way to achieve balance. It will happen, be patient.
2. If you're Joe Armstrong the amount of time you're wasting is negligible, and you don't need to worry about working with an unneeded dependency. I mean I would be wasting a significant amount of time because I'm not Joe Armstrong but there are things I get asked to do that other people would use a library for and I think what, why?
3. Opacity is in the eye of the beholder. If you're working on a project only you and a few people you know can handle it will ever look at, then you are that beholder. If you're working for a big organization as a consultant and your contract runs down in a month, think of the worst guy you have ever worked with that was still a programmer - that guy is the beholder.
4. If you write a comment then they won't make the assumption.
Later: Why the downvote?
Even Later: Upvoted, so I guess someone didn't like or maybe just clicked wrong.
But 99% of us are not at his level. And more importantly, 99% of the people who come after to maintain this code are not either.
2. You are wasting effort evaluating libraries. And I wonder what you lose in compile time?
3. You are calling into a black box, and you talk about opacity, a simple comment: /* this inverses the matrix */ will deal with your perceived issue.
4. Other programmers will assume there is a pertinent reason you included yet another dependency. Wonder what it does that 5 lines of code couldn't have done (surprise: nothing!).
I'm not sure that's right in this case. For inverting a matrix, it makes a huge difference whether the determinant is exactly zero or just very close, and (IIRC) floating point math may give the wrong answer. A library will have worked through this issue.
If there is some situation that is somewhat likely to come up and needs special code, but it is not immediately obvious (and, going by OP's evident lack of realizing it, that's some proof to say it is not immediately obvious)? It may not be there. And even if it is the common path, it can still be buggy.
Is is less likely that a library with some actual use out there has these bugs vs. handrolling it, but, if you handroll it, you're writing it for _your_ case, specifically with what you have in mind, whereas the library is more general than that. That's usually upside, but there's also downside, in what may be obvious to your usecase may not be obvious to the library.
The central point of the top comment here, that opinions differ and that there isn't actually an obvious answer even if you think there is one, holds true here.
This is a great example of the fallacy of believing that a library automatically solves arbitrary issues.
Not if they come in as integers that have to be coerced at some point.
>What is the library supposed to do?
Compare ad to bc instead of ad - bc to zero, for one.
>This is a great example of the fallacy of believing that a library automatically solves arbitrary issues.
I didn’t say that and don’t believe it. My point only depends on he library having avoided more domain-specific rookie mistakes than I would have caught in five minutes, when it comes to computational linear algebra. Like the sibling commenter, I don’t think libraries are a panacea, and the answer depends on the specifics of the case.
In which case you can calculate the determinant exactly with your own roll, but a library will likely force you to convert everything to doubles.
>Compare ad to bc instead of ad - bc to zero, for one.
That is the same thing, a compare instruction is basically just a subtract that doesn't write the output, so you get zero/equality in exactly the same cases. Most compilers will probably turn your code into doing just one subtract and then check the flags, no matter which version you write.
I quite doubt that, though I used to feel the same.
The one feature I need in the library is probably implemented for a specific use case which is undocumented or misrepresented; and use cases outside of that are likely to be egregiously unsupported. Fixing the issues is almost always more work than just implementing X directly and looking up gotchas.
(aside: to understand this first-hand, go pick any JavaScript[0] animation library, and then go add a library that integrates it the core framework you're using (or not)... et voila, enjoy the descent into hell.)
A simple implementation of X that doesn't lock you into patterns you don't need or will one day harm you is worth its wait in gold. Seriously, I'd rather sacrifice DRY code than fuck myself eight months down the road by using a shitty over-engineered pattern that doesn't actually fit my needs.
> 2. You're wasting effort reinventing the wheel
Probably not; it turns out most of the wheel was invented and given to you in the language and runtime. For the article, it's 100% true, BEAM is pretty much an operating system and a great one at that.
We often abstract for the sake of abstraction; I honestly think Elixir/Erlang has helped break me of quite a bit of that. Rust is also nice because it'll make things so very painful for you when you abstract for no reason.
> 3. You risk adding unnecessary opacity by littering the program with dense math
Opaque libraries are the exact same thing but are even harder to understand since the source code doesn't comply with your same coding standards, methodologies, or test suite.
> 4. Other programmers will assume there is a pertinent reason you hand-rolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!).
Documentation is the answer here, it doesn't have much to do with library or DIY.
Why go through the trouble to properly evaluate a solution, decide the best solution was to implement your specific use case (or otherwise), and then not document it? You're just asking to repeat yourself or forget.
On the flip-side, its really easy to assume that library authors did something brilliant when in fact they've done the opposite. Or someone will assume you've chosen this library because you took the time to properly evaluate it and never question it...
Ex.
After eight months of having the library run slow and leak memory some junior engineer pulls you aside and softly says "I have a minor in mathematics, I noticed that library Y is only doing simple matrix multiplication... I can do the same thing with this [shows you four lines of code] and its a lot faster... I just assumed we had to use that library since its in our code base."
Adding libraries before you need them causes more headaches than not because now you've littered a dependency throughout your code which may not be sufficient for future use cases. This goes back to I'd rather have less DRY code than be bound to a shitty inflexible pattern.
> The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded).
I don't totally agree with either of those statements fully but I don't strongly disagree with them either. I'd like to add for me these would be of much high significance:
1. Your future use case are unknown.
2. You do not know if the library will be maintained; so, by using it, you're going to own it anyway. There are no free lunches.
3. You're using only a small, defined subset of some standard (i.e. RFC or specification). The existing solutions provide too much cruft for ...
My approach to deciding if using an external library or write the code goes more or less as follows:
a) Limit yourself to a few big libraries that are widely used and well supported.
Especially if you work with a language where it is trivially easy to publish a new library, let's say javascript, you will find that even if you are just an average programmer half of the libraries you have access to have worse code that you would have written (so your point 1. is not necessarily correct)
Also using a library has a number of costs with it: increase in code size, time wasted updating it when needed, learning how to use it, understanding its documentation (if there is any at all), understanding why it doesn't behave how it is supposed to, checking github for bugs not yet fixed, etc. (so the effort spent in your 2. point is not necessarily bigger)
This cost is reduced is the library is a quality one, and also if you use a good amount of it, so my second point is
b) try to avoid a library if you just you need to use little of its functionality
Not only it may be cheaper to re-implement the part that you need, but adding a whole library means that you give charte blanc to other developers to use all the other parts of that library. This may be something that you want to avoid, expecially if the library has varying quality and not all parts are well-written. On the other side, if you know that you will probably need other parts in the future, it may be better to use the library instead.
My third rule is
c) keep in mind your team
If most people in your team already know a library, the cost of using it is greatly reduced. On the other side, if it new for most of them, it may be easier for them to maintain custom code (relates to your point 3.).
Do it by hand. No dense math involved. One line of code (or so, depending on coding style).
If you want to implement or call LINPACK to do a 2x2, you've got other problems.
I think there's a balance between rolling your own and using a lib, and it boils down to the domain of the problem being solved. If you're really just flipping a 2x2 matrix, the linear algebra lib will have way too much junk you don't need, and learning to use it would be more work than just solving a math problem. On the other hand, if you need a lot of common functionality (like multitouch events on the frontend), then why reinvent the wheel when there are a handful of great libraries for it.
1. Add an "invert2x2Matrix(theMatrix)" function.
2. Implement the function myself (no library).
3. Add a comment at the top of the function to the effect of "if this ever ends up doing much more than what the name says, consider replacing it with a library."
I can't think of a similar scissor the majority of programmers both already have a solid understanding of and would be likely to be divided on - but I'd love to be out thought!
IMO, the best way to write the code (pulling in a lib vs doing it yourself) is to consider the context in which it’s going to be read, maintained, and used. If that’s by a bunch of applied scientists or linalg experts who can read a formula more easily than debugging and maintaining integrated libraries, then roll it yourself.
In the general case look at complexity of the options and then look at context of who has to maintain that complexity don't just assume 3rd party means it'll be easier for others to grok while bug hunting.
Unless they're not really an engineer. Engineering is pragmatic, not dogmatic. Your job is to do the good thing for your system, not to apply first principles to everything. Approaching it from a position of dogma is not engineering.
In particular, I wonder whether ill-conditioned edge cases (https://en.wikipedia.org/wiki/Condition_number) might need special-casing.
Having said that, I’m not convinced I could easily judge the quality of a third party solution, find a library that favors accuracy over speed, if I needed one, or whether existing libraries properly document how much overhead calling a routine taking nxn matrices for 2x2 matrices takes, if I needed a fast implementation.
... but I remember it being super simple and I’d probably be in the “just write it” camp.
Schools don't teach formulas. That's the misunderstanding of a student that only sees the next exam. Schools teach you where to look when you don't have the answer.
And a good programmer should be able to learn a new domain if they want to excel in the craft, otherwise we’re back to software factories and code monkeys.
One kind of programmer will look up what it is and how the algorithm to invert it works, then implement that.
The other kind of programmer will search for libraries implementing these features.
The end result is that the first kind of programmer will know how a lot of things work, while the second kind of programmer will know a lot of libraries.
But this is really another version of the same question: When faced with a task you don't understand, do you Google "How to do X", or do you search "library to do X in Y language".
In fact even if LAPACK does all linear algebra operations correctly 100% of the time, it also introduces a new paradigm, meaning I have to learn how to do things the LAPACK way which may not be intuitive for the class of problem I have. Or LAPACK may do things correctly for the general case at the cost of CPU time whereas my problem needs to be calculated in real-time, in which case including LAPACK is a net-negative for me.
I think the two schools of programmer divide pretty cleanly into people who assume infinite resources and no CPU time constraints and people who have to make trade-offs between resources and time constraints. Serious embedded programmers will almost never want to just include LAPACK and be done with it unless their problem domain involves a ton of linear algebra. Whereas I've yet to meet a web developer who wouldn't just pull in 400 megabytes of libraries doing god knows what to invert a 2x2 matrix with a function call. There are web developers out there who have to optimize pretty close to what the system is capable of, but most web developers don't really care that much about that because they have infinite computing resources(just spin up more instances!) but they need to release their website "like, yesterday."
So it's not about camps per se. As is everything in engineering, it's about trading a resource you have in abundance for a resource that is scarce. For embedded programmers, the real-time requirements and storage capabilities of the MCU are king and they'll gladly spend all the time in the world rolling their own 2x2 matrix inverter. And for web developers, their development time is king so it's better to just cobble together something working using LAPACK and move on because they can just reserve a bigger, faster server.
I’d give good odds that the person doing a 2x2 matrix didn’t even notice it was linear algebra. I wrote a trivial image rotation, zoom and crop feature a number of years ago and I bet if you looked at that code you’d tell me I fucked up by not using linear algebra. Instead I had a series of geometry diagrams on my whiteboard trying to figure out unit tests for my hand written code. Which the next guy promptly broke but that’s another story for a different day.
If I know it’s linear algebra, it will probably come up again. Now my decision is somewhat informed and also more difficult.
"I might have to change it to use a THREE by THREE matrix!!"
So, how sure are you that the formulas you used handle massive cancellation and roundoff correctly?
But things can depend on stuff not mentioned in your questions.
For example, for a one of script deployed in Pythonista on iOS, I would probably use a library if It is available/installable via pip with a single command, rolling out my own otherwise for 2x2 cases with a single google search unless I need to handle special cases.
Don't most people do this?
I think for most of us can make this mistake, and reasoning in higher order (i.e. using a function from a tested library) helps avoiding this kind of error.
Which seems to me like it assumes away a big part of the difficulty -- you'd need to follow up with: for this problem domain, what am I doing with the answer? How do I need to handle a singular matrix? Should do I throw an exception, and how should the caller handle it?
If you wanted to use equations in your code and I was reviewing it, you're going to have a LONG session explaining the finer points of floating point and how you avoided all the problems that can cause.
I can quote sections of IEEE-754 from memory, and I would use a library. I simply have zero confidence that my algebraic equations would handle massive cancellation and roundoff correctly in all cases.
And these are the WORST class of bugs to introduce into your program--a function that works right 99.9% of the time but fails catastrophically at random times.
I think the show Silicon Valley made two perfect archetypes of this divide:
Gilfoyle: the proudly independent coder who prefers to build everything himself rather than be subject to trusting other's code. Documentation is minimal. Code is intense, efficient, and requires a (too) high level of intelligence and effort just to make sense of. Gravitates towards back-end bare-metal systems code and hardware. Background is self-taught pragmatic hacker. Refuses management roles, preferring to do a department's work himself rather than trust underlings.
"Why would I use a library to perform elliptic curve cryptographic functions I am fully capable of writing myself? Do you think I'm a fucking moron?"
Dinesh: the extremely needy but clever programmer whose code is a massive pile of dependencies and abstractions. Unhindered by pride and happy to trust any workaround or outside source if it means him doing less "work" (he does not view learning about new libraries/codebases as work). Documentation is conversational and verbose, with huge doc blocks above the simplest functions. Code is highly abstracted and spread across many files but with an overall working logic to it - often following various (obscure) engineering standards rather than using personal pragmatism. Loves to communicate, and so has a flair for front-end-facing code and making things pretty and understandable rather than completely efficient. Background is more academic and group project based, with a deep understanding of standards and love of new technologies. In a management role, tries to befriend, integrate, and understand all his employees programming styles - and sends many memos with emoticons.
"Oooh, look, I found this neat library that animates my system font whenever I press the 'r' key! Arrrrrr! Hah. Pirates..."
(May our industry RIP)
I like to implement everything from scratch (compilers, TCP/IP/arp/rarp stack, databases). But I like to use recommended libraries, not my own code. By implementing everything myself, I have a ready list of questions about how certain problems were handled, and a firmer understanding of where to look when the software misbehaves.
The code and steps needed to compile/bundle/start/maintain for many other languages would be far more than in erlang or other beauties.
Where are the tests? Even in erlang, unit tests are useful. Why would I trust my code? What about security? You might not want this server to allow nasty things to happen if unintended use takes place.
So even in this particular case, I think it is worth evaluating opensource solutions, their maintenance, their license, and have a proper server in place that will be far easier to maintain and trust its robustness (security and availability)
> With enough eyes, all bugs become shallow.
So now we've got some simple code to copy files between machines, because we hired someone who was too academic to figure out how to set up a simple FTP server. Here's what comes next:
1. We need to open firewall ports so these two machines can communicate. What are they? Who knows. If it were FTP, the sysadmin would know.
2. We need authentication; we can't just allow any old machines to connect. If we were using FTP, we'd have that built-in. And it would have been audited, so we can have at least baseline confidence in its safety and accuracy.
3. We can't go having the disk overflow. We need something to manage quotas, because even though we trust our users, sometimes software goes rogue. Maybe the default FTP server can't handle this, but some other product can.
4. We need logging so there's an audit trail, in case something goes wrong and we need to track it down.
And on and on and hey look you've reinvented FTP and now no one else on your team knows how it works, it's full of bugs, you've hardcoded a bunch of inefficiencies, there's no documentation, we have to explain what it is and how it works to the operations team, and hey here comes the boss asking us to make sure it works over SSL.
If you think you can do it better, make sure you think about what you're doing.
You came up with a bunch of things to complicate the scenario that do not apply to the author's problem, and those things are precisely the kind of things that make a general solution more complex.
Here, by writing their own tool, Joe could avoid all the complexity that does not apply to his scenario.
I don't think the point of the example was to demonstrate a complete and general solution to the problem of transferring files across computers, but a specific solution that solves the immediate problem in the specific environment Joe was operating in, faster than you'd get it done by trying to apply a more complex (and more general) solution.
Another point I'd like to make (that the original post doesn't) is that they could easily extend this custom solution with additional logic that doesn't exist in any FTP server off the shelf. In fact I've had to reject libraries and programs on similar grounds; they almost kinda do what we need, but then a vital piece of the puzzle is missing and forking the project & implementing our special sauce (and then maintaining it) is going to be much more painful than rolling a custom solution from scratch.
He didn't need an ftp server, he just needed to manually copy some files. Is it really wrong to do that without setting up an FTP server with authentication, disk quotas, logging etc?
It's that learning curve. . .
He evaluated 3 options, then downloaded, compiled, configured and integrated the best choice into the project over the course of a full day. Its limited database integration configuration necessitated some odd quirks like duplicating a db column, but oh well.
When finally done, integration tests revealed that it didn't actually solve the entire problem. It only solved 90% of the problem. Not only that, it had a significant but elusive bug that only occurred in 5% of situations. He filed a bug report, but the project hadn't received any updates in over a year. Damn.
When it became clear neither of the other two readymade libraries would solve 100% of the problem either, the programmer wrote his own solution. It took 3 days of work but he learned a LOT about exactly how it works, and the result solved 100% of the problem with no major bugs.
All of the quirks disappeared. The solution fit the problem exactly, and remained flexible and maintainable. It eventually served as a solid foundation for future developments which had no readymade solution.
When I bring in a third-party solution for something that might be feasible for me to implement myself, it must 1) be actively maintained, 2) be decoupled from my code at the function-interface boundary, and 3) fully cover my use-case, at least as it's known at the time of introduction.
In the ecosystems I work with, finding all of these qualities is not uncommon.
We had our own snowflake wrapper around the saucelabs client. I eventually figured out how to file a PR to do the same. Now I get to use that solution on every other project I ever work on. I have similarly used my own Stack Overflow answer to solve a problem I also had three years prior. And I recently found my SO answer to another question verbatim in our own codebase. Ironically, from a guy who often ignores my advice.
Get as many things out of your head and into the group memory as you can. Things get forgotten otherwise. Speaking of, did you guys ever consider open sourcing his solution?
However, there are 10 dependencies, some of which are non-trivial. You spend 4 hours changing versions of libraries, downloading a newer compiler than the one you are using, and never get the thing running.
You could have slapped some code together to read the file, and then draw it yourself. It would have taken a half hour.
However, later on you are tasked with with representing many more objects, which have corner cases you didn't account for. This eats up hours and hours of development. Also, you're then tasked with drawing those objects in many different ways, more and more development has been added.
The two take aways are choose based on what you have to do, and make that choice based on knowing 100% what you have to do. If a customer can't make up their mind, they're hurting development whether they know it or not.
BTW, the objects were GIS shapefiles.
The easy solution was to just use Conda.
Heavily agree. Shameless self plug, this is the philosophy behind Distilled[0]: provide a good-enough, low-abstraction tool so you can build on top of it without it getting in your way or forcing you to memorize configuration options.
Adding unnecessary black-boxes to your code can occasionally come back to bite you. I try not to be religious about this, but I am a little biased towards avoiding the abstractions of the 3rd-party libraries unless they come with a lot of benefits. A big "aha" moment for me was trying to teach interns how to use Grunt/Gulp, and eventually realizing that just teaching them Bash was easier. Then I started noticing other stuff -- for example, that the documentation for our inline documentation library was longer than some of the documentation pages we were generating with it.
There's a principle here that's true for abstractions in general[1], but I find it is especially true for some 3rd-party dependencies. I regularly need to read the source code of 3rd-party dependencies that I use, so I no longer treat them as free.
Obviously this stuff isn't a hard rule. I still use 3rd-party dependencies. But there's a balance here. You should be at least a tiny bit cautious about ready-made solutions.
[0]: https://distilledjs.com
[1]: https://peertube.danshumway.com/videos/watch/2daaee22-4d92-4...
- guess whether or not this is an isolated case, or whether or not this will become core functionality
- a self-assessment of the true difficulty of the problem
- a self-assessment of their own skills and knowledge in the area
- security reasoning
- API access/readabilty for other developers to use this code
- maintainability of new code
I have personally seen personal implementations that lead to bug, after bug, that have already been reasoned about in equivalent libraries.
Often for the simple fact that other devs who have to work on this code, its likely that the abstraction and readability of a third party library is probably greater than the 'quick-and-dirty' implementation.
The person who reaches for the “New File” button is often not used to asking if the thing they need already exists. They can end up duplicating or triplicating business logic which results in other engineers confidently declaring that a problem has been fixed when they only fixed the obvious occurrence.