HN Explorer

Ask HN: How do you work with Dependabot?

43 points by dynamite-ready ↗ HN
I like the idea of Dependabot. A to that actively tracks down dependency updates can be useful. Where I work, we have a daily CI job that creates a PR for each new dependency and runs a build in both our UI (JavaScript) and API (Python) projects. If the build passes, "Happy Days", we can merge the PR, and the app is all the more secure and effective for it.

What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something.

Dependency 'spam' is also a very real issue - https://news.ycombinator.com/item?id=27929596 - the daily deluge of often insignificant updates is a trudge to deal with, especially when coupled with the risk of these sly runtime errors.

Dependabot is a great idea, but no-one appears to have anything bad, or practical, to say against it. But it does clearly have flaws.

I don't think I'd want to switch the bot off, but I would be interested in hearing how other people get on with the tool.

Thanks. :]

45 comments

[ 3.8 ms ] story [ 102 ms ] thread
Raed667
At a previous job, we loved the idea of depandabot but in practice it didn't match the way we work or review PRs. And as you said, just because a test passed doesn't mean that the update is 100% safe.

So instead, we identified our critical dependencies, then included dependencies update task to the list of tech-debt tasks we handle every week manually.

dynamite-ready
Yes. This is exactly what I would prefer to do. My current plan is to round up all the PR requests, at the start of each new sprint (every two weeks), and make it the first development task in the sprint.

The problem though, is that so far, I can't point to any literature online, to support this idea.

I would still like to keep Dependabot, because the diagnostic step it performs is useful. But introducing new dependency upgrades daily, even minor upgrades, seems like a recipe for trouble. Minor upgrades are just as likely to introduce a vulnerability, as they are to patch them, after all.

Raed667
We ended up developing a (naive) internal tool that allowed to see how many versions (and days) behind each dependency is, sorted by how much we care about each package.

This gave us a quick dashboard to checkout before running `yarn upgrade-interactive --latest`

dignati
> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something.

Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

dynamite-ready
That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test.

On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually raise.

Which is why I don't want to throw the baby out with the bathwater, as one or two people have suggested.

But I can say that I think that the reality of working with Dependabot, is not very well reflected in popular online articles.

damienwebdev
> Python Django specific sub dependency, that broke the admin interface, which obviously, we don't explicitly test.

There's your problem.

cr3ative
> the admin interface, which obviously, we don't explicitly test

This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.

dynamite-ready
You routinely write unit tests for dependent packages in your apps?
4ndrewl
But once you've imported that package it's part of _your_ codebase.
KronisLV
While this is true, many will probably disagree, just because they don't want to consider the maintenance burden that external dependencies will introduce.

So between choosing to write everything themselves (and getting nothing done), writing tests against dependencies (and getting little done due to the overhead), or claiming that external dependencies should have tests of their own, many will pick the latter.

Then again, in a world where create-react-app results in 180 MB of dependencies and about 1500 modules (probably different numbers now, using some older ones from my blog post), auditing security is an uphill battle, not even talking about actual testing.

The situation in the back end development, isn't that much better either, to be honest, because once you look into the complexity of any framework like Spring, Laravel, Django, Rails etc., it becomes apparent that creating a fully featured framework like that is a huge undertaking.

That said, you should at least test the bits where the external dependency is integrated with your codebase.

manfre
This is not testing some other package. It's testing functionality provided by your app that happens to rely upon a 3rd party package. The 3rd party package has its tests, but doesn't know or care about the specific integration environment of your site.

With regards to writing tests of the Django admin, you need to do it if your site is customizing and/or depending upon it.

numbsafari
No. This is what integration tests are for.

These kinds of failures are super annoying, but it’s the sort of thing that e2e and canary releases should help you get some coverage on.

Despite the marketing, unit tests are pretty fragile, especially in code with lots of deps (another good reason to limit deps)

madeofpalk
Yes, we routinely write tests that interact with dependencies to make sure our application works.
Narretz
Was it an update to the test runner or test specific packages that broke the test runner? I would ignore infrastructure/testability/tooling packages in dependabot and do them manually to prevent these errors.
dignati
> one that broke the test runner, and ignored a number of tests

That's unfortunate! For the project I'm working on, we've "solved" that by showing the number of test and the difference to the number of tests that ran on main.

FWIW, at previous jobs, upgrading Java dependencies was a major pain because they were all outdated and the latest versions introduced too many breaking changes for us. At my current job, we pretty much instantly merge all PRs from dependabot because we trust our CI. Upgrades rarely introduce problems and if they do, they are easy to fix.

viraptor
Most test runners have an option (or can be easily modified) to fail when 0 or less than X tests have been run. You should use it for situations like this.
lucasyvas
But what if there's also a bug where that feature doesn't work :)
viraptor
What if there's a bug where your CI ignores all steps? There's always some situation possible and we can go all the way to nasal demons. You have to accept the risk at some point.
[deleted]
(comment deleted)
blibble
I ignore everything it says, for me it's just noise
eurasiantiger
Someone knowledgeable of the codebase reviews changelogs of individual packages. They merge all simple cases and flag any breaking changes and anything that affects features used in the codebase. These can then be tested/fixed by others not so deeply familiar with the codebase, which fuels knowledge transfer.
cpswan
My main gripe is that Dependabot can end up raising multiple PRs for the same dependency bump in the same repo (especially with Dockerfiles). I really wish I could tell it to do rollups e.g. `@dependabot rollup #1234 #1235 #1236` or something like that.

To save having to do multiple rounds of merge PR, rebase next PR, wait for CI... I end up doing my own rollup PRs by merging the various Dependabot branches. At least Dependabot is smart enough to close all of the original PRs when the rollup is merged.

dynamite-ready
Yes. 'Rollups' would also help the situation where multiple upgrades in combination will cause an issue. Ideally though, you'll be merging master into each automated pull request regardless. But rollups would save a bit of time.
TruthWillHurt
I set it up on a repo, then delete emails I recieve from it.
[deleted]
(comment deleted)
ncphillips
Same feelings. I like the idea but in practice I don’t trust it.

Or rather, I don’t trust package maintainers to adhere to semver. I prefer to manually go through dependencies updating one at a time and reading the change logs. I usually do this in batch. Peace of mind is worth more than the hour saved every week or two.

I do really like the tool that flags security issues with packages though.

hypeatei
This is what I do, too.

Dependabot creates the PRs and I review the changelog / commits to make sure I'm not introducing a bug or security issue.

aeyes
You can still use dependabot to assist you with this. Let it only open PRs and it will show you the commits and changelog or at least a link to the source project in the PR description.

Having a PR already open with a full test run done in CI saves a ton of time, at least in repos with a lot of dependencies.

kristiandupont
My strategy is to blindly merge developer dependencies like linter tools etc., with the rationale that if CI succeeds, the dependency didn't break anything. Obviously, that is not completely guaranteed but the risk feels small enough that it's acceptable.

Any update to production dependencies I will want to test manually unless it happens to be a part of the code where I feel exceptionally confident in our test coverage.

marekdedic
I use it in 2 different ways depending on the project.

1) For more important projects I have high test coverage (over 95%) and integration tests. In this setting, I have dependabot set to check once a week and the PRs get automatically merged if everything works. I have never had a problem with this setup, if the update breaks something, the tests fail and I get to fix it manually.

2) For less important projects without CD, I usually merge dev dependencies and minor versions by hand if there isn't something obviously wrong in the CI. Yeah, there's a risk it breaks something, but I can fix that before a release - obviously these are projects that only get worked on occasionally. Major versions I usually check manually before merging.

I think the ultimate answer is good testing as part of CI. Either you trust your tests and then I think it's ok to auto-merge, or you don't and then all bets are of...

viraptor
> occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application

That will then happen whether you use dependabot or another approach. If you don't test this manually and don't have enough test coverage, you will fail there eventually.

Dependabot helps you fail here early when you have N packages that changed and it's easy to figure out what happened, rather than wait until you're upgrading 10*N packages together and have to isolate things manually.

shaftoe444
Dependabot made me move some side projects away from React cos it really highlighted how many dependencies I was bringing in without realising.
drawkbox
Software went from DLL Hell [1] to Dependency Hell [2]. Only now there is a dependency war so it is much more dangerous, and so many more dependencies.

Dependabot has to be used because of the threats. However everyone automating and moving to latest is also a threat. SolarWinds/VMWare/USGov hack [3] was all related to CI builds and automated "trust", ended up infecting tens of thousands of systems that thought they were secure with SOC2. SOC2 ends up making enterprises "trust" many third parties. What happens when dependabot is an attack vector as well...

The log4j/Log4Shell [4] issue shows how long exploits can go on without detection or automated fixes. Node is filled with dependency issues and that is just the known exploits besides all the "telemetry". [5]

Any third party or dependency is a potential attack vector, and dependency saturation is adding lots of tedium to shipping. So much time goes to just updating libs it is a bit of a tragic comedy.

[1] https://en.wikipedia.org/wiki/DLL_Hell

[2] https://en.wikipedia.org/wiki/Dependency_hell

[3] https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

[4] https://en.wikipedia.org/wiki/Log4Shell

[5] https://en.wikipedia.org/wiki/Npm_(software)#Notable_breakag...

mschwaig
I have not used Dependabot myself, but I wrote a tool that runs in a nightly CI job and creates or updates a single PR per repo with all of the dependency updates. You can then merge those with the push of a button. I think that's better than the more granular approach of having one PR per update.

This still lets me know when an update to ANY internal or external dependency will break the build, so that I can take a look.

My tool only works with Nix Flakes, so it updates dependencies of Nix packages, but you often have other language-specific package managment tools wrapped inside of Nix (e.g. nix calling cargo) and I think it would be a cool extension to add a configuarable language-specific shell command where the results of running that command are added to that single PR as well. That could document the exact update workflow a dev should go through while also regularly exercising it.

https://github.com/serokell/update-daemon is a better tool than mine that does the same thing and I have been thinking about adding the feature there. I'd be interested in discussing that idea further, maybe I'll open an issue for that.

hackandtrip
Did anyone try Renovatebot[1]? It should be OpenSource and endorsed from OpenSSF and Google. Main advantages I see is batched updates, which reduce a bit the dependency update spam.

1: https://renovatebot.com/

andrew_
Disabling it on a repository is one of the first things I do. I tend not to like how noisy it is on Node project. Instead, I have scheduled tasks which audit periodically.
jraph
A more useful bot would be named Undependabot. Cuts complexity caused by the dependency bloat by suggesting to remove random dependencies and linking to http://vanilla-js.com/. Flags any PR that introduces new dependencies, and computes the cost in time and money these new dependencies should cost in the long term, and the time and bandwidth usage it adds to each (CI) build. Adds badges to developers and teams that reduce their dependencies or have few or none. Rejects any PR introducing (transitive) dependencies to packages from jonschlinkert.
chrismorgan
Mind you, adding dependencies does not imply increasing build artefact sizes. Suppose A depends on B and C, and B and C contain similar functionality, then collaborate to extract that into D, so that A ends up with one more dependency, but less code net. (This is the theory of the tiny package philosophy widely found in npm. But in practice, they often seem to clog things up awfully with runtime type checking and the likes, or just adding a pointless level of indirection, so that they make things worse. But the underlying idea is sound.)

—⁂—

For the rest, most of the pieces already exist and are not particularly difficult to put together, though a more polished presentation of a holistic experience (if you’ll pardon the expression) could be desirable.

For suggesting dependencies to remove and flagging undesirable dependencies, that’s entirely a data problem: you want another audit/code review/advisory database, really. (You could take it further and identify automatic code translations, but that’d be quite a bit more work, though again the bones that you need generally already exist.)

For pointing out dependency costs: such metric monitoring is already widely used, especially in larger projects that want to catch regressions immediately.

geoffharcourt
I've been using Depfu for a while and I think it handles some of the biggest pain points with Dependency spam.

- Package releases don't get PRs in their first 24 hours unless they are for security issues, so you don't get noise if there's a yank or a quick patch for a bug in the latest release

- You can set development (or production!) packages to only update once a week

- Packages that are known to have a very frequent release cadence (AWS SDK subcomponents, looking at you)_get pushed to a much slower PR pace so that you only update them 2x/month, etc.

- This might be fixed now, but it had much nicer auto-merge behavior for releases that passed CI.

- With Yarn, it can run `yarn-deduplicate` after updates to trim down shared dependency bloat.

FWIW we still use Dependabot for security patches only because they seem to get picked up a few hours earlier. We also have much tighter lock rules on some JS packages which seem to make breaking changes on patch/minor releases.

oweiler
Dependabot only works reliably with lots of integration tests.
chrismorgan
I’ve only really experienced Dependabot for libraries, rather than applications, and in that context I don’t really understand why it exists or why people use it.

If you’re making a library and it bumps the lower version bounds of your dependencies, it’s doing something that’s actively slightly harmful. It’d be fine to notify you about major/incompatible version bumps, maybe even try them to see if they work (though certainly don’t apply the change automatically—a major version bump on a dependency is regularly a breaking change even if your tests still pass). But minor/patch/compatible releases that are already accepted by the version specifier? You shouldn’t be bumping those. Test against them if you like, maybe update your testing lockfile, but generally speaking the versions of libraries are an application concern, and your library should not be trying to dictate it.

(I must admit that the whole concept of compatible ranges is in practice slightly broken through and through, though it seldom causes trouble. I would love to have tooling that actively minimised dependency versions: “1.2.3? Turns out the newest functionality you’re depending on is from 1.1.0, so I’ll reduce the spec to ‘1.1’.” This would, of course, need to be paired with checking that you don’t accidentally use newer functionality—at the least, running your tests on minimum versions, preferably stronger API checks. Overall it’s the sort of thing that is difficult to make robust by any means in the likes of JavaScript and Python, but which could sanely be done in Rust, and there’s enough interest that I think something will probably happen within the next decade.)

But applications? Sure, this is where Dependabot has an actual sane purpose. But presuming people tend to have it configured in similar ways to how they configure it in libraries… ugh. I’d still rather just do roughly what it does manually from time to time.

tolkienfanatic
For people complaining that Dependabot is too noisy, it is easy to configure it to only open PRs for major versions.

I am a big fan of Dependabot, but maximizing its effective is highly dependent on the culture of the team/codebase.

fmorel
We don't use it on our Node app because it would be too noisy. Another noisy library was the AWS SDK for .NET. In that case, I actually use a wildcard version range `3.7.*` to always have the latest patch release and cut down on Dependabot noise.

We have it on for everything else (.NET/Nuget, Ruby, Docker, GitHub Actions, Git Submodule). It's great at keeping us up to date with little effort. For major (and some minor) releases, I'll read release notes to see if there's anything to watch out for.

It's especially useful for updating the git repo we use as a submodule in 5 other repos. Submodules were a source of frequent merge conflicts before as different developers updated it in their PRs. Now we pretty much don't think about it, and Dependabot keeps it up to date with our latest database models and shared libraries.

edit: We have it set to weekly for everything but the submodule, which is daily.

antifa
It would be really helpful if dependabot mentioned how much bigger my build size would be relative to the main branch.