GitHub PM here. We have tried this, but we weren't able to get results that we were satisfied with. Of course, you have to revisit these things regularly, as the models and wider state of the art are evolving so quickly!
This is cool, it looks to me like you're integrating static analysis on the user's codebase and the underlying dependency. Very curious to see where it goes.
We've found dependency upgrades to be deceptively complex to evaluate safety for. Often you need context that's difficult or impossible to determine statically in a dynamically typed language. An example I use for Ruby is the kwarg migration from ruby 2.7->3 (https://www.ruby-lang.org/en/news/2019/12/12/separation-of-p...). It's trivial to profile for impacted sites at runtime but basically impossible to do it statically without adopting something like sorbet. Do you have any benchmarks on how reliable your evaluations are on plain JS vs. typescript codebases?
We ended up embracing runtime profiling for deprecation warnings / breaking changes as part of upgrading dependencies for our customers and have found that context to unlock more reliable code transformations. But you're stuck building an SDK for every language you want to support, and it's more friction than installing a github app.
12 comments
[ 2.7 ms ] story [ 29.3 ms ] thread(a) they’re broadly similar across companies,
(b) they aren’t time-sensitive, so the agent can take hours without anyone noticing, and
(c) customers are already accustomed to using bots here, just bad ones
(I'm one of the maintainers on Renovate)
We've found dependency upgrades to be deceptively complex to evaluate safety for. Often you need context that's difficult or impossible to determine statically in a dynamically typed language. An example I use for Ruby is the kwarg migration from ruby 2.7->3 (https://www.ruby-lang.org/en/news/2019/12/12/separation-of-p...). It's trivial to profile for impacted sites at runtime but basically impossible to do it statically without adopting something like sorbet. Do you have any benchmarks on how reliable your evaluations are on plain JS vs. typescript codebases?
We ended up embracing runtime profiling for deprecation warnings / breaking changes as part of upgrading dependencies for our customers and have found that context to unlock more reliable code transformations. But you're stuck building an SDK for every language you want to support, and it's more friction than installing a github app.
0: https://fossabot.com/