Launch HN: Infield (YC W20) – Safer, faster dependency upgrades

(founder here). The interaction between high level language package managers and system binaries or compiled extensions is definitely on our roadmap. For example the ruby net-ssh gem relies on specific versions of openssl being compiled on the machine. Do you think this is particularly difficult in ML?

Is your frustration mostly from needing compile all this non-python stuff across environments?

I don't think you'd print money just by telling people if a dependency is hard or easy to install though. You'd need to actually solve the problem of getting the dependency installed or suggesting some alternative solution that gets the code written.

E.g. if I'm banging my head against a wall trying to upgrade to the newest version of PyTorch for some graph embeddings feature and everything breaks, knowing it's high risk doesn't help me. I still need to get the thing installed so I can use the feature.

Do you work at a company forcing you to develop on Windows? You couldn't pay to do to that.

Only Ruby right now. We want to make it work really well for one language before expanding. Which language would you want it in?

We have a command line tool you can use to send your lockfile to our server so we can process it, but you have to use our web app to view the results.

Thanks! We actually pivoted to working on this problem at the beginning of last year. We were previously called Syndetic and were building software for data providers. As we considered pivoting I spent some time thinking about the biggest problems I've faced maintaining software and started doing personal consulting. I upgraded a bunch of Rails and React apps for various companies over the course of 2022.

Toward the middle of last year I re-connected with Andy (our third cofounder) who I went to school with and have wanted to work on a company with for a long time. We did a bunch of customer discovery / product work to figure out how to take what I learned doing this by hand and turn it into a software product. That was exciting enough that we decided to bring Andy on as a third co-founder and pivot our YC company.

Allison and my background is in building data businesses. Before Infield and Syndetic we worked at a startup in the beverage industry where we standardized inventory data for every alcohol product sold in the US. As we got into building Infield we didn't expect to use LLMs at all. We imagined a similar human-in-the-loop expert system to what we've built before.

I've been extremely impressed with recent language model's ability to handle unstructured changelog text. For example, consider the following snippet of a changelog:

  Security:
  - Address an issue with password validation

  Breaking change:
  - The `foo?` method now returns a boolean instead of int

Language models can carry the context through, so we are able to not just parse this apart into discrete changes (which I could figure out how to do with a regex) but also bring in context and categorize them. It can do this generically and really feels like something new.

Andy (Infield co-founder) here - agreed that wording was a bit clunky. Changed now. Appreciate the feedback!

Thanks for the feedback. We know we're tackling a big problem here. Our ultimate goal is that if you use Infield we keep you on the latest version of every dependency, automatically. The road to get there can go a number of ways.

> My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares.

Planning is one way to make those major upgrades go wrong less often. Our idea is that we can break large upgrades down into incremental changes that are individually safe and mixed in to your regular dev cycles, so that when you go to do the large upgrade you're not making such a big change all at once. This means things like fixing every deprecation ahead of time and making sure all the other dependencies you use are compatible with the version you're upgrading to.

We're calling this "upgrade paths" in our tool. Something like a major rails version upgrade is terrifying because of all the moving parts. Often you may have a dozen other gems that need to be upgraded in order to upgrade Rails. You said "If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.". We want to make that the flow for major framework upgrades, so you're merging in incremental improvements in the blocking dependencies ahead of time.

Undocumented breakages are for sure on our roadmap. We have two types in mind today: - Changes that are missing from the changelog - Incompatibilities across gems (for instance there was a time recently where if you had both datadog and newrelic installed and used elsaticsearch upgrading the newrelic gem would break prod)

We have some ideas on how to figure this out automatically (like reading code diffs with GPT rather than just changelogs), but the best way is to see upgrades out in the wild. As we see more and more upgrade experiences from our customers we'll be able to catch these issues and build this dataset.

I'd love to hear other ideas for making major framework upgrades safer.

> * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail.

Would this be something like `git bisect` for upgrades? We run your CI to figure out where you can upgrade to without something breaking? We're trying to figure this out statically by reading changelogs and building up a database of breaking changes. Our roadmap is to make this database better over time by pulling in more and more undocumented changes (by seeing customer upgrade experiences and by sourcing information from places like github issues). In my experience it's more common (and much more dangerous) that things break when your dependency changes in a way that wouldn't be caught by CI.

It's a bit buried, but if you go to https://app.infield.ai/hn and click "I don't use Ruby" we'll ask for your email/language and put you on our list.

Can you expand a little? Here's some technical background on what we're doing:

We have our own database of every version of every rubygems package alongside its runtime dependencies (like you see at https://rubygems.org/gems/pundit).

Then we parse your Gemfile and Gemfile.lock. We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code; we use bundler's APIs to parse your Gemfile.lock). This gives us all of the dependencies your rely on.

Then we let you choose one or more package that you want to upgrade and the version you want to target (let's say Rails 7.0.4.3).

Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything.

If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version). This is where the optimization part comes in. The problem becomes "what is the optimal set of versions of all your dependencies that would resolve with the next version of Rails". Currently we solve for this set trying to optimize for the fewest upgrades. As our dataset of breaking changes gets better we'll change that to optimizing for the "lowest effort".

Happy to elaborate.

  if RUBY_VERSION < "3"
    gem "minitest", ">= 5.15.0", "< 5.16"
  else
    gem "minitest", ">= 5.15.0"
  end

Andy here (Product at Infield). I've also been impressed with what I've seen from Renovate - particularly on the configuration side. How has it worked for you on major / breaking upgrades?

Thanks! We wanted to start with a language that's relatively consolidated around a framework before expanding to more balkanized ones like JS.

Definitely. I'm very interested in govulncheck and building tooling into the ecosystem to make dependency use safer.

I generally agree with you, but I'd caveat that a critical CVE is easier to take if you're on the latest version of that dependency (so you can easily apply just the patch and not other changes). We want to help you weave package upgrades into ongoing maintenance rotations so that this is the case.

You mentioned that implementing an upgrade is difficult because of the actual code change but also the testing and identifying what the changes are in the first place. We're starting trying to do a really good job of the last one - what are the changes in this next version, are they breaking, do they break for your codebase, and what should you do if they are? We want to move from there to automating as many of the code changes as we can, but my sense is the biggest value is in giving you the confidence that we at least captured all of the changes you need to be concerned about.