Ask HN: What tool(s) do you use to code review and deploy SQL scripts?
Say you're an engineer that needs to deploy a SQL script to a production database; you need it peer-reviewed and (hopefully) approved by a chain of approvers who gate-keep production databases; what tool do you currently use?
I'm assuming you'll also need this script/change to go through QA and staging before it reaches Prod. The only tool I found, short of daisy chaining a bunch of GitHub actions, that does this is RedGate Deploy.
Curious to learn how y'all are achieving this today.
22 comments
[ 5.1 ms ] story [ 60.4 ms ] threadAnother is migrations as code using an ORM library.
Last gig was doing that and diff’ing sql schema and bootstrap files, depending on the db. We eventually turned them into one time docker containers which actually made life easy
What we are not good about is making sure the changelogs have revert capability... some devs do it right and others not so much
Funny you mention changeset rollback. I tried doing that and got it working for a wide variety of SQL types of changes, but I had a hard time at the end of it convincing myself all the extra code writing and testing of all that was worth imposing on other developers for our fairly centralized SaaS-supporting database.
Liquibase supports a lot of databases but not always all the data warehouse/lake latest ones I’m interested in, e.g. Athena/iceberg.
If it is for a CMS / Web based thing that modifies the structure of the database or inserts some seed data - Use Migrations
If it is for some ETL type job where you are using SQL then migrating to a tool like DBT is probably wise.
We commit schema file alongside the rest of the code, and it gets applied as part of the CI run to our dev and QA/test DBs.
So not a direct answer I suppose, but it really cuts down on what you need to do manually against the DB.
We use GitHub actions for deployment and database migrations are just one step of the pipeline. The step invokes sqitch deploy which runs all the pending migration files.
Then, all the approval process is standard for the environment. We require approvals in pull requests before merging to the main branch.
You can further check out this tutorial to get a feel of our GitOps solution
https://www.bytebase.com/docs/tutorials/database-change-mana...
I’m not sure anything more complex is needed. I tried using Liquibase at some point and it was a nightmare compared to just using SQL.
[1]: https://github.com/jackc/tern
Active Record is a Ruby library that provides ORM support in Ruby on Rails, and includes an elegant way for creating reversible migrations.
Capistrano allows you to configure and run tasks on a remote server, such as pulling the latest version of code from the git repo and running the newest migrations.
The key here is to put _everything_ into a 100% replayable transaction log, which is what said combination does.
So ours is pytest that builds up a fake database + tables to unit test the scripts. And otherwise it is the normal review process for pull requests.
So ours is daisy chained as well, but I am quite happy with it.
(Though I don't think it is "complete" or "perfect", either.)
EF Migrations are in a really good place now if you like/don't mind C# as a language (and you can easily embed SQL inside the C#, too, but there are benefits to being able to also run high level C# code). With today's tooling you can package your migration "runner application" as a single deployable executable for most platforms. You can build the executable once and run it in all your environments. (The same tool that updates your QA and Staging updates your Prod, testably running the same migrations.) Given the single executable deployable I might even consider using it for projects not themselves written in C#.
I personally use SQL Server Data tools (SSDT) Which gives a full visual studio project and can be automated in CI/CD. It has some drawbacks but after learning the ins and outs you can pretty much do anything with it.
A lot of the same goes for testing down migrations, but even more overlooked how disincentivized they sometimes get. If a developer never has a reason to use down migrations, they never get tested, and likely are full of bugs. I know too many developers that don't believe in down migrations for that very reason that they "never" see a use for them and so will never use them and will never bother testing them. I've also worked on applications where rollback plans were critical and rollback testing something of a norm even in developer local environments and down migrations, especially loss-less ones (though that is true hard mode for SQL), were a critical part of CI/CD and the frequent testing made for good down migrations. (If for some reason you need to test a new bug in an old version application as it was, and you can clone a current production database and trust that you can run down migrations to get it to the state that old version expects, there's some useful debugging things you can do, sometimes.) I sometimes don't think down migrations are worth the effort either even having seen them used at all, much less kind of well.
Direct advice that I can offer specific to EF Migrations as it sounds that you do still have those hundreds of migrations: if you haven't already, switch from scripts to EXEs. The "Idempotent" scripts especially are nice when a project is just starting out and you've got a few or a few dozen at most migrations, and they are "normal" T-SQL files so there's a lot of DBA comfort there, but it is in practice just about impossible to keep them working after more than a couple dozen migrations or so due to T-SQL parsing bugs and I've seen too many migrations break the script files entirely for silly parsing reasons. Today's EF Migrations offer a way to build a self-contained EXE for any platform .NET supports (so you can build a linux-x64 if you need it for your cheapest build agents in your CD pipeline) to run through migrations. The EXEs are faster and more reliable, use good transactions, only run the migrations that are needed, and can handle a lot of migrations including start-to-finish runs that you might currently see failing as scripts (because T-SQL parsing has weird quirks and they get weirder the larger the file is).
I was a fan of SSDT for a number of years, it's always been a good option. At this point SSDT aren't often my first option given the Visual Studio requirement and SQL Server specific nature of it. (EF Migrations can be used now for other databases, such as Postgres. Also, not that I don't like Visual Studio, but there are benefits to more cross-platform options like VS Code.) (For some of those years I wished I had DBAs I was working with that understood SSDT better because of silly reasons that SSMS worked just fine with dacpac/bacpac files as deployment artifacts but DBAs I encountered seemed to have a hard time learning those tools in SSMS and refused to try SSDT.) I know Azure Data Studio had been s...