HN Explorer

Ask HN: How do you manage one-off internal scripts?

11 points by rolandtshen ↗ HN
For a recent project, I've found myself writing quite a few one off scripts for database adjustments, SQL queries for analytics, etc. I've been manually running them myself in the command line, but it's a very messy process. New team members also don't know what scripts are available and where to look. Anyone know of any tools, or have any workflows to better organize these scripts?

16 comments

[ 2.1 ms ] story [ 24.2 ms ] thread
MattGaiser
At a prior company we added them to a folder of misc scripts along with notes on what they were for.

Not the greatest solution but it did ensure they never got lost.

rolandtshen
As more engineers joined and people needed to reuse old scripts, was there some process to discover them on their own? My scripts are also dumped in a repo, but team members are having trouble finding them and running them especially since there are quite a few now.
MattGaiser
A combo of group experience and manually reading them.

Most scripts were never re-used, so it was not a problem all that often.

deeteecee
you could just put them in a repo
willcipriano
Find the closest related repo and add a scripts folder has been my approach.
villasv
If they were truly one-off, you wouldn't need to manage them. Categorize and document. SQL queries for analytics go into something like Metabase. Database migrations go into git. Frequent adhoc scripts go into something like Jenkins so you can trigger them easily. So on and so forth. All of those documented somewhere, like the team's Confluence/Notion/whatever.

But don't just leave them at that, do work on eliminating their need. Sisyphean task but necessary nonetheless.

rolandtshen
I'm curious about your workflow for reusable scripts with Jenkins. What are the advantages to putting it in Jenkins? How do you set up Jenkins to run these individual scripts?
villasv
Jenkins is not the ideal place. We take some scripts and create pipelines that just invoke them. Some of these are configured to be scheduled jobs. We have separate declarative pipelines for each one.

But this isn’t good, we have Airflow and that’s where these things should be. Their stay inside Jenkins is technical debt, but at least is much better than not having it automated. The advantage is that you made an ad-hoc process less unstructured, you can add post-failure alarms, you don’t have to worry about long running scripts being interrupted by shabby notebooks, you don’t need to spread production credentials to every developer and so on.

tcbasche
The classic is the /scripts directory in your codebase (presumably in VCS). The key is also to trawl through occasionally and remove anything that hasn't been used in a while.
giorgioz
I do the same. a /scripts folder in the root of the git repo. I keep old scripts mainly to copy-paste the structure and/or pieces for new one-time scripts. It feels a bit dirty but I haven't found a better solution yet.
SirensOfTitan
If it’s truly one-off (like codemods), I put the scripts in relevant tasks and diffs for reference.

…otherwise we just throw everything into a scripts directory. Keep it flat until it feels messy, then start organizing. All documentation must live with the code, otherwise it gets out of date too easily.

marto1
I mostly..don't. For one liners, I keep them in a git controlled org-mode file. But for everything else it's all over the place.
fiftyacorn
I have a repo and raise a jira for each script and record any history or background. Then mark the git history against the jira.
yokoprime
If it's one-off but has to be worked on for some time, with testing and whatnot it may be placed in a folder for maintenance scripts in the repo. Once it's been executed it can be deleted from source control
Jugurtha
Add them to the repository and treat them like you would treat other code. Chances are, they'll become some sort of tooling later when people tweak them, improve them, make them modular, generalize them.

We introduce many things that way. First it's just a few commands that do something useful. Then included in the repo. Then tweak. Next thing you know there's an endpoint with that functionality because it's useful and frequently needed.