HN Explorer

Ask HN: How do you automate file upload to GitHub?

8 points by sirspacey ↗ HN
I’ve been exploring Git workflows and curious how others would approach this challenge:

We need to upload a file to GitHub on demand (a table routing JSON file to minimize API calls) and want to automate it so non-technical users can push the file.

Our ideal workflow is to generate the JSON and push automatically via webhook or API call

How would you approach it? Things you’d be concerned about with this approach?

Some additional context:

- We have to solve this problem in this way due to limitations with another code base we rely on.

- this is a private repo so we aren’t concerned about merge issues, just overwriting the file is fine

13 comments

[ 2.5 ms ] story [ 34.5 ms ] thread
codingdave
I mean, we're talking about git here - so you don't need to re-invent pushing a file to the repo. What you need to focus on is building a UX that abstracts it so the non-tech folk don't need to do it via command line.

So the question is less about pushing to git and more about what causes the file to update in the first place. If it is an automated update, just have whatever task updates it also run the git command to push it. If it is a manual update, what tool are the non-tech people using that makes it happen? Add a button to that tool which runs the command.

sirspacey
Agreed - I'm new to git so the question is what code that button is firing? Ideally in a cloud app not desktop app.
alemanek
I wrote an ansible pipeline that pushes config backups, json and ndjson, to a GitHub repo. It just uses the “git” cli tool to clone the repo, commit any changes, and push those commits to “origin/main”.

Only piece that you have to be a bit careful with the guarding credentials or token you will be using for the “git” commands. In my case we use Hashicorp Vault for secrets management so I can just checkout the token to use from there.

sirspacey
Got it. CI/CD tools coming up often as a possible solve. Time to do some YT vids!

Appreciate the guidance on credentials, fully agree on using a vault.

solardev
You also don't have to use `git` directly for this (the command-line tool / protocol).

Because you're on Github and wanting to edit it from another cloud app, it might be easier to just use the Github API (which is its own thing, not git-based) to edit the file directly: https://docs.github.com/en/rest/repos/contents?apiVersion=20...

Then you can skip all the CI/CD stuff (it's overkill for something like this) and just send a plain fetch from your app's backend (NOT the frontend/client!! Don't expose your Github tokens to your users!). You can either use their JS SDK or just send the raw HTTP commands.

hotdogs
If it's JSON it sounds more like config than code. Does it actually need to live in the GitHub repo? Could your application read it from an external location (where ever the rest of your configs live) and reload it on an API call or periodically?
sirspacey
We tried this but the performance hit was too high

* edit: we're building an integration with a SaaS product that doesn't allow us to reference external files due to security, hence having to upload direct to git

mystified5016
#!/bin/bash

commandToGenerateFile > file.json #or wget some.server.tld/autogen.json file.json git add file.json git commit -m "Automatic upload <date>" git push

You can do this in any number of ways. Powershell, batch script, bash, you could even build a little GUI if you want.

If you want to get fancier, throw the script on a server and give users an SSH script they can double-click, or hook it up to some intranet web interface.

This isn't a particularly complex problem, and git was designed to handle use cases just like this.

Of course you could absolutely overcomplicate and burn dozens of engineer hours building some kubernetes cluster or AWS swarm, but you can do this with a script so simple that I wrote it on my phone.

sirspacey
yeah running a bash script is one option

the k8 approach is tempting though haha

dan_can_code
I'm not sure if it's overkill but decapCMS [0] (formerly netlifyCMS) uses git for its publishing method to store static files and content. It's quite simple and open source, and might even just serve as good inspiration for your final solution.

[0] - https://github.com/decaporg/decap-cms