219 comments

[ 2.4 ms ] story [ 261 ms ] thread
(comment deleted)
Did Microsoft switch GitHub to AGI as part of the OpenAI deal?
Q: Can it automatically reject bad code or would that be impossible because of the halting problem?
it could automatically reject bad-looking code perhaps
Brings into perspective how essential git is for my workflow - I am waiting for `scp` to transfer my files from my laptop to work computer and can't push anything into the CI/CD pipeline.
For what it's worth, the "D" in "Distributed Version Control System" is useful here. You can `git init --bare foo` on your work computer, `git remote add workcomp username@hostname:path/to/foo` on your laptop, and `git push workcomp master` to push everything over using pure Git. (And the first steps only have to happen once for these two machines.)

(This creates a bare repo on your work computer, meaning there's no associated working directory -- you'd probably want to add that same repo as a remote from whichever existing repository on your work computer you have. The bare repo, in this scenario, is just a means of passing commits from your laptop to your work computer in a Git semantically-meaningful way.)

Your comment is probably going to inspire a lot of people to start self hosting remote repos during the downtime today.

As with most things, it'll start great, but a lot of those people will be in tears in a few weeks.

Great power; great responsibility.

True! It's best to consider that kind of repo as a downstream fork, with all the responsibilities of upstreaming changes once the canonical repository comes back up.
Having a bare repo is certainly one option. Some may find having two copies of a repo on one system to confusing.

Here's an alternate setup that doesn't use a bare repo. It does require some git hygiene/discipline though.

Setup: Desktop: git config --local receive.denyCurrentBranch updateInstead This will let the laptop push the desktop, updating its files, as long as the desktop doesn't have uncommited things (aka working dir is clean). On the laptop: git remote add desktop...

Working with it: Desktop: commit everything Laptop: commit everything git pull --rebase desktop git push desktop (assuming there are no issues w/ the pull).

Not saying either workflow is better, merely providing an alternative.

imo the reason git took over has some to do with being unopinionated about workflow - there's some tooling, but whatever workflow is managable with that tooling is "supported" - as long as the team can agree to use said workflow.

Just when it was time for beergarden. Thank you!
I'm just wrapping up the work to migrate my company away from Gitlab to Github and this happens. I did it because I figured Github has to have better reliability / uptime than Gitlab. Someone joked that as soon as the migration is done Github will have some major downtime.

sigh

(comment deleted)
You can't spell GitLab and GitHub, so you deserve it.
If we are being pedantic they were both spelled correctly.
HN is not typically receptive to snide corrections/remarks. There are several more productive ways to address a minor capitalization error - the best of which is to not correct it at all because it's so trivial and unimportant.
Besides, why would I care if how I capitalize a company's name isn't perfectly in line with their marketing style guide? Github is lucky if I even feel like capitalizing the first letter.

What a weird thing to hold over others.

I highly recommend running at least a local, self-hosted git mirror at any tech company, just in these cases. Gitolite + cgit are extremely low maintenance, especially if you host them next to your other production services.

Not to mention, if you get the self-hosted route you can use Gerrit, which is still miles better for code review than GitHub, Gitlab, bitbucket and co.

You don't even need gitolite, if you're going the self-hosted route:

apt install git-all

is enough to host your own git server. Put it behind a firewall to limit access and use standard linux users with ssh keys for access control if you don't need anything fancy. For small companies I'm not sure you need anything else. Of course if you need different levels of access etc then you'll need more sophisticated tools, but many people won't.

Code review I do using local tools (the editor) face to face, again not sure you need an online service for that unless you're a larger company with lots of developers coordinating (in which case it becomes pretty essential).

> Code review I do using local tools (the editor) face to face, again not sure you need an online service for that unless you're a larger company with lots of developers coordinating (in which case it becomes pretty essential).

I mostly like online code review services because they offer an audit trail and semantic history that's easier to navigate than email. And of course, to let CI automation check tests, coverage and lint. Not because I don't trust my coworkers, but because otherwise I would forget to run tests and lint myself.

Lots of different ways to do it, and of course github and online code review is tremendously useful to people, particularly on large projects with lots of collaborators, and where a history of reviews is required.

For lots of small projects though, it's perhaps not as necessary as people think. I run tests and linting locally on save and don't really use the code review/CI features of online hosts much. That won't suit everyone of course, but it is one possible path.

Could you explain why Gerrit is better than the rest for code reviews? I have not used Gerrit in years (before I ever used github PRs) and I guess I don’t miss it, but also don’t know what I am missing :)
For me there are two killer features:

1) dependent reviews/change requests. I will work on some feature, submit it for review as one CR, and then I can immediately start working on a feature that depends on that. When I submit this one for review, it will be always shown as dependent on the first, and show a diff against master after the first is merged. This also means you can split large changes into multiple CRs, have them reviewed (possibly independently), then submit them all at once. It makes changes across large repos fantastically easy.

2) very powerful rule engine for approvals. It's based on Prolog, and basically allows you to define arbitrary, turing complete rules on what labels added by whom must be present on a CR for it to be submittable. Using the 'owners' plugin, you can also make it depend on OWNER files that define ownership in subtrees of the repository. This can lend to rules like 'product A must be approved by an owner of A but cannot be self-approved; in addition, someone who is fluent in the languages used must approve it, but that can be self-approval'.

Without those two working in Git monorepos is painful. And since I like monorepos for other reasons (like ease of deployment and testing), I like Gerrit, too :).

It also offers, in my opinion, a much better UI for actually reading and commenting on code. High contrast, fast keyboard navigation, marking of files as reviewed and a very readable history of patchsets, comments, approvals, etc.

The learning curve is much steeper than a GitHub PR, as it's a somewhat weird abstraction (CR/patchset vs git commit/branch), but in my opinion it's worth it. I guess it's my general tendency to use less beginner friendly but more powerful tools. ^^

Github's PRs are pretty bad at letting you comment on code near the diff lines (you can do it if it's within 5 lines, but if you have to click to expand the entire file, you can't comment on the expanded parts). I also like how Gerrit lets you comment on specific parts of the line, rather than the entire line.

Finally, I'm a big fan of the various labels that are common. +2 Code Review means I reviewed the code, +1 Verified means that I ran it and it worked. Those are different things and having to have both makes the responsibility clear, even if the author is adding +1 Verified.

> dependent reviews/change requests

This is really nice in Gerrit. On GitHub you can simulate this by changing the base of the PR yourself, but it's not as smooth experience.

you can also self-host the GitLab community edition :D
>I did it because I figured Github has to have better reliability / uptime than Gitlab

Why the hell would you think that after the Microsoft acquisition?

(comment deleted)
(comment deleted)
Kinda irritating when a binary you need is only hosted on github (...together with the files to build from source).
Even worse if libraries / packages are only hosted on Github. Then the main ecosystem of a language dependent on those packages can grind to a full stop from there.

I'm looking at all of you CocoaPods, SwiftPM, npm, Cargo, vgo.

AdoptOpenJDK builds are hosted there too; found out about the outage trying to download one.
The npm registry itself does not depend on GitHub.

It's true that most developers rely on GitHub for pushing to the registry, but there are a handful that use GitLab or self host that would be totally unaffected.

I'm in the same position trying to build Neovim, but I have all the sources downloaded already since libmpack always times out. Maybe someone browsing this thread who can't work right now can help: does anyone know how to force/suggest to Cmake that the dependencies it needs are already downloaded and in some src directory?
I heard that maven is also considering to move it's central repo to github.com.
push/pull seems to be working for me. Getting 500 server error while trying to browse repositories on the website
Push and pull doesn't work for me. For a second I thought this was my company's subtle way of terminating my employment.
That explains a sudden access error when I tried to open an issue in a repo.
I feel like this whole year has served as one big reminder of how fragile the internet really is...
The great AWS crash of 2017 convinced me to distribute a bit more, thankfully
It's only fragile when you centralize all of the information in giant websites. It didn't use to be like that. And now there are a ton of decentralization technologies that we should be taking advantage of, but almost everyone ignores them.

For example https://medium.com/@alexberegszaszi/mango-git-completely-dec...

Absolutely. I'm (cautiously) optimistic that some good can come from all this downtime in that sense.
It certainly helps make the argument not to move some stuff to the cloud, there is still a case for on premise as always the answer is "it depends".
Well, the internet didn't stop working, so it's no so fragile after all.

You probably mean centralized internet based services.

Good that git is a distributed VCS. At least our master branch history is available on our local PCs, and we can freely work. Yupi.
Hope you dont have any dependencies hosted in github :)
Java and JavaScript programmers have solved it years ago. Maven Central and npm do the job :)
Do you do PRs? or have CI/CD pipelines hooked with GitHub?
not being able to submit or review PRs for a day is much less impact than not being able to checkout for a day.
I guess it depends on the definition of done of each team.

We normally can't complete tasks until they are code reviewed, so checking out locally would be the analogous of working offline in a CVCS

You can finish the first task as much as you can, then start on the next one...
The CI/CD pipeline could also be done with a post-receive hook on another remote.
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -Leslie Lamport
You've got to wonder what the regex that caused _this_ downtime is going to be
This is the perfect time to take a break. Kick back, have a coffee, contemplate your life choices. That commit can wait, that PR (i was about to merge) can wait too. It's not the end of the world.
(comment deleted)
well actually I spent the holiday weekend not working and now I have a time window to work with our east coast contractors before they stop work for the day (they don't work on the weekends or after 5PM) so having the system down isn't really end of the world but it will impact our productivity.
Why not just keep working? That next commit will work just fine without a central server.
merging work with contractor right now...
That works too, believe it or not. Just add another remote and pull it.
I only coordinate my remotes through github. I don't use git in "decentralized mode".
That's not something you decide once at the beginning and get stuck with it. You can reevaluate that decision at any point and easily add more remotes to a git repo on the fly.
So, uh, it's not clear what exactly people are expecting me to do here. Wire up my repo to my contractors temporarily (not even sure how I'd do that as i don't have access to their machines or filesystems) then detach when github is back (it's already back)?
A single repo can have many remotes. Adding additional remotes does not prevent you from also using old remotes, nor do you ever have to remove a remote that you intend on using again in the future.

Obviously it's a moot point for you now, but getting comfortable with how git remotes works is something that could pay dividends for you in the future.

I have multiple remotes already and know how they work. What I'm asking is, if github is down, and I don't have ssh or other access to my contractor's repo on their machine, what is my remote supposed to attach to?

Also, the additional effort to manage multiple remotes is entirely nontrivial.

(comment deleted)
They can always tar up their repo and send it over. You then unpack it on your end, then set it as a remote, pushing/pulling from it as you see fit. Admittedly, not the greatest workflow, but it doesn't require any new tools if you are already familiar with remotes.

As another alternative, git-format-patch/git-apply or git-bundle may be a quicker way of shipping changes around.

You don't need write access to a remote for that remote to be useful. Two or more people can collaborate using public repos that others may fetch from but not push to. Then, in lieu of github PRs, you request other people pull from you with email or however you like.

If you need access control without VPNs or whatnot (or if you simply want something closer to the github workflow), there are other services similar to github that can almost certainly provide what you need, such as gitlab.

> "Also, the additional effort to manage multiple remotes is entirely nontrivial."

With practice, I'd say it asymptotically approaches trivial.

1. Send a series of patches in an email.

2. Send a branch in whatever file exchange service you use. (Also formatted as patch series)

3. Setup ad-hoc vpn (zerotier?) and setup remote via SSH.

4. Push the branch into a private repository on a different service (private gitlab?)

5. Spin up a free t2.micro on AWS and push the branch there.

There are lots of options.

At some point there has to be centralization. We are deploying one product, not N products for N commits. If I can't push, I can't have CI build my branch, and I can't submit a PR for review.
It should be easy to move the center. Having a single 3rd party be the only central point you have available is asking for trouble.
But you could continue working and only have to apply minor patches forward if you later find your starting point needed to be fixed.

You don't have to wait for CI, just create a new branch and continue your work as though CI passed, and your PR was accepted. If that comes true you don't need to worry much, just rebase and continue.

On the other hand if you find out you needed to make changes - do those on the branch you made those changes, and finish your PR/CI cycle. Then go to the new branch you continued work on and rebase, and continue.

Is there something I am missing?

Heaven forbid we find a reason to take a break
(comment deleted)
In the old days at least we had compiling. Not so much anymore.
No, today we have something else to wait for, it is called the runtime.
But users wait for that, I can get on with finding a new framework instead.
npm install
And when you need to delete your node_modules folder for the upteenth time, it's a nice 5 minute break.
But you can technically spin up another git server, you can spin up as many git servers as you need and the same code will be there. I do understand what you mean though, been there. Organizations may not be happy with hearing that they need multiple places to store code.
you can totally push to each other dev machine as needed, so that covers reviews. and just get the guy responsible for maintaining ci to fetch from everyone machine and push to the build server, or you can configure that to pull from this guy pc if needed depending on your ci
Because we forget how to let go of all this work stuff and get sucked into it. There is a life out there too. some one us can balance, some find it hard. this is an opportunity to just pause for a few minutes. Mentally pause.
(comment deleted)
Right. There's an entire world out there of of personal projects and open source contributions you can be doing when not working.

People get so caught up in their daily coding they forget about all the other fun coding they could be doing outside of work.

It's a shame.

Or, not coding.
I think there was a joke in there ;-)
(comment deleted)
Of course, you know that, I know that, we all know it - but our bosses don't ;)
They are living in the technical stone age then. Time on keyboard does not equate good productivity. Happiness, balance, freedom - that increases output.
Because programmers always need a break. Don't go giving our bosses ammo. It's compiling, it's uploading, Github is down. Have some compassion.
And reflect why keep using git when we still work the good ol' centralized workflow style of SVN and the likes.
(comment deleted)
I wonder if all of these downtimes are related to the summer holiday season...
They're probably still moving stuff to Azure after the acquisition. If my memory serves, there was an uptick in outages ~6 months after the buyout.
6 months after an IPO is often the conclusion of a typical employee lockout period on shares, which can also be a time of more frequent team members leaving.

I wonder what the retention incentive period was for Github employees; it's often a year, and the Github acquisition was mid-2019...

I think the comment is referring to the other outages like Cloudflare, Stripe and so on.
Is my project lost now?
Damn. Thought I was going nuts when I couldn’t browse certain repos. Everything I checked said GitHub was up
There was a certain repo I was trying to access and I was concerned the content had been taken down. I pulled up the repo in Google Search cache, copied the git URL, and cloned it from my terminal. That worked just fine, only the webpage was completely down.
Github goes down on Monday around 9am pacific time - must be totally random.
The funny thing about this is the idea that west coast engineers actually start work at or before 9am
I'm rarely in before 9AM. The 10 minute standup at 10:30 AM is the only daily requirement for most staff.
I bet a bunch of PR's were built up from the weekend which weren't deployed, and some guy who came in at 9 decided to deploy them and broke things. Always scary to be the one to deploy if no one has deployed in a while.
Although it's nice that Git is Git and we can all mostly still work, it still seems foolish to rely on a single point of failure like Github. I've been toying around with the idea of creating a tool that would map the Git api to work with two+ hosting services at the same time. The effect would be something like, run "$git push" and it pushes to Github, Bitbucket, and Gitlab. I can't imagine something like this would be too difficult and would eliminate having to twiddle your thumbs while you wait for things to come back up.
You mean like?

    git remote set-url --add --push origin git@github.com:Foo/bar.git
    git remote set-url --add --push origin git@gitlab.com:Foo/bar.git
:-)

see: https://git-scm.com/docs/git-remote#Documentation/git-remote...

Hah, today I learned! It's a great feeling when you don't have to reinvent a wheel, thanks.

Now I'm just annoyed that more teams I've been on haven't set this up!

Me too. That's awesome: I've just suggested it to our team. Thanks for sharing GP!

One serious question though: how do you deal with PRs when you do this? That's one area where it feels like things could be quite messy, especially if you have quite a few PRs going in throughout the day.

Pull requests could be done through email using the git format-patch, git send-email, and git am commands.
You could merge the feature branch into master (or whatever), resolve conflicts, commit, push and deploy.
There have been various proposals over the years for how to integrate issues and reviews in the distributed git tree itself (http://dist-bugs.branchable.com/software/), but I don't think any of them have really gone anywhere, certainly not in terms of support by the hosted git vendors.

Having looked briefly into it now, git-dit does look promising in its approach. I'd be interested to hear from someone who had actually used it and bumped up against the limitations: https://github.com/neithernut/git-dit/blob/master/doc/datamo...

Merge the PR on github, pull to your local copy (now you're ahead of one of the urls of origin), push (and it should just push to the origin that's behind)

If you have any discrepencies in between them, you'll need to merge locally of course.

Now that you solved your problem. Let me guess your next question:

I have two git repositories which somehow got into an inconsistent state: How can I reconcile changes in both repositories and resolve conflicts between mutable-metadata (branches, tags) in a sane way?

Tags are going to be ugly, but branches can simply be merged like any other.

Alternatively you decide the one of the repositories is the primary one, set up a remote called `mirror` and set-up a `post-receive` hook to:

    git push --mirror mirror
Now just ensure no one pushes into the mirror directly. Of course this only works if you control the primary repository.
Wasn't the original point to be able to push to the second copy when the first is down .. what's the point (other than backup) of a second "working" copy that you can't use.
A repo admin or script could enable pushing permissions to the mirror while the primary is down. The when the primary is back up, fast-forward it and change back. Or, just allowing pulling from it and wait to merge until the primary is fixed.
Branches: Let each branch owner deal with that. They likely have the most information about it. Create a new temporary branch, merge both sides into it and see what happens.

Tags: Don't have a process which can result in tags pushed into different places. It's a path to madness. Same applies to master/release branches.

> Same applies to master/release branches.

Yes. This was the case I had in mind. Path to madness.

You can just push and pull directly to/from your colleague's computers. The main advantage (for an established team) of github/gitlab/bitbucket are pull requests, issue management, CI etc., and that's not easily synchronized across multiple providers.
> Now I'm just annoyed that more teams I've been on haven't set this up!

For most people, it would be just a read-only copy. And the value of that is fairly small.

Wait, what?! You can have 2 different urls for the same remote so a single push will push to both?
Yes, exactly. Here's an example for a repository hosted on my server and in Keybase Git. Pulls / Fetches will use the repository on my server. Pushes go to both.

    [timwolla@/s/xxx (master)]g remote show origin
    * remote origin
      Fetch URL: git@git.example.com:xxx.git
      Push  URL: git@git.example.com:xxx.git
      Push  URL: keybase://private/timwolla/xxx
      HEAD branch: master
      Remote branch:
        master tracked
      Local branch configured for 'git pull':
        master merges with remote master
      Local ref configured for 'git push':
        master pushes to master (up to date)
Neat! I was also unaware of this.
Whoa, cool, Keybase should definitely have this mentioned on their own page.
I will be that one, who will remind that git server does not have to be a gitea/gogs/gitlab/onedev/pagura/git-remote-{keybase,s3,codecommit,...} - you might provide a path to the WebDav server[0][1][2], if you need a very simple git server (i.e. served internally over 192.168/16 in office network, let's say RaspberryPi with an external USB drive or temporal repository share from own laptop to your colleagues).

[0]: https://cets.seas.upenn.edu/answers/git-repository.html

[1]: https://blog.osdev.org/git/2014/02/13/using-git-on-a-synolog...

[2]: https://git-scm.com/book/en/v1/Git-on-the-Server-The-Protoco...

Or you can just provide a URL to any host with a SSH server and Git installed. You only need `git init --bare /some/path` on the server and `git remote add origin myserver:/some/path` on clients. If the repo is used by multiple users, you’ll need the `--shared` flag.
TIL how is it this is! Thank you.
Yes, but you can also push/pull from a filesystem location. To be able to push it, it is simpler to init the repo with `git init --bare`.

I've used this on NFS drives, but also SMB shares from windows, and just about anything that can be mounted to a folder. Having an external hard disk drive or usb stick also works.

And lastly, git also comes with a daemon mode which makes it easy to temporarily host a server for a repo. Just connect multiple laptops trough Wi-Fi, and work together (with a pull workflow rather than a push workflow). That's quite useful [1]

[1]: https://stackoverflow.com/questions/377213/git-serve-i-would... Further reading: https://git-scm.com/book/en/v1/Git-on-the-Server

Number of alternatives to popular git repository hosting services looks indeed awesome.

> Yes, but you can also push/pull from a filesystem location. To be able to push it, it is simpler to init the repo with `git init --bare`.

Personally, I have build my very own and simple solution to sync encrypted files over the Internet using git with git-remote that uses filesystem location. The implementation evolved over time but initial idea[0] was to combine restic, pass and git with simple scripts to pull/push the git-remote repo (located in /tmp/repos) to S3 bucket via restic that takes care of upload, deduplication and encryption. Thanks to restic I also don't care much if I'd commit to stale (outdated) master branch, because it uses snapshots and it's quite easy to navigate between them.

[0]: Year-old PoC of encrypted repository share with B2 as a storage: https://gist.github.com/piotrkubisa/dece2fc71399efa56e2d0b8f...

Cool.

But why not just have different remote names other than the default of “origin”? Somebody else on the thread mentioned that it might be a bit complicated to clean things up after an outage on such a “multiplexed” remote.

Is this safe? The Git docs explicitly say not to do this:

> Note that the push URL and the fetch URL, even though they can be set differently, must still refer to the same place. What you pushed to the push URL should be what you would see if you immediately fetched from the fetch URL. If you are trying to fetch from one place (e.g. your upstream) and push to another (e.g. your publishing repository), use two separate remotes.

which seems to imply that weirdness might happen if the two happen to get out of sync, or if one (specifically, the one pointing to the repository you're fetching from) fails.

For something that may be a bit safer, I believe it's possible (but haven't tested) to have multiple values for branch.whatever.pushRemote-- that should do the same thing, and has the added bonus of making the secondary remote easily fetchable.

I don't see how the parent comment does anything different from what's advised there? It's setting two push URLs for the same remote, not a push and a fetch URL. Presumably for fetch you would have a separate remote. I think the idea is that every time you push you push to both.
(comment deleted)
Isn't this the idea of multiple origins? You can already set up different origins, including filesystem origins (like Dropbox).
There's not much more to do than create a quicker remote wrapper. The Git flow here is 2+1 steps: add another remote, push to that remote (+1 is to create the remote). It would be cool to see it built into a Git plugin or wrapper!
> I can't imagine something like this would be too difficult

Pushing to all three isn't that difficult. The hard part is reconciling after one of them suffers an outage or partition.

(comment deleted)
Git is fine, and the outage does not affect you and your team if you already have the source tree anywhere.

What it does affect is the ability to do code reviews, work with issues, maybe even do releases. All the non-DVCS stuff.

t my last company when our internet connection went down a bunch of team members said they couldn't work because they couldn't get too GitHub. They were shocked to learn they could still collaborate by pushing their changes back and forth with other colleagues.

Perhaps what they really meant is that they couldn't get to stack overflow :-(

Its up n running for me now, everything works. Location Sweden.