Pushes are failing for me with this error: "remote: fatal error in commit_refs". I'm seeing it across various repositories so it's not just one particular repo.
Does anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance.
edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people.
edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.
Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).
Also, I don’t seem to be able to properly run it in docker (I’m trying to deploy it on kubernetes), do you, by random chance, have experience with deploying gitea?
Sorry, I don't know about running it in docker. My usecase is personal lightweight repos management, so I just download the binary and run it backed by SQLite. Never faced any issues with that. I never needed to go fancy with docker.
Github has added all sorts of proprietary integration points to prevent it from being this easy, like push based checkin notifications. Then developers get lure by the drag'n'drop CI servers that integrate with them.
Unfortunately people only care about API's now instead of standards.
With each GitHub failure people learn some good lessons. Redundancy is the key. Push to multiple remotes and fall back to backup remotes in your scripts.
It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all.
However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server.
This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.
Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently).
What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.
Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.
How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.
naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.
Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.
"Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth?
The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.
But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.
Not if you're using more than git from github, for example downloading releases as zip files for deployment.
There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.
Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.
> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?
It's not a runtime dependency, it's a stupidity to rely on random
repositories from the internets when building and deploying (remember the
left-pad farca that halted work for half of the web programmers). You should
always have your local copy of necessary sources or use repositories that can
trivially be swapped with some other mirror.
We used to have the infrastructure for those things (in the form of binary and
source package repositories), but it was mainly used by sysadmins, and
programmers usually have an alergy for such infrastructure.
Speaking from an operations perspective, yes, it's a runtime dependency. From a development perspective, you might think of it as a "deploy-time" dependency. Think of it in contrast to compile errors. If your program has errors that your compiler can catch, then it will refuse to compile the program and you must fix the error before executing the program. A runtime dependency is one that may change after your program is compiled -- these are harder to test, especially when the program may fail depending on its environment. In the case of dependencies on services such as Github at deploy time, it's a "runtime dependency" for the operations programs. Since the operations programs are the ones with the runtime dependency, it's also in the operations team's purview (and responsibility) to revise their program(s) and eliminate the runtime dependency such that testing (and reproducibility of) deployment is deterministic and reliable.
>
Your question seems to indicate that you have a runtime dependency on github.
Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.
But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing.
Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.
So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.
> you can instead push your artifacts to an s3 target
This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).
Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.
You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.
Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration and deployment. It's still early days, though, and there's a lot that a new user can get wrong in setting such a system up. Ansible telling servers to git pull from Github may give you better uptime once you factor in your own mistakes.
> then some automated script will tell the cluster to pull from Github
GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.
Could you elaborate on how Kubernetes makes it easier to deploy code changes directly to the cluster? Are you talking about the Docker/container image build?
A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.
Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket.
Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).
You need to either ensure you control the resources you depend on, or plan for outages.
At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos.
Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.
Most items of ours that have a production dependency on GitHub actually point at a simpler (and much stupider) mirror. When commits are made pushes are mirrored to those repos. These repos can even be naively replicated since git's nature allows for this.
If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors.
Also make sure your tools don't lock on only one item.
At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked? Get your popcorn out. It will happen.
For some projects, yes. The biggest well known US examples are the 18F open source efforts and http://code.gov / http://code.mil open source efforts, most of which are public, open source on GitHub.
There's e.g. https://aws.amazon.com/govcloud-us/details/ these days. Azure and Google have government offerings as well. Safe to say neither would be bothering if the government didn't use cloud products these days.
It was in lunchtime in New Zealand, so doesnt really matter. ;) I am just back with our world famous wellingtonian flat white, and looks everything perfect. :)
Reinvent VMS-style clustering on BSD/Linux boxes or just buy some VMS servers. They measured uptime in years to decades for tech that started in the 80's. There should be several, modern solutions like that given companies were developing them a long time ago when I first looked into such things. I just haven't surveyed to see what exists, is reliable, and is affordable.
They could also have a base version of the site that's read-only for source or binaries to allow customers access to the data to fall back to whenever problems happen in more complex functions. Keep it running in some form while they fix things. They might also similarly sell Git appliances w/ high-availability that themselves plug into multiple colo's or cloud providers mirroring changes in the repo to the Github site. Just a wild idea as I have no idea if that's marketable.
You can run a local copy on a server of something like Github. For example BitBucket Server (which used to be called Stash). I worked somewhere that was on GitHub but cried when it was down, then went to BitBucket and cried when it was down. It's a big point of failure.
When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.
Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).
This seems like a pretty common response for a breaking incident for a scale app. Requests flow through to a failing system and trigger HTTP 500. Those requests may pachinko through the stack, making a variety of calls that can compound the degradation of a system weathering an unplanned failure state.
Engineers stop the bleeding by 503'ing requests at the perimeter or putting up a static maintenance page. This allows things like caches or DBs or app servers to cool off while a rollback or a revert goes out. Then, when the system is stable again, let requests flow through again (slowly, of course).
Exactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.
Then what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.
Push and pull to your own server running Gitea/Gogs or Gitlab, and mirror to Github if you feel the urge? Or, if you insist on having your workflow around Github, do it the other way around: have your local server mirror your Github repositories. That way, if Github goes belly-up you can just switch to your local server for the time being.
If your workflow depends on centralization, run your own centralized service. If you can't live without the GUI, Github will even sell you Github Enterprise. Or just run Gitolite or one of the other tools.
You can set up another git server and webhook pull from the main one whenever someone pushes there, so you have a backup replica. Coders just need to setup two remotes for the repos and use the second one when main one is unavailable.
I always thought that an IM-style tool with online/offline presence information of remote repositories could be a fascinating tool for DVCS source control repositories.
Tried to prototype on XMPP at one point in college but never got very far in the prototype.
(That was back when I was a heavy darcs proponent too and had other crazy things like automated consensus branching as a great distributed way to coordinate efforts on such a distributed project, which would work with the darcs push/pull/merge model but not so well in git's.)
Yes, exactly. It's great that we can continue along editing and committing. The freedom that DVCSs bring has really changed the world of development. However, feeling the pain of not being able to access other repos shows both the value that Github brings and also the downside of siloed services.
Centralization is an artifact of human societies (programmers & organizations) decision making process not result of existence/lack of decentralization features.
Fossil [1] has always been my go-to DVCS for ages. Not only can I host it myself. I can also run it when completely offline. It offers Bug Reporting, a Wiki and both a web interface and CLI.
Created by Richard Hipp, creator of SQLIte, it's open source and free, has zero dependencies and works across all operating systems.
And even if you don't want to host it yourself, there's always chiselapp.com [2]
196 comments
[ 3.3 ms ] story [ 77.3 ms ] threadOnce is an accident. Twice is a coincidence. Three times is an enemy action.
edit: confirming, push doesn't work either
edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people.
edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.
If you're really concerned, you run it yourself. GitLab insider a docker container isn't hard to manage. You can even 1-click Gitlab onto a DO Droplet in just a few minutes: https://www.digitalocean.com/community/tutorials/how-to-use-...
[0] https://gitea.io
thank goodness I'm on bitbucket :) One of the things they offer is your own local server synched with their cloud servers for redundancy.
then architect your build system be able to switch to another service with the flip of a switch, if its that important to you.
Unfortunately people only care about API's now instead of standards.
Obviously, this doesn't trivially scale to many repos/users/etc, which is why Github exists in the first place.
However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.
This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.
What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.
If Github being down means you're not fixing your site today, I'd call it a runtime dependency.
(Hint: sans hug of death, because a runtime dependency brought it down.)
Deploying/scaling is somewhere in the middle.
The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
It's not a runtime dependency, it's a stupidity to rely on random repositories from the internets when building and deploying (remember the left-pad farca that halted work for half of the web programmers). You should always have your local copy of necessary sources or use repositories that can trivially be swapped with some other mirror.
We used to have the infrastructure for those things (in the form of binary and source package repositories), but it was mainly used by sysadmins, and programmers usually have an alergy for such infrastructure.
Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.
Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.
So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.
This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).
Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.
Github is an off-site backup, and a nice interface for pull requests and code browsing.
There's just no reason to make github a critical part of your infrastructure.
GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.
Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket.
Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).
At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos.
Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.
If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors.
Also make sure your tools don't lock on only one item.
http://fortune.com/2016/09/02/us-government-embraces-cloud/
Looks like even the CIA is using some AWS.
"U.S. government agencies are moving to cloud computing and away from their own data centers faster than private corporations"
They could also have a base version of the site that's read-only for source or binaries to allow customers access to the data to fall back to whenever problems happen in more complex functions. Keep it running in some form while they fix things. They might also similarly sell Git appliances w/ high-availability that themselves plug into multiple colo's or cloud providers mirroring changes in the repo to the Github site. Just a wild idea as I have no idea if that's marketable.
When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.
Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).
> 18:19 CDT Major service outage.
https://news.ycombinator.com/newsguidelines.html
Engineers stop the bleeding by 503'ing requests at the perimeter or putting up a static maintenance page. This allows things like caches or DBs or app servers to cool off while a rollback or a revert goes out. Then, when the system is stable again, let requests flow through again (slowly, of course).
Tried to prototype on XMPP at one point in college but never got very far in the prototype.
(That was back when I was a heavy darcs proponent too and had other crazy things like automated consensus branching as a great distributed way to coordinate efforts on such a distributed project, which would work with the darcs push/pull/merge model but not so well in git's.)
Created by Richard Hipp, creator of SQLIte, it's open source and free, has zero dependencies and works across all operating systems.
And even if you don't want to host it yourself, there's always chiselapp.com [2]
[1] http://fossil-scm.org/index.html/doc/trunk/www/index.wiki [2] http://chiselapp.com/