174 comments

[ 5.0 ms ] story [ 274 ms ] thread
And so is all Github-dependent deployment...
Got here because of that haha, can't deploy. EDIT: Why downvote fellas?
Downvotes are most likely because – if you are already using git – you should just be able to switch over to a local repo?
Depends how his deployment pipeline is plumbed in. If he's dependent on Travis for builds (for example), he can't just switch.
Depending on services from others for your own business without having a SLA is just a disaster waiting to happen.

Along the lines of "two is one, one is none", anything that you don’t own or where you don’t have a SLA has to be considered not existing for a business.

"Move fast and break things." Including your own bones.

People are getting too spoiled by perceived reliability. Internet is not electricity yet. And especially third-party services on the Internet.

Because your comment adds nothing.
It adds something. It shows an example of an actual person affected by this. Yours here, however, I'm not sure what that adds :)
It shows an example of someone who caught the "why downvotes?" bait, and enables a comment like yours.

INB4: my comment shows an example of someone going meta.

Someone was apparently confused as to why his comment was downvoted. I answered that question.
We're using satis to create a mirror with all the required composer packages and we're deploying from there. https://github.com/composer/satis

The main reason that we switch to that was exactly to avoid deployment issues. Security is also a bonus (in case somebody modifies existing packages).

Probably there are tools like this for all type of packages.

You can also follow githubstatus on twitter: https://twitter.com/githubstatus
Interesting that one of the tweets uses identical wording to one of the status messages, and one doesn't. I would have assumed the tweets would be automated, but maybe there's a thinking that the message for the status bar isn't necessarily appropriate for Twitter, so it can be 'overridden'?
If only we had a version control system that didn't rely on a central hub...
We do. It's called Git.
Now if only we had a convenient way of using a distributed VCS without a central hub.
That sounds like a joke, but that's exactly the problem: the VCS has been distributed, but not all the services enmeshed around it: all the discussions (issues, pull requests, design debates, …).

Github makes it easy to host git repositories, but so do a bunch of other services and it's not that hard to host your own either way or to set up trees of repository or multiple mirrors or what have you. The reason people use a central hub is all the meta-information and exchange about the code and the contributor communities.

The repository on its own has very little value.

And lest you think it's different for the kernel or whatever, a mailing list or an IRC channel are still centralised SPOF[0], when the ML provider goes down so does the project.

[0] though possibly more resilient and easier to switch out of ones

Could Ethereum (or a similar concept) be the solution of this kind of problems?
Together with systems such as ipfs.io, possibly yes. Have a look at https://github.com/whyrusleeping/gx

But there still are many open problems: - incentivised file storage - fair name spaces - search and indexing and more.

What about BitTorrent DHTs?

Or dat http://dat-data.com/

Or Camlistore? https://camlistore.org/

The problem of content-addressing — aside from the opaqueness to "shallow" contributions (nobody is going to install a content-addressed distributed filesystem just to report a bug to your software, don't kid yourself) — is what content-addressing means for resource updates. How do I get the most recent comment on the most recent bug report on one of my projects? In fact how do I even discover their existence from knowledge of the project's existence?
Has anybody tried to store a project's metadata (issues and PRs) within the project's git repository itself? Seems like a logical step.

If you don't want special git semantics around it, you'd have to be clever about how you store it so you don't have conflicts within the metadata. I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.

> Has anybody tried to store a project's metadata (issues and PRs) within the project's git repository itself? Seems like a logical step.

There were a number of tentative distributed bug trackers a few years ago, they sucked and fizzled out. IIRC Fossil is an attempt at an entire distributed project management system, I don't know how it fares.

> I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.

That only works for your own personal project where you're the only user and contributor. Bug reporting by editing a markdown file (or even something actually usable like an org-mode file or an sqlite or BDB file) isn't going to scale very high and is way more effort than most bug reporters (even technical ones) will be willing to put in.

And even if your users were willing to subject themselves to that, they still need a way to send back those contributions somehow.

> The repository on its own has very little value.

Depends for who. When I'm a user of a project, I don't care at all about anything other than the repository. The source code is the only thing that matters.

Anyway, you can't really avoid centralization in most of the projects. Barring some extreme social experiments, a project always has identity, a "canonical version". That's your SPOF right there. Of course we have tons of experience with mitigating potential issues - from caching and mirroring to not doing stupid shit like telling your CI server to download dependencies from GitHub every time it builds your project. Temporary GitHub outages seem to be a problem mostly for people doing the latter kind of stupid shit.

What would be interesting to see though is some kind of torrent network for Git repos. The kind of where I ask for a particular commit of a particular project, and I get a way to download it from some random computer that has it (with all the magic crypto code signing for confirming the code was not maliciously changed, etc.). Node.js developers have local mirrors of third of GitHub anyway, so that would pretty much solve "GitHub is down" issues.

Though it would not solve the problem of people too lazy to have a local cache of their dependencies...

> Depends for who. When I'm a user of a project, I don't care at all about anything other than the repository. The source code is the only thing that matters.

The repository doesn't have any more value for that use case, tarballs would be just as good if not better.

You could have separate centralized servers for each project, though, instead of one big point of failure for all the projects.
That would make the downtime less correlated, but if it was higher in aggregate, would it be a net benefit?
I'm talking about a traditional sort of model, where each project hosts its own issue tracker, code review, etc. Unless you're working across multiple projects, you'd only care about downtime for your own server, not other people's (except perhaps for dependencies, which should be cached locally anyway).
So where's the 'big point of failure'?

Unless you're working across multiple projects, it doesn't matter to you that a GitHub outage takes out everyone else's projects at the same time.

Can you beat GitHub's uptime with a self-hosted solution? Sure. But you'll waste more time setting it up & managing it than you'd lose from GitHub downtimes.

We do. It's called git instaweb && ngrok 1234
Did someone say GitTorrent...?

https://github.com/cjb/GitTorrent

Looks exactly like what's needed. Is it safe enough though? I.e. can I rely on getting a non-maliciously-modified repo when I ask for particular SHA1?

Also:

sudo npm install --global gittorrent

Cool! It automatically installs a local copy of half of GitHub! ;).

> Is it safe enough though?

I'm not so sure about that. I've only used it once, an only to test its functionality, I didn't do any in-depth testing. Well I wouln't use it in production, but in a non-critical, small-scale project it could be worth a try...

> I.e. can I rely on getting a non-maliciously-modified repo when I ask for particular SHA1?

Someone could serve you a maliciously-modified repo, but Git itself would then reject those objects, since they won't lead to the SHA1 that Git knows that it's looking for -- you would keep trying other peers until someone gave you bits that match the correct SHA1.

(comment deleted)
I think he was joking, he was referring to stackoverflow.com :D
We do, it's called Fossil! http://fossil-scm.org/

Fossil tracks issues (tickets), documentation, and wiki pages at the same level as it tracks code, so when you clone a Fossil repo, everything gets downloaded to your local computer. You can then respond to tickets and perform repo maintenance offline, syncing everything to the server when you're back up.

Edit: A few weeks ago I got a job a one-hour train ride away. I can commit, close tickets, and write docs on my way in, sync everything to the internet when I get to work, then do the same on my way back - even though I don't have a net connection during my commute. It works great. (Ok, not as good as not having to travel at all, but great nonetheless!)

the parent commentator was joking that git _is_ decentralised, but that we made it centralised again by relying on a small number of hosters (github, bitbucket and others)
Heh, don't worry, I got the joke. But I also got the joke the last time this was posted, and the time before that, and the time before that...

This just keeps happening. It's not GitHub's fault that they keep getting DDoSed or have a network outage every now and again. But it's not a problem with no solution.

To be honest, I can cope with Github's downtime, it is probably less than if I was hosting a Git solution myself and the service is much more reliable than all the other solutions I've tried (which are either lacking features or have bugs or have massive downtimes).

Github is trying their best as far as I can see, they manage to fix the majority of issues within three hours and even during those three hours its not the end of the world because you can still code.

Most developers will not be able to write code without being able to look up stuff on the Internet.
<3 Fossil. I wish it were more popular so I could use it on non-personal projects and still have a hope of attracting collaborators...
Yeah, fossil is all I use. It's so clean, tiny and self contained with no install or bloat. I've always been in love with software written that way. It makes me look at things like git as just a product of engineers that can get the job done, but don't know what it means to write good software or use imagination.

Being fair, fossil performance on giant scale projects might not compare quite as well against git or proprietary solutions like Perforce, but for the large majority of projects I expect it is more than adequate.

> but don't know what it means to write good software or use imagination.

You can't do patches. You can't rebase. You can't rename files.

Tell me all about this good software and imagination you speak of?

> You can't do patches.

You can create bundles and submit them for approval. Very similar.

> You can't rebase.

You have alternatives. The work philosophy is different, though.

> You can't rename files.

False.

>You can create bundles and submit them for approval. Very similar.

Bundles do not have anywhere near the same flexibility and nuanced capacity that patches do.

>The work philosophy is different, though.

That's an apology if I've ever seen one. Again, not the same.

>False.

Rename a file. Then try to merge. Tell me what happens.

>Bundles do not have anywhere near the same flexibility and nuanced capacity that patches do.

Interesting. That's basically what the people who suggested bundles (a Hg feature) said about patches.

I've used both as appropriate. Handy that Fossil has both.

Rename: https://www.fossil-scm.org/xfer/help/rename - Merge only merges content, but the content does get updated between renamed files. Also note that while rename and mv work similarly, only mv can be used with --hard to also rename the actual file on the drive. If you know of an issue with renaming, you should report it on the bug tracker.

Patches: You can select any 2 commits and create a patch between them ( http://fossil-scm.org/xfer/vpatch?from=cbda43e71bda52f0&to=e... ). If you want to apply the patch, you can just use patch. Whether patch itself should be integrated directly into fossil is I guess something people can argue over. I don't use patches often, but haven't had issues with them.

Rebase: This is what Hipp has to say - http://permalink.gmane.org/gmane.comp.version-control.fossil...

These things may not be exactly how you prefer them, but you're misinterpreting my point about good software and imagination. Almost no software in existence is perfect for everyone in every scenario, but that doesn't also mean that no software is good.

Fossil leaves me with no reason to use it. What it can do, Git and Mercurial can do better. Saying that no software fits the use case for everyone, everywhere is completely irrelevant. Other software not fitting into every mold completely, while fossil doesn't really bring anything to the table to cover what other solutions don't, doesn't really justify its existence as a solution in any way.
If you need something that works in exactly the way you're used to and that's how you evaluated fossil, then I guess fossil isn't what you need, because fossil is different. I'm not sure why you're being toxic and venting frustration, though. Use whatever you like, these aren't football teams. :P
>I'm not sure why you're being toxic and venting frustration

Please do not devolve the conversation by making outlandish statements that have no bearing on the conversation.

>If you need something that works in exactly the way you're used to and that's how you evaluated fossil

That's the exact opposite point I made though. Fossil constrains me in how I work and how I accomplish certain tasks -- with no benefit. Neither Git or Mercurial do that. What Fossil can do, both Git and Mercurial can do better. So why would I choose Fossil?

They're not outlandish statements, you are frustrated.

If you don't get any benefit out of a tool, then don't use that tool. I get value out of it and am not meaningfully impacted by the issues you have with it in practice.

I'm not a salesman and it's free software, so do what makes sense to you.

>They're not outlandish statements, you are frustrated.

As an expert on myself, I must disagree.

>If you don't get any benefit out of a tool, then don't use that tool.

I don't use it. But I think you need to remember what I replied to, as that's an ill-placed sentiment in this thread.

> Fossil tracks issues (tickets), documentation, and wiki pages at the same level as it tracks code, so when you clone a Fossil repo, everything gets downloaded to your local computer.

Issue tracking and wikis are not a version control problem. Sure, their data can be version controlled. And documentation has been version controlled since Sccs, why would it require special care?

> why would it require special care?

It doesn't, really, but it's in the same place as the code and everything else (this is what I meant as being "on the same level"). You don't need to clone the code and then clone the tickets, for instance. When you run "fossil timeline" on your local machine, you'll see entries for tickets being closed and wiki pages being updated interspersed among the actual code commits. This is useful to me as they're as much a part of the project as the code itself. Does this answer your question?

No, unfortunately. How is this different to and / or better than having a "wiki/" subfolder, and, e.g. an Org-mode file (need not be org-mode, even a plain text file is o.k., and there are wiki systems and todo managers that use plain text) for issues on the project root? I can easily "git/hg/cvs log project.org" to see the history. And the format is not tied to the VCS software, so, if I switch to sth. else, it's still perfectly usable.

An basic example of an org file for tracking issues:

  #+title: Project issue tracker
  #+description: descr...
  #+todo: OPEN(o!) | CLOSED (c@)
  
  This is the issue tracker for Project.  In order to add a
  new issue, add it to the top of the entries as a plain org
  entry, then hit =C-c C-t o= to mark it as open.  In order
  to close an issue, on it's headline, hit =C-c C-t C=,
  this will prompt for a closing comment, see the buffer that
  pops up for instructions.
  
  Common tags: docs, features, bugs, security, ideas
  
  * CLOSED Write README. :docs:bookkeeping:
  - State "CLOSED"     from "OPEN"       [2016-03-22 Tue 02:16] \\
    Written.
  - State "OPEN"       from              [2016-03-18 Fri 22:33]
  * OPEN Implement new features :features:
  - State "OPEN"       from              [2016-01-27 Çrş 23:35]
Ah, I see. In that case, you're right, it's not: if you track issues and docs like like in the same way as you track source files, then functionality-wise they're no different.

The differences that remain are purely in the interface. For example, although they're synced the same way, VCSes tend to display issues differently than source; it's easier to filter out issue changes from the timeline, and harder to accidentally delete other people’s comments. It also makes it possible for people who aren't actual members of your project to raise and comment on the issues, which wouldn't usually be possible (unless your VCS can manage permissions for each file).

One particular use case of yours that Fossil does support is the ability to have a "wiki/" subfolder and have the web interface automatically format and display the pages inside, so you can link to them more easily if you're using the HTTP server for hosting. I think I use this feature more than the built-in wiki because it’s possible to version the docs along with the source, ensuring they never go out of sync.

Is it easy to fix merge conflicts on something like an issue?
Argh, I don’t actually know, and I probably should. I’ll ask on the mailing list and reply with what I find out.
Oh, I agree. But I still think Git, even over IPFS, has significant failings. Mainly, how do I share the issues and other problems that Github has did their own GUI for?

Fossil SCM seems to fit that bill. As another user said, they can address issues in a local, non-connected manner, and then merge the issues back to main.

The last piece would be to use IPNS so you can have the same key for a changing package. I just hope versioning support gets added in soon (it's in the spec, just not done yet with go-ipfs).

> Mainly, how do I share the issues and other problems that Github has did their own GUI for?

I think this should not be done in the same system as where people store their source repos. Mixing the two is convenient, but can stifle innovation. Better to have flexible and well documented abstractions and good separation of concerns.

> Mixing the two is convenient, but can stifle innovation.

Could you elaborate?

I think I've heard the argument before, but I don't really see how. These outages show up again and again, partly because people cannot access the project issues.

Well, the argument holds for any big system. It is better to keep things modular and simple, so anybody can work on each of the components (in parallel), and improve each of the components, and replace them at will.

Also keep in mind that protocols are much more important than the actual code implementing them (see git for a good example!)

I guess I'm still not following.

If I sync a git repo from random location, I have the files to make that package. However none of the documents that aren't explicitly in the git repo are provided. With Git[hub/lab/...], this is metadata outside of the repo. Ideally, I want and need that as well. If there's an issue, I can research it quickly and see if it applies to me... Or if I'm a maintainer, I can make a solution offline and then merge it when online.

I understand that it may not be the perfect tool to include everything... but it does seem to fit the bill. Fossil does seem to do this correctly. I'll have to look into it.

its better to use fossil locally and have a mirror sync to github to run in the background. Its sad that git being distributed and github being a central place , sort of undermines the very purpose git was built for
The Web UI is still accessible for me, just extremely slow.
(comment deleted)
You'd think by now GH would have a more resilient system in place ... I mean it isn't like it is the first time it happens to them.

Thanks for the downvote, my hearts goes to you. :/

I guess you will get what you pay for... which is nothing for most of their users.
> I guess you will get what you pay for...

1. There is literally no amount of money which will get you 100% uptime. You may be able to buy guarantees of 100% uptime, however...

2. ... which will not change the fact that things still go down, and that if something goes down in a way which violates your signed contract, the most you can hope for is some kind of monetary compensation and, maybe, an apologetic-sounding employee to scream at until you feel a bit better. If you expect to buy someone's care, you're woefully naïve.

My point is that the paying customers don't necessarily get a better service. All they get is limited compensation for enduring the same bad service all of the rest of us get.

It's true that you can't get 100% uptime. And yes 99.999% is more of an SLA agreement/goal than a generally achievable metric. But Github aren't even managing two nines - they only have 98% uptime over the past month. That's bad for any service.
Ok, but I thought the business model here was to be the facebook of code repository. And I don't see the later suffering as many outage as the former (maybe because I am not all day long on FB, mind).
Because people aren't DDOSing Facebook by telling their CI servers to constantly refetch all the dependencies every goddamn build.
Hm.

Git clients probably supply some anonymous but consistent identifying information when doing a standard pull... per-client rate limiting, anybody?

It's a nice idea[l] at least; practically speaking there's enough bandwidth to keep the current disaster ticking over, so... :(

NPM and Composer only check for updates and cache locally.
...which is why we switched to a local instance of Gitlab: stuff still works, no matter whether Github is in business, whether their servers are up, or even whether our internet pipes are up. CLOUDCLOUDCLOUD!!!
And when you local instance did not work becaus of a disk failure? BACKUPBACKUPBACKUP!!!
Haven't had a disk failure kill a server for years (spoiler: RAID); and we do follow backup SOP, thankyouverymuch. Moreover, this is far easier to manage than "Github is down, we don't know anything - what happened, why and how, what is affected, when (or whether) it would come back again."
(comment deleted)
(comment deleted)
it's "working" now , still very slow
SourceForge is doing fine ;)
And so are my free private repositories on Bitbucket. :)
and my two independent gitlab instances.
I'm guessing your gitlab instances see as many visitors as SourceForge
That's why I call it private.
And my local copies of all dependencies I use.
keeping a server running with a load average of 0 is easier than keeping a server running with a load average of 40.
GitLab to the rescue!
Momentarily, yes. But it's just another central hub.
You can host your own.
Which can go down too.
It's turtles all the way down.
Still, a lot less risky. Self-hosted hub is something you have more control over and is less likely to be DDoS'ed by some Chinese harm-wishers.
Not just that, but if it's self-hosted, that also implies that it might be local to your work-area. I.e. It's on-premise, so even the entire internet going down wouldn't necessary stop local operations.
But you can do something about it then.
With enough copies you can push the probability of all of them going down simultaneously to arbitrarily low levels.

Or at least down to "fuck, the power grid went out" levels.

Or gogs, which is actually simpler to manage.
(comment deleted)
Haha, I always feel guilty when this happens, I'm doing something and suddenly start getting errors, which just escalates on refresh and now everything is down :P
has any one tried setting up a fossil-scm locally and work with that instead
Yeah. I've been using it on and off for a while, but git works fine locally, so it was mostly for private bug tracking.
we need a gittorrent.
storj can be a nice way of making that possible
No. Git is easy to distribute, that's pretty much the point, and even if your central repository is down you can generally work: you can easily commit locally and can even use ad-hoc servers for P2P collaboration. Anything based on your VCS (CI, deployment) can be rerouted or handle multi-master sources.

The problem is all the shit that is not in the VCS and not directly based on it: issues, code reviews, discussions, … and the community around them if you have external contributors.

"Gittorrent" attacks a problem which by and large is already solved, it might further improvement on it but it's improving something which is not the bottleneck.

I hope the world realizes that clouds are not just fluffy white things in the blue sky, but can create destructive storms too, with wind, rain and so on.

Make sure your infrastructure is ready to weather such storms - have local backups and fallbacks, make sure your code is available even if github (or the Internet) goes down.

The more I hear 'cloud', the more I try to depend only on tools that work offline.

Metaphors like that only makes it even more confusing. It's a marketing scam, let's call it for what it is. Servers. They are servers.
There can be difference between hosting on bare metal boxes and hosting on cloud providers.

If you are connecting different services (message queues, databases, storage, transcoding), and spinning up and down instances based on current demand and spot prices, then cloud hosting is totally different. On the other hand, if you are just renting a couple of dedicated cloud instances, then yeah - they're just (virtualized) servers.

Yeah. The problem is, "cloud" now means both of the things you described and then some. Probably the best definition corresponding to present usage of this word would be "not on localhost".

(Because yeah, if I host a Dropbox clone on my Raspberry Pi sitting on the wardrobe, I have my files "in the cloud" now.)

We need a new word for automated, dynamic allocation/deallocation/management of remote instances for various services. Something that would differentiate it from buying a VPS.

Agreed. Amongst non-technical people, 'in the cloud' just means 'somewhere on the internet'. For example, if I backup my iPhone onto iTunes on my laptop then that's not on the cloud, but if backup to iCloud then it's on the cloud.

I guess it's frustrating that cloud means quite different things depending on the context, but I guess that's true of a lot of language.

It's one of the most successful marketing scams I've ever seen - the whole tech world bought it, and here I was thinking at least engineers are mostly bullshit-proof.
Cloud have silver linings. They also evaporate or blow away without warning.
Well, then you need RainCloud.io™!!

It's the cloud-based offline tool storage platform that gives you access to your offline tools...from the cloud!!

Maybe it's time to use Git with Storj ?
Sorry github, I know my code is unbearable to watch :(
I pinged GitHub and it seems to be responding but the web UI is taking a few minutes to respond.
(comment deleted)
Facebook and Twitter seems to have sorted their once frequent outages pretty well over the years. Come on, GitHub - you can do better!
Taking bets on if this is the chinese government testing out GreatCannon v2.0.0
Once the office router was down, we just pushed our commits to a USB stick. Worked pretty well.
GitHub seems to be back up again now.
(comment deleted)