Tell HN: Read up on your GitHub Support SLA
* "Check again." If the turnaround on a support ticket is truly 11 days, we would have been facing a 22 day outage (as we'd expect yet another 11 days after responding "it's still happening").
* "You have no SLA."
* "If you want support, I can direct you to our sales team."
If, god forbid, Git access (or everything) had been down, we would have been scrambling to continue business, we wouldn't have had a number to call, and we likely would have had to pony up the cash.
I strongly recommend you take a good look you fall here: https://github.com/premium-support. Note that "< 8 hours" under "Enterprise" mean absolutely nothing, as they aren't guaranteed (per your contract). A more honest value in the column would be "N/A."
You have to get hold of sales to learn about premium support pricing, it isn't publicly disclosed. This is likely to prevent you from budgeting for premium support only in the event that you need it.
If your business continuity depends on GitHub Enterprise, and you don't have Premium support, you need to pay, plan, or change.
139 comments
[ 2.7 ms ] story [ 218 ms ] threadDon't know anything about Drone CI, but it seems to me like a lot more work to manage two separate systems than it is to manage a single Gitlab instance, where everything is already nicely integrated.
I'm not a sysadmin by any stretch of the imagination, but I was still able to set up Gitlab, Gitlab CI with a bunch of runners, Gitlab Pages, and offsite backups with very little effort on my self-hosted instance. All I had to do was edit one config file to enable the various services (backups were with borg and systemd via the "gitlab-ctl" CLI)
Been running that setup for probably 2 years no with the only outages being external internet/power outages. I'm the only user, so my usage isn't at all going to reflect OP's needs, but it's still more reliability than I ever got from Gitea.
Guaranteed SLA?
No
But it doesn't say so on the page you get to if you select "Pricing" from github's homepage. Nor, as far as I can tell, does it say so on any page linked to from that page.
Standard Support - GitHub Support can help you troubleshoot issues you run into while using GitHub. Get support via the web.
Premium support is listed as "available" (so not included):
With Premium, get a 30-minute SLA and 24/7 web and phone support. With Premium Plus, get everything in Premium plus your own Support Account Manager and more.
Off topic, pedantic, but still...
That is horrible writing. And that's not even going into the "and more".
Just today, I've been asked for what availability for a meeting I have between the hours of 9pm and 6am (and no, that's not a typo!).
You can't paste errors into the ticket. Their support system doesn't even support all of printable ASCII. The reps don't read the ticket before responding. If you can't replicate the error in front of their eyes, it doesn't exist, and even if you can, they won't believe you. I've literally had them defend 56 kpbs as being "good enough throughput", I've had "What is your ISP?" on "we get 500 Internal Server Error from $service" (and … our ISP is you, Azure…) … yeah. The list of idiocy we've seen just goes on and on and on.
Their support team wants to talk about "how can we architect a cloud system that'll blah blah enterprise blah", I just want valid API calls to succeed. I'm an eng, I can do the building, I just need APIs that aren't hot garbage.
(The outage I allude to in the above comment is actually mentioned in passing in [1]. I had to link to that blog post in the support ticket! (So thank you, Scott Helme, for writing it.))
[1]: https://scotthelme.co.uk/lets-encrypt-root-expiration-post-m...
Why would you expect TLS to work correctly on a HTTP load balancer that can’t do HTTP properly either?
Microsoft App Gateway is incompatible with Microsoft’s own SharePoint server for example because it doesn’t send a user agent header in the health monitor probes, which about 50% of all web apps require. Similarly it can’t send authentication headers either so it can’t monitor web apps that enforce security.
It’s a security product incompatible with security and encryption.
Let that sink in and then look at what you’re paying for this on your next monthly bill.
Well, it doesn't exactly advertise that on the tin, now does it?
And we're not using it as a security product (I know there are people out there after "WAF"s and … stuff … but that's not us); AFAICT, it is Azure's offering equivalent to AWS's ELB.
(It is tempting to remove it from the architecture, but unfortunately, we're integrating with a third-party in this case that wants it that way. Everywhere else we need an HTTP proxy we use nginx…)
Speaking of Front Door: it's a CDN, TCP accelerator, and TLS offload performance product that slowed down web site performance in every test that I've ever done with it. It doesn't even begin to approach the features of some of its competitors. For example it still can't do HTTP/3 or TLS v1.3. Or Brotli, or ZStandard, or multi path TCP, or anything really that helps performance.
Communication matters a lot. A response within 24 hours means they would be ten days ahead. Yes, that is better than being 11 days behind.
Waiting is what the customers do, the provider needs to communicate, at the very least, that they are aware of the problem and looking into it.
EDIT: a Threadripper will do for CI. Quick as you like.
Replacing all that with something as scalable, flexible and agreeable with potentially thousands of global developers is far more than '15 minutes' of work. Several orders of magnitude more.
Even on the git repo question alone, if you're an enterprise of some size, you'll have hundreds or maybe thousands of repos that could be potentially gigabytes in size (for any one repo) for code alone. Moving to a self hosted solution requires far more than just throwing some threadrippers and enterprise drives at the problem. And that's assuming the best outcomes.
A competent UNIX sysadmin would be the one yelling not to throw the baby out with the bathwater here, because they would know just how hard this stuff is at scale.
Pop in a self-hosted GitLab install, configure SAML or AD auth for SSO. It's all GIT so importing all commits (and not losing history) isn't hard - just tedious.
For testing pipeline, use Selenium on a 32 core threadripper running linux, with 1/4 TB ram. You can get upwards of 400 headless chromes on that.
Throw in NodeRed for overall process automation (think: tying in disparate APIs with a low code environment).
I've done this, with exclusion of the selenium checks themselves (there was a qa team for that), in like 2 weeks.
Just throwing up a server somewhere running git and a few software packages is nowhere near the same thing.
https://github.com/pricing
https://docs.github.com/en/enterprise-server@3.2/admin
Active/passive HA is possible: https://docs.github.com/en/enterprise-server@3.2/admin/enter...
Spin up a Ubuntu machine and install the Omnibus and you have the basic functionality running in about half an hour, plus another half an hour for the CI Runner.
If you already have internal infrastructure and a moderately competent operations team for that infrastructure, the calculus for you may be different. Blindly assuming that I’m wrong is not a sign that you’re aware of the tradeoffs.
Say you don't understand opportunity cost in software development without saying it.
I won't delve too deeply on the obvious: most "competent Linux sysadmins" have a very over-inflated sense of their own skill set, and tend make for toxic team members.
Most software development shops are in the business of developing their particular software, not deploying and self-managing DVCS, much less hosting, monitoring etc...
Sure, could one person set up a Git/GitLab system? Absolutely. Can they operationalize it effectively? Not really... the bus problem is a thing and anyone that thinks tying the entirety of a system's uptime to one individual is an operational improvement over GitHub's outage SLA is deluding themselves.
And it's definitely possible to run gitlab or any other git hosting solution on-site with little downtime. There's no magic or arcane knowledge involved. It just takes serious effort to do so - more than a single lone wolf sysadmin can provide. All their skills are worth nothing if they're sick and in hospital or on a beach holiday.
Maybe, but also maybe not. And then that still doesn't mean I want them to focus on running git/gitlab. I mean we're doing stuff that revolves around the rust compiler and we have operations people easily capable of running gitlab around, but their primary task is something else - they're building systems on top of that. Do I want to re-task - or even just side-track them - into running gitlab?
Once you reach a certain size, you can have an internal ops team that's responsible for providing internal infrastructure, but to what extend is that really different from giving github/gitlab money? They'll be about as far removed from the individual teams they're serving as github is. Is that really something I want to put organizational effort into, distracting the org from achieving the goal? It's all tradeoffs.
When you pay someone else to handle your data, there is a lot that can go wrong. GitHub could go down, they could lose (or corrupt) your repos, they could accidentally delete your account. The nice thing about git is that it's absolutely trivial to clone repositories. There is _zero_ reason not to have a machine or VPS _somewhere_ that does nightly pulls of all of your repos. When Github goes down, you'll lose a lot of functionality, but at least you have access to the code and can continue working on the most urgent things.
I'm not clear on the details but OP's issue seemed to be around a broken CI system. At its heart, CI is just the automatic execution of arbitrary commands. Every repo (or project consisting of multiple repos) _should_ have documentation for building/testing/deploying code outside of whatever your CI system is. If your source of truth for how to use your code is in the CI system itself, then your documentation is very lacking and yes, you are susceptible to outages like these.
¹e.g., the devs "don't" have access to the credentials, except they're in the CI workflow, so technically they do. But I've worked at a number of companies where security will happily bury their head in the sand on that point.
Because if the things you really need actually keep running when your provider(s) disappear or go down for an extended period of time, you're running them locally anyway, and might as well get the benefits of that effort all the time.
Many orgs have difficulties with escalating issues outside of the support organization. AWS never _ignored_ a ticket I opened.
Good fucking luck getting them to call you back or respond if you aren't adding 1000+ seats. It's unbelievable.
There was recently a thread about someone asking about password management solutions for an organization of 10k employees, and someone recommended 1Password, which I already find hard to believe it could ever make financial and operational sense. [0]
Now, these prices from GH can make sense if you are a team of 10-50, but I wonder what would be the point where it wouldn't take a bean-counter to say "can't we just hire someone else do that in-house, part-time? It would cost as much or less, but at least we would have a lot more control over the system and we wouldn't be at the mercy of a company that treats us as cattle."
I see on your profile page you are on Matrix. Would you mind if I add you there to chat (or add me @raphael:communick.com)? Seems like your company could have so many different cheaper and better solutions for their needs, I've been thinking for a while whether there could be a market in "open source strategy consulting".
[0]: https://news.ycombinator.com/item?id=31582161
Ha, these costs are a DROP in the bucket for businesses at 10k employees.
> "can't we just hire someone else do that in-house, part-time? It would cost as much or less, but at least we would have a lot more control over the system and we wouldn't be at the mercy of a company that treats us as cattle."
Good luck with this. You're missing things like:
- Maintaining the software (upgrades, etc.)
- Maintaining the hardware (you buying bare metal? cloud? who's upgrading servers? etc.)
- Hiring/firing costs
- Do developers want to work with a system that isn't GitHub and be productive?
- etc. etc.
Even $20 a month is not worth wasting time over. It is a margin of error compared to the salary of that user.
1Password for 10k users? What would that be, 80k/month? There are probably pretty big volume discounts too.
Payroll for 10k people is like $30m-70m.
Not just that... if I'm working at the IT department of this company (surely they have one) and hear about such deal, I'll have three thoughts:
1. Can I do it myself? Give me a raise and I'll take the extra responsibility. Everyone wins.
2. If they are throwing this much money out of the window, I'll go knock on my managers door and ask for a raise.
3. If they say they can't give me a raise for any bullshit reason, I'll immediately lose trust in upper management and I'll start looking for a job the next minute.
I think you're not doing a very good job of updating your expectations based on what other folks are saying. Across multiple threads you have a whole variety of folks trying to explain their reasoning, and you're dismissing them all out of hand. You don't seem to be giving much consideration to the idea that you could be wrong in at least some reasonable organizations (rather than all the folks you're replying to).
Ownership is a burden, ownership at scale is a nightmare. Paying a third-party to own something for you is fantastic value in the default case, until you have a strong business case for taking on that ownership burden. “One of our nerds says he’ll own it for 50% of the cost” is a terrible option, it’s nightmare waiting to happen.
If I had 10k people, and I could pay $50k to offload ownership of some critical infrastructure to a third party, I wouldn’t even blink. That’s great value.
Well, this is still called Hacker News. Am I in the wrong place?
Anyway... you've created many strawmen here, where should I start?
> you decide to leave? Then what?
My "hiring" for it would imply defining a proper budget for it and a set of conditions negotiated a priori. It's not really "I will just do the job myself", I'm talking about "ok, you are willing to pay $80k/month to have this solved. Here are the 5 other different plans and solutions that we can implement and that will cost less than that, which one gets the go-ahead from upper management?"
> make a mistake that takes 24 hours to fix: that’s thousands of people unable to work
It's still coming out ahead of Github that took 11 days to solve?
Also, what is that joke about HackerNews overloading with traffic whenever there is a github outage? Or that one of how half of the internet GDP being tied to AWS?
Seriously though, the answer would be "you don't migrate everyone at once". You'd start with these migrations on a project-by-project basis, starting with the less critical projects on the new system and slowly weaning off on your dependency of the big vendor.
Bonus: by migrating your systems you will have some kind of redundancy. If GH goes down, the teams could use the opportunity to move to the new system. It it works, the teams gain confidence and can accelerate the migration. It it doesn't, it becomes an opportunity to learn something out of a sunk cost.
> If I had 10k people, and I could pay $50k to offload ownership of some critical infrastructure to a third party
Paying $50k is not giving you any guarantee that your business is robust. You are just paying for CYA.
There’s not much more I can say because you’ve just outlined an operationally expensive strategy without appreciating the costs. I recommend spending some time in a big organisation, it’s hard to appreciate the enormity of the challenge in orchestrating people until you’ve witnessed it first hand.
Amazon, for example, has thousands of employees dedicated just to running systems to support the other employees! And that’s out of necessity, because third-parties cannot meet their needs — if a third-party could, Amazon would absolutely use them in a heartbeat (as they do already in many cases).
After my time at Amazon, I gleefully pay for 1Password for my org because the thought of what you outlined, in a growing org, would keep me up at night.
My very first comment started with "I'm really not cut out to work for a big company". I am well aware that I do not want to do this.
> After my time at Amazon, I gleefully pay for 1Password
But that's exactly the type of cultural issue that I am talking about. So many people going to work at FAANGs and when they leave they think that mentality should be applied everywhere.
The first issue in this thinking is straightforward: I know that in your mind your company is the greatest thing in the world, but it is not Amazon. YAGNI applies beyond software development.
The second problem: FAANGs can operate like this because they are making so much money per employee that it simply does not matter to them. But this mentality when applied to a smaller company, can be the difference between 6 months or 2 years of runway. And every time that a company outsources this is a missed opportunity to learn how to do it more effectively. Instead of thinking "this would keep me up at night", I'd rather think "we are doing it the hard way, but that makes us more resilient and increases our chance of survival".
The reason businesses like Amazon can be successful is because they focus on what matters, and that’s what startups need to embrace too.
It's 1Password, it's Jira, it's Github, it's Salesforce, it's Tableau, it's Google Docs, it's Dropbox, it's Figma. It's all the services that have viable alternatives, but you just don't want to try out because... it's easier to think it's not worth it?
> The reason businesses like Amazon can be successful is because they focus on what matters, and that’s what startups need to embrace too.
That says more about our different views of what constitutes "success" than anything.
But then I became the boss. And we pay for many of the services you listed and more (google docs, gitlab, tailscale, 1password, JIRA, zendesk, Okta, gusto, quickbooks, etc, common tools used by startups). We have 11 employees.
I’ve spent more time typing this comment than the time I’ve ever spent wondering whether they’re worth the cost.
And they all total probably $200 a month per employee or more. And they still pale in comparison to our payroll.
They’re all no brainers especially given how small we are.
Those tools allow us to perform at the same level security and compliance-wise as much larger companies. And they liberate our time so we can focus on adding value to our customers rather than futzing around with unstable internal systems.
Again, I get it, the 80k number seems like a lot. What we’re all trying to tell you in different ways is that 80k is _nothing_ to a company with 10k employees. They probably spend that much on stationary and cleaning supplies.
My argument is more against the former than the latter. And it's not just about cost. It's about lock-in. It's about shitty customer support that takes 11 days to respond to a ticket. It's about building your systems on the "no-brainer cloud provider" that leaves you empty handed on the semi-yearly outage, and all you can do is pray to be resolved it quickly and console yourself that your competitors are offline just like you.
The point on the second issue was not just "they could do it in-house and cheaper". It is also "they are paying this much money and they are still in a weak position, where they have no control about some critical piece of the organization. I would understand if someone takes this route as a temporary measure while better processes are being developed and putting in place, but if this paying $1M/year is accepted as the "natural way of things" of things, it seems like management is saying "we are incapable of doing it ourselves, and we are too lazy to even care. Let's just hope that the money keeps coming, and if doesn't we just lay some people off".
That way I have a path to migrate to a different product. That’s the true lock in for me.
Never noticed any difference or cared between gitlab being “open source” vs GitHub being closed source. As long as it’s an unaltered git repo and there are apis for my data.
Depends on the structure of your org. Is the 0.5 FTE being able to maintain addons for all the browsers, dealing with on and off boarding, multiple apps for multiple platforms, write end-user documentation at the same level as 1Password does?
Contribute $800/month to the vaultwarden developers, one year later you (with the help from all the other companies that understand the benefits of open source) will end up with a FOSS product that can be as good or better than 1Password: https://news.ycombinator.com/item?id=31582369
I'm not sure who's the bean counter here. Maintaining a service, with the same availability as Github, is 3 FTE minimum, unless you are expecting your 0.5 FTE to also be on call. "I can hire an intern to replace $X" seems like the bean counter attitude. Do you have an SLA? Can management claw your raise back when your bespoke system has more than a day of downtime? If you go on vacation and the system goes down, can management expect you in the office on the next day? Remember they are paying you an $80k/yr "bonus" for three nines of availability. Or are you just going to tell the other 10k employees "hey it won't go down while I'm in Tahoe, just trust me."
Keeping Github is a complex issue, because they are trying to offer the same service to millions of people. It is a totally different beast than having to manage a service to thousands of people. Github needs multiple datacenters in multiple regions. A company self-hosting needs one server and can handle thousands of users. A lot less complexity, a lot less moving parts.
> when your bespoke system has more than a day of downtime?
What is "bespoke" about Gitea? Or Gitlab if you want an all-stop-shop solution? They can also do all the enterprisey things that one expects from Github.
Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.
The same as the cost per hour of GitHub going down, I agree, but GitHub will have 20 people working on fixing it. We'll just have to wait for you?
What do I tell my 10k users?
How about security? Are you going to manage the provisioning of those users, the access control? How about audits? How about scaling? Your magical .5 FTE will do all this on top of your daily duties?
The calculus is never "just give me a raise" – it's not paying 1Password 80k vs paying some IT person that much. It's just that that money supports a core function for 10k people.
A former-boss once told me when he became a CTO of a very large company – he said, "at our scale, most of our conversations revolve around risk rather than cost. we even start to wonder, will this vendor be able to cope with our level of demands? what if they go out of business? what if they want to quit? what is the cost to our organization in time and money to have to switch to a different vendor/product/etc?"
Who knows, a company of that size might save half that money in cybersecurity insurance premiums just by adopting 1Password. (guessing here...)
It's just way more complex than you think.
Already answered in another place. Github is infinitely more complex to keep up than an internal company service. If Github goes down, can be for a few hours. Or worse, it fails in a way that affects only a handful of users and it takes them 11 days to respond your support ticket.
If an internal service fails, the sysadmin (not the "guy who was brought in to setup gitea") can run a troubleshoot playbook and in the worst case can have a whole new server from the latest backup running in some minutes. Even better, your internal monitoring service detects that the service went down and it can be reprovisioned automatically.
> Are you going to manage the provisioning of those users, the access control?
Sigh...
A company at this size would have already a SAML/LDAP/AD/SSO solution for their other services. You just integrate your service with it.
> most of our conversations revolve around risk rather than cost
I don't disagree. I just think that is just a way to say "I want to pay to buy some CYA".
I fully believe that someone managing a larger organization has other things to look at in terms of priority, but at the same time I think accepting this blindly leads to some complacency.
It's like software developers who are so used to their top-of-the-line workstations hooked to their 3-monitor and fiber internet who forgets that their users might be connecting to their website through a $50 phone on a 3G connection.
There are other ways to reduce your risks. Prioritizing open source is one of them, as that gives some sort of "built-in" protection against vendor lock-in. Giving more autonomy to your departments so they can independently choose their IT solutions to eliminate the chance of systemic risks. Adopting a solution that let's you both outsource or bring it in-house without expensive switching costs. And all of those could be applied no matter the size of the organization, yet it seems that everyone wants to think they are some Silicon Valley unicorn and feel justified in spending ~$1M/year in a password manager.
Just because you have one server doesn't make you immune to downtime. If the server goes down at 8am EST (5am PST), who is on call to fix it? Or is the server just down for 3 hours until you get into work (assuming you live on the West Coast)? That's what I mean by 3 FTE. If this is core to your company and you require 24/7 uptime, then either one person is on call 24/7 or 3 people doing 8 hour shifts.
>Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.
And if it happens at 1:30am? Or if it happens while that Sysadmin is on vacation? I don't think you are considering that your solution has a bus-factor of 1 for a service that is depended on for a 1000 other people. The idea that an $80k/year cost is preposterous for a core service only applies to people who think humans are fungible, cheap robots who don't sleep, get sick or take vacations.
I'm not saying Gitea or Gitlab are bad products. Plenty of companies self-host Gitlab with their own teams. But the idea that it costs _much_ less than Github Enterprise once you get company sizes of 1,000+ is absurd. We haven't even considered what happens when Gitea or Gitlab starts serving you 500s because your company hits some use case that the OSS developers hasn't thought of. Who fixes that? Now you are looking at sponsoring development of that. Or do you fix that? Can you guarantee any SLA on fixing any bug in Gitea? That company is moving to Ubuntu 420.69 LTS. What's the timeline on getting Ubuntu 420.69 in CI? Is that .5FTE engineering hours?
Presumably, your IT team will have many other projects to work on, some of them will be on-call, some of them will be on vacation, etc, etc. But when allocating the resources, I'd guess that 20h/week is plenty to such a service, even for a corporation of thousands of people.
Does that help?
Someone should tell every employer I've ever had this.
I think you're underestimating what it would take to solve the problem (even one as "simple" as this one). But, let's assume there is an off the shelf open source solution that is perfect for your environment and requires no changes and integrates easily in your corporate processes without any work (this is already very unrealistic...).
Then, you need to run it and provide 24/7 uptime for your 10k users, which are presumably spread across multiple timezones. Your 0.5 FTE is going to be on call 24/7? Good luck filling that position.
Finally, how are you going to support the 10k users? Is that 0.5 FTE going to answer all the support requests?
I'm not saying it's bad to do things in house - and there are really good reasons to do so. You just have to be realistic. It's going to take a whole lot more than 0.5 FTE to run even a "simple" service like this with the kind of reliability and support your 10k users will demand.
Suddenly that $80k/mo to make it someone else's problem doesn't sound so bad :)
I know I'm cherry picking but the outsourced solution is taking 11 days to say "try again" so maybe whatever that employee takes is an improvement, heh
And your logic is so out of touch with reality that I don't know whether you're trolling or not.
First, it's not just the cost. It's the fact that a company of 10k employees do not need to be at the mercy of a third-party vendor. And if the company is going to go with the line of "we want to outsource everything that is not core to the business", then they should be asking themselves what is their "core" business that warrants 10k employees in the first place.
Second, if it's just the occasional SaaS that they need and they can't find or train people to do it, I'd understand it. The problem is when you look at the typical early-stage startup who raises $500k for a seed stage round and they think it is normal/expected to burn $10k/month on Github/Jira/CI/Contentful/Dropbox/Figma/etc/etc/etc, when there are alternatives that could work and people don't even try them.
> You're missing things like:
No, I am not. Check the comment that I referenced. I suggested a scenario where you can hire someone to implement the solution and have an ongoing support contract for a fraction of the cost from 1Password.
> Do developers want to work with a system that isn't GitHub and be productive?
Have they even tried? Surely any Engineering Manager worth anything should be able to at least back up their choices based on more than "we are using this because this is what everyone else is using"?
Oh man, where do I even start here. Have you ever worked in a company with 10k employees? In IT at a company with 10k employees?
> Github/Jira/CI/Contentful/Dropbox/Figma/etc/etc/etc, when there are alternatives that could work and people don't even try them.
So let me get this straight. You expect every organization with 10k employees to:
- Find talent to build, operate, and maintain on said systems
- Keep that talent
- Train employees on said systems. Cheryl in accounting has only used SAP and you want her to understand how to use XYZ?
- Spend the time NOT having the solution available to them while they wait for said system to be built/implemented.
- And and and...
> No, I am not. Check the comment that I referenced. I suggested a scenario where you can hire someone to implement the solution and have an ongoing support contract for a fraction of the cost from 1Password.
You absolutely are. You clearly have no understanding when it comes to what is required to "just hire someone to do it".
> Surely any Engineering Manager worth anything
Yaaa, no true scotsmen!
The trope of "just hire someone" needs to die. Orgs (of all sizes) regularly do buy vs build analysis all of the time. Saying that one is better than the other unilaterally is just plain wrong.
The point is not "buy vs build". The point is that one does not exclude the other.
Sally from accounting has only used SAP? Fine, keep paying for it, but also invest in the development of an alternative. Also go to Sally and offer training in this alternative. Ask Sally what is missing in the alternative compared to what she's used to. Take her feedback to the developers and tell them "If you solve X, Y and Z, we might be able to drop SAP and switch to you".
> You expect every organization with 10k employees to find talent to build, operate, and maintain on said systems.
No. I'm expecting that some of them will be able to it in-house. Others will look for a third-party vendor that does not lock them in. Others will continue using the proprietary solution BUT will set aside part of their budget to invest in open source alternatives, as a way to create an open source alternative that can help negotiate with the proprietary vendor.
And others could do all of it, or none of it.
This whole discussion is literally "buy vs build" and the fact that you don't know that means you're way out of your league here.
The one thing that bugs me though... Maybe it is me being out of my league, but why can't bigger companies at least foment the develop of alternatives that reduce their dependency on third-party, closed service providers?
Another thing that I fail to grasp: why is it that smart and distinguished people like you always put a response in absolute, all-or-nothing terms? Case in point: when presented with one possible strategy, your counterargument was "do you expect all companies to do this?". The answer is (obviously) negative. In my childish mind, I thought it was possible to have different companies doing different things. Can you explain why this type of thinking is so clearly wrong? What do they teach in the Big Boys League that show how that absolute conformity and total adherence to the rules is the surest way to win?
They do. This is literally called "buy vs build".
> why is it that smart and distinguished people like you always put a response in absolute, all-or-nothing terms?
Because you started the conversation with absolute, all or nothing terms, which I countered. Read these two statements and tell me which is absolute, all-or-nothing and which isn't:
> What do they teach in the Big Boys League that show how that absolute conformity and total adherence to the rules is the surest way to win?Lay off of the conspiracy kool-aid will ya? Lots of reasonable people are disagreeing with you because we've all experienced this, not because some holier-than-thou entity is telling us to.
> Something troubling you?
Not the slightest, thanks though.
> This is literally called "buy vs build".
"Buy vs build" presupposes an either/or decision. Like there is no point in building an alternative once a company has decided to buy something or vice-versa. It also seems to implicate that if a company chooses to "build", that they will be taking all of the burden and costs of development to themselves.
What I'm talking about is beyond the buy/build dichotomy. What I am advocating is for companies to treat all and any systems that they depend on as a risk, and never stop looking or financing the development of F/OSS alternatives that can mitigate these risks.
This does not mean that all companies should stop using Github and implement their own source repo/CI systems. It just means that companies should look for a way to hedge their bets.
And if you think that this is something that companies do "all the time", I'd be eager to know of any, e.g, Design Agencies that support/contribute to the development of Gimp/Inkscape as insurance against their heavy dependence on Adobe Suite. Or companies (of any size) that allocate part of their budget to fund F/OSS alternatives to the SaaS solutions that they use.
> Lots of reasonable people are disagreeing because we've all experienced this
What I am seeing is "lots" of people attacking a strawman like the one you created. Maybe my phrasing was off, but my original question was "what would be the size of the team where having someone in-house to mitigate the risk of this dependency becomes clear?", and somehow this got translated (by you and others) to "why don't they just do everything in-house?".
> but I wonder what would be the point where it wouldn't take a bean-counter to say
Name-calling you say?
> Maybe my phrasing was off, but my original question was "what would be the size of the team where having someone in-house to mitigate the risk of this dependency becomes clear?"
It wasn't.
> It's the fact that a company of 10k employees do not need to be at the mercy of a third-party vendor.
Re-read what you wrote again. You're just not discussing in good faith...we're done here.
> Re-read what you wrote again.
I read these two sentences you put there, and I still can not see what is contradictory about them.
The idea I am trying to convey from the very beginning (even from last week's conversation) is "I think that buying from a closed SaaS when there is an open source alternative is stupid. Not only it costs more, it also does not really mitigate risk and it just gives an excuse to a manager to punt the problem to someone else. If however the open source solution is lacking some key functionality or you think that the best solution at the moment is to go with the established player, at least throw some money to the developers of the open alternatives as a hedge and eventual way out. The open source solution can give you optionality. You can do it in-house, or you can outsource it, etc."
> You can do it in-house, or you can outsource it
You are 100% correct companies should do this AND better yet this is exactly what is referred to as "Buy vs Build"[0]. You keep trying to convince me and others that companies ARE NOT DOING THIS ANALYSIS. I'm telling you that they most certainly are because it's literally what I do for a living (tech strategy consulting). And guess what, the reason companies keep coming back to closed-sourced providers (GitHub, etc.) is because over and over again we conclude it makes financial sense to do so.
> at least throw some money to the developers of the open alternatives as a hedge and eventual way out.
Now, this part is a hilarious suggestion. No one does this or would do this, because it makes zero financial sense.
[0] - https://medium.com/adobetech/when-to-build-vs-buy-enterprise...
I am not arguing that they don't do "buy vs build" analysis. What I am saying is that they stop here, where it would be smart if they look for other options.
> No one does this or would do this, because it makes zero financial sense.
This is what I am saying that companies DON'T DO, and I am arguing that not doing this is short-sighted. There are plenty of strategic reasons that make it reasonable to invest in open source projects:
- It is good PR. [0]
- It can help with hiring: [1]
- It can be used as a negotiation tactic when dealing with different vendors. This is just an extension of the "Smart companies commoditize their complements" [2] idea from Joel Spolsky. Going back to the 1Password example: even if you still want to continue using 1Password, the "threat" of a viable open source alternative is enough for them to not try to jack up prices. By investing small sums in open source projects, you might be able to negotiate the price down on the closed services that you depend on.
- It reduces the risk of becoming locked into solutions from third-party vendors. Even if your company "keeps going back to Github", it might be wise to contribute to the development of an alternative as a backup plan. So, if push comes to shove and Github outages become so problematic that the they affect your business, there is a way out.
Not to mention that it can be tax-deductable, so you can get all these benefits without really affecting your bottom line.
[0]: https://blog.sentry.io/2021/10/21/we-just-gave-154-999-dolla...
[1]: https://news.ycombinator.com/item?id=31679118
[2]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
It does exist, but not suprisingly people who want to pay less usually aren't keen on paying lump sums (if at all) someone who could teach them on how to do it.
So that is 0.1%, I'm sure the productivity boost from GH can be greater than that for knowledge workers. Even for people getting 25k the productivity boost could be more than 1%.
That said this kind of support turn around is just bad and should be called out forcefully.
$250/user/year; for $50k yearly salary is 200 users.
I don't think $50k salary is a lot. On the other hand, I've heard you should roughly double salary costs to approximate cost to employer (including all overhead). So then a $50k/yr local gitlab (or somesuch) admin only becomes viable salarywise at 400 paying users. And if you have 400 users dependent on that one employee for their daily work, you need backup. At least another .5 FTE - so 600 users before the business case makes sense salarywise.
So that's not really soon - if you're reaching that size, you want professional support, not what a team of 1.5 employees can cobble together.
But when I say "hire someone else to do it", it wouldn't have to be an employee. It could be a consultant, it could be a third-party IT support service. I really don't think that this kind of work would require a FTE. You'd be hiring someone to get the initial setup and you'd just run a support contract for the solution to ensure that things keep going. I think such a system you could be paying less than what the vendor charges and you also get more control over your operations.
You could also architect your business not to go down when an upstream vendor goes offline, but maybe I'm crazy.
This is what I'm wondering the most about honestly. With Azure DevOps we have slots, so if a deployment fails, there is no switch that happens and the current available online version stays online. Something about putting all your eggs in one basket, but if you're going to do it anyway, make sure you handle a pipeline failure cleanly. A failure to build should not take down production, ever.
Used to use Rietveld and now using Gerrit. You have to buy in to that code review methodology and you lose a lot of the other things (issue tracking, CI), but those can be handled elsewhere by better/more-specific tools.
If they really give you worse service than free customers if you don't have a good SLA, that's a little troubling and they should maybe rethink their support strategy - but I also imagine that helping with outages is a little more complex than regular support tasks(account recovery, sponsors, etc).
Support has refused to so much as acknowledge the problem for two years now, if not longer.
It got so bad I stopped paying for GitHub entirely and will never give another cent.
It's possible that there is a bottle neck in the system that is invisible, possibly intentionally by someone in the middle, to people higher up in the chain. This is a way for them to break that blockage, and put some spot light on it.
What this does now is create a specific marker 'Hacker News follow up' that cannot be swept under the rug. There will be eyes on this particular issue, both inside and outside the company.
Yeah, you want to "take this offline" and handle OP's case individually, and perhaps OP gets their demands met, some sort of service credit, or something.
But what the rest of us want is real systemic change, to where managed services like this actually give us real value, and not "marginal value, but nothing works like it ought to and you're going to spend a lot of time working around the bugs and gluing the parts together", where every bug report (which has to be filed as a support ticket) is met with "you're holding it wrong", or where the service goes down and the status page gaslights us, or …
Buying managed services instead of letting engineers do their jobs is today's "nobody got fired for buying IBM".
I agree with this perspective significantly more than I don't. My (again, smaller) disagreement lies with cost: the entire "aaS" industry generally is in the ballpark of $15-$30/user/year [edit: /user/month]; you are getting what you pay for. $250/user/year is in a completely different class.
That's $1-3/user/mo. That's very low for SaaS. You may get that for indie products or bulk pricing (for thousands to millions of users, e.g. end-users).
[edit: fixed in parent comment]
But it's also not "$30/user/mo", either. It's that, plus (my salary * time spent on support tickets and outages), plus storage costs, plus compute costs.
To be fair, what do you expect the first representative to do/say? They responded in 3 hours, and it's unlikely they can do much beyond escalate this issue internally. We can't expect them to say "We're going to do sweeping changes" at this point.
Github in the past has done sweeping changes for things such as youtube-dl. They created a large blog post about it, including having both programmers and laywers review every DMCA request, and allowing the most minimal amount of changes to comply, etc. That type of response takes time and coordination.
Even cloudflare with their CEO/CTO can't offer sweeping changes in a HN comment. There's layers to this. You can only really expect damage control from a HN comment.
Disclosure: I have no specific experience with GitHub support, but I have experience with other support organizations and "send us a new ticket" can easily result in a repeat of the original bad experience. I'm not saying this would necessarily happen to the OP, but we also don't have any assurance at this point that it wouldn't happen.
For a company as large as github (and microsoft), I would expect support metrics (mean time to response, max time to response) to be known to management.