But seriously, what's going on with all the service disruptions? From slack to all of facebook to GitHub (who had 7 incidents in September). Maybe centralizing the whole internet into a few providers isn't a good idea after all?
Running services at scale is hard! And having everything in the cloud does put all of your eggs in a single basket. Also having all on-prem, home-built stuff is the same problem (eg. Facebook). Hybrid cloud solutions should help somewhat with impact when a service provider goes down, hopefully you can shunt your services locally and isolate yourself from the problem. But I doubt anyone is really doing that at scale either.
> Maybe centralizing the whole internet into a few providers isn't a good idea after all?
Everything's tradeoffs, economies of scale for centralising- easy to blame the provider if a fuckup happens.
But when it goes down it stops everything; same issue with AWS, Azure's ADFS, Cloudflare, ad infinitum. Half the internet can go away at a moments notice or the provider can turn on you personally because an automated bot dislikes your username and decides to blacklist your company.
(is my bias showing?)
Anyway, Late September/Early October is rife with outages for as long as we've been running internet services.
People speculate that it's the influx of new sysadmins/programmers, but that's debated.
> Anyway, Late September/Early October is rife with outages for as long as we've been running internet services.
Is there any particular source for that or is it just something that you've noticed? Feel like I do see more around this time of year but would be real interested if there is data backing it up.
I literally set up some GitHub actions today for the first time in my life, I was really pumped about getting into the CI/CD game with my latest open repository. A couple of hours ago everything was going great, running a workflow needed maybe 40 seconds max. Now, they are not running at all. That all happened while I was thinkering with the yaml workflow file, so the disruption of the service made me believe that it was me who broke something :P it's been a nice first experience with CI/CD though, never going back to building my self.
Check my GitHub repo (see my bio) to see it's true haha
Think of the poor grandma yesterday who was finally convinced to get a Facebook account to see the grandkids and right as she hits "create account" Facebook goes down worldwide.
No force on earth will convince her she didn't do it.
In CI (Github Actions) , as generally is recommended I have limited with max-workers=1. With a single worker 5 min doesn't sound it unreasonably long for 2,000 tests? Most CI runners are going to only give lower performance single core vCPU in their runtime so it doesn't seem too bad time taken for this type of workload.
On my laptop I run 6 tests in parallel. It runs in 60ish seconds however the balance is how much cores I can spare for the workers while still being able to do something else.
The challenge is if you switch to another task you don't come back always after the minute and it takes some time to review and commit, or some tests failed now you are fixing those and also working on other items.
Both tasks go poorly, because after each run you want to go back to other task so sometimes you fix one thing and that breaks something else in the next run, so you keep doing few runs and that new task is going poorly because you are losing focus constantly, the
Or eventually you end up with work that is really few completely different commits getting actually pushed as one big item, If you like frequent atomic commits this is just breaks your flow all the time.
P.S. the pet peeve: when the laptop fans starts spinning hard to handle the work, it feels like the same frustration that I feel when I am out of shape and start huffing and puffing in a simple 2k run.
FYI you might want to make the company logos & names more generic since big companies are pretty ruthless when it comes to trademark usage and their brand.
42 comments
[ 0.18 ms ] story [ 111 ms ] threadBut seriously, what's going on with all the service disruptions? From slack to all of facebook to GitHub (who had 7 incidents in September). Maybe centralizing the whole internet into a few providers isn't a good idea after all?
Just don't go 'all in' on one service without backups or some second plan.
Everything's tradeoffs, economies of scale for centralising- easy to blame the provider if a fuckup happens.
But when it goes down it stops everything; same issue with AWS, Azure's ADFS, Cloudflare, ad infinitum. Half the internet can go away at a moments notice or the provider can turn on you personally because an automated bot dislikes your username and decides to blacklist your company.
(is my bias showing?)
Anyway, Late September/Early October is rife with outages for as long as we've been running internet services.
People speculate that it's the influx of new sysadmins/programmers, but that's debated.
Is there any particular source for that or is it just something that you've noticed? Feel like I do see more around this time of year but would be real interested if there is data backing it up.
No force on earth will convince her she didn't do it.
Now gran really thinks something is up
If your test suite takes 40 seconds, is it not quicker and easier to run locally than the overhead of another git commit to get feedback?
On CI it takes about 5 minutes to run.
My laptop which while not top of the line is fairly well powered (8core 32 gb) the fan starts spinning and slows down a bit whenever I run the suite.
I find both these times incredibly distracting and inefficient. It breaks my concentration, faster feedback would be really nice.
On my laptop I run 6 tests in parallel. It runs in 60ish seconds however the balance is how much cores I can spare for the workers while still being able to do something else.
The challenge is if you switch to another task you don't come back always after the minute and it takes some time to review and commit, or some tests failed now you are fixing those and also working on other items.
Both tasks go poorly, because after each run you want to go back to other task so sometimes you fix one thing and that breaks something else in the next run, so you keep doing few runs and that new task is going poorly because you are losing focus constantly, the
Or eventually you end up with work that is really few completely different commits getting actually pushed as one big item, If you like frequent atomic commits this is just breaks your flow all the time.
P.S. the pet peeve: when the laptop fans starts spinning hard to handle the work, it feels like the same frustration that I feel when I am out of shape and start huffing and puffing in a simple 2k run.
If I'm making a change and I want feedback from the test suite, of course I'd run it locally.
By the way, just discovered a nice library to test GitHub Actions locally: https://github.com/nektos/act
https://status.dev.azure.com/_event/265268533
Related? Geography is US.
`host gist.github.com` returns - "Host gist.github.com not found: 5(REFUSED)"
Disabled adblock. Still same. I'm using from India.
Our whole business[1] relies on GitHub Actions functioning. It just sucks.
[1] https://buildjet.com/for-github-actions
Github Action 2vCpu / 8 GB RAM
BuildJet Runner 4 vCpu / 16 GB RAM
Also, should probably be capitalized as "vCPU".
Anyways, Twitter will be down next week. You've read it here first.