16 comments

[ 2.8 ms ] story [ 50.9 ms ] thread
We're having problems pushing to repositories (git over ssh). Currently blocking after the following

    Counting objects: 24, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (22/22), done.
    Writing objects: 100% (24/24), 2.87 KiB | 2.87 MiB/s, done.
    Total 24 (delta 16), reused 0 (delta 0)
Status page shows no issues. How reliable...
https://status.bitbucket.org/ shows:

All HTTP-based services unresponsive Subscribe Investigating - We're seeing high response times on all HTTP-based services, and are investigating now. Mar 2, 14:49 UTC

When I wrote the comment there were no such notice. Also SSH has shown operational all along.
Our Jira Cloud instance is still up, for now, so it might be isolated to one DC.
https://status.aws.amazon.com/rss/directconnect-us-east-1.rs...

We are investigating increased packet loss possibly impacting some AWS Direct Connect customers in the US-EAST-1 Region.

Bitbucket briefly blamed the issue on AWS us-east-1 too, until they scrubbed it and just said "cloud provider". Here's what they said then:

Identified - Component services in AWS us-east-1 are currently unreachable due to upstream incident. We're attempting to route as much as possible away from the affected components now.

"investigating increased pocket loss" is a nice way to put it.

it's been 100% packet loss here all day :)

This is becoming a common thing for bitbucket? Think I will be switching to GitHub.
our jira cloud instance remains down at 17:37 UTC
...and we're back online at 18:33 UTC.

from AWS:

> The root cause of this issue was network reconvergence following the loss of power at a network facility. Connectivity to instances and services within the Region was not impacted by the event. The issue has been resolved and the service is operating normally.

David from the Atlassian SRE team here. AWS Direct Connect is experiencing an outage in their US East Region: https://status.aws.amazon.com, which is causing connectivity issues for most Atlassian products and services. We're working hard to get everything back up and running. Please check http://status.atlassian.com for the latest updates. We're posting regularly and will continue to provide updates there.
Will there be a postmortem blog post or some other summary about this?