I think generally when every alarm bell in your monitoring system goes off the first thing you do is question whether monitoring is broken. When you confirm there is a problem this big, you panic and try to fix it really fast. Then you call your other on-call guys and tell them you actually have an "oh, shit" situation.
Once you recognize there's a serious problem, THEN you make the public announcement. Ah, the life of ops.
They also don't seem to count the "partial degradation" periods as downtime; Stripe.js and Checkout.js both have a 9m red bar, but Checkout.js has had a lot of "partial degradation" periods. Both sit on 99.992% uptime, though. Maybe that's not so bad; it would depend on how severe a "partial degradation" can be and still be classified that way, I suppose.
I work at Stripe. We're back up now. Your customers can safely retry any failed payments. We'll post a public postmortem over the next few days with more information.
If you need any help, please drop us a line at support@stripe.com (and feel free to CC me; I'm michael@stripe.com).
Oh give me a break. Be upset that it went down and you maybe lost a few sales, not that this guy is trying to be helpful and confirms that it is back up and OK to retry transactions. There are plenty of people using stripe for non-ecommerce things.
Yes, I totally hear you -- this is why we treat any API unavailability as such a huge deal and feel so bad about it. Whether customers can or should retry is a question we sometimes get from users after API errors and so I wanted to be proactive about answering it. (And our users often have Stripe.js tokens or customer tokens that they can retry.) But, you're totally right; this definitely doesn't solve all cases.
In the unlikely event there's anything I can do to help, definitely drop me an email.
There was some question as to what "partial degradation" means on the status site. Is that a " takes longer than x milliseconds to respond" type measurement? Can you clarify?
"Elevated error rates" is now status page speak for "shit is totally down". Amazon does the same thing with a small note next to a green check when their api is also broken. I wonder what hellscape on earth would be necessary for amazon to drop to a red icon.
For payment processing, I've built in a system which uses stripe (credit cards only) and bitcoin (using a different API. But that is only to detect whether payment has received. If the API is down, the detection of payments can still be done manually thanks to the fact that the bitcoin network is decentralized).
I never rely on a single point of failure. And of course, always good to have a "plan B" in place.
What I'm about to say is definitely NOT IN ANY SENSE Schadenfreude, but...
I get really excited when amazing services we've come to take for granted goes down. I'd put places like Amazon, Netflix, Gmail, etc in this category. The reason? Well, I really really look forward to reading the post-mortems and getting a sneak-peek at the inner workings of said services.
27 comments
[ 3.2 ms ] story [ 69.3 ms ] threadHmm!
Once you recognize there's a serious problem, THEN you make the public announcement. Ah, the life of ops.
That sounds like something Homer Simpson would ogle at.
Alternative: accepton. Allows me to use stripe and others.
If Stripe's down, then it will allow me to accept payments via Paypal or other. Good for these scenarios.
If you need any help, please drop us a line at support@stripe.com (and feel free to CC me; I'm michael@stripe.com).
In the unlikely event there's anything I can do to help, definitely drop me an email.
I never rely on a single point of failure. And of course, always good to have a "plan B" in place.
I get really excited when amazing services we've come to take for granted goes down. I'd put places like Amazon, Netflix, Gmail, etc in this category. The reason? Well, I really really look forward to reading the post-mortems and getting a sneak-peek at the inner workings of said services.
https://support.stripe.com/questions/outage-postmortem-2015-...