AWS Team deleted one of our Elasticache clusters by “mistake”

22 points by shrey-sinha ↗ HN
Reply from them: "Upon further review, it has been determined that the cluster was deleted inadvertently due to a human error while a corrective action was taken to recover the cluster from the modification stage."

And this deletecluster action didn't even show up in our CloudTrail logs.

7 comments

[ 2.7 ms ] story [ 22.8 ms ] thread
Sorry that happened. Legally, you are paying Amazon for the privilege of taking your money, they don't actually have to deliver any services you pay for. They can nuke your entire business (if it is based only on AWS) and have zero liability. From the AWS Customer Agreement: https://aws.amazon.com/agreement/

> 10. Disclaimers.

> THE SERVICE OFFERINGS ARE PROVIDED “AS IS.” EXCEPT TO THE EXTENT PROHIBITED BY LAW, OR TO THE EXTENT ANY STATUTORY RIGHTS APPLY THAT CANNOT BE EXCLUDED, LIMITED OR WAIVED, WE AND OUR AFFILIATES AND LICENSORS (A) MAKE NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE REGARDING THE SERVICE OFFERINGS OR THE THIRD-PARTY CONTENT, AND (B) DISCLAIM ALL WARRANTIES, INCLUDING ANY IMPLIED OR EXPRESS WARRANTIES (I) OF MERCHANTABILITY, SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, OR QUIET ENJOYMENT, (II) ARISING OUT OF ANY COURSE OF DEALING OR USAGE OF TRADE, (III) THAT THE SERVICE OFFERINGS OR THIRD-PARTY CONTENT WILL BE UNINTERRUPTED, ERROR FREE OR FREE OF HARMFUL COMPONENTS, AND (IV) THAT ANY CONTENT WILL BE SECURE OR NOT OTHERWISE LOST OR ALTERED.

> 11. Limitations of Liability.

> WE AND OUR AFFILIATES AND LICENSORS WILL NOT BE LIABLE TO YOU FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR EXEMPLARY DAMAGES (INCLUDING DAMAGES FOR LOSS OF PROFITS, REVENUES, CUSTOMERS, OPPORTUNITIES, GOODWILL, USE, OR DATA), EVEN IF A PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. FURTHER, NEITHER WE NOR ANY OF OUR AFFILIATES OR LICENSORS WILL BE RESPONSIBLE FOR ANY COMPENSATION, REIMBURSEMENT, OR DAMAGES ARISING IN CONNECTION WITH: (A) YOUR INABILITY TO USE THE SERVICES, INCLUDING AS A RESULT OF ANY (I) TERMINATION OR SUSPENSION OF THIS AGREEMENT OR YOUR USE OF OR ACCESS TO THE SERVICE OFFERINGS, (II) OUR DISCONTINUATION OF ANY OR ALL OF THE SERVICE OFFERINGS, OR, (III) WITHOUT LIMITING ANY OBLIGATIONS UNDER THE SERVICE LEVEL AGREEMENTS, ANY UNANTICIPATED OR UNSCHEDULED DOWNTIME OF ALL OR A PORTION OF THE SERVICES FOR ANY REASON; (B) THE COST OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; (C) ANY INVESTMENTS, EXPENDITURES, OR COMMITMENTS BY YOU IN CONNECTION WITH THIS AGREEMENT OR YOUR USE OF OR ACCESS TO THE SERVICE OFFERINGS; OR (D) ANY UNAUTHORIZED ACCESS TO, ALTERATION OF, OR THE DELETION, DESTRUCTION, DAMAGE, LOSS OR FAILURE TO STORE ANY OF YOUR CONTENT OR OTHER DATA.

Worth noting that larger accounts may supersede this contract with their own agreement. Not that there's any way to know which type of account OP has or is representing, but the standard customer agreement is only applicable if AWS and the client haven't negotiated something specific.

And oftentimes the sections you've highlighted are some of the first to be tweaked and the last over which one party will blink.

(not a lawyer; my only qualification here is having provided software-security-specific consultation to vendor management teams. An actual attorney will probably give a better answer)

is there a tutorial for that?

how much money do we need to burn to remove those two?

> And this deletecluster action didn't even show up in our CloudTrail logs.

Are you bound by anything from posting both the relevant reply and the snippets from cloudtrail logs, either publicly or privately? not that this would prove much (screenshots can be faked trivially), but it gives something to go off of for larger accounts who need to ask specific questions and implement non-technical controls e.g tailored contract language outside the standard AWS customer agreement.

Ahh, human-powered Chaos Engineering as a Service (CEaaS): AWS Lolwut.
At least they can solve LeetCode medium / hards, though.