Tarsnap outage 2023-07-02

3 points by voytec 3y ago ↗ HN

    Subject: Tarsnap outage 2023-07-02
    From: Colin Percival
    Date: Sun, 2 Jul 2023 10:13:26 -0700
    
    The important information first: All data stored on Tarsnap is safe, but you
    can't access it right now.  I'm working to bring it back online.
    
    The detailed story:
    
    At approximately 2023-07-02 13:07 UTC the main Tarsnap server suffered a
    "failed system status check" in Amazon Web Services vernacular.  In plain
    language, something broke -- either the server hardware, the power, or its
    connection to the rest of the EC2 network (including network-attached disks).
    
    At approximately 2023-07-02 13:52 UTC the server rebooted (probably running
    on new hardware, considering that it's a virtual machine), but had filesystem
    corruption affecting cached metadata.  This cached metadata essentially tells
    the Tarsnap server where to find user data in Amazon S3; without it, the
    Tarsnap server cannot service requests.  I *think* that the metadata present
    is accurate as far as it goes, but it definitely lacks information about
    archives stored in the last few seconds before the outage started.
    
    While I could probably "replay" the most recently stored data from the backing
    store in S3, the nature of filesystem corruption means that I can't guarantee
    that this would work; since Tarsnap is a backup service I have to prioritize
    safety, and so I've started the process of reconstructing the Tarsnap server
    state from the "known good" data stored in Amazon S3.
    
    If anyone needs data urgently, I can bring the Tarsnap server up in read-only
    mode, subject to the caveat that the most recently stored data may not be
    available.  Please contact me if this would be useful to you.
    
    It's too early in the state reconstruction process to provide an accurate ETA;
    my best guess is somewhere between 12 and 24 hours.
    
    As always, feel free to email me if you have any questions.
    
    Sorry about the headaches,

6 comments

[ 40.3 ms ] story [ 1683 ms ] thread

disadvantage 3y ago ↗

Tarsnap just uses S3 for storage. Would it not be cheaper to just use that yourself, and ditch the abstraction layer of Tarsnap?

voytec 3y ago ↗

They use deduplication to upload differential backups for costs reduction, let you pay for months or years in advance, donate (iirc) 10% revenue to The FreeBSD Foundation and have more professional support.

cperciva 3y ago ↗

donate (iirc) 10% revenue to The FreeBSD Foundation

Tarsnap contributes an amount equal to its December operating profit to open source software. This largely takes the form of sponsoring conferences throughout the year and then the FreeBSD Foundation gets whatever is left in the budget (once I know how much profit Tarsnap made in December).

_DMN_ 3y ago ↗

Is this an email you received?

I (at work) did not get an email about this, tried a manual `tarsnap --list-archives` after seeing all the failed backup jobs, checked https://www.tarsnap.com/news.html after that failed and found nothing. I had to Google "Tarsnap network outage" and find this post to have any idea something was actually wrong with the service, less than ideal.

You should subscribe to the tarsnap-announce mailing list. That's where tarsnap announcements are emailed out.

(And yes, the above text is what I sent to the list.)

Thanks, I will do that. I do think that a backup service being down for 12-24 hours is pretty critical information that shouldn't require an opt-in though.