I Accidentally Deleted 7TB of Videos Before Going to Production (blog.thevinter.com)

Things are complicated, people are human and forget things, there are pressures to "get it done" and override the guardrails. Everybody has horror stories. Some worse than others. Welcome to the OP's day of horror. I would think "Silicon Valley" dev-ops horror stories make this one seem like a triviality.

It's not even necessary to the story.

Thanks for the comment :)

It's like the difference between "do what I say" and "do what I mean"...

Instead of performing the dangerous action outright, just log a message to screen (or elsewhere) and watch what is happening.

Alternatively, or subsequently, chroot and try that stuff on some dummy data to see if it actually works.

This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder.

In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.

   SELECT COUNT(1) FROM table 
   -- UPDATE table SET col='val'
   WHERE 1=1

- make sure the sum of their lengths == number of total current items

- make sure items_to_be_kept.length != 0

- make sure no two items appear in both lists

- check some items chosen at random to see if they were sorted in the correct list

At this point the only possible mistake left is to confuse the lists and send the "to_be_kept" one to the delete script; a dry run of the delete list can be in order.

https://news.ycombinator.com/item?id=29083367

It defaults to not doing anything so you can gradually and selectively have it do something.

Learned about when I posted my command line checklist tool on HN: https://github.com/givemefoxes/sneklist

(https://news.ycombinator.com/item?id=25811276)

You could use it to summon up a checklist of to-dos like "make sure the collection in the dictionary has the expected number of values" before a "do you want to proceed? Y/n"

Yes, that can be a simple but powerful live on screen log. I developed a library to use an API from a SaaS vendor, in much the same way as the author. It was my first such project & I learned the hard way (wasted time, luckily no data loss or corruption) that print() was an excellent way to keep tabs on progress. On more than one occasion it saved me when the results started scrolling by and I did an oh sh*t! as I rushed to kill the job.

Make sure you got everything out and off before you pull up your pants, or else you better be prepared to deal with all the shit that might follow!

Automated tests are awesome :)

Good decisions come from experience. Experience comes from making bad decisions.

As they say, measure twice, cut once.

Don't feel bad, I think every professional in IT goes through something similar at one time or another.

    (step1) Download video from URL.  Include the Id in the filename.
    (step2) Grab the list of files that have been downloaded and parse to get the Id.  Using the Id, delete the original file.

This pattern has the added benefit that it makes it really easy to write unit tests, which is something often sorely lacking in these sorts of batch scripts. It also makes full automation down the line a breeze, since you have nice shearing layers between your components.

http://www.laputan.org/mud/mud.html#ShearingLayers

    Decide, then act.  

Another of my favorites is egregious use of caching, because traversing a DAG can result in the same decision being made four or five times, and the 'obvious' solution is to just add caches and/or promises to fix the problem.

As near as I can tell, this dates back to a time when accumulating two copies of data into memory was considered a faux pas, and so we try to stream the data and work with it at the same time. We don't live there anymore, and because we don't live there anymore we are expected to handle bigger problems, like DAGs instead of lists or trees. These incremental solutions only work with streams and sometimes trees. They don't work with graphs.

Critically, if the reason you're creating duplicate work is because you're subconsciously trying to conserve memory by acting while traversing, then adding caches completely sabotages that goal (and a number of others). If you build the plan first, then executing it is effectively dynamic programming. Or as you've pointed out, you can just not execute it at all.

Plus the testing burden is so drastically reduced that I get super-frustrated having to have this conversation with people over and over again.

If both paths are on the same disk moving files is a fast operation - and if you discover a screw up, you can easily undo it. On the other hand if everything still looks fine after a few days, you just `rm -rf` that folder and purge the files.

    During buildup of the our_id list: assert (vimeoId not in our_ids). 
    After creating the list:  assert len(set(our_ids)) > 10000 and assert len(set(our_ids)) == len(our_ids)
    Before each final deletion: assert id not in hardcoded_list_of_golden_samples. 
    Depending on the speed required you could hit the api again here as an extra check. 

After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have the infrastructure in place, if you are in a hurry. I doubt that's the case for a 10 heads shop.

The article hardly comes across as 'blaming' them for the core issue but they were definitely not helpful.

‘Judgment comes from experience, and experience comes from poor judgment.’

:-)

Also, would you rather everyone only ever posted about all the times they were successful?

A million times better than your comment.

(We do this sort of thing to protect users, usually as the result of an emailed request, and you can tell when we've done it because of the word 'redacted' in square brackets.)

    List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars) 

Say Foo and Bar have fields in common, such that you can say a Foo object "equals" or "matches to" a Bar object, like if they have name and dateOfBirth fields or something else that are the same (nothing like a common ID between the two). Now say there are some other fields too, like amountSpentThisYearOnDogFood that you know is always accurate for Bars, but might be out of date for Foos. How do you get the list of all the Foos to update?

Initially I did the nested for loop solution that's like

   List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars)
   {
    List<Foo> returnList = new List<Foo>();
    foreach (var foo in foos)
    {
     foreach (var bar in bars)
     {
      // check if "equal" or "matching" based on some criteria
      // if equal, update foo dog food expenditure with bar dog food expenditure, add to returnList, and break
     }
    }
    return returnList;
   }

The solution with a Dictionary is obviously better. All you need to ensure is that you have a method for both the Foo and Bar classes that will produce the equivalent hash for both, if they would be considered equal or matching by whatever criteria you are using.

So you could have something like

    int GetHashOfFoo(Foo foo)
    {
     string firstName = foo.FirstName;
     string lastName = foo.LastName;
     DateTime dob = foo.Dob;

     return (firstName, lastName, dob).GetHashCode(); // convenient c# method
    }

    int GetHashOfBar(Bar bar)
    {
     string firstName = bar.FirstName;
     string lastName = bar.LastName;
     DateTime dob = bar.Dob;

     return (firstName, lastName, dob).GetHashCode();
    }

   List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars)
   {
    List<Foo> returnList = new List<Foo>();
    Dictionary<int, Bar> barsByHash = new Dictionary<int, Bar>(bars.Count);

    foreach (var bar in bars)
    {
     int barHash = GetHashOfBar(bar);
     barsByHash[barHash] = bar;
    }

    foreach (var foo in foos)
    {
     int fooHash = GetHashOfFoo(foo);
     if (barsByHash.ContainsKey(fooHash) 
     {
      returnList.Add(foo.CopyWith(dogFoodExpenditure: barsByHash[fooHash].DogFoodExpenditure))
     }
    }
    
    return returnList;
   }

I actually messed up something like OP with this, but with doing undesired additions instead of undesired deletions.

You can think of it as having two endpoints, both expecting a .csv with rows being the things you were updating/changing/deleting.

The problem was, there was a column to indicate (with a character) whether the row was for an edit, or addition, or deletion, but this was only with one of these endpoints. For the other, there was only addition functionality, but I thought changes and deletions were also options for the other kind of .csv due to some unwise assumptions on my part (thinking that the other .csv would have the same options as the other). That's how we accidentally put in over 100 additions that should have been changes that had to be manually deleted. Luckily I had a list of all the mistaken additions.