The 1–10–100 Principle or How We Stopped Neglecting Data

[A Case Report on Fuzzy Name Matching]

In 1992, George Labovitz and Yu Sang Chang proposed the 1–10–100 principle, which suggests that the longer one takes to correct bad data, the more expensive it will be to fix it eventually. A data-centred way of putting it is this:

"It will cost $1 to verify the quality of your database, $10 to clean each item in it and $100 to store each item that is not cleaned."

[100 Dollars]

I worked for a small organisation which I will call ACME from here onward. It is social security organisation that manages funds for the Government. The law of the land is such that every employee was required to save a certain portion of their salary with ACME on a monthly basis. This money can then be accessed upon reaching the retirement age or under special circumstances which are beyond the scope of this post. At that time, ACME had the list of expected monthly contributions from almost everyone in the labour sector, but their list was compromised due to several factors surrounding how they managed data (let’s call this List_A). Concurrently, ACME would receive multiple salary schedules from hundreds of organisations around the country (we’ll call this List_B) and their task was to compare both lists to come up with a clean, properly formatted and reliable master copy. It is also important to note that List_A and List_B usually had cases of the same individuals appearing on both lists, but with their names spelled differently.

Initially, matching each name in List_A to the names in List_B was done manually. This usually took many days and resulted in plenty of mismatched names because of human error, to the point that our error rate was hovering around 30%. In other words, our team was mismatching hundreds of thousands of dollars each day. Further on, project costs were also running high. Each person working on the project was earning about $15 an hour and for a team comprising about 70 people (not counting top management), it is easy see how overall project costs could easily have got out of hand if the project had dragged on for too long.

[CONTINUE READING]

"The 1–10–100 Principle or How We Stopped Neglecting Data and Started Investing in Data Cleaning" > https://getflookup.medium.com/the-1-10-100-principle-or-how-we-stopped-neglecting-data-and-started-investing-in-data-cleaning-b07a7dbf92d6