Why Doesn't Anyone Want to Solve Duplication?
There are always exceptions (variations) that need to be applied. For example, the default field title of a given field might be too long for a given screen listing, being the user wants to see a lot of info on a single listing. Thus, an exception (override) is needed for that title on that page. A centralized "data dictionary" is useful, but cannot carry the entire load. Further, sometimes the variations need to be computed dynamically.
A way to manage these variations is needed that doesn't confuse developers. Perhaps if an industry standard were devised, then coders would just learn the factor-friendly system and we'd end the duplication. It feels to me the industry spends too much time chasing the latest UI trends/fads instead of factoring.
I find our tools are too rigid to even propose specifics of such a stack. Code and/or references to code would need to be managed in something akin to an RDBMS, and not a hierarchical file system nor OOP inheritance hierarchies. Trees are just too rigid. There are tools to automate duplication, but few that avoid it. Auto-dup only helps initial coding, not maintenance. Should we be expected to "just live with" duplication, or are our standards and tools lacking?
6 comments
[ 3.7 ms ] story [ 24.4 ms ] threadOne take of the vision of the Object Management Group is that the application should almost write itself once you've figured out the data structures. The designers of Visual Studio Team Edition wrote this book:
https://www.amazon.com/product-reviews/0471202843
which describes a hypothetical product which is much much better than VSTE.
It is a hard problem. The structure of common sense knowledge is something like: "A(x) is true" but "A(3) is false". In principle this kind of system would be good
https://docs.jboss.org/drools/release/7.17.0.Final/drools-do...
but the error messages it returns often make no sense at all and I haven't learned the source code for the engine enough to be able to read them. Products like that have several different mechanisms that can be used to model "default logic" (X is true unless...) but there is no simple standard mechanism that makes people happy.
(Look at my HN profile and contact me if you want to chat)
I realized that in order to be 10x better than the alternatives, I was going to need to solve some very tricky AI problems. For example, acurately deduplicating a customer record "John Doe" vs "Johnathon Doe" is not straightforward. Maybe it's two different people?. Maybe it's just a spelling mismatch? The system must have a great deal of context to accurately determine if the data is indeed duplicated. And even if it does, perhaps there's a perfectly good reason for the spelling mismatch. (e.g. perhaps one table is his preferred name, while the other is just referential, etc). In the end, deduplication often comes down to the requirements of the company and it's hard to generalize.
I think there's space in the market for this kind of business, but it'll be a slog. Unless you have a 10x solution (i.e. super AI), you'll be competing with the likes of Trifacta, etc. And it's hard to compete with that kind of sales force.
Really good question. Thanks for posting.
I think the question is about line of business software and issues there are very different.
For instance there is a literature on record matching and good techniques exist, but without an exception handling workflow you don't have a way to deal with the unusual cases the code works up.
I would love to talk and share notes about what you did.
Think of it this way: one could build a detailed Entity Relationship diagram (or OOP equivalent) in a machine-readable format with all the relationship and column-size constraints defined. One could then push a button and have a machine generate a working version of the software. Those tools do exist. But they are usually missing useful details and result in UI's poorly tuned for how employees will likely be using the system.
Many of the tweaks to make it "practical" will be exceptions or local customizations to the original ER diagram data. Those customizations/deviations are the bottleneck such that in practice most stacks use duplication of info instead. See DRY ("Don't Repeat Yourself") in software engineering slang sites.