Season's greetings,

I'm the co-founder of a very small start-up which needs to process lots of external tabular data and produce visualisations in real time. We currently store all of our data in a monolithic database. Our biggest challenge at the moment is data change management, and I'm a little over my head. I'd be most grateful if anyone has reading/resources/solutions to recommend, especially if they're not massively overengineered (we don't need FAANG scale scaling).

By data change management, I mean things like the following:

- When our schemas change, knowing which data falls under the previous vs new schema (e.g. if we add a column, being able to distinguish between NULLs that came after the new schema launches vs before).

- When our schemas need to change, identifying all the code that depended on the previous schema

- Being able to incrementally update schemas without having to rewrite all code with dependencies on the old schema in a big-bang (i.e. the two can coexist for a time)

- Ideally, avoiding duplicating our data for every change

- Sometimes the data we ingest has errors. We'd like a standardised way of tagging where we're aware of these errors (e.g. to exclude certain time ranges from ML models), without just overwriting/cleaning the data

- An external data provider is going through a big migration and rebuilding their systems. There will be a dual-run period where we'll get the same data set from different systems at different times, and we need a constant view of what is the latest view. We have a huge amount of code that reads from the first data set directly, so if we're going to do a big refactor, I want to build in a way that will be less-huge of a rewrite in future. I could just create a merged table / view and use that as the source of truth in future, but I suspect there's a more elegant way I'm missing

Our main selection criteria are:

- Fast (for a small number of end-users)

- Fit-for-purpose: Whiz-bang solutions for non-tabular data like images aren't strictly necessary yet, although no harm if they have that flexibility

- Affordable: We currently spend around $250pm. While that can grow, we can't really justify huge, enterprise-scale investments

- Plausibly maintainable by a small team

I appreciate that these criteria can conflict with one another, so I guess the biggest trade-off we can make is that we don't need super high scalability.