Ask HN: Resources to plan organizations data strategy?

2 points by dav43 5y ago ↗ HN

I'm looking for resources to learn about development of a wholistic data strategy for large organizations (>1,000) people - to store data, share data, create automatic/realtime reporting?

Starting from a very low base - I've learnt to mockup R Notebooks to analyze data sets and used PowerBI to create (somewhat) informational dashboards and KPI measurement for organizations.

E.g. For an organization with +10,000 users, what options are available to setup datastores/share and develop pipelines for a large organization - hosted/local. E.g what tools are used? Whats industry best practice? Whats open source vs proprietary? Security implications/trade-offs?

Any directions helpful. I've briefly looked at Power BI and its hosted platform but looking for what else is around.

4 comments

[ 2.9 ms ] story [ 22.8 ms ] thread

coderintherye 5y ago ↗

A common stack in industry now is: Fivetran (or Stitch) -> Snowflake (or BigQuery) -> Looker (or Mode)

For an organization with 1000+ employees you are going to be paying a lot for whatever service you use, which means that you will get a lot of support from the vendor you go with and you don't have to try to answer all of these questions ahead of time. Plus, you are going to hire someone on your team to manage this, it's not something that someone just manages on the side.

dav43 5y ago ↗

Thanks. I believe the organization already has the powerBI service available - just not being used. But when I look at their use case - and ability - I think it might be over kill.

Agreed that its definitely not someones side project on a Friday afternoon!

brudgers 5y ago ↗

At >1000 the ordinary resources is consultants because person-centuries of experience is usually warranted at that scale. Consultants are also the best way to get in-house staff up to speed quickly because they will participate in specifying the system and learn what are the right questions to ask and important factors to consider. Good luck.

dav43 5y ago ↗

Agreed it would take a large team to deploy and execute - especially as its a public/government run team. I probably wasn't clear in my description of the goal.

I want to get up to speed and distill what options are available - at a high level. I won't code it/deploy it, but I'm good at reading and understanding setups and seeing the pros and cons.