Ask HN: Good tech talks on how analytics systems are implemented?
I am doing a new sub-system, for analytics, which I can design / implement from scratch. I get a bunch of unique users (say a few thousands). Now I need to track each of these users and do some analytics (Which places are they logging in from ? How long do their sessions last on average ? What are the usual links/endpoints that they visit ? etc.) in my API server. I have a few thousand active users and about two dozen parameters on which I want to track and analyze these parameters.
I have never implemented such an analytics type system. I want to learn about people who have already implemented similar systems. Are there any good tech talks, engineering blog posts, video courses, etc. that highlight the design/technology/architecture choices and their benefits.
86 comments
[ 3.0 ms ] story [ 156 ms ] threadWhen this starts creaking at the seams it'll mean that you either have bigger analysis and/or scalability requirements and it'll much clearer what you need to look for.
Writing to Google Analytics, Amplitude, Mixpanel (all of which have free tiers) or equivalent all should handle your case well.
There's a lot of context that's missing from your post, some questions that can help us guide you in the right direction:
1) Can your website call out to external services, or are you limited to operating behind a company network?
2) Is this more of an ad-hoc analysis or do you want to invest in a framework to be able to track such metrics systematically over time?
3) How important is data accuracy? Adblock can easily mess with client-side metrics.
4) How real-time do metrics need to be? The big trade-off here is speed vs accuracy.
5) How long do you intend to keep this data? This is a pretty big concern with regards to privacy and storage costs.
If you'd rather not share some of these answers on a public forum, feel free to shoot me an email.
2) We want to invest in building a good framework to track such metrics systematically over time
3) We have some non-web API clients too. Adblock is not a problem.
4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed.
5) The data will be kept for a few months at least, if not years. The storage costs are not a big problem. This is for enterprise based solution, where the deployment may happen in a private network.
You can use different technologies based on your use case, but you probably need all the pieces outlined above. As someone else has mentioned, if you're looking for trade-offs between different technologies, I'd recommend "Designing Data-Intensive Applications" by Kleppmann.
This is the hard part of analytics for an app that is more back office oriented, understanding what will be needed to get truly accurate and useful information to support the reports people will want in 2 years, 5 years, whatever time frame is long enough for things to really change in your environment.
Try to think in an adversarial way, what question could someone come up with that I can't answer. The user who has seen the most errors? Usage trends by department?
One place this might lead is wanting to put a way to link request/error logs back to an application level user account (in a way that respects security and privacy), this can become great debugging info too.
My thought process for solving your problem would be the following. First, you need to understand what's good for you and for your company might not be the same. You want the challenge, you want to implement something that could scale and you want to use exotic tools for achieving this. It's interesting and looks good in your CV. Your company might just want the results. You need to decide which is more important.
If we prioritize your companies needs over keeping you entertained, I'd follow this thought process:
Can't you just use Google Analytics? You can also connect it to BigQuery and do lots of customizations. Maybe time would be better spent learning GA. It's powerful, but most of us cannot use it well.
Second question: if for some reason, you don't want to use Google Analytics, can you use another, possibly open-source and/or self-hosted analytics solution? Only because you can design it from scratch, it doesn't mean you should.
Third: Alright, you want to implement something from scratch. For this scale, you can probably just log and store events in an SQL database, write the queries, and display it in a dashboard.
Then, if you really want to go further, there are many tools that are designed to scale well and perform analytics, "big data". By looking for talks about these tools, you will get a better understanding of how things work. There are various open-source projects you should read more about: Cassandra, Scylla, Spark, Storm, Flink, Hadoop, Kafka, Hadoop, Parquet, just to name a few.
I am aware of these tools like cassandra/flink/spark/kafka etc. But I am more curios about the best tools and architectural patterns that work well with each other.
Well you can go with:
Fancy: Hdfs(distributed file system) as storage - oozie as workflow scheduler for your load(python/hive/scala/spark) - Tableau for visualization (your business ussers will love it.
Mid range: SQL Server as storage - Informatica for your workload - power BI /SSRS for visuals
Open/low budget: PostgreSQL / Cassandra for storage - make your own scheduler/ there is a post for ETL open score yesterday that might help - for visuals you can make it from scratch but hire a good designer!
This is based on my experience on industries like Gambling, Banking and Telecom
I'd imagine you could use Segment (or source(s) that plugs into it) to accomplish much of this.
Then you can take the db offline for maintenance and upgrades and not lose data.
You also get HA for free; your solution depends on local node disk persistence.
We have reached a point where people think that local node disk persistence is an exotic property.
If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.
It covers all the basic needs, and even if you're not using Rails, I think you can draw inspiration from it!
[0] https://github.com/fishtown-analytics/snowplow/
Having spent some time on this I'll just say that don't overthink it. Over-engineering such system is way too easy while the actual benefits might not be that great. Sure if you're receiving a lot of data there might be some pitfalls to be aware of eg using proper bucket partitioning with Athena for queries.
I’d say go with something like Matomo (formerly Piwik) https://matomo.org. If you wanted to build your own, I’d suggest keeping it simple. Look at Matomo’s architecture and replicate https://github.com/matomo-org/matomo.
It's a distributed system, the mobile and web SDKs batch the user events on their devices and push it to our API in JSON format. The API servers enrich & sanitize the data, validate the schema, convert it to a serialized AVRO binary, and push it to a commit-log system such as Kinesis or Kafka (It's pluggable).
We have another project that fetches data from Kafka & Kinesis in small batches, converts the data into columnar format and stores it in an S3 bucket / Google Cloud Storage. Then, we integrate their preferred data-warehouse into their distributed filesystem. That way they have all their raw data in their infrastructure for other systems such as fraud detection, recommendation, etc. but they have SQL access to their data as well.
That being said, this architecture is for >100M events per month. If your data is not that much, you can actually ingest your data into an RDBMS and it just works fine. We support Postgresql at Rakam and you need is the API server and a Postgresql instance in that case. Our open-source version supports Postgresql so you can look into the source code from here: https://github.com/rakam-io/rakam Would love to get some contribution. :)
For the analysis part, all these metrics can be created using just SQL, the modern data-warehouse solutions (BigQuery and Snowflake) also support javascript and it's relatively easy to build funnel & retention queries that way. It requires more work but now you have more control & flexibility over your data.
2) Find your visualisation tool of choice. This is more important than any architecture choice for the tracking because this makes your data useable. [1]
3) Select your main data storage that is compatible with your visualisation tool, data size, budget, servers, security, ... SQL is always better because it has a schema the vis tools can work with. For a low amount of data you might just want to use your existing database (if you have one) and not build up new infrastructure that has to be maintained.
4) If you need higher availability on the data ingress than your db can provide use a high availability streaming ingress [2] to buffer the data.
5) Design a schema to connect your db to the visualisation tool. Also think about how you will evolve this schema in the future. (Simplest thing in sql is: Add colunms.)
I hope this helps. If you have selected some tools it is fairly easy to search for blog posts and tech talks. But don't think to big (data). "A few thousands users" and "two dozen parameters" may be handled with postgres and metabase. Also in most enterprise enviroments there already exists a data analytics / data science stack that is covered by SLAs and accepted by privacy officers. Ask around.
[1] https://github.com/onurakpolat/awesome-bigdata#business-inte... [2] https://github.com/onurakpolat/awesome-bigdata#data-ingestio...
2 and 3 are tied to each other. You could have visualizatoin drive storage or visa versa. Just understand the tradeoffs.
I'd suggest for who's done it before / talks, etc. there are a ton out there. There's those chats from the FAANGs and various groups in the valley (Lyft, etc.). Tons of blog posts there. Vendors have (largely predisposed towards them) builds. Finally the talks/slides at datacouncil and strata often contain lots of more .. "pointed" information. The high level bible that lots of folks would say look at is Kleppmann's "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems".
I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building, they're going to waste your time and hurt your stability.
Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue and then flush them to DB in a batch?
And yes typically you also have a job queue for the same reason. So e.g. API calls don't take longer due to database event writes.
If your needs are similar to the original questioner (still starting out, few thousand users), I'd say even the separate database isn't necessary, just the job queue. Heck even the job queue is probably optional at that scale, but most hosting platforms make job queues easy enough to integrate so I say why not.
You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement, upselling, growth, etc) and the types of analysis you'll need to understand what changes you need to make to produce those outcomes. Protip: this is actually really hard, and people underestimate it by orders of magnitude. A blog post by Roger Peng (with indirect commentary from John Tukey) ... https://simplystatistics.org/2019/04/17/tukey-design-thinkin...
One other immediate tip is to start thinking about correlating your telemetry with user surveys - again, strongly focusing on outcomes and the controllable aspects of those outcomes.
Don't let the data lead the discuisson; decide on the question you're asking, and the implications of all of the possible answers to that question (clearly yes, clearly no, mixed, etc) before you ask it.
Then engineer the lightest weight system possible to ingest, process, store, analyze, and visualize that data.
For me, that would just be:
1. Log data in whatever logging tool you like. Persist the raw stuff forever in a cheap data lake. 2. Batch at some fixed interval into a staging area of a relational DB. 3. Transform it with stored procedures for now (while you figure out what the right transforms are) into a flat fact table. 4. Visualize in Superset or PowerBI or even plain old Excel.
Once you've got the patterns of analysis at least fundamentally right you can consider stream processing (Flink or Kafka Streams are fine) to replace 2 and 3.
Persist it as long as you need it to obtain value from it, but after that, you're just holding on to something you can lose some day in a data breach, or build up enough of that people start to get mistrustful (Facebook, Google, etc.) Data expiration policies should definitely be something you think about and have justification for, even if you do decide that your expiration is "Never".
Structure your code so you crunch your historical data once, store in redis, and then new data gets shoved in the redis cache as your time dimensions on your metrics progress based on business logic.
Until your data is at enterprise volume, you really don't need an OLAP system.
https://www.youtube.com/watch?v=-70wNNrxf6Q
I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make sure you have identifiers in each row for each entity a row is part of: user, session, web request. If you're not building replicated PostgreSQL (which you probably won't need for this), log to files first, and build another little process that tails the files and loads the rows into PostgreSQL. That log then load advice is hard learned experience from working at Splunk.
The SDKs provide ways to send the event data with the following format:
rakam.logEvent('pageview', {url: 'https://test.com'})
The event types and attributes are all dynamic. The API server automatically infers the attribute types from the JSON blob and creates the tables which correspond to event types and the columns which correspond to event attributes and inserts the data into that table. It also enriches the events with visitor information such as user agent, location, referrer, etc. The users just run the following SQL query:
SELECT url, count(*) from pageview where _city = 'New York' group by 1
All the project is open-source: https://github.com/rakam-io/rakam Would love to get some contribution!
- Who will be viewing these reports when they are done? Who do you want to have a view of the data eventually?
- How fresh do you need the data to be? Is 24 hours, 4 hours, or 4 seconds okay to wait?
- Do you need to be alerted of anomalies in the data?
- How long do you intend to store the raw data? Aggregated data?
- Does your data need to contain anything that could personally identify a user in order to make a useful analysis? Do you serve customers in the EU?
I'll check back later today and see if I can provide any insights based on your response.