rshrin
No user record in our sample, but rshrin has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but rshrin has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
Not for this use case. Other use cases at Netflix use AWS Managed service when it makes sense from a use-case and cost perspective. In this case, using TimeSeries opens the door to a lot of other potential future use…
For the Eventually Consistent counter, the low millisecond requirement is for reads and writes, not for the convergence of counts. For this category, the convergence is in the order of seconds (user-personalization,…
Use cases fetching counts directly in the path of Netflix users/streaming, e.g. user-personalization, feature-gating > what features are shown when you load the home page, dictated by how many times these have been…
Yes, this is one of the approaches mentioned in the article and is indeed a valid approach. One thing to keep in mind is that we are already operating the TimeSeries service for a lot of other high ROI use cases within…
Added a note on why we chose EvCache for the "Best-Effort" use-case instead of probabilistic data structures like HLL/CMS. Appreciate the discussion.
Just to add, we are also trying to support idempotency wherever possible to enable safe hedging and retries. This is mentioned a bunch in the article on Accurate counters. So take that into consideration.
Querying each instance can lead to availability and latency challenges. Moreover, HLL is not suited for tasks like increments/decrements, TTLs on counts, and clearing of counts. Count-Min Sketch could work if we're…
True. You could do decrements that way. We trimmed this article as the post is already quite long. But considering the multiple threads on this, we might add a few lines. There is also something to be said on operating…
5 people (who also maintain a lot of other services like the linked TimeSeries service). The self-service to create new namespaces is pretty much autonomous (see attached link in the article on "Provisioning"). The…
HyperLogLog (or even Count-Min Sketch) will not support some of the requirements of even the Best-Effort counter (clearing counts for specific keys, decrementing counts by any arbitrary number, having a TTL on counts…
Yes, once the events are aggregated (and optionally moved to a cost-effective storage for audits), we don't need them anymore in the primary storage. You can check the retention section in the article. The rollups…
Fwiw, we didn't mention any probabilistic data structures because they don't satisfy some of the basic requirements we had for the Best-Effort counter. HyperLogLog is designed for cardinality estimation, not for…