Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?
> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API.
For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.
7 comments
[ 11.3 ms ] story [ 486 ms ] threadWhat are some use cases that you found are useful?
For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.
[0] https://aws.amazon.com/builders-library/reliability-and-cons...
[1] https://brooker.co.za/blog/2021/05/24/metastable.html
https://github.com/databricks/dicer/blob/master/dicer/extern...
(btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/extern...)