Ah, I think you may be talking about the repeatable reads property? All readers in LogDevice are guaranteed to see the same records in the same order (aside from trimmed data).
LogDevice clients do notify sequencers if they have seen newer epochs, which would cause a sequencer reactivation, which indeed resolves the issue within the context of a single client. However, there can still be…
> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read.…
Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority. For a simple…
A log in LogDevice is roughly equivalent to a Kafka partition.
yes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and…
splitting data into partitions would mean there's no total order on that data anymore, right?
LogDevice has many-to-many rebuilding as well, and typically data for a log (similar to partition in Kafka) is spread relatively uniformly over the (potentially large, much bigger than replication factor) set of shards…
No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.
Some strengths of LogDevice include: - It's designed to work with a large number of logs (roughly equivalent to partitions in Kafka), hundreds of thousands per cluster is common. - Sequencer failover is very quick,…
LogDevice is payload-agnostic and doesn't inspect the value of the binary blobs it stores. If your writer is allowed to write according to ACLs, LogDevice will happily take writes from it regardless of their content…
Ah, I think you may be talking about the repeatable reads property? All readers in LogDevice are guaranteed to see the same records in the same order (aside from trimmed data).
LogDevice clients do notify sequencers if they have seen newer epochs, which would cause a sequencer reactivation, which indeed resolves the issue within the context of a single client. However, there can still be…
> Which are the cases where consistency is compromised then? If a client of the log needs consistency, it needs to ensure that it has seen all previous updates to a log before making a new update, which implies a read.…
Yes, reads are not released (i.e. are blocked) until sealing is complete. We call the minimal set of nodes sufficient to serve reads for a log (the same set is needed for sealing to complete) an f-majority. For a simple…
A log in LogDevice is roughly equivalent to a Kafka partition.
yes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and…
splitting data into partitions would mean there's no total order on that data anymore, right?
LogDevice has many-to-many rebuilding as well, and typically data for a log (similar to partition in Kafka) is spread relatively uniformly over the (potentially large, much bigger than replication factor) set of shards…
No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.
Some strengths of LogDevice include: - It's designed to work with a large number of logs (roughly equivalent to partitions in Kafka), hundreds of thousands per cluster is common. - Sequencer failover is very quick,…
LogDevice is payload-agnostic and doesn't inspect the value of the binary blobs it stores. If your writer is allowed to write according to ACLs, LogDevice will happily take writes from it regardless of their content…