Show HN: Lockable – sync locks for distributed systems (lockable.dev)
Hi guys, creator of lockable here - the easiest way to think of lockable is as `flock` for when you don’t have a shared file system. You can use it to control concurrent access to resources or to ensure only a single instance of a process runs at any given time.
Your processes can acquire, refresh and release locks via simple HTTP requests, so it’s language/framework agnostic. E.g. with `curl`:
$ curl https://lockable.dev/api/acquire/my-lock-name
{
"response": true //or false, if the lock wasn’t available
}
$ curl https://lockable.dev/api/release/my-lock-name
Feel free to play around, the free tier is fully functional. Happy to hear any feedback you might have.
51 comments
[ 3.7 ms ] story [ 99.6 ms ] threadIt's not a difficult conceptual task to keep track of some locks in a Postgres DB (or use PG advisory locks), but you still need to:
unless the only thing you are trying to protect with your lock is access to other rows in the same postgresql server
The trickiest part is correctly writing the logic that spawns the secondary thread used for heartbeats.
But if it does synchronous replication without using a consensus algorithm like Paxos or Raft then the system become unwrittable if an instance go down.
In a similar vein, I guess you can ask what happens if a client first checks if a lock is available then tries to acquire it, in two separate steps; but in that case, there's no guarantee that the lock wasn't acquired in between checking and acquisition.
https://redis.io/commands/setnx/
How do you release a lock reliably? How do you solve the problem of releasing accidentally while using a resource?
Can the lock jam locked if the process dies?
I would use Consul for this or I would try avoid needing to lock to begin with.
Even better is to use a language such as bloom Lang.
> How do you ensure that a lock isn't acquired by two requests?
Indeed, the compare and set is done atomically. It's guaranteed that the lock can only be acquired by at most one process.
> How do you release a lock reliably? How do you solve the problem of releasing accidentally while using a resource?
A lock is released in one of two conditions:
I'm not sure what you mean by "releasing accidentally" - if nobody calls /release then the lock won't be released.> Can the lock jam locked if the process dies?
Locks come with a lease which expires after a set amount of time. If lockable doesn't receive a heartbeat to renew the lease, the lock is released automatically.
> I would use Consul for this or I would try avoid needing to lock to begin with.
For sure Consul is an alternative, so are ZooKeeper and things like ETCD - lockable is intended to be a no-setup alternative to something like that.
This happened on a project I was on, two processes would join a RabbitMQ on a service that was not safe to load balance.
I suspect it could happen if the program using your lock service is not implemented properly and the lease expires but the program doesn't realise.
The lockable server guarantees that e.g. if multiple /acquire requests come in for the same lock in a short time span, only one request will be successful. You are correct in that, without care, there can be pathological cases where a client may not realize their lease has expired.
You open a session and the session open many locks.
if you can’t talk to server the session expire and all lock are released by the server but also the client library notify that the sessions expired.
Also etcd and consul and zookeeper use The Raft consensus algorithm. it’s literally impossible to make a lock server that is safe without using a consensus algorithm for replication.
You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.
Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.
edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
How is it dangerous? What danger would the user be in?
There's plenty of critical software in the world, some of which has real world consequences if there are bugs.
This implementation has "heartbeats" so I wonder whether it solves the problem.
See this amazing article by Martin Kleppman, author of Designing Data-Intensive Applications.
"How to do distributed locking"
https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
In order to deal with long-running processes, the client Python implementation uses a separate thread for sending periodic heartbeats to the lockable server, which serves to do 2 things:
The GP's point was that the heartbeat thread can hang in pathological cases, which means the main worker thread would not be notified that it has lost the lock.This can be addressed in a few ways - one way being by adding fencing tokens[0]. However, that requires modifying the underlying resource you are accessing.
[0]: https://ebrary.net/64834/computer_science/fencing_tokens
The Python client implementation can be improved, for sure. In particular, the pathological case that is difficult to deal with is the one where the heartbeat thread pauses at the worst possible time (the "pathological hang" you mention) and so the main thread doesn't notice its lease has expired.
Lockable is designed as an easy-to-use drop-in locking mechanism which requires minimal changes client-side. The downside is that this puts the onus on the client implementation to fully cover all pathological cases.
I agree that the documentation should be clearer about these limitations and requirements on the client.
Thanks for the feedback, it's really useful!
Numerous AWS (really early) internal eng talks I saw, hammered home the point that distributed locking / leasing was one of the hardest problems to engineer for, if not the hardest, often quoting Chubby and Paxos Made Live papers.
Today, if I were to need a form of distributed co-ordination, I'd reach for Durable Objects to see if it fits my needs.
However, TTLs can be set to be arbitrarily long (e.g. multiple days), which means, in practice, you can avoid losing locks due to networking issues.
Any reason Azure Blob Storage couldn't be used as a no-setup, managed, cheap way to acquire distributed locks?
if you look at azure event hub library they use blob as lock to partition the consumer instances at runtime.
You'd also need to "sync" the lock resource, which is accessed over REST. At the very least, you need some sort of Idempotency-Key for your REST API.
(Imagine the scenario where /acquire succeeds on the server, but the network response fails before the client gets to read it. The client retries. How does the server know it's the same request?)
If the service goes down and restarts, it will it lose all locks currently held. At this point, a new client could obtain the lock, while an older client that is still blissfuly unaware of the service going down may proceed assuming that its lease is still valid.
If it is implemented as a single server, then it must persist the lock info to disk before granting a lock. Or one can get durability using replication. In which case I would just as soon use zookeeper.