Ask HN: How does Booking.com guarantee consistency of reservations on scale?
I have been wondering consistency guarantees at scale, specially with a massive number of concurrent users.
It seems that Booking.com mainly uses MySQL for most product features and use-cases and I am wondering how do they ensure that a listing is not booked by multiple users? Is the reservation system a single service with it's own storage and they run serializable transactions on it? Or do they utilize other means of distributed transactions, such as: distributed locking, 2PC, 3PC or Sagas?
33 comments
[ 4.2 ms ] story [ 86.3 ms ] threadIf the worst case scenario the hotel could be totally screwed up (IT error, bankruptcy, nature disaster, …) and you might not be able to stay there. Booking.com has to have something in the contract they serve you under that says in some cases they will put you up in another hotel, offer a refund or something but maybe in others they don’t.
Even in the short term though it is the real reservation at the hotel that matters, not the state of their system, so there has to be some kind of concurrency control for the system as a whole.
> In the (hopefully rare) event of a problem, they probably try and keep people happy through a standard playbook of things like refunds, giving future coupons or discounts, and trying and arranging alternative lodging.
> Even if their databases could somehow guarantee to be perfectly consistent and no random booking problems occur, they still need those standard policies in place to deal with random hotel problems like flooding or other problems.
Business-wise, if you are worried about contention, the locking is achieved within hotel reservation systems at two levels. One level is the channel, and another level is the hotel itself. The hotel itself holds the ultimate record of note with respect to forward room allocations, re-allocation powers, deferment and cancellation powers, etc. They are the arbiter of truth. However, because hassling staff or systems at the hotel are not always optimal distribution solutions, large channels such as major booking sites will often be hard or soft-allocated groups of rooms in bulk. They are then able to provide their own confident sub-assignment of these rooms before communicating the ultimate guest details to the hotel. Conceptually, this functions similarly to exchange-based trades in crypto, broker-based book allocations in the conventional financial services world, and so on.
The ultimate client-facing business process is generally handled with a soft booking at agreed rates, and then a hard confirmation of the receipt and agreement to booking by the hotel receiving the notification of such from the reservation channel. Different hotels offer different interfaces for this, but not so long ago a huge amount was still done by fax. It wouldn't surprise me if a great deal are still done manually by humans using email. Larger operations use APIs, but all fall back to each hotel's reservations department who have the power to issue contracts, change rates, and override agreements. A web-booking consumer would typically pay and receive a confirmation of payment immediately, and for hotels not offering real time reservation confirmation APIs receive a deferred confirmation via email.
FWIW IIRC booking.com was one of the higher-vis perl holdouts globally, at least 10 years ago.
Source: Around 2007-2009 I developed from scratch a hotel reservation system operating in six human languages which grew to over 3,000 partner hotels. This is not so big these days but at the time was the same size as multiple NASDAQ listed operations.
If we put the entire 100M/day traffic evenly distributed in a single hour we're in the order of 10k qps. Isn't that the same order of magnitude as the absolute max writes you can buy from AWS RDS Postgres for a single-primary setup without sharding? That seems to me to be a worryingly close hard limit that suggests you'd want to start working on something a little more complex
Whereas, if you don't have much traffic, you're probably better off going with a simple solution (sqlite on an average computer) than prioritising scalability.
If you don't know the distribution of requests, assuming a constant rate simplifies calculations; and can be interpreted as "what if the traffic gets as high as this".
sqlite can handle 50k inserts per second. -- Sure, if you need more than 50k inserts in a second, then you can't use sqlite. But, the rate of 50k/second sustained for a day would be half the world's population booking a hotel every day. (Since "seconds in a day" is much smaller than "half the world's population": that the rate for one day is staggeringly high indicates that that rate for one second is also staggeringly high).
Once the user chooses the reservation it's blocked off directly in the Central Reservation System at the hotel so it can't be double booked. If the rooms fill up between the availability being presented and the user deciding the CRS won't accept the reservation then booking.com then has to handle that use case with the user.
So Booking.com itself probably doesn't really have any "floating" reservations where the user is told it's booked and the hotel doesn't know about it. It's all pretty much done in real-time.
Even for a tiny hotel there are cheap and extremely well made booking systems that can connect with booking.com, AirBnB, Expedia, as well as a bunch of other resellers in real-time, keeping everything synchronized and avoiding overbooking.
Of course, you can opt to not have any booking system of your own and put your listing on booking, and then yes booking.com will sell rooms without checking anything - since there's nothing they can check against. If you arrange things like that, you'd have to manually remove rooms from bookings system when you make a sale yourself. But that's self-inflicted harm.
I used to run a hotel a long time ago, and a couple of the large regional resellers tried to scam me into giving them allotments. I said "Sure, will you pay for your allotted rooms wether you sell them or not?", they said "Of course not!" - and that was the end of it.
The "manual" way to interact with booking.com from a hotel is simply to create on booking.com a list of available rooms while on the hotel side those rooms are pre-booked to booking.com (or if you prefer removed from the local list).
At any time you (the hotel) can grow or reduce the list.
As soon as booking.com gets a reservation it confirms it (to the customer), removes that room from the available rooms list and notifies the hotel of the reservation.
The hotel can then decide to leave the list "as is" or add to it one or more rooms.
The claim is not wrong, for flights. There are big systems (Amadeus, Sabre, mostly) who manage flight-reservations for big airlines. But it's a bit expensive, and airlines usually also offer their own reservation-systems now. There are also lowcost airlines and charter flights which are normally not available through them. But probably 90% of commercial flights in the western world happen through those reservation-systems.
And Hotels have a similar situation. You can theoretically book hotels, and even trips with train, ships, buses or cars through the centralized systems, but usually only the big companies are available through the reservation-systems. This has mostly historical reasons, but also because it's expensive, and is a bit complicated.
This is why alternatives like booking.com emerged, who are cheaper, and could adapt faster to the changing market. Amadeus & Sabre are annoyingly old and "heavy" in how they operate. Though, this all is changing slowly. The travel-industry is trying to move to new APIs for handling modern demands of travelers. And The old gangsters have started to branch out to web-stack, with JSON and restful APIs, offer free documentation on Github and youtube, and even have discord now.
Sure, but we are talking of hotel rooms and booking.com, not of flights.
And surely large hotel chains (national and international) have their own systems and interfaces to whatever booking system, but booking.com is AFAIK mainly about small, independent hotels, at the most these latter ones have softwares that integrate booking.com reservations into their internal reservation tableau, but (again AFAIK) they also work with the same mechanisms I described for manual interaction, there is a pool of pre-allocated rooms on which booking.com can make reservations.
Why are you doubting the truth, when you can easily verify? Even most small hotels have connected reservation systems. Booking will simply ping the system for available rooms, put in a reservation if the client makes a reservation, and now there's one room less available for that date. Nobody that is sane will pre-book any of their rooms to booking.com, that is a thing of the past.
I think that's where you're wrong... first of all, the number of hotel booking transactions is nowhere near the number of interactions on a social media site/app, for example. It's not massive at all -- it's something any modern database server can handle easily, with replication for reads as required.
And secondly, even if it were massive, it's not massive per-property which is the only thing that matters in terms of transactions here. Database sharding in this case is incredibly straightforward, compared to many other types of websites. And in both cases, just use standard database transactions.
Now, in reality, as other commenters have attested, it's actually an optimistic communications process with each property that has nothing to do with databases at all, and overbooking can absolutely happen when both Booking.hom and Hotels.com book the same last room before the hotel has confirmed one or the other. And then it gets sorted out manually, by the hotel, because the hotel is aware of it.
This is such an important point that a lot of software devs don't get. If you think about the shape of the data, you can often find ways to naturally partition the data so that you are dealing with much smaller scale.
The backend credit card of the reservation company declined the charge, so the hotel room had been booked, but the hotel hadn’t been paid.
The hotel manager sat on hold for 10 minutes, and the website fixed it.
But to answer your question, it's probably more about hotels advertising truthful number of rooms than booking's system, and I believe booking makes sure hotels don't overbook by penalizing them.
Their tech model is very simple (I know quite a lot of people there), but it works well.
But this is way beyond your level here now
Here's a secret: most of the world runs on BASE. Even banks.
https://devinz1993.medium.com/book-notes-system-design-inter...
Extracted from the post:
Chapter 7. Hotel Reservation System
Functional requirements: - View hotels and rooms. - Reserve a hotel room. - Add/delete/update a hotel room. - Support overbooking.
Non-functional requirements: - Support high concurrency. - Moderate latency (a few seconds for reservation).
Recommendations: - Choose a relational database for the read-heavy workload and ACID properties. - Include an idempotency key in the API for making reservations to avoid double booking. - Use the microservice architecture and use remote procedure calls for inter-service communication. - Keep a pre-populated room type inventory (for 2 years ahead) to support reservations by room type (instead of by room id).
Concurrency control: - Pessimistic locking is prone to deadlocks and tends to have bad performance. - Optimistic locking is usually faster but performance dramatically drops when contention is heavy.
Scaling: - Shard the database by hotel since most queries need to filter by it. - Cache inventory information (aggregates) in Redis to reduce database load and improve read performance.
Tip: - Use the same service to manage reservations and the inventory to avoid data consistency issues that require 2PC or Saga.