I can't speak for Jespen. But I am using it from last several years and it never ditched me even at the transaction data flow rate of 20K per second.
Database alone isn't the deciding factor, how you used it in your application architecture matters the most. No database is perfect but some are more flexible than others.
In fact we replaced Kafka with Change Streams( little over 7k messages per second) and ES with text search (2K queries per second) in MongoDB on 16 CPU core nodes in our cluster.
I don't believe in any benchmark unless it matches the business use case and development practices I use at work. I recommend the same to others.
HSBC Commerical might have just asked everyone to put their data into a mongodb data store that nobody ever uses, which would make the headline technically true but kind of irrelevant. One thing you can be sure of is that they didn't stop using 65 relational databases.
The article explains quite in depth the rationale behind this. Basically, they think it's too difficult to manage their current relational databases differently in each country, but they need that due to the different regulations and particulars on each country.
With one global DB, they hope they can have one single application for all countries, with country-specific things hidden in only a few sub-documents.
_ It's now a one service environment, one database and one execution path for all the countries. This is made possible because of MongoDB's document model and the ability to map all the different table requirements for each country into a single collection, using sub-documents. Everything is simplified into one collection using country specific identifiers._
How they will address race conditions, eventual consistency and so on is not addressed at all though. I guess they trust Mongo's new transactions will take care of that (at which point, it would be nice to know whether that actually worked and what the performance will be when compared to the old relational system).
These are definitely not source of truth customer transactions. Likely auxiliary databases to power various experiences. Like, a customer walks into the bank and the agent needs to know what services to upsell. Or the customer logs into the website and is shown their current portfolio value.
a) Salesforce is usually out of the question due it not being on-premise.
b) Dynamics and SAP are both ridiculously expensive especially since you would be need to also buy additional software e.g. Windows to run it on. Plus of course it's much harder to find specialist engineering talent who know these products.
c) All three are far too slow both on the ingestion side e.g. millions of mutations a second and on the read side e.g. feeding into a real-time ML model for an upsell prediction. MongoDB is specifically designed to cater to both of these technical requirements since you can mutate sub-documents and retrieve the full document extremely quickly.
a) I'm not suggesting this for banking transactions- that'd be crazy. However customer service, interactions whilst at a branch, handling all sorts of loyalty programs and etc. To my knowledge, Barclays in the UK use Salesforce for some of these.
b) Don't know SAP licensing model but I can't see a major bank not being able to afford $100/month per user for Dynamics/Salesforce.
c) They do some offering for these kind of things,but I can't comment on how itd work on such a scale.
So you have no idea about the business and non-functional requirements or the architectural or engineering considerations that went into this decision. But yet you believe it's such a bad decision that you physically cringe.
Did you ever actually read mongo db code? I had to, and only the drivers, but when I learned that the C++ driver would just nope out of town in some cases and literally exit() the application, I made the decision to get rid of that stuff ASAP. There are other instances, pymongo cannot be forked, for instance. Good luck using it inside celery. The connection string parsing is another matter. All in all their driver's code looks good at the surface, everything documented etc. But the comments are more or less superfluous and the design is a mess.
I'm not in the banking industry but this sounds like the type of industry where you really need strong DB schemas in place and not something mongo-like ...
But the database doesn't need to be the component enforcing those schemas, right? They could validate the schema of a document on storage and retrieval. And keeping the schema version within a document would allow for multiple schema versions to exist within a single collection, which then would allow lazy migrations.
I'm not saying I'd choose Mongo specifically for this job (if for anything), but the lack of DB enforced schema might not be the biggest problem here.
This is the problem I’m alluding to in another comment about NoSQL in this thread. Why would you want to enforce the schema in code when the DB could do it for you? It becomes such a mess and such a burden for the developer!
Until you need to do an emergency transfer of some recovered data volume and there’s a few dozen slightly corrupted JPEGs scattered throughout a few million files.
Also, I don’t think I’d want my filesystem deciding what qualifies as a valid JPEG.
That means either only one place accesses the database (and becomes a bottleneck for change) or everywhere that accesses it has to keep their schemas in line.
Neither of those sound like great situations to be in.
I’m a big proponent of schema-rich SQL but I don’t see the argument here. Enforced schemas can be a bottleneck for change no matter where it’s enforced.
And just because some parts of a schema need to be “in line” doesn’t mean all of it must be. Maybe I care that “Customer ID” is enforced, but not “Australian Tax File Number” or “Document revision annotations”.
I know things about the banking industry and let me tell you, just downgrade your expectations coming from the tech industry. Then downgrade them again, just to be sure.
Banks have notoriously terrible uptime! See, for example, the wide range of multi-hour to multi-week outages suffered by many UK banks in the last few years.
Most of the legacy retail banks regularly take down their online banking for hours at a time to do 'planned maintenance'. I've also heard that this happens to things like payment gateway APIs that aren't directly visible to consumers. Sometimes a Faster Payment might take a suspiciously long time and that's because the API fell over or was down for maintenance.
A lot of what banks do is filing forms. Thousands of ever-changing forms from millions of sources. Even things that aren’t forms are still very document-like: transcriptions, attachments, receipts, invoices, statements, letters... all of it more suited to document storage rather than tables.
Mongodbs newer versions support schema validation by allowing you to register a JSON-Schema description against a collection and validate writes against It.
probably safer to buy put options than short the stock (shorting it exposes you to unbounded losses if the stock price rises), but as you say, you'd need to get the timing right. https://finance.yahoo.com/quote/HSBC/options/
if they've got a decent QA process and they can't get the new system to actually work reliably there's a fair chance this project would get stuck in QA for a year or two, blocked from going live and cancelled, so it wouldn't cause catastrophic data loss, just reallocate a bunch of money from shareholders to contractors / employees / vendors.
HSBC's own website lists dozens of exchange traded warrants with various strike prices and gearing. Need a brokerage with access to the HK market though.
I'm not sure about their choice of Mongo but the initiative seems like it should be helpful for customers like myself who have accounts in a few countries. We can access them all from one app but the experience is inconsistent at the moment. I'm cautiously optimistic.
I've worked contracts in UK banks for a number of years now. Nothing on the mainframes but I know some of the guys that do work on those and, despite the constant stream of flavour-of-the-month tech that we talk about constantly in places like HN, the mainframes work well and will not be replaced any time soon.
The dev cycle for releases and whatnot is incredibly long due to testing and, lets be honest, fear, in case something breaks but when it comes to battle-tested technology, mainframes and DB2 are right up there.
I do know of some code that's over 40 years old and still runs at the core of one of the big banks...
MongoDB, despite its capabilities, is... an odd choice imo! Not trying to second-guess but if I was in that meeting I would have certainly raised an eyebrow.
My prediction is something will go horribly wrong, then they will go "see, we tried modern tech and it fucked us over", and they will commit to seeing their code and tech stack turn a century old before considering any updates.
I read the article and couldn’t figure out what they were actually putting in mongo. If it’s their core processing/ledger then that would be very surprising. But banks have dozens of different systems, some of them would actually almost be suited to Mongo, and some of them have rather loose SLAs. Unless I missed something, the article didn’t really make that clear.
There's another thread on the homepage right now about Crux [0], which is a database designed specifically to be able to operate downstream from multiple legacy systems using bitemporal indexes to stitch everything together. This pattern is very common in banks e.g. for calculating and reporting on risk positions.
I expect you're right that this story is purely about using Mongo for downstream systems.
That thought did occur to me as well but, in taking the article at face value it's most certainly some core aspect of their banking systems at the very least...
If they are intent on replacing the mainframe then, imo, MongoDB would be a bad choice but who knows... the new, challenger banks don't use mainframes.
Any ideas what some of the challenger banks are using? I've not seen any examples beyond Nubank (~$10B valuation), who have bet big with Datomic: https://www.youtube.com/watch?v=VYuToviSx5Q
Not sure exactly but any job ads I've seen for the likes of Monzo and Starling talk about backend engineers that have a deep knowledge of CI/CD and you don't do that on a mainframe.
Also, Monzo are heavy users of Kubernetes, so again, no mainframe. [0][1]
Starling use AWS and GCP [2]
In addition, the cost of entry of a mainframe system is a high bar when you can spin up a server on Azure in seconds and deploy code to it in a handful more seconds and then scale it to the heavens in yet just a few more seconds... all for less than the hourly cost of the IBM salesman.
I've yet to see any jobs that mention mainframes, Cobol or the like.
> Not sure exactly but any job ads I've seen for the likes of Monzo and Starling talk about backend engineers that have a deep knowledge of CI/CD and you don't do that on a mainframe.
The part of the bank that handles core processing is going to be a very small portion of their engineering team. Kube and DevOps ads don’t provide any hints about how much they might rely on DB2, COBOL, etc...
The article gave me the impression it might have been for some sort of CRM-like system. I don't know if it would be even possible to replace DB2 with Mongo, the consistency guarantees are too weak. Banks are accountable to the central bank in every jurisdiction in which they operate, I wouldn't see them getting that past one single central bank, let alone 60+ of them.
If you remove the specific technical choice from mind (a great engineering team can move mountains with any data layer, something I've also learnt working in numerous contract teams at multi-billion dollar companies) - the simple act of rewriting and consolidating huge amounts of data/infra (and stacks) can yield enormous gains.
I will be tracking their technical reporting more closely over the next year.
"I will be tracking their technical reporting more closely over the next year."
Just curious, but what specifically do you mean? ie. I'd love to keep an eye on this also, but have no idea where to find these types of (public) reports.
The problem with 40y old code, is where I work we had to pull engineers from retirement to make a regulatory change. In 10y, I am not sure we will have that option.
I’d wager it would cost a whole lot less to train devs up on the existing system than to rewrite. Without understanding the existing code, you’re going to have a tough time replacing it.
I’ve read that in some cases the initial requirements and source code was lost, so making changes to the existing software becomes a reverse engineering problem in addition to a “language proficiency” problem.
This seems like a hiring failure to me, not a tech failure. I've been hired on several occasions to work with a language I had zero prior exposure to, and while there is of course some ramp-up time I managed just fine and eventually became skilled and knowledgable in those languages.
I don't see why COBOL, mainframes, or whatnot would be any different. Perhaps the ramp-up time would be a bit higher, but most skilled programmers will manage. I actually wouldn't really mind working with any of that – I just never had the opportunity.
Really depends what apps the databases are supporting. Big banks have thousands of internal apps (as opposed to client facing services) to support workflows, business analysis and management reporting. Might be a fine choice for some of that stuff.
"is looking to" makes this sound aspirational.
The absence of operational lessons learned that would come out of a deployment make me wonder if this actually happened.
The whole premise of this seems wrong. I've worked on large financial systems that have happily supported different rules for different countries using a relational database. The database was never the issue, it was how to handle testing and feature enablement. This seems like the wrong cure to the problem, and a cure that will end up causing more problems than it solves. i.e. a hack. I'd love to hear more about the rational. I just don't understand it.
A fortune 100 financial institution I once worked for had an initiative to migrate their extremely large, fairly well normalized DB2 database to schema-less dynamoDB.
All new tables had to go into dynamo, and if your service begins consuming an old DB2 table, it had the job of migrating that data to dynamo.
The end result of this was applications that half pointed to DB2, half dynamo, confusing new data with old, and tons of bugs relating to losing entries into dynamo tables under outdated keys, not being able to clearly match up customer data, and myriad problems that were never a consideration using a relational database. Needless to say, I pulled all of my assets out of this institution as soon as I had the chance.
I've spent decades in enterprise space and unfortunately developers often don't have a clue how actually things work.
Companies like IBM specialise in royally screwing companies through their licensing deals and with their legal enforcement teams. They are nothing like dealing with a normal startup vendor or with someone like AWS. They play hard ball.
So sure you can complain that they should've just stayed with DB2. But usually that means paying tens of millions which then increase each year since they know you won't migrate.
That's why companies take drastic decisions and move to the cloud even if it's technically not the best option.
I have no actual information about this migration but I wouldn’t be surprised to learn that the databases that are being consolidated are for some minor role or are separate instances with duplicate data for analytics or whatever. It sounds wildly unlikely that they are going to smush together all their customer accounts in one MongoDB instance. As well as being a huge technic challenge I’m sure regulators would have something to say about this.
This is very likely. Most banks have stupid-huge quantities of products and services, each with extremely elaborate requirements and data storage needs. “65 relational databases” might well only represent a few percentage points of HSBC’s data footprint.
TBH I think so, too. I guess this is just a glorified global network drive to put all the different, country-specific documents like Excel sheets, Word docs, PDFs and some such.
Sounds like a win-win for them. They get to market themselves as staying up to date with tech (although the HSBC personal banking app I use indicates otherwise). And when they "lose track of" the money they move for the cartel and terrorists, they can blame MongoDB and spare some execs the jail time.
There’s been a lot of talk about the Jepsen-Mongo affair on HN lately, but the problem I have with Mongo and NoSQL in general is much more basic - it’s modeling relationships, which always end up appearing in every data model I’ve ever designed.
I’m aware you can keep references from one doc to another but it always ends up being a messy affair even at smallish scales, and it feels as if the cognitive burden of managing these relationships ends up falling on the dev, instead of being managed by the DB.
For those of you using MongoDB in large scale large production apps, is this not a problem? Is your underlying business domain really non-relational, or do you manage to comfortably run highly relational models on a doc-based DB? How?
Mongodb now supports effectivly "Joins" in its aggregation framework, so it can do some relational style data representation. The latest versions also support transactions.
Yeah but they do not recommend using that. I think the one nosql afficianado in my team spoke with mongo support and they recommended reassessing our document structure for it. (So a migration after all... )
Also in their blogpost [1], about $lookup they say:
We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area.
The whole thing sounds like a terrible idea. Why would you want to put all the data from at least 65 different applications in the same table? In MongoDB? To save cost?
I simply don't get it. It doesn't make sense to me, but perhaps they know something we don't.
360 customer view as an example. If you are feeding a real-time decisioning, support, advertising system then you need all of the customer features accessible as fast as humanly possible. Ideally a consistent O(1) time.
Or it could be a real-time fraud system where you need every bit of customer data in one payload to feed into an ML model.
You can't do any of that if you are having to do multiple joins across federated databases.
Lots of negative comments about MongoDB for folks who don't use it, but perhaps give NoSQL a chance and keep an open mind? there is no need to structure the data into tables and schema does not need to be enforced at the DB level.
Google has been running its entire infrastructure on a NoSQL database for years.
Perhaps folks need to keep some open mind and see if that works instead of dismissing a new trend?
For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not. Neither is Cassandra. Kafka is boring, even more so if they kill of Zookeeper.
NoSQL has proven to be hard to get right than was first assumed and at the same time RDBMS’es have improved and adopted many ideas from the NoSQL world faster than expected, often with better results.
What’s no clear from the article is why MongoDB was picked, so maybe there’s something that made it the obvious choice, it’s just hard to see what the might be.
"For MongoDB it’s mostly a case of it had issue years ago when it was the new hotness, but it should be a boring choice by now, and it’s not"
SQL has been around since birth of computing, give NoSQL some time.
"NoSQL has proven to be hard to get right than was first assumed"
Nothing has been proven, again give it a chance, SQL had years of optimization.
Also, no need to downvote if you like SQL or your job depends on it, or that is all you know, just debate with reason and evidence, I've used both and open to change my mind with evidence and case studies which we don't have many yet and folks here predicting doomsday to HBSC without keeping an open mind.
“Google has been running their entire infrastructure...”
I’m skeptical about this - Google’s entire infrastructure is so large that surely they must be using a wide variety of db technologies?
I took a databases course that did mention that part of the Google search infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a doc-based approach, it was more akin to a relational DB but with flexible columns (can’t remember the details sorry).
At any rate I don’t feel that this is any vindication for NoSQL, as Google builds much of its underlying tech from scratch and its hardly comparable to the out-of-the-box solutions that the rest of us, even massive corporations like HSBC, have to work with.
"Bigtable development began in 2004[3] and is now used by a number of Google applications, such as web indexing,[4] MapReduce, which is often used for generating and modifying data stored in Bigtable,[5] Google Maps,[6] Google Book Search, "My Search History", Google Earth, Blogger.com, Google Code hosting, YouTube,[7] and Gmail.[8]"
"Bigtable is one of the prototypical examples of a wide column store. It maps two arbitrary string values (row key and column key) and timestamp (hence three-dimensional mapping) into an associated arbitrary byte array. It is not a relational database and can be better defined as a sparse, distributed multi-dimensional sorted map."
I'm aware of that, but read the other comments, most of the criticism is around the nature of NoSQL versus using SQL databases, the most common criticism is no scheme and the underlying common assumption is that banks are better off with SQL, after all, that is what they have been using for years... maybe not?
The other statements are not even well reasoned, you read things like "this is madness" or "they up to disaster", "lol just lol", "they're gone", not explaining why. You would expect better arguments from the folks here.
Some pointed flaws in the design or implementation, and I'd argue with time and money, it will get sorted out, again SQL had years optimization and research.
I'd go further, I'd argue is that NoSQL is more flexible and easier to work with than SQL and I think it is a good thing that large organizations are given it a real try, so we have a large scale case studies, instead of completely dismiss the effort as a failure from the get-go.
Because I think there are a lot of highly educated and tech people this forum. Thus, I'd assume they would more inclined to value reasonable and logical arguments in contrast to an online space like a general Facebook group for example.
291 comments
[ 3.2 ms ] story [ 108 ms ] threadDatabase alone isn't the deciding factor, how you used it in your application architecture matters the most. No database is perfect but some are more flexible than others.
Are you using ACID transactions?
In fact we replaced Kafka with Change Streams( little over 7k messages per second) and ES with text search (2K queries per second) in MongoDB on 16 CPU core nodes in our cluster.
I don't believe in any benchmark unless it matches the business use case and development practices I use at work. I recommend the same to others.
This is the video if anyone is wondering:
https://www.youtube.com/watch?v=b2F-DItXtZs
I remember specific flaws/hacks with bitcoin websites coming down to mongo race conditions/inconsistency issues.
_ It's now a one service environment, one database and one execution path for all the countries. This is made possible because of MongoDB's document model and the ability to map all the different table requirements for each country into a single collection, using sub-documents. Everything is simplified into one collection using country specific identifiers._
How they will address race conditions, eventual consistency and so on is not addressed at all though. I guess they trust Mongo's new transactions will take care of that (at which point, it would be nice to know whether that actually worked and what the performance will be when compared to the old relational system).
Clearly they have some expertise, I'd love a more in depth look at how they arrived at the decision.
b) Dynamics and SAP are both ridiculously expensive especially since you would be need to also buy additional software e.g. Windows to run it on. Plus of course it's much harder to find specialist engineering talent who know these products.
c) All three are far too slow both on the ingestion side e.g. millions of mutations a second and on the read side e.g. feeding into a real-time ML model for an upsell prediction. MongoDB is specifically designed to cater to both of these technical requirements since you can mutate sub-documents and retrieve the full document extremely quickly.
Classic HN.
I'm not saying I'd choose Mongo specifically for this job (if for anything), but the lack of DB enforced schema might not be the biggest problem here.
Also, I don’t think I’d want my filesystem deciding what qualifies as a valid JPEG.
But yes, I can see the theoretical appeal.
Neither of those sound like great situations to be in.
And just because some parts of a schema need to be “in line” doesn’t mean all of it must be. Maybe I care that “Customer ID” is enforced, but not “Australian Tax File Number” or “Document revision annotations”.
Most of the legacy retail banks regularly take down their online banking for hours at a time to do 'planned maintenance'. I've also heard that this happens to things like payment gateway APIs that aren't directly visible to consumers. Sometimes a Faster Payment might take a suspiciously long time and that's because the API fell over or was down for maintenance.
At least the data integrity is pretty good.
Ps. I think they never heard of database per service when looking at their microservice graph. Lol
if they've got a decent QA process and they can't get the new system to actually work reliably there's a fair chance this project would get stuck in QA for a year or two, blocked from going live and cancelled, so it wouldn't cause catastrophic data loss, just reallocate a bunch of money from shareholders to contractors / employees / vendors.
> microservice graph
banks have graphs of macroservices.
https://www.warrants.hsbc.com.hk/en/tools/search/ucode/00005...
I'm not sure about their choice of Mongo but the initiative seems like it should be helpful for customers like myself who have accounts in a few countries. We can access them all from one app but the experience is inconsistent at the moment. I'm cautiously optimistic.
The dev cycle for releases and whatnot is incredibly long due to testing and, lets be honest, fear, in case something breaks but when it comes to battle-tested technology, mainframes and DB2 are right up there.
I do know of some code that's over 40 years old and still runs at the core of one of the big banks...
MongoDB, despite its capabilities, is... an odd choice imo! Not trying to second-guess but if I was in that meeting I would have certainly raised an eyebrow.
Good luck to them.
I expect you're right that this story is purely about using Mongo for downstream systems.
[0] https://news.ycombinator.com/item?id=23493163
If they are intent on replacing the mainframe then, imo, MongoDB would be a bad choice but who knows... the new, challenger banks don't use mainframes.
Also: https://www.datomic.com/nubanks-story.html
Also, Monzo are heavy users of Kubernetes, so again, no mainframe. [0][1]
Starling use AWS and GCP [2]
In addition, the cost of entry of a mainframe system is a high bar when you can spin up a server on Azure in seconds and deploy code to it in a handful more seconds and then scale it to the heavens in yet just a few more seconds... all for less than the hourly cost of the IBM salesman.
I've yet to see any jobs that mention mainframes, Cobol or the like.
[0] - https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...
[1] - https://www.8bitmen.com/an-insight-into-the-backend-infrastr...
[2] - https://blog.container-solutions.com/starling-how-to-build-a...
The part of the bank that handles core processing is going to be a very small portion of their engineering team. Kube and DevOps ads don’t provide any hints about how much they might rely on DB2, COBOL, etc...
I will be tracking their technical reporting more closely over the next year.
Just curious, but what specifically do you mean? ie. I'd love to keep an eye on this also, but have no idea where to find these types of (public) reports.
What's a good example of this working out well?
I don't see why COBOL, mainframes, or whatnot would be any different. Perhaps the ramp-up time would be a bit higher, but most skilled programmers will manage. I actually wouldn't really mind working with any of that – I just never had the opportunity.
I wouldn't claim they are any good, just mentioning that grads are being trained in COBOL.
Geniuses!
Edit: In case you missed the story https://www.bbc.co.uk/news/business-18880269
All new tables had to go into dynamo, and if your service begins consuming an old DB2 table, it had the job of migrating that data to dynamo.
The end result of this was applications that half pointed to DB2, half dynamo, confusing new data with old, and tons of bugs relating to losing entries into dynamo tables under outdated keys, not being able to clearly match up customer data, and myriad problems that were never a consideration using a relational database. Needless to say, I pulled all of my assets out of this institution as soon as I had the chance.
Companies like IBM specialise in royally screwing companies through their licensing deals and with their legal enforcement teams. They are nothing like dealing with a normal startup vendor or with someone like AWS. They play hard ball.
So sure you can complain that they should've just stayed with DB2. But usually that means paying tens of millions which then increase each year since they know you won't migrate.
That's why companies take drastic decisions and move to the cloud even if it's technically not the best option.
https://youtu.be/GI5kwSap9Ug?t=63
I’m aware you can keep references from one doc to another but it always ends up being a messy affair even at smallish scales, and it feels as if the cognitive burden of managing these relationships ends up falling on the dev, instead of being managed by the DB.
For those of you using MongoDB in large scale large production apps, is this not a problem? Is your underlying business domain really non-relational, or do you manage to comfortably run highly relational models on a doc-based DB? How?
Also in their blogpost [1], about $lookup they say:
We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area.
[1] https://www.mongodb.com/blog/post/revisiting-usdlookup
I simply don't get it. It doesn't make sense to me, but perhaps they know something we don't.
Or it could be a real-time fraud system where you need every bit of customer data in one payload to feed into an ML model.
You can't do any of that if you are having to do multiple joins across federated databases.
Pretty common these days to just rely on Git and not DDLs.
Google has been running its entire infrastructure on a NoSQL database for years.
Perhaps folks need to keep some open mind and see if that works instead of dismissing a new trend?
NoSQL has proven to be hard to get right than was first assumed and at the same time RDBMS’es have improved and adopted many ideas from the NoSQL world faster than expected, often with better results.
What’s no clear from the article is why MongoDB was picked, so maybe there’s something that made it the obvious choice, it’s just hard to see what the might be.
SQL has been around since birth of computing, give NoSQL some time.
"NoSQL has proven to be hard to get right than was first assumed"
Nothing has been proven, again give it a chance, SQL had years of optimization.
Also, no need to downvote if you like SQL or your job depends on it, or that is all you know, just debate with reason and evidence, I've used both and open to change my mind with evidence and case studies which we don't have many yet and folks here predicting doomsday to HBSC without keeping an open mind.
I’m skeptical about this - Google’s entire infrastructure is so large that surely they must be using a wide variety of db technologies?
I took a databases course that did mention that part of the Google search infrastructure runs on a tailor-made NoSQL technology, but it wasn’t really a doc-based approach, it was more akin to a relational DB but with flexible columns (can’t remember the details sorry).
At any rate I don’t feel that this is any vindication for NoSQL, as Google builds much of its underlying tech from scratch and its hardly comparable to the out-of-the-box solutions that the rest of us, even massive corporations like HSBC, have to work with.
"Bigtable development began in 2004[3] and is now used by a number of Google applications, such as web indexing,[4] MapReduce, which is often used for generating and modifying data stored in Bigtable,[5] Google Maps,[6] Google Book Search, "My Search History", Google Earth, Blogger.com, Google Code hosting, YouTube,[7] and Gmail.[8]"
"Bigtable is one of the prototypical examples of a wide column store. It maps two arbitrary string values (row key and column key) and timestamp (hence three-dimensional mapping) into an associated arbitrary byte array. It is not a relational database and can be better defined as a sparse, distributed multi-dimensional sorted map."
Yes, keep an open mind, but please do not use a database that has a history of just losing data in mission critical software.
> there is no need to structure the data into tables and schema does not need to be enforced at the DB level.
There is no need to, just like your car doesn't need safety belts to function.
A database scheme is mostly a safety mechanism as it will prevent you from doing something stupid.
> Google has been running its entire infrastructure on a NoSQL database for years.
Do you have a source for this claim? I would be inclined to believe that the data for my Google account sits in a relational database somewhere.
The other statements are not even well reasoned, you read things like "this is madness" or "they up to disaster", "lol just lol", "they're gone", not explaining why. You would expect better arguments from the folks here.
Some pointed flaws in the design or implementation, and I'd argue with time and money, it will get sorted out, again SQL had years optimization and research.
I'd go further, I'd argue is that NoSQL is more flexible and easier to work with than SQL and I think it is a good thing that large organizations are given it a real try, so we have a large scale case studies, instead of completely dismiss the effort as a failure from the get-go.
You would? Why?
You don't think I should keep that expectation?