There are only atomic clocks in some master servers: "TrueTime is implemented by a set of time master machines per datacenter and a timeslave daemon per machine. The majority of masters have GPS receivers with dedicated antennas; these masters are separated physically to reduce the effects of antenna failures, radio interference, and spoofing. The remaining masters (which we refer to as Armageddon masters) are equipped with atomic clocks. An atomic clock is not that expensive: the cost of an Armageddon master is of the same order as that of a GPS master."
The timeslave daemons running on each machine keep them synchronized with the master time servers, and maintain tight bounds on their inaccuracy.
> Unlike most wide-area networks, and especially the public internet, Google controls the entire network and thus can ensure redundancy of hardware and paths, and can also control upgrades and operations in general
I know this is a single system, but I'll still say it. This seems like another step in a scary trend for our internet.
The parent comment author seems to be disturbed by the phrase "controls the entire network." Setting aside his unwarranted paranoia, it does make me wonder which organization has the largest and most reliable corner of the internet. Google certainly qualifies by sheer volume of hardware and network infra.
This is no different from a myriad of other companies that have private networks on private fiber that span large geographical paths. Google's is probably bigger than most (both in terms of geography and throughput), but it's nothing new to have private fiber to ensure latency/throughput/reliability.
I am an employee, and have my biases, but I’ll always prefer a customer's data stay on our backbone and not be passed through the public internet. Our customers also prefer it, and it's not something other Cloud providers can fully cover.
That probably came off as more offensive than I intended. I just feel a more decentralized internet is a stronger one. Not assuming any nefarious intent.
> Today, we’re excited to announce the public beta for Cloud Spanner, a globally distributed relational database service that lets customers have their cake and eat it too: ACID transactions and SQL semantics, without giving up horizontal scaling and high availability.
This is a bold claim. What do they know about the CAP theorem that I don't?
Separately, (emphasis mine):
> If you have a MySQL or PostgreSQL system that's bursting at the seams, or are struggling with hand-rolled transactions on top of an eventually-consistent database, Cloud Spanner could be the solution you're looking for. Visit the Cloud Spanner page to learn more and get started building applications on our next-generation database service.
From the rest of the article it seems like the wire protocol for accessing it is MySQL. I wonder if they mean to add a PostgreSQL compatibility layer at some point.
"High availability" is not the same thing as the "A" in the CAP theorem. Spanner chooses consistency over availability, but it is a HA system in the sense that it can tolerate datacenter outages.
What do they know about the CAP theorem that I don't?
I don't know your expertise on the subject, but they do have a post on this topic.
Some highlights:
"Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA."
"The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system."
They write that it's a CP system. So A is not guaranteed, but in practice they are able to provide A most of the time, with a private network and only 1 failure in all of 2016 (whatever that means).
[disclosure: I'm the CTO at NuoDB, one of the NewSQL systems discussed in [4] above]
Totally agree with your comment. While the "NewSQL" category is pretty broad and hard to define cleanly, one common theme is understanding the trade-offs that matter to your applications & operators as you distribute & scale SQL services.
For instance, with NuoDB we've decided that standard ANSI SQL with complete DML/DDL support is a must-have for migrating existing applications. As you distribute the database, the ACID contract you expect with those tools also has to be maintained, and we do that. As others have noted in this thread, however, there are many variants of "consistency." I believe if you're trying to exploit distributed scale, and low-latency through locality of reference, then you're probably willing to trade-off serializability. Using MVCC, logical timestamps (not globally coordinated clocks) and a few other well-understood techniques we provide a scale-out solution that maintains standard levels of isolation and consistency. In practice our customers are traditional enterprises using these capabilities to migrate legacy, high-value systems into modern architectures.
What Google has done is something very few other organizations can do: exploit their massive scale and hardware capabilities to offer something unique. From that perspective, they've also chosen to make trade-offs around syntax, indexing, transaction models etc. There's good commentary from my peer at Cockroach about this already on this thread, but the point is that Google picked a view of the database that makes sense for some applications and not others. Personally, I'm excited that Google has opened up this service and I'm sure it will lead to a lot of good introspection on what we need from modern data management services.
For those who haven't read it yet, I'd highly recommend the Quizlet blog referenced elsewhere on this thread. It's a really good discussion about some trade-offs that Cloud Spanner has made, and why they would or wouldn't map well a given application.
Bottom-line: both Cloud Spanner and NuoDB are examples of the modern, distributed, scale-out systems we're moving towards that focus on consistency, fault-tolerance and modernizing traditional SQL applications.
> This is a bold claim. What do they know about the CAP theorem that I don't?
It's right there in the article:
"Remarkably, Cloud Spanner achieves this combination of features without violating the CAP Theorem. To understand how, read this post by the author of the CAP Theorem and Google Vice President of Infrastructure, Eric Brewer."
Theoretically it means they are giving up on being Partition Tolerant. There was a popular post a while ago about how the P can't be sacrificed. Because if it is... everything else will fail.
Being Google they are probably prideful enough to think their servers could never have an outage. Which yes, I agree with you, that is a very scary claim.
This is thoroughly wrong. Cloud Spanner sacrifices the "A", not the "P." The cool thing being accomplished here is that the sacrifice to the A is greatly reduced (five or more 9s). There are several documents on the subject linked right off that page and elsewhere in these same comments, like this one: https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...
In terms of CAP, Spanner claims to be both consistent and highly available despite operating over a wide area, which many find surprising or even unlikely. The claim thus merits some discussion. Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA.
The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system.
However, no system provides 100% availability, so the pragmatic question is whether or not Spanner delivers availability that is so high that most users don't worry about its outages. For example, given there are many sources of outages for an application, if Spanner is an insignificant contributor to its downtime, then users are correct to not worry about it.
> The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system.
> However, no system provides 100% availability, so the pragmatic question is whether or not Spanner delivers availability that is so high that most users don't worry about its outages.
> From the rest of the article it seems like the wire protocol for accessing it is MySQL. I wonder if they mean to add a PostgreSQL compatibility layer at some point.
> From the rest of the article it seems like the wire protocol for accessing it is MySQL. I wonder if they mean to add a PostgreSQL compatibility layer at some point.
I doubt it, the spanner is not even offering full MySQL capabilities, so it's unlikely to to support any advanced PG SQL.
> clients can do globally consistent reads across the entire database without locking
How is this possible across data centres? Does it send data everywhere at once?
Seems too good to be true of course but if it works and scales it might be worthwhile just not having to worry about your database scaling? Still I don't believe it ;-)
EDIT: further info...
> Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down. Data is divided into groups that form the basic unit of placement and replication.
So it's SQL with Paxos that presumably never get's confused but during a partition will presumably not be consistent.
> In terms of CAP, Spanner claims to be both consistent and highly available despite operating over a wide area, which many find surprising or even unlikely. The claim thus merits some discussion. Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA.
The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system.
No, he's saying it's effectively CAP because the A downtime is so small.
It's one thing to do that for a key-value store. Entirely another to support joins on a globally distributed database. This ain't just one availability zone. Spanner is amazing.
It took them a few years to make it a service, but when they announced its use internally a few years ago, it seemed like the nail in the coffin for in-house database hosting.
There's nothing wrong with saying it's CP, but since we control everything there's extremely rare P. Then he can show availability numbers (which he kinda does).
No, it's engineering. It's the recognition that if periods of unavailability are too small and too rare to be noticed, then the system behavior is indistinguishable from an "available" system in the sense of the CAP theorem.
It's like the "Retina" display you're probably reading from. There are pixels, you just can't see them.
Another point is that since all records are globally timestamped, you can do a read that is consistent at a timestamp in the past (i.e. read data as the database was 1 second ago, or something like that).
If data from other places has synchronized to your zone, you may be able to do this globally-consistent read while only touching your local datacenter (because TrueTime guarantees that no other records anywhere in the system will be created at the time you are querying).
Note: I work at Google, but I don't know more about Spanner than the Spanner paper.
I wonder why they charge a minimum of $0.90 per node-hour when they offer VMs for as little as $0.008/hr. This is hugely useful even for single-person startups, so why charge a minimum of ~$8,000 per year?
Hugely useful but also hugely different from an engineering/coverage perspective, perhaps.
Companies with more data than can fit in a single-instance RDBMS system (like >3TB of hot data, more throughput than a single node can handle) but still seeking transactional consistency are a clear use case. Single-person startups could definitely benefit, but it's a less-likely scenario that they would require the level of coverage Spanner provides.
Because you're paying not for cost plus, but value added. To make a meaningful cost comparison, the best alternative must be considered, which is you spending your own labor to engineer and build a similar system yourself.
Concur. The pricing makes sense for their "target market" of folks who currently have a "bursting at the seams" MySQL or PgSQL instance, but it locks out folks just getting started with a tiny database and low load. This seems like bad positioning: the "bursting" folks will have to decide between the cost of re-hosting their whole system on the Cloud Spanner and trying to incrementally keep their current platform running; the small folks who would like to organically grow on a platform without scaling limits are locked out of the low end, so by the the time they are big enough to need Cloud Spanner... they too with be forced into the "re-host or muddle on" decision.
This seems like a real problem. The "bursting at the seams" people will need a pretty significant rewrite to run on Spanner, which seriously limits the appeal.
The ideal market for Spanner seems to be new projects developed by big companies that know that big traffic will arrive on day 1. In other words, Google. Which I guess shouldn't be too surprising...
To be fair, you shouldn't even run MySQL on an f1-micro. This is more on par with an 18 vcpu raw server, before you'd even consider any value that the software provides.
I'd certainly love to see us get to a world where we can split up a single spanner "install" in an isolated, multitenant manner, but even for a small company, $8k/year is admittedly a small fraction of one engineer. At a company with several, you can share your single Spanner instance just like you would any other database.
Disclosure: I work on Google Cloud (but not Spanner).
> even for a small company, $8k/year is admittedly a small fraction of one engineer
This is certainly true, but overlooks the fact that many major products start out as experimental projects, and $8k/year is a significant investment for an experiment.
If there's a reasonable upgrade path from traditional databases like MySQL and Postgres this shouldn't be a big deal, but if the answer is "rewrite your app" it will probably be a friction point for adoption.
A Spanner node is not related to a VM instance, they are entirely different concepts. A Spanner node refers to a collection of regional (and eventually global) compute and storage resources, while a VM instance is just a virtual machine running on a single physical host.
They say Spanner is "both consistent and highly available despite operating over a wide area". So not 100% availability but they've got it to "more than five 9s of availability (less than one failure in 1066)."
I didn't pick up on it until just a moment ago, but when you say "They say", it's actually Eric Brewer saying that -- who's most known for coming up with the CAP theorem. I think they've got a pretty good understanding of it!
ha whoops good catch! I knew the post was written by Brewer, but slipped up with the they... yeah I think we have it on pretty good authority that they system doesn't so how violate CAP yet addresses it's challenges.
The team here at Quizlet did a lot of performance testing on Spanner with one of our MySQL workloads to see if it's an option for us. Here are the test results: https://quizlet.com/blog/quizlet-cloud-spanner
What's the SQL and wire compatibility level? MySQL?
EDIT: Found quite a bit of my answers in your linked article:
> Cloud Spanner uses a SQL dialect which matches the ANSI SQL:2011 standard with some extensions for Spanner-specific features. This is a SQL standard simpler than that used in non-distributed databases such as vanilla MySQL, but still supports the relational model (e.g. JOINs). It includes data-definition language statements like CREATE TABLE. Spanner supports 7 data types: bool, int64, float64, string, bytes, date, timestamp[20].
> Cloud Spanner doesn't, however, support data manipulation language (DML) statements. DML includes SQL queries like INSERT and UPDATE. Instead, Spanner's interface definition includes RPCs for mutating rows given their primary key[21]. This is a bit annoying. You would expect a fully-featured SQL database to include DML statements. Even if you don't use DML in your application you'll almost certainly want them for one-off queries you run in a query console.
> Though Cloud Spanner supports a smaller set of SQL than many other relational databases, its dialect is well-documented and fits our use case well. Our requirements for a MySQL replacement are that it supports secondary indices and common SQL aggregations, such as the GROUP BY clause. We've eliminated most of the joins we do, so we haven't tested Cloud Spanner's join performance.
This seems like it'd prevent any kind of easy switch over to Spanner.
Just to be clear, the JOINS were removed for the vertical sharding prior to looking at Cloud Spanner. Cloud Spanner fully supports complex JOINS of many times (e.g. INNER, OUTER)
How did you go from 10s of ms for an update for 2012 Spanner, to sub-ms update (best case) for Cloud Spanner according to Quizlet? Did TrueTime get an order of magnitude better? Or is Quizlet measuring the wrong thing?
> We've eliminated most of the joins we do, so we haven't tested Cloud Spanner's join performance.
The join performance is by far the most interesting part of this to me. A more traditional NoSQL solution sounds like it would have worked just as well for you, sans all the atomic clock fanciness. Joining across geographically disparate data is a real trick, and it seems like there would be some physical performance limits?
> Not every application can handle Spanner's ~5ms minimum query time, but if you can, then you can have that latency for a very high-throughput workload
>> Cloud Spanner doesn't, however, support data manipulation language (DML) statements. DML includes SQL queries like INSERT and UPDATE. Instead, Spanner's interface definition includes RPCs for mutating rows given their primary key[21].
Does this mean I need to rewrite my application?
My application uses an ORM and it typically converts my logic to SQL statements and fires them off to Postgres. Would I need to change it such that it doesn't issue INSERT / UPDATE statements?
No, though it does imply you would need to write adapter code to have your ORM call the equivalent stored procedures for inserts or deletes. Most ORMs can do this to varying degrees.
> So a query that accesses 10 rows in disparate parts of the primary key space will take longer than one where the keys reside on the same splits. This is expected with a distributed system.
No, why? Query can be executed in parallel.
BTW, isn't 20k/sec is a very very small performance for 30 node installation. Cassandra can handle 50k+ (both writes and read) on a single node. When in most queries you are trying to collect data from many nodes it will scale almost linearly.
I don't think that comparison holds. It's easy to push 50k+ on a single node, you're basically only resource bound on that machine. Pushing 20k+ on something that's globally consistent spread out over so many instances is a different exercise entirely. It also depends on the level of consistency you're asking from Cassandra. You'd probably need to set this to EACH_QUORUM or ALL to mimic the behaviour Spanner gives you.
And yes Cassandra will scale linearly-ish as long as you're in the same datacenter. Try running a geo-distributed 30-node Cassandra ring and it's a whole different story at that level of consistency and availability.
Sure, but you can make geo-replication and even implement proxy for providing consistency across DC, sure it won't be that fast in terms of latency, but throughput will be the same.
How does this proxy that provides cross-DC consistency work? And how does it guarantee that you'll get globally consistent reads?
Most cases I've seen latency can affect throughput plenty so I doubt your assertion that it won't affect throughput quite a bit. Even more so for anything that relies on TCP/IP.
I'd highly recommend reading the Bigtable[0] and Spanner[1] papers first and maybe then we can have a sensible and fruitful argument.
Just wanted to say thanks for this writeup. This is really excellent, to the point I was passing around your blog post in lieu of the GCP announcement.
I'm curious about how you have MySQL configured. If the query cache is enabled you will see MySQL plateau. There's also the all important innodb_buffer_pool_size and innodb_flush_log_at_trx_commit. See this for more info https://www.percona.com/blog/2013/09/20/innodb-performance-o...
We typically do yearly performance audits with Percona to ensure our databases are optimally configured. We disable the query cache. We set innodb_buffer_pool_size based on a % of total memory (as MySQL will use more than that in total, much for allocating connection structs and things like that). We set innodb_flush_log_at_trx_commit to 0 which is not ideal for data integrity but gives us more performance. In practice, because of Google's Live Migration technology we have never experienced a crash due to hardware on our master nodes (LM will run an emergency migration for any server that has hardware that is detected to be going bad before it becomes a problem) and disks are abstracted away even further. Our main risk is the kernel crashing or MySQL crashing which we've been fortunate enough to not happen on our masters. Spanner provides ACID compliance by default with all the scaleability and performance we get out of it.
I wonder how this will affect adoption of CockroachDB [1], which was inspired by Spanner and supposedly an open source equivalent. I'd imagine that Spanner is a rather compelling choice, since they don't have to host it themselves. As far as I know, CockroachDB currently does not support providing CockroachDB as a service (but it is on their roadmap) [2].
I imagine the globally distributed database market is big enough for more than one winner. The presence of competitors can sometimes even be a boon, increasing the visibility of a market's goods relative to other similar goods.
Capital expenditures might shed some light on this. I don't think there's enough public data to be clear but in 2015 Amazon (4.8B), Google (9.9B) and Microsoft (5.9B) were at least on the same order of magnitude in terms of CapEx, whereas other major "datacenter" companies like Rackspace (475M) are much smaller.
I don't think you can draw any definitive conclusions from this, but calling it a class of size 1 or 2 is probably an overstatement of Google (+/- Amazon)'s advantage over Microsoft at least.
> A simple statement of the contrast between Spanner and CockroachDB would be: Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception.
CockroachDB is waiting for time keeping hardware to improve.
Eric Brewer's post on Cloud Spanner mentioned that Google intends to expose TrueTime to customers at some point. If/when that happens, it would be very interesting to see CockroachDB's performance on Google Cloud. (They might have to do some engineering work to accomodate whatever TrueTime API is exposed, but when timekeeping is fundamental to your product, that seems worthwhile.)
If the clock offset is too high (more than 250ms), we should use another transaction model, Google Percolator is a good fit before the unforeseeable improvement in the hardware. Based on the monitoring of clock offset on cloud, TiDB chose to use the timestamp oracle to allocate timestamp, which is much faster.
Google launching Spanner is generally a positive thing for our industry and our product. It's more proof that what we're aiming for is possible and that there's demand for it. We expect that in five years, all tech companies will be deploying technology like ours.
One of the big differences is that Spanner only uses SQL for read-only operations, with a custom API for writes. We use standard SQL for both reads and writes, which means we also work with major ORMs like GORM, SQLAlchemy, and Hibernate (docs should be live today or tomorrow). Spanner's custom write API will make it difficult to work with existing frameworks, or to convert an existing application to Spanner.
Cloud Spanner only works on Google Cloud and is a black-box managed service. CockroachDB is open source and can be run on-prem or in any cloud on commodity hardware. (We don't offer CockroachDB as a service yet, but may in the future)
At this point, both products are still in beta and are still missing features like back-up and restore (according to the Quizlet blog post). We plan to launch CockroachDB 1.0 with back-up / restore enabled.
> Google launching Spanner is generally a positive thing for our industry and our product. It's more proof that what we're aiming for is possible and that there's demand for it. We expect that in five years, all tech companies will be deploying technology like ours.
Echo on this! It's truly exciting moment for each and everyone in the field.
(CockroachDB CTO here) We haven't implemented everything in the standard yet (Nor will we by 1.0 - there's a lot of stuff there!), but we are aiming to ultimately be compliant with the SQL standard. For example, when we introduced "time travel queries" (https://www.cockroachlabs.com/blog/time-travel-queries-selec...) we adopted the SQL-standard syntax "AS OF SYSTEM TIME" (as opposed to the non-standard out-of-band parameter used in Cloud Spanner)
That's not very much given the capabilities and managed service. Anything cheaper probably means the single-node managed SQL offerings are more than enough.
I am also interested in how it compares to NewSQL databases like NuoDB. NuoDB has been positioning itself as a very similar type solution (no compromise relational distributed database) to Cloud Spanner for a while (minus the cloud hardware provided for you).
> The underlying time references used by TrueTime are GPS and atomic clocks. TrueTime uses two forms of time reference because they have different failure modes... TrueTime is implemented by a set of time master machines per datacenter and a timeslave daemon per machine. The majority of masters have GPS receivers with dedicated antennas; these masters are separated physically to reduce the effects of [GPS] antenna failures, radio interference, and spoofing. The remaining masters (which we refer to as Armageddon masters) are equipped with atomic clocks. An atomic clock is not that expensive: the cost of an Armageddon master is of the same order as that of a GPS master.
Here's a very rough (and non-practical due to the large time error) implementation of TrueTime that I wrote a while back that uses the regular linux apis:
> Today, we’re excited to announce the public beta for Cloud Spanner, a globally distributed relational database service that lets customers have their cake and eat it too: ACID transactions and SQL semantics, without giving up horizontal scaling and high availability.
This sounds too good to be true. But it's Google, so maybe not. Time to start reading whitepapers...
They answer your question just two lines under the one you are quoting:
> For distributed systems over a “wide area,” it's generally viewed that partitions are inevitable, although not necessarily common. If you believe that partitions are inevitable, any distributed system must be prepared to forfeit either consistency (AP) or availability (CP), which is not a choice anyone wants to make. In fact, the original point of the CAP theorem was to get designers to take this tradeoff seriously. But there are two important caveats: First, you only need to forfeit consistency or availability during an actual partition, and even then there are many mitigations. Second, the actual theorem is about 100% availability; a more interesting discussion is about the tradeoffs involved to achieve realistic high availability.
A CA system is not a system that doesn't have partitions, it's a system that works under the condition that there are no partitions (ie. it is a non-partition-resistant system).
> , it's a system that works under the condition that there are no partitions
But that is not a choice with distributed systems, unless as soon as a partition happens the system shuts down immediately. That is it effectively disappears and never re-appears again. But that's not a CA system then?
Or saying that it is not partition resistant is also difficult because the system in case of a partition will do _something_. The typical choices is that it either responds to clients (trying to be AP) or it doesn't (trying to be CP).
That is why I understand CA systems as equivalent to the belief that "partitions can't happen". Which I think is unrealistic.
This is a system which is guaranteed to be consistent and available as long as there are no partitions. If there are partitions, the system isn't guaranteed to be either.
Trivially, by never allowing either half of a partition to make progress while the partition is in place. Since the CAP theorem, by itself, doesn't put a cap(oof) on latency, it is valid to consider a system CA, if it if always available to listen to requests while partitioned, but never able to fulfill them.
This, of course, is effectively useless in practice, and is dependent on an infinite buffer of pending operations, etc.
> Trivially, by never allowing either half of a partition to make progress while the partition is in place.
Doesn't availability mean getting a response on success or failure. If during a partition there is no response on success of failure how is the system available? It seems re-writing a term like "x will happen" to "x will happen after an infinite timeout" should not be valid
It does, but within what bounds? The CAP theorem doesn't specify. One could assume that it means before the partition is restored, but that is only one possible valid interpretation. The PACELC theorem, which is by no means the last word on the story, clarifies this well:
"PACELC builds on the CAP theorem. Both theorems describe how distributed databases have limitations and tradeoffs regarding consistency, availability, and partition tolerance. PACELC however goes further and states that a trade-off also exists, this time between latency and consistency, even in absence of partitions, thus providing a more complete portrayal of the potential consistency tradeoffs for distributed systems."
And I would take that argument one step further and say that latency and partitioning are effectively identical, and from the point of view of any given operation, it is impossible to say whether the system is in partitioned state until max lateny (timeout) has elapsed, because failure to make progress within timeout is the only meaningful definition of partion-induced unavailability.
Sometimes it's helpful to consider a distributed system through the lens of "harvest vs yield" where harvest is the proportion of information in a system reflected by a response, and yield which is the probability of receiving a response at all.
That's a good summary by Aphyr he said it better than me of course.
Now I know it is Eric Brewer himself who wrote that announcement. So I make no claim disproving him or knowing better than, I am just struggling to understand the implications. That CA terms has caused a lot of confusion (at least to me) and so far from what I have read talking about CA systems doesn't make sense.
Never heard of harvest & yield that's an interesting perspective. Thanks for the link, I'll need to go and ponder that for a while. It seems to take a probabilistic approach to these constraints.
But even in that article the description of CA systems is confusing. They describe it it means system can work only in the absence of a partitions. Ok, but then what does it do in the case of partition? The system will presumably do something accept requests, refuse to reply, ... I don't see how it can be both C and A though. That is why I equate that choice with a statement of "I believe partitions will never happen", which I personally think is not realistic.
A useful perspective for me has been that C, A, and P are spectra rather than all-or-nothing. A system may bend a bit on availability when some of its replicas are partitioned, because there are fewer servers that can respond at all, smaller aggregate queue depth etc... It has been said that consistency was specified rather poorly in Brewer's original paper, and we've since seen work that dives into the consistency spectrum. Peter Balis's "HAT not CAP" paper is an often-cited example of such work. Aphyr again comes through with a summary of some of these ideas in https://aphyr.com/posts/313-strong-consistency-models .
We know that we can't have 100% harvest and 100% yield in the presence of partitions. We can play with different things though to shape the curve of these things as conditions worsen, however
Amazon likes to respond to Google with it's own price drops and product launches. It's telling that their announcements are orthogonal instead of direct competition with Spanner.
When Google announced Spanner back in 2012, I'm sure Amazon and Microsoft started teams to reproduce their own versions.
Spanner is not just software. The private network reduces partitions. GPS and atomic clocks for every machine help synchronize time globally. There won't be a Hadoop equivalent for Spanner, unless it includes the hardware spec.
And because of that, Aurora's multi-zone replicas are read-only.
I just noticed Google says the cross-region feature is coming later in 2017. Amazon might be planning to announce a similar change for Aurora in the coming months.
Actually, there are lots of reasons why it's hard to share a service externally. Making a service public means you have to deal with lots of new problems like billing, abuse, dealing with lots of small users instead of a few large ones, stronger backwards compatibility requirements and so-forth.
That's kind of the point, when you make a service like Cloud Spanner public you create new scaling dimensions. New scaling dimensions means new problems to solve that are not trivial.
It's not fair to compare these since Aurora is a traditional relational database. It does not have a horizontal scalability solution aside from read replicas.
Not sure. If you need to scale beyond a single master, Aurora won't help in the same way Spanner does though. You can dial up the number of nodes in Spanner dynamically under load with good results.
Really a CP system but with the Availability being five 9s or better (less than one failure in 10^6)
How:
1)Hardware - Gobs and Gobs of Hardware and SRE experience
"Spanner is not running over the public Internet — in fact, every Spanner packet flows only over Google-controlled routers and links (excluding any edge links to remote clients). Furthermore, each data center typically has at least three independent fibers connecting it to the private global network, thus ensuring path diversity for every pair of data centers. Similarly, there is redundancy of equipment and paths within a datacenter. Thus normally catastrophic events, such as cut fiber lines, do not lead to partitions or to outages."
2) Ninja 2PC
"Spanner uses two-phase commit (2PC) and strict two-phase locking to ensure isolation and strong consistency. 2PC has been called the “anti-availability” protocol [Hel16] because all members must be up for it to work. Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down."
You're making some assumptions. For example if you use epaxos for the quorums there is no unavailability due to a leader failure and re-election. Even then, re-election is likely going to be a lot faster than any sort of fault tolerant 2PC coordinator recovery protocol. You're exaggerating the write amplification. The above offering is single region. Google says they do 5 way geo in their various papers/etc, and there's no way they're paying some cumulative 15 copy write amplification as you imply.
epaxos is not general purpose replication. It leaves the problem of reconciling multiple replication streams to another layer. I've heard that they do 20+WA (distributed, before you consider on-disk WA) for much of it. shrug
9x write amplification isn't that high. If you are setup with a master slave configuration, that creates write amplification and for reliability, you probably have some sort of raid, that increase the number of places writes go to as well.
Congratulations to the Spanner team for becoming part of the Google public cloud!
And for those wondering, this is why Oracle wants billions of dollars from Google for "Java Copyright Infringement" because the only growth market for Oracle right now is their hosted database service, and whoops Google has a better one now.
It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers!
That's vague. AWS also powers huge websites and Amazon is recommending Aurora as the "default choice" for most workloads.[1] There are certainly significant architectural differences but I would say we can definitely make a direct practical comparison.
And their needs are reasonably complex. They use machine learning and big data analytics to generate the list of videos that you should be watching. In order for those to work they need to capture a whole raft of end user metrics e.g. at what point you paused video X.
It is not close to equivalent. But I do want to get a better feel for if Google really has figured how to do basically the impossible. I want to see if this truly scales horizontally but of it does then competitors better hope for a much more detailed paper :)
It's equivalent, with different (unknown) constraints. Aurora is specifically for scaling workloads in the same way. You can say it's horizontal (machine) over vertical (resource) but it's all a matter of accounting.
The big nono is the Spanner pricepoint. I will stick with Aurora for scaling based on traffic I use, over pricey timeslices.
You would have to have quite a load to justify the switch from cheaper de jour solutions right now (AWS). Relying on the few that do, is a risk.
Aurora is very cool but won't help you much after you vertically scale your master and still need more write capacity. With Cloud Spanner you get horizontal write scalability out of the box. Critical difference.
Yes, Aurora has a single write master, though it does have automatic write failover -- i.e. if the Aurora primary dies, one of your read replicas is promoted to the primary and reads/writes are directed to the new instance. That does constrain your primary's capabilities to the largest instance size (currently a db.r3.8xlarge).
I don't have a good idea what the upper limit is for an Aurora database setup.
AWS uses heartbeats for detecting liveliness. If x heartbeats fail the failover procedure is started. Generally 10s - 5minutes. In practice (for me) the failover has been less than 15s.
Aurora's read replicas share the underlying storage that the primary uses, so AWS claims that there's no data loss on failover. They also claim -- and I've never heard anyone say they were wrong -- that Aurora failovers take less than a minute. So the pain should be limited to under a minute of lost writes, which most applications can handle (with an error). It can still be painful depending on the application.
Per their pricing page[1] it looks like the largest instance available is a "db.r3.8xlarge", which is a special naming of the "r3.8xlarge" instance type[2] which is 32 cpus and 244gb of memory.
That's a hell of a lot of capacity to exhaust, especially if you're using read replicas to reduce it to only/mostly write workloads. Obviously it's possible to use more than this, but the "sheer scale" argument is a bit of a flat one.
You can disagree on that if you'd like, but note that I explicitly acknowledged the possibility of exceeding these limits. In my opinion, for most cases/workloads, it's highly unlikely that you will and designing for that from the outset is a waste of time and resources.
While Aurora doesn't provide true horizontal scalability, the same-node scalability seems so strong it might allow many companies to stay single-node for quite a while.
> the only growth market for Oracle right now is their hosted database service
This is not true. Oracle is far more than a database company nowadays in the same way that Microsoft is more than Windows. Oracle has been acquiring high-growth startups at a significant rate.
Which has their 'cloud services' doubling their contribution to revenue year over year and licenses losing 50% of their contribution to revenue year over year.
It'll be interesting to see how well customers will adopt this. When I was at one of the two companies you mentioned above, we tried adding global snapshots (a la TrueTime, which is the real innovation in Spanner not the clocks) and demoed it to our DBAs/MVPs. They didn't understand what on-earth was going on. Wanted something that worked with existing clients. We just gave them classic 2PC and they went home happy. I think that's the reason why Oracle will keep on chugging. There just aren't that many workloads that need this sort of scale. It is real cool technology though and we always used to wonder why Google wasn't offer Spanner as a service.
As a bit of a veteran in the database industry, I concur (at least about the impact on Oracle's database business). There is a lot of pent-up demand for anything that offers distributed consistency.
We've been seeing this demand at Fauna. FaunaDB offers distributed consistency, based on Raft and the Calvin protocol instead of depending on specific networking and clock hardware. We've seen a big part of our appeal is the ability to run FaunaDB across multiple cloud services.
The serverless cloud is pay-as-you-go. There is no minimum spend, unlike Spanner's $1000 per month (apparently). And it's cheaper than operating any open source on cloud hardware.
On-premises is licensed by core.
We have a developer edition you can use on your local machine, but we don't currently have plans to open source FaunaDB itself.
While I'm not familiar with Spanner's inner workings, I would guess that they recommend 3 instances for quorum establishment in case a region becomes unreachable. If that's the case, using fewer than 3 instances could cause major problems.
Since Spanner currently only supports single-region deployments, it clearly isn't recommended as protection against a region becoming unavailable.
It may be recommended as protection against an availability issue on an instance, though, which is, after all, a big reason why you'd want a distributed DB in production.
I suppose the loss of a region doesn't apply (yet), but yes, the quorum requirement would still apply even if you only had instances in a single region.
> It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers!
There are also plenty of choices evolving for developers who aren't looking for hosted solutions (which can sometimes be a showstopper for enterprise on-prem deployments). There's a growing ecosystem of distributed open-source databases to look out for too.
Take Citus, for instance – a Postgres-compatible distributed store which automatically parallelizes normal SQL queries across machines. It's as easy to set up as adding an extension, and people are doing some staggering things in prod with it.
Different audience from BigQuery and Spanner, but no less exciting.
Disclaimer: no professional association, but love their product and the team.
Craig from Citus here. Thanks for the kind words. We've seen a lot of people scale-out transactional workloads with Citus as well. In particular, we've seen a lot of multi-tenant apps that need to keep scaling beyond a single node when they're running into memory or compute issues.
If you are looking for something that is more Postgres flavored (meaning we're just an extension to it so you get all the good stuff of Postgres such as JSONB, PostGIS, etc.) then we hope we'd be a good fit. And we run a managed service on top of AWS as well (https://www.citusdata.com/product/cloud) built by the team that built Heroku Postgres. If curious on pricing you can find it at https://www.citusdata.com/pricing/
It would be very interesting to have your product at the 200-300$ pricepoint. Currently, the lowest tier starts at almost 2000$ per month for the high availability version.
I'm not trying to compare on a per-mb level, but it would be nice for smaller scale workloads.
Helpful feedback, we do have a development plan for $99, but it's not really intended for production workloads. If you only have 10 GB of data we'd heavily recommend going with something like RDS or Heroku Postgres. At that amount of data single node Postgres works great.
Right? Apps written to use Postgres aren't just going to be re-written to use Spanner.
If anything, hosted Postgres from Google Cloud will be priced in a way that makes Spanner some what more attractive, as a way to get conversions to Spanner in the long-run.
There are several managed hosting companies that will run Postgres (and other databases) for you on public clouds. Compose, Aiven, ElephantSQL, Database Labs, Heroku, etc. There are all kinds of price points and GCP is working on supporting postgres internally.
How many nodes are you looking to run for $300/month? Unless you have more than 150gb/node of data, you don't really need a distributed database which is what Citus is for.
not at the price point that RDS is at. the starting cost of a multi - az deployment is lower. For a startup thats just starting out, it is the best and safest choice. Even heroku, but im not very sure about its reliability versus rds.
Please note what im paying for is availability and reliability.. not for a database per se.
And im not even talking Aurora. That stuff is going to blow every other price point out of the water at probably higher reliability metrics.
> Its pretty much a monopoly, now that Google seems to have officially closed the book on ever supporting postgresql.
Uh, how? I wouldn't be surprised to see a Cloud SQL-like managed Postgres service from Google.
While there's obviously some overlap in the potential market for any relational datastore service, Spanner doesn't really overlap with a cloud Postgres service as much as Cloud SQL does.
I would be. its been years and google and has been building out various pieces of infrastructure around mysql - including cloud spanner.
the issue is that the migration path of self hosted mysql to cloud sql to spanner is pretty well defined. I dont see postgresql being strategically important or relevant to google for anything.
if I was a startup deciding on my database, there's a lot less compelling reasons to choose postgresql from the point of view of long term viability.
hell, I can pretty much do a back of the envelope calculation on how much will it cost me to support 100 million users on mysql.
Is it safe to think that Evernote and Snapchat - startups who are giant success stories - are google mysql hosted? (in some form.. maybe even spanner)
So uber, snapchat, Google, Evernote and a clear cut path for upward scale.
I have very less hope for postgresql on google cloud.
> its been years and google and has been building out various pieces of infrastructure around mysql - including cloud spanner.
What does Cloud Spanner have to do with MySQL? It's neither API nor SQL-dialect compatible with MySQL. If there are MySQL bits used somewhere in the implementation, they are well hidden, and irrelevant to users.
> the issue is that the migration path of self hosted mysql to cloud sql to spanner is pretty well defined.
So what? Were there a Cloud SQL-like Postgres offering, the same would be true; Spanner is no closer to MySQL than Postgres. (If anything, it's SQL dialect is a little closer to Postgres's dialect than MySQL's, though not so much that you'll get away without doing substantial conversion going from either.)
I really like your attitude towards something like this, when your product would be an overkill for a use case and you just recommend a different product. I also really like your blog posts about Postgres, we use it a lot for developers explaining a bunch of internals, like the one with how to paginate in Postgres.
As someone who works (in part) in the MS SQL field, is it irrational to be a bit worried about the effects some of these platform advances might have one one's career?
For example, being a MSSQL performance tuning expert requires years of experience and probably pays very well, but just the other day I read an anecdotal story where someone switched a large BI database to use columnar indexes, allowing them to replace very complex (extreme manual tuning to achieve acceptable performance) queries with just standard SQL with comparable performance.
How long until the scale, pricing, and now transparent & full(?) sql compliance offered by these cloud platforms starts to make traditional RDBMS platforms a niche platform?
Microsoft has a history of sales and support that will allow them a certain longevity. They also have less "brand hate" than Oracle. I dont think MSSQL is going to be like Sybase any time soon, but I probably wouldn't focus on that stack starting now if you are into the startup or california scene. For many places in the USA, MS is the way to go.
EDIT: Also, most DB users don't need global-scale databases.
Not sure if this is what you intended, but you are aware that SQL server was developed in partnership with Sybase until the mid-90s (when they were substantially the same product) right?
Neat history! I was not aware of that! I meant to imply that sybase's rdbms offering is no longer a big player and I would not want to bet the future of my career on being an expert in it.
I'm not saying MS is going to abandon the platform, but to me it seems entirely possible that "very soon" these cloud platforms using cheap, shared, commodity hardware might be so affordable and technically capable that it might be a no brainer choice unless you have a very good reason to use MSSQL (kind of the opposite today, on-prem by default, cloud if necessary).
I write software as a developer. This is how I earn my livelihood.
Four years ago, I determined that while development work might seem to be near the top of the food chain, there will at some point where my work will be replaced by AIs.
This is not so different from how word processors replaced the specialist job of typesetters. Word processors make "good enough" typesetting. You can still find typesetters practicing their craft; the rest of us use word processors and don't even think about it.
At the time, I was learning to put the Buddhist ideals of emptiness and impermanence to practice, and to become more emotionally aware: the _main_ reason I had thought I would never be replaced by an AI writing software has more to do with wishful thinking and attachment than any clear-sighted look at this.
I also made a decision to work on the technologies to accelerate this. Rather than becoming intoxicated by the worry, anxiety, and existential anguish, I decided try to face it. Fears are inherently irrational, but just because they are irrational does not mean it is not what you are experiencing. Fears are not so easily banished by labeling them as irrational. Denial is a form of willful ignorance.
Now, having said all that, whether our tech base will come to that, who can say?
Since then, I have been tracking things like:
Viv - a chat assistant that can write it's own queries
DeepMind's demonstration of creating a Turing-complete machine with deep learning using a memory module.
I watched a tech enthusiast write a chat bot. He does not write software professionally. Talking with him over the months when he tinkers with in his spare time, I realized that in the future, you won't have as many software engineers writing code; you would learn how to _train_ AIs when they become sufficiently accessible to the masses. Skills in coaching, negotiation, and management becomes more important then some of the fundamental skills supporting software engineering. And like typesetting, I can see development work being pushed down the eco-ladder.
It's not surprising to me to see that Wired article about how coding becoming blue collar work. And even that will eventually be pushed down even further.
It's not surprising to me about Google's site-reliability engineering book, branding, and approach. I have done system admin work in the past, and I can already see traditional, manual sysadmin work being replaced.
It's easy to get nihilistic about this, but that isn't my point here either. I know the human potential is incredible, but I think we have to let go of our self-serving narratives first.
I find this fascinating. There are a few ideas that are at play. One is the march of progress seeking to automate everything. The rationale of automation is to improve productivity. But what happens when everything is automatic? I don't see a corollary being played out at the moment. There are a small number of people reaping the benefits, and huge swathes of the population being marginalised and disenfranchised as a result.
The second idea that interests me is this idea of very high technology. It is built upon layer after layer of very clever tech year after year that I wonder how long it would take to start again from scratch if some disaster rendered a large part of one of these layers unusable.
For instance, if you were on a desert island, could you (would you want to?) build some piece of tech? An electric generator would be useful, perhaps. How long would it take to build? You'd need knowledge, raw materials, plant, fuel etc. It's not an easy solve. And that's way down the tech stack before you start talking about AIs. I suppose what I'm saying is, that the AI layer is based upon such high tech, that is inherently fragile, because it is so hard to do.
> There are a few ideas that are at play. One is the march of progress seeking to automate everything. The rationale of automation is to improve productivity. But what happens when everything is automatic? I don't see a corollary being played out at the moment.
I don't know! :-D
I don't know what society would look like from a purely technological point of view. From a spiritualist point of view, though, it could either go very well or very badly. When everything is automated, would people have enough time and space to really start asking the really big questions? Or would it accelerate and intensify existential anguish?
> There are a small number of people reaping the benefits, and huge swathes of the population being marginalised and disenfranchised as a result.
Yeah. Arguably, this has already happened.
> The second idea that interests me is this idea of very high technology. It is built upon layer after layer of very clever tech year after year that I wonder how long it would take to start again from scratch if some disaster rendered a large part of one of these layers unusable.
The stuff of sci-fi :-D Among them, alt-history novels (what happens when someone drops into a lower-tech era; you'd have to start from 0 ... literally, 0, as in Arabic numerals).
Open Source Ecology is trying to preserve some of this tech base. I find their aims awesome, though I am not sure how effective it is.
The flip side are things being spoken from well outside the techno-sphere, (for example, shamans and mystics) It is the perspective that the further evolution of human consciousness will, at some point, no longer require a technology or artifacts. Technology seen as the last crutch. The collapse of a high-technic civilization then sets the stage for a removal of that crutch, and humans learn to stand with two feet (so to speak).
Anything that you can easily specify and describe in detail can be automated. In practise the world is filled with computers that need programming to cope with ever changing chaotic actions of our users. Personally I'm long on software developers, despite being very excited by how AI has blown up lately.
Agree. I remember someone advising me against getting a degree in computer science, back in 2000. Argument was -- look at MS Word. what else you want to add to this? It has more features than you need.
Not a fair argument against the point made above, however, I believe we will find the next big challenge for software to solve as soon as traditional problems are commoditized/automated and considered solved. Also, just knowing how to code is not going to be enough. You must complement it with domain expertise to solve challenging unsolved real world problems.
People are abusing databases like MSSQL to do things they may not be not good at. Large scale analytics is an example where databases like Infobright give amazing performance.
> As someone who works (in part) in the MS SQL field, is it irrational to be a bit worried about the effects some of these platform advances might have one one's career?
There's a reason why regulated (including self-regulated) professions have continuing education requirements; progress happens and you become obsolete if you don't keep up with it.
Just because tech isn't regulated doesn't mean it's any more sensible to expect to remain valuable without keeping up with progress in the field.
That being said, MSSQL experts will likely have good-paying opportunities for quite a while, for the same reason thats then case for any well-established enterprise technology: lots of systems are going to be around using it long after it has become distressingly uncool to spend time learning.
I would say: quite long enough, but the scale/importance of traditional deployments will be going down through that time. If you're looking around the market regularly, you should have ample time to notice that gigs are not what they used to be, so maybe it's time to change area.
I would say: quite long enough, but the scale/importance of traditional deployments will be going down through that time. If you're looking around the market regularly, you should have ample time to notice that gigs are not what they used to be, so maybe it's time to change area.
The absolute level of computation available isn't changing at the consumer level. What's happening for the next decade is the destruction of businesses hosting their own IT infrastructure and moving it to a couple of core centers.
So, the computational "Gini index" is increasing, but no one is being thrown into computational poverty.
The parent comment doesn't seem to specify "consumer level" and the loss of businesses having their own infrastructure is equally troubling. Everyone is putting a lot of eggs in a very small number of baskets.
I would disagree about the character of the situation. This isn't about people putting eggs in a few baskets, it's that it's more efficient to have centralized chicken coops instead of every family in the world owning their own chickens.
Now, you could play with that analogy further and see some issues as well, but I don't think the issue here is centralized failure; all these data centers/"clouds" are at least good. The Cloud is about businesses focusing on core business and not supporting functions.
[Disclosure, I work on the Google Cloud team, I'm biased]
> it's that it's more efficient to have centralized chicken coops instead of every family in the world owning their own chickens.
Only if the owner of the chicken-coop has everyone else's best interests in mind. Protip: They don't.
The Cloud isn't about efficiency, it's about data control. Getting people's systems and data into Google/AWS/etc helps with data mining, vendor lock-in, etc. Often times that can be efficient, but also it often isn't.
Let's look at your metaphor. It's more efficient for the raising of a large overall number of chickens. It's less efficient when I need fast access to a single egg.
Hence we get caching. There's the farms, then the inbound warehouses, then the distribution centers, then the grocer, then our refrigerators by the dozen or dozen and a half. When your local cache is empty of eggs, though, it requires a trip back out to the grocer to get an egg even if you need nothing else that trip. Then you generally have to buy at least half a dozen if not a dozen or more eggs just to get the one you wanted.
If I have my own couple of hens, I can go out into the yard and get an egg. If that's the whole of my fetch list, it's much more efficient for this single egg to have the hens laying right out back.
This whole few baskets metaphor breaks down from another point of view, though, when we consider that by the very nature of using a globally distributed hosted service we're actually eliminating a single basket problem. Yes, there's not much choice among just Google, Amazon, and Microsoft. (That they are the only options is a bit of a strawman, but lets grant this one legs.) However, putting just your own employees in charge of all your infrastructure in just your own datacenter(s) in just PostgreSQL or just MySQL is another single-basket problem. Spreading it out so that someone else gets to manage the hardware and the service and replicating your data widely within that service is from that point of view more baskets. You get more datacenter baskets, more employee baskets, and more software baskets. Using standard SQL means you can move among compliant software later, too, so you're not as tied to those baskets.
Now, back to your coop analogy. What's stopping me from having my application talk to Cloud Spanner and a local database proxy (or a work queue that sits between the app and the DB or whichever) so I can use Google's reliability for transactions and my local cached replicant for query speed when I'm querying older data? Why can't I keep a few eggs around?
Also, why would I be scared of Google or Amazon "having my data"? Why would I put sensitive data into my own database in plaintext and then replicate it among multiple datacenters that way?
>focusing on core business and not supporting functions.
Having a devops team with the necessary expertise in Google Cloud or AWS is still a supporting function. You've just traded one skill (managing physical servers) for another (managing proprietary virtual resources).
But hopefully a smaller team, and one that keeps diminishing in size over the years if the trend continues. At least for the same level of service (in availability, security, etc.).
> The Cloud is about businesses focusing on core business and not supporting functions
Yep, it inevitably trends towards commodity infrastructure. As such it will require regulation (and eventually even some form of nationalisation in some cases)
That's like being sad about the emergence of banks, because everybody's money is being kept in a small number of vaults instead of under each one's mattress.
A good point, but there is an up and downside to everything. The centralization of IT does impact civil liberties and possibly innovation - unlike FOSS and other local systems, aspiring hackers can't tinker with Facebook code and see how it works.
> Aspiring hackers couldn't tinker with MS Word 2000 code either.
They could tinker with the binaries, something many did with game binaries. But your point is well taken; open source is also very valuable to innovation.
Web apps were also very useful for learning JS and browser APIs, before everybody started minimizing and obfuscating it. I learned how to write a rich-text editor just by looking at the code of Hotmail's email editor.
Fair enough, but think of that free and open stack: (layer 1), Ethernet, IP, TCP/UDP, HTTP/SMTP/DNS/etc, HTML/JavaScript. How many cut their teeth on those?
The apps on top, Facebook, Snapchat, etc., are not so open and much of what they do is out of reach from the user.
Also, I meant to add above: People could tinker with data files (e.g., Word docs), configurations, etc. The whole system was local and accessible. You could write local code, such as VB or for Windows, that integrated with those systems.
That strategy resulted in the Great Depression and later 2008 crises. Damage was so high that country had to be rescued by the federal government. So, banking is a decent example of how such consolidation into private hands can go wrong. Now we just apply that to IT services and data.
That's a ridiculous argument: Banks started being a thing at the end of the Middle Ages. The Great Depression and the Great Recession were not caused by banks emerging, nor by people putting their savings in them.
Not emerging. Just being themselves with all their schemes and an economy dependent on them. A distrust of banks and their schemes at a national level might have reduced their ability to cause those problems. On top of the smaller stuff such as them delaying deposits or withdrawing stuff for bogus reasons.
Putting your savings under the mattress instead of in a bank account wouldn't have prevented the Great Recession. It was caused risky mortgages (debts, not savings) being sold as low risk from bank to bank, and then defaulting.
Putting your savings under the mattress instead of in a bank account wouldn't have prevented the Great Depression either.
The only thing it would have accomplished is making your savings easier to steal.
Storing gold or other valuables instead of Gederal Reserve notes for sale or bartering wouldn't have helped during Great Depression? I havent heard the angle that there was nothing to barter with on top if worthless dollars.
> The absolute level of computation available isn't changing at the consumer level.
Maybe not, but consumers increasingly use centralized computation resources. I would guess that by now most applications used by consumers run in their web browser, such as Facebook.
>What's happening for the next decade is the destruction of businesses hosting their own IT infrastructure and moving it to a couple of core centers.
Yes, and this will be disadvantageous over the long run for people that want to run things themselves. Ultimately companies like AMD/Intel go where the big money is at. As things centralize further and further, there will only be 3 customers they care about in the server market.
> The centralization of computation is likely not a good thing in the long run.
I agree. It only makes sense if you need special data for statistics, AI training, etc.
In all other cases the classic way of programming on pc and notebook is smarter. If you do everything in the cloud, what if you lose Internet connection? I had that experience several times over the last years.
* Computers are much more stable than they used to be
* Much of the world lives in places with less stable connections
* The most expensive spec in an Internet connection is availability. You can get a low-end 15 Mbps connection with no availability guarantee for $40/month; a T1 is one-tenth the speed and costs 10 times as much (all numbers are rough estimates).
The movement from ownership to renting on the web is absolutely terrifying to me. Within the span of a few years we've gone from owning our technology to renting it out from a big players for monthly fees that we cannot completely predict or control.
The advantages of owning your own hardware will never go away, but soon this will be made quite intentionally impossible as the big players coalesce and continue building their walled gardens.
This is already happening. All the big players own their hardware and rent it out to everyone else, while trying to convince everyone it's not worth owning your own hardware at the same time.
These companies have already begun closing off server platforms by developing custom hardware and software systems that cannot be bought for any price, only rented. These systems represent a new breed of technology with unbreakable vendor lock in.
Theses same companies compete with each other and countless other companies across the space. Take for example a start-up that wants to run their own app store. Google, Amazon, and Microsoft all run app stores. Where will this company go for cloud services? Their only big name options are to host their software on the hardware of a direct competitor. Their host has full visibility on how their system works, and control over the pricing and reliability of their machines.
It's laughable to think their "cloud partner" will give them any chance to compete if they enter the same market.
We've seen UEFI BIOS and un-unlockable mobiles enter the market in droves the last few years. A lot of new PC's can't run anything except windows. A lot of new phones can only run the carrier's version of android. We have all these general purpose CPUs that can no longer run general purpose programs because "security", and a lot of lobbyist pushing to make it actually illegal to run your own software on these with "anti tampering" laws, again for "security" . Soon the big guys (same companies, MS and Google) will make it impossible to run your own software on any reasonably inexpensive devices and the walled market will be complete.
Mark my words, I've never seen an industry with a couple big players where growth and innovation doesn't eventually turn into collusion, higher prices, and market stagnation. Once MS, Google and Amazon have their slice of the pie and they've killed off everyone else, we will see the death of general purpose computers and mobile devices. Everything you buy will be "android computer" "windows computer" and "apple computer". Anything general purpose will be massively more expensive because individual companies can't get the kind of volume discount of the giant behemoths that increasingly control large swaths of the world's computing power. We've already seen the endgame, with Amazon trialing an "on premesis" version of their compute platform which is basically a super locked down server that you can't buy, only rent endlessly. The future of on premesis will be a cloud in a black box if these companies have anything to do with it. Why? Because once they've got you locked in it makes no sense to sell you anything for keeps. Why keep improving their product so you buy the new version when they can just make it incompatible with everything else and force you to rent it forever, for whatever price they feel like charging?
One day running your own servers will be like running your own ISP . Massively impractical because the free market has been manipulated to the point that it effectively no longer exists
> One day running your own servers will be like running your own ISP . Massively impractical because the free market has been manipulated to the point that it effectively no longer exists
What? People use cloud computing because it already is massively impractical to run your own servers. Hardware is hard to run and scale on your own and experiences economies of scale. This principle is seen everywhere and can hardly be viewed as something controversial. Walmart for instance can sell things at a really low price because of the sheer volume of their sales. Similarly, data centers also experience economies of scale.
As someone who cares about offering the best possible, reliable user experience, cloud computing is absolutely the next logical step from bare metal on-prem servers. When your system experiences load outside the constraints of what it can handle, a properly designed app that has independently scaling microservices horizontally scales.
Even if you had the state of the art microservice architecture running on a kubernetes cluster on your own hardware, you still wouldn't be able to source disk/CPU fast enough if your service happens to experience loads beyond what you provisioned.
And there is the rub, buying your own hardware costs money, and no one wants to buy hardware they may not ever use. Another advantage of cloud computing.
You are seeing the peak of free market right now, because of cloud computing, which enables people with little upfront cash to invest to form real internet businesses and scale massively.
You think a game like Pokemon Go can exists and do the release they did without cloud computing?
Have you run your own servers in a colo? I've done it myself.
One person, with maybe 3 hours a week of time investment after a few weeks of setup and hardware purchase. Using containers I can move between the cloud and my own servers seamlessly, and long as I never bite the golden apple and use any of the cloud's walled garden "services" like S3. If I need more power I can spin up some temporary servers at any cloud provider in a few hours. For me the cloud is a nice thing because I don't use too much of it. If AWS disappeared tomorrow it would be a mild inconvenience, not devestating like it would be to many newer unicorns.
Go ahead and try to use the cloud you're paying for as a CDN or DDoS sheild, or anything amounting to a bastion of free speech. You'll quickly find out that your cloud provider doesn't like you to use all the bandwidth and CPU you pay for, and they don't like running your servers when they disagree with your views. They quietly overprovision everything pulling the same crap as consumer ISPs where they sell you a 100mbps line and punish you if you use more than 10 of that on average. That's the main reason the cloud is so cheap.
Hardware is cheap, colo's are cheap, software is largely easy to manage. The economy of scale they enjoy is from vendor lock-in and overprovisioning more than anything else.
Is it really that hard to double the amount of servers you own every few weeks? No! If you're using containers or managed KVM you can mirror nodes basically for free over the network as soon as the Ethernet is plugged in. Your time amounts to what it takes to put the thing in a rack, plug in the Ethernet, and hit the "on" button. Everybody in SV land thinks you have to use cloud to "scale massively" but they forget that all of today's technology behemoths were built years ago when the cloud didn't exist. Oh yeah, they all still run all of their own hardware too and have from the early days. Using their model as a template, you should own every single server you use and start selling your excess capacity once you get big enough.
Did you ever read about how Netflix tried to run their own hardware but can't because they have so much data in AWS that it would basically bankrupt them to extract it? Look at how these cost models work. Usually inbound bandwidth is extremely cheap or free but outbound is massively more expensive than a dedicated line at a datacenter, 50-100 times the cost if you're saturating that line 24/7. The removal fees from a managed store like S3 or glacier are even more ludicrous. The cloud is like crack and as soon as you start using it more than a few times a year you will get locked in and unable to leave without spending massive $$$. Usually companies figure out this shell game once they're large enough, but by then it's far too late to do anything about it.
Why are they marketing these things so heavily to startups? Because lock in is how they make their money. They make little or nothing on pure compute power, but since you don't have low level hardware access they can charge whatever the hell they want for things like extra IP's, DDoS protection, DC to DC peering, load balancing, auto scaling. You give massive discounts to new players using these systems and inevitably some of these will become the next Uber or Netflix. Then you are free to charge whatever exhoribitant rates you please once it's so impractical to move that it would require a major redesign of the business.
I see it a lot like franchising. By building on Amazon's cloud services you become "Uber company brought to you by Amazon". Like franchising, your upside is limited because any owner with a significant share of total franchises will begin to put pressure on the service owner itself.
Did you ever read about how Netflix tried to run their own hardware but can't because they have so much data in AWS that it would basically bankrupt them to extract it?
Where did you read this? You can have Amazon send you a truck full of hard drives. I doubt it costs more than Netflix can afford.
Nevermind, I misremembered the story I read about them. They moved the main site to AWS with the huge omission of their movie streaming system. Their own Open Connect servers are far cheaper to use for this becuase of massive AWS outbound data costs.
Also, the truck is for data in, not data out. Getting data out of AWS is far more expensive than putting it in. That's the lock in.
To be honest, you sound like conspiracy nut hell bent on hating the Cloud. Maybe you should try taking a deep breath, and try to open up to the possibility that the Cloud is actually a good thing, and Cloud providers aren't the illuminati trying to "lock you in". Well maybe they are. Of course every cloud provider wants you to use their services.
You can architect your system in a way that it'll run on any cloud provider. All the major Cloud Providers support kube for orchestration.
To be honest I don't think you know what you're talking about. You should refrain from making uninformed opinions on hacker news, especially on a throwaway.
"Even if you had the state of the art microservice architecture running on a kubernetes cluster on your own hardware, you still wouldn't be able to source disk/CPU fast enough if your service happens to experience loads beyond what you provisioned." That basically means you never planned. As everyone moves to cloud what makes you think AWS, Azure wont have same issue. If entire region is down do you think other regions can handle the load. If you think so you're kidding yourself. Unless you have business where you dont know your peak number then cloud does not matter.
You can plan all you'd like, failures happen not necessarily due to poor planning but because in real life, shit happens. Pokemon Go for instance experienced like 50x the amount of traffic they planned for.
Secondly, software companies like Microsoft, Google and IBM might know a thing or two about running data centers. Due to economies of scale, these companies are inherently in a better position to supply hardware at scale.
> If entire region is down do you think other regions can handle the load. If you think so you're kidding yourself
Netflix routinely does just this to test the resilience of their systems. They pick a random AWS region, and they evacuate it. All the traffic is proxied to the other regions and eventually via DNS the traffic is routed entirely to the surviving regions. No interruption of service is experienced by the users.
Here's a visualization of Netflix simulating a failure on the US-east-1 region and failing over to US-west-1/US-west-2
The top right node is the one that fails. As the error rate climbs, traffic starts getting proxied over to the surviving nodes, until a DNS switch redirects all traffic to the surviving nodes. Netflix does this monthly, in production. They also run https://github.com/Netflix/SimianArmy on production.
The cloud enables fault tolerance, resiliency and graceful degradation.
I think you missed the point, Netflix evacuating a region is not the same thing as that region failing. If the whole region goes down, their (AWS's) total capacity just took a major hit and unless they have obscenely over-provisioned (they haven't), shit is going to hit the fan when people start spinning up stuff in the remaining regions to make up for the loss.
You did not ever own your own globally consistent, massively scalable, replicated database. The fact that you can now rent one by the hour is strictly an improvement for you, if you need that kind of thing.
Cassandra also does that without requiring the "magic" of a system you can only get from a single vendor and never buy. At the same time these walled gardens have come up free software has grown to fill the gaps
Spanner is unique in a lot of ways, but it still trades off consistency for speed.
The most unique thing about spanner is the use of globally synchronized clock timestamps to guarantee "comes before" consistency without the need to actually synchronize everything.
There is nothing stopping startups and open source developers from building the same thing in a few years. The missing ingredient is highly stable GPS and local time sources which will hopefully be available on cloud instances sometime soon. This is a new piece of hardware so it will be interesting to see if cloud providers make one available or use the opportunity to sell their own branded "service" version you can't buy. Unfortunately I think we'll see the latter far before the former, it it ever even exists. Without a highly stable timesource doing what spanner does will be completely impossible.
Yes spanner is special right now but that's even more reason to not go near it. Google has a complete monopoly on it, the strongest vendor lock in you can possibly have
Only "new" in the sense that it is currently not commonly offered, the devices themselves have been available for ages. (If you are a large enough customer you apparently can get at least some colo-facilities to provide you with the roof-access and cabling needed for the antennas). If cloud providers make precise time available I don't see much potential for locking you in with their specific way of providing it, as long as it ends up as precise system time in some way.
I'm saying I doubt they will ever offer it precisely because it will conflict with their paid offerings. The fact that it takes its hardware is a great excuse to not give your customers the option.
I know GPS time sources have been available forever but a fault tolent database needs a backup. The US GPS is incredibly reliable but there have been multiple issues with both Glonass and Galilio.
It sounds like Google has an additional time source making this possible, probably a highly miniaturized atomic clock, possibly on a single chip. There's no way they're running on GPS alone
Yes, they clearly say that they use atomic clocks in addition, but that's commercially available as well. Atomic clock for frequency stability short- to mid-term, GPS to keep it synced to global time. E.g. in many cases, mobile-phone base stations contain just such a setup, and the data-center versions should fit in a few HE.
A system build on top of it? Possibly, but thats the trade-off if you don't want to pay for/be lock-in to somebody else running it. For just the timing stuff: not really. Of course it adds complexity, but these things are established and should be quite stable.
We've already been through it. People eventually abandoned mainframes for everything they could. Many of the current customers are interested in better solutions but just stuck due to lock-in of piles of COBOL, etc.
I didn't downvote you. It is important to note though that the Spanner project isn't related to MySQL and there is some discussion of that in a the stories around Spanner. It would nominally compete directly with Oracle's flagship database product.
it's actually hard to beat MySQL for a lot of things. i was skeptical about this when I joined google, but as an SRE on the MySQL team around this time, I gained a lot of respect for it.
That is an interesting way to look at it; I have wrestled with MDB[1] while working at Google, it was a ginormous MySQL database (possibly one of the world's largest). And I would characterize Spanner's relationship this way, "If you think you are actually going to build an ACID database that scales, then make sure you can support the MySQL api that MDB uses and we'll see just how well it scales."
I don't know if anyone put it to them that way but as Spanner was just getting started when I left I know that one of its success criteria was to be able to be a scalable replacement for MDB. Given the white paper and other papers on their results, I'm sure it managed that requirement.
[1] MDB, Machine Data Base, used throughout the org but especially in Platforms and SRE to keep track of machines and their parts.
It seems like Oracle could have a play here, working on adapting cloud infrastructure tools for managing on-premises data centers. That keeps them in play for customers who can't put their data in the cloud and those that haven't because they're already Oracle customers.
This pendulum swings. We're pretty near the apex now. A little work on ergonomics and these tools could be turn-key, and back we go to decentralized hardware.
455 comments
[ 3.5 ms ] story [ 276 ms ] threadUpd: Downvoting this warning will only increase that number.
Edit: not every server has an atomic clock; see replies by Google employees
The timeslave daemons running on each machine keep them synchronized with the master time servers, and maintain tight bounds on their inaccuracy.
(Disclaimer: I work at Google)
I know this is a single system, but I'll still say it. This seems like another step in a scary trend for our internet.
This is a bold claim. What do they know about the CAP theorem that I don't?
Separately, (emphasis mine):
> If you have a MySQL or PostgreSQL system that's bursting at the seams, or are struggling with hand-rolled transactions on top of an eventually-consistent database, Cloud Spanner could be the solution you're looking for. Visit the Cloud Spanner page to learn more and get started building applications on our next-generation database service.
From the rest of the article it seems like the wire protocol for accessing it is MySQL. I wonder if they mean to add a PostgreSQL compatibility layer at some point.
I don't know your expertise on the subject, but they do have a post on this topic.
Some highlights:
"Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA."
"The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system."
https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
Their statement, for what it's worth: https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
Furthermore, there are already more than a few attempts underway to build scalable relational databases ("NewSQL") outside Google.[4]
1: https://research.google.com/pubs/pub36971.html
2: https://research.google.com/archive/spanner.html
3: http://datascienceassn.org/sites/default/files/F1%20A%20Dist...
4: http://db.cs.cmu.edu/papers/2016/pavlo-newsql-sigmodrec2016....
1 - https://www.cockroachlabs.com 2 - https://github.com/pingcap/tidb
Totally agree with your comment. While the "NewSQL" category is pretty broad and hard to define cleanly, one common theme is understanding the trade-offs that matter to your applications & operators as you distribute & scale SQL services.
For instance, with NuoDB we've decided that standard ANSI SQL with complete DML/DDL support is a must-have for migrating existing applications. As you distribute the database, the ACID contract you expect with those tools also has to be maintained, and we do that. As others have noted in this thread, however, there are many variants of "consistency." I believe if you're trying to exploit distributed scale, and low-latency through locality of reference, then you're probably willing to trade-off serializability. Using MVCC, logical timestamps (not globally coordinated clocks) and a few other well-understood techniques we provide a scale-out solution that maintains standard levels of isolation and consistency. In practice our customers are traditional enterprises using these capabilities to migrate legacy, high-value systems into modern architectures.
What Google has done is something very few other organizations can do: exploit their massive scale and hardware capabilities to offer something unique. From that perspective, they've also chosen to make trade-offs around syntax, indexing, transaction models etc. There's good commentary from my peer at Cockroach about this already on this thread, but the point is that Google picked a view of the database that makes sense for some applications and not others. Personally, I'm excited that Google has opened up this service and I'm sure it will lead to a lot of good introspection on what we need from modern data management services.
For those who haven't read it yet, I'd highly recommend the Quizlet blog referenced elsewhere on this thread. It's a really good discussion about some trade-offs that Cloud Spanner has made, and why they would or wouldn't map well a given application.
Bottom-line: both Cloud Spanner and NuoDB are examples of the modern, distributed, scale-out systems we're moving towards that focus on consistency, fault-tolerance and modernizing traditional SQL applications.
It's right there in the article:
"Remarkably, Cloud Spanner achieves this combination of features without violating the CAP Theorem. To understand how, read this post by the author of the CAP Theorem and Google Vice President of Infrastructure, Eric Brewer."
The post they are referring to: https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
Seems like they might know a lot :)
Being Google they are probably prideful enough to think their servers could never have an outage. Which yes, I agree with you, that is a very scary claim.
In terms of CAP, Spanner claims to be both consistent and highly available despite operating over a wide area, which many find surprising or even unlikely. The claim thus merits some discussion. Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA. The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system. However, no system provides 100% availability, so the pragmatic question is whether or not Spanner delivers availability that is so high that most users don't worry about its outages. For example, given there are many sources of outages for an application, if Spanner is an insignificant contributor to its downtime, then users are correct to not worry about it.
https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp... (written by Eric Brewer, who proposed CAP)
https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
> The purist answer is “no” because partitions can happen and in fact have happened at Google, and during some partitions, Spanner chooses C and forfeits A. It is technically a CP system.
> However, no system provides 100% availability, so the pragmatic question is whether or not Spanner delivers availability that is so high that most users don't worry about its outages.
It looks like the wire protocol is Protocol Buffers and client libraries will likely use GRPC: https://cloud.google.com/spanner/docs/reference/rpc/google.s...
I doubt it, the spanner is not even offering full MySQL capabilities, so it's unlikely to to support any advanced PG SQL.
Your's is a bold claim also :) What do you know about the CAP theorem that Eric Brewer doesn't?
https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...
How is this possible across data centres? Does it send data everywhere at once?
Seems too good to be true of course but if it works and scales it might be worthwhile just not having to worry about your database scaling? Still I don't believe it ;-)
EDIT: further info...
> Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down. Data is divided into groups that form the basic unit of placement and replication.
So it's SQL with Paxos that presumably never get's confused but during a partition will presumably not be consistent.
"CA except when there are partitions" is CP. It's not "effectively CA".
It's one thing to do that for a key-value store. Entirely another to support joins on a globally distributed database. This ain't just one availability zone. Spanner is amazing.
It took them a few years to make it a service, but when they announced its use internally a few years ago, it seemed like the nail in the coffin for in-house database hosting.
There's nothing wrong with saying it's CP, but since we control everything there's extremely rare P. Then he can show availability numbers (which he kinda does).
Saying it's "effectively CA" defeats the point of the CAP theorem, which says you have to make tradeoffs. See: https://codahale.com/you-cant-sacrifice-partition-tolerance/
No, it's engineering. It's the recognition that if periods of unavailability are too small and too rare to be noticed, then the system behavior is indistinguishable from an "available" system in the sense of the CAP theorem.
It's like the "Retina" display you're probably reading from. There are pixels, you just can't see them.
If data from other places has synchronized to your zone, you may be able to do this globally-consistent read while only touching your local datacenter (because TrueTime guarantees that no other records anywhere in the system will be created at the time you are querying).
Note: I work at Google, but I don't know more about Spanner than the Spanner paper.
for anyone interested
Companies with more data than can fit in a single-instance RDBMS system (like >3TB of hot data, more throughput than a single node can handle) but still seeking transactional consistency are a clear use case. Single-person startups could definitely benefit, but it's a less-likely scenario that they would require the level of coverage Spanner provides.
My intuition, which I hope is wrong, suggests this is a small market.
The ideal market for Spanner seems to be new projects developed by big companies that know that big traffic will arrive on day 1. In other words, Google. Which I guess shouldn't be too surprising...
I'd certainly love to see us get to a world where we can split up a single spanner "install" in an isolated, multitenant manner, but even for a small company, $8k/year is admittedly a small fraction of one engineer. At a company with several, you can share your single Spanner instance just like you would any other database.
Disclosure: I work on Google Cloud (but not Spanner).
This is certainly true, but overlooks the fact that many major products start out as experimental projects, and $8k/year is a significant investment for an experiment.
If there's a reasonable upgrade path from traditional databases like MySQL and Postgres this shouldn't be a big deal, but if the answer is "rewrite your app" it will probably be a friction point for adoption.
Having your database problem solved however, one less thing to worry about, if you're bigger.
Have they documented the wire protocol? I couldn't find it.
RPC: https://cloud.google.com/spanner/docs/reference/rpc/ Rest: https://cloud.google.com/spanner/docs/reference/rest/
https://github.com/GoogleCloudPlatform/google-cloud-node#clo...
In any case this is much better than Amazon's offerings... when they actually ship it. :)
https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Sp...
Of note:
They say Spanner is "both consistent and highly available despite operating over a wide area". So not 100% availability but they've got it to "more than five 9s of availability (less than one failure in 1066)."
EDIT: Found quite a bit of my answers in your linked article:
> Cloud Spanner uses a SQL dialect which matches the ANSI SQL:2011 standard with some extensions for Spanner-specific features. This is a SQL standard simpler than that used in non-distributed databases such as vanilla MySQL, but still supports the relational model (e.g. JOINs). It includes data-definition language statements like CREATE TABLE. Spanner supports 7 data types: bool, int64, float64, string, bytes, date, timestamp[20].
> Cloud Spanner doesn't, however, support data manipulation language (DML) statements. DML includes SQL queries like INSERT and UPDATE. Instead, Spanner's interface definition includes RPCs for mutating rows given their primary key[21]. This is a bit annoying. You would expect a fully-featured SQL database to include DML statements. Even if you don't use DML in your application you'll almost certainly want them for one-off queries you run in a query console.
> Though Cloud Spanner supports a smaller set of SQL than many other relational databases, its dialect is well-documented and fits our use case well. Our requirements for a MySQL replacement are that it supports secondary indices and common SQL aggregations, such as the GROUP BY clause. We've eliminated most of the joins we do, so we haven't tested Cloud Spanner's join performance.
This seems like it'd prevent any kind of easy switch over to Spanner.
Details - https://cloud.google.com/spanner/docs/query-syntax#join-type...
Disclaimer: I work on Cloud Spanner
And yeah it makes it sound like writing an OEM adapter will be much more difficult.
The join performance is by far the most interesting part of this to me. A more traditional NoSQL solution sounds like it would have worked just as well for you, sans all the atomic clock fanciness. Joining across geographically disparate data is a real trick, and it seems like there would be some physical performance limits?
>> Cloud Spanner doesn't, however, support data manipulation language (DML) statements. DML includes SQL queries like INSERT and UPDATE. Instead, Spanner's interface definition includes RPCs for mutating rows given their primary key[21].
Does this mean I need to rewrite my application?
My application uses an ORM and it typically converts my logic to SQL statements and fires them off to Postgres. Would I need to change it such that it doesn't issue INSERT / UPDATE statements?
No, why? Query can be executed in parallel.
BTW, isn't 20k/sec is a very very small performance for 30 node installation. Cassandra can handle 50k+ (both writes and read) on a single node. When in most queries you are trying to collect data from many nodes it will scale almost linearly.
And yes Cassandra will scale linearly-ish as long as you're in the same datacenter. Try running a geo-distributed 30-node Cassandra ring and it's a whole different story at that level of consistency and availability.
Most cases I've seen latency can affect throughput plenty so I doubt your assertion that it won't affect throughput quite a bit. Even more so for anything that relies on TCP/IP.
I'd highly recommend reading the Bigtable[0] and Spanner[1] papers first and maybe then we can have a sensible and fruitful argument.
[0]: https://research.google.com/archive/bigtable.html
[1]: https://research.google.com/archive/spanner.html
[1] https://www.cockroachlabs.com/docs/frequently-asked-question...
[2] https://www.cockroachlabs.com/docs/frequently-asked-question...
No startup will be able to replicate that anytime soon, a lot of time (and money) has been put into it by a lot of people over a long time.
Could any government? Has any government?
My impression is that, infrastructure wise, Google is genuinely in a class of size one.
How much more infrastructure do they have besides AWS? How much does Google have besides GCP?
I don't think you can draw any definitive conclusions from this, but calling it a class of size 1 or 2 is probably an overstatement of Google (+/- Amazon)'s advantage over Microsoft at least.
NSA's annual budget is $50bn. U.S. military budget is about $600bn.
Google's revenue is $90bn and they don't spend all of it.
> A simple statement of the contrast between Spanner and CockroachDB would be: Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception.
CockroachDB is waiting for time keeping hardware to improve.
https://www.ntpsec.org/
Google launching Spanner is generally a positive thing for our industry and our product. It's more proof that what we're aiming for is possible and that there's demand for it. We expect that in five years, all tech companies will be deploying technology like ours.
One of the big differences is that Spanner only uses SQL for read-only operations, with a custom API for writes. We use standard SQL for both reads and writes, which means we also work with major ORMs like GORM, SQLAlchemy, and Hibernate (docs should be live today or tomorrow). Spanner's custom write API will make it difficult to work with existing frameworks, or to convert an existing application to Spanner.
Cloud Spanner only works on Google Cloud and is a black-box managed service. CockroachDB is open source and can be run on-prem or in any cloud on commodity hardware. (We don't offer CockroachDB as a service yet, but may in the future)
At this point, both products are still in beta and are still missing features like back-up and restore (according to the Quizlet blog post). We plan to launch CockroachDB 1.0 with back-up / restore enabled.
* For anyone wanting to know more about how we make CockroachDB work without TrueTime, see our blog post: https://www.cockroachlabs.com/blog/living-without-atomic-clo...
> The underlying time references used by TrueTime are GPS and atomic clocks. TrueTime uses two forms of time reference because they have different failure modes... TrueTime is implemented by a set of time master machines per datacenter and a timeslave daemon per machine. The majority of masters have GPS receivers with dedicated antennas; these masters are separated physically to reduce the effects of [GPS] antenna failures, radio interference, and spoofing. The remaining masters (which we refer to as Armageddon masters) are equipped with atomic clocks. An atomic clock is not that expensive: the cost of an Armageddon master is of the same order as that of a GPS master.
Source: https://static.googleusercontent.com/media/research.google.c...
https://stackoverflow.com/questions/18384883/why-is-googles-...
I suppose if you spent some serious effort with ntpd (maybe hook it up to a PCI ClockCard), you could get this approach to work.
Except that it's not. CockroachDB uses MVCC. Spanner uses 2PL and F1 uses OCC. Not the same at all.
This sounds too good to be true. But it's Google, so maybe not. Time to start reading whitepapers...
What is a distributed system that is CA? Can you build a distributed system which will never have a partition.
> For distributed systems over a “wide area,” it's generally viewed that partitions are inevitable, although not necessarily common. If you believe that partitions are inevitable, any distributed system must be prepared to forfeit either consistency (AP) or availability (CP), which is not a choice anyone wants to make. In fact, the original point of the CAP theorem was to get designers to take this tradeoff seriously. But there are two important caveats: First, you only need to forfeit consistency or availability during an actual partition, and even then there are many mitigations. Second, the actual theorem is about 100% availability; a more interesting discussion is about the tradeoffs involved to achieve realistic high availability.
How does that answer it? Are they implying that partitions will not happen if you don't believe in them?
But that is not a choice with distributed systems, unless as soon as a partition happens the system shuts down immediately. That is it effectively disappears and never re-appears again. But that's not a CA system then?
Or saying that it is not partition resistant is also difficult because the system in case of a partition will do _something_. The typical choices is that it either responds to clients (trying to be AP) or it doesn't (trying to be CP).
That is why I understand CA systems as equivalent to the belief that "partitions can't happen". Which I think is unrealistic.
Yes, it's a pretty much useless system.
This, of course, is effectively useless in practice, and is dependent on an infinite buffer of pending operations, etc.
https://brooker.co.za/blog/2014/07/16/pacelc.html
Doesn't availability mean getting a response on success or failure. If during a partition there is no response on success of failure how is the system available? It seems re-writing a term like "x will happen" to "x will happen after an infinite timeout" should not be valid
https://en.wikipedia.org/wiki/PACELC_theorem
"PACELC builds on the CAP theorem. Both theorems describe how distributed databases have limitations and tradeoffs regarding consistency, availability, and partition tolerance. PACELC however goes further and states that a trade-off also exists, this time between latency and consistency, even in absence of partitions, thus providing a more complete portrayal of the potential consistency tradeoffs for distributed systems."
And I would take that argument one step further and say that latency and partitioning are effectively identical, and from the point of view of any given operation, it is impossible to say whether the system is in partitioned state until max lateny (timeout) has elapsed, because failure to make progress within timeout is the only meaningful definition of partion-induced unavailability.
Sometimes it's helpful to consider a distributed system through the lens of "harvest vs yield" where harvest is the proportion of information in a system reflected by a response, and yield which is the probability of receiving a response at all.
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24...
Now I know it is Eric Brewer himself who wrote that announcement. So I make no claim disproving him or knowing better than, I am just struggling to understand the implications. That CA terms has caused a lot of confusion (at least to me) and so far from what I have read talking about CA systems doesn't make sense.
Never heard of harvest & yield that's an interesting perspective. Thanks for the link, I'll need to go and ponder that for a while. It seems to take a probabilistic approach to these constraints.
But even in that article the description of CA systems is confusing. They describe it it means system can work only in the absence of a partitions. Ok, but then what does it do in the case of partition? The system will presumably do something accept requests, refuse to reply, ... I don't see how it can be both C and A though. That is why I equate that choice with a statement of "I believe partitions will never happen", which I personally think is not realistic.
We know that we can't have 100% harvest and 100% yield in the presence of partitions. We can play with different things though to shape the curve of these things as conditions worsen, however
When Google announced Spanner back in 2012, I'm sure Amazon and Microsoft started teams to reproduce their own versions.
Spanner is not just software. The private network reduces partitions. GPS and atomic clocks for every machine help synchronize time globally. There won't be a Hadoop equivalent for Spanner, unless it includes the hardware spec.
You're right that there's literally nothing else out there that has tight synchronization using atomic clocks, though.
I just noticed Google says the cross-region feature is coming later in 2017. Amazon might be planning to announce a similar change for Aurora in the coming months.
Single region, limited backups and replication topologies, limited performances.
There is yet to see if Spanner can achieve the expectations, if it does, it's a game changer.
It has for Google internally. No reason why they can't share the service externally.
How: 1)Hardware - Gobs and Gobs of Hardware and SRE experience
"Spanner is not running over the public Internet — in fact, every Spanner packet flows only over Google-controlled routers and links (excluding any edge links to remote clients). Furthermore, each data center typically has at least three independent fibers connecting it to the private global network, thus ensuring path diversity for every pair of data centers. Similarly, there is redundancy of equipment and paths within a datacenter. Thus normally catastrophic events, such as cut fiber lines, do not lead to partitions or to outages."
2) Ninja 2PC
"Spanner uses two-phase commit (2PC) and strict two-phase locking to ensure isolation and strong consistency. 2PC has been called the “anti-availability” protocol [Hel16] because all members must be up for it to work. Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down."
Anyone know how exactly this is defined for them? (Time? Queries? Results?)
https://en.wikipedia.org/wiki/High_availability
Five-9s means 5 minutes of downtime per year.
>> This feature is not covered by any SLA
So I would guess that you don't get _any_ guarantess. Not five nines and not even one nine.
MTTR is bounded by reelection latency, rather than replica recovery, although you still may eat a write amplification cost for rereplication.
write amplification is 3-5x of non-quorum-backed 2PC system, depending on replication ensemble size.
google further multiplies write amplification with geo-redundancy, so bump that WA by another 3x+.
it's an insanely high cost to pay for availability, but for an advertising company it's important to count the beans accurately.
And for those wondering, this is why Oracle wants billions of dollars from Google for "Java Copyright Infringement" because the only growth market for Oracle right now is their hosted database service, and whoops Google has a better one now.
It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers!
This is a globally-available, nearly-CAP-beating datastore that powers one of the biggest websites on the internet.
It's not quite apples and oranges, but this is definitely a different problem they are solving.
[1] http://www.computerworld.com/article/2953299/cloud-computing...
Doesn't strike me as a business with complex logic.
Netflix was given as an example of scale. I guess for another example, spanner could be used to store every visa transaction
And their needs are reasonably complex. They use machine learning and big data analytics to generate the list of videos that you should be watching. In order for those to work they need to capture a whole raft of end user metrics e.g. at what point you paused video X.
To be fair, Spanner's cross-region service is coming "later 2017".
It's equivalent, with different (unknown) constraints. Aurora is specifically for scaling workloads in the same way. You can say it's horizontal (machine) over vertical (resource) but it's all a matter of accounting.
The big nono is the Spanner pricepoint. I will stick with Aurora for scaling based on traffic I use, over pricey timeslices.
You would have to have quite a load to justify the switch from cheaper de jour solutions right now (AWS). Relying on the few that do, is a risk.
Do you have a sense of what that limit is?
There's a pretty big price difference between Spanner and Aurora at the entry level so it's useful to explore this.
I don't have a good idea what the upper limit is for an Aurora database setup.
For the former, I don't think they specify beyond "automatic".
For the latter, "service is typically restored in less than 120 seconds, and often less than 60 seconds": http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora...
See here for more info: https://aws.amazon.com/rds/aurora/faqs/#high-availability-an...
Per their pricing page[1] it looks like the largest instance available is a "db.r3.8xlarge", which is a special naming of the "r3.8xlarge" instance type[2] which is 32 cpus and 244gb of memory.
That's a hell of a lot of capacity to exhaust, especially if you're using read replicas to reduce it to only/mostly write workloads. Obviously it's possible to use more than this, but the "sheer scale" argument is a bit of a flat one.
[1] https://aws.amazon.com/rds/aurora/pricing/ [2] https://aws.amazon.com/ec2/instance-types/#r3
Let me present a quote to you: 512 kb ram ought to be enough for everybody
For example, see this benchmark:
http://2ndwatch.com/wp-content/uploads/2016/09/Graph-3.jpg
from this article:
http://2ndwatch.com/blog/benchmarking-amazon-aurora/
Thoughts?
This is not true. Oracle is far more than a database company nowadays in the same way that Microsoft is more than Windows. Oracle has been acquiring high-growth startups at a significant rate.
Which has their 'cloud services' doubling their contribution to revenue year over year and licenses losing 50% of their contribution to revenue year over year.
There 'cloud' collateral is pretty opaque though.
You're still buying the same stuff, but you're outsourcing your dev ops to them on top of it (which may not be a bad thing).
We've been seeing this demand at Fauna. FaunaDB offers distributed consistency, based on Raft and the Calvin protocol instead of depending on specific networking and clock hardware. We've seen a big part of our appeal is the ability to run FaunaDB across multiple cloud services.
On-premises is licensed by core.
We have a developer edition you can use on your local machine, but we don't currently have plans to open source FaunaDB itself.
Edit: Looks like maybe you're referring to the recommended 3 node minimum in production mentioned at https://cloud.google.com/spanner/docs/instance-configuration
It may be recommended as protection against an availability issue on an instance, though, which is, after all, a big reason why you'd want a distributed DB in production.
There are also plenty of choices evolving for developers who aren't looking for hosted solutions (which can sometimes be a showstopper for enterprise on-prem deployments). There's a growing ecosystem of distributed open-source databases to look out for too.
Take Citus, for instance – a Postgres-compatible distributed store which automatically parallelizes normal SQL queries across machines. It's as easy to set up as adding an extension, and people are doing some staggering things in prod with it.
Different audience from BigQuery and Spanner, but no less exciting.
Disclaimer: no professional association, but love their product and the team.
If you are looking for something that is more Postgres flavored (meaning we're just an extension to it so you get all the good stuff of Postgres such as JSONB, PostGIS, etc.) then we hope we'd be a good fit. And we run a managed service on top of AWS as well (https://www.citusdata.com/product/cloud) built by the team that built Heroku Postgres. If curious on pricing you can find it at https://www.citusdata.com/pricing/
I'm not trying to compare on a per-mb level, but it would be nice for smaller scale workloads.
Imagine.. RDS is literally the ONLY place where you can buy a 10 GB data multi-AZ replicated, snapshotted and managed postgresql.
Its pretty much a monopoly, now that Google seems to have officially closed the book on ever supporting postgresql.
Not true. I was looking for a hosted Postgres provider and discovered these two:
https://aiven.io/postgresql (tried it, worked excellently)
https://www.elephantsql.com
I would be hesitant to say that this is a fact.
Er, that's a bet I'd strongly suggest you didn't make.
If anything, hosted Postgres from Google Cloud will be priced in a way that makes Spanner some what more attractive, as a way to get conversions to Spanner in the long-run.
How many nodes are you looking to run for $300/month? Unless you have more than 150gb/node of data, you don't really need a distributed database which is what Citus is for.
Please note what im paying for is availability and reliability.. not for a database per se.
And im not even talking Aurora. That stuff is going to blow every other price point out of the water at probably higher reliability metrics.
Uh, how? I wouldn't be surprised to see a Cloud SQL-like managed Postgres service from Google.
While there's obviously some overlap in the potential market for any relational datastore service, Spanner doesn't really overlap with a cloud Postgres service as much as Cloud SQL does.
the issue is that the migration path of self hosted mysql to cloud sql to spanner is pretty well defined. I dont see postgresql being strategically important or relevant to google for anything.
if I was a startup deciding on my database, there's a lot less compelling reasons to choose postgresql from the point of view of long term viability. hell, I can pretty much do a back of the envelope calculation on how much will it cost me to support 100 million users on mysql.
Is it safe to think that Evernote and Snapchat - startups who are giant success stories - are google mysql hosted? (in some form.. maybe even spanner)
So uber, snapchat, Google, Evernote and a clear cut path for upward scale.
I have very less hope for postgresql on google cloud.
What does Cloud Spanner have to do with MySQL? It's neither API nor SQL-dialect compatible with MySQL. If there are MySQL bits used somewhere in the implementation, they are well hidden, and irrelevant to users.
> the issue is that the migration path of self hosted mysql to cloud sql to spanner is pretty well defined.
So what? Were there a Cloud SQL-like Postgres offering, the same would be true; Spanner is no closer to MySQL than Postgres. (If anything, it's SQL dialect is a little closer to Postgres's dialect than MySQL's, though not so much that you'll get away without doing substantial conversion going from either.)
Though hopefully you'll find many more useful ones about Citus and Postgres broadly on our blog: https://www.citusdata.com/blog/
For example, being a MSSQL performance tuning expert requires years of experience and probably pays very well, but just the other day I read an anecdotal story where someone switched a large BI database to use columnar indexes, allowing them to replace very complex (extreme manual tuning to achieve acceptable performance) queries with just standard SQL with comparable performance.
How long until the scale, pricing, and now transparent & full(?) sql compliance offered by these cloud platforms starts to make traditional RDBMS platforms a niche platform?
EDIT: Also, most DB users don't need global-scale databases.
That is not necessarily a foregone conclusion. In tandem with their marketing of Azure, MS is pushing SQL Server on Linux heavily [0].
"SQL Server is Windows-only" is no longer a valid argument to choose another RDBMS if a startup uses lots of MS tooling but deploy on Linux servers.
[0] https://www.microsoft.com/en-us/sql-server/sql-server-vnext-...
Four years ago, I determined that while development work might seem to be near the top of the food chain, there will at some point where my work will be replaced by AIs.
This is not so different from how word processors replaced the specialist job of typesetters. Word processors make "good enough" typesetting. You can still find typesetters practicing their craft; the rest of us use word processors and don't even think about it.
At the time, I was learning to put the Buddhist ideals of emptiness and impermanence to practice, and to become more emotionally aware: the _main_ reason I had thought I would never be replaced by an AI writing software has more to do with wishful thinking and attachment than any clear-sighted look at this.
I also made a decision to work on the technologies to accelerate this. Rather than becoming intoxicated by the worry, anxiety, and existential anguish, I decided try to face it. Fears are inherently irrational, but just because they are irrational does not mean it is not what you are experiencing. Fears are not so easily banished by labeling them as irrational. Denial is a form of willful ignorance.
Now, having said all that, whether our tech base will come to that, who can say?
Since then, I have been tracking things like:
Viv - a chat assistant that can write it's own queries
DeepMind's demonstration of creating a Turing-complete machine with deep learning using a memory module.
I watched a tech enthusiast write a chat bot. He does not write software professionally. Talking with him over the months when he tinkers with in his spare time, I realized that in the future, you won't have as many software engineers writing code; you would learn how to _train_ AIs when they become sufficiently accessible to the masses. Skills in coaching, negotiation, and management becomes more important then some of the fundamental skills supporting software engineering. And like typesetting, I can see development work being pushed down the eco-ladder.
It's not surprising to me to see that Wired article about how coding becoming blue collar work. And even that will eventually be pushed down even further.
It's not surprising to me about Google's site-reliability engineering book, branding, and approach. I have done system admin work in the past, and I can already see traditional, manual sysadmin work being replaced.
It's easy to get nihilistic about this, but that isn't my point here either. I know the human potential is incredible, but I think we have to let go of our self-serving narratives first.
The second idea that interests me is this idea of very high technology. It is built upon layer after layer of very clever tech year after year that I wonder how long it would take to start again from scratch if some disaster rendered a large part of one of these layers unusable.
For instance, if you were on a desert island, could you (would you want to?) build some piece of tech? An electric generator would be useful, perhaps. How long would it take to build? You'd need knowledge, raw materials, plant, fuel etc. It's not an easy solve. And that's way down the tech stack before you start talking about AIs. I suppose what I'm saying is, that the AI layer is based upon such high tech, that is inherently fragile, because it is so hard to do.
I don't know! :-D
I don't know what society would look like from a purely technological point of view. From a spiritualist point of view, though, it could either go very well or very badly. When everything is automated, would people have enough time and space to really start asking the really big questions? Or would it accelerate and intensify existential anguish?
> There are a small number of people reaping the benefits, and huge swathes of the population being marginalised and disenfranchised as a result.
Yeah. Arguably, this has already happened.
> The second idea that interests me is this idea of very high technology. It is built upon layer after layer of very clever tech year after year that I wonder how long it would take to start again from scratch if some disaster rendered a large part of one of these layers unusable.
The stuff of sci-fi :-D Among them, alt-history novels (what happens when someone drops into a lower-tech era; you'd have to start from 0 ... literally, 0, as in Arabic numerals).
Open Source Ecology is trying to preserve some of this tech base. I find their aims awesome, though I am not sure how effective it is.
The flip side are things being spoken from well outside the techno-sphere, (for example, shamans and mystics) It is the perspective that the further evolution of human consciousness will, at some point, no longer require a technology or artifacts. Technology seen as the last crutch. The collapse of a high-technic civilization then sets the stage for a removal of that crutch, and humans learn to stand with two feet (so to speak).
Not a fair argument against the point made above, however, I believe we will find the next big challenge for software to solve as soon as traditional problems are commoditized/automated and considered solved. Also, just knowing how to code is not going to be enough. You must complement it with domain expertise to solve challenging unsolved real world problems.
There's a reason why regulated (including self-regulated) professions have continuing education requirements; progress happens and you become obsolete if you don't keep up with it.
Just because tech isn't regulated doesn't mean it's any more sensible to expect to remain valuable without keeping up with progress in the field.
That being said, MSSQL experts will likely have good-paying opportunities for quite a while, for the same reason thats then case for any well-established enterprise technology: lots of systems are going to be around using it long after it has become distressingly uncool to spend time learning.
A sad choice though. The centralization of computation is likely not a good thing in the long run.
So, the computational "Gini index" is increasing, but no one is being thrown into computational poverty.
Now, you could play with that analogy further and see some issues as well, but I don't think the issue here is centralized failure; all these data centers/"clouds" are at least good. The Cloud is about businesses focusing on core business and not supporting functions.
[Disclosure, I work on the Google Cloud team, I'm biased]
Only if the owner of the chicken-coop has everyone else's best interests in mind. Protip: They don't.
The Cloud isn't about efficiency, it's about data control. Getting people's systems and data into Google/AWS/etc helps with data mining, vendor lock-in, etc. Often times that can be efficient, but also it often isn't.
Hence we get caching. There's the farms, then the inbound warehouses, then the distribution centers, then the grocer, then our refrigerators by the dozen or dozen and a half. When your local cache is empty of eggs, though, it requires a trip back out to the grocer to get an egg even if you need nothing else that trip. Then you generally have to buy at least half a dozen if not a dozen or more eggs just to get the one you wanted.
If I have my own couple of hens, I can go out into the yard and get an egg. If that's the whole of my fetch list, it's much more efficient for this single egg to have the hens laying right out back.
This whole few baskets metaphor breaks down from another point of view, though, when we consider that by the very nature of using a globally distributed hosted service we're actually eliminating a single basket problem. Yes, there's not much choice among just Google, Amazon, and Microsoft. (That they are the only options is a bit of a strawman, but lets grant this one legs.) However, putting just your own employees in charge of all your infrastructure in just your own datacenter(s) in just PostgreSQL or just MySQL is another single-basket problem. Spreading it out so that someone else gets to manage the hardware and the service and replicating your data widely within that service is from that point of view more baskets. You get more datacenter baskets, more employee baskets, and more software baskets. Using standard SQL means you can move among compliant software later, too, so you're not as tied to those baskets.
Now, back to your coop analogy. What's stopping me from having my application talk to Cloud Spanner and a local database proxy (or a work queue that sits between the app and the DB or whichever) so I can use Google's reliability for transactions and my local cached replicant for query speed when I'm querying older data? Why can't I keep a few eggs around?
Also, why would I be scared of Google or Amazon "having my data"? Why would I put sensitive data into my own database in plaintext and then replicate it among multiple datacenters that way?
Having a devops team with the necessary expertise in Google Cloud or AWS is still a supporting function. You've just traded one skill (managing physical servers) for another (managing proprietary virtual resources).
Yep, it inevitably trends towards commodity infrastructure. As such it will require regulation (and eventually even some form of nationalisation in some cases)
They could tinker with the binaries, something many did with game binaries. But your point is well taken; open source is also very valuable to innovation.
The apps on top, Facebook, Snapchat, etc., are not so open and much of what they do is out of reach from the user.
Also, I meant to add above: People could tinker with data files (e.g., Word docs), configurations, etc. The whole system was local and accessible. You could write local code, such as VB or for Windows, that integrated with those systems.
Putting your savings under the mattress instead of in a bank account wouldn't have prevented the Great Depression either.
The only thing it would have accomplished is making your savings easier to steal.
Maybe not, but consumers increasingly use centralized computation resources. I would guess that by now most applications used by consumers run in their web browser, such as Facebook.
Yes, and this will be disadvantageous over the long run for people that want to run things themselves. Ultimately companies like AMD/Intel go where the big money is at. As things centralize further and further, there will only be 3 customers they care about in the server market.
I agree. It only makes sense if you need special data for statistics, AI training, etc.
In all other cases the classic way of programming on pc and notebook is smarter. If you do everything in the cloud, what if you lose Internet connection? I had that experience several times over the last years.
* Most Internet usage is via smartphone
* Computers are much more stable than they used to be
* Much of the world lives in places with less stable connections
* The most expensive spec in an Internet connection is availability. You can get a low-end 15 Mbps connection with no availability guarantee for $40/month; a T1 is one-tenth the speed and costs 10 times as much (all numbers are rough estimates).
It's only so many heavy industries that definitely need some sort of local infrastructure (probably not master tho) to be locally.
The advantages of owning your own hardware will never go away, but soon this will be made quite intentionally impossible as the big players coalesce and continue building their walled gardens.
This is already happening. All the big players own their hardware and rent it out to everyone else, while trying to convince everyone it's not worth owning your own hardware at the same time.
These companies have already begun closing off server platforms by developing custom hardware and software systems that cannot be bought for any price, only rented. These systems represent a new breed of technology with unbreakable vendor lock in.
Theses same companies compete with each other and countless other companies across the space. Take for example a start-up that wants to run their own app store. Google, Amazon, and Microsoft all run app stores. Where will this company go for cloud services? Their only big name options are to host their software on the hardware of a direct competitor. Their host has full visibility on how their system works, and control over the pricing and reliability of their machines.
It's laughable to think their "cloud partner" will give them any chance to compete if they enter the same market.
We've seen UEFI BIOS and un-unlockable mobiles enter the market in droves the last few years. A lot of new PC's can't run anything except windows. A lot of new phones can only run the carrier's version of android. We have all these general purpose CPUs that can no longer run general purpose programs because "security", and a lot of lobbyist pushing to make it actually illegal to run your own software on these with "anti tampering" laws, again for "security" . Soon the big guys (same companies, MS and Google) will make it impossible to run your own software on any reasonably inexpensive devices and the walled market will be complete.
Mark my words, I've never seen an industry with a couple big players where growth and innovation doesn't eventually turn into collusion, higher prices, and market stagnation. Once MS, Google and Amazon have their slice of the pie and they've killed off everyone else, we will see the death of general purpose computers and mobile devices. Everything you buy will be "android computer" "windows computer" and "apple computer". Anything general purpose will be massively more expensive because individual companies can't get the kind of volume discount of the giant behemoths that increasingly control large swaths of the world's computing power. We've already seen the endgame, with Amazon trialing an "on premesis" version of their compute platform which is basically a super locked down server that you can't buy, only rent endlessly. The future of on premesis will be a cloud in a black box if these companies have anything to do with it. Why? Because once they've got you locked in it makes no sense to sell you anything for keeps. Why keep improving their product so you buy the new version when they can just make it incompatible with everything else and force you to rent it forever, for whatever price they feel like charging?
One day running your own servers will be like running your own ISP . Massively impractical because the free market has been manipulated to the point that it effectively no longer exists
What? People use cloud computing because it already is massively impractical to run your own servers. Hardware is hard to run and scale on your own and experiences economies of scale. This principle is seen everywhere and can hardly be viewed as something controversial. Walmart for instance can sell things at a really low price because of the sheer volume of their sales. Similarly, data centers also experience economies of scale.
As someone who cares about offering the best possible, reliable user experience, cloud computing is absolutely the next logical step from bare metal on-prem servers. When your system experiences load outside the constraints of what it can handle, a properly designed app that has independently scaling microservices horizontally scales.
Even if you had the state of the art microservice architecture running on a kubernetes cluster on your own hardware, you still wouldn't be able to source disk/CPU fast enough if your service happens to experience loads beyond what you provisioned.
And there is the rub, buying your own hardware costs money, and no one wants to buy hardware they may not ever use. Another advantage of cloud computing.
You are seeing the peak of free market right now, because of cloud computing, which enables people with little upfront cash to invest to form real internet businesses and scale massively.
You think a game like Pokemon Go can exists and do the release they did without cloud computing?
One person, with maybe 3 hours a week of time investment after a few weeks of setup and hardware purchase. Using containers I can move between the cloud and my own servers seamlessly, and long as I never bite the golden apple and use any of the cloud's walled garden "services" like S3. If I need more power I can spin up some temporary servers at any cloud provider in a few hours. For me the cloud is a nice thing because I don't use too much of it. If AWS disappeared tomorrow it would be a mild inconvenience, not devestating like it would be to many newer unicorns.
Go ahead and try to use the cloud you're paying for as a CDN or DDoS sheild, or anything amounting to a bastion of free speech. You'll quickly find out that your cloud provider doesn't like you to use all the bandwidth and CPU you pay for, and they don't like running your servers when they disagree with your views. They quietly overprovision everything pulling the same crap as consumer ISPs where they sell you a 100mbps line and punish you if you use more than 10 of that on average. That's the main reason the cloud is so cheap.
Hardware is cheap, colo's are cheap, software is largely easy to manage. The economy of scale they enjoy is from vendor lock-in and overprovisioning more than anything else.
Is it really that hard to double the amount of servers you own every few weeks? No! If you're using containers or managed KVM you can mirror nodes basically for free over the network as soon as the Ethernet is plugged in. Your time amounts to what it takes to put the thing in a rack, plug in the Ethernet, and hit the "on" button. Everybody in SV land thinks you have to use cloud to "scale massively" but they forget that all of today's technology behemoths were built years ago when the cloud didn't exist. Oh yeah, they all still run all of their own hardware too and have from the early days. Using their model as a template, you should own every single server you use and start selling your excess capacity once you get big enough.
Did you ever read about how Netflix tried to run their own hardware but can't because they have so much data in AWS that it would basically bankrupt them to extract it? Look at how these cost models work. Usually inbound bandwidth is extremely cheap or free but outbound is massively more expensive than a dedicated line at a datacenter, 50-100 times the cost if you're saturating that line 24/7. The removal fees from a managed store like S3 or glacier are even more ludicrous. The cloud is like crack and as soon as you start using it more than a few times a year you will get locked in and unable to leave without spending massive $$$. Usually companies figure out this shell game once they're large enough, but by then it's far too late to do anything about it.
Why are they marketing these things so heavily to startups? Because lock in is how they make their money. They make little or nothing on pure compute power, but since you don't have low level hardware access they can charge whatever the hell they want for things like extra IP's, DDoS protection, DC to DC peering, load balancing, auto scaling. You give massive discounts to new players using these systems and inevitably some of these will become the next Uber or Netflix. Then you are free to charge whatever exhoribitant rates you please once it's so impractical to move that it would require a major redesign of the business.
I see it a lot like franchising. By building on Amazon's cloud services you become "Uber company brought to you by Amazon". Like franchising, your upside is limited because any owner with a significant share of total franchises will begin to put pressure on the service owner itself.
Where did you read this? You can have Amazon send you a truck full of hard drives. I doubt it costs more than Netflix can afford.
Also, the truck is for data in, not data out. Getting data out of AWS is far more expensive than putting it in. That's the lock in.
Also, the truck is for data in, not data out. Getting data out of AWS is far more expensive than putting it in. That's the lock in.
This is also not true. The bulk transfer service is bi-directional.
They cache popular content close to the users, they don't manage their catalog at the edges.
But any "lock in" is totally up to you. Take a look at this: https://kubernetes.io/
You can architect your system in a way that it'll run on any cloud provider. All the major Cloud Providers support kube for orchestration.
To be honest I don't think you know what you're talking about. You should refrain from making uninformed opinions on hacker news, especially on a throwaway.
Secondly, software companies like Microsoft, Google and IBM might know a thing or two about running data centers. Due to economies of scale, these companies are inherently in a better position to supply hardware at scale.
> If entire region is down do you think other regions can handle the load. If you think so you're kidding yourself
Netflix routinely does just this to test the resilience of their systems. They pick a random AWS region, and they evacuate it. All the traffic is proxied to the other regions and eventually via DNS the traffic is routed entirely to the surviving regions. No interruption of service is experienced by the users.
Here's a visualization of Netflix simulating a failure on the US-east-1 region and failing over to US-west-1/US-west-2
https://www.youtube.com/watch?v=KVbTjlZ0sfE
The top right node is the one that fails. As the error rate climbs, traffic starts getting proxied over to the surviving nodes, until a DNS switch redirects all traffic to the surviving nodes. Netflix does this monthly, in production. They also run https://github.com/Netflix/SimianArmy on production.
The cloud enables fault tolerance, resiliency and graceful degradation.
No, tooling to failover and spin up new instances does that. An enterprise with 3 data centers can do that.
"the cloud" is just doing it on someone else's hardware.
The most unique thing about spanner is the use of globally synchronized clock timestamps to guarantee "comes before" consistency without the need to actually synchronize everything.
There is nothing stopping startups and open source developers from building the same thing in a few years. The missing ingredient is highly stable GPS and local time sources which will hopefully be available on cloud instances sometime soon. This is a new piece of hardware so it will be interesting to see if cloud providers make one available or use the opportunity to sell their own branded "service" version you can't buy. Unfortunately I think we'll see the latter far before the former, it it ever even exists. Without a highly stable timesource doing what spanner does will be completely impossible.
Yes spanner is special right now but that's even more reason to not go near it. Google has a complete monopoly on it, the strongest vendor lock in you can possibly have
Only "new" in the sense that it is currently not commonly offered, the devices themselves have been available for ages. (If you are a large enough customer you apparently can get at least some colo-facilities to provide you with the roof-access and cabling needed for the antennas). If cloud providers make precise time available I don't see much potential for locking you in with their specific way of providing it, as long as it ends up as precise system time in some way.
I know GPS time sources have been available forever but a fault tolent database needs a backup. The US GPS is incredibly reliable but there have been multiple issues with both Glonass and Galilio.
It sounds like Google has an additional time source making this possible, probably a highly miniaturized atomic clock, possibly on a single chip. There's no way they're running on GPS alone
GOOGLE.
it's actually hard to beat MySQL for a lot of things. i was skeptical about this when I joined google, but as an SRE on the MySQL team around this time, I gained a lot of respect for it.
I don't know if anyone put it to them that way but as Spanner was just getting started when I left I know that one of its success criteria was to be able to be a scalable replacement for MDB. Given the white paper and other papers on their results, I'm sure it managed that requirement.
[1] MDB, Machine Data Base, used throughout the org but especially in Platforms and SRE to keep track of machines and their parts.
Everything you need to know is here: https://research.google.com/pubs/pub38125.html
it walks through the architecture of the Ads DB, the issues with replacing MySQL, and some of the heroic efforts to implement it via Spanner.
At the time, I asked the team why they used MySQL instead of Postgres (which I prefer) and the short answer was: MySQL replication worked at the time.
This pendulum swings. We're pretty near the apex now. A little work on ergonomics and these tools could be turn-key, and back we go to decentralized hardware.
That's what the ExaDatas are supposed to be.
It could end up that way, but lacking INSERT and UPDATE will likely limit this to a niche market for now.