146 comments

[ 3.4 ms ] story [ 219 ms ] thread
Annual flat rate plan is going from 2.3c per slot hour to 4.8c. On demand pricing increasing by 25%. Seems like they are trying to squeeze the users who are most locked in.
That always seemed like the long term play to me for GCP. They were beating AWS/Azure on price but it wasn't profitable. All of the companies who signed up for GCP were incredibly short-sighted if they thought those prices would last forever.

What option do you have though? Migrating off of BigQuery would likely take months for any decent sized deployment.

Considering GCP is still a distant 3rd, I would have thought they'd aim to get more market share 1st :(

BigQuery was 1 of the main reasons anyone would use GCP.

At my previous job, BigQuery was the only Google Cloud service we used -- everything else was AWS.
Seems similar to ours. We only used bigquery and it seems they are not getting those money.
Does aws not have something similar?
BigQuery is much, much better than Redshift.

Mind you, both of them are worse than Snowflake, but not everyone has a spare child to offer up to get Snowflake discounts ;)

Wouldn't Athena be the service closest to BigQuery?
I don't really know, tbh. I've been at Snowflake shops for a long time now, but from a quick overview it looks like the serverless part is Athena, so like BQ in that sense (and SF also).

But given that Athena uses HiveQL for DDL I'm gonna stand by my original statement.

Yes, Athena is the serverless data querying alternative to BigQuery, not Redshift. When I used Athena for some large scale data querying just a couple years ago it was extremely slow, like two orders of magnitude slower than BigQuery for the same queries. And the UI, documentation, and just general user experience is nowhere near as good as BigQuery's. Nothing from AWS competes with BigQuery, it's genuinely a good product.
There is a Redshift serverless offering now, not sure how it compares though. I think it's not 100% there when it comes to not having to think about the underlying infra yourself...
Redshit Serverless still has a minimum cost. Doesn't scale to 0 or even close.
Not really.

Athena lets you run SQL queries on datasets in S3. You can do something similar in BQ with external tables. BQ is much faster. Athena will easily be more expensive on larger datasets, because you pay $5 per terabyte scanned.

AWS doesn't have a service similar to BQ in terms of performance.

I think the 3 main players in the Analytics space are BQ, Snowflake and Databricks.

Note, compute is going up in price, but storage should be cheaper[1]. So it depends on your workload if and by how much this will impact you

[1]https://cloud.google.com/blog/products/data-analytics/introd...

Definitely. But in our case, compute makes up more than 80% of costs, even though we store a large amount of data. I suspect most big customers will come out much worse from this.
> Seems like they are trying to squeeze the users who are most locked in.

That's my understanding too. I'm reminded of the classic Eagles song, "Hotel California."

My immediate answer is:

So? It's not like the customers didn't knew this was a REAL possibility. Given the track-record of Google and companies similar to Google, this was a known quantity even at that time.

Now pay up.

> Given the track-record of Google and companies similar to Google

What does “similar to Google” mean? All for profit companies? My rent doesn’t increase by 100% only because it’s illegal to do so. Otherwise, I don’t see anything immoral about it.

Let me introduce to the World of corporate IT licenses and contracts..

Exhibit number 1: Oracle..

When I called Oracle to inquire about a license and they set up a 1 hour meeting with 5 people in it I knew it would not be cheap. I was right.
I needed an Oracle test database lately to try and get "off" Oracle and they came in and did a licence review.

Ended up having to pay £1m due to some legacy non-compliance and couldn't even afford the test DB anymore.

>> What does “similar to Google” mean? All for profit companies?

Did you really think the comparison of "similar to Google" would amount to "all for-profit companies?"

I think what perhaps was meant by "similar to Google" was:

- Mature technology company - Large enough to be at or near the top worldwide in terms of revenue, employees, users, etc. - Involved in many different technologies rather than a single technology

> What does “similar to Google” mean? All for profit companies?

AWS has a reputation for not raising prices [1, 2, 3].

Presumably this is a deliberate business strategy - customers are much more willing to risk vendor lock-in when they're pretty sure they won't get screwed over.

So given the context, I assume "companies similar to Google" means "corporate IT vendors that haven't chosen a never-increase-prices strategy"

[1] https://www.reddit.com/r/aws/comments/yr8ggq/comment/ivste15... [2] https://aws.amazon.com/blogs/aws-cost-management/amazon-ec2-... [3] https://www.stayclassyinternet.com/articles/investigating-AW...

Will be interesting to see what snowflake does. Does this demonstrate why it’s important you trust a company whose main product is what powers your core offering? Or does it suggest the opposite?
it was well publicized that Google was turning the screws on its GCP sales staff...maybe this is a put-up-or-shut-up moment for GCP to bring in more money...assuming the sales drives didn't yield
Beyond the meme, I don't see how Google could kill GCP without looking like an extremely weak company. They have to make it work to show that they can still deliver results.
I predict Google will be going through major changes in the next twelve months...they've been caught off-guard by OpenAI/Bing...Google isn't used to playing defense and having to fight for search marketshare

by some measures, Google is number 4 in cloud...definitely no better than number 3...you have to wonder if it is worth it for Google to be an also-ran

No, it's just how GCP works and has worked. They've been tweaking pricing like this for a long time already.
Google Cloud is their portal into AI/ML workflows as a hosted service. There's no chance they kill it.
(comment deleted)
(comment deleted)
Slot hour? It's been a year or two since I worked with bigquery but I enjoyed that you only paid for data read.

It was such a simple way to think about it.

At small scales, yes. But at more than that it's much cheaper to use dedicated slot reservations.
They have on-demand pricing, which is metered by data read, and also reserved slots, which are metered by time. The reserved slots come with a considerable discount to on-demand, so it's worth it when you are consuming compute most of the time.
They still have on-demand pricing. It's going from $5/TB to $6.25/TB (a 25% increase).
It’s not just BigQuery. Pricing for cloud services is increasing across every provider. Azure increased prices in the EU recently https://news.microsoft.com/europe/2023/01/05/consistent-glob...

For AWS, one of the people behind https://ec2instances.info showed that the price/performance ratio for newer instances is rising, https://github.com/patmyron/cloud/#compute--memory-unit-pric... meaning cloud users are paying more for less.

AWS has never increased prices though, like since they started. Atleast I can't find any references to any price increase. The link showing "price/performance ratio for newer instances is rising" is deceptive. It just shows vcpu/hour, which doesn't really measure real world performance. Like m5.large is a significant performance improvement over m4.large. Newer instances also carry improvements not shown in the link.
They haven't and agree at the end of the day over the years performance has increased and price has decreased especially with Graviton.
S3 prices increased dramatically for some workloads when AWS introduced per-request fees.

Admittedly that was a very long time ago, but it has happened.

I think the fact that change has already disappeared from our collective memory means people weren’t too bothered.

Meanwhile, Google can’t seem to take a step without tripping over a price increase.

It hasn't disappeared from my memory! There were people who did very, umm, "creative" things with S3 (classic example, creating a virtual disk by storing each 512-byte sector as an S3 object) who suddenly changed direction after the pricing changed.

I agree that AWS does much better than Google though -- the S3 change was a big wakeup call to them about pricing all the dimensions rather than assuming people will use a service the same way as Amazon uses it internally.

That's right -- on 2007-06-01 they switched from:

    Data Transfer
    -------------
    $0.20 per GB - data uploaded
    $0.20 per GB - data downloaded
to:

    Data Transfer
    -------------
    $0.10 per GB - all data transfer in

    $0.18 per GB - first 10 TB / month data transfer out
    $0.16 per GB - next 40 TB / month data transfer out
    $0.13 per GB - data transfer out / month over 50 TB

    Requests
    -------------
    $0.01 per 1,000 PUT or LIST requests
    $0.01 per 10,000 GET and all other requests
    (No charge for delete requests)
While keeping storage at $0.15 per GB-Month.

See https://web.archive.org/web/20070502160305/http://www.amazon... (can't find the announcement page)

Thanks, I had a feeling it was in 2007 but I couldn't remember the exact date.
Interesting that the differential between different regions is remaining the same.

You would imagine that if some big user were slurping up all the spot instances worldwide, then it would level out regional pricing differences. Or, if not level it out, change it based on that users sensitivity to bandwidth costs in/out of each region.

That makes me think that the spot pricing change is a gradual AWS algorithm change instead - I'm sure they'd be smart enough to apply a pricing change with some smooth function based on date.

The problem is there's been growing demand across the world and/or more people optimizing costs and moving from on-demand -> spot. It's not a single user doing this.

There were hardware shortages earlier. I think it's just a case of AWS can't expand fast enough to meet demand for now.

You can shop around a bit for AWS for spot instances. Spot prices in eu-west-3 (Paris) are not that wild yet for the same c5a instances. Somteimes, a region has all its spot instances near the on-demand price. At least they seem to be smoothing out their changes over days. (I think it used to be more wild in the past.)
> Somteimes, a region has all its spot instances near the on-demand price.

That happens? I’ve pretty consistently had all spot prices at 30-40% of on-demand.

Important to note that AWS doesn’t control spot prices: it’s a market where you bid on what you’re willing to pay. Customers control the cost through their bids.

edit; and prices tend to surge at end of quarter, regardless of provider. I’ve seen that for years at AWS and it’s only increased year over year.

Well hold on now. AWS has never, to my knowledge, increased prices for an existing service. They'll introduce new EC2 instance types that aren't as cheap as you were expecting, but the existing types you're already using don't get more expensive. Nobody gets hit with a price increase unless they choose to switch types. If you keep doing the same thing, you'll keep getting the same prices.

On the other hand, this Google BigQuery price increase is a real, honest-to-god price hike for existing customers. If you keep doing the same thing, the cost will change.

> AWS has never, to my knowledge, increased prices for an existing service.

Really? Seems semi-plausible but still surprising

AWS has never raised their prices
They've consistently lowered them. But that makes sense, as they want as many companies to go all-in as possible. Because the egress is still egregiously high, there's terrific lock-in for big clients. They wouldn't raise prices until the exact right moment.
Actually Lightsail halved their prices a few years ago.
What is the best alternative for BigQuery? What is the most economical? Once you are hooked, it is really hard to imagine going to e.g Redshift
Snowflake seems most similar, plus you would get to stay on GCP.
Export your data via avro files and try using Athena
Ideally you’d used Parquet or ORC if querying from Athena/Trino/Presto. Since they are columnar formats you will enjoy considerably faster queues and lower query costs for most query patterns other than “select *…” since the query engine can just retrieve the columns needed for your query instead of the entire row.
100 percent correct.

Ideally you will also partition the data, so your queries can use partitions. Athena charges $5 per terabyte of data scanned, so it's important to get this right.

Athena and Redshift Spectrum are the best alternatives on the AWS side.
Buy the biggest server on Hetzner and install ClickHouse (/s?)
There's also ClickHouse Cloud now.
Even better, there are lots of ClickHouse clouds! ClickHouse Cloud, Altinity.Cloud, DoubleCloud, Aiven, TinyBird...plus many others. This is the best assurance for users that prices for ClickHouse will stay competitive over time.

Disclaimer: I work at Altinity.

Snowflake is easily the best alternative. It's a little expensive, but it just works.
I could never understand how to go about guesstimating what snowflake would cost, their pricing abstraction feels like lunacy as an outsider.
They're the Oracle of cloud databases.

Great product, pricing models are pretty insane.

And despite the pricing, they're still losing money!

That is because they figure out what you will pay and that is what they charge. It is completely disconnected from the costs involved.
Slowflake isn’t really an alternative if someone is switching due to costs. It’s generally more expensive for the same workloads.
Snowflake is much slower than BigQuery in my experience.
For which workload?
(comment deleted)
A little expensive is an understatement. It's black magic to figure out how much it's going to cost. Our test instance of snowflake costs 10x production BigQuery costs, which stores an order of magnitude more data and queries even more
May be a slightly different use case but moving from BigQuery to managed Clickhouse on clickhouse.com has been pretty great. Easy data injection, super fast querying, lots of control over the environment, and a nice SQL console for quickly looking at the data.
We at https://www.gumlet.com use bigquery a lot, but now it seems we will have to move to click house. Can’t trust google with anything. First Kubernetes price intro, then multi region storage price increase along with inter region price introduction and now this.

I would trust AWS but not Google.

When you run out of resources on an elastic cloud: "here's a bill for 2-5x what you thought it would be, good luck paying it!"

When you run out of resources on metal: "your application is slowing down, becoming unresponsive, and falling over, good luck getting it back up!"

It's almost like competing for finite resources favors those with more means to pay for them.
Not necessarily. Combined with the move to compressed storage, really they're shifting the cost from storage to compute. Storage heavy customers may see their cost go down. Probably this just better aligns the pricing with the cost model.

You might also see a bit of savings by using Autoscaling which wasn't available before.

Net/Net it depends on the specific usage pattern whether your cost will go up or down.

The per-GB price for compressed is higher, but more than offset by the fewer GB being measured, depending on your compression ratio. If the per-GB price doubles and your compression ratio is 8:1, that's a 75% reduction on the storage side.

You started with “not really” but then clarified that it does indeed change the costs to the worse to an unquantified fraction of its customers, so what you should have said is “yes really” both literally and figuratively.
No, he started with “not nessesarily”. Words matter.
The article already clarified that not every user will see prices double with the use of “many” rather than “all.”

Thus saying “not nessesarily” changes what? It’s like saying you’re wrong and then repeating what someone said.

Aggressive equivocation.
Good call out, goalpost shifting is one of the primary reason why discussion quality is as poor as it is on the internet.
Title is intentionally misleading even if it's technically accurate (given that "many" has no technical meaning).
Yep. "Many Americans are killed by lightning every year!" Technically true, but obviously sensational.
What's sensationalistic about that? It seems accurate and not loaded to me. "Many" just means "more than a few" and implies "less than a majority".
I wouldn't consider 28 "many" in a context where it's less than one in a million.

(https://www.cdc.gov/disasters/lightning/victimdata/infograph...)

That is the point that gp is making. Many has no definition and many (see what I did there?) disagree about where it’s appropriate.
I don't think it's even technically true in the lightning case
We’ve achieved recursion.

Edit: how can something be “technically” true if the word has no definition?

I said 'many' because 'everyone who uses reserved capacity instead of on-demand pricing' was too long for a headline.
Thats why comments allow more characters. Editorializing belongs in comments.
Anyone who uses reserved capacity will see their prices double?

That's still not true.

You should be particular about the words you choose to be particular about because those words weren't said.
For my company, compute is over 80% of the cost, so we would come out much worse on this even if storage dropped to zero.
And think of it on the positive side, at least they didn't cancel the product in this announcement! Yet!
the title of this post is very editorialized and biased. Why didn't the OP just post the actual title of the announcement?

Compression based pricing is huge for data warehousing loads and will save clients quite a bit of money.

As someone responsible for pricing decisions at my company, it has been very interesting reading through this thread and processing the reactions.

First, I will say that - I feel for users that are reading this blog and trying to process the changes. I completely understand the reaction of those to whom at a glance it may appear that compute bills would go up by 2x as a result of this pricing change. I am sure that is shocking, especially to companies on short budgets. The change is effective on July 5 2023, which is really not that far away. If you have a mature data warehousing deployment, it could take may way longer than 3 months to plan and execute a migration.

However, speaking from experience, it is challenging to come up up with a perfect usage-based pricing model for infrastructure heavy product like BigQuery that suits everyone’s needs, and even harder to communicate it well.

Some of the reasons are…

* There is tension between keeping the pricing simple and yet have it be fair to every workload. If you really want a fair pricing model, you will need to add more and more pricing dimensions to make users pay more granularly for each resource that makes up the infrastructure cost. But a pricing model with 10-20 dimensions is really hard to understand, even if the final price tag would have been lower for each workload, than a more “simple looking” model.

* Deciding on pricing changes of an existing offering is also not easy. I am going to guess the reason for this pricing change wasn’t because Google decided to “nickle and dime” their users (could be of course, but I would hope that is not how most companies think). I would guess it was the autoscaling feature that was the catalyst for the change. Why did the compute price point have to change? Probably because current users are over-provisioning by quite a bit and the current price point is lower to reflect that, so the end price tag is still within reasonable budget and market expectations. If the autoscaling feature is going to work as perfectly as their charts in the blog show, the price point must come up to make up for the margin loss. As some point out, in that scenario, bursty compute workloads that are reserving too much capacity may actually come out ahead.

* Once you do one pricing change, you may as well bundle in other important pricing changes that have “piled up” on the product team backlog. I am guessing that is how the storage change came up - users were confused about paying for uncompressed storage, while some alternatives charged on compressed, and so they made the change. Changing pricing from one dimension to another is tricky. As some pointed out, storage price point went up, but the dimension moved from uncompressed to compressed storage, so the effective price change for most users may actually be lowered, if the compression rate make up for the price point increase.

Anyway, very interesting, I am really curious to see how this change plays out in the market in the coming months.

Bigquery is one of the tools i’ve been most impressed with the last 20 years. It just works so much faster than I assumed it would.

Unfortunately for Google I think this will push some folks to self host some of the OSS alternatives, but maybe not.

When I think of tools I'd want to pay for outside of barebones VMs in a cloud, the things at the top of my mind are Newrelic and BQ.

So they have decided that the growth period for GCP is over and now it needs to squeeze all the profits it can. I thought with all that search money pouring down, they would bet a little longer on GCP growth.

I remember many large companies that migrated to GCP explicitly cited BigQuery cost effective performance. This should change that calculation significantly.

But if GCP loses all cost effectiveness, why would companies chose it, knowing that unlike AWS, GCP products might be priced like a loss leader for some time and can increase any day.

Also, if Google becomes content to be just a small cloud provider, do they just bet their entire future on search. What if search gets disrupted (by something like a next gen LLM).

This isn't the 1st time GCP has re-evaluated pricing. I wouldn't conclude that it's "over". Seems to just be how they work.

E.g. a few months ago pricing of cloud storage was "changed".

Yes and before that GKE engine pricing.

Once you do it enough times it removes any pretence that these increases were one off and establishes it clearly that products are not sensibly priced and prices would increase when the time to drop the hammer comes.

I still am getting a better value than I would at AWS.. and in my view, those are currently the only two games in town.
Agree - "only two games in town".

Tell us more - what specifically is better value and by how much?

Yeah, Google Cloud Storage pricing increases were a meaningful (negative) change.

AWS and Azure storage costs are roughly the same at the list prices. Some SKU unit costs are a bit cheaper on Azure, but you could also probably realize the same difference through other means using slightly different AWS SKUs. Overall, they feel pretty comparable. (Disclaimer: I haven't run the full price comparison. This is only based on comparing the most common SKUs at list price)

But GCP now feels significantly higher in storage costs than both AWS and Azure. I mean significant more in the statistical sense -- the difference might be small for many use-cases, but it's very likely that GCP storage is more expensive for almost all users and all use-cases now.

Being sort of integrated with the rest of Google workspace is nice.
I'm no expert here, but I absolutely guarantee that nobody there decided "the growth period for GCP is over."

All of the big cloud providers are still seeing 20-40% growth annually, and there's still a huge pile of not-yet-on-cloud customers to fight over. Are things doubling annually? No. But there's still a lot of growth left. The bad financial results from Google last quarter were that its cloud revenue growth dropped down to just 32% year-over-year. AWS's last update was a "disappointing" 20% revenue growth (on their $80 billion revenue). Nobody's giving up on growth anytime soon.

I would personally try Snowflake before BigQuery unless I knew BigQuery was cheaper. They do store things in some other cloud but you can choose between the big three.
Something has to pay for the AI Race they are losing right now.
They make enough profits in a quarter to outspend OpenAI for years. They don't need to penny pinch cloud (which was their only promising bet out of ads) to do anything in the AI race. This is just a leadership failure of not knowing what they want to do. Like bet on cloud or bet on AI, bet the company on search for all it's worth, but atleast sit and decide what businesses Google wants to be in.
It was rather expensive (but I'd still use it again) last year when we migrated a custom solution to BigQuery. Now it's even more expensive? It was easily 10k/month for a medium startup.
What was your solution?
A custom set up on top of Postgres and Timescaledb. Which worked excellent when being small but would have involved a DBA b/c of company growth.
Like big oil, it sounds like there is enough "infrastructure" out there that allows them to move the price fixing to third party companies that "recommend" prices for stuff.
(comment deleted)
It's interesting to see GCP raise prices when AWS constantly do the opposite
(comment deleted)
(comment deleted)
(comment deleted)
Google also just added new SKUs last year related to multi-region storage buckets, too. I wonder if we'll start to see similar price increases from AWS/Azure. Usually the cost of tech trends down, not up.
Except this isn't just tech. It's a managed service. The advancement of technology can no longer cover the increase in electricity and salaries.
(comment deleted)
GCP's biggest problem is trust. People (rightfully) don't trust that GCP won't do a rug pull with their production workloads. GCP has done that with both pricing and killing off services.

If you are an infrastructure provider, the first thing people are paying you for is to be a stable foundation. If there is no stable foundation there is no trust.

Its really hard to imagine GCP fixing this without new leadership. Hire someone from AWS who knows what they are doing. AWS doesn't do this shit. AWS goes so far out of its way to not break customer workloads.

So now you are saying infrastructure providers can't raise prices? Is PG&E not allowed to raise prices on my electricity once I signed up? Is my water company not allowed to raise prices despite the drought where I live?

You have way-underestimated the flexibility and resilience of most people. Raising prices isn't the end of the world; for most businesses they just pass it on to their customers.

Uh, PG&E can't just raise their rates. They are regulated by the California Public Utilities Commission and have to get approval for rate changes. But thats not really the point.

If I am choosing between a service provider that has a history of making large pricing changes to customers and one that has a history of pricing stability, I'm going to take the pricing stability every time. Why is AWS so much better at this than GCP? Is AWS simply better at doing financial modeling and projections? Does no one at GCP think about margins before they launch a product?

I think it is more fundamental than that. I think AWS has a respect for its customers that GCP lacks.

For GCP, customers are an inconvenient detail that they would prefer to ignore if they could. If there is pain that will either be felt by GCP or by its customers, GCP tends to push it onto its customers. In contrast AWS goes out of its way to not do that to its customers. Old workloads continue to run. Prices are stable. They treat their customers with respect.

There's a number of things with Amazon's employee culture that I find distasteful. But when they say "they work vigorously to earn and keep customer trust," that is clearly demonstrated in a way that is simply lacking at GCP.

(comment deleted)