I think Firebase was kind of douchebags assuming the story is correct, if you change how people are billed you should give people time to adjust their software.
Stuff like that makes me want to host everything myself as it will for sure be the cheapest option.
Hosting by yourself is always the safest long-term solution, even if you are hosting it in the cloud, you at least protect yourself from this "Company reserves the right to change the TOS at any time..."
It doesn't sound like it's the 'serverless' hosting that is their issue, rather that they have no direct control over the endpoint that their difficult to upgrade client software is hitting. They actually fix it in future versions of the client software by putting controllable serverless endpoints in front of the firebase endpoints they're hitting, so if something like this occurs again they could just change their endpoint to not hit the third-party resource that is costing them so much money.
The more general problem with serverless architectures that this post illustrates is that you're essentially running on a series of FAAS/SAAS style services which can change their billing at any time.
So you either have to code a portability layer so you can move to another one in the event of an unwanted change (which could be quite a lot of work), or you accept the risk that a 3rd party could cost you a lot of money until you can change off their service (which could be quite technically challenging depending on the complexity of your use case and the availability of comparable alternatives)
Whilst with things like container services you retain more control (ultimately you can self-host if need be) as soon as you step onto pure serverless you'll always be at the mercy of the providers of the services you use.
I think they make it pretty clear the problem is with their inability to change the endpoint (i.e. the URL). Switching from lambda to something else could be a pain in the ass but they'd be hitting their own domain and changing where that points is relatively trivial. Right now they have clients out there hitting a Firebase endpoint that they have no control over and that they can't update quickly.
I think Google gets the public cloud. What they struggle with is lower margin, revenue driven, customer obsessed businesses.
First, it's important to understand unless you're buying ads from Google, or renting their machines, you are not their customer. Users are inventory and they spend a lot of money trying to procure inventory (users) for their customers (advertisers).
When you look at it through this lens, you can see how Google as an institution can tolerate experimenting with ideas that are disconnected from revenue models and dropping failed ideas like hot potatoes. It makes sense for them to behave that way.
Amazon's services are almost always designed with either a concrete or direct path to revenue upfront and they have a huge customer support infrastructure they can tap into. They move a little slower in certain areas of R&D because customer obsessed businesses are tolerant of deprecating technology when it doesn't work out. That means they'll do more market research, take less risks and focus on tenaciously steady growth rather than trying to always hit it out of the park.
When choosing a vendor, it's important to understand how their offerings affect the solvency of their business and whether it fits into their core competency.
In this case, Firebase's prices were too low. That should have been a warning sign. The other warning sign is Firebase is a such a small percentage of Google's revenue, Google doesn't have a huge incentive to fix customer service issues.
Have you used them recently? Their support has been extremely good in my experience, and I'm far from a premium customer. Email/phone support readily available. Loads of their engineers available over slack.
Using Firebase or similar is akin to opening a restaurant that serves take-away pizzas from a nearby Pizza Hut, after nicely putting them in a porcelain plate.
If Pizza Hut changes the pricing, you either close shop or adapt.
Pricing mistake 101: You never ever change those old plans. Instead, you grandfather them. Especially so if you're still a young platform and have most growth ahead of you, the "loss" of not charging the new - and supposedly higher - pricing is gonna be trivial 2 years from now, with lots of new projects coming in. But the NPS hit from pissing your most loyal users off and the subsequent damage to your growth curve are huge.
Also, I do love Bezos mantra of always becoming cheaper over time. That is the core of Amazon's brand. Whereas Oracle, IBM, and the old guard always make you pot committed and charge an arm and a leg, the Amazon guys will leave money on the table. That is how you earn trust.
What? they're a business like any other and they're not interested in minimizing profit, there's lot of thought that goes into their pricing and you can be sure that your opinion and trust is an outcome of their strategy to make more money.
I think Amazon pulls this off because their prima donna is the paying customer, not the programmer who thinks talking directly to customers is the worst possible reflection of their ability to automate. Automation is great and all, but Google has a major problem when it comes to empathizing with the customers.
Also, is there any possibility of legal action against Google for this bait and switch, which seems to be quite clearly the case here?
> they're a business like any other and they're not interested in minimizing profit
Actually they are. Minimizing profit is sacrificing short-term gains for long-term gains. Increasing prices helps your short term profit, but you tend to lose customers to competitors. Decreasing prices means less profit right now, but you tend to convert more customers.
Companies aren't purely RIGHT NOW profit-driven. That's why many companies burn cash and take on debt. They're driven by growth (whether vertical or horizontal). Bezos is obsessed with growth. That's why they've only recently started making money. Enormous investments in AWS, FBA and a host of other products put them in the red, but in doing so, Amazon has incredibly horizontal growth.
The timeframe does not change between my first phrase and the last. Overall (as in lifetime) the company is maximizing profits (as companies are designed to do). They certainly aren't "leaving money on the table" unless it's a plan to extract even more, so I don't see why that's so great as the GP comment says. They're good competition for the industry but I don't understand how their pricing automatically engenders trust. Also value is not equal to price and many major companies understand that absolute price is rarely the significant factor.
> Minimizing profit is sacrificing short-term gains for long-term gains.
> The timeframe does not change between my first phrase and the last.
The timeframe dictates behavior. But, you responded to a comment that said Amazon will leave money on the table, arguing that they won't. They will, and do, leave money on the short-term in order to benefit their long-term strategy.
Short-term vs Long-term profit motivations are actually MASSIVELY different sets of behaviors.
> That would actually be maximizing profit...
Yes, if you assume a company is infinite that's true. But since there's currently not a company that's existed forever, I can safely assume that such a thing doesn't exist.
You can't argue profit without arguing time. Particularly when your argument was that all companies focus on profit at the exclusion of all other things. Growth companies focus on growth over profit. And while, yes, over time that can result in more profit, it also may not. If pure profit were the only motivation, short-term MASSIVE profits would be the only goal ever, since it's vastly more real than potential profits if everything works out.
Bezos is worth 80B. Amazon is a global empire operating in practically owns the ecommerce industry and extracts an enormous amount of profit. What exactly are they minimizing?
Charging less to make more is actually maximizing profit for the company.
Tell that to FastMail, who have pissed me off by deciding to end-of-life the "lifetime" 16Mb member account I set up for my father with a one-time $15 payment.
16Mb is modulo zero these days, but it's enough for him - he just deletes some emails when it gets full. He also gets imap access and FastMail's spam filtering, which is really very good. I can also assign him an email address from a domain that I own that is associated with my FastMail enhanced account.
I don't want my $15 back. What I DO want is for FastMail to honour our existing agreement, which was a LIFE-TIME 16 Mb member account in exchange for $15.
Back in January we contacted you with some important information about your @.* FastMail account.
We're now contacting you again to remind you that from 31st July, 2017 our 'Member' level plans will be discontinued.
To continue using your FastMail account after this time you'll need to select one of our current plans. Please note: your email address will remain the same.
As a long-time FastMail user we're offering you 50% OFF all upgrades until the end of July, which includes our already discounted multi-year subscriptions.
And if you upgrade now we'll also add all the remaining time until 31 July for free!
As an added bonus we've also given you $15 credit to put towards your subscription, which is equal to the amount you originally paid for your account.
This means if you were to upgrade to a Basic account today for one year, with the discount and account credit you are effectively getting FastMail at no charge until 31 July 2018 with a massive 2 GB of storage (up from 16 MB)!
There could be technical issues with keeping grandfathered plans around. Maybe the 16mb email accounts were on an old system that was getting increasingly difficult to maintain?
FastMail made an agreement - $15 in exchange for a lifetime member account.
Now they don't want to honour that agreement.
And they've created work for me: What to do about my father's email account?
I have his email address as part of the domain that I have associated with my (legacy) FastMail enhanced account. They've discontinued family accounts for new signups.
Yeah but you could move their account over physically but continue to not bill them. I doubt 16MB is anything more than a properly enforced column value in a CRM.
Thanks for sharing this. I had recently set up an account with FastMail, but now that it's clear that they won't abide by their own promises, I'll look elsewhere.
I would be cautious about extrapolating from that. Yes, it sounds like an unfortunate decision, but I've been a FastMail user for many years and have been very happy with them.
I think you're being a bit hasty. This is literally the first negative thing I've seen anyone post about FastMail, and my experience as a customer of theirs has been fantastic.
Yeah, don't be so fast. I've had a legacy Enhanced account which they show no signs of removing - I went to renew it today and could (though I went to the Standard).
I've had nothing but excellent, and personal, service from them, for years. Very high availability, no canned responses.
Even when bitching about Google's support of "this address used to be with Gmail but is now external, but we'll make it problematic for other users to send it calendar invites", though in no way did I imply Fastmail was at fault, they jumped into the HN thread to say "Hey, let us know if we can be of assistance figuring out the issue".
I'm a long-time and happy Fastmail user. I'm not defending their decision, but I personally never trust "lifetime" plans of any kind -- it's impossible for a business to know they can sustain you forever, and dishonest for them to say they can.
I am a long-time Fastmail user who also was on the "lifetime" 16MB plan. Last year I switched because I wanted some features that were only available with a newer plan. I was a little annoyed to have to give up my lifetime plan but in my case it was more understandable.
I've been very happy with Fastmail for over a decade and this seems like one of the few missteps they've made. New "lifetime" 16MB accounts have not been available in years. It really doesn't seem that there could be very many of them still in active use that it would hurt anything to just continue to grandfather them, especially since they were sold explicitly as "lifetime" one-payment accounts.
Curious - how could FastMail ever properly close/sunset these accounts? Were they going to regularly check on all of their customers to see if they had passed away? Was there some 'we close the email if you don't log in for 5 years straight'?
Serious question - how should these type of lifetime promises ever be properly managed? (especially when the customer is an individual)
I am so surprised why people buy into some SAAS software that does simple things instead of hiring 2 programmers from cheap location (somewhere between India and Poland western border), getting Hetzner VPS or dedicates server (even the second option is cheap, including a few administrator hours) and be done.
It is not so hard to predict failure/buy out of some SAAS provider, it happened so many times. I am curious why Venture Capital investors are not looking at that problem closely.
Nothing in the entire VC Funded startup scene makes sense when you try to apply basic logic, much less a sane way to operate a sustainable business.
But that's just it: These companies are funded by people who are simply in it to make a profit, (as opposed to people who want to actually achieve something with their efforts). Users become dollar signs, and it's all about how quickly you can get enough new users to monetise the bejeezus out of their attention/personal information/etc, before they dump you for the next new flavour of the week.
> Still cheaper for a single dev starting a startup to hook up to Firebase/AWS/Azure than hiring 'cheap' devs from Poland.
Yes but you gotta be smart here. It's not like Firebase is the only database out there. And once your product grows, and you hit Firebase limits which forces you in the "pay as you go" plan, whatever cheapness you thought you bought into quickly becomes a huge expanse.
Of course it's not and I feel sorry for the author, but being smart doesn't mean that you know everything from dev to setting up servers. Most startups are just quickly glued up. What the author and team could do after they started growing is to start slowly moving to own infrastructure/backend. I'd advise any startup anyways to start with Saas and when money starts showing up then hire the remote guy to set up infrastucture.
They have already 'issues' by paying 2k per month for firebase. I don't think you start at some provider for under 500 euros per month for support only. The initial setup is probably much higher.
And than you don't have any control over your infrastructure as well because someone else has theknowledge not you.
I don't see any advantages doing this as long as you are small. Those few hundred dollars are better investet in simple SAAS.
I find it hard to reconcile this suggestion at the bottom of the article
> Always build your architecture in a way that will avoid becoming trapped into a specific service. Amazon’s AWS Lambda sitting between any services and your app is a strongly recommended path!
- Build without depending on external services
- You can do this by depending on this external service
I don't feel that relying on an external service is always bad as long as there's no vendor lock-in with technologies that are open standards. Case in point the managed postgresql and mysql services that Amazon, MS, and Google all support. That said I'm extremely wary of Cloud Spanner and Azure Cosmos DB or any specialized service that the big 3 offer.
> Always build your architecture in a way that will avoid becoming trapped into a specific service. Amazon’s AWS Lambda sitting between any services and your app is a strongly recommended path!
Isn't using AWS Lamba as your gateway, trapping you into using a specific service, just as much as firebase was?
Solution here is to make sure all api endpoints and URLs used in your code (eg into services such as lambda) get mapped through a domain that you can control / change via DNS. Then you can always cut off a service or redirect all requests to a different endpoint of your choosing (eg another serverless system or your own backend). This is better than hardcoding an unalterable endpoint / url / domain in your code that is beyond your own control, and it is doubly important if updating code on your product is difficult or impossible.
This times and again.
If you're hitting an API hosted on a domain you don't control, you're bound to get screwed.
Otherwise they would just be able to deploy a simple Firebase API proxy (for the single call that they use) that uses TLS tickets and keep alive, change the DNS records and call it a day.
You wouldn't have clients talk to AWS Lambda directly. You'd put it behind API Gateway, which would allow you to use your own domain in front. Your clients would just know to send requests to https://myapi.mycorp.com.
Since you control the domain, you could then stop something like this more quickly.
Reminds me of getting Bait-and-Switch'd by Google App Engine in 2011. From 10's of euros to 1000's.
> They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options.
Sounds like Google did a great job with Firebase. Definitely gives you that Google-feel of unresponsive support and no hope when you have problems with any of their services. I bet they also have a "community" support forum or Google Group full of helpless people talking to a virtual wall. (Edit: Yep, they recommend Stack Overflow, Quora, and then their Google Group.. lol. Some things never change)
Yeah, I thought I'd miss GMail, but I really didn't. I used Google Inbox for a while too and thought I'd miss that, but now I use Spark as my mail client so I don't even miss that either.
It was more of a mental "but but my decade-long history of gmail" thing, but it turned out that really was less of a big deal than I thought. I now have a nice custom domain and I forward my gmail to my fastmail, updating subscriptions every so often. Eventually gmail will receive no more mail I care about and I will remove the forwarding altogether. Its getting there!
This. I too dreaded the switch because of all the history, but the migration to fastmail was quick and painless, and I lost nothing. (Arguably, losing everything would've been better, so much useless noise in my history.)
Looking back on it now, about a year after migrating, my only regret is I didn't do it sooner.
Exactly the same experience for me. It's so much faster, more convenient and useful, and the migration tool was a breeze to use, I should have done it years ago.
I just stopped by to join the little FastMail fan crowd over here.
If you aren't a FastMail customer yet, you're gonna assume we're shills. We're not, it's just the sort of quality product that makes you want to tell others. It's like having found Gmail in 2005.
That seems like a minor issue in comparison to the issues with google discussed here. I mean, sure, they should honour the lifetime membership, but is it really worth losing sleep over something you paid $15 for 15 years ago (at the time of that message)?
Sure, but the sentiment in this entire discussion is how unreliable googles paid for service is (never mind their free stuff - have you ever tried to get support from google for gmail or other feee service? From what I've heard it's like talking to a wall). I'd rather pay for good service than get unknown service for free.
I pay for G Suite and basically the only thing anyone uses is Gmail. I don't have hard proof but I feel that all of my emails from my domain never go to spam. I one time had to use Godaddy's email system and ALL emails went to people's spam folder. It is like Google makes you pay for email or your emails won't get delivered properly. Maybe it was Godaddy's mail system's IP that was blacklisted but I think you have to pay to play these days.
Is there some criterion by which GoDaddy is not the worst service provider? Given their popularity with domain parkers and other less savory parts of the IT world, I would assume that the blame was entirely on their shoulders. That said, configuring an email server correctly seems to be rather an involved process[0] so we should perhaps not rule out some measure of incompetence as well as shadiness.
The reason we're all here, support. One of the reasons GoDaddy is a company I still use and have used for years is that they have 24/7 telephone support that's reasonably decent. I have called and gotten domain issues sorted by GoDaddy at 2 AM on a Sunday by a dude who didn't have a foreign accent.
I'm not a big account with them, surely trivial compared to those mass domain parkers you mention, but they've never made me feel like I was less valued as a customer. I have received calls from humans just to ask me if I am currently satisfied with my service with them.
Especially if you have a business-critical service, this should simply be a baseline requirement for support, but it's becoming shockingly rare these days.
I've had the complete opposite experience with GoDaddy support. Many of them are not so knowledgable and can easily waste time, and billing matters in particular can be quite disorganized with them if something goes wrong. I moved all my domains way from GoDaddy.
I did that. It took me two years to move everything over (email took the longest time), but it's been totally worth it. I still have my @gmail address, but it's only there to forward email to my real address, and I never log in or send email. Besides junk mail and spam, I am now at a point where I get maybe 1-2 real emails on my @gmail account, and then I tell the sender to use my new address.
My google usage is now
1. Occasionally watching Youtube videos in an incognito window
2. Very rarely using !g to search Google via DDG
3. Using Gmaps in an incognito window a couple times a month
and that's it. I don't find that I miss it at all. If I had to stop using Youtube and Gmaps, I won't miss 'em.
Now, if only good and valuable startups weren't getting bought by google by the dozens :( I guess it's your cue to start migrating to something in house when it happens.
I use GCP for "small stuff" and never have any issues at all: using GAE with the maximum number of instances set to 1 on a paid plan, using cloud storage to host static web sites, and occasionally spin up a really large VPS (their maximum memory and CPU core offering) for a short time periods to run machine learning and other large tasks.
Yes, for that reason a POC I made for a side project costed almost 100 bucks to me (I was the single user)... I never checked the current bill because I was sure the total cost would have been around 15-20... when I got charged I noticed all the previous pushed versions had a single instance running, so the costs were incremental...
Let's just say that testing out the platform by pushing a 1 page test app a few times and then being burned by a $200 bill has put me off the platform for a while.
It may be my fault for not adequately reading all their caveats, but it seems to me that the default experience shouldn't be like that.
I love Google, but their Cloud Platform is still incomprehensible to me.
AFAIK this doesn't happen by default, since by default the autoscaling will reduce instance count to 0 if the old versions aren't receiving any traffic.
Yep. For me it was running instances that didn't respond to requests to delete or stop. They kept charging anyway.
Support was somebody in a different timezone who complained when I didn't answer my phone in the middle of the night. I quite liked the Google Cloud offering, but to get so burnt so quickly... stay away.
Sorry to hear about this. I work for our cloud platform support team ( we don't cover firebsse) and this is not the impression we want our customers to get.
We don't want to be calling customers in the middle of their night while they are sleeping. Its not an effective way to resolve issues. We need help identifying when we fall short so we can fix the issues.
Ways to let us know:
1) fill out the survey when your case is closed. We review every survey but less then 1% of cases get the survey filled out.
2) tweet your case id at us and it also goes through our review process
3) email me (tsg@Google.com) and I can correct the problems.
4) GCP slack community has a whole bunch of active Googlers (and the community will flag one us down) if you are having problems with the support team.
Funnily, I was working on cloning a particular piece of Firebase functionality, and inquired if they could help me with that (or if they had that particular piece in open source), and got more replies from them for that than apparently this paying user ever got.
Yes. They are in the "Embrace" phase of the "Embrace, Extend, Extinguish". Sure they are helpful. Have to be (considering the API drift between the Tensorflow releases and how late they were - Theano is the same exact idea and great implementation, only first released in 2009-ish).
But it's pretty easy to look a bit into the future, and predict what we would see during "Extend and Extinguish" phases of TensorFlow project.
Never quite understood what's "appalling" about it. They've never had strong support for any dev/cloud-oriented services. I know some folks who've had good experiences, but those were exceptions, not the rule.
I'll say, I get the frustration with it, but it's why I've never considered using their cloud services for anything. I had a client who asked me to migrate all their stuff to google 'cloud compute' and their 'sql cloud' offering last year. It was a performance disaster; queries that were minor .1ms on localhost or in the same network were suddenly 18-20ms hitting google cloud. Spent a few days trying to dig in, and a few responses from various groups were "shouldn't be that slow" and "works for me" and that was about it. Gave up trying to get any real response from google (we may have got a 'looking in to it' response - I don't remember).
Thing was, I knew going in it wasn't going to end well, I just wasn't sure exactly why (maybe that's just too cynical of a view, but I prefer 'realistic').
Just as a restaurant can be appalling from when it was first opened and even if you don't visit it ever, other services can. Appalling doesn't depend on personal experience or lack of prior knowledge.
It's appalling because you're a paying customer. I can understand free services not having support, but any service that's costing money should at the very least have a human being responding through email.
a) hundreds and thousands of horror stories from people saying "I did not get any support from google". You can counter that with "here are people who did get support", but the existence of the thousands of support nightmare stories out there gives you enough evidence to know that this isn't really "supported" in the way you might expect.
b) there are no phone numbers.
https://firebase.google.com/support/ - no phone numbers. That's, of course, not the sole criteria you should use to judge potential support, but... it's google.
It's 2017. We have multiple years of knowing what Google's track record is re: support for paying customers. Outside of large adwords advertisers (where they make a lot of money), it's non-existent from a practical standpoint.
The big problem here is that firebase was fine until google acquired them.
I'm reminded of a time when I had a bad experience at a restaurant. Local place I went to with friends from work. We went... probably 4-5 times per year, and I loved the chicken sandwich they did. Last time I went, it was ... bad. Not what I'd expected at all. It was different.
Server asks "how's everything?". I said I was disappointed that the chicken sandwich had changed - it was not as good as before. Now... I wasn't actually complaining as in "comp the meal", but I was not a satisfied diner.
"Nothing's changed sir".
"um... yeah - this is nowhere near as good as it used to be".
"You're wrong, nothing's changed".
We had a back and forth, and they went back and came out "the cooks say the recipe is exactly the same".
This was getting a bit ridiculous. Manager came over, same exchange. He leaves, the comes back and says "they're right - the recipe is 100% the same. We have changed a few ingredients last month, but it's the same".
Maybe that doesn't actually relate to the google service situation at all now, come to think of it, but I was reminded of that situation when I started typing it :)
Oh yeah - nothing changed from OP's end, except how the new owner (google) decided to start monitoring and accounting for usage. Think of a "$9.99 all you can eat buffet" getting loyal diners, then changing owners, and the new owners giving some people a $95 bill at the end of the meal because they'd decided to also charge for any visits to the salad bar over 2. Most diners would never hit that limit, but a few - loyal diners for years - would have been (rightly) upset at the after-the-fact change.
I know this but to say everyone does or should isn't exactly fair. As well, you assume I am holding the purse strings, that I'm in any way responsible or even have a say in procurement. This may be true in my own business (it is) but not necessarily elsewhere, like my clients (it more often than not isn't) and you'll just have to live with poor decisions made on your behalf.
Also quite often, there's a sunk cost that makes switching that much harder. So no, it's not fair to say "Google has shitty support but that's ok because we all kind of know this." Paying customers can and should complain about this, loudly, shouting it from the roof tops. Ideally they vote with their wallets and just leave Google, but again that's just easier said than done in many cases.
Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less.
"Ideally they vote with their wallets and just leave Google, but again that's just easier said than done in many cases."
That's the ONLY thing that will ever make this change, but it rarely happens because "sunk cost" rationalizations (and "maybe it'll get better next year!").
Shouting from the rooftops for years has not worked. This is not anywhere near a 'new' problem.
Yes, possibly some people don't know it, and yeah, I totally get that "you'll just have to live with poor decisions made on your behalf.".
I live with poor decisions all the time (mine and others'). But the notion that we can bitch about it on forums, or make 'formal' requests for better support - none of that makes a lick of difference to them (or, demonstrably hasn't for several years). It's "google", so some people will flock to it because of the brand, or because it's not azure, or aws, or whatever. Or to tick the 'cloud' box on their spreadsheet. Or because google gave them thousands in free credits to switch.
"Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less."
But... it demonstrably is, and has been.
And... you'll get a client who will say "I can't believe this, email someone, get this fixed". At some point you have to push back and say "no, this is simply not fixable, you have to live with XYZ, or pay the cost of moving somewhere else".
An extremely large customer, maybe, possibly, might have a bit of say, perhaps. But even if you're dropping millions with them... for a company that measures revenue in hundreds of millions of dollars per day, you probably won't hold much sway. If you had that sort of budget to spend, and it was that critical, you'd likely be doing enough due diligence that research in to actual support/service levels (vs what the sales contact tells you) would be on the agenda.
Vote with the wallet is the only viable/impactful approach, and people have to be willing to forgo the sunk costs.
And for people that don't know up front - they always had the opportunity to search (google) beforehand. If they choose not to - caveat emptor.
10 years ago, we could possibly be forgiven for choosing google's services (GAE, etc) because there wasn't as much of a track record to look at re: support, etc. There's no good excuse for starting a project today on google compute and claiming ignorance of their support levels.
Ok, I've missed your point now – what are you actually trying to say? That Google being unprofessional and delivering on the whole sub-par services is OK because a lot (but not all) people already know this, and so really paying customers should just suck it up or go elsewhere, regardless of whether or not they even can in the first place?
If I've misrepresented what you're trying to say here please do correct me, because I clearly don't get it. If I haven't, I suppose we'll just have to agree to disagree – I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere.
But to avoid it, you generally do research ahead of time. And/or be willing to actually walk away from your 'investment' on their platform and start again somewhere else.
In the OP case, they didn't sign up with google in the first place. It's doubly shitty for them, because they didn't ask for this, and could not have known in advance that google would end up being the support provider. Similar to when I refinanced a house, and 6 month later, my 'servicer' sold my loan to someone else with much crappier service (charge me $3/month to take an electronic payment from me? I wouldn't have chosen this as a service option).
Yes, paying customers should go elsewhere. Complaining about it to google directly and indirectly for a decade has not noticeable improved their support for their cloud offerings.
Suck it up or go elsewhere. You can keep registering complaints with them, or organize digital protests, or whatever, but to get better service, you will likely have to go somewhere else.
"I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere". Of course they can. They just need to realize it will not change the quality of support. Someone earlier (you?) said to 'vote with your wallet'. Shouting from the digital rooftops about how crappy service X is, but continuing to pay them, is a waste of time at best, and enabling at worst. Leaving for another service is voting with your wallet/budget, and it needs to happen.
Sure but voting with your wallet isn't, as mentioned previously, always an option for various reasons. Shouting from the roof tops is little recourse, I agree, but if nothing else it can be cathartic. I don't think that should be discouraged.
"as mentioned previously, always an option for various reasons. "
In the very short term, agreed - it doesn't do much good. But shouting from the rooftops internally (at client, management, stakeholders, etc) may do more good in the medium term.
So, you agree that Google's service is appalling, but claim that it is not appalling because the degree of appal has not changed over time? I cannot follow this logic.
totally agreed that it wasn't google. people signed up for firebase and expected whatever level of service they had with firebase to continue. google acquiring them, then changing things for the worse for existing customers with little-to-no documentation/warning/etc - it's inexcusable. but people will stay with them because "we're already here", and they'll continue the abusive cycle for years.
And yes, on the comic - ok, not everyone "knows" everything up front. But... you have search engines (google, even!) to do actual research beforehand.
I see small business folks and small consultancies do far more research on comparing various $7/month hosting plans via hosting review sites than I see people researching their decisions to use AWS vs Google vs Azure.
Nothing you've said makes that any more acceptable. I don't care if we "know going in"; it's still unacceptable, and they should be investigated for it.
I believe that the standard approach for getting support from google is to hire an ex-googler who has good relationships with relevant people still at the company. It's ludicrous, but I've seen it work.
I don't think it's fair you're being down voted, I get your point of view, so I gave you an upvote. But the thing is, this isn't some kid-in-a-basement free-but-ad-supported service where you-get-what-you-pay-for. Google's cloud offering isn't free (for any use beyond hello world anyway) and as a paying customer I'd expect:
1. Things to work
2. People to answer when things don't work
3. Things to start working again in a timely manner, with timely and concise updates along the way
Points two and three carry a lot of nuance, but Google's support does not. It's a virtual wall that people cry to and get no response back from, except for the rare few occasions when the Google deities decide to grace forums or mailing lists with their presence. It's just not a serious offering.
Now, you can pay extra for support, but even so it's going to be appalling. You probably will get to talk to someone at least, but in my experience you rarely get ahold of anyone who'll actually read what you've sent them, much less actually try to help.
My take away is that support and customer service is just not a priority for Google, what so ever. It's not even a low priority thing, it's a no priority thing.
is there a typo here? 18ms seems like soemthing that could be designed into their whole infrastructure for some reason you don't know about (such as partitioning of networks, whatever) -- after all, it's a cloud solution, right? Maybe 18 ms tacked onto every request is fine for them.
To be honest, if all of my Internet requests were 18 ms slower I wouldn't even notice.
if I read you correctly and it wasn't a typo, could you mention why you cared so much? The .1 ms you quoted on localhost is extraordinarily quick and I wouldn't expect it to translate to a cloud...
Because we had a shitty codebase that made 350 SQL calls on one page (the user dashboard), so the primary page that everyone used went from < 500ms to > 8 seconds, and the end client who had to test and accept this change didn't like it. It became unusable.
To (re?)clarify - this was 18ms per query to the SQL cloud (from a google compute engine).
Yes, were there things that could be done to reduce that? Of course. The other guy on the project put some caching in, reduced duplicate DB calls, etc, and got it down to around 3-4 seconds, but without severely diving in and learning far more about this inherited codebase, there wasn't much that could be done to make it much faster.
The other steps involved heavier caching, which changed the actual functionality of the service that was contracted before. "real time status updates" had a functional definition of being within X seconds, IIRC. Caching everything so 'new' information wasn't necessarily available to an end user for, say, 3 minutes, wasn't an option, without getting a lot of buy-in from a lot of people. Not saying it wouldn't work technically, but the client relationship was already strained.
The move to google was primarily to 'save money' (because someone had $10k service credit with google), but the effort to make it work well on google cloud and SQL combo (wasn't my call) was too much.
You can say this isn't google's fault entirely - you'd probably have a strong case - but not everyone is moving pristine clean/strong/good code over, and problems with the original base were amplified by seemingly small issues.
I hope you don't think I'm blaming your code base, but just out of pure curiosity, as an amateur question could you mention why the SQL calls couldn't be done in parallel -- did each one depend on the next or something?
At the high level I get everything you just said, just curious about this point, if you remember.
I made it a point a long time ago to never use any SaaS, PaaS, or cloud solutions from Google for anything important. First off, their offerings tend to be much more expensive than competitors. Second, Google tends to randomly drop features, services, and support for applications without warning whenever they want. Third, they don't have any actual support at all for anything. I use Linode for all of my hosting, and I can get tech support on the phone in less than 5 minutes even without the upgraded support plan. On top of this, Linode and DigitalOcean offer extensive community curated documentation for almost everything.
The above reasons are also why I don't use any open source software developed or maintained solely by Google. There's no guarantee that Google won't just stop maintaining it after a year or less. For a somewhat recent project, my co-founder and I evaluated using Angular2 for the front-end of our application. During our evaluation, we learned about Angular4. Knowing about what happened between Angular1 and Angular2, we decided to steer clear of the angular ecosystem. This decision also made more sense from the performance and ecosystem side of the argument. Most groups that develop open source software do their best to ensure that there is a well documented and simple upgrade path. Look at Django, rails, React, Vue, postgresql, etc for example. As far as I have experienced, Google products offer no such simple upgrade path.
To be honest, I'm not really sure what value Google is offering anymore. The only quality product from them I still find any use for is Gmail. Even Gmail has competitors that do Email better. I guess there's also youtube, but Google are kind of shitting that up as well.
I think the tech space needs to walk away from worshiping Google and Facebook. In all reality, the technical quality of their products is lacking and their entire goal is to lock both consumers and developers inside their own walled gardens.
I hear you. I've made an exception on angular2, partially because it's being baked in to some other tools I use, and I think those communities will help support it as well. And from what I can tell, the 'a2->a3->a4' is much more minor support numbering vs the "everything is redone" from a1->a2.
But this is a somewhat easier choice to make because I still have the code locally. Google could stop supporting it tomorrow, and it would not be fun, but things wouldn't stop working, unlike the firebase situation above, where the option is "turn off your app".
That said, I've generally been using knockout and vue (more vue recently than knockout) and both have been fine.
"Third, they don't have any actual support at all for anything". There's a lot of people on this thread that feel like because they're paying for something they should get support. And... while I'm sympathetic to that viewpoint, and agree they should, we've had 10+ years of seeing google consistently under- or non- support many services that people seem to want to base their entire businesses on.
The easiest way of dealing with Google is to just cancel your credit card... You won't get your money back (unless a CC dispute gets it back). Google are impossible to get any customer service from, which is why I refuse to use any of their non-free offerings (and have limited my use of their free stuff to hangouts and google docs).
Take them to small claims court, if you have a legitimate grievance. In most jurisdictions a lawyer cannot represent a defendant. At best it will get them moving. It will be sure to at least make them take notice, even if you you have at least costed them and made them account for your grievance.
edit: Meant, even if you lose you have cost them significantly an that has the intended effect of signaling they should improve customer service.
Does that matter if every Google service is run like the one in the article? That sounds a bit like saying you'll ruin any potential friendship you could have if you fought back against a bully
If it was the only way to recover several thousand dollars that they stuck me with I'd absolutely do it, and make sure to do a couple orders magnitude more damage to them with widespread recommendation of AWS, Azure, Linode and basically anyone but Google. It'll catch up to them eventually.
Well, it sort of makes sense, Google is literally "speak to the wall". Isn't that what we do all day long while googling? We're no longer asking humans and asking questions to a "talking wall".
I learnt this lesson the hard way. Solution: get a separate debit (not credit!) card for all cloud stuff. Make sure you only transfer enough money on to it each month to cover what you reasonably expect your bills to be (and that you can quickly top it up in hours if you legitimately need to).
Worst case, if AWS or Google decide to fuck you over, let the bill bounce. This way you've still got funds on hand to deal with the fall out and some leverage to negotiate a solution.
Or the bank can just automatically give you an overdraft for that amount, add some fees because you accessed that facility, and require you to pay the full amount.
After all, they're just being helpful in managing your money and paying your suppliers.
This is why no reasonable person should use debit cards for recurring payments.
I'd advise not using debit cards for anything, under any circumstances, and relying on the better protections that credit cards provide, but it's a losing battle. My debit card is ATM-only.
You're right and the trick is setting up a separate account for the debit card (ideally with a different bank).
This is generally a good approach for all sorts of reasons, not just protection from over enthusiastic cloud bills.
Keep most of your funds in one bank (and never use the cards). Transfer from this account to one or more other accounts with other banks and use the cards on these secondary accounts to pay for things. This way you're protected from suppliers over billing and banks going crazy with unarranged overdrafts (which aren't secured so absolute worst case you just tell them to fuck off and default on it).
Basically, the trick is holding on to your cash. Much harder to fix things rapidly if you're having to negotiate with a third party between you and the supplier (which is the case with a credit card company).
Yes, you can. The last few banks accounts I've created have all asked if I wanted overdraft protection. You say "NO! decline ANY transaction that would cause an overdraft"
But they seem to be able to delay debits for several days, perhaps a week? At least on WellsFargo. Isn't this a backdoor/loophole to get around the no overdraft issue?
Now in the USA as of 2012, banks cannot turn on overdraft protection on a debit card unless the customer requests. Prior to this they were taking your day's debits and applying the largest one first. So suppose you bought lunch for $10 and a coffee for $3 then your cable company later in the day billed you for an amount in excess of your account. They would then apply the cable company bill first, then the 2 smaller charges to maximize overdraft fees and hit you with 3 $30 charges.
The order wouldn't matter in that case, would it? Either way it's still deducting the sum of those three transactions at the end of the day.
I thought what they were doing is applying all your debits, charging you if that puts you into overdraft, and then applying any credits. If they applied the credits first, you wouldn't go below zero.
The order still matters. If you have $20 in your account and get hit with three debits for $6, $10, and then $30, processing them in order means the $6 and $10 debits go through, leaving you with $4 in the bank, then the $30 debit causes an overdraft and a single fee. Comparatively, processing the $30 debit first immediately causes an overdraft, and the $10 and $6 debits cause two more overdrafts for three total fees.
Either way, you're trying to charge $46 against an account with $20, but one method results in 1 fee and one method results in 3 fees. The issue of credits vs debits you mentioned can compound this issue.
I had this happen to me in real life, back as a broke college student. The difference is the number of transactions you get hit with the overdraft fee for.
Consider a bank account with $20 in it, the day before payday, when you forget your World of Warcraft subscription hits today instead of tomorrow. That's $15. But you buy:
If the transactions happened in-order, the subscription causes the overdraft, and you're charged $25 or whatever (yeah they are very high). But in the way banks used to do it, they'd take your total ledger for the day, and order it by highest $ first. So, subscription first, $5 left. Lunch, $0 left. Coffee, dinner, and snack are three transactions when overdrawn, so that's $25 * 3 = $75 in fees instead.
Banks claimed that this process was because you'd probably want your highest-amounts paid first, as it'd be something like rent money. In the end, it means way more fees for them, though.
$75 in fees is especially brutal when you don't have much money in the first place; it pretty much caused me to not be able to even go to 7-11 for a candy bar for two months.
I agree, let the providers take the risk. Typically, the providers will let it bounce a few times before cutting things off as well. This might just give you another week or two to sort things out, without having to lay out possibly significant funds that you may or may not be able to reclaim.
I only wish setting up virtual cards were as easy as setting up a new mail box. I know there are services out there that lets you do this, but I've yet to find one that works globally and is easy to work with.
To be honest, this all sounds like it should be a primary bank feature. I should be able to cap a recurring payment at $X for a specific vendor otherwise have it go through automatically.
That South African hackable (in the good sense of the word) bank that's been making the rounds on HN might be able to do this. (I forget the name of the bank.)
My personal bank sort of gives me that with their fraud protection. If a one off huge charge appears from a company that regularly bills me (but at much smaller amounts), their fraud system sends a text asking if I want to authorise or block the charge.
The second bank account is really more for protecting you against black swan events. If all your accounts are under the same bank and have the same owner, there's nothing stopping the bank taking funds from one of your accounts to make up for shortfalls in another.
CaptialOne does this pretty well. So far it is the only bank I have come across would alert me they are blocking the transaction for a few days so I have a chance to review and if I don't they will release the block. Every 6 months or so they have someone called me up to verify recent transactions.
Hell, their mobile app alerts me the moment a recurring charge posts for a different amount than normal, if I put my NYT delivery on hold for travel or something I can guarantee the next two months I'll get an alert to review the transaction (though they don't outright block it)
This works well in India, with 2-factor auth — Indian companies can't charge my debit or credit card just because they have the card number, expiry and CVV. I have to approve each transaction with my bank.
I'm wary of giving my card information to companies outside India because I don't have this protection.
>Indian companies can't charge my debit or credit card just because they have the card number, expiry and CVV. I have to approve each transaction with my bank.
True for Indian companies but what if you get charged via Forex abroad? For me - Google or Steam - don't required 2FA. All I have to do is to enter my number, confirm CVV then zap. It gets debited. I do get a call from my bank (HDFC) asking if this transaction has been initiated by me, if I say yes, they let go.
My bank (in Sweden) lets me generate virtual debit card numbers with a specified cost cap and expiration date. It's some ancient-looking Flash widget though so I suspect it's a legacy feature...
I know that bank of america offers this feature on all its personal credit cards. You can create a temporary card number (and related info) as well as how much can be charged in a given month and how many months the card number should be active for.
The only reason I don't use them for everything is because I get better cashback/points with other cards.
Well, that's pretty much what a virtual card is – you transfer funds to it like you would with any other bank account, and it's not in your bank. It'll be a debit card with no overdraft, so any charges over what is in the account will be denied. This won't fly with places that require a credit card, like car rental agencies for instance, but that's a non issue in this case obviously.
Definitely use a different bank. A friend had a separate checking account for their PayPal purchases. And when PayPal clawed back some money (as they are wont to do), the bank manager helpfully transferred money into the account to avoid an overdraft charge.
Looks great, but like many other services like it, it's US only[1]:
> Is Emburse available outside of the U.S.?
> Only companies incorporated within the U.S. may register for an Emburse account. In addition, the primary account owner must be a resident of the United States with a Social Security number and a verifiable physical U.S. street address (no P.O. Boxes). We do not offer Emburse accounts to residents of Puerto Rico, Guam, the U.S. Virgin Islands, American Samoa, and other U.S. territories.
I just started using privacy.com and so far it has been an excellent experience. They provide everything my previous VAN-providing CC company did, and more. The one and only downside is that at present, the VAN uses ACH debit to draw immediately any funds used; there's no credit as such (although they are looking into that).
EDIT: I have no affiliation with privacy.com; I'm just a satisfied user.
Of course, this approach only works if you're happy to have your service cut off more or less abruptly if your card bounces, something the author of the article explicitly mentions isn't an option.
The problem (in the article at least) isn't being surreptitiously billed a large amount, it is cost of a critical service changing dramatically without warning.
They won't cut you off immediately and the gain you get is some breathing space/wriggle room to decide what your next step is going to be once you've been landed with a huge, unexpected bill.
Also, nothing gets them talking to you faster than a bounced invoice :)
Perhaps. Or perhaps they block or throttle your account. Stopping payment is the nuclear option, you can't expect any cooperation with outstanding payments on your account (not that you necessarily can with a fully paid up account), and your position, if you should choose to pursue legal action, just got a lot more complicated.
To be clear, the only time any of this matters is when a huge, unexpected bill arrives. Day to day, you should anticipate any big, legitimate bills and make sure funds are there to settle them (or you scale back usage to avoid getting smacked).
It's when a big bill lands in error or is introduced sneakily that having a separate "cloud only card" can save you. In that case, it's the supplier that's gone nuclear and you're just responding in kind.
Sure, but that's just not what we're discussing here. The author of the original article clearly authorised the increased spend (but obviously not the sudden change causing it), and could have declined (presumably at the cost of having his app throttled or shut down). Nothing about a "cloud only card" would have changed his situation the least.
Totally agree. If he/she authorised it after being given an opportunity to decline then, yes, it's absolutely on them. /me clearly didn't read the article ;)
I've had an invoiced account with AWS since 2007. We have been 9 months behind on payments at times and they have never suspended the account. Keep a line open with then and they will be reasonable. We are currently current with then but have been behind for more that 2/3 the history of the account.
> The problem [...] isn't being surreptitiously billed a large amount, it is cost of a critical service changing dramatically without warning.
Sorry, but this is the risk of using somebody else's service -- especially
Google's -- as a critical component of a business. It may be marvellous for
developers not to bother with infrastructure for their own service, but this
strategy likes to come back and bite in the ass.
You always have to use someone else's service - even if it's just the electricty supply. But different services come with different risks, and you have to try to gauge them.
No, this is the risk of using somebody else's service without getting your rates locked in with a contract. You know, a Service Level Agreement? Those things companies used to sign with their hosting providers, before all this IaaS stuff made developers feel like they could "start using now, talk to legal later"?
Interesting! For AWS, DO, Linode etc, I use a debit card with no problem.
Edit: just remembered we're using Google's Cloud Vision API and that's billed to the same debit card. So, for UK accounts at least, they take debit cards.
Maybe it's a regional thing? Most banks here (Ireland) issue debit cards by default and most people don't go out of their way to get a credit card, so they'd be cutting off a large chunk of their customer base by banning debit cards.
Are your debit cards really just debit cards (i.e. bank-specific "ATM" or "client" cards), or do they work using credit-card payments infrastructure? If you plugged your Irish bank debit card number into e.g. Stripe as a credit card number, would it take it?
Google stopped accepting my Debit card a year ago! Even after 14 months of calls / emails the issue is still here. I don't have other options rather than throw Google Cloud to the garbage bin (where actually it live).
For Google Cloud can't you set a billing limit for it to not exceed? That's what I have done but I have not come close to exceeeding it yet so IDK if there is something I am missing.
Should be fine I think providing there are no circumstances where Google can decide to ignore the limit (don't have much experience with their cloud stuff).
Ahh I think you are right. Just checked and you can set limits for App engine, but there are budgets in Google Cloud which is confusing and doesn't seem like it actually limits anything, it just keeps you notified of how much you are spending relative to your budget
You can. Beware though! It can take 24 hours to increase the limit - horrible if you get a sudden spike in traffic from, say, a techcrunch post or something.
In Sweden we have at least one bank that has e-cards where you can set up a new card for each transaction. You decide how long it is valid in months and for how much total and recieve a unique card number and ccv. I have been using this at least 15 years for payments online. You set it up, make the payment and then immediately close the card again if you want to be extra safe.
Unfortunately my bank (Nordea) removed this excellent feature and instead refer to having Visa 3D Secure verification, which basically only works in Sweden and is voluntary for the entity charging the credit card. Now I have to pay for two cards just to be able to do this.
Or at least they used to. I used to use it all the time on my Bank of America credit card but haven't used it in a few years. I just tried to use it last week and it seems like the option has disappeared. Maybe it's just bad web design, but I can't find it anywhere now.
This is excellent advice. In general, I've personally imported lessons about accounting controls I learned running a company to personal finance. Yes, it is a pain.
But Paypal, for instance, can only empty a bank account that is used for nothing else, any service like this (open-ended liabilities with shitty/no support) is isolated to debit cards, etc. Nobody gets permission to charge open-ended amounts to credit cards, and nobody gets automation rights to the "real" bank accounts, period.
There's a business model here somewhere for someone to automate all of this. It involves substantially more time to keep organized than my prior two-banks/two-credit cards model. Personal finance apps just aren't designed to treat, for instance, debit cards as transient.
As others note, sure, services that do something like this can still come after you. But who has the money in their pocket right now matters a lot when resolving disputes like this.
> They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options.
I don't understand why the larger internet/tech community keeps giving Google a pass on this. Anything where money comes in or out should ultimately have a support line where ultimately a human customer service agent can respond.
Is it because so many of Google's customers would then feel obligated to also offer support or service? While it's easy to pike at Google as a huge company which should provide this level of service, where do you draw the line? As I look across the collection of services I pay for, I don't see many of them with much more than "contact us" forms on sites, and in-app, no way to contact them at all.
Is there a certain size of vendor or price point where we expect to have human support and an SLA for response?
I don't know if Firebase has a reseller program, but for other cloud offers from Google, my advice is: if you want someone you can scream at, buy from a reseller instead of buying directly from Google.
Last week at thegoodfellas.com.br a disgruntled ex-employee managed to erase all their Gmail accounts and change the administrator account in order to prevent recovery. You can recover a deleted account in 5 days, but you need access to that administrator account. Unfortunately they could not recover access to the administrator account in time and lost all their G Suite data - you may blame them for their flawed firing process but shit happens and when it does a reseller can be fast enough to prevent a tragedy.
I administer a couple G Suite accounts for small business owner friends purchased direct from Google.
G Suite has incredible phone support. I've called 3-4 times over past 5 years and never waited more than 30 seconds to talk to a knowledgeable and capable tech who resolved the issue immediately. No BS troubleshooting steps, just straight to the heart of the issue.
I recall a couple Irish guys, one Eastern European and one Indian, but none with too-strong to parse accents (these were their locations -- relevant to make the point that Google clearly does have multiple live phone support centers around the world). All fully willing and capable of resolving the issue without passing me around. These weren't incredibly complex issues, but at least a couple were glitches requiring fixes on the backend (vs. mistakes on my part).
Has anyone with a G Suite account and a Cloud account ever tried contacting G Suite Support for the cloud side? Might be worth a try -- they've been helpful for me in one case where the issue wasn't strictly related to G Suite.
G Suite support from Google improved a lot, but for good reasons it is not easy to recover access to an administrator account that was modified to prevent recovery. In a case like this, how support personel can tell a legitimate customer from someone trying to gain access through social engineering?
A local reseller can pay you a visit and verify the situation but hands are tied for the friendly Kumar sitting at a helpdesk facility in India.
Here in Brazil it is even harder, because most customers are unable to communicate well in English and support in Portuguese is not done by native speakers.
> In a case like this, how support personel can tell a legitimate customer from someone trying to gain access through social engineering?
Pretty simple: opt-in KYC. Give people the option to email/fax you their passport or birth certificate or whatever, at any time after they set up their account but before the account is compromised. Extract the relevant ID numbers from the images; then store those numbers, encrypted, the same way you'd store "password recovery" info.
If the account is later compromised on their end, just ask the person attempting to do the recovery to send through the same stuff again, and compare with your recovery fields.
It's essentially the pseudo-biometric, pseudo-"something you have" equivalent of a Secret Question.
So naive. That's not how you get support in this day and age. You get support by kicking up a storm on social media.
Shout far and wide (twitter, facebook, reddit, etc) that [Big corporation] is screwing you. If you can give it some sort of spin - like racism or gender equality - all the better. Just kick up as much of a storm as you can.
I guarantee that [Big corporation] will be forced to respond.
(I say this semi-jokingly, but it is sad that this a) works, and b) that sometimes it is the only way to get their attention)
GCP sells support as a separate product. If you're a hobbyist, then that's great, as you can keep your costs lower.
If you're a business, there's no excuse not to pay Google for support. It's $150/mo; significantly higher than the $25/mo infrastructure bill the OP was discussing, but not unreasonable if you want to have a chance of guaranteeing QoS to your customers.
There NOT getting a pass ... I don't know anyone who is willing to bet their business on Google Cloud, but plenty who will do so with AWS and Azure. Google needs to replace whoever is running this section of the business with someone who can emulate what Amazon & Microsoft are doing.
Google needs to fix this kind of thing if they ever want to be a real threat to AWS. AWS has publicly stated their goal to only ever lower prices and do so continuously/periodically. This kind of lock in make people leery to even try a GC service or at the least build an abstraction layer to easily be able to swap.
The detail that amazed me is the lack of visibility of the issue in reporting tools - it's all very well changing the terms, but not giving your customer any ability to see what is increasing the bill seems massively unfair.
Exactly the reason why I decided not to pay for Firebase and instead went with gradually rolling my own solutions.
Their Realtime Database is extremely convenient and pretty useful for web apps, but there's no way to limit the connections and track bandwidth usage in real time. It's basically an open door for anyone with only your Security Rules to block data access. There's no logs to even see if your rules are working as expected, nor where your connections are coming from. They didn't even have that simple blue line showing basic usage in OP's post until recently. The profiling tool mentioned in the OP is also recent and doesn't give a proper picture.
It's just really pathetic. They sell themselves as a service company for startups, but one of the biggest things startups need to do is manage their costs, which Firebase doesn't provide in any form. And now since Google bought them, their support will be near non-existent as with all Google products.
Having said all that, their Free tier is great for live prototypes and doesn't require adding payment information. Just make sure you're not tying yourself in if you want to progress. Make thin wrappers around their libraries and export your data regularly if it's not throwaway.
"Welll, we have this there chalkboard that we write what we think your price is. And some intern draws lines on a graph to match what we think it is. Oh, proof? Nahh, we dont do that. Oh, logs? Nope, you'll just have to trust us..."
I use Firebase and find it easy, useful and constantly improving (cloud functions etc.). Their support, however, is a joke.
5 questions can be logged per year for one-on-one email support and then you're on your own. And this is for urgent requests.
I can understand not wanting to be inundated with wasteful questions but when Firebase is changing their API and pricing plans and you're trying to scale a product, and paying them lots of money in the process, support should be forthcoming.
They also don't respond on their Twitter handle, which is a further crime.
For technical questions Puf seems to be telepathically connected to StackOverflow. He has answered any of my questions within a an hour or two at the most.
Dan from Firebase here. I’d like to clarify a few details about the support options that are available. First, the limit of five questions per year applies only to technical troubleshooting questions (i.e. some indeterminate issue in your code that you’d like help with). Developers are always able to contact us an unlimited number of times for issues relating to identified bugs or to request features. Additionally, any questions relating to your account or bills (such as the issue in the original post) are also unlimited.
With that said, it’s been a year since we introduced the five-a-year limit on technical troubleshooting questions. I’ll see if this is actually a useful limit to keep in place and remove it if not.
Thanks for explanation, Dan. My issue is that the switch to the 3.x SDK and moving my assets to Firebase hosting depleted 4 of my 'technical troubleshooting questions' (two of those were bug reports which seem to no longer deduct from the five-a-year limit).
So now I'm looking at a message telling me that 'for all urgent requests...you have 1 question remaining' and hoping that nothing bad happens more than once between now and whenever you decide to allow me, a paying customer, to ask for help again.
To make clear: 99.9% of the time Firebase just works, which is great. And when I did receive support, it was excellent. It's the support model that's weak. Making billing and bug reports unlimited while limiting urgent requests to five a year is poor policy, and not very useful - for your customers at least.
Yes, I understand completely. The intent of the limit is obviously not to discourage legitimate issues from reaching our team. Limits like this are a blunt instrument, and it's possible that the time has come to remove it.
I can't promise a change immediately, but I am taking a very close look at what we can do here.
I authored the post and yes you are correct about the AWS Lambda being another service. I have since modified it based on a users suggestion to reflect.
I guess the point was that by adding the middleman between it there was more control --- We are self-funded and small group of people. We can't afford to hire a bunch of people to know about containers and servers and every little detail.
Thanks for your comments! Didn't expect so many views so quickly on this...
Yeah, no kidding. It means someone nefarious (or bored) can easily run up a huge bill by just directing a ton of invalid requests to their API.
Many inexpensive hosting providers provide you with a lot of bandwidth (e.g. Scaleway offers unlimited bandwidth at ~200MBit/s). Assuming you can saturate the link, that's around 64TB per month of traffic. Quite the Firebase bill they'd have.
well it's not like anybody has the business model of doing denial of service attacks that stop when the ransom is paid, because if someone did have that business model this seems like a great way to make sure people will pay you. I mean just do like enough to cost $500 and say pay me $1000 today or you will pay google $10000 tomorrow.
I'm guessing that spamming invalid request on all of firebases customers endpoints is the most efficient way to deal with the whole ordeal... If a large enough customer base gets the problem they'll have to act... Unless they can happily close up shop because Google owns them... Ah well :(
Wowow! In this position my first step would be to determine if there's any legal basis for disputing the charge (an informal conversation with a lawyer or even just someone who's been through a similar experience is a fine start). Even when a billing change is disclosed in advance and the customer actually uses the bandwidth, massive spikes in metered pricing are a tricky area and have attracted the attention of regulators in the past--for instance telecoms have come under fire by charging sky-high overage fees for mobile data and not warning you until you've already racked up a few extra zeroes worth of charges on your next bill.
Also, immediately dispute the charge with your credit card company, as you want this on their radar. Their interests are aligned with yours to at least some degree (if Google's causing financial distress for one of their customers with thousands of dollars in unanticipated charges, how many others may get hit?).
Aside from that, if things played out as described: major dick move, Google! Is "do no evil" a bad joke these days? Seriously, though, Google is not a dumb company and my guess is that if the right person was made aware of this case (e.g., anyone who understands that the most important word in "customer relationship" is the second one), they would make it right.
I'm totally on the side of the author: when possible, it's worth a little extra elbow grease to build your business on top of free, self-hosted software instead of proprietary SaaS. Think of it as insurance against cases like this one.
I would also doubt that it would be worth it even without keeping in mind the risk of loosing the service. Google is usually pretty clever when it comes to TOS Agreements.
"Don't be evil." was dropped in 2015 or thereabouts IIRC. I'd guess when they realised it was becoming more of a joke than a motto: "Google, the company who's motto is 'don't be evil' angered customers again today when ...
...
The new code of conduct has a close approximation of the philosophy—though perhaps more formally phrased—in the very first sentence of the preface: "Employees of Alphabet... should do the right thing – follow the law, act honorably, and treat each other with respect."
> In this position my first step would be to determine if there's any legal basis for disputing the charge
Let's calm down.
The first step is to pay the bill, set up better infrastructure monitoring on your end, so you don't act surprised that you are consuming 7000% more resources than third party service is reporting.
Then, if you want to burn some money, you can file a petition and tell the court that you deserve to get free service because you did not know that you have been going over the reported data by 50%, I mean 7000% for... two years.
As far as I understood, Firebase didn't report it before. As mentioned, the meter "suddenly jumped" to 100GB and then 180GB (if I remember the numbers correctly).
The author decided to figure it out with support later, which makes sense if the previous number was something like 20-30GB and you guess it's simply in error.
> telecoms have come under fire by charging sky-high overage fees for mobile data
Telecoms is a very heavily regulated industry.
> Also, immediately dispute the charge with your credit card company
No, don't do that. Most credit card companies require you to try and resolve the issue with the merchant before filing a chargeback. Sometimes they give you a time period, e.g. at least 30 days.
Google is famous for not having any humans in support. One robot will write abuse on you and another robot will happily oblige and ban you with no appellation possible. Building whole business on one of such platforms takes some "courage" :) .
I know - because I am one of those humans in support grins. I work in the support organisation for GSuite (formerly Google Apps). And I work across the hall from the guys who support GCP (Google Cloud Platform). And there are an entire army of us, helping support many, many enterprise customers.
Even our consumer products have real people manning support phones - I know, because I've had to call them before (mostly Google Play Music issues - this was before I joined Google, so I couldn't use my internal pull...lol).
Regarding your second point - abuse is a real problem for any cloud provider. I know, because I happen to work in anti-abuse. It's a difficult problem to crack, and there's a lot of smart people (at many companies) trying to solve it. But please understand there are real humans at the other end, trying to provide a good service - and it's not very kind to take cheap shots at them.
There usually are appeal processes available for nearly any kind of action taken - if you were actually personally affected by an issue, or feel unsure about how something was handled - please feel free to reach out to me. I can't guarantee I can wave a magic wand and fix it, but I'm certainly willing to reach out to see what can be done.
Disclaimer: I work for Google, but the views expressed here are purely my own.
Apparently time after time stories arise with Googlr lack of support and are very slow at answering ans helping. In this very damn case no freakin' response. Often people have to get a Googler to initiate a Hello wothin the organization. So while we respect you doing your day work, we think your organization as a whole fail at customer support. Yeah I have read sicceyss stories too but many of them seen to be businesses with really good pocket or have deep connection with Google already or just being lucky.
Your competitor AWS would refund in less than an hour with a billing support, without going through any phone calls. Your competitor has a support subscription service at least for people to purchase (expensive but worth it). Yours?
Often, these types of blog posts only present one side of the story - so I would caution a bit of patience, to see the full story. This applies no matter what company is at the other end - whether it's us, Amazon, Facebook etc. In this case, I know there's a few technical details missing - I won't comment further on this specific issue, as far more qualified people than me are already on this issue =).
Many times, I've seen it's due to a misunderstandings, or crossed-wires regarding technical issues.
And yes, our support team does provide refunds in many cases - I know, because these sometimes get escalated to me to approve (e.g. if they're borderline). I think one was actually offered already in this case, but there may have been a misunderstanding or mix-up around that.
And yes, we have a "paid" support service - it's actually comes free with any GSuite subscription (even at the most basic level, which I think is like $5 a month). You can chat, email, or phone. I work as an engineer in the team that supports this.
For GCP (our cloud offerings), the support offering is a paid offering:
I hired a freelancer from the Philippines on Upwork to work with me on some Google App Engine issues because Google's documentation is incredibly unhelpful at times and as well as bizarrely organized.
He was from the Philippines, and it turned out he was a contractor providing "support" for Google Cloud. So I guess Google is outsourcing their support? Isn't this something Dell tried to do in the 90's with disastrous results?
http://i.imgur.com/Gu8GzLB.png (If you click it, it will pop up this modal form, which also allows you to just screenshot the offending part).
Did you try using this link on any of the pages that you found unhelpful, or bizarrely organised?
I'd encourage you to use this - as all of this feedback comes through to us via an internal system, and it does get reviewed (although I'm sure you can imagine the volume).
Otherwise feel free to reach out to me if you have specific concerns and feel more comfortable doing it that way, or you're unsure about the response you received. (Although quite frankly, I think the form is best, as that gets tracked, and anybody can jump on it).
Actually a PM I reached out to a couple of months ago did get back to me. I was disappointed at first because I heard nothing from Google Next could be uploaded online.
That had been my complaint...that the documentation was hard to get and that when I saw the images presented at Next I understood things that I had previously given up on.
Synchronizing data in real-time is computationally expensive (this applies to both Operational Transform and Differential Synchronization approaches).
The price of Firebase was probably kept artificially low to attract new users but this was a ticking timebomb.
Those people/companies who are using Firebase as their primary database are going to suffer now.
It should really only be used for specific applications where datasync makes sense such as collaborative text editing.
I wouldn't even recommend using it for chat because it's really easy for the complexity to get out of hand there and you can usually achieve similar results using a combination of pub/sub and REST.
> Synchronizing data in real-time is computationally expensive
No, its not. I was responsible for implementing, deploying and managing the infrastructure at lever.co when we were a tiny fledgeling startup. The entire application is built on top of JSON OT. I took some measurements at one point when we had ~thousands of active browser sessions of our app. At the time we were seeing about 1-2 OT merges (transform + re-apply) per day. All the other non-concurrent operations can get sent straight to the database. I don't know what the numbers are now, but I'd be shocked if OT ever becomes the bottleneck.
For text based OT you'll see more concurrent edits. But for text based OT, I have a little C library that can comfortably do 20 million simple text OT transforms/second on a single core of my old 2012 macbook air. Good luck making that a bottleneck.
I guess it somewhat depends what kind of OT we're talking about (text or JSON). In any case, I don't think that the bottleneck would
be the OT algorithm itself - More likely, the bottleneck would be the number of messages (HTTP requests or WebSocket frames) required to send each individual operation between the client and server.
If you have text-based OT and you send an operation over the wire each time a user presses a key, and if you do this for every single input field in your app, it's going to add up.
Not saying it's not feasible but it's not going to be as fast as making a plain REST call for those use cases that don't require synching.
That said, I wouldn't be surprised if OT turns out to be much faster than differential transform in terms of raw algorithm speed but again I think the bottleneck will be the number of frames/requests that need to go over the wire.
> Not saying it's not feasible but it's not going to be as fast as making a plain REST call for those use cases that don't require synching.
The whole point of OT is that it lets you fearlessly merge concurrent edits. You can lean on that to rate limit messages to or from the client, if you need to. Because the client can always apply its own edits immediately to its own local model, there's no perceived latency. So, if you design your system right you can batch up changes at any granularity you like (per page, per form, per second, dynamically based on load, whatever). Of course, the tradeoff is that you lean on the OT system more by batching up edits. But it can be made effectively free to transform if the edits modify different parts of the database (using range trees to cull, etc).
But yeah - it is much faster than differential transform. I'm not sure how good diffing algorithms are in practice, but best case they need to scan the entire document. On the other hand OT only requires size & computation proportional to how much was changed. If we're collaboratively editing a 10kb text file, changes will mostly only be a few bytes each. And applying each change using a good rope library is a O(log n) operation.
Of course, all this requires a database which supports OT out of the box whistles innocently
Did a Firebase team member already comment on this? In a few hours Google will present some new Firebase features at io17, and I would expect them to avoid bad press now as much as possible...
I think it's kind of weird to see Google/Firebase team members swarming around in dozens on HN when they present a new feature, which seems to me like a coordinated marketing effort by Google. But once problems arise or they get criticized by their own users, no one seems to be available for a comment.
I was about to post the same thing. Of the two Google cloud stories I saw on here yesterday, it felt like one in three posts had "disclaimer: I work on the Google cloud team" or something to that effect. As of this writing, there's 100 comments over 4 hours and not a peep from any of them that I see.
Sometimes I am wondering who is actually steering the discussion to a certain point. I wouldn't go so far to call it censorship, but if the Google employees take over the comment section, it is very hard to call the flow of information "unbiased".
YCombinator shouldn't let companies use Hacker News as a marketing tool, at least not to the extent Google does.
I was thinking the same thing. There's a giant circle of patting each other on the back every time they roll something out, and always have that very precise disclaimer that they work for the big G.
Then again, as a meta comment I will point out that the fact that both you and I so clearly notice this suggest that we are following these updates much more closely then most of the folks here.
True, I used to be really interested in Firebase and their set of products. But because of several pretty bad experiences related to vendor lock-in and (very) high prices I switched over to an open source alternative.
It might not be so well known, but I found http://www.deepstream.io to have a relatively similar feature set, without having to deal with bad support (if it bothers you, just fix it yourself) and high prices, since you are the one who picks what suits you best.
Or it's just engineers that swarm around since that's the demographic here but this needs a response from PR, not from engineering. What do you want google/firebase engineers to say?
The engineering response is pretty clear-cut. Polling every minute from an uncached SSL connection is expensive, don't do that. That's pretty obvious and the author admits this was a mistake. The complaint is about everything that's not the engineering part - support & communication.
I'm sorry, I can have 50'000 devices poll every minute via an uncached SSL connection on my 16$/month server from Online.net, including bandwidth and traffic costs.
I'm not sure what you're smoking, but I want that, too.
Seriously, at Google, TLS is terminated in hardware, and bandwidth costs less than a cent per terabyte for ISPs or Google.
The costs that this caused are basically nil.
(This, btw, is why I'm using kubernetes on rented bare metal, paying 30$ a month, for what would cost me, including traffic, around 900$ (Google Cloud with rented servers) to 63'000$ (Firebase) a month.)
I would assume you get the support if you pay for it. Google, Azure and AWS seem to have all gone the same way. You select the service level and pay monthly fee. I don't know if this support covers Firebase.
Yes and no. If an AWS customer ran up a bill like this, Amazon would absolutely cover them. They've done it hundreds of times. There's a very different culture between Google and Amazon when it comes to customer service.
We've also gone from ~$25/mo to $1000-2000/mo with the pricing change. While it's not the end of the world (we're near shipping replacements for FB stuff anyway), there's also been 3-4 incidents of downtime (> 1 hour) in the last 2 months, with no way to contact anyone except their stupid web form.
It all depends how dependant customers on a product. At one point I worked for a company that successfully charged customers for fixing obvious bugs in a software on top of expensive support contructs. The sad part it worked for some time.
I think there's a major difference between say, support figuring out how to implement their product, which is a you problem, their service is working fine.
And a wholly different one if they screw up and you can't talk to them about it without paying them.
The former is an incentive to self-serve and investigate yourself, the latter is an incentive for the company to provide shoddy service to people pay to resolve it.
Tiered support contracts are common in enterprise software. If the base level of service were bad, nobody would use the software, and that's clearly not the case here.
(More common is the Oracle/IBM-style approach of making your docs vague enough that you need professional services to actually integrate the product).
A free level of support that requires answering within of 14 days to any issues via email, letter, or phone, and providing an address that a letter can be sent to, is required by law in the entire EU for any paid services.
No - how do you opt in? Can't find it on any of the billing or pricing pages, and can only find the message "we do not currently offer Enterprise contracts, pricing, or support".
That said, we'd only need support to get ETAs on outages being resolved ><
This is nothing new to me. I'm still working on my GAE/datastore migration...Price spikes, platform changes and deprecation are real risks when you use proprietary services/platforms.
That's the same, old vendor lock-in approach. It has happened with the servers, databases, network equipment etc. now it happens with cloud -- nothing unexpected, prepare to see more of these when cloud market stabilizes and clear leaders emerge. That's why I always prefer open solutions, that can be moved/migrated/replaced easily and advise others to do the same.
Eh, it looks like the service provider hadn't taken into account an oddball use case in the pricing / metering, and then suddenly did.
They certainly should have communicated that better and their tools / charts should have a way of showing the billable data, but it's a bit of a stretch to call it vendor lock-in. It's not like everyone on Firebase is complaining and the author admits they made major mistakes.
Just "communicating better" is not a reasonable response to making such a dramatic change in how much money you charge someone. If I tell you I'm going to punch you in the face, that doesn't make the punch acceptable behaviour.
Cloud services that pull this kind of bait and switch deserve to be ridiculed and lose lots of business over the negative PR. Cloud services that unintentionally pull this kind of bait and switch but then don't make it right or provide meaningful support deserve no special treatment.
From the GGP and some of your other comments here, it seems like you are arguing that communication is the main problem here rather than the change itself. Sorry if that was a misunderstanding.
But why do providers get a free pass when they do a major mistake (à la GitLab losing tons of not-backed-up data), but customers are required to accept whatever the provider throws at them? It seems pretty unfair to me.
I don't think they should get a free pass, but I don't think they should be beholden forever to be under-metering traffic if they made a mistake and weren't counting part of it (as it seems to be the case here).
To that extent, they should have communicated this better, ideally 90 days out perhaps even with phone calls for customers whose bill will increase notably.
Providers to some degree get a free pass when they do their best to fix their mistakes.
In Gitlab's case, their response was excellent and their total data loss was less than a day's work - and even that only for a full time manipulator of gitlab metadata (project managers? some parts of QA?): "Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately." ( https://about.gitlab.com/2017/02/10/postmortem-of-database-o... )
Considering that "[...] out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." ( https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx... ), this was a pretty amazing result, and about the best a response they could make. That, combined with their transparency, earns something of a pass (especially when data loss I care about is already limited by git's DVCS nature, even if they had obliterated the repository data.)
In Firelab's case, if no other action is taken, they've actively mislead their customer in failing to follow through on crediting their account, charged them an obscene and unsupported price hike, and gone incommunicado. That's like... an anti-pass.
Chances are they also knew when they fixed the bug that it would result in price hikes. They could have easily generated a report showing which customers would be impacted, allowing them to contact them ahead of time.
I'm not implying that this was made on purpose -- vendor lock-in might happen "naturally" as well, when company becomes big enough that they can start unilaterally change prices/conditions/services knowing that costumers will accept that (even grudgingly), because they have nowhere else to go or it is too expensive/complex. Google is the perfect example of this strategy. Smaller player or startup would probably go bankrupt very fast if acted with such disregard for their costumers.
So far I've seen only one customer notably affected, and (judging by the post) I doubt they'll be sticking around on Firebase. There are a lot of take-aways for this story, but I don't think "vendor lock-in" really fits.
515 comments
[ 3.3 ms ] story [ 346 ms ] threadStuff like that makes me want to host everything myself as it will for sure be the cheapest option.
Building on top of a proprietary stack with no guarantees on pricing and future availability will lead here.
So you either have to code a portability layer so you can move to another one in the event of an unwanted change (which could be quite a lot of work), or you accept the risk that a 3rd party could cost you a lot of money until you can change off their service (which could be quite technically challenging depending on the complexity of your use case and the availability of comparable alternatives)
Whilst with things like container services you retain more control (ultimately you can self-host if need be) as soon as you step onto pure serverless you'll always be at the mercy of the providers of the services you use.
E.g. shutdown Prediction APIs, charge 70x, not enough support.
I believe Google got much better tech than AWS, but the culture, support and strategy has blind spots.
I get the impression Google is doing a lot of things different for the sake of doing things differently. AWS is much more pragmatic and transparent.
First, it's important to understand unless you're buying ads from Google, or renting their machines, you are not their customer. Users are inventory and they spend a lot of money trying to procure inventory (users) for their customers (advertisers).
When you look at it through this lens, you can see how Google as an institution can tolerate experimenting with ideas that are disconnected from revenue models and dropping failed ideas like hot potatoes. It makes sense for them to behave that way.
Amazon's services are almost always designed with either a concrete or direct path to revenue upfront and they have a huge customer support infrastructure they can tap into. They move a little slower in certain areas of R&D because customer obsessed businesses are tolerant of deprecating technology when it doesn't work out. That means they'll do more market research, take less risks and focus on tenaciously steady growth rather than trying to always hit it out of the park.
When choosing a vendor, it's important to understand how their offerings affect the solvency of their business and whether it fits into their core competency.
In this case, Firebase's prices were too low. That should have been a warning sign. The other warning sign is Firebase is a such a small percentage of Google's revenue, Google doesn't have a huge incentive to fix customer service issues.
If Pizza Hut changes the pricing, you either close shop or adapt.
Who would open such a restaurant?
Also, I do love Bezos mantra of always becoming cheaper over time. That is the core of Amazon's brand. Whereas Oracle, IBM, and the old guard always make you pot committed and charge an arm and a leg, the Amazon guys will leave money on the table. That is how you earn trust.
What? they're a business like any other and they're not interested in minimizing profit, there's lot of thought that goes into their pricing and you can be sure that your opinion and trust is an outcome of their strategy to make more money.
Is exactly what the parent comment said...
I think Amazon pulls this off because their prima donna is the paying customer, not the programmer who thinks talking directly to customers is the worst possible reflection of their ability to automate. Automation is great and all, but Google has a major problem when it comes to empathizing with the customers.
Also, is there any possibility of legal action against Google for this bait and switch, which seems to be quite clearly the case here?
Of course, they are a business. But they look at minimising the price, increase the scale, have a monopoly and earn money.
Actually they are. Minimizing profit is sacrificing short-term gains for long-term gains. Increasing prices helps your short term profit, but you tend to lose customers to competitors. Decreasing prices means less profit right now, but you tend to convert more customers.
Companies aren't purely RIGHT NOW profit-driven. That's why many companies burn cash and take on debt. They're driven by growth (whether vertical or horizontal). Bezos is obsessed with growth. That's why they've only recently started making money. Enormous investments in AWS, FBA and a host of other products put them in the red, but in doing so, Amazon has incredibly horizontal growth.
> Minimizing profit is sacrificing short-term gains for long-term gains.
That would actually be maximizing profit...
The timeframe dictates behavior. But, you responded to a comment that said Amazon will leave money on the table, arguing that they won't. They will, and do, leave money on the short-term in order to benefit their long-term strategy.
Short-term vs Long-term profit motivations are actually MASSIVELY different sets of behaviors.
> That would actually be maximizing profit...
Yes, if you assume a company is infinite that's true. But since there's currently not a company that's existed forever, I can safely assume that such a thing doesn't exist.
You can't argue profit without arguing time. Particularly when your argument was that all companies focus on profit at the exclusion of all other things. Growth companies focus on growth over profit. And while, yes, over time that can result in more profit, it also may not. If pure profit were the only motivation, short-term MASSIVE profits would be the only goal ever, since it's vastly more real than potential profits if everything works out.
Charging less to make more is actually maximizing profit for the company.
16Mb is modulo zero these days, but it's enough for him - he just deletes some emails when it gets full. He also gets imap access and FastMail's spam filtering, which is really very good. I can also assign him an email address from a domain that I own that is associated with my FastMail enhanced account.
I don't want my $15 back. What I DO want is for FastMail to honour our existing agreement, which was a LIFE-TIME 16 Mb member account in exchange for $15.
See also here:
http://www.emaildiscussions.com/showthread.php?p=599512
Text of their latest email is below the line:
=============================================
Back in January we contacted you with some important information about your @.* FastMail account.
We're now contacting you again to remind you that from 31st July, 2017 our 'Member' level plans will be discontinued.
To continue using your FastMail account after this time you'll need to select one of our current plans. Please note: your email address will remain the same.
As a long-time FastMail user we're offering you 50% OFF all upgrades until the end of July, which includes our already discounted multi-year subscriptions.
And if you upgrade now we'll also add all the remaining time until 31 July for free!
As an added bonus we've also given you $15 credit to put towards your subscription, which is equal to the amount you originally paid for your account.
This means if you were to upgrade to a Basic account today for one year, with the discount and account credit you are effectively getting FastMail at no charge until 31 July 2018 with a massive 2 GB of storage (up from 16 MB)!
But then it's not my problem.
FastMail made an agreement - $15 in exchange for a lifetime member account.
Now they don't want to honour that agreement.
And they've created work for me: What to do about my father's email account?
I have his email address as part of the domain that I have associated with my (legacy) FastMail enhanced account. They've discontinued family accounts for new signups.
https://news.ycombinator.com/item?id=14357859
I'm not sure whether my chosen title follows HN guidelines - admins please feel free to edit
I've had nothing but excellent, and personal, service from them, for years. Very high availability, no canned responses.
Even when bitching about Google's support of "this address used to be with Gmail but is now external, but we'll make it problematic for other users to send it calendar invites", though in no way did I imply Fastmail was at fault, they jumped into the HN thread to say "Hey, let us know if we can be of assistance figuring out the issue".
I've been very happy with Fastmail for over a decade and this seems like one of the few missteps they've made. New "lifetime" 16MB accounts have not been available in years. It really doesn't seem that there could be very many of them still in active use that it would hurt anything to just continue to grandfather them, especially since they were sold explicitly as "lifetime" one-payment accounts.
Curious - how could FastMail ever properly close/sunset these accounts? Were they going to regularly check on all of their customers to see if they had passed away? Was there some 'we close the email if you don't log in for 5 years straight'?
Serious question - how should these type of lifetime promises ever be properly managed? (especially when the customer is an individual)
https://news.ycombinator.com/item?id=14357859
My point is simply this:
FastMail unwisely offered lifetime email service for one-time $15 payment. I accepted that offer on behalf of my father.
FastMail should now honour that agreement.
They will go bankrupt sooner or later, it's only a matter of when.
It is not so hard to predict failure/buy out of some SAAS provider, it happened so many times. I am curious why Venture Capital investors are not looking at that problem closely.
Nothing in the entire VC Funded startup scene makes sense when you try to apply basic logic, much less a sane way to operate a sustainable business.
But that's just it: These companies are funded by people who are simply in it to make a profit, (as opposed to people who want to actually achieve something with their efforts). Users become dollar signs, and it's all about how quickly you can get enough new users to monetise the bejeezus out of their attention/personal information/etc, before they dump you for the next new flavour of the week.
Yes but you gotta be smart here. It's not like Firebase is the only database out there. And once your product grows, and you hit Firebase limits which forces you in the "pay as you go" plan, whatever cheapness you thought you bought into quickly becomes a huge expanse.
-installing MySQL
-Installing Apache
-installing PHP MyAdmin
-running some commands from shell and SQL
-done.
If you create a startup it is wise to know the basics as such and in case that you grow to hire people to pimp it up.
And than you don't have any control over your infrastructure as well because someone else has theknowledge not you.
I don't see any advantages doing this as long as you are small. Those few hundred dollars are better investet in simple SAAS.
> Always build your architecture in a way that will avoid becoming trapped into a specific service. Amazon’s AWS Lambda sitting between any services and your app is a strongly recommended path!
- Build without depending on external services
- You can do this by depending on this external service
Isn't using AWS Lamba as your gateway, trapping you into using a specific service, just as much as firebase was?
Since you control the domain, you could then stop something like this more quickly.
But, either way I agree. You should always have all requests to any services going through your own domain.
> They have no phone number to contact, no way to dispute this other than email — which they have ignored us for over a month now without replying to our continued requests. Trapped. Doomed. We have no further options.
Sounds like Google did a great job with Firebase. Definitely gives you that Google-feel of unresponsive support and no hope when you have problems with any of their services. I bet they also have a "community" support forum or Google Group full of helpless people talking to a virtual wall. (Edit: Yep, they recommend Stack Overflow, Quora, and then their Google Group.. lol. Some things never change)
"What do you mean, you retain a running instance of every previous instance I pushed?!"
Fun times.
What? Seriously? Is this documented anywhere?
Would you recommend against using GAE?
I've been burned so hard by them, I'd recommend against using anything Google.
It was more of a mental "but but my decade-long history of gmail" thing, but it turned out that really was less of a big deal than I thought. I now have a nice custom domain and I forward my gmail to my fastmail, updating subscriptions every so often. Eventually gmail will receive no more mail I care about and I will remove the forwarding altogether. Its getting there!
Looking back on it now, about a year after migrating, my only regret is I didn't do it sooner.
If you aren't a FastMail customer yet, you're gonna assume we're shills. We're not, it's just the sort of quality product that makes you want to tell others. It's like having found Gmail in 2005.
No angels anywhere.
[0] https://serverfault.com/questions/218615/mail-server-checkli...
I'm not a big account with them, surely trivial compared to those mass domain parkers you mention, but they've never made me feel like I was less valued as a customer. I have received calls from humans just to ask me if I am currently satisfied with my service with them.
Especially if you have a business-critical service, this should simply be a baseline requirement for support, but it's becoming shockingly rare these days.
SPF is essentially a self-maintained whitelist.
Blacklists are public and searchable. Something to check after dealing with an infected machine, for example.
Ironically, commercial spammers have the best SPF records. And DKIM. And all the other goodies.
My google usage is now
1. Occasionally watching Youtube videos in an incognito window
2. Very rarely using !g to search Google via DDG
3. Using Gmaps in an incognito window a couple times a month
and that's it. I don't find that I miss it at all. If I had to stop using Youtube and Gmaps, I won't miss 'em.
Now, if only good and valuable startups weren't getting bought by google by the dozens :( I guess it's your cue to start migrating to something in house when it happens.
So, for some applications, GCP works very well.
Yes, for that reason a POC I made for a side project costed almost 100 bucks to me (I was the single user)... I never checked the current bill because I was sure the total cost would have been around 15-20... when I got charged I noticed all the previous pushed versions had a single instance running, so the costs were incremental...
It may be my fault for not adequately reading all their caveats, but it seems to me that the default experience shouldn't be like that.
I love Google, but their Cloud Platform is still incomprehensible to me.
Support was somebody in a different timezone who complained when I didn't answer my phone in the middle of the night. I quite liked the Google Cloud offering, but to get so burnt so quickly... stay away.
We don't want to be calling customers in the middle of their night while they are sleeping. Its not an effective way to resolve issues. We need help identifying when we fall short so we can fix the issues.
Ways to let us know: 1) fill out the survey when your case is closed. We review every survey but less then 1% of cases get the survey filled out. 2) tweet your case id at us and it also goes through our review process 3) email me (tsg@Google.com) and I can correct the problems. 4) GCP slack community has a whole bunch of active Googlers (and the community will flag one us down) if you are having problems with the support team.
Very typical from Google.
Unfortunately Google as a company and their corporate practices are nothing like that.
But it's pretty easy to look a bit into the future, and predict what we would see during "Extend and Extinguish" phases of TensorFlow project.
This is Google support summed up perfectly in a single sentence. It really is appalling.
Never quite understood what's "appalling" about it. They've never had strong support for any dev/cloud-oriented services. I know some folks who've had good experiences, but those were exceptions, not the rule.
I'll say, I get the frustration with it, but it's why I've never considered using their cloud services for anything. I had a client who asked me to migrate all their stuff to google 'cloud compute' and their 'sql cloud' offering last year. It was a performance disaster; queries that were minor .1ms on localhost or in the same network were suddenly 18-20ms hitting google cloud. Spent a few days trying to dig in, and a few responses from various groups were "shouldn't be that slow" and "works for me" and that was about it. Gave up trying to get any real response from google (we may have got a 'looking in to it' response - I don't remember).
Thing was, I knew going in it wasn't going to end well, I just wasn't sure exactly why (maybe that's just too cynical of a view, but I prefer 'realistic').
You know this by
a) hundreds and thousands of horror stories from people saying "I did not get any support from google". You can counter that with "here are people who did get support", but the existence of the thousands of support nightmare stories out there gives you enough evidence to know that this isn't really "supported" in the way you might expect.
b) there are no phone numbers. https://firebase.google.com/support/ - no phone numbers. That's, of course, not the sole criteria you should use to judge potential support, but... it's google.
It's 2017. We have multiple years of knowing what Google's track record is re: support for paying customers. Outside of large adwords advertisers (where they make a lot of money), it's non-existent from a practical standpoint.
The big problem here is that firebase was fine until google acquired them.
I'm reminded of a time when I had a bad experience at a restaurant. Local place I went to with friends from work. We went... probably 4-5 times per year, and I loved the chicken sandwich they did. Last time I went, it was ... bad. Not what I'd expected at all. It was different.
Server asks "how's everything?". I said I was disappointed that the chicken sandwich had changed - it was not as good as before. Now... I wasn't actually complaining as in "comp the meal", but I was not a satisfied diner.
"Nothing's changed sir".
"um... yeah - this is nowhere near as good as it used to be".
"You're wrong, nothing's changed".
We had a back and forth, and they went back and came out "the cooks say the recipe is exactly the same".
This was getting a bit ridiculous. Manager came over, same exchange. He leaves, the comes back and says "they're right - the recipe is 100% the same. We have changed a few ingredients last month, but it's the same".
Maybe that doesn't actually relate to the google service situation at all now, come to think of it, but I was reminded of that situation when I started typing it :)
Oh yeah - nothing changed from OP's end, except how the new owner (google) decided to start monitoring and accounting for usage. Think of a "$9.99 all you can eat buffet" getting loyal diners, then changing owners, and the new owners giving some people a $95 bill at the end of the meal because they'd decided to also charge for any visits to the salad bar over 2. Most diners would never hit that limit, but a few - loyal diners for years - would have been (rightly) upset at the after-the-fact change.
I know this but to say everyone does or should isn't exactly fair. As well, you assume I am holding the purse strings, that I'm in any way responsible or even have a say in procurement. This may be true in my own business (it is) but not necessarily elsewhere, like my clients (it more often than not isn't) and you'll just have to live with poor decisions made on your behalf.
Also quite often, there's a sunk cost that makes switching that much harder. So no, it's not fair to say "Google has shitty support but that's ok because we all kind of know this." Paying customers can and should complain about this, loudly, shouting it from the roof tops. Ideally they vote with their wallets and just leave Google, but again that's just easier said than done in many cases.
Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less.
That's the ONLY thing that will ever make this change, but it rarely happens because "sunk cost" rationalizations (and "maybe it'll get better next year!").
Shouting from the rooftops for years has not worked. This is not anywhere near a 'new' problem.
Yes, possibly some people don't know it, and yeah, I totally get that "you'll just have to live with poor decisions made on your behalf.".
I live with poor decisions all the time (mine and others'). But the notion that we can bitch about it on forums, or make 'formal' requests for better support - none of that makes a lick of difference to them (or, demonstrably hasn't for several years). It's "google", so some people will flock to it because of the brand, or because it's not azure, or aws, or whatever. Or to tick the 'cloud' box on their spreadsheet. Or because google gave them thousands in free credits to switch.
"Some semblance of professionalism shouldn't be too much to ask from one of the world's biggest corporations, an ask coming from paying customers no less."
But... it demonstrably is, and has been.
And... you'll get a client who will say "I can't believe this, email someone, get this fixed". At some point you have to push back and say "no, this is simply not fixable, you have to live with XYZ, or pay the cost of moving somewhere else".
An extremely large customer, maybe, possibly, might have a bit of say, perhaps. But even if you're dropping millions with them... for a company that measures revenue in hundreds of millions of dollars per day, you probably won't hold much sway. If you had that sort of budget to spend, and it was that critical, you'd likely be doing enough due diligence that research in to actual support/service levels (vs what the sales contact tells you) would be on the agenda.
Vote with the wallet is the only viable/impactful approach, and people have to be willing to forgo the sunk costs.
And for people that don't know up front - they always had the opportunity to search (google) beforehand. If they choose not to - caveat emptor.
10 years ago, we could possibly be forgiven for choosing google's services (GAE, etc) because there wasn't as much of a track record to look at re: support, etc. There's no good excuse for starting a project today on google compute and claiming ignorance of their support levels.
If I've misrepresented what you're trying to say here please do correct me, because I clearly don't get it. If I haven't, I suppose we'll just have to agree to disagree – I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere.
But to avoid it, you generally do research ahead of time. And/or be willing to actually walk away from your 'investment' on their platform and start again somewhere else.
In the OP case, they didn't sign up with google in the first place. It's doubly shitty for them, because they didn't ask for this, and could not have known in advance that google would end up being the support provider. Similar to when I refinanced a house, and 6 month later, my 'servicer' sold my loan to someone else with much crappier service (charge me $3/month to take an electronic payment from me? I wouldn't have chosen this as a service option).
Yes, paying customers should go elsewhere. Complaining about it to google directly and indirectly for a decade has not noticeable improved their support for their cloud offerings.
Suck it up or go elsewhere. You can keep registering complaints with them, or organize digital protests, or whatever, but to get better service, you will likely have to go somewhere else.
"I think customers should shout from the rooftops till they're blue if they want to, even if it seems to lead nowhere". Of course they can. They just need to realize it will not change the quality of support. Someone earlier (you?) said to 'vote with your wallet'. Shouting from the digital rooftops about how crappy service X is, but continuing to pay them, is a waste of time at best, and enabling at worst. Leaving for another service is voting with your wallet/budget, and it needs to happen.
In the very short term, agreed - it doesn't do much good. But shouting from the rooftops internally (at client, management, stakeholders, etc) may do more good in the medium term.
https://xkcd.com/1053/
Also, from the article, Firebase wasn't google when they went in, plus it was a side project that suddenly took off, not a thought-out business plan.
And yes, on the comic - ok, not everyone "knows" everything up front. But... you have search engines (google, even!) to do actual research beforehand.
I see small business folks and small consultancies do far more research on comparing various $7/month hosting plans via hosting review sites than I see people researching their decisions to use AWS vs Google vs Azure.
1. Things to work 2. People to answer when things don't work 3. Things to start working again in a timely manner, with timely and concise updates along the way
Points two and three carry a lot of nuance, but Google's support does not. It's a virtual wall that people cry to and get no response back from, except for the rare few occasions when the Google deities decide to grace forums or mailing lists with their presence. It's just not a serious offering.
Now, you can pay extra for support, but even so it's going to be appalling. You probably will get to talk to someone at least, but in my experience you rarely get ahold of anyone who'll actually read what you've sent them, much less actually try to help.
My take away is that support and customer service is just not a priority for Google, what so ever. It's not even a low priority thing, it's a no priority thing.
To be honest, if all of my Internet requests were 18 ms slower I wouldn't even notice.
if I read you correctly and it wasn't a typo, could you mention why you cared so much? The .1 ms you quoted on localhost is extraordinarily quick and I wouldn't expect it to translate to a cloud...
To (re?)clarify - this was 18ms per query to the SQL cloud (from a google compute engine).
Yes, were there things that could be done to reduce that? Of course. The other guy on the project put some caching in, reduced duplicate DB calls, etc, and got it down to around 3-4 seconds, but without severely diving in and learning far more about this inherited codebase, there wasn't much that could be done to make it much faster.
The other steps involved heavier caching, which changed the actual functionality of the service that was contracted before. "real time status updates" had a functional definition of being within X seconds, IIRC. Caching everything so 'new' information wasn't necessarily available to an end user for, say, 3 minutes, wasn't an option, without getting a lot of buy-in from a lot of people. Not saying it wouldn't work technically, but the client relationship was already strained.
The move to google was primarily to 'save money' (because someone had $10k service credit with google), but the effort to make it work well on google cloud and SQL combo (wasn't my call) was too much.
You can say this isn't google's fault entirely - you'd probably have a strong case - but not everyone is moving pristine clean/strong/good code over, and problems with the original base were amplified by seemingly small issues.
I hope you don't think I'm blaming your code base, but just out of pure curiosity, as an amateur question could you mention why the SQL calls couldn't be done in parallel -- did each one depend on the next or something?
At the high level I get everything you just said, just curious about this point, if you remember.
The above reasons are also why I don't use any open source software developed or maintained solely by Google. There's no guarantee that Google won't just stop maintaining it after a year or less. For a somewhat recent project, my co-founder and I evaluated using Angular2 for the front-end of our application. During our evaluation, we learned about Angular4. Knowing about what happened between Angular1 and Angular2, we decided to steer clear of the angular ecosystem. This decision also made more sense from the performance and ecosystem side of the argument. Most groups that develop open source software do their best to ensure that there is a well documented and simple upgrade path. Look at Django, rails, React, Vue, postgresql, etc for example. As far as I have experienced, Google products offer no such simple upgrade path.
To be honest, I'm not really sure what value Google is offering anymore. The only quality product from them I still find any use for is Gmail. Even Gmail has competitors that do Email better. I guess there's also youtube, but Google are kind of shitting that up as well.
I think the tech space needs to walk away from worshiping Google and Facebook. In all reality, the technical quality of their products is lacking and their entire goal is to lock both consumers and developers inside their own walled gardens.
But this is a somewhat easier choice to make because I still have the code locally. Google could stop supporting it tomorrow, and it would not be fun, but things wouldn't stop working, unlike the firebase situation above, where the option is "turn off your app".
That said, I've generally been using knockout and vue (more vue recently than knockout) and both have been fine.
"Third, they don't have any actual support at all for anything". There's a lot of people on this thread that feel like because they're paying for something they should get support. And... while I'm sympathetic to that viewpoint, and agree they should, we've had 10+ years of seeing google consistently under- or non- support many services that people seem to want to base their entire businesses on.
edit: Meant, even if you lose you have cost them significantly an that has the intended effect of signaling they should improve customer service.
http://m.huffpost.com/us/entry/213176
For a funnier variant, https://lmgtfy.com/
Worst case, if AWS or Google decide to fuck you over, let the bill bounce. This way you've still got funds on hand to deal with the fall out and some leverage to negotiate a solution.
After all, they're just being helpful in managing your money and paying your suppliers.
I'd advise not using debit cards for anything, under any circumstances, and relying on the better protections that credit cards provide, but it's a losing battle. My debit card is ATM-only.
This is generally a good approach for all sorts of reasons, not just protection from over enthusiastic cloud bills.
Keep most of your funds in one bank (and never use the cards). Transfer from this account to one or more other accounts with other banks and use the cards on these secondary accounts to pay for things. This way you're protected from suppliers over billing and banks going crazy with unarranged overdrafts (which aren't secured so absolute worst case you just tell them to fuck off and default on it).
Basically, the trick is holding on to your cash. Much harder to fix things rapidly if you're having to negotiate with a third party between you and the supplier (which is the case with a credit card company).
I thought what they were doing is applying all your debits, charging you if that puts you into overdraft, and then applying any credits. If they applied the credits first, you wouldn't go below zero.
Either way, you're trying to charge $46 against an account with $20, but one method results in 1 fee and one method results in 3 fees. The issue of credits vs debits you mentioned can compound this issue.
Consider a bank account with $20 in it, the day before payday, when you forget your World of Warcraft subscription hits today instead of tomorrow. That's $15. But you buy:
Then, the subscription hits at midnight.If the transactions happened in-order, the subscription causes the overdraft, and you're charged $25 or whatever (yeah they are very high). But in the way banks used to do it, they'd take your total ledger for the day, and order it by highest $ first. So, subscription first, $5 left. Lunch, $0 left. Coffee, dinner, and snack are three transactions when overdrawn, so that's $25 * 3 = $75 in fees instead.
Banks claimed that this process was because you'd probably want your highest-amounts paid first, as it'd be something like rent money. In the end, it means way more fees for them, though.
$75 in fees is especially brutal when you don't have much money in the first place; it pretty much caused me to not be able to even go to 7-11 for a candy bar for two months.
I only wish setting up virtual cards were as easy as setting up a new mail box. I know there are services out there that lets you do this, but I've yet to find one that works globally and is easy to work with.
The second bank account is really more for protecting you against black swan events. If all your accounts are under the same bank and have the same owner, there's nothing stopping the bank taking funds from one of your accounts to make up for shortfalls in another.
I'm wary of giving my card information to companies outside India because I don't have this protection.
True for Indian companies but what if you get charged via Forex abroad? For me - Google or Steam - don't required 2FA. All I have to do is to enter my number, confirm CVV then zap. It gets debited. I do get a call from my bank (HDFC) asking if this transaction has been initiated by me, if I say yes, they let go.
The Flash UI looks like this http://mayoyo.tokyo/jJ7.png
They used to have a mobile app but it was discontinued since it didn't support 2FA.
The only reason I don't use them for everything is because I get better cashback/points with other cards.
> Is Emburse available outside of the U.S.?
> Only companies incorporated within the U.S. may register for an Emburse account. In addition, the primary account owner must be a resident of the United States with a Social Security number and a verifiable physical U.S. street address (no P.O. Boxes). We do not offer Emburse accounts to residents of Puerto Rico, Guam, the U.S. Virgin Islands, American Samoa, and other U.S. territories.
[1]: https://www.emburse.com/faq
> What countries are you available in?
> We're only available in the United States at the moment.
[1]: https://privacy.com/faq
EDIT: I have no affiliation with privacy.com; I'm just a satisfied user.
The problem (in the article at least) isn't being surreptitiously billed a large amount, it is cost of a critical service changing dramatically without warning.
Also, nothing gets them talking to you faster than a bounced invoice :)
It's when a big bill lands in error or is introduced sneakily that having a separate "cloud only card" can save you. In that case, it's the supplier that's gone nuclear and you're just responding in kind.
Sorry, but this is the risk of using somebody else's service -- especially Google's -- as a critical component of a business. It may be marvellous for developers not to bother with infrastructure for their own service, but this strategy likes to come back and bite in the ass.
Edit: just remembered we're using Google's Cloud Vision API and that's billed to the same debit card. So, for UK accounts at least, they take debit cards.
https://privacy.com
1. https://getfinal.com
But Paypal, for instance, can only empty a bank account that is used for nothing else, any service like this (open-ended liabilities with shitty/no support) is isolated to debit cards, etc. Nobody gets permission to charge open-ended amounts to credit cards, and nobody gets automation rights to the "real" bank accounts, period.
There's a business model here somewhere for someone to automate all of this. It involves substantially more time to keep organized than my prior two-banks/two-credit cards model. Personal finance apps just aren't designed to treat, for instance, debit cards as transient.
As others note, sure, services that do something like this can still come after you. But who has the money in their pocket right now matters a lot when resolving disputes like this.
I don't understand why the larger internet/tech community keeps giving Google a pass on this. Anything where money comes in or out should ultimately have a support line where ultimately a human customer service agent can respond.
Is there a certain size of vendor or price point where we expect to have human support and an SLA for response?
Yes, when they control the switch that can kill your business.
Last week at thegoodfellas.com.br a disgruntled ex-employee managed to erase all their Gmail accounts and change the administrator account in order to prevent recovery. You can recover a deleted account in 5 days, but you need access to that administrator account. Unfortunately they could not recover access to the administrator account in time and lost all their G Suite data - you may blame them for their flawed firing process but shit happens and when it does a reseller can be fast enough to prevent a tragedy.
Disclaimer: I run a G Suite reseller operation
G Suite has incredible phone support. I've called 3-4 times over past 5 years and never waited more than 30 seconds to talk to a knowledgeable and capable tech who resolved the issue immediately. No BS troubleshooting steps, just straight to the heart of the issue.
I recall a couple Irish guys, one Eastern European and one Indian, but none with too-strong to parse accents (these were their locations -- relevant to make the point that Google clearly does have multiple live phone support centers around the world). All fully willing and capable of resolving the issue without passing me around. These weren't incredibly complex issues, but at least a couple were glitches requiring fixes on the backend (vs. mistakes on my part).
Has anyone with a G Suite account and a Cloud account ever tried contacting G Suite Support for the cloud side? Might be worth a try -- they've been helpful for me in one case where the issue wasn't strictly related to G Suite.
A local reseller can pay you a visit and verify the situation but hands are tied for the friendly Kumar sitting at a helpdesk facility in India.
Here in Brazil it is even harder, because most customers are unable to communicate well in English and support in Portuguese is not done by native speakers.
Pretty simple: opt-in KYC. Give people the option to email/fax you their passport or birth certificate or whatever, at any time after they set up their account but before the account is compromised. Extract the relevant ID numbers from the images; then store those numbers, encrypted, the same way you'd store "password recovery" info.
If the account is later compromised on their end, just ask the person attempting to do the recovery to send through the same stuff again, and compare with your recovery fields.
It's essentially the pseudo-biometric, pseudo-"something you have" equivalent of a Secret Question.
Shout far and wide (twitter, facebook, reddit, etc) that [Big corporation] is screwing you. If you can give it some sort of spin - like racism or gender equality - all the better. Just kick up as much of a storm as you can.
I guarantee that [Big corporation] will be forced to respond.
(I say this semi-jokingly, but it is sad that this a) works, and b) that sometimes it is the only way to get their attention)
If you're a business, there's no excuse not to pay Google for support. It's $150/mo; significantly higher than the $25/mo infrastructure bill the OP was discussing, but not unreasonable if you want to have a chance of guaranteeing QoS to your customers.
https://cloud.google.com/support/#comparison
Also AdSense, but there's obvious reasons they support that.
Their Realtime Database is extremely convenient and pretty useful for web apps, but there's no way to limit the connections and track bandwidth usage in real time. It's basically an open door for anyone with only your Security Rules to block data access. There's no logs to even see if your rules are working as expected, nor where your connections are coming from. They didn't even have that simple blue line showing basic usage in OP's post until recently. The profiling tool mentioned in the OP is also recent and doesn't give a proper picture.
It's just really pathetic. They sell themselves as a service company for startups, but one of the biggest things startups need to do is manage their costs, which Firebase doesn't provide in any form. And now since Google bought them, their support will be near non-existent as with all Google products.
Having said all that, their Free tier is great for live prototypes and doesn't require adding payment information. Just make sure you're not tying yourself in if you want to progress. Make thin wrappers around their libraries and export your data regularly if it's not throwaway.
Author's issue was with the commandline profiler.
(A Firebase dev)
5 questions can be logged per year for one-on-one email support and then you're on your own. And this is for urgent requests.
I can understand not wanting to be inundated with wasteful questions but when Firebase is changing their API and pricing plans and you're trying to scale a product, and paying them lots of money in the process, support should be forthcoming.
They also don't respond on their Twitter handle, which is a further crime.
With that said, it’s been a year since we introduced the five-a-year limit on technical troubleshooting questions. I’ll see if this is actually a useful limit to keep in place and remove it if not.
So now I'm looking at a message telling me that 'for all urgent requests...you have 1 question remaining' and hoping that nothing bad happens more than once between now and whenever you decide to allow me, a paying customer, to ask for help again.
To make clear: 99.9% of the time Firebase just works, which is great. And when I did receive support, it was excellent. It's the support model that's weak. Making billing and bug reports unlimited while limiting urgent requests to five a year is poor policy, and not very useful - for your customers at least.
I can't promise a change immediately, but I am taking a very close look at what we can do here.
So in the end it was a coding problem, because you forgot to add Keep-Alive = true? If you use the provided SDK's it should be no problem then right?
And it's hard to call this a coding problem if the requirements of the API never specified that session tickets and keep alive were necessary.
I wonder how many HN startup founders go around telling their customers on how to pay less.
keep-alive keeps the HTTP/TLS socket open.
TLS session tickets allow to close and "cheaply" reopen TLS session on a new socket for a new HTTP connection.
Using both keep-alive and TLS session tickets is optimal.
I guess the point was that by adding the middleman between it there was more control --- We are self-funded and small group of people. We can't afford to hire a bunch of people to know about containers and servers and every little detail.
Thanks for your comments! Didn't expect so many views so quickly on this...
Ouch!
Yeah, no kidding. It means someone nefarious (or bored) can easily run up a huge bill by just directing a ton of invalid requests to their API.
Many inexpensive hosting providers provide you with a lot of bandwidth (e.g. Scaleway offers unlimited bandwidth at ~200MBit/s). Assuming you can saturate the link, that's around 64TB per month of traffic. Quite the Firebase bill they'd have.
Also, immediately dispute the charge with your credit card company, as you want this on their radar. Their interests are aligned with yours to at least some degree (if Google's causing financial distress for one of their customers with thousands of dollars in unanticipated charges, how many others may get hit?).
Aside from that, if things played out as described: major dick move, Google! Is "do no evil" a bad joke these days? Seriously, though, Google is not a dumb company and my guess is that if the right person was made aware of this case (e.g., anyone who understands that the most important word in "customer relationship" is the second one), they would make it right.
I'm totally on the side of the author: when possible, it's worth a little extra elbow grease to build your business on top of free, self-hosted software instead of proprietary SaaS. Think of it as insurance against cases like this one.
This will result in the shutdown of their service, and ruin their app.
Thanks.
... The new code of conduct has a close approximation of the philosophy—though perhaps more formally phrased—in the very first sentence of the preface: "Employees of Alphabet... should do the right thing – follow the law, act honorably, and treat each other with respect."
Sounds like a loophole, not just 'more formally.'
Alphabet created a new code of conduct when it was formed that didn't use the phrase, but it was never dropped from Google.
Let's calm down.
The first step is to pay the bill, set up better infrastructure monitoring on your end, so you don't act surprised that you are consuming 7000% more resources than third party service is reporting.
Then, if you want to burn some money, you can file a petition and tell the court that you deserve to get free service because you did not know that you have been going over the reported data by 50%, I mean 7000% for... two years.
As far as I understood, Firebase didn't report it before. As mentioned, the meter "suddenly jumped" to 100GB and then 180GB (if I remember the numbers correctly).
The author decided to figure it out with support later, which makes sense if the previous number was something like 20-30GB and you guess it's simply in error.
Telecoms is a very heavily regulated industry.
> Also, immediately dispute the charge with your credit card company
No, don't do that. Most credit card companies require you to try and resolve the issue with the merchant before filing a chargeback. Sometimes they give you a time period, e.g. at least 30 days.
I know - because I am one of those humans in support grins. I work in the support organisation for GSuite (formerly Google Apps). And I work across the hall from the guys who support GCP (Google Cloud Platform). And there are an entire army of us, helping support many, many enterprise customers.
Even our consumer products have real people manning support phones - I know, because I've had to call them before (mostly Google Play Music issues - this was before I joined Google, so I couldn't use my internal pull...lol).
Regarding your second point - abuse is a real problem for any cloud provider. I know, because I happen to work in anti-abuse. It's a difficult problem to crack, and there's a lot of smart people (at many companies) trying to solve it. But please understand there are real humans at the other end, trying to provide a good service - and it's not very kind to take cheap shots at them.
There usually are appeal processes available for nearly any kind of action taken - if you were actually personally affected by an issue, or feel unsure about how something was handled - please feel free to reach out to me. I can't guarantee I can wave a magic wand and fix it, but I'm certainly willing to reach out to see what can be done.
Disclaimer: I work for Google, but the views expressed here are purely my own.
Your competitor AWS would refund in less than an hour with a billing support, without going through any phone calls. Your competitor has a support subscription service at least for people to purchase (expensive but worth it). Yours?
Many times, I've seen it's due to a misunderstandings, or crossed-wires regarding technical issues.
And yes, our support team does provide refunds in many cases - I know, because these sometimes get escalated to me to approve (e.g. if they're borderline). I think one was actually offered already in this case, but there may have been a misunderstanding or mix-up around that.
And yes, we have a "paid" support service - it's actually comes free with any GSuite subscription (even at the most basic level, which I think is like $5 a month). You can chat, email, or phone. I work as an engineer in the team that supports this.
For GCP (our cloud offerings), the support offering is a paid offering:
https://cloud.google.com/support/
He was from the Philippines, and it turned out he was a contractor providing "support" for Google Cloud. So I guess Google is outsourcing their support? Isn't this something Dell tried to do in the 90's with disastrous results?
How else do you provide 24/7 support?
There should be a huge "Send Feedback" widget at the bottom of every single documentation page:
http://i.imgur.com/MhWbY0W.png
http://i.imgur.com/Gu8GzLB.png (If you click it, it will pop up this modal form, which also allows you to just screenshot the offending part).
Did you try using this link on any of the pages that you found unhelpful, or bizarrely organised?
I'd encourage you to use this - as all of this feedback comes through to us via an internal system, and it does get reviewed (although I'm sure you can imagine the volume).
Otherwise feel free to reach out to me if you have specific concerns and feel more comfortable doing it that way, or you're unsure about the response you received. (Although quite frankly, I think the form is best, as that gets tracked, and anybody can jump on it).
But Google did come through and they posted the images here: https://cloud.google.com/storage-options/
That had been my complaint...that the documentation was hard to get and that when I saw the images presented at Next I understood things that I had previously given up on.
But man this is business, this is the stuff the global economy runs on 24/7, I hope you all are tougher than that...
No, its not. I was responsible for implementing, deploying and managing the infrastructure at lever.co when we were a tiny fledgeling startup. The entire application is built on top of JSON OT. I took some measurements at one point when we had ~thousands of active browser sessions of our app. At the time we were seeing about 1-2 OT merges (transform + re-apply) per day. All the other non-concurrent operations can get sent straight to the database. I don't know what the numbers are now, but I'd be shocked if OT ever becomes the bottleneck.
For text based OT you'll see more concurrent edits. But for text based OT, I have a little C library that can comfortably do 20 million simple text OT transforms/second on a single core of my old 2012 macbook air. Good luck making that a bottleneck.
The whole point of OT is that it lets you fearlessly merge concurrent edits. You can lean on that to rate limit messages to or from the client, if you need to. Because the client can always apply its own edits immediately to its own local model, there's no perceived latency. So, if you design your system right you can batch up changes at any granularity you like (per page, per form, per second, dynamically based on load, whatever). Of course, the tradeoff is that you lean on the OT system more by batching up edits. But it can be made effectively free to transform if the edits modify different parts of the database (using range trees to cull, etc).
But yeah - it is much faster than differential transform. I'm not sure how good diffing algorithms are in practice, but best case they need to scan the entire document. On the other hand OT only requires size & computation proportional to how much was changed. If we're collaboratively editing a 10kb text file, changes will mostly only be a few bytes each. And applying each change using a good rope library is a O(log n) operation.
Of course, all this requires a database which supports OT out of the box whistles innocently
Which is pretty sad.
YCombinator shouldn't let companies use Hacker News as a marketing tool, at least not to the extent Google does.
Then again, as a meta comment I will point out that the fact that both you and I so clearly notice this suggest that we are following these updates much more closely then most of the folks here.
It might not be so well known, but I found http://www.deepstream.io to have a relatively similar feature set, without having to deal with bad support (if it bothers you, just fix it yourself) and high prices, since you are the one who picks what suits you best.
I have yet to see a platform that's as flexible and robust, with the added bonus of being real-time.
The engineering response is pretty clear-cut. Polling every minute from an uncached SSL connection is expensive, don't do that. That's pretty obvious and the author admits this was a mistake. The complaint is about everything that's not the engineering part - support & communication.
I'm sorry, I can have 50'000 devices poll every minute via an uncached SSL connection on my 16$/month server from Online.net, including bandwidth and traffic costs.
I'm not sure what you're smoking, but I want that, too.
Seriously, at Google, TLS is terminated in hardware, and bandwidth costs less than a cent per terabyte for ISPs or Google.
The costs that this caused are basically nil.
(This, btw, is why I'm using kubernetes on rented bare metal, paying 30$ a month, for what would cost me, including traffic, around 900$ (Google Cloud with rented servers) to 63'000$ (Firebase) a month.)
https://aws.amazon.com/premiumsupport/pricing/
https://azure.microsoft.com/en-us/support/plans/
https://cloud.google.com/support/
Note that this was posted to HN around midnight Pacific, so it's quite possible nobody from the Firebase team had seen it yet.
I like the fact I can pay one price for self support and another for 15 minute resolution. Why force people to pay for support if they don't need it?
And a wholly different one if they screw up and you can't talk to them about it without paying them.
The former is an incentive to self-serve and investigate yourself, the latter is an incentive for the company to provide shoddy service to people pay to resolve it.
(More common is the Oracle/IBM-style approach of making your docs vague enough that you need professional services to actually integrate the product).
Just FYI.
Google doesn’t even provide that.
That said, we'd only need support to get ETAs on outages being resolved ><
They certainly should have communicated that better and their tools / charts should have a way of showing the billable data, but it's a bit of a stretch to call it vendor lock-in. It's not like everyone on Firebase is complaining and the author admits they made major mistakes.
Cloud services that pull this kind of bait and switch deserve to be ridiculed and lose lots of business over the negative PR. Cloud services that unintentionally pull this kind of bait and switch but then don't make it right or provide meaningful support deserve no special treatment.
To that extent, they should have communicated this better, ideally 90 days out perhaps even with phone calls for customers whose bill will increase notably.
But then they failed to follow through.
In Gitlab's case, their response was excellent and their total data loss was less than a day's work - and even that only for a full time manipulator of gitlab metadata (project managers? some parts of QA?): "Database data such as projects, issues, snippets, etc. created between January 31st 17:20 UTC and 23:30 UTC has been lost. Git repositories and Wikis were not removed as they are stored separately." ( https://about.gitlab.com/2017/02/10/postmortem-of-database-o... )
Considering that "[...] out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." ( https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx... ), this was a pretty amazing result, and about the best a response they could make. That, combined with their transparency, earns something of a pass (especially when data loss I care about is already limited by git's DVCS nature, even if they had obliterated the repository data.)
In Firelab's case, if no other action is taken, they've actively mislead their customer in failing to follow through on crediting their account, charged them an obscene and unsupported price hike, and gone incommunicado. That's like... an anti-pass.
Sounds like they've processed the credits as well, so that's good.