“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
> According to the support, the notification emails to the users were sent on Feb 23, Apr 6 and May 15th. However, we did not receive those at all.
If true, this is concerning. One message getting lost in spam understandable. But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken.
Do serious companies not have canaries or other checks in place to ensure their notifications are correctly delivered to customers?
If a spam service erroneously marks one email as spam, chances are it will also marks other very similar emails as spam. So it's not too surprising all three were marked.
For these kind of automated emails getting all emails consistently being delivered to everyone is really hard, almost impossible.
The problem here isn't really that emails weren't being delivered, it's that they seem to have tried only one method to contact people, didn't check how successful that was (e.g. by seeing how many customers were still on those regions), and seemingly never tried anything else (such as notifications on the dashboard, a temporary brown-out to alert people, etc.) – "we tried one way to contact you and that didn't work, so we just deleted your service sucks to be you lol kthxfuckitybye"
> Do serious companies not have canaries or other checks in place to ensure their notifications are correctly delivered to customers?
Even with a canary, you have no guarantee that a particular customer will receive your email. In this case, maximum CYA is a certified letter sent return receipt requested. There are services out there to mass mail those.
Even if all three emails were properly delivered, that is not sufficient notice for a storage service. Why is there not also a reminder on the dashboard?
A financial service I use was recently purchased by another. The company has been aggressive in keeping me on the loop about what is upcoming. Maybe six months before the actual event a heads up. Again at two months. Then at one month. Then every week, along with countdowns to the deadline. "Are you ready? This is really happening. Here are relevant docs on how to ensure your transition goes smoothly."
Even the dashboard isn't really enough... if I'm running a one-off application or many, I may not log into every dashboard for every single thing regularly. A scream test would have been most appropriate, combined with a backup and at least 30 days retention for migration.
Customers ignore emails newsflash. For our SASS service it isn’t uncommon for customers who are administrative contacts and technical contacts to spam their emails and/or generally ignore emails on all varieties of topics. If they’ve supplied valid phone numbers typically they don’t work either.
But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken.
Plus some enterprises change their mail policy out of nowhere. I have a lot of subscribers from enterprise companies, but sometimes every mail delivered to some company comes back with something like "Your address is not on our whitelist. We only accept approved emails." and every subscriber is hard bounced off the list and that's that. (They aren't paying me, though, so less of a big deal.)
Hard to reverse actions need multiple safety switches, for example, turning off the machines in that region for 2 weeks before deleting them, which would bring support issues to attention ahead of the no-going-back step of deleting data.
This kind of thing really does need a cooling off period.
Assume that your users won't see your emails. How do you help them avoid data loss when you shut down a service like this?
One option that I like is to take the service down (hence loudly breaking things that were depending on it) but keep backed up copies of the data for a while longer - ideally a month or more, but maybe just two weeks.
That way users who didn't see your messaging have a chance to get in touch and recover any data they would otherwise lose.
I'm not sure how best to handle the liability issues involved with storing backups of data for a period of time. Presumably the terms and conditions for a service can be designed to support this kind of backup storage "grace period" for these situations.
That seems like the worst of both worlds, during the brown-out you have to keep paying for the compute while your customers don't get a reliable service, even if they have a plan to migrate.
Also you probably can't keep charging customers for that period since you offer a crippled service on purpose.
Just shut it down for real (after proper early warnings), so you save on compute and no one is confused about the state, and offer data retrieval for the grace period.
Brownouts are great for API changes, but not very useful before a full shutdown.
That's why I'm saying to take it offline. Purposefully broken service is not very valuable, can't really be sold, and yet can still be missed; it also costs you money.
Much better is to stop the service but add button "Resume" that re-enables service for two more weeks with no data loss. That way you give users opportunity to gracefully migrate away.
Stopping service and immediately delete data is just callous.
When ovh sunsetted a class of VPSen and I'd completely failed to notice they were going to do that, I asked nicely in the support ticket I'd sent in and they turned it back on for a few days while I shifted the data to a replacement (which was still an ovh VPS, it had been Just Working long enough that I didn't feel like I'd been mistreated, more lulled into complacency by the lack of problems).
I think requiring a ticket might be a worthwhile trade-off compared to just adding the button, because that allows you to engage with customers to make sure they can (in a case like this) migrate to a different region of your own service, and the activation energy of sending a ticket means a customer's less likely to click 'Resume' and then forget about it again until it's too late.
I mean this is why you do these projects on two different timelines: The internal timeline and the external timeline.
Externally you communication: Different announcements each month, final notices at T+5M, System will be deleted at T+6M, data will be lost at that point and so on.
Internally (at least at work) such a timeline is more that at T+6M, we cut access to the systems. Afterwards, systems not accessed for 2-4 weeks are removed periodically and the hard removal is planned for T+9M. Customer support and account managers can manage if systems need to be accessed. If a customer needs the system for a longer time, they can, but then they pay for it. Entirely with all necessary infrastructure, not renting a few licenses on the system.
Call it a bit callous, but this allows our customer support to appear nice and in control. And it leaves the customer happy and relieved that we have left some slack and leeway. But they've been shaken and woken up and can get to migrating.
The biggest challenge here is to stay on it and to not allow customers to become complacent again. This can be done by e.g. limiting the reactivation time to a week or so so they have to get on it.
Yep, and in certain in house situations it's best to keep a backup around for ~13 months in case there's an obscure business process that only gets done once a year. (I'm aware that some people reading that sentence are going to go wtf at the idea that that's anybody's problem except whoever didn't tell you said business process even existed, but if it's a sufficiently critical finance or HR thing it tends to rapidly become everybody's problem so I like to have options)
Agree absolutely wrt complacency, I believe I asked for less than a week because I actively preferred a situation where I had to get on it immediately.
yeah no, the people complaining about their deleted data clearly wanted to keep using it - so much so, they didn't expect anything close to migrating away to another location
I wonder if they’re open in other EU regions? If you wanted to shut down a region, as a database provider, is it even possible to send snapshots to another region without user consent?
It feels like that could be a good practice, or not, depending on the laws in question.
Wow, the incredibly callous 3-word explanation of the issue by pointing to a docs link with no other context. Really gives off "it's your fault for not reading the wiki." Is this how InfluxDB treats their customers?
Incidentally at work we've been evaluating a new hosted observability provider, looks like we can rule out Influx as an option.
> Really gives off "it's your fault for not reading the wiki." Is this how InfluxDB treats their customers?
I don't see any indication that the person who posted that is associated with InfluxDB. In fact, it doesn't seem like any staff member has posted in that forum in the past week. Up to you if you consider that better or worse.
It's weekend and I assume people designated to answer questions on forums do not work 24/7.
It could be them not sending notification about it early enough or at all, it could be that note being stuck in spam or something, it could be person complaining not reading the e-mails.
I used Influx Enterprise at a previous startup. Support was so bad after our first year we switched back to the OSS Influx and added HAProxy with manual replication and round robin load balanced. Was so much smoother, wish we had done that from the start since that was our original plan.
This seems really poor. They should have been able to see what customers are still using the service in that region, send them additional reminders that they need to get off, and only after several of those and a grace period remove access. Maybe even have a phase of read-only access before full removal.
It's tough to believe that the turning off of the service couldn't have involved at least a week of 'soak' time, where if you contacted them they then helped you move to another location. After all the additional cost/benefit ratio of keeping the VMs around, but not using CPU or bandwidth vs. retaining a few customers would indicate it is the right thing to do for both the customers and the business.
I like Influx, and have been using it for all our TSDB needs for ~7 years. It sounds like there have been a lot of shake-ups internally, all their hopes for Flux (the InfluxQL language) have been aborted. I'm wondering about how they're doing.
This situation doesn't seem like it is specifically them doing anything wrong, if indeed they did send out multiple notifications over 6 months. It sounds like it's caught many people by surprise though, which makes one wonder if there was a problem with their announcements of this change. Definitely seems like it could have been handled better than "deletion with no recovery" though, some sort of "shutdown, wait a week or month, delete" would have been better.
3 emails over six months is pretty lousy notification rate. At minimum, something sent the month of the shutdown would be appropriate. More ideally, would be an aggressive notification push of increasing frequency the closer to the shutdown period. Was there any notification of the shutdown in the admin panels? The person with the ability to migrate data may not be the person receiving the messages.
EDIT: Also, does the company have a Sales team? Why is that not a top line item for representative to contact those who have active service in the region to emphasize the shutdown? Or was the Sales team similarly in the dark about the messaging?
It is pretty amazing. Business of storing other people's data, and evidently notice of deletion is treated as a non-event. Instead of a Huge Big Deal that should be the top line communication to impacted customers until a migration plan has been enacted.
Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influxdata.com.
Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications.
I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.
However, tech companies will often not have your physical address--unlike your bank. And I'd probably block phone calls from some tech company I was a customer of.
We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.
As a buyer I have come to expect good vendors to design systems so mistakes (my team or yours) don't cost me sleep or you business.[3] i.e.
- they do soft-deletes before hard
- have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w
- don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst.
- customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects.
- also small things like training, certification
- Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication.
Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider
- End customers (your customer's customers) may need to be notified and may need to sign off
- Compliance and GDPR DPA changes - both end customers and internal ones
- DR, BCP concerns have to be planned for , not all GCP regions are equivalent.
- Documentation and certifications like SoC, ISO, PCI, HIPAA etc usually mean ton of paperwork to modify
- SRE/Devops may have to move other services along with telemetry on InfluxDB, may need network whitelisting from their customers, things typically break when moving, need to plan dry runs, rollbacks and so on.
A better way to handle service closure would be to shut down but not delete on the planned date[1] , and offer data export separately for few weeks/month after[2].
You can definitely do better than shutting down service and deleting data at the same time .
[1] I would do this for internal customers let alone external paying ones
[2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it.
[3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.
> don't instantly nuke the account if a payment goes astray or delayed
Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short warning time is common practice among hosting companies, or if it’s unique to Hetzner.
I has an almost identical experience with https://virmach.com/. I will never recommend them.
After 5 years, they deleted everything 2 weeks after the first payment failure.
Sure it was the cheapest VPS. But still, you don't just delete your customers' data.
I was away from emails and the service during those two weeks. As far as I can tell it might have been some race condition in their payement processing system. They couldn't figure it out. They had no backup. They refused to reinstate the service anew to restore my own backup.
It doesn't look too far fetched from their point of view, they saw a payment failure and they may have assumed that you decided to stop paying and didn't bother to send a cancellation request.
It looks like you had your own backup, which is always a good idea, hopefully you were able to restore your data elsewhere.
I've had the exact same experience with them. After 10 years using that server, one payment failed, about ten days and they nuked the machine. German efficiency I suppose.
Some companies dont mark you as a debtor if you are under a certain threshold (say 10 dollars), because the cost of processing this unpaid amount is not worth their time.
Also it is smart to have the threshold set to at least 1 cent, because this way you dont ask someone to pay you is supposed to pay you a fraction of a cent due to some rounding error. There are those stories where a company sends you a registered mail, where they ask you to repay a fraction of a cent - what is impossible. Also the cost of the letter (snail mail) made it not worth it. Even if you get an email that is "free" you cant pay 0.0001 cent. I mean you can pay a whole one and then ask to get 0.999 back - the time required by the bookkeeper to process it, then pay it (probably with a fee) is not worth it.
I've been 3 weeks late for a Hetzner payment (also for medical reasons) more than once, and my servers are still running. They sent several emails, one was a reminder to pay and another was a warning about what date they would shut down service. So I guess their notice system isn't as straightforward as one week for everyone.
Perhaps it's because I pay for several bare metal servers, or because I have a business account with them. Perhaps it's because I pay their invoices by bank transfer manually instead of by credit card. Who knows! You have made me wary of changing to a credit card now, because those do fail from time to time!
What worries me more is Hetzner's reputation for suddenly dropping customers with no warning and no way to retrieve data from the servers. That's always on the back of my mind.
I know that this is a stressful time and it’s all hindsight, but there are two different contact methods that don’t rely on email available to you in migrations like this:
Going read-only, waiting 2 weeks, and then deleting. The contact method is peoples’ alerting systems as writes stop working.
Putting a message on your service dashboard indicating the upcoming action. The contact method is exactly what it sounds like, and it’s the only other place you can stick text and know for sure all your customers can access it.
It will probably help customer relations if you don’t hide behind the defense of only having email - there are a few strategies for this that you can use in the future. Best of luck on the road ahead - I know this must be a particularly stressful time.
Basically, you just turn stuff off, and make people scream, while they can turn it back on. You could have did this a month ago, as a critical warning of impending termination of systems.
You didn't do that.
Instead, you sent a few emails, which itself isn't even a guaranteed delivery. Again, as a CTO, you should know that email is not a guaranteed deliverable.
>You can't turn system off if customer paid for it. That's the breach of contract.
It sounds like they did turn the system off for paying customers though, why would it be any worse to do a scream test a few days before they pulled the rug out?
If it's just a matter of billing cycles (does everyone's billing cycle end on the same day?) , it seems like they could've handled it better. Just give impacted users a prorated and shortened final month (or even given it out for free for goodwill)
You've literally just told the world "you shouldn't rely on us for your data. When our business needs to drop you, we will and you might not receive notice."
> After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss.
Using billing contacts for this is a mess. The billing contact could be an accounts payable department that will check the invoice against the contract or PO and pay it. It could be an outsourced office that has no idea what a database is. It could be someone who only catches up on email once every few weeks.
What is isn’t in a technical contact who knows what the shutdown of a database means.
Ignore for a moment what you think your customers should have done, and look at the actual outcome. Some customers did not know about the shutdown and deletion, and have now lost data. You telling them "well you should have read your email" is not going to satisfy them, even if you think it should.
All you've done is told your customers that their data isn't safe with your service. This was an easily-avoidable "own goal" situation.
> All you've done is told your customers that their data isn't safe with your service.
And not only Influx's current customers, but also their future customers. I really like Influx for my homelab. But with this attitude, I would be really hesitant for a real world production deployment.
What future customers? After seeing this astoundingly terrible behaviour for a company with "DB" in their main product's name, I can't imagine anyone ever making the decision to trust InfluxData again. I know I certainly won't, nor will any company I work for.
Should get a phone number too. Shit like that happens. Next time lay down the law to the marketing drones from UX that want to "reduce friction".
Explain carefully so their thick minds will understand that NOBODY is lazy enough to quit subscribing to your service just because you added an additional field to your onboarding.
Which most people would be reluctant to provide because everyone hates sales spam, and what else would a SaaS need your phone number for in regular times (impeding data deletion is a good one)? On HN making a phone number for a signup is regularly criticised.
Taking the region off line and making the data inaccessible at the same time was a big wrong call. I won’t hash over the other good suggestions here, but I would throw out there you should have turned off all data plans APIs for at least a month before deleting a byte of data. Nothing wakes up the customers attention than everything suddenly failing.
I think the attitude that “you pay us money so you better read every email we send” is at odds with reality.
1) why do you believe a single human being has that email address? As a company of any size I would never assign a human to a vendor email address. Turnover and rogue employee risks are too high for that. Usually these vendor emails are black holes only used to establish the account and recover credentials if needed. Or, it ends up in the hands of a vendor relations person who is more an accountant than engineer. Do you get the emails from GCP and AWS directly in your inbox?
2) because I pay you money I expect the opposite of the relationship you articulated. I’m not here to read your emails. If it’s really important then use my account manager to contact me. Make a phone call. Email is for spam - ESPECIALLY when it’s from a vendor. I view it as “I’m paying you money, vendor, so you need to go out of your way to give me excellent service” not “I’m paying you money so your emails are incredibly important to me I hang on your every dispatch”.
What baffles me is why on earth did you guys delete the data? I get you couldn’t afford to run the region for whatever reason. But you should have retained all the data. Storage isn’t that expensive.
> Did anyone consider that bob@company.com left months ago, but since autopay still works, no one considered potential problems?
Everyone knows it happens, meanwhile every single company with high turnover is like this (those I've had personal encounters with):
Datadog: our domain has changed, but I cannot change my login. I've changed the email address in my profile, but I'm not sure if my login (which is an email address) is just a name or it may be used as an email address in some context.
Intuit: good luck changing your name
Apple Developer: still addresses the account as Bob No-Longer-Working-Here. It's not very clear how to change that name.
Apple ID: no, you cannot change the email address that had been primary back when you created it. And it better be a valid email address.
Orange: my address has changed twice, they are aware of that, they swore they updated my address everywhere; the invoices are coming with my old old address in their headers despite everything. Good thing they are sending them electronically at least so at least I receive them.
You likely can change the data there if you really need to, but it's very involved.
Someone should tell the IT/CRM drones that sometimes people not only leave the company, but also get incapacitated or die. In their Teletubby universes it doesn't seem to happen to anyone ever.
I once worked for a company that after a merger had a bank account running just to pay services it didn't know what they were for.
If you grow, have a merger, people moving, it is easy for email addresses to no longer be read. Yes there are best practices to prevent this, but most companies I have seen don't do that.
In one company important emails went to the email address (private!) of the founder, who left after M&A.
> if people can’t respond to vendors they pay money to, we’re really at a loss
No, you're legally obliged to keep the service running. They are paying customers and even if you can't reach them through email there are other means of communications. If a business fails on account of your one-sided deleting of the data then you're going to be in for a very hard time, for instance a damage claim for gross negligence and breach of contract. This isn't just going to blow over. The onus for reliable communications is on you and if the channel you've got fails then you seek another one.
>After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss
This is going to sound counterintuitive, like the Birthday Problem or Bayes Rule but at least for me it's true: most of the spam that gets into my inbox is from vendors I have a relationship with. Email isn't always ideal. Did you consider doing any other methods, like turning off writes a day or so before reads were disabled? That would trigger a much more immediate "oh shit" response than an email (unless the subject line is super clear and informative, and the email doesn't go to my span folder)
I find your tone here quite condescending. We've never received those mails you've mentioned and you make it out like it's our fault that we didn't react. I mean, you managed to send us marketing mails in april and failed to mention you're gonna discontinue the service. So much for honest effort...
Lmao are you serious? What about emails buried in spam? What if contact x left the company and the emails are black holes? There are a million valid reasons for emails to go poof. "But we emailed you" is weak.
You actually need to read your spam to check and if a company didn’t bother transition an employee out properly (i.e figure out what their email address was attached to), why is that on the supplier?
Why do they need to move mountains so that you can avoid any seriousness about your own operations?
This attitude you have is not appropriate for a vendor. Clients do not care that you think your substandard practices are fair, they will find someone else with actual "seriousness."
I missed an email that a train I was scheduled for had changed schedules on a trip earlier in the year. Why? Because Eurostar sends me maybe weekly email marketing messages that get filtered to one of my Google tabs because I maybe take them every couple years. It's probably unreasonable to expect that I'll see a reasonably last minute update though I'm not sure what a good alternative is.
I get probably 100+ emails a day that hit my inbox in some form and occasionally fairly important ones get mixed in with the mostly dross though Gmail does a pretty good job overall.
At least you got a notification your schedule had changed… I was just 5-6 hours late.
Had the same issue with amazon though. In the flood of “Information about your order” emails one had some slightly different content (but the same subject): “We haven’t received your entire return. Please contact us in 14 days or we’ll trash it and charge you.”
When I contacted them a month later I was not very pleased.
I get slammed with emails. In fact, nearly all of my email is automated content or junk. I would hope that these emails would catch my attention, but I can easily see how'd I'd miss them.
Further, they might be going to some alias/group that's not frequently monitored. If a vendor is going to delete all of my data, I expect way more noise than 3 random emails blasts.
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
Because if a company has 20000 engineers which one is the one that gets the vendors email? Answer is usually none and the email to that address goes to /dev/null.
Or suppose an employee did have the email on file and left the company.
Or suppose people assume vendor email are spam because they’re almost always spam.
What? Each and everyone of our suppliers have a dedicate address their notification are sent to. Those automatically goes into the service desk as a ticket and is read by the service desk team, which can either correct billing information if required, or escalate to the correct team if action is require or there's any doubt about the content of the email.
If you have 20.000 engineers (or even 200) you have a functional service desk and I assure you that no individual engineers email is given as the email for vendors to contact. Even for large contract where you have a preferred contact on each end, there's an escalation path.
Really? I mean good on you, but I’ve never seen such an arrangement in my 30 years of working in various mega corps. Usually they end up in the hands of vendor relations, but typically in no one’s hands and the expectation is the vendor works with us through TAMs. Most companies have their vendor relationship model based around negotiated licensing agreements and software delivery, and the saas delivery model is fairly recent. At a certain scale and age these things are pretty hard to change, so things like this aren’t well accounted for. It gets more complex when we have a federated model where we have one global relationship with the vendor but teams use the saas individually. Then the email address on the account is subsumed either in some automation or onboarding process used to ensure no engineer has the ability to reset credentials unilaterally.
Your model is a smart one. It’s smart enough it tells me you’re either a small company or a newer company, or both, or a company with a rarely together vendor management team.
"Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data."
Because the suppliers have abused this form of communication to the point it's not useful for serious communication anymore. I cant read you 1000 marketing emails to find out that one single important service related one.
At my company we used to only ask for email address. No names, no phone numbers, no mailing addresses. Because we understood that technical people don't like spam and don't like to give out their data. So we don't ask for them.
We also didn't send any reminders for them to check whether their email address was up to date. No account update reminders. To prevent annoying people with spam.
So other than sending emails and hoping that they read it, there was nothing else we could do.
> So other than sending emails and hoping that they read it, there was nothing else we could do.
but there are other ways. you can put a big red popup that can only be dismissed by typing "i agree" when the customer logins, you can put the service into read only mode, even with email you can send daily reminders for the last 30 days with a subject like "your data will be deleted in 21 days", etc there are so many things that could have been done.
What we sold was software that customers deploy locally. We don't have any of their data. But the software would stop working if their license is no longer valid, resulting in downtime. That already made people angry enough.
Now we have changed it so that the software never turns off even if license has expired (though it will continue to nag an email address). Updates also cannot be installed.
Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading.
Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees.
Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.
I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else
There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.
I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone.
To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed.
In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior.
See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
Re: Hyrum's law
This is why user facing code should have (at least) two classes of tests:
1) is it doing what the developer intended it to do
2) is it doing the same thing it did on the last release version with typical user requests?
That sounds the same, but it is not.
The first class is a set of simpler "happy paths" of intended specific behaviors.
The second class is like wargaming. A good way to do this is to replay user requests against your API and see that they return the same results release to release. You may also uncover interesting unintended behavior / conversations to have with users this way.
I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)
I think the analogy is to station people at various floors in a big office building while you disconnect (unmarked) cables in a legacy patch panel to see who starts screaming. Then you plug it back in and label the cable...
> I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.
I love the scream test, but the analogy you bring up actually - this seems unfair. The cost of Fedex'ing everyone is astounding (for many businesses).
But I like the concept. Definitely a sort of "shut off the server for like an hour" and then see who yells.
Phone calls for any account that still operating on the location.
The point of the story was that if you have something critical to communicate, you can't do it using the same methods that are also used for low-priority mostly junk and expect it to be acted upon. Surely a mortgage servicing company can afford to FedEx an envelope. I wouldn't expect Influx to do so.
For that matter, registered mail is another option that isn't as expensive as fedex... there's other options that are slightly more costly, and more noticeable than sending via the same structure that junk mail goes through.
I expressly don't use automated payments for my mortgage and auto loan(s) as I don't want to have an account miss... I didn't know my mortgage was sold/changed a couple months ago until I went online to change. The same happened with an auto loan a few years back as well.
When you buy a large, registered debt such as a mortgage the cost of Fedex'ing everyone should be factored into the sale. If that's too much money you shouldn't be buying such assets. Notifying those that are affected properly would seem to be the least you can do in such situations.
Whatever costs are imposed broadly an industry are covered by customers of that industry. If mortgages are more costly to buy, they’ll be more costly to originate.
The consumer doesn't bear 100% of the costs though. If mortgages cost $1mm to buy, they wouldn't cost $1mm to originate. They'd cost a little extra by being illiquid - the same amount as if they cost $2mm to buy.
The consumer nearly always bears 100% of the costs - because most companies sell at a profit.
The final price is a combination of all costs incurred + a profit percentage imposed on the consumer.
If all customers would receive fedex mail, then the costs of this fedex would be pushed on them somehow. Probably by making them pay slightly higher interest rates or by introducing some one off handling fee.
Of course not, but financial products are priced and offered with a financial outcome (usually a margin) in mind. If you make mortgage processing more expensive, you'll find the offers for origination are worse than if mortgage processing were less expensive.
Sorry, it's a ridiculous argument. Mortgages get bought and sold all the time and clearly the buyers are on the hook for the communications costs and these do not pre-emptively get priced into future products by the sellers. I'm sure that there are situations where your argument has merit but this isn't one of those.
No one is saying that existing mortgages would be re-priced. Those are contracts and you can't unilaterally change them. What I and others are saying is that if you changed the obligations of buyers such that mortgages became more expensive to service, that those servicing cost increases would ultimately be borne by the mortgage borrowers rather than eaten by mortgage lenders out of the goodness of their hearts.
> these [communications costs] do not pre-emptively get priced into future products by the sellers
They 100% do get priced in. Whenever you buy a product, you're paying all the costs of that product. When someone originates a mortgage, they're aware of the secondary market for mortgages. If that secondary market is eroded by a significant increase in communications costs, that reduces the willingness of a secondary buyer to bid for your book of mortgages. That erosion reduces your projected profit on originating, so you take a little longer to lower your offered rate to 5.250%, or you charge a bit higher origination fee, or whatever to ensure you maintain a viable business. So long as these fees hit the entire market, the other originators are all making the same calculations.
It seems odd that you [seem to] think that money for these costs would just result in reduced profits for the financial services companies rather than in increased borrowing costs.
> What I and others are saying is that if you changed the obligations of buyers such that mortgages became more expensive to service, that those servicing cost increases would ultimately be borne by the mortgage borrowers rather than eaten by mortgage lenders out of the goodness of their hearts.
I got what you and others are saying but I've been trying - and failing, apparently - to point out that the costs for a mortgage in case of a sale are born by the buyers who are not even the same kind of institutions as the parties that sell them, and so they are in no position to charge the subjects, nor are the sellers going to price the mortgages any higher in the future because the costs aren't born by them.
There is no such thing as 'projected profits' that go into this because mortgages can be sold (and are sold) more than once, the number of times is not known when they are issued first. And it is going to be only a small fraction of the audience that is going to be hard to reach for whatever reason. The presumption that there is some kind of free market mechanism that will ultimately pass those costs back to the original mortgage underwriters is not in any way evidenced by present day mortgage prices. On the sum total of mortgages out there and the - exorbitant - profits they create for the lenders we're talking about such small amounts that it will make zero difference.
A 'significant increase in communications costs' would translate into that being something that is some noticeable percentage of the total yield over the remaining time and it just simply isn't. Typical mortgage rates and amounts utterly dwarf the costs of a one time notification, especially if you don't have to notify everybody like that, and you can try cheaper channels first until you have a hit. Besides, the original mortgage 'service charge' is already a large multiple of the various costs and tends to be mostly pure profit for the initial lender.
It seems odd that you [seem to] think that mortgages are priced such that the mere cost of communications is going to show up in the prices, they are amongst the most profitable financial products.
> On the sum total of mortgages out there and the - exorbitant - profits they create for the lenders we're talking about such small amounts that it will make zero difference.
If it made zero difference, they would Fedex the documents, but they dont.
Something similar happened to me when I lived at a condo in Boston and the management company changed and I somehow missed the memo.
I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a lawsuit.
You know, I don't really care that mortgages get sold, but the fact that I need to go out of my way every time to make sure I update my information in their system is astounding.
Neither should be a random third party defrauding a bank, but banks are masters at outsourcing work and responsibility onto their customers, and that's why it's called "identity theft" and suddenly I am at fault somehow.
> Not everyone is going to see those emails and even those that do may not understand what they are reading.
If that's the case, these companies/people have no business using cloud services. Fair enough that you might not understand the ramification, in that case you contact support. If you don't see those email... that's on you. We operate out of a number of datacenters, they all communicate via email, giving us one month to three notice regarding service windows. If we fail to plan for an outage because we didn't see an email, that's our problem. I don't know why anyone would expect more from a SaaS company.
For really large customers, I would assume that they have a customer service representative and yes that person should have called. If you're just a small customer (even if you might be big in your own mind) and just have an account that get billed to a company credit card each month, it's a little naive to think you'd get anything more than an email.
We've already seen a number SaaS company just shutting of customers for little to no reason, even AWS has done this. Running things in the cloud is a risk, and it's you job as the operations team to stay on top of things, have backup plan, because you cannot expect cloud vendors to care about some random customer who just signed up using a credit card and a nondescript email. They should, but they don't.
A good rule is: Don't expect a SaaS/cloud company to put in more effort contacting you than you did signing up.
> > Not everyone is going to see those emails and even those that do may not understand what they are reading.
>
> If that's the case, these companies/people have no business using cloud services.
Cloud services are responsible for this. I've signed up to many cloud services where I purposefully unchecked all the newsletter/updates/... notifications.
But I still receive notifications for stuff unrelated to what I use. These emails are full of marketing/PR jargon, where it's unclear whether I'm affected by the change or whether there is even a change!
Cloud services are lazy, don't look at their customer use, spam everybody, and blame their customers when they missed an important update due to noise-fatigue.
Unless the cloud provider can provide proof that the person received and read such a notice, then they can still be sued for actual damages... and I'd be surprised if that doesn't happen in this case.
The fact is, there are many options from a cooldown, scream test, automated backup for migration/recovery... this organization did none of those things and absolutely deserves to lose massively as a result. This is a DATABASE as a Service... RETENTION should be one of the highest priorities.
For that matter, it would have been better if they auto-migrated in an OFF status, or otherwise backed up... just hitting the DELETE ALL DATA button is wrong. Several of the posters in the thread indicate they received no such emails.
Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this.
I would expect:
* Those 3 "email blast" notifications. I'm guessing one of two things happened here:
* You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP.
* You sent them as a transactional email, but blasted them too quickly and got pegged for spam. Never hit the inbox.
* Increasingly common "you haven't migrated emails" if you still detect traffic on these instances. This is pretty critical since some companies might not realize they have affected They should, but things get complex.
* Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.
* A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data.
Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.
Another problem is that service providers frequently poison the email channel with important sounding engagement dreck and we are now conditioned to ignore it.
* Important: migrate to this new feature immediately or you risk missing out!!
Vs
* Important: migrate your data immediately or you risk losing it!!
I was also thinking that salesmen will cold-call/email me at least three times before they give up. People with whom I have no business relationship try harder than a corporation taking someone's money.
Welcome to modern economy. This applies to B2C, and as we can see it, B2B too. Getting a new customer is all the shitty vendors care about. Keeping existing customers is apparently just cost.
Plus, many companies abuse what I assume is an exemption to spam rules by pretending that their marketing emails are “service related”.
Xfinity is the worst about this. They’ll send me a so-called service-related email exhorting me to download their app. Same with Capital One and their monthly emails asking me to turn on automated texts.
> Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.
If the data is moving between countries then this is not an option. Your clients may have legal or contractual obligations with respect to data location.
We are talking EU here. apart from some rare archaic remains of past rules or really "national security related" (and of this last one I don't think anyone is using a cloud provider) clients should be free to store their data in any Country inside the EU common market.
From the sounds of it, it was impossible for them to keep data after the 30th of June, so not sure a soft-delete at the deadline would work. Of course the better option probably would've been a soft delete before the deadline, but you know that some customers would've said "why'd you do this before the deadline, you could've waited an extra month!"
And the shutdown isn't even a month / quarter end, so seems even less like a billing cycle thing.
There's so many more mature ways to do a graceful paid service shutdown. Disable reads first so people get errors & contact support. Then disables writes as well. Somewhere a few weeks later you can consider deleting data.
As others have said, I've worked at megacorps with internal systems that had more mature migrations than this. Honestly I have run internal apps at sub-1000 employee firms where we did various forms of scream tests and soft deletes before MONTHS later, even daring to delete data on disk.
Contrary to the majority of the thread here, I find this to be an architectural issue. For whatever reason the system was designed without a way to communicate important service and maintenance issues to the customer. That’s part of the good architectural design of a system – it must include human factors, communication among them.
> We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions.
What other communication methods were attempted beyond just emails? Big, red obnoxious banners and warnings in various UIs? Phone calls?
The fact that it seems as though quite a few customers didn't get your emails, what was the thought process when looking at the workloads that were clearly still active before nuking it from orbit? Or was there no check and it was just assumed that people got the email and migrated?
Of the customers who were in that region, how many actually migrated? Was someone tracking these statistics and regularly reporting them to leadership to adjust tactics if there weren't enough migrations or shutdowns happening?
This screams either gross incompetence or straight up negligence. This is such a solvable problem (as many here have already mentioned various solutions), but I'm honestly just flabbergasted that this is a problem that is even being discussed here right now.
As a DBaaS, the data of your customers should be your number one priority. If its not, y'all need to take a hard look at what the heck your value proposition is.
We weren't impact by this directly, but you can be sure that this is going to be one of the topics for discussion amongst my teams this week. Mostly how we can either move off InfluxDB Cloud or ensure that our DR plans are up to date for the rug being pulled out from under us from you guys in the future.
Yup. It's literally a green status page that no one would give a second glance. That unreadable white on green? Oh. It's a deprecation message. It even has a subscribe link so that people would immediately and completely dismiss it as an ad due to ad/banner blindness.
Edit 2: Someone replied in the thread and added more context for the absolute lack of communication.
Hi Paul, email is one-way communication and not guaranteed to be delivered. At a minimum you should have monitored who did and did not respond to the email with some kind of action and those that did not should have more effort expended to be able to reach them. Finally, you should have kept the data for a reasonable amount of time (say 90 days) post shut-down so users that did not get the notification could download it. What you've done is super rude and if I were still a customer in an unaffected region it would definitely be reason enough to leave because it's pointless to sit and wait to see how you'll deal with my data when the time comes. Better to preempt that and leave while I still have control.
> we made our best efforts to notify affected users
You call three emails (the last of which was almost 2 months ago) "best efforts"?
I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?
They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc.
Multiple comments in the linked issue report not receiving an email.
Did you use the same email you use for spam/"marketing" for this notification?
The correct course of action is to shutdown the service and give people time to fetch data, not to erase the data as the first indication of shutdown.
A few emails are not sufficient if the end result is dataloss, a comment in documentation or release notes is not sufficient (the only reference at least one person in the referenced issue found).
I should not have to say this, but "best efforts" is not enough and is very offensive to every user still relying on your services. You had a duty of results, not merely "best efforts", to reach every-single-one of your active users before hitting shift-delete on their data.
Lol great attitude. why would anyone pay money to any company you manage now or in the future if you deliberately trash user data and justify it with 'but we emailed you a couple of times first'
Three emails are not "best efforts". 4.5 months notice is not "best effort".
My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.
Paul, are you actually for real right now? Did you really just say "We deleted all your data, and its your fault. We did whisper into the wind three times, you should have heard it. No, there is no chance of recovery"?
You might have literally deleted people's whole businesses, companies, who employ real people, who have families, now need to figure out how to continue. Not least of which, your own. If the company survives until Christmas I will be shocked; no one can trust your company ever again - your core business is storing other people's data, and you deleted it, for many, completely without warning.
I guess people still use Mongo even after finding it doesn't achieve any property of the CAP theorem, maybe some people will keep using a database provider with a track record of intentionally deleting their paying customers' data.
There just aren't enough adjectives for astonishment to adequately describe this situation.
I hope you offer Jay Clifford some support, he's clearly been put in the awful situation of having to explain the decisions of others and deliver the awful news. If I were him, I would be in need of serious mental health support, this is an absolutely awful thing to have responsibility for without any ability to rectify.
Hope they didn’t have any big corp customers impacted. Some big corps would very easily use that to cancel contracts and void out payments. Then let the lawyers deal with it
Why would you even need to log in if your service is running swimmingly? I run some things off of Backblaze and I haven't seen their console in over 2 years.
There's a small increase in cost compared to the cheapest US regions, but the European regions are still significantly cheaper than the expensive Australia/Brazil regions.
What I don't understand is why they have not auto migrated all customer data to the closest region. Why ask the customers to do this migration instead of transparently handling it for them? yes the cost may be different but this was their decision, they should absorb these costs until the customers decide.
99% of cases there are no legal implication of moving data in another country inside EU. Amsterdam is the closest in Geographic and Political sense. These are cloud data. already subject to EU Policy and not country specific. Those that really have the country condition can not even use a cloud provider but need to rent/collocate servers (not services) in the country.
Imagine if github was fired upon from the Death Star. The disturbance of the force would be epic as millions of voices suddenly cried out in terror and were suddenly silenced.
It seems this probably happened due to some regulation or other. The sunset date for the service should have been a month prior so that influx could have kept the data legally until the 30th in case of this situation happening.
They wanted to have the euros flowing in right until the last minute.
1. Flash messages on all user facing consoles.
2. No new resource able to be created for 6 mo this.
3. Emails.
4. Service end date should have been at least a month prior to mandatory shut down.
5. Any people still running workloads in May should have had aggressive contact attempts made to ensure they were aware.
6. The console in the region should have switched to a final backup that can be exported by the user or moved to another region. This should have been available for 30 days.
You don’t do this because it’s fun, you do this because you need to save reputation. If I can’t trust you with business critical data then why would I use you for my critical business?
Also, as someone who works for a large enterprise, if you really believe email is the way to inform them of these changes, well I’d reconsider your beliefs.
You wonder if we will see more of this from all these high burnrate SaaS startups right? It saves money to shutdown even paid services if they are cashflow negative. The difference between paid services and unpaid only matters if costs are below prices they sell the services for..
> "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions."
So it's worse than you believe. Yes, the handling is a scandal for all the reasons you say. But they weren't even pushed into this by some regulatory issue; it's pure cost-cutting.
They still should have at the very least done a backup of each customer DB in those regions and created an option to download and/or restore to a new region and kept those for at least 30-90 days.
A scream test would have been a better option in addition to the above.
For cases like this email is not enough. You need to do trial shutdowns where you disable access to the db unless someone acknowledges via web. Do it for maybe 24 hours so you know you catch everyone.
I've seen better communications around company-internal services that have been deprecated and for which a replacement exists that we need to migrate to. Heck, I've seen this a couple of times.
We had even tried out Influx a few different times. It was always ok, but never quite good enough. Now with this, I think, this seals the deal on me ever considering Influx either as a product or as a service.
I've been bitten by the old Influx and had to migrate to something we could trust... (Influx basically tacitly admitted that the original architecture was pretty poo and they've since swapped out the engine (twice?), but it smells a bit like mongodb trying to reinvent itself and distance itself from all the early web-scale claims, so I'm kinda skeptical).
So I've rolled our own with MySQL+tokudb, but that's not a good choice for a new system as tokudb is disappearing. When I tried to migrate to MyRocks we discovered the newer kids like MyRocks don't really work nearly so well for specifically this kind of use-case.
Something I haven't personally tried, but have heard rave reviews of, is Timescale. Its a special storage engine for Postgres and it has a lot of nice features like auto-maintained rollups. And they have lots of deep technical blogposts that I find myself agreeing with, so it must be good! :D
I use Timescale and I can recommend it. The reason why I'm still using influx too (1.7) is because it's unmatched in its data storage efficiency and query performance. You can get close with Timescale, but its main power is having the query power of PostgreSQL, if you have room for the extra hardware resources it requires.
It's unusual to read that InfluxDB is fast and efficient. Did you try VictoriaMetrics? It usually needs 10x less RAM than InfluxDB for the same workload, especially when the number of active time series is high. It also uses less CPU and disk space on the same production workload. [1]
I've heard of VictoriaMetrics before but haven't had time to play with it. InfluxDB is also now ingrained in a production system so replacing it is not straightforward. The query language is also different meaning everything that uses it will need to be updated too, and coming from a mainly SQL background, PromSQL/MetricsQL looks oddly weird.
Agreed that PromQL and MetricsQL have limited querying capabilities comparing to SQL or InfluxQL. But they cover the most of use cases for analyzing time series measurements, and allow writing much simpler queries than InfluxQL or Flux for these particular cases [1].
(My memory is that we kept on influx including 1.7, but it was a while ago now so memory might be fuzzy)
I guess influx perf and efficiency is really depending a lot on your data shape then :)
Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data.
In fact, it lost data generally. When we were replacing it we dual-ran an ACID DB version (which, with tokudb, was fast enough to keep up (although we didn't index every tag column)). So we did a diff and discovered just small random holes in the influx data that we'd never noticed before.
We had other considerations when we went mysql, as in, we were already using it. If shopping for a standalone solution to start a new project on, I'm thinking Timescale is the go-to these days?
I’ve been self-hosting InfluxDB in the hundreds of GB range for several years. I wouldn’t say I’m super happy with it, but… let’s say we’ve reached an understanding, the software and I. We’re on the latest patch of 1.8 and content to stay there.
I agree with GP about storage efficiency, which is superb. Query performance is good as long as a single query doesn’t deal with more than ~dozens of series. And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable. Storage is optimized for long reads of a single series, not for short reads of many series (but in fairness, you have to choose one or the other, that’s just the physics of the thing).
If I were starting from scratch, I’d probably pick Timescale. Or maybe DuckDB… I wonder if it would work for our use case.
> And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable.
I think I went properly mad while trying to troubleshoot this. The same query sometimes pulls 5GB, sometimes 20GB, sometimes 50GB and sometimes OOMs at 200GB memory pulled beyond base load of the system. And there's no query planner, no execution log, no metrics to help you. And most documentation or threads about it can be summarized as "well sucks to be you, eh? Maybe less data would be an option I guess"
We don't do that anymore and just roll up a very small select number of metrics.
And yeah, we've committed to Postgres as our main DB 2 years ago or so, and currently time is clearing up to start work with TimescaleDB. Zabbix is supposed to be great with it.
However, I do feel that they are trying to really do the right thing with the new 3.0 architecture, addressing the deficiencies (most importantly performance and full-fledged SQL) while keeping the stuff that works (InfluxQL for simple and legacy queries). Also leveraging open-source projects and contributing to their upstream is a plus. Thus I’m hoping for them to succeed delivering on that promise.
Agreed, embracing battle-hardened open source tech will be a win for them and for customers.
However, once your storage layer is parquet and your query layer is SQL, well... DuckDB is also basically parquet+SQL, and it won't be long before there's a nice Postgres wire protocol adapter in front of it. What's the advantage of continuing to use InfluxDB if you don't need clustering or HA?
First time: we chose Timescale over Influx (and a few other competitors). I really liked Timescale. This was ~2.5 years ago and obviously Timescale was much less mature than today.
We also sent data to AWS Timestream. We're heavy AWS users and got to try out the product. I found it ok, but it was expensive even for us (Disney+).
Second time: team migrated from Influx to Clickhouse for server and network metrics. Services had been relying on that data being correct and timely in order to route traffic, and well...there were issues. The simplest solution was simply to replace Influx with a product more suited to handling high volumes of metrics...yeah, ironic.
Unfortunately we’ve been bitten by influx operation issues a few times before. We adopted influxDB long time ago and always had to deal with breaking changes for each upgrade and every time we had an issue their answer would be: upgrade the latest version and see if it continues.
Then recently they made a change to Telegraf that broke all our data collection because they changed the environment variable replacer and their Jsonnet parser broke.
Now this. Shutting down a region without brownouts and only emails is not operationally acceptable.
We’ve moved on from influxDB for a while and only rely on telegraf now.
Which problem? Of the massive breaking changes between 0.8 and later, and then between 1.x and 2.x? Not to mention InfluxQL to Flux?
Also, they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it - years ago it was Prometheus vs InfluxDB for (non-SaaS) observability metrics, nowadays the only question is which backend for Prometheus to choose.
VictoriaMetrics. Don't have experience with InfluxDB, but had done rudimentary evaluation for popular backend. VictoriaMetrics stood out mainly due to low comparative operational maintenance.
Not sure why the downvotes here, a database service misusing semantic version is itself a red flag in my opinion. If major releases don't indicate a breaking change I'm not as confident in what might be part of a minor or patch release.
> they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it
I still remember this. We were ready to standardize on InfluxDB when we got a taste of their business practices.
Same, we were just about to choose InfluxDB, and ended up using it only for a niche low criticity scenario (VMware vSphere metrics, mostly for troubleshooting). We were never going to purchase Enterprise though, so they haven't lost anything outside of mindshare and champions, which can be evaluated to between 0 and infinity.
Out-of-band data, like emails, is bound to be ignored by some.
I'd incorporate warnings in the application-specific responses; you can also return a different response code to make sure many clients do not blindly ignore warnings. The HTTP standard includes a 207 Multi-Status response code, which is mostly used with WebDAV.
The funniest thing about this to me is how they officially responded that if you want to find out the reason why this happened, you can look it up on their Slack chat.
276 comments
[ 2.9 ms ] story [ 186 ms ] thread“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
If true, this is concerning. One message getting lost in spam understandable. But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken.
Do serious companies not have canaries or other checks in place to ensure their notifications are correctly delivered to customers?
For these kind of automated emails getting all emails consistently being delivered to everyone is really hard, almost impossible.
The problem here isn't really that emails weren't being delivered, it's that they seem to have tried only one method to contact people, didn't check how successful that was (e.g. by seeing how many customers were still on those regions), and seemingly never tried anything else (such as notifications on the dashboard, a temporary brown-out to alert people, etc.) – "we tried one way to contact you and that didn't work, so we just deleted your service sucks to be you lol kthxfuckitybye"
Even with a canary, you have no guarantee that a particular customer will receive your email. In this case, maximum CYA is a certified letter sent return receipt requested. There are services out there to mass mail those.
A financial service I use was recently purchased by another. The company has been aggressive in keeping me on the loop about what is upcoming. Maybe six months before the actual event a heads up. Again at two months. Then at one month. Then every week, along with countdowns to the deadline. "Are you ready? This is really happening. Here are relevant docs on how to ensure your transition goes smoothly."
Plus some enterprises change their mail policy out of nowhere. I have a lot of subscribers from enterprise companies, but sometimes every mail delivered to some company comes back with something like "Your address is not on our whitelist. We only accept approved emails." and every subscriber is hard bounced off the list and that's that. (They aren't paying me, though, so less of a big deal.)
Assume that your users won't see your emails. How do you help them avoid data loss when you shut down a service like this?
One option that I like is to take the service down (hence loudly breaking things that were depending on it) but keep backed up copies of the data for a while longer - ideally a month or more, but maybe just two weeks.
That way users who didn't see your messaging have a chance to get in touch and recover any data they would otherwise lose.
I'm not sure how best to handle the liability issues involved with storing backups of data for a period of time. Presumably the terms and conditions for a service can be designed to support this kind of backup storage "grace period" for these situations.
Also you probably can't keep charging customers for that period since you offer a crippled service on purpose.
Brownouts are great for API changes, but not very useful before a full shutdown.
Stopping service and immediately delete data is just callous.
I think requiring a ticket might be a worthwhile trade-off compared to just adding the button, because that allows you to engage with customers to make sure they can (in a case like this) migrate to a different region of your own service, and the activation energy of sending a ticket means a customer's less likely to click 'Resume' and then forget about it again until it's too late.
Externally you communication: Different announcements each month, final notices at T+5M, System will be deleted at T+6M, data will be lost at that point and so on.
Internally (at least at work) such a timeline is more that at T+6M, we cut access to the systems. Afterwards, systems not accessed for 2-4 weeks are removed periodically and the hard removal is planned for T+9M. Customer support and account managers can manage if systems need to be accessed. If a customer needs the system for a longer time, they can, but then they pay for it. Entirely with all necessary infrastructure, not renting a few licenses on the system.
Call it a bit callous, but this allows our customer support to appear nice and in control. And it leaves the customer happy and relieved that we have left some slack and leeway. But they've been shaken and woken up and can get to migrating.
The biggest challenge here is to stay on it and to not allow customers to become complacent again. This can be done by e.g. limiting the reactivation time to a week or so so they have to get on it.
Agree absolutely wrt complacency, I believe I asked for less than a week because I actively preferred a situation where I had to get on it immediately.
https://news.ycombinator.com/item?id=36663023
https://news.ycombinator.com/item?id=36661448
It feels like that could be a good practice, or not, depending on the laws in question.
They might not have been able to do it automatically as the region name were hardcoded in the hostname.
Incidentally at work we've been evaluating a new hosted observability provider, looks like we can rule out Influx as an option.
I don't see any indication that the person who posted that is associated with InfluxDB. In fact, it doesn't seem like any staff member has posted in that forum in the past week. Up to you if you consider that better or worse.
It could be them not sending notification about it early enough or at all, it could be that note being stuck in spam or something, it could be person complaining not reading the e-mails.
I wouldn't jump to conclusions here
This situation doesn't seem like it is specifically them doing anything wrong, if indeed they did send out multiple notifications over 6 months. It sounds like it's caught many people by surprise though, which makes one wonder if there was a problem with their announcements of this change. Definitely seems like it could have been handled better than "deletion with no recovery" though, some sort of "shutdown, wait a week or month, delete" would have been better.
EDIT: Also, does the company have a Sales team? Why is that not a top line item for representative to contact those who have active service in the region to emphasize the shutdown? Or was the Sales team similarly in the dark about the messaging?
I literally get more notification for DNS expirations (and those are scheduled and predictable).
I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.
- they do soft-deletes before hard
- have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w
- don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst.
- customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects.
- also small things like training, certification
- Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication.
Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider
A better way to handle service closure would be to shut down but not delete on the planned date[1] , and offer data export separately for few weeks/month after[2].You can definitely do better than shutting down service and deleting data at the same time .
[1] I would do this for internal customers let alone external paying ones
[2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it.
[3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.
Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short warning time is common practice among hosting companies, or if it’s unique to Hetzner.
After 5 years, they deleted everything 2 weeks after the first payment failure.
Sure it was the cheapest VPS. But still, you don't just delete your customers' data.
I was away from emails and the service during those two weeks. As far as I can tell it might have been some race condition in their payement processing system. They couldn't figure it out. They had no backup. They refused to reinstate the service anew to restore my own backup.
It looks like you had your own backup, which is always a good idea, hopefully you were able to restore your data elsewhere.
Strange. I've been at least eight days late with a VPS payment at Hetzner (3 euro) and the server is still up.
Also it is smart to have the threshold set to at least 1 cent, because this way you dont ask someone to pay you is supposed to pay you a fraction of a cent due to some rounding error. There are those stories where a company sends you a registered mail, where they ask you to repay a fraction of a cent - what is impossible. Also the cost of the letter (snail mail) made it not worth it. Even if you get an email that is "free" you cant pay 0.0001 cent. I mean you can pay a whole one and then ask to get 0.999 back - the time required by the bookkeeper to process it, then pay it (probably with a fee) is not worth it.
Perhaps it's because I pay for several bare metal servers, or because I have a business account with them. Perhaps it's because I pay their invoices by bank transfer manually instead of by credit card. Who knows! You have made me wary of changing to a credit card now, because those do fail from time to time!
What worries me more is Hetzner's reputation for suddenly dropping customers with no warning and no way to retrieve data from the servers. That's always on the back of my mind.
Going read-only, waiting 2 weeks, and then deleting. The contact method is peoples’ alerting systems as writes stop working.
Putting a message on your service dashboard indicating the upcoming action. The contact method is exactly what it sounds like, and it’s the only other place you can stick text and know for sure all your customers can access it.
It will probably help customer relations if you don’t hide behind the defense of only having email - there are a few strategies for this that you can use in the future. Best of luck on the road ahead - I know this must be a particularly stressful time.
https://www.v-wiki.net/scream-test-meaning/
Basically, you just turn stuff off, and make people scream, while they can turn it back on. You could have did this a month ago, as a critical warning of impending termination of systems.
You didn't do that.
Instead, you sent a few emails, which itself isn't even a guaranteed delivery. Again, as a CTO, you should know that email is not a guaranteed deliverable.
If you keep customer data after payment period, you're losing money.
So scream test will cost you money.
They decided that this cost is not appropriate payment to compensate for possible reputation loss.
Whether that's a good business decision or not, time will tell.
It takes years to build a good reputation and minutes to destroy it.
"We did our best - we have sent 3 emails". I wonder how shitty the product was when sending few emails is their best.
The guy either sounds like a full fleged VC psychopath, or someone very unexperienced.
It sounds like they did turn the system off for paying customers though, why would it be any worse to do a scream test a few days before they pulled the rug out?
If it's just a matter of billing cycles (does everyone's billing cycle end on the same day?) , it seems like they could've handled it better. Just give impacted users a prorated and shortened final month (or even given it out for free for goodwill)
You've literally just told the world "you shouldn't rely on us for your data. When our business needs to drop you, we will and you might not receive notice."
You're hurting your company's reputation by denigrating your customers like that
Using billing contacts for this is a mess. The billing contact could be an accounts payable department that will check the invoice against the contract or PO and pay it. It could be an outsourced office that has no idea what a database is. It could be someone who only catches up on email once every few weeks.
What is isn’t in a technical contact who knows what the shutdown of a database means.
All you've done is told your customers that their data isn't safe with your service. This was an easily-avoidable "own goal" situation.
And not only Influx's current customers, but also their future customers. I really like Influx for my homelab. But with this attitude, I would be really hesitant for a real world production deployment.
Explain carefully so their thick minds will understand that NOBODY is lazy enough to quit subscribing to your service just because you added an additional field to your onboarding.
Which most people would be reluctant to provide because everyone hates sales spam, and what else would a SaaS need your phone number for in regular times (impeding data deletion is a good one)? On HN making a phone number for a signup is regularly criticised.
I think the attitude that “you pay us money so you better read every email we send” is at odds with reality.
1) why do you believe a single human being has that email address? As a company of any size I would never assign a human to a vendor email address. Turnover and rogue employee risks are too high for that. Usually these vendor emails are black holes only used to establish the account and recover credentials if needed. Or, it ends up in the hands of a vendor relations person who is more an accountant than engineer. Do you get the emails from GCP and AWS directly in your inbox?
2) because I pay you money I expect the opposite of the relationship you articulated. I’m not here to read your emails. If it’s really important then use my account manager to contact me. Make a phone call. Email is for spam - ESPECIALLY when it’s from a vendor. I view it as “I’m paying you money, vendor, so you need to go out of your way to give me excellent service” not “I’m paying you money so your emails are incredibly important to me I hang on your every dispatch”.
What baffles me is why on earth did you guys delete the data? I get you couldn’t afford to run the region for whatever reason. But you should have retained all the data. Storage isn’t that expensive.
I can empathize with this, but also would expect a good product organization to consider failure modes here and work around them.
Did anyone consider that bob@company.com left months ago, but since autopay still works, no one considered potential problems?
Did anyone consider Bob in accounting is paying the bill but ignores email that doesn’t have “balance due” in it?
… and a million other scenarios that are quite likely and need consideration.
Everyone knows it happens, meanwhile every single company with high turnover is like this (those I've had personal encounters with):
Datadog: our domain has changed, but I cannot change my login. I've changed the email address in my profile, but I'm not sure if my login (which is an email address) is just a name or it may be used as an email address in some context.
Intuit: good luck changing your name
Apple Developer: still addresses the account as Bob No-Longer-Working-Here. It's not very clear how to change that name.
Apple ID: no, you cannot change the email address that had been primary back when you created it. And it better be a valid email address.
Orange: my address has changed twice, they are aware of that, they swore they updated my address everywhere; the invoices are coming with my old old address in their headers despite everything. Good thing they are sending them electronically at least so at least I receive them.
You likely can change the data there if you really need to, but it's very involved.
Someone should tell the IT/CRM drones that sometimes people not only leave the company, but also get incapacitated or die. In their Teletubby universes it doesn't seem to happen to anyone ever.
If you grow, have a merger, people moving, it is easy for email addresses to no longer be read. Yes there are best practices to prevent this, but most companies I have seen don't do that.
In one company important emails went to the email address (private!) of the founder, who left after M&A.
No, you're legally obliged to keep the service running. They are paying customers and even if you can't reach them through email there are other means of communications. If a business fails on account of your one-sided deleting of the data then you're going to be in for a very hard time, for instance a damage claim for gross negligence and breach of contract. This isn't just going to blow over. The onus for reliable communications is on you and if the channel you've got fails then you seek another one.
This is going to sound counterintuitive, like the Birthday Problem or Bayes Rule but at least for me it's true: most of the spam that gets into my inbox is from vendors I have a relationship with. Email isn't always ideal. Did you consider doing any other methods, like turning off writes a day or so before reads were disabled? That would trigger a much more immediate "oh shit" response than an email (unless the subject line is super clear and informative, and the email doesn't go to my span folder)
Please make sure you can implement a TCP-type communication with your customers for this kind of critical movement.
Why do they need to move mountains so that you can avoid any seriousness about your own operations?
I have far bigger fish to fry than monitoring my inbox for shitty practices.
Deleting their data first and then complaining that your customers don't run a serious enough operation is not the way to keep the best reputation.
> Why do they need to move mountains so that you can avoid any seriousness about your own operations?
You won’t stay in business for long with that attitude.
This kind of attitude will ensure you lose your customers.
I get probably 100+ emails a day that hit my inbox in some form and occasionally fairly important ones get mixed in with the mostly dross though Gmail does a pretty good job overall.
Had the same issue with amazon though. In the flood of “Information about your order” emails one had some slightly different content (but the same subject): “We haven’t received your entire return. Please contact us in 14 days or we’ll trash it and charge you.”
When I contacted them a month later I was not very pleased.
Further, they might be going to some alias/group that's not frequently monitored. If a vendor is going to delete all of my data, I expect way more noise than 3 random emails blasts.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
Because it's burried among lots of other similar looking emails that's just marketing garbage?
Because the email wasn't delivered?
Because the customer was on vacation and they expect a reply very quickly?
There's plenty of valid reasons to miss an email.
Or suppose an employee did have the email on file and left the company.
Or suppose people assume vendor email are spam because they’re almost always spam.
If you have 20.000 engineers (or even 200) you have a functional service desk and I assure you that no individual engineers email is given as the email for vendors to contact. Even for large contract where you have a preferred contact on each end, there's an escalation path.
Your model is a smart one. It’s smart enough it tells me you’re either a small company or a newer company, or both, or a company with a rarely together vendor management team.
"Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data."
Plenty of past explanation here:
https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
We also didn't send any reminders for them to check whether their email address was up to date. No account update reminders. To prevent annoying people with spam.
So other than sending emails and hoping that they read it, there was nothing else we could do.
but there are other ways. you can put a big red popup that can only be dismissed by typing "i agree" when the customer logins, you can put the service into read only mode, even with email you can send daily reminders for the last 30 days with a subject like "your data will be deleted in 21 days", etc there are so many things that could have been done.
Now we have changed it so that the software never turns off even if license has expired (though it will continue to nag an email address). Updates also cannot be installed.
Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees.
https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-...
Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.
To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed.
In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior.
See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
That sounds the same, but it is not.
The first class is a set of simpler "happy paths" of intended specific behaviors.
The second class is like wargaming. A good way to do this is to replay user requests against your API and see that they return the same results release to release. You may also uncover interesting unintended behavior / conversations to have with users this way.
I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)
I love the scream test, but the analogy you bring up actually - this seems unfair. The cost of Fedex'ing everyone is astounding (for many businesses).
But I like the concept. Definitely a sort of "shut off the server for like an hour" and then see who yells.
Phone calls for any account that still operating on the location.
100% agreed 3 emails is... hardly anything.
I expressly don't use automated payments for my mortgage and auto loan(s) as I don't want to have an account miss... I didn't know my mortgage was sold/changed a couple months ago until I went online to change. The same happened with an auto loan a few years back as well.
The final price is a combination of all costs incurred + a profit percentage imposed on the consumer.
If all customers would receive fedex mail, then the costs of this fedex would be pushed on them somehow. Probably by making them pay slightly higher interest rates or by introducing some one off handling fee.
> these [communications costs] do not pre-emptively get priced into future products by the sellers
They 100% do get priced in. Whenever you buy a product, you're paying all the costs of that product. When someone originates a mortgage, they're aware of the secondary market for mortgages. If that secondary market is eroded by a significant increase in communications costs, that reduces the willingness of a secondary buyer to bid for your book of mortgages. That erosion reduces your projected profit on originating, so you take a little longer to lower your offered rate to 5.250%, or you charge a bit higher origination fee, or whatever to ensure you maintain a viable business. So long as these fees hit the entire market, the other originators are all making the same calculations.
It seems odd that you [seem to] think that money for these costs would just result in reduced profits for the financial services companies rather than in increased borrowing costs.
I got what you and others are saying but I've been trying - and failing, apparently - to point out that the costs for a mortgage in case of a sale are born by the buyers who are not even the same kind of institutions as the parties that sell them, and so they are in no position to charge the subjects, nor are the sellers going to price the mortgages any higher in the future because the costs aren't born by them.
There is no such thing as 'projected profits' that go into this because mortgages can be sold (and are sold) more than once, the number of times is not known when they are issued first. And it is going to be only a small fraction of the audience that is going to be hard to reach for whatever reason. The presumption that there is some kind of free market mechanism that will ultimately pass those costs back to the original mortgage underwriters is not in any way evidenced by present day mortgage prices. On the sum total of mortgages out there and the - exorbitant - profits they create for the lenders we're talking about such small amounts that it will make zero difference.
A 'significant increase in communications costs' would translate into that being something that is some noticeable percentage of the total yield over the remaining time and it just simply isn't. Typical mortgage rates and amounts utterly dwarf the costs of a one time notification, especially if you don't have to notify everybody like that, and you can try cheaper channels first until you have a hit. Besides, the original mortgage 'service charge' is already a large multiple of the various costs and tends to be mostly pure profit for the initial lender.
It seems odd that you [seem to] think that mortgages are priced such that the mere cost of communications is going to show up in the prices, they are amongst the most profitable financial products.
If it made zero difference, they would Fedex the documents, but they dont.
I wouldn't be surprised if the cost difference is negligible, maybe a couple bucks per mortgage? FedEx bulk discounts can be pretty massive
I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a lawsuit.
https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-kn...
If that's the case, these companies/people have no business using cloud services. Fair enough that you might not understand the ramification, in that case you contact support. If you don't see those email... that's on you. We operate out of a number of datacenters, they all communicate via email, giving us one month to three notice regarding service windows. If we fail to plan for an outage because we didn't see an email, that's our problem. I don't know why anyone would expect more from a SaaS company.
For really large customers, I would assume that they have a customer service representative and yes that person should have called. If you're just a small customer (even if you might be big in your own mind) and just have an account that get billed to a company credit card each month, it's a little naive to think you'd get anything more than an email.
We've already seen a number SaaS company just shutting of customers for little to no reason, even AWS has done this. Running things in the cloud is a risk, and it's you job as the operations team to stay on top of things, have backup plan, because you cannot expect cloud vendors to care about some random customer who just signed up using a credit card and a nondescript email. They should, but they don't.
A good rule is: Don't expect a SaaS/cloud company to put in more effort contacting you than you did signing up.
>
> If that's the case, these companies/people have no business using cloud services.
Cloud services are responsible for this. I've signed up to many cloud services where I purposefully unchecked all the newsletter/updates/... notifications.
But I still receive notifications for stuff unrelated to what I use. These emails are full of marketing/PR jargon, where it's unclear whether I'm affected by the change or whether there is even a change!
Cloud services are lazy, don't look at their customer use, spam everybody, and blame their customers when they missed an important update due to noise-fatigue.
This is the main reason why I stopped using SaaS.
The fact is, there are many options from a cooldown, scream test, automated backup for migration/recovery... this organization did none of those things and absolutely deserves to lose massively as a result. This is a DATABASE as a Service... RETENTION should be one of the highest priorities.
For that matter, it would have been better if they auto-migrated in an OFF status, or otherwise backed up... just hitting the DELETE ALL DATA button is wrong. Several of the posters in the thread indicate they received no such emails.
I would expect:
* Those 3 "email blast" notifications. I'm guessing one of two things happened here:
* Increasingly common "you haven't migrated emails" if you still detect traffic on these instances. This is pretty critical since some companies might not realize they have affected They should, but things get complex.* Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.
* A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data.
Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.
* Important: migrate to this new feature immediately or you risk missing out!!
Vs
* Important: migrate your data immediately or you risk losing it!!
I ignore most of my vendor's emails because they're simply trying to spam me at this point.
Xfinity is the worst about this. They’ll send me a so-called service-related email exhorting me to download their app. Same with Capital One and their monthly emails asking me to turn on automated texts.
If the data is moving between countries then this is not an option. Your clients may have legal or contractual obligations with respect to data location.
Many people don’t have that option now since their data is dead and gone.
And the shutdown isn't even a month / quarter end, so seems even less like a billing cycle thing.
There's so many more mature ways to do a graceful paid service shutdown. Disable reads first so people get errors & contact support. Then disables writes as well. Somewhere a few weeks later you can consider deleting data.
As others have said, I've worked at megacorps with internal systems that had more mature migrations than this. Honestly I have run internal apps at sub-1000 employee firms where we did various forms of scream tests and soft deletes before MONTHS later, even daring to delete data on disk.
This is just regular old incompetence/negligence/greed.
> We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions.
What other communication methods were attempted beyond just emails? Big, red obnoxious banners and warnings in various UIs? Phone calls?
The fact that it seems as though quite a few customers didn't get your emails, what was the thought process when looking at the workloads that were clearly still active before nuking it from orbit? Or was there no check and it was just assumed that people got the email and migrated?
Of the customers who were in that region, how many actually migrated? Was someone tracking these statistics and regularly reporting them to leadership to adjust tactics if there weren't enough migrations or shutdowns happening?
This screams either gross incompetence or straight up negligence. This is such a solvable problem (as many here have already mentioned various solutions), but I'm honestly just flabbergasted that this is a problem that is even being discussed here right now.
As a DBaaS, the data of your customers should be your number one priority. If its not, y'all need to take a hard look at what the heck your value proposition is.
We weren't impact by this directly, but you can be sure that this is going to be one of the topics for discussion amongst my teams this week. Mostly how we can either move off InfluxDB Cloud or ensure that our DR plans are up to date for the rug being pulled out from under us from you guys in the future.
Depends on where and how this message was added.
It also means that they had no monitoring in place to see how many people migrated.
Edit: They also say that this is reflected on the status page. Here's how their page looks: https://i.imgur.com/xlO4Ik2.png
Yup. It's literally a green status page that no one would give a second glance. That unreadable white on green? Oh. It's a deprecation message. It even has a subscribe link so that people would immediately and completely dismiss it as an ad due to ad/banner blindness.
Edit 2: Someone replied in the thread and added more context for the absolute lack of communication.
What amateur hour service is that?
It is astonishing that you have literally completely ignored one of the primary USPs of your product.
You call three emails (the last of which was almost 2 months ago) "best efforts"?
I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?
They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc.
They also had a notice on the status page
Did you use the same email you use for spam/"marketing" for this notification?
The correct course of action is to shutdown the service and give people time to fetch data, not to erase the data as the first indication of shutdown.
A few emails are not sufficient if the end result is dataloss, a comment in documentation or release notes is not sufficient (the only reference at least one person in the referenced issue found).
truly mind blowing behavior.
Then again, I was only using the free tier, so I guess I got what I paid for.
Trying to assert that you were doing what you thought was right only presents the image that your company is run poorly.
The correct thing to do is to admit that your best efforts were not aligned with best practices, and look into remediation.
Not “well, we tried”
Not ideal???
You backed up everyone's DB and moved that to another region so they can just restore and change DB endpoints, right?
I don't believe that someone along the chain didn't suggest a scream test or similar. If they did, they must have been ignored.
My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.
You might have literally deleted people's whole businesses, companies, who employ real people, who have families, now need to figure out how to continue. Not least of which, your own. If the company survives until Christmas I will be shocked; no one can trust your company ever again - your core business is storing other people's data, and you deleted it, for many, completely without warning.
I guess people still use Mongo even after finding it doesn't achieve any property of the CAP theorem, maybe some people will keep using a database provider with a track record of intentionally deleting their paying customers' data.
There just aren't enough adjectives for astonishment to adequately describe this situation.
I hope you offer Jay Clifford some support, he's clearly been put in the awful situation of having to explain the decisions of others and deliver the awful news. If I were him, I would be in need of serious mental health support, this is an absolutely awful thing to have responsibility for without any ability to rectify.
Hope they didn’t have any big corp customers impacted. Some big corps would very easily use that to cancel contracts and void out payments. Then let the lawyers deal with it
Three emails clearly wasn't enough right? Now their name is in the dumps, customers are pissed and my only exposure to influxdb is a negative one.
I hope other saas guys learn this very expensive lesson.
I know ap-southeast tends to be pricier, I'm assuming Belgium was the same.
https://www.concurrencylabs.com/blog/choose-your-aws-region-...
They wanted to have the euros flowing in right until the last minute.
1. Flash messages on all user facing consoles. 2. No new resource able to be created for 6 mo this. 3. Emails. 4. Service end date should have been at least a month prior to mandatory shut down. 5. Any people still running workloads in May should have had aggressive contact attempts made to ensure they were aware. 6. The console in the region should have switched to a final backup that can be exported by the user or moved to another region. This should have been available for 30 days.
You don’t do this because it’s fun, you do this because you need to save reputation. If I can’t trust you with business critical data then why would I use you for my critical business?
Also, as someone who works for a large enterprise, if you really believe email is the way to inform them of these changes, well I’d reconsider your beliefs.
> "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions."
So it's worse than you believe. Yes, the handling is a scandal for all the reasons you say. But they weren't even pushed into this by some regulatory issue; it's pure cost-cutting.
A scream test would have been a better option in addition to the above.
We had even tried out Influx a few different times. It was always ok, but never quite good enough. Now with this, I think, this seals the deal on me ever considering Influx either as a product or as a service.
I've been bitten by the old Influx and had to migrate to something we could trust... (Influx basically tacitly admitted that the original architecture was pretty poo and they've since swapped out the engine (twice?), but it smells a bit like mongodb trying to reinvent itself and distance itself from all the early web-scale claims, so I'm kinda skeptical).
So I've rolled our own with MySQL+tokudb, but that's not a good choice for a new system as tokudb is disappearing. When I tried to migrate to MyRocks we discovered the newer kids like MyRocks don't really work nearly so well for specifically this kind of use-case.
Something I haven't personally tried, but have heard rave reviews of, is Timescale. Its a special storage engine for Postgres and it has a lot of nice features like auto-maintained rollups. And they have lots of deep technical blogposts that I find myself agreeing with, so it must be good! :D
[1] https://valyala.medium.com/insert-benchmarks-with-inch-influ...
[1] https://valyala.medium.com/promql-tutorial-for-beginners-9ab...
I guess influx perf and efficiency is really depending a lot on your data shape then :)
Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data.
In fact, it lost data generally. When we were replacing it we dual-ran an ACID DB version (which, with tokudb, was fast enough to keep up (although we didn't index every tag column)). So we did a diff and discovered just small random holes in the influx data that we'd never noticed before.
We had other considerations when we went mysql, as in, we were already using it. If shopping for a standalone solution to start a new project on, I'm thinking Timescale is the go-to these days?
I agree with GP about storage efficiency, which is superb. Query performance is good as long as a single query doesn’t deal with more than ~dozens of series. And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable. Storage is optimized for long reads of a single series, not for short reads of many series (but in fairness, you have to choose one or the other, that’s just the physics of the thing).
If I were starting from scratch, I’d probably pick Timescale. Or maybe DuckDB… I wonder if it would work for our use case.
I think I went properly mad while trying to troubleshoot this. The same query sometimes pulls 5GB, sometimes 20GB, sometimes 50GB and sometimes OOMs at 200GB memory pulled beyond base load of the system. And there's no query planner, no execution log, no metrics to help you. And most documentation or threads about it can be summarized as "well sucks to be you, eh? Maybe less data would be an option I guess"
We don't do that anymore and just roll up a very small select number of metrics.
And yeah, we've committed to Postgres as our main DB 2 years ago or so, and currently time is clearing up to start work with TimescaleDB. Zabbix is supposed to be great with it.
However, I do feel that they are trying to really do the right thing with the new 3.0 architecture, addressing the deficiencies (most importantly performance and full-fledged SQL) while keeping the stuff that works (InfluxQL for simple and legacy queries). Also leveraging open-source projects and contributing to their upstream is a plus. Thus I’m hoping for them to succeed delivering on that promise.
However, once your storage layer is parquet and your query layer is SQL, well... DuckDB is also basically parquet+SQL, and it won't be long before there's a nice Postgres wire protocol adapter in front of it. What's the advantage of continuing to use InfluxDB if you don't need clustering or HA?
First time: we chose Timescale over Influx (and a few other competitors). I really liked Timescale. This was ~2.5 years ago and obviously Timescale was much less mature than today.
We also sent data to AWS Timestream. We're heavy AWS users and got to try out the product. I found it ok, but it was expensive even for us (Disney+).
Second time: team migrated from Influx to Clickhouse for server and network metrics. Services had been relying on that data being correct and timely in order to route traffic, and well...there were issues. The simplest solution was simply to replace Influx with a product more suited to handling high volumes of metrics...yeah, ironic.
Our team maintains an internal CRM. When we plan to delete data or deprecate features, what we usually do (beyond sending emails):
- hide features/data from the UI without actually deleting them - if no one complains, after a few weeks, proceed with the removal
- for critical data, make sure there are backups, store them for about a month - if no one requests them, delete them
Then recently they made a change to Telegraf that broke all our data collection because they changed the environment variable replacer and their Jsonnet parser broke.
Now this. Shutting down a region without brownouts and only emails is not operationally acceptable.
We’ve moved on from influxDB for a while and only rely on telegraf now.
Also, they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it - years ago it was Prometheus vs InfluxDB for (non-SaaS) observability metrics, nowadays the only question is which backend for Prometheus to choose.
I sometimes wonder if vendors realize that they put their customers into a buying mode when they do this, when our options are:
- Upgrade to the new version of product X.
- Change to vendor or tooling completely as we're already changing everything.
We might pick another system if we feel like it is more stable.
Either there should have been a breaking change in there or they don't understand symantic versioning
I still remember this. We were ready to standardize on InfluxDB when we got a taste of their business practices.
Out-of-band data, like emails, is bound to be ignored by some.
I'd incorporate warnings in the application-specific responses; you can also return a different response code to make sure many clients do not blindly ignore warnings. The HTTP standard includes a 207 Multi-Status response code, which is mostly used with WebDAV.