HN Explorer

Fly.io deleted all my apps without notifying me

111 points by lukas_b ↗ HN
I noticed my app was down and went to check it out. Lo and Behold my account had been flagged by their automatic fraud detection algorithm. This is despite my account having close to $500 of credit and being active for around a year now. I was not informed about this at all. I followed the prompt to allow them to charge my credit card in order to verify that I'm not a bot and now I have control of my account again. However none of my apps were restored. Has anyone else experienced something like this in the past?

53 comments

[ 4.6 ms ] story [ 84.7 ms ] thread
frafart
I haven’t had my apps deleted but the fraud detection system flags me almost every time I interact with the platform. It’s very disruptive.
add-sub-mul-div
Dropbox deleted all my data because I didn't log in for a long time. Guess you're supposed to check in on your completely static data occasionally. Whatever, the lesson is not to let yourself seriously rely on any of these services.
cpach
No matter what service provider one uses, one should always have backups and a plan B/disaster recovery. The 3-2-1 rule is as relevant as ever.
altairprime
What were your apps?
lukas_b
a personal vpn and a company website
yellowapple
I'd guess that the personal VPN had something to do with it. Did you do anything on it that put Fly.io at risk of getting in trouble?
yellowapple
Well that just evaporated the interest I had in using Fly.io. Thanks for the heads-up.
oliwarner
That's what makes me so deeply suspicious of posts like this. It costs nothing but a few minutes to spring up a fake account and dump on the competition.

Who knows if Lukas is legit, but they don't seem to have asked the service provider where their data is yet. Or what happened. You're not going to get those answers here.

Going straight to social media before you've found anything out stinks.

lukas_b
This is a reasonable criticism. I don't know how I can prove it as screenshots are not hard to fake, but here they are in any case. Also I emailed them and still haven't gotten a response, so I'm not inclined to be very understanding seeing as this appears to be happening quite frequently to people.

https://imgur.com/a/xLj3tqd

my company site is infoproc dot co. it is currently down. but if you google it you can see it has been indexed. And you can search for my company on linkedin if you want to know who I am.

oliwarner
I really wasn't directly attacking you or demanding proof, just pointing out why I don't like the normalisation of complaining in public as an early step in a technical support.
bananapub
you've made such a big change based on one post from a basically brand new account on HN? why?
yellowapple
It ain't really a big change; fly.io looked cool, but it ain't like I'm already a customer. And lukas_b doesn't seem to be all that brand-new of an account, either.

As for "why", it's because I've been burned by this sort of thing before.

Arjuna144
Thanks for telling. It may be a bug in their system, but the way this seems to have been handled by them is really ugly
r0ks0n
that bug could even be... a fly...
bmitc
It's my understanding that Fly.io heavily uses Elixir and has several Elixir bigshots on staff. But I only see mention of Fly.io for people are moving off of it or having major issues. I'm wondering of that's really the case and how given the tech stack.
g15jv2dp
How is the programming language they use at all relevant?
taspeotis
I forgot functional programming in Elixir meant no bugs! It's built on Erlang which halves bugs tenfold too!

So 0 bugs because Elixir / 10 = 0

QED

strzibny
The issue is political and has nothing to do with Elixir (which is pretty great btw). Please read the post again.
bmitc
Political? It was performed by their system. It doesn't sound like a human was in the loop at all, besides those who developed the system. And like I said, it isn't just this post alone. Read my comment again.
nomilk
> Has anyone else experienced something like this in the past

My account was marked as high risk and was prevented from deploying for a few hours: https://stackoverflow.com/a/76400473/5783745

But it didn't wipe out my apps. Took a couple hours to figure out what was going on and how to fix it (purchase some credit and run an esoteric and possibly unrelated command in the stack overflow answer). That (and a dozen other issues) was why I left fly. For now it's just not production-ready.

I do hope they build a successful product (as defined by happy customers, not by fund raising). The more players there are in this space, the more options we have as developers, which is good even if the path there is bumpy.

lukas_b
Thanks for this tip. I tried it and I keep getting the error: "Error: Your account got flagged by our internal fraud protection service" even after verifying my credit card.
yieldcrv
well guess I’m keeping my current solution

I use the free tier of vercel or netlify and host all the static assets on IPFS, I keep everything pinned using filecoin for free. lol yup I use filecoin to pin IPFS CIDs

and because IPFS gateway CDNs get super unreliable, I created a js script that does a series of promises to every ipfs gateway and after one actually loads it cancels all the other promises, so that ensures availability to users and can handle spikes in traffic, without messing with the free tier

zacksiri
The thing about fly.io / heroku / vercel is you are gaining convenience by giving up control over the underlying infrastructure.

When you give up control of the infrastructure you also give up the ability to decide what's right and wrong.

If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.

ec109685
Unless you build your own data center and run your own fiber, you’re at the mercy of someone else’s policies and could be shutdown at their whim.
johnchristopher
Don't forget to bring a diesel generator if you forgot to build your own electric grid.
qhwudbebd
Whilst this is true in theory, the experience of being a data centre customer, taking transit from one or two providers and peering with a handful of others is very different to being a user of a bulk cloud infrastructure provider.

I think the difference is automation, or rather lack of it. You build personal relationships with real people. Senior network engineers at the supplier don't consider it beneath them to ring up similarly competent people at the customer and chat directly.

In the past I've hosted kit on a larger scale for my businesses, but it seems you get the same personal experience as a tiny individual customer. Even with just a couple of racks of kit of for hobby projects and friends' stuff, I know the network engineers at the DC and transit providers and can email or call direct to discuss things when I need to.

More importantly, if (for example) one of my friends starts gobbling 10Gbps of $$$ transit because of a typo in a script, they ring my mobile to chuckle about it while we fix in realtime, rather than some automated process pulling the plug with the real engineers hiding behind support tickets and first-line support staff that can't string a sentence together.

zacksiri
I think it's important to realize, each level of abstraction has it's pros and cons. Concluding to have to go all the way down the stack and run your own data center and fiber is extreme.

In life it's always going to be a balance of what's feasible that you can do to keep control and giving up control because something is not feasible.

In this case obviously the author is having a problem high up the stack. The best solution would be to have more control in order to keep his business going.

miki123211
Even if you build your own datacenter and your own fiber, other ISPs might still either depeer from you or threaten to do so if you don't boot customer x off your service.
strzibny
This is where having Kamal config at hand is great.

Even if I would lose my VM and access, I can be up and running again pretty quickly on a different host.

I just change the IP in my config/deploy.yml

ec109685
And I guess offsite backups too?
alexedw
Would love to set up everything on Kamal but just feels like slightly too much management cost in maintaining a database yourself for mission critical stuff. If I'm going to use a managed DB, might as well do the same for the container?

Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.

tasuki
You mention Kamal as if it was some kind of a ubiquitous standard. It is the first time I hear about it. In what way is Kamal better than the hundreds of alternatives?
rizalp
thanks for informing this, will avoid fly.io at all cost
[deleted]
(comment deleted)
deely3
Why not mention the name of the company? Then post will not become flagged, I suppose.
lukas_b
the name is fly.io. it's in the title
deely3
Name of the posters company. Which site was erased by fly.io.
lukas_b
infoproc dot co. which is currently down.
mrkurt
This account was not banned in error. Lukas, if you’d like me to explain publicly, I'm happy to do so.
tptacek
Further: we do not have an automated system that deletes flagged apps.
lukas_b
Maybe you could start by replying to my email? I would not have complained publicly if there had been any form of communication from you but there was none.
tptacek
I think here will do nicely. That OK with you?
bbg2401
How is that an acceptable proposal? I use Fly, but the policy to communicate with customers only via public forum is a disaster.
tptacek
That's not a decision we made. Anyways: I really don't think we have much more to say without this person's consent to proceed, beyond what was already said: we do not have a system that automatically deletes flagged accounts, and this was not an error.

(What you're describing as our policy is certainly not our policy; these are extraordinarily distinctive circumstances.)

[deleted]
(comment deleted)
tptacek
Things got real quiet here, didn't they?
yellowapple
Did y'all respond to him via email like he asked? If so, then that'd explain his lack of replies here.

Y'all are doing some cool technical work, but this insistence on "either give us consent to air your dirty laundry in public or else we won't respond to you at all" raises a pretty big red flag. I'm sure there was a perfectly good reason for banning his account (I'd guess something to do with the VPN endpoint he said he was running on it), but this doesn't seem like a particularly reasonable way of handling it.

tptacek
Anybody else getting the impression that I'm not about to let this go? Want to double down on their behalf? By all means, do.

Just to keep this perfectly clear: we do not have a system that automatically deletes flagged accounts, and what happened here was not an error on our part.

yellowapple
Yeah, we all got that the first three times y'all said it. That doesn't make your responses any less weird. The only one "doubling down" here is you.