HN Explorer

Ask HN: My web app was hacked today – Bad experiences and how to prevent them?

90 points by ngranja19 ↗ HN
Today my web app ViralQuotes was hacked. My DB was erased, instead, there was a new table called Warning with a message 0.08 BTC to recover my DB.

After 5 hours I was able to rebuild my DB, but unluckily I lost all my historical data within my 400+ users data among them.

Of course, there are some lessons learned about it:

Don't forget to set up regular backups. I know, I was really silly for not doing it, but I never thought that someone would hack my insignificant website. So, stop what you are doing, and go and set up some way to back up your DBs and significant files at least once a day. Some providers offer it for a few extra bucks a month. Don't be like me, maybe your product is not generating thousands of dollars and you think that no one will waste his time hacking your site, but remember that is important for you and that's is enough.

After looking into how that could happen, I realized that I pushed to my server my .env file with all the database credentials in it, which Is pretty simple access to it especially if you use Laravel. I found out that is a pretty common mistake, If you google DB_USERNAME filetype:env you will find thousands of Laravel env files exposed Therefore, remember to actually set your variables from the .env file as environmental variables in your server and destroy any .env file that is around there.

For sure there are more lessons to learn about this, but I realized that I Would like to hear if some of you have had bad experiences like this one, and what do you recommend to prevent them?.

Cheers, Nico

49 comments

[ 4.9 ms ] story [ 117 ms ] thread
alexpetralia
Just wanted express my condolences, that sounds like a huge pain. It at least sounds like you are using the experience profitably with some thoughtful reflection. I wish you the best in a speedy recovery!
ngranja19
Thanks a lot for your kind words. And you are right, it was awful to discover that my first product was hacked and I lost a userbase that it was really difficult to collect being nobody in internet. But at the same time I realized that maybe is more common than we imagine, and there are several stories around there that I would like to hear to be more prepared in the future. That's why I write about it. (I don't post really often because I don't feel confident with my writing) Thanks again for the Good vibes!
ryanmccullagh
To maintain dev ops best practices such as prohibiting nginx from sending dotfiles (.env, .git, etc) takes time. It’s understandable that you made a mistake. But how did they get access to the actual dB? Did you expose it over the public internet?
ngranja19
Exactly. You could access to it like viralquotesonline.com/.env. I did several mistakes for sure. It was my first project launched to the world and the first time I used Laravel, is a bit embarrassing to be honest, but hopefully helps other to realize that they are in a risk too.
ryanmccullagh
Well it’s a good lesson and definitely don’t pay them. but I’m still not sure how they accessed the DB. Did you expose port 3306 over a public IP?

Small things like this are one of the reasons I founded the PHP hosting platform, Amezmo where secure production ready container instances can be launched in seconds.

jlis
What I don't get is how the .env file was even served when Laravel uses the public/index.php as starting point, so your root Folder should have been the public/ folder, not the application folder.

Mistakes like this happen, but it takes courage to post about it. Thanks for sharing it.

ngranja19
I'm not totally sure I'm not a Laravel expert I just use it as a backend api so I believe I exposing the entire Laravel folder instead just the public. And yes is embarrassing haha.
saluki
I was curious too how this could happen, since the .env wouldn't normally be in the public folder.

Laravel is a great framework, check out Laracasts.com and forge.laravel.com, it's a deployment tool that will spin up a VPS on AWS, Digital Ocean, etc. with the proper configuration.

tluyben2
But how did that work in the first place? Not that it is good practice etc however, normally your webserver for Laravel would be pointing to $MYSITE_DIR/public and .env would be $MYSITE_DIR/.env. So how did you end up getting it exposed there?
priom
Sorry to hear about your situation, it really sucks. Curious, if you can shed more light on how the .env was exposed?
thegoleffect
I don’t know for certain, not op, but sounds like since they pushed the env file, it may have been accessible via URI and since it had all the credentials and host info to access the database, thats all the attacker messed with... and that is more than enough.
ngranja19
Haha exactly that was what happened (I guess). My mistake was not only pushing the .env file but also how I stored my Laravel files in the server leaving everything in a public directory where everyone can access to it... :Facepalm:
gentleman11
I usually make sure my .env files are in my .gitignore, but honestly its just my memory that prevents me from slipping up one day.

Is there an automated strategy that protects against slipups like this?

enitihas
Use a git template for .git/info/exclude. This will make sure that every new repo you create or clone will have by default exclude list. Git templates can be used for other stuff too. I use them for having a pre-push hook to run a clean build.
throwaway8941
Put them into your global .gitignore

    echo -e >>~/.config/git/config '[core]\n    excludesFile = ~/.config/git/gitignore'
    echo >>~/.config/git/gitignore '.env'
makison
Laravel should/does have it on ignore by default.
mc3
Sorry to hear that. The obvious lesson is "backup", maybe the less obvious lesson is to pay a bit more for a service that does this for you (or a cloud provider might have that included) and have peace of mind.

For similar reasons I use a cloud backup for home PC files that just syncs it all up. When I was doing it manually it was a once a year affair because: too busy!

ngranja19
Totally, I use digital ocean and I can set up a daily backup just for 2$ extra by month ( my doplet costs $10).
mc3
Yes I use that feature too. Seemed like a no brainer. I can even recover the image pretty easy anyway by typing in the 10 or so commands again.
allthetime
How is your db setup? If you are running your own instance, just make a cron script that dumps the db every so often (more often if its small) and uploads it to s3 or something.

You don't really need server backups/images if you document/script your server setup properly.

ngranja19
I'm thinking on doing exactly this. Now that this happened I updated all the scripts and is really easy to rebuild it if something happen again. But I need to backup the data
michaelmcmillan
That sucks! But the real lesson should be that you do not expose your database on the internet (0.0.0.0 vs 127.0.0.1). That way it doesn’t matter if you leak your .env via your webserver. Never expose sensitive services.
ngranja19
If I use 0.0.0.0 I'll still be able to access my DB remotely? sorry for the ignorance...
terwey
You can setup an SSH tunnel to your remote machine. Some DB explorers offer this tunneling out of the box (e.g. Sequel Pro for Mac).

Top hit on Google for how to setup a SSH tunnel for MySQL: https://linuxize.com/post/mysql-ssh-tunnel/

ngranja19
Oh, but that is what I do, I set up an ssh tunnel, so I'll change it, thanks!
GauntletWizard
You should post again on what you're going to change, because i (at least) read that advice both ways, and I want to make sure you're reading it the way that was intended.

If your database is open to 127.0.0.1/{8,24,32}, it is open only to the local host; that is the non-routable localhost address and it's not accessible to the internet.

If your database is open to 0.0.0.0/0, it is open to the internet and anyone can attempt login. This is bad. Presumably you also still need a password, but never assume. Always have defense in depth - at least two things need to go wrong/be wrong before you're compromised.

mc3
The real lesson is there are multiple lessons. Layered security and risk management. Back up automatically, don't expose database, don't expose configuration files, use tools to try and exploit your own site first, try to avoid passwords in files at all if there ways to do so such as credential stores, encrypt data at rest, no plaintext passwords. When one layer fails the next layer turns this from a catastrophe into a mere headache.
tortasaur
After reading your post, I thought "these things happen; they'll figure it out with experience."

Tried going to clientsite.com/api/.env on a whim, and leapt out of bed.

redstripe
Bots also regularly check for .git in common folders:

404 - GET /admin/.git/config 404 - GET /shop/.git/config 404 - GET /beta/.git/config etc..

or adhoc backups using the current date -- stuff that you think nobody would even try.

404 - GET /2020-02-27.zip 404 - GET /backup20200227.zip 404 - GET /backup.zip etc...

No webserver should be serving .folders in it's default config.

imvetri
Sorry to hear that. As a request please remove the Google keyword suggestion in your post. More people will be aware of the technique now.
jlgaddis
> More people will be aware of the technique now.

Good. Perhaps more people will learn how to properly configure/ deploy their servers and apps.

ngranja19
But I believe that hackers already know this vulnerability, so it's better that everybody is aware of this issue so they can be prepared
rutthenut
Some useful points made in other comments, tied-in to your particular problem/exploit.

I would recommend you get familiar with OWASP, and especially the 'top ten' security issues that they publish. See https://owasp.org/www-project-top-ten/

Operyl
In addition to setting up backups, there’s another step you should probably think about: make sure whatever credentials you are using to store those backups (if using an object storage service) are only allowed to add/append/upload and _not_ delete. I have seen attackers wiping backup locations a few times now, and it isn’t immediately obvious to people.

Sorry it happened to you, but it’s a good lesson learned. Better luck in the future.

ngranja19
Thanks! I'm learning a lot with all these tips!
dana321
Its a suckerpunch.. If there is a way in, they will find it. You have to vet everything.

I had a side project compromised through some supplementary php files that came with a javascript library. Luckily, i didn't have any users on it.

Keep everything up to date, including your server software, composer, javascript libraries etc.

Make sure you run a clamav on linux, it will catch any intrusion writing scammy files and rename them.

All passwords unique and made using a password generator. That way, if you are compromised it is only one password not the keys to everything.

Don't keep non-user stuff with predictable uri (like /admin/ /phpmyadmin/ etc.)

One-way backups.. Allow your backup server behind a firewall to login to your site and do hourly backups of all the data.

Disallow access to any files beginning with dot in your webserver configuration. I'm pretty sure nginx does this by default.

ngranja19
Thanks for all the recommendations, so useful!
machinecoffee
In your haste to get everything back up and running again, don't forget to change the database credentials or you'll be back in the same boat again :)
dana321
And block access to .env files too!!
ngranja19
Haha yeap that was the first I did. Thanks
sergiotapia
Woah what, you're not kidding:

https://www.novochem.net/.env

Laravel should not even render this .env file by default, way too many results.

Laravel has no github issues, so I don't know how to let the author know. https://github.com/laravel/laravel Someone ping him on twitter

pier25
But that would be an Apache or Nginx config, no?

Afaik PHP is not the webserver.

makison
Exactly. Nginx/apache is not pointing to /public/ directory as root of the application/domain. That's why .env is accessible.
franzwong
Where do you host the webapp? Some cloud platforms provide security protection and you should use it.
longtermd
I would love to see a blogpost "security best practices for SPAs" (single page applications)
woranl
If the hacker gained access to your database, he/she can also create a backdoor file to gain shell access via the output sql. You might want to check for any malicious/suspicious files in your server that will take shell command as http parameters.