Ask HN: How do you continuously monitor web logs for hack attempts?
what is the generally accepted best practice to be monitoring web logs for anomalous accesses ?
do you guys just throw cloudflare in front and forget about it ? Or do you have engineers who work like data scientists - eyeball the logs ?
I have heard suggestions of using a firewall - but I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you.. or is there a generally accepted practice that has evolved for even regular engineers to do this ?
P.S. I'm personally more interested in an API-focused answer, but I guess the same thing applies for websites.
231 comments
[ 3.0 ms ] story [ 311 ms ] threadSince I'm this bad, I assume that any monitoring tool I could install on the boxes themselves, up to and including all my logs are compromised. I'll usually still install monitoring tools for compliance, basic diligence and because they make wicked graphs that impress my superiors.
Since I'm that bad, I'd rather not invest 40 hours a week combing through fake logs unless I'm validating that I've taken care of the basic security fundamentals for my threat level.
Rather, I like to make sure that my security fundamentals are taken care of so that I can stop the simple attacks. If my threat/compliance model dictates, from there I'll possibly step things up into a web application firewall (WAF).
But, through all of this, I know that there are people out there who are so much smarter than me that the best I can do is little more than an annoyance for them. So, again, I like to assume that I'm badly compromised. This implies that I'm very careful about what data I keep, and I take efforts to protect my data at rest.
Beyond that, tripwire or it's modern equivalent. Traffic monitoring, "deep packet" inspection on the network side. Run all outbound through a proxy.
There is this Linux thing called IMA which in some ways reminds me of tripwire (Integrity Measurement Architecture).
Unless something has recently changed it is some of the most poorly documented software on the planet.
I’ve never been able to configure the thing or figure out what it’s doing or... really anything besides find the homepage.
It’s also just not worth the trouble. Globally open SSH port? That’s like 1992 level Linux box in your parent’s basement stuff.
The documentation isn't the greatest, but this page explains the concepts well enough and was easy to find using Google.
https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
1.) Read through the documentation.
2.) Pour myself a scotch.
3.) Read through more documentation.
4.) Start drinking right out of the bottle.
5.) Try something.
6.) Get an error or experience a spectacular failure.
7.) Use StackOverflow.
8.) GOTO 5.
-------------
In good news, it worked - the light went on and now I'm confident that I could adapt Fail2Ban to suit any use. But, in bad news, honestly, I feel your pain and understand where/why you are getting stuck. Fail2Ban is a very complicated, yet extremely useful piece of software.
I'm not going to post the exact configuration files I use, but the GitHub repo for fail2ban contains examples.
https://github.com/fail2ban/fail2ban/blob/0.11/config/filter...
That's a filter that protects Apache against the 'shellshock' (https://en.wikipedia.org/wiki/Shellshock_(software_bug)) vulnerability, for example.
http://www.kylheku.com/cgit/txrban/tree/
It consists of several processes written in the TXR language. One for monitoring Apache, one for sshd and one for Exim (mail server).
TXR's standard library has "tail streams": streams which automatically follow a rotating log file.
These txrban scripts simply open tail streams on their respective log files, use TXR pattern matching to scan for what they are a looking for, and then use functions in the tiny txrban utility library to do the banning. Banning is instigated by issuing iptables commands.
The banning isn't persisted, but when you restart the scripts, they retroactively replay everything by scanning the available logs from the beginning.
Banning events are reported via syslog. A "dry run" configuration variable allows testing for the rules without actually modifying your iptables.
I never packaged this program into a redistributable form. It contains some hard-coded log-file paths that work on my server, and no documentation.
Every month or whenever you choose, review your access log and see what kinds of attacks are happening, then use fail2ban to block those.
If you target 404s for non-existing PHP files, especially things like "setup.php" you can nail most would-be hackers before they can find vulnerabilities. Use a low-retry count and a high ban time, that'll kill the scanner-bots.
Other than that, make sure your webserver is secure. If your server is secure then you don't need to do anything because nobody can get in. Secure first. Aggressive next.
Of course, there some basic security practices. Firewall, fail2ban or similar to stop/ban misbehaving IPs, monitoring systems metrics to detect abnormal activity/usage (not just for intrusion detection), and maybe feeding logs to i.e. ELK to notice new/unknown patterns.
I suppose you could add a honeypot page that took form submissions and looked for any sequence attempting a CSRF, XSS, or SQL attack and ban the source there as well.
I started working at a cybersecurity company - our product is ridiculously stupid and I try and work as little as possible. In the time I’m not working I try to at least learn stuff. Since I’m at a cybersecurity place I sometimes try to learn about cybersecurity. It seems that even though for profit cs companies are making crazy dough, a lot of open source solutions out there do the same thing. There are so many attacks - they can be anywhere from the application level to the level of physical devices like routers having some kind of exploit. For any type of attack there is a good or multiple good open source tools that can be used to execute or defend (depending on what color hat you have), and a ton of them come standard installed on Kali Linux, but before you learn these you have to understand networks. As a python/Django dev I didn’t really have a deep knowledge and still don’t have a deep knowledge of this. Nor have I done more than dabble with any of these tools. But I know that there are a lot of videos on how to use these - including this 15 hour (!) one https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM - if you were to go through that you’d probably be a security wizard
There is nothing I love more than the "just throw {Cloudflare,Akamai,Fastly,etc} in front of it" crowd. If you are going to deploy a WAF [1] it _needs_ to be at the edge of your datacenter or cloud environment. No matter what fancy feature your provider wants to sell you to fix this, there is a way to bypass it.
* Make sure you are logging centrally, and logging the right things.
* Make sure someone is looking at the logs, it doesn't have to be a full time person, but it needs to be someones job.
* Depending on how big you are, hire people to break in on some regular basis. It could be anything from hiring a cheap contractor to run a bunch of automated tools, to full on 10 person adversarial teams. They will find some stuff for you to fix, but more importantly you will know whether the first two points are working based on what you see.
1. https://www.owasp.org/index.php/Web_Application_Firewall
Could you please elaborate on what some of the right things generally are?
Okay this sounds incredibly interesting. Are there any open source groups that do this "for free" sorta? Not asking to get free working, asking so I could possibly jump in and learn.
imho try to hack yourself first
https://hackyourselffirst.troyhunt.com/
https://microcorruption.com/ is a quite a good one to get started with. (Because embedded systems are smaller and do less, they are easier to reason about, which makes it easier to see vulnerabilities.)
See also: https://www.reddit.com/r/securityCTF/
More broadly, the (with-permission) attackers are known as "the Red Team", and the concept is used against computer networks as well as in literal war-games by the US government.
From your profile it looks like you work for a big company. Ping some people in security and find out who runs your "Red Team" (if you have one). Make friends with them.
If your WAF is the edge of your datacentre, it will likely be ran by some sepperarte team with a generic ruleset, literally be MITMing all your connections, but more importantly it will be attackable via your company VPN or some other side channel like B2B connections generally, that you probably have no control over.
Your WAF, if you choose to have one, should sit directly in front of your app and have rules tailored to your app. Even those on VPNs, or in the datacentre , should be forced to traverse the apps WAF.
That said, the WAF logic is largely what you're going to do in your input validation layer, so unless you feel you need to double bag it try and to mitigate unknown framework or library issues, or are having another person write the rules, then you're just repeating work aimlessly.
When some dickhead manager buys an insecure http service they want on the web, you should probably quit. Failing that put a WAF in front of it.
I generally recommend having the WAF live in front of of apps, but with a small gap that allows your scanners to still access the app. It is important to still be identifying vulnerabilities that your WAF might block, because they aren't perfect.
Just curious, how this is covered from legal standpoint? Do you ask clients to sign some consent document to release you from liability from damage from hacking by automated tools?
I like to remind people that having someone on retainer that breaks your stuff with an automated tool is better than having some random guy on the internet break your site with the same tool. Either way it is eventually going to happen, but at least I will immediately notice and contact the right folks to help get it fixed.
Just good web logs is enough to detect all kind of malicious anomalies.
Splunk machine learning functions can cluster data and detect anomalies in anything without rules or signatures:
https://imgur.com/a/IFhUZuH
If anything happens, THEN you look at the logs. The signal/noise ratio of the average web host logs makes combing through them just not worth it.
There are tons of products out there - have a look at OSSEC (https://www.ossec.net/), it's open source and free.
We use CloudFlare so we feed the IP bans into CloudFlare's firewall through their API as well as local IPTables. (We also use CloudFlare's WAF - we like redundancy in our paranoia.) If you had more demanding requirements you could look at mod_security for Apache or Nginx.
We also implement rate-limiting using Fail2Ban, which helps to trip the dumber automated scans.
Every time Fail2Ban blocks an IP address, it emails our admin team with the matching log entries and a whois on the IP address, which we then eyeball to make sure there's nothing weird going on.
TBH, easily 90% of the IP addresses we block got caught looking for 'wp-admin/admin.php', which is probably a good reason to never use WordPress.
99% of IP addresses we block are from outside our target market - we're a very local/regional company - so we don't really have to worry much about blocking legitimate customers. Very rarely our support team will get a request from someone that's been blocked and we'll unban the IP address, or set it to a JS/Captcha challenge at CloudFlare.
Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.
But yeah, like every tool, there's potential risks/down sides/things to watch for. There's no silver bullets!
These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.
If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
Isn't it lovely when every other referrer is trying to promote some online pharmacy or testing to see if you have a wp vulnerability?
Sure you can filter it, but I'd rather deal with garbage at the source.
Our technical team is very small and the company entirely dependent on our website and our reputation, so we take multiple approaches to securing the site and servers. Each little thing we add helps my boss sleep better, and that alone is probably worth the initial effort.
I definitely wouldn't do what some people are suggesting, manually reviewing logs/fail2ban matches and blocking IP addresses by hand. That's a waste of effort - and would probably be a full time job, too.
It is not a replacement for the actually security of keeping things updated and having good passwords/client certificates.
Log monitoring is good for finding bad access patterns, but can't be trusted to find a problem as it happens.
Look for repeated errors. Hitting the same service over an over? Login attempt failures from the same few IPs? It's either a script, or the Bing crawler. Write application level code and network level configuration to mitigate the problem.
How about legitimate users doing social engineering attacks on each other? Sybil attacks in ratings or voting? Attempt to reuse mobile purchase tokens on different accounts? Some new SSHD credential leak zero day? Using your cat picture upload service for storing pictures of dogs? Each of these need different approaches, and a good logging system that can change with your needs is super helpful.
But you can't read the logs in real time, looking for bad patterns. That needs customized analytics.
OSSEC [1] is my go-tool for this type of tasks, it offers a language-agnostic interface to monitor any type of events. I used it along with my team to power the early version of the Sucuri Firewall, which later became the GoDaddy's security system. The project has matured a lot and is heavily tested using millions of HTTP requests every day. We also implemented a behavior-based algorithm to detect malicious requests, this part is not open-source but I can tell you that it was relatively easy to integrate it with the core interface.
Splunk [2] is also a good option, slightly more difficult to configure though. I have had the opportunity to meet some of the developers working in this company during a short visit to their Vancouver office and was genuinely surprised with the level of engineering that goes around this product. Definitely, worth checking it.
NewRelic [3] while the purpose has nothing to do with security, it also offers an easy-to-use interface to monitor events in the system. With some extra configuration you can make it work as a rudimentary firewall and benefit from all the great features that the product provides.
Disclaimer: I am an active contributor to the core features, but I am neutral on my recommendation.
[1] https://en.wikipedia.org/wiki/OSSEC
[2] https://en.wikipedia.org/wiki/Splunk
[3] https://en.wikipedia.org/wiki/New_Relic
Most of the things you can easily set up to watch for security events are looking for problems you simply shouldn't have.
For instance: you can set up fail2ban, sure. But what's it doing for you? If you have password SSH authentication enabled anywhere, you're already playing to lose, and logging and reactive blocking isn't really going to help you. Don't scan your logs for this problem; scan your configurations and make sure the brute-force attack simply can't work.
The same goes for most of the stuff shrink-wrap tools look for in web logs. OSSEC isn't bad, but the things you're going to light up on with OSSEC out of the box all mean something went so wrong that you got owned up.
Same with URL regexes. You can set up log detection for people hitting your admin interfaces. But then you have to ask: why is your admin interface available on routable IPs to begin with?
What you want is an effectively endless sink for developer logs (Splunk is fine but expensive, ELK is fine but complicated to manage), and you want to add structured instrumenting logs to your applications in all the security-sensitive places, like authorization checks (which should rarely fail), and especially a very solid, very long-term audit trail of admin-like actions, so you have some prayer of tracking insider misuse. You could instrument your SQL queries and catch any query failures --- which should also be rare. That kind of stuff. I don't think there's a good shrink-wrap package that does that.
But really: focus on removing attack surface, rather than continuously monitoring it.
You buried the lede on this one.
Don't use any of those scanners, all of which look almost exclusively for vulnerabilities you don't have. Do look into getting a Burp license for every developer on your team.
By all means, harden Nginx. Don't pay SpiderLabs to do it for you.
Your message is being lost/ignored as a result of your delivery.
We want to understand your message, but you aren't giving enough background. I don't trust your message because there is no supporting background. If you're going to tell me to not do something, either:
A) provide a compelling alternative (not doing $it is not a compelling alternative to $it in nearly all cases) B) explain in detail why you shouldn't do $it. Provide more details if asked.
I want to believe what you are saying and be convinced by it, please help me. (Note: I'm not trying to be a dick, I've just known a lot of people who are perpetually ignored because of this communication style and take forever to learn that lesson because no one bothers to say anything)
You are succeeding without trying. You're essentially complaining some free advice does not fit your exact expectations and specifications. Which is fine, lots of free advice doesn't. Trying to pass this complaint off as some kind of favour to the person giving you free advice, though, is... less than great.
Please speak for yourself alone. tptacek's comments in the thread are quite clear.
Example, same author: Cryptographic Right Answers.
(Disclaimer: I work with tptacek.)
The sense of giving rational arguments lies not only in the persuasion of the parent poster but also on the information of anybody else who is reading. And simple "Don't do most of this." and "You almost certainly don't want to set up a WAF." are just crappy answers which are based on authority and therefore meaningless.
I'd also like to add that you may want to work on the tone of your comments if you'd like to get enough karma to really participate in the community. Your comment history seems a bit confrontational and argumentative. There are better, friendlier ways to make the same points. I'll defer to the guidelines[0] to explain how you may improve your communication skills.
[0]https://news.ycombinator.com/newsguidelines.html
I hope your day gets better.
Or you go to the doctor and expect him to justify your treatment down to the minor detail? You sound like you talk to your doctor about $overpriced_drug.
It's just reasonable to ask for rational arguments. That's the whole point of the rants here.
First of all - it takes less than an hour to set up a WAF, and it doesn't worsen your security position. Not only is it free, it's fast and easy and will demonstrably be better than no security at all.
Saying "don't put up a firewall because it isn't perfect" is stupid. You don't have to be a security guru to get that, which obviously several people have, or they wouldn't all be giving you crap about it.
Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous.
If I have a computer on the internet, I put a firewall up. Is it defense in depth? No. But that doesn't mean it won't stop the most basic, most common attacks, and it takes no time and no skill to apply it.
I don't know what weird thing you have against WAFs other than apparently they aren't cool or sophisticated enough, but 1) they remove attack surface, 2) they filter the most common drive-bys, 3) they can be both passive and active, 4) they're free, 5) they're easy, 6) they're quick to deploy.
How is it 2018 and I have to explain to you of all people that the web equivalent of iptables might be a good idea if you have zero security?
They do what you configure them to do. Without knowledge of what's allowed and what's not, they're going to have either false positives or false negatives. And if you have that knowledge, you can fix those issues in the web app rather than hide them with the WAF. A lot of preconfigured WAFs will remove specific matches you don't care about anyway rather than attack surface.
> 2) they filter the most common drive-bys
Common as in known proxy or webapp exploits? If that's the case, why not patch them? (There's a reason to use WAF if you can't patch but that costs time/money to get right and test)
> 3) they can be both passive and active
I'm not even sure what passive WAF means... just logging?
> 4) they're free
Only for very small traffic.
> 5) they're easy 6) they're quick to deploy.
See point 1. If you can't put the right restrictions on the app, why do you think you can put them in WAF? If they're not in the app in the first place, why do you think this way is quicker? It's not a magic box will know your requirements. You need a proper setup or the only thing you achieve is removing obviously silly cgi-bin requests from your log files.
I'm not even against WAFs. Just think the gain from them is been none and minimal if you reduce it to "turn it on, it's cheap and easy". It's really similar to iptables - you can turn it on, but why are you doing it? (On a server anyway) If you have a specific reason/route to protect - great. If not, make sure you bind non-public services to local interfaces and you don't need iptables.
> it takes less than an hour to set up a WAF
Not if you want it to do anything useful. Default WAF rules are mostly useless and come with bypasses -- kinda like XSS filters in browsers, and browsers are in a way better position to judge if XSS is going on or not. You can sort-of get useful WAF rules if you encode most of the invariants of your system in your WAF -- but your app is in a strictly better position to do that sort of validation for you instead.
> and it doesn't worsen your security position.
Yes, except for the shiny new TLS MITM, the pathological parser performance cases, the new attack surface in that WAF itself (hands up if you've seen XSS in a WAF interface) and the continued operational cost of having new infrastructure that sees plaintext... Sure! And then I'm being generous in assuming we're not talking about a virtual appliance or as I like to call them "mystery *nix box you can't patch or audit".
> Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous.
Also, unlike WAFs, it's effective.
To be clear: I am not conceding your point about cost. It does not take "months" to identify systemic appsec issues. It also does not take an hour to configure a WAF and have it do something vaguely useful for your application.
Anecdotal evidence: Latacora literally does this systemic-discovery-and-remediation-plan thing for a living as a part of our engagement. "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS.
> ... the web equivalent of iptables might be a good idea
Are you really sure you want to stand by that equivalence? How often does one come across iptables bypasses?
Yes. Both require some knowledge of how to use the tool and rules, but both are not difficult for even a novice to pick up quickly. Not to mention both can be done by either a dev or a sysadmin in the cloud right now, neither of them require in-depth knowledge of the code, and the scope of the change is simple enough that it can be tested and applied quickly.
> "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS.
The discussion was about a dev or sysadmin fixing all this themselves ("taking appsec seriously" etc). Not just running Burp on their website. After they identify the vulns they have to have meetings to discuss fixes, figure out how to patch them and test that it works, go through QA and then roll into production. A fix may involve several systems and most people's deploy process is painful. I've never seen that take less than three weeks, and six is more reasonable. Can you do it in less time? Of course. Does that actually happen? Not without paying a consultancy $300k to hold your hand for two weeks.
It is completely plausible to get a WAF up and running and have it provide real protection immediately. CloudFlare's is probably the best example of this, considering the extensive add-ons they provide based on the intrusions they've seen. I don't like CloudFlare's TLS MITM either, but tens of thousands of their customers don't seem to mind.
Just like iptables, ModSecurity doesn't do anything at first. You have to actually make some decisions. Does that mean you shouldn't spend the extra 15 minutes to read a manual and enable some protection?
Earlier it was suggested that protecting against Drupal or Wordpress vulns is unnecessary. I disagree! There's no reason not to protect against drive-by vulns. Filters for XSS/SQLi, unnecessary HTTP features, malicious bots and scanners, directory traversal, RCE, etc are well worth preventing whenever possible.
As for false positives, they're not as bad as they used to be. There are thorough guides designed to help you identify and remove them from your site. https://www.netnea.com/cms/apache-tutorial-7_including-modse... https://www.netnea.com/cms/apache-tutorial-8_handling-false-... For a sysadmin this may take more time, just like it would if you were using iptables to secure a black box. A dev can change the filters immediately to suit their app.
But even if you don't do any active filtering, the passive filters can at least alert you that something is happening so you can go fix it. This is so standard that Kubernetes includes it as an option in their Nginx Ingress Controller. This guide can get you up and running on your existing K8s install in 10 minutes: https://karlstoney.com/2018/02/23/nginx-ingress-modsecurity-...
This points out how WAF/NIDS is just a speed bump for a serious attacker.
You may conclude that they don't really 1) remove attack surface 2) Unclear if they really do this, and if they do what value there is 3) See the papers for detail on the differences 4) Zero cost to purchase, but what cost to maintain, run, keep up? 5) see #4 6) even if you have a large web farm?
This paper was writen 20 years ago.
Is this advice assuming that it's devs who want the WAF, rather than ops or management?
For devs clearly it's not what they should be focusing on. But saying something isn't worth doing because it shouldn't work is a terrible security principle.
So, I'm just going to go with: no, if you're really building an application and care about application security, no, you shouldn't waste time setting up a WAF tool designed to help 10,000 employee insurance companies make sure their Wordpress marketing site isn't exposing some vulnerability from 3 years ago.
Not wasting time with dumb stuff like this isn't "terrible security"; every dumb thing you do exacts a cost from good things you could be doing instead.
If your counterexample is just reinforcing the parents erroneous point that WAFs only protect against specific applications with known vulnerabilities, and that not all websites have "doors", then you are missing the point that many WAFs have detectors for fairly generic sql injection and path traversal techniques that could very well find problems with even custom application endpoints....
So, no, not really interested in generic SQL detection rules in WAFs.
It's like throwing gauze over an open wound. Some companies don't care enough to put thought into healing the wound altogether, whereas some just wrap it in gauze and hope they don't bleed out. And it usually works - at least for a while, until you get big enough/become a lucrative enough target to a persistent hacker.
It isn't a case of "more security tools == better security", it's a case of "why do I need these tools in the first place". If you aren't analyzing attack surfaces and building security into your software development process, I can only hope the code you write isn't accessible externally.
* using 'you' incorrectly here, speaking to those looking to apply these tools to their setup.
What's wrong with that use case though? If you get hired as the new CTO, you're not even going to know all the managers, let alone all the developers.
Clearly auditing all code, doing developer training, etc. is the right longterm approach. But putting a security culture in place and actually getting things right in this kind of environment is a multi-year process.
I'm writing for an audience of people whose businesses own the applications they're deploying. If you control code, you should have at least the security staff required to cut out the extremely low-hanging fruit attack surface that things like WAFs address. Most startups, even with 20+ developers, don't have any full-time security people, and that's fine, because even at the ~20 engineer scale, it's still perfectly feasible to cut out all the WAF-addressed attack surface for your environment.
On rare occasions I'll write advice or give an answer from the perspective of enterprise secops, but unless I say I'm doing that, I'm never really writing from the enterprise perspective. Reddit r/netsec is the right place for that POV.
Company X, a content publisher, has a wordpress site with a bunch of plugins. They hear that the security record of such a setup is less than ideal. They buy a WAF solution to protect them.
Company Y, also a content publisher in the same space, realizes that they don't need dynamically generated content. They don't need tons of JS. They start publishing static content and what little dynamic functionality they have is well-compartmentalized, with interactions to the outside world carefully audited.
Company Y could also get a WAF, but why? It increases the attack surface, Company Y probably don't have the time/expertise to audit it, it is not clear what benefits it will have.
Thanks !
If you follow the Geekflare link to modsecurity, and then follow the link there to a download page, there's this warning:
>NOTE: Some instabilities in the Nginx add-on have been reported (see the Github issues page for details). Please use the "nginx_refactoring" branch where possible for the most up to date version and stay tuned for the ModSecurity version 4.
That's not something I would suggest throwing in production, given that warning has been there for years.
The post itself describes Shadow daemon as "probably defunct".
Ironbee's last commit was two years ago and I get an nxdomain looking for their website.
AWS WAF is both expensive and and extremely manual. You won't get anything out of it without a major labour investment.
Nginx's built in WAF is only in the pro edition, and outside the price of many people.
This whole discussion however depends on your customer. I have an agency I support where I have to provide a report on every single hack attempt. That means if my Rails application gets hit with said three year old Wordpress exploit, and the web server "404", I have to report on how we "blocked" it. And no, "not applicable" is not an answer.
So we throw money at commercial products. And sometimes I recommend doing so. But like security can involve just being aware of your risks, this use of a WAF is about being aware of the real problem you're solving, which is a paper work based one.
Edit: updated to be more clear
Yeah, I should have been more explicit. When I say build secure applications, I imply the full stack including infrastructure. I updated my comment accordingly.
Remove? Probably not. There are many options for minimizing the attack surface with http.
The first two use cases are all about traffic visibility, which has been, and remains a big problem. If we're talking about improving security, the third use case—virtual patching—is the exciting part. The story goes something like this: you discover a vulnerability in your code but you can't fix it immediately, for whatever reason (take your pick). In practice, it may be weeks until the problem is fixed. However, if you have ModSecurity installed, you can write a virtual patch for your problem, often to fix it 100% reliably.
However, it's very important to understand that virtual patching doesn't mean just slapping together some rule that appears to be doing the job. Blacklisting doesn't work. To properly virtual-patch, you have to fully understand the vulnerability and the entry point to the application. You have to carefully whitelist all parameters in that particular location. In other words, you have to almost replicate the work that should have been done by the developer.
Even in this context, it's not all rosy. Virtual patching requires a great deal of expertise to do properly. It's almost always better to spend your time fixing the actual problem. Sometimes, however, that's not possible to do in a reasonable time frame, and that's when a WAF (according to my definition), can be a life saver.
- - - -
There's few people I am more pleased to see commenting here on a topic like Sec more than yourself Ivan, your work's exceptional. And the tools you've provided like Hardenize and SSL Labs have saved me countless hours.
Your books have been an immense source of knowledge for me (and I'm relatively a newcomer by my colleagues standards).
Really pleased to see you here on HN. Though, I'm not entirely surprised. The best of the best seem to haunt this place.
I Also have some mod_rewrite rules in to 403 some URL’s that bots currently hit, mostly because I don’t want to spend 200ms of PHP time for the bot to discover that it’s exploit didn’t work.
I accept that when you’re hosting websites for a load of other people you’re gonna need backups too, that’s the bottom line.
Converting our WordPress sites to static sites greatly simplified and improved our infrastructure.
You have railed against network isolation as defense in other threads. Locally-scoped IPs, VPNs, and ACLs are all about network isolation. Care to shed some light on the inconsistency?
* Your logs should be there to help you find out what happened in the event of a compromise, not to detect it in progress.
What OP should be monitoring is the activity that occurs on the system - logins, processes spawned (e.g. php or mysql suddenly spawning shells for no reason), changes to content, resource volume spikes and as the parent said, configuration changes.
Much of this can be done with a variety of tools but there isn't really a one-size fits all approach. I'd start with resource spikes as it's easiest and work up to an OSSEC configuration reflecting the actual environment.
Also by way of middle ground on the log front, Graylog2 is a happy medium between Splunk and ELK, but it's better to log fewer (but more useful) things than a load of noise.
In my experience, monitoring this takes about as much effort as adding a new apparmor / selinux / firejail / ... rule which would block it in the first place. I mean, with processes spawned we're almost at auditd level already.
I vehemently disagree with this assertion. I also disagree with tptacek's asseration that any admin interface should be inaccessble from the public Internet. That's just not how a lot of small businesses, non-profits, etc. operate. 2FA? Sure. Geo-fencing? Keeps out noise. Basic auth / some front-end to minimize app-specific attack surface? Perhaps.
Saying "you shouldn't look at logs to detect anomalous activity before the fact" is awful. So is "If you have and admin availability publicly, you're screwed".
My saying is "Security is a spectrum". A bank should not have their firewall admin interface publicly available. A community newsletter blog would probably be alright.
There are organizations on AWS that have publicly-routable admin interfaces. They should all be working on plans to migrate away from that, and, in the meantime, they should be extra careful about authentication and monitoring.
If you're too small to have a non-routable admin interface, you are definitely too small to pay for security products.
As for the content of your comment, it's hard to distinguish what's aimed at me and what's aimed at him, so I'll assume everything aside from your first sentence is aimed at him.
If you do security for an enterprise with dozens or hundreds of apps that you don't develop yourself, you might not have the luxury of only trying to reduce attack surface, or instrumenting 3rd-party supplied code.
In a realistically achievable world for virtually every startup, no routable SSH. No security rule allows 22/tcp from a routable address into a VPC. You want to SSH into a host, you VPN to your network first. This is how most organizations with security teams set up SSH.
If you can't get there, at least get to jump box SSH --- 22/tcp is allowed to exactly one isolated host from the outside world (some people filter source addresses to their jump boxes, I don't bother); to get to real hosts, you bounce through that one.
A network where you can randomly connect to the sshd on real servers is one most security professionals will assume isn't being run competently.
When I was young I too used to try implement detection and banning but realised it's a waste of time. Key only access to ssh and make sure you at least subscribe to security updates for all packages is enough.
For our own app we went the ELK route due to the number of log messages coming from our own application. But yes as you said it's a pain to manage. Also creating kibana graphs/views/desktop for each type of log is also painful.
A good service I believe would be someone to offer a turnkey/shrink wrapped managed ELK for self hosting. Not only managing the system side of it, but also offering services to help create custom kibana graphs.
Using a short ban time keeps the load of running fail2ban to a minimum. I at one point was banning the kiddies/bots for a week and 40% of the server load was chewed up just running fail2ban.
The solution I came up with to this problem is to use a custom ban action that adds an IP to the blocked ipset list with a given TTL, but set fail2ban to think that the IP has to be only banned for 5 seconds or so.
The result is that ipset purges the IP from the list after the TTL expires, but fail2bans working set is a 5 second rolling windows of blocked IPs.
This works great and it's being used in production for a couple of years.
Of course there are ways to deal with it, but that's just extra work.
You don't really need fail2ban at all. Just don't log failed password attempts. Key access only, no one is going to get in.
Just out of curiosity has anyone with a cryptographically strong ssh password ever been hacked via the password authentication path unless they gave this password to someone else?
Do you know how they were owned? Is there a write up anywhere as this is a super interesting topic.
By having their password brute forced, or some other means?
"just" that? That's a very bold statement that most of the security community would disagree with.
An example: you don't monitor network traffic on the border because you trust your server not to be compromised. Then one gets compromised and the incident goes undetected due to lack of monitoring.
> If you remove attack surface altogether
That's a big if. Services can have a complex life, with software and configuration changing often. Unless you pull the plug you attack surface is never going to be zero and, on top of that, there so guarantee that it stays small. Hence organizations usually have different people taking care of different layers.
I'm pretty happy with sumo. A couple things I wish they'd change, but their core competency is searching logs and metrics and they do that really well. It is paid (there is a free tier but we outgrew it quickly) bit it's well worth it IMHO.
> But really: focus on removing attack surface, rather than continuously monitoring it.
So much this. Logs are great for telling you something bad has already happened and what that was, both security and faults. Reducing the attack service will get you more mileage from a security pov (again, in my opinion). Logs need to he there and some basic trawling of them is also necessary, but is purely reäctive.
Amen to that. I stopped caring about these sorts of entries in my HTTP logs when I realized these automated attacks target software packages written for language runtimes I don't even have installed. I'm sure not losing any sleep at night that someone is trying to hit phpMyAdmin or wp-admin on my server that doesn't even have a PHP interpreter.
we used mod_security with common rulesets + a custom list which was reviewed/updated based on what hacks were coming in (with hacks both passively monitored by customer base/compromised spam reports and via active scans for malware deposited by attackers which were then cross-correlated/traced back to application logs to find the entry point/compromise.
support staff was ~600 (including front end call center and billing people), with probably 250 of those doing L2+ support and ~50 doing the security work which fed into the ruleset, for a server farm of ~2-4000ish (and exponentially more sites/accounts since it was shared webhosting). In some ways our network could have been viewed as a giant honeypot because of all of the outdated webapps that were being run by the customers, plus their compromised PC's saving passwords, etc.
(I don't pick on you, so I hope this comment doesn't come off that way.)
What are "malicious events" and how will you know them when you see them?
I would argue that, instead, your system should automatically analyse logs for benign (a.k.a., completely normal, expected, innocent) events, toss them out, and leave humans to review anything that's left over.
With any new "system", it will take a bit to build up a complete "whitelist" so that the amount of data to review is down to a manageable level but you'll be rewarded with much more meaningful and actionable data when you get there.
I suspect they are simply defined as "rare" events.
I work in a Fortune 100 Security Operations Center (SOC). We have a dev team that builds tools for our Incident Response team, Threat Intel team, etc, etc and an engineering team to manage those tools. And lots of other teams that specialize on lots of other things (malware reverse engineering, spam/phishing, red team, etc). We have to build a fair amount of our own stuff because we operate at levels above what a lot of commercial tools can do. We have hundreds and hundreds of custom alerts that fire based on various network traffic and sysmon activity that our threat team has written rules around.
This, of course, is out of the question for most.
So, what should you do? First, get your logs all in one place. The ELK stack is probably the easiest way to do that. ES makes things searchable and Kibana is a pretty good UI for a lot of this stuff.
Second, if you aren't monitoring your internal network, you really have no idea what is going on, so look at what you can accomplish quickly with distributions like Security Onion (which includes the ELK stack): https://securityonion.net/
Third, there's a fair amount of Open Source Intel (OSINT) on threats. You'll need to find a way to integrate that information and scan your logs for it. Malware, like Coinminers, will have certain Indicators that will tell you if they are on your network. If you're on the ELK platform, look at something like ElastAlert. You write rules in Yaml and then set up Alerts to fire into your Slack channel or Email you or whatever. There is no real shortcut here.
Commercial tools that can do a lot of this stuff includes FireEye's TAP, but they definitely aren't free: https://www.fireeye.com/solutions/threat-analytics-platform....
And, as mentioned elsewhere, absolutely remove any attack surface that you can. Scan your IP ranges and domains with tools like Shodan.io, HackerTarget.com's DNS tools (DNSDumpster.com has a nice, free DNS search), etc and make sure you have a handle on what, exactly, you're exposing.
And, finally, if you're big enough then every single day a computer will be compromised in some way. There is no company in the world who doesn't experience this so you need to have a clear set of tools and playbooks to handle that. If you can respond to most incidents in days (not weeks), you'll be better than 90%+ of the companies out there.
Edit: clarified acronym
There is room in the industry for players that aggregate logs in interesting ways and can share threat intel and track attack patterns between customers, but a lot of the time with MSSPs you will just find yourself paying for a really expensive splunk instance.
Logs are fairly useless without occasional review (to at least verify you are collecting the things you think you are) and a decent understanding of the infrastructure they support.
I don't know your business model or capabilities though so this might not apply in your case. Hopefully not.
But there are plenty of MSSPs out there that will ingest logs, charge out the %]%{^{ for "monitoring" and add very little value. After the nth time you get alerts for PHP vulns on stuff which is clearly not running it, or something stopped sending logs months ago and no one noticed, or you find that firewalls logs from anything not in an unpublished support list don't really get parsed (yeah they will ingest it but just not tell you that nothing meaningful will happen with it..)...
To be fair, monitoring by third parties is one of those things that you can't "do and forget about it", you need shared responsibility and a clear understanding of who is doing what. None of this is cheap though.
Sadly, it is too easy to get a new IP address now. I think as part of our security mindset, we should consider it trivial to scan for several hundred vulnerabilities using relatively unique IP addresses.
I'm not saying that you shouldn't ban traffic as it's required under several compliance programs and it is another form of demonstrated diligence. But, unfortunately, banning individual IP addresses only works against the absolute lowest hanging fruit of the criminal world.