People will stop deploying WAFs when the compliance standards are rewritten to not require them. They are prominent in lots of installations because the box ticking exercise of compliance frameworks, namely PCI or HIPAA, require a WAF-like component to reach compliance. It took long enough for them to be written in that now everyone knows they need one. It will be even longer for them to be phased out, and no one with risk assessors wants to be the first to remove them. Too much risk they say, regardless of how strenuously the tech component say they're unneeded.
We are still waiting on compliance standards to update their password change policies to reflect what most people have been saying for over a decade, that frequently changing passwords are a security risk..
NIST discourages it, but most of the US government still requires it. In fact, I used to have a 60-day password rotation policy with such brilliant requirements as "no more than three characters of one class sequentially."
I soon realised that "ShitFuck!", thus, was a valid password -- and off I went. Meanwhile, a teammate who had previously been a technical writer contracting with the NSA told of the "waterfall method" -- not of software developments, but of passwords. It was common in his world, apparently, and you could always hear he was typing his password by the staccato of QazWsxEdc123, etc.
The PCI-DSS website itself requires that you change password every 12 months. At the same time the period for recertifying PCI-DSS is ... every 12 months. I have a systematic way to create a new password each time, which probably isn't secure.
Got into a quarrel about this with our IT director at a previous company. I wish there was a security best practices FAQ I could link to for this sort of stuff.
1) WAF do far more than just prevent SQL injections.
2) Many companies don't own the software they run and so they can't guarantee that it is free of SQL injections or that the version of ORM libraries are secure. WAF protect against this.
3) Auto-scaling is just as much about high availability than performance. Database indexes do not help with the former.
I did not downvote you, but as the article explains, WAFs don't protect against anything assuming the attacker bothers to spend five minutes bypassing them with one of a thousand well-known tricks.
8KB is a small amount of data. Many apps will need more, so will not be able to blanket block these. Even 64k, what appears to be the absolute limit, might not be enough.
Continuing, not blocking, is the default when not using the console, making this insecure by default.
It's not about competency but the fundamental nature of WAFs. It's like using a regular expression to sanitize input parameters you concatenate into a SQL query, except you also can't make it specific to the SQL dialect used. It doesn't matter how competent you are or how much money you spend on engineering the regular expression.
Look, if you want a "real" WAF capability you buy something like Imperva and manage the care and feeding of a team of say 2-4 ppl who understand web app vulnerabilities in depth, AND know the tool. Your SOC/NOC will need training and procedures too. Fully loaded an average enterprise will pay > $1M a year to maintain the capability if you look at the TCO carefully.
There are environments where this makes sense. Banks are the classic - they have the money, they care about the risk, lots of places have systems still on java 8 and take > 1 month to deploy code fixes.
The biggest practical problem in my experience is that it's almost impossible to keep good people maintaining an enterprise WAF. Anyone with a deep enough understanding of web app security and network infra to do the job properly will get bored in the role and leave. WAFs are usually barely maintained for this reason.
Another major issue is alert handling - there's nobody with enough context. The SOC staff usually understand nothing (not the vuln, not web, not the app). The devs don't understand the vuln and the security team don't understand the app so tickets go round in circles with nobody able to understand if an alert is an FP or an actual issue. Eventually alerts start getting quietly dropped on the floor.
Most enterprises have moved public facing sites to the cloud. And so are making use of the cloud provided WAF solutions all of which are trivial enough for someone to manage part-time.
Also in 20+ years in enterprises have never heard of firewalls being left unmaintained. I don't know how that would pass security audits, why such a critical piece of security architecture would end up in this state or how any in-house development would work given the constant need to make changes to it.
I do agree lightweight cloud WAFs can have positive ROI. You have to be realistic about their capabilities however and what you can expect from a part-timer managing one.
You end up choosing between a heavyweight "advanced" WAF (either cloud or on-prem) that requires lots of tuning and response, which most orgs can't do properly, or a lightweight cloud WAF with vendor rules that doesn't do much but hey at least it's easy to look after.
Re: enterprise WAF maintenance, I know this is anecdotal but I was a pentester for 10 years and I could count the "non-dysfunctional enterprise WAF setups" I saw on one hand. Were you in a role where you talked to the staff who looked after them or dealt with the alerts? Often the org lacked the insight to understand there was a problem. "We bought the magic security box, we applied the patches, what's the issue?"
Can confirm. I work in financial services, we employ Imperva for WAF + DDoS.
Initial onboard was 2 FTE for ~6 months.
We are now at probably .5-.75 FTE. Onboarding new sites, responding to “waf broke the site”, type things and doing exceptions (for pen tests and what not).
Not sure of the TCO, but I shudder to think what would happen if we didnt have it.
> 1) WAF do far more than just prevent SQL injections.
They largely don't prevent SQL injections.
> 2) Many companies don't own the software they run and so they can't guarantee that it is free of SQL injections or that the version of ORM libraries are secure. WAF protect against this.
Only if the WAF somehow understands the internals of that software better than that software itself. Which, sure, sometimes happens, but there's no systematic reason to believe it. Why should the WAF have a better hit rate than the makers of the actual application? Does the WAF vendor offer a guarantee that systems behind it wil never be hacked?
The code hasn't, but it would be a lie to say that the application hasn't, and as a leader of ops teams while I can't directly influence code security & quality, I can damn sure influence overall security by demanding a WAF.
Defense in depth is an important concept in security for a very good reason.
It's basically fizzbuzz. If you have a WAF, it proves you know how to add a WAF. Presumably even one you know how to configure when security needs change.
Compliance auditors are mostly there to underwrite posture, not actual risk.
Auditors pretty much only certify that you have told them you are doing what you are supposed to be doing. They are not logging in to your servers and verifying that an AWF is running in front of your web server. They are not probing your network from the outside to see if an AWF is blocking their activity.
Just as financial auditors are only confirming that your financial statements match what your accounting department tells them.
If you lie to your auditors, there's a good chance they won't catch it because that's not what they are looking for.
I've audited a lot of networks for compliance, and we always actually check that the protections that are meant to be in place are in place. I don't think I've done an audit where I wasn't using nmap to some degree.
You'd think that, but I had an auditor that had guidance requesting confirmation of security cameras on the servers. They wanted to drive to the AWS data center to see the cameras. If the drives got stolen, you'd better make sure AWS shares that video with you.
Ok, but what if someone bypasses the cameras on your servers with looped footage? Did the auditor think about having security cameras on the security cameras?
The article notes the problem of the attack surface of WAF products themselves, and raises questions about the security posture of the products and the vendors. Compliance standards need to address the risks and weigh them against the benefits of WAFs. If, as the author argues, they have reached a point where they do more harm than good, the compliance bodies need to be honest about that and reevaluate recommendations.
Here is a big lesson that I "learned": actual security does not matter for the purpose of avoiding risks and fines. What matters is the ability to say "we got hacked despite following the accepted best practice" or "we can shift the blame HERE". Having a paid-for WAF ticks both boxes.
Only if the WAFs themselves become a significant attack vector, to the extent where blame for breaches affects the bottom lines of the WAF vendors, the situation will start to change.
Especially when you only have the product in your pipeline in the first place because of some security compliance checkbox that needs to be checked ...
Qualyss WAS - don't get me started. Generic accounts, short passwords you can't change, incredibly slow and not-able-to-debug scans. Numerous false positives. And with a crawler you don't truly know if you've covered everything.
Hallelujah. Also, with many single-phase apps, WAFs don't make any sense - the HTML/CSS content is just served statically, so the potential vulnerabilities are in the API, which IMO is much easier to harden. Without going into too much of a tangent, this is one reason I'm a big fan of GraphQL. It's strong typing and support for custom scalar types means malformed content gets rejected before it even gets to your code. For example, most injection attacks require the use of some "special" characters like < or ;, but many field types have no need to support those characters, so instead of just typing "strings" everywhere, you can have things like Email or Date or SSN or Name scalar types that are more restrictive in the characters they allow.
Pretty much every SQL injection attack is going to need to be injecting single quotes with some uncompliant lack of parameterization someone put together.
Simply using parameterized queries solves this problem, no amount of semicolons can escape it.
Yes, totally agree. There are many good SQL libraries now that used things like tagged templates (e.g. sql`SELECT * FROM foo WHERE bar = ${zed}`) that make it virtually impossible to not use parametrized queries.
But my primary point is that I still believe it makes sense to type string inputs as restrictively as possible, not just for SQL injections but also for other types of potential vulnerabilities. E.g. if you're taking a date string that you expect to be in YYYY-MM-DD format, it's best to type that string as such as furthest out as the edge as possible.
All of that is possible with plain old REST APIs as well. For example, FastAPI and pydantic will do this sort of parsing and validating at the application edge.
Nicely written and the author clearly has more experience than myself. I did, however, get hit with a data breach via SQL injection, and everyone I spoke to (not vendors or sales folks) seemed to agree that a WAF would have blocked the attack outright.
Yes, those are needed too. And static analysis and dynamic analysis, etc.
Despite all of that we just found a SQL injection that existed for years somehow. Luckily the WAF blocked attempts to exploit it until we could issue a fix.
Sometimes one has to host an application and has no control over the details of how that application is developed or configured related to parameterized SQL queries.
Most large companies have too many developers and too many teams to expect/assume that each team will do the right thing for security when putting something in production on the public Internet.
Why? Because most software developers are bad at security (I said most not all).
So yes do all the things at the bottom of this article! Teach security-by-design to all your teams. Make sure they know what OWASP is at least. Make sure you test all the things. Either own or rent red teams.
But if you're a big enough company, you probably also need something centralized like a WAF, because you want defense in depth.
WAFs are far from perfect, but in my experience they are better than not having one in 2023.
This is my personal opinion but any developer building software that runs on the public Internet should not need to be incentivized to care about security.
Let me rephrase: software development (the action of engineers and the whole process in general) is actively insensitvised to not care about security.
The consequences of poor security are often way, way lower than the costs of doing it properly. Add on to that, that security problems are contingent risks that only "pay out" in a small number of cases and you have a recipe for low expected value for investment into security.
Software engineers often want to develop a secure product, but they don't know what they don't know, and their employer is not interested in investment in their security capabilities, both the companies security capabilities and the capabilities of their employees.
So, I guess I hear you on this, although I think it's a generalization..
There's certainly a trope among developers that "the business" doesn't give us time to do work properly, including security.. and that's it's just "ship features fast"..
My take on this is that it's on us, as professional developers and particularly technical leaders, to force that issue and advocate for better practices.
I've advocated for better technical practices at several jobs I've been at, and in many cases there was no intentional desire to incentivize bad practices, it was simply that the leaders weren't aware of all the requirements, and of the consequences of cutting too many corners.
When framed in the context of business value and risk, it's not as hard as you think to introduce better standards to most software development teams. Smart leaders are open to listening and changing if it means better outcomes.
That said, if your technical leadership isn't interested in supporting this kind of improvement or helping you advocate, then maybe it's time to dust off the CV if you personally care about it.
I've fought against WAFs in the past, but in govt departments, they are often a check-box in your security checklist that must be checked. I've even had to fight a security guy telling me I needed two WAFs because he wasn't convinced the first WAF was good enough (because it was a managed service and the vendor didn't want to share the specific configuration.)
In saying that, there are definitely use-cases where a WAF is required - especially if you have a legacy app on an older code-base that needs to be exposed to the internet.
First off, most large applications need something on the edge to help deal with volumetric attacks. If you've already got something there, adding a light WAF engine isn't exactly a huge ask. It's (almost) free.
Also, WAFs let you "fast patch" against vulnerabilities. The Log4j example the author gives as a negative is actually a positive. Your vendor can help prevent you from being attacked while you have time to respond. Those rules given in the example are bad, for sure, and probably have false positives - but, a few days of a slightly higher rate of false positives while you patch is probably worth it to most organizations.
Lastly, WAFs let you increase "risk scores" of requests and IPs, which lets you turn up captchas and other roadblocks against malicious IPs. This raises the bar from the floor to somewhere about knee height. Not a lot, but one more thing for an attacker to have to step over.
I do agree, though, that people treat WAFs as a magic solution or make them really heavy. Against application attacks, I view them as a tool of medium importance. Also, there are also better and worse vendors out there. Personally, in a WAF, I view "less as more".
The more rules and complexity, the more problems you're going to have. Adding rules should be temporary, and there's very few reasons to have blocking rulesets for long running issues.
I think ultimately acknowledging that there are no magic solutions, only a variety of options that can each contribute to reducing the probability of something bad happening, is critical to approaching security issues effectively.
My first experience with WAFs was part of a check-the-box security/compliance process, and I thought they were dumb. Easy to work around! Just regexes! With time I've come to appreciate that they're pretty low effort to operate, can be fairly lightweight, and wind up being one more thing for an attacker to deal with, in a world where each thing they have to deal with decreases their chances of success and increases the chance of someone noticing.
If we assume that basically everything can be bypassed by a sufficiently motivated attacker, the best approach is defense-in-depth where there are multiple barriers they need to traverse, and little opportunity to do so "quietly". WAFs can be evaded with clever approaches, sure, but getting to that point means they initially triggered a block, and have to make additional requests to test their evasion payload, each of which increases the signal we have to block more aggressively, trigger an alarm, and get a human involved.
We use AWS WAF for a bunch of this which is necessary because the other AWS components are so dumb. We definitely don't want to overcomplicate our WAF setup. Good enough is good enough.
This sounds so much like music to my ears. Where I work a WAF is mandatory, Azure Application Gateway in our instance, and they enable ALL of the rules because the outsourced colleagues they follow the rules set by infosec like sheep. The consequences are that many requests are blocked as a false positive.
Completely legitimate requests, like for instance an OpenID redirect from Azure AD (Microsoft Online) login to the application. They then get blocked by the SQL Injection rules in AAG. So user friendly.
At one point we were seriously considering base64 encoding all request bodies from javascript before sending them to the server and decoding them there - just to get around this bullshit.
Such a shame that you're forced to use this. We run Azure Function apps directly to public traffic and the experience is really nice & simple. Not once have I had a "I wonder if a middleman ate my request" experience. OIDC works flawlessly for us.
I would quickly grow to hate my tech stack if I had to cram stuff like AAG into it. Right now, we can spin up a very robust stack with ~3 products. The moment I have to start playing with network policies, I feel like the security level of my solution goes down not up. Manual routing, firewall or certificate management is a canary to me in 2023. I don't want to touch any of that stuff anymore - I'll probably screw it up at some point.
Perhaps you could reframe the AAG as creating an emergent situation that is less secure than the alternative without. It certainly sounds like an honest possibility based upon the workarounds you seem to be entertaining.
Large companies have WAFPlus-as-a-service(load balancing +WAF+ SSO: any team can provision one and put their app behind a WAF. Is there any alternative to replace that?
These are pretty weak arguments. Making nice graphics in games also increases frame times therefore we shouldn’t make nice graphics? Yeah wafs will slow down network requests the question is does that matter?
The answer is no. The argument about cap1 is also extremely weak. It was a bad incident but it’s a single example of a waf being a vector and most of the damage was caused by IAM misconfiguration.
The WAF increasing hypothetical attack surface was the closest thing to a good argument on there, and since their “alternatives” amounted to “don’t misconfigure anything or deploy a vulnerability”, which solution would also have solved their single example of WAF-as-an-attack-vector actually happening… yeah, that still made the piece less convincing, overall.
So there are a few issues with this, WAFs do have their uses, generally speaking yes rules based on regexes looking for sql injection are silly.
But they do have their useses. For example tarrgeted blocking, https://confluence.atlassian.com/security/cve-2023-22515-pri... . While waiting for the patch, a WAF can quickly block all requests to the /setup endpoint.
I would also say that static analysis as a panacea for SQL Injection is laughable. SAST tools have a hard time finding sql injection in code. As they quickly loose track of user controlled data. They almost always create false positives / false negatives when Parameterised queries are used incorrectly. For example when user controlled data gets into the SQL query rather than the parameter of a paremeterised query.
And that completely ignores SQL Injection attacks that do not occur within your code directly, but in libraries you are using.
depends on the org. The appsec team, may not have access to the webserver in production atleast not quickly. But will have access to modify a WAF they own.
WAFs are also bring on risks that are not considered. At least in scenarios where things get logged and no one considers how they are sanitized. And then there is also GDPR in Europe.
Lot of these issues could be avoided by good API design. But if you need WAF you might not be doing that... So you end up passing tokens as parameters and they end up in logs. Then again if someone has access to them they probably also have other access. But it still is often forgotten aspect.
WAFs are a good way of masking some issues but they tend to not help against more sophisticated attackers, as in a bit more sophisticated than just using metasploit.
Here is my opinion: WAF is a tool, use it where it makes sense. Don’t use it where it doesn’t make sense.
I’ve seen many use cases where deploying a WAF solution was completely reasonable and the problem couldn’t have been solved in the application code. And let’s not forget often you have to run applications that were written by other companies, and you still need an additional layer of security.
—-
These “stop using X” articles are really pointless (and boring). They also misguide the people who have less experience in a particular field.
PCI-DSS does not mandate the use of a WAF. It is one of two ways you can fulfill requirement 6.5 or 6.6. WAF + OWASP Top Ten ruleset is typically easier to get evidence for your auditor, but you can show that continuous scanning using a DAST scanning engine to meet requirements.
I would have a WAF installed with very few highly tuned rules against mostly SQLi. Why? Because the damage of letting that through and praying that the developer or web-app framework does it right are significant. The rules for SQLi are pretty easy to get right and dropping that traffic before it gets to your web server is a reasonable thing.
I would have a WAF installed with no rules too. It is nice to have something there where you can drop in a Log4J rule and get protection relatively quickly for attacks of that nature. There have been a number of these over the years and a small performance penalty seems worth the big picture safety net.
I am against the pricey models that the cloud vendors push. WAF can get expensive. They typically are bundled with other cloud services, but hey, if you've gotten that far, you are probably outsourcing most things to the cloud provider anyway.
I do not like WAF pragmatically because it lets the developer off the hook in many ways. There is something there doing their work for them and another reason for some developers to not understand or care about the security of their applications. Something else will do it for me whether I know this or not.
Basically every service of any size is terminating SSL before the request gets to the “real” recipient, anyway. They’d do it with or without a WAF. CDN, load-balancing, centralized routing, you’re doing it somewhere whether or not a WAF is involved.
I mean, you know TLS will have to be terminated at some point anyways. Might as well rage against NGINX terminating the TLS instead of whatever back end you're using (Node, Django, whatever).
Seems like Coraza also has some more recent benchmarks, since from what I can tell they more or less aim to replace ModSecurity in some regards: https://coraza.io/docs/reference/benchmarks/
I wonder whether anyone has undertaken the effort to compare the performance of the self-hosted WAF options in 2023, or at least in the past few years.
Personally, I think the performance tradeoff might sometimes be worth it if security does indeed improve, the Swiss cheese model (defense in depth) and all that: https://en.wikipedia.org/wiki/Swiss_cheese_model
However, for 3rd party code we run/host but don't really own I see value to a WAF. For example, we unfortunately run WordPress, and I don't have time to manually audit all of the stupid plugins people want to be installed beyond a checker for known vulnerabilities, so a WAF is some comfort/protection.
I think WAF is really a bigger set of tools now (bot protection, IP reputation, L7 DDoS/rate limiting, API restrictions) than just signatures. Virtual patching is also incredibly important and there's really no other security tool that gives you the granularity to restrict something like the values of some param on a specific path of your app, but only when some cookie exists.
I don't think the performance concerns here are accurate. I think these days most people are using vendors own cloud infra (Akamai, Cloudflare, F5, Imperva, etc), but even if you are using WAF on-prem, F5 and Imperva sell purpose built hardware that have no problem handling tons of requests. Most WAF's also have weighted signatures these days and won't just fire on ${jndi. "${jndi" might give 5pts, while "org.apache.*" gives another 5, and maybe their threshold is set for 10 for blocking.
I have plenty of issues with WAF's and I would invest a lot more in developer training, but I think they still have their place.
I think of WAFs as an extra safety net. Defense in depth.
The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.
Or even do what what CloudFlare did [1] and transpile all the slow ModSecurity rules to Lua and deploy OpenResty at the edge. Run them in nginx+luajit.
> I think of WAFs as an extra safety net. Defense in depth.
The WAF itself is a complex codebase written in a performance-critical domain, so they're generally implemented in memory-unsafe languages. If the services behind the WAF are implemented at all competently, you're probably increasing the attack surface by more by adding the WAF than you're saving.
Show me an alternative that I can deploy despite developers having both the lack of knowledge and complete indifference of security. They don't care and they aren't forced to conform to any security standards, so a WAF is literally the only thing I can do to try to improve things. I can't rewrite all their code for them. Management hears about a WAF and makes that a requirement and moves on.
If software development was a professional trade group, we could make membership require security training, and industry standards for ensuring security. But that'll never happen. It would mean them giving themselves more work to do, and we all know how lazy devs are.
I cannot argue with this too much. A WAF will protect you from unsophisticated attackers - at great cost.
In my group, I am considered to be the WAF SME. Enough that I wrote a training course to get into ruleset tuning.
What I see a lot is customers who are security-focused demanding "OWASP Top 10" protection and then, somehow, not understanding that it is not 10 rules you enable on the WAF. These are people often with application security and other credentials.
Most people I have seen running WAF's are in "Set it and forget it" mode. Tune the rules until it no longer blocks legitimate traffic and call it a day. I think few really understand what it is, and the why of using them.
Another funny anecdote: I had one of these customers talk about how amazing Akamai WAF was, because it never had false positives. Never? Really? That looks like a red flag to me, but they were not concerned.
Similarly I've met a worrying amount of people, often with titles like "Senior Architect", thinking that adding Sonarqube to your pipeline will magically eliminate all security bugs and you'll no longer have to think about security again
Feels like you could replace WAF with DLP, EDR or most any other 3 letter abbreviation... Or indeed bike helmets. Vendor push + industry ‘best practice’ tribal knowledge is tough to resist.
144 comments
[ 3.3 ms ] story [ 210 ms ] threadI soon realised that "ShitFuck!", thus, was a valid password -- and off I went. Meanwhile, a teammate who had previously been a technical writer contracting with the NSA told of the "waterfall method" -- not of software developments, but of passwords. It was common in his world, apparently, and you could always hear he was typing his password by the staccato of QazWsxEdc123, etc.
https://pages.nist.gov/800-63-3/sp800-63b.html#memorizedsecr...
You sprinkle it on top of code riddled with SQL injections nobody could be bothered to avoid or fix, and now magically the code has a become secure!
You’ll find it on the shelf next to auto-scaling cloud wizardry, which can similarly be used to fix the total absence of indexes in the database.
2) Many companies don't own the software they run and so they can't guarantee that it is free of SQL injections or that the version of ORM libraries are secure. WAF protect against this.
3) Auto-scaling is just as much about high availability than performance. Database indexes do not help with the former.
I didn't realise AWS, Cloudflare etc were so incompetent that their product can be bypassed in 5 minutes. I assume you have an example of this.
Maybe you should read it.
https://docs.aws.amazon.com/waf/latest/developerguide/waf-ov...
For oversize requests WAF can be configured to reject it.
8KB is a small amount of data. Many apps will need more, so will not be able to blanket block these. Even 64k, what appears to be the absolute limit, might not be enough.
Continuing, not blocking, is the default when not using the console, making this insecure by default.
And having an insecure default is a huge difference from 100% of requests are allowed.
This guide, linked from the article, shows some simple evergreen bypass techniques (section "WAF bypass cheat sheet"):
https://habr.com/en/companies/dsec/articles/454592/
There are environments where this makes sense. Banks are the classic - they have the money, they care about the risk, lots of places have systems still on java 8 and take > 1 month to deploy code fixes.
The biggest practical problem in my experience is that it's almost impossible to keep good people maintaining an enterprise WAF. Anyone with a deep enough understanding of web app security and network infra to do the job properly will get bored in the role and leave. WAFs are usually barely maintained for this reason.
Another major issue is alert handling - there's nobody with enough context. The SOC staff usually understand nothing (not the vuln, not web, not the app). The devs don't understand the vuln and the security team don't understand the app so tickets go round in circles with nobody able to understand if an alert is an FP or an actual issue. Eventually alerts start getting quietly dropped on the floor.
Also in 20+ years in enterprises have never heard of firewalls being left unmaintained. I don't know how that would pass security audits, why such a critical piece of security architecture would end up in this state or how any in-house development would work given the constant need to make changes to it.
You end up choosing between a heavyweight "advanced" WAF (either cloud or on-prem) that requires lots of tuning and response, which most orgs can't do properly, or a lightweight cloud WAF with vendor rules that doesn't do much but hey at least it's easy to look after.
Re: enterprise WAF maintenance, I know this is anecdotal but I was a pentester for 10 years and I could count the "non-dysfunctional enterprise WAF setups" I saw on one hand. Were you in a role where you talked to the staff who looked after them or dealt with the alerts? Often the org lacked the insight to understand there was a problem. "We bought the magic security box, we applied the patches, what's the issue?"
https://accelerationeconomy.com/cloud/amazon-shocker-ceo-jas...
And the article you posted is about total spend not just infrastructure.
Initial onboard was 2 FTE for ~6 months.
We are now at probably .5-.75 FTE. Onboarding new sites, responding to “waf broke the site”, type things and doing exceptions (for pen tests and what not).
Not sure of the TCO, but I shudder to think what would happen if we didnt have it.
We have a lot of…old…java…
They largely don't prevent SQL injections.
> 2) Many companies don't own the software they run and so they can't guarantee that it is free of SQL injections or that the version of ORM libraries are secure. WAF protect against this.
Only if the WAF somehow understands the internals of that software better than that software itself. Which, sure, sometimes happens, but there's no systematic reason to believe it. Why should the WAF have a better hit rate than the makers of the actual application? Does the WAF vendor offer a guarantee that systems behind it wil never be hacked?
The code hasn't, but it would be a lie to say that the application hasn't, and as a leader of ops teams while I can't directly influence code security & quality, I can damn sure influence overall security by demanding a WAF.
Defense in depth is an important concept in security for a very good reason.
It could become less secure too, as WAF software itself can have vulnerabilities or misconfiguration.
Honestly AWS WAF is perfect for this. It's not a great WAF but it is ideal if you need "pretend to have a WAF" as a service so you can tick the box.
Compliance auditors are mostly there to underwrite posture, not actual risk.
Just as financial auditors are only confirming that your financial statements match what your accounting department tells them.
If you lie to your auditors, there's a good chance they won't catch it because that's not what they are looking for.
There's no audit checkbox for "WAF actually does something useful", so it's fine.
Security auditors are (quite literally) accountants.
I guess there's just a decent market for the product everyone trying to check that box, though.
Amateurs.
What matters, of course, is money.
Only if the WAFs themselves become a significant attack vector, to the extent where blame for breaches affects the bottom lines of the WAF vendors, the situation will start to change.
Simply using parameterized queries solves this problem, no amount of semicolons can escape it.
But my primary point is that I still believe it makes sense to type string inputs as restrictively as possible, not just for SQL injections but also for other types of potential vulnerabilities. E.g. if you're taking a date string that you expect to be in YYYY-MM-DD format, it's best to type that string as such as furthest out as the edge as possible.
Parameterised queries however don’t need an annual fee and a team of security engineers to babysit it.
Cloudflare, AWS, GCP etc offerings are basically just one click and for smaller sites will be free.
And over the years there have been many security flaws in how SQL libraries actually handle parameterisation.
Eliminating false positives is a significant effort.
Despite all of that we just found a SQL injection that existed for years somehow. Luckily the WAF blocked attempts to exploit it until we could issue a fix.
Defence in depth is the win here.
Why? Because most software developers are bad at security (I said most not all).
So yes do all the things at the bottom of this article! Teach security-by-design to all your teams. Make sure they know what OWASP is at least. Make sure you test all the things. Either own or rent red teams.
But if you're a big enough company, you probably also need something centralized like a WAF, because you want defense in depth.
WAFs are far from perfect, but in my experience they are better than not having one in 2023.
In my experience it is that most software engineers are not incentivized to care about security.
It's a fundamental part of the job.
The consequences of poor security are often way, way lower than the costs of doing it properly. Add on to that, that security problems are contingent risks that only "pay out" in a small number of cases and you have a recipe for low expected value for investment into security.
Software engineers often want to develop a secure product, but they don't know what they don't know, and their employer is not interested in investment in their security capabilities, both the companies security capabilities and the capabilities of their employees.
There's certainly a trope among developers that "the business" doesn't give us time to do work properly, including security.. and that's it's just "ship features fast"..
My take on this is that it's on us, as professional developers and particularly technical leaders, to force that issue and advocate for better practices.
I've advocated for better technical practices at several jobs I've been at, and in many cases there was no intentional desire to incentivize bad practices, it was simply that the leaders weren't aware of all the requirements, and of the consequences of cutting too many corners.
When framed in the context of business value and risk, it's not as hard as you think to introduce better standards to most software development teams. Smart leaders are open to listening and changing if it means better outcomes.
That said, if your technical leadership isn't interested in supporting this kind of improvement or helping you advocate, then maybe it's time to dust off the CV if you personally care about it.
In saying that, there are definitely use-cases where a WAF is required - especially if you have a legacy app on an older code-base that needs to be exposed to the internet.
First off, most large applications need something on the edge to help deal with volumetric attacks. If you've already got something there, adding a light WAF engine isn't exactly a huge ask. It's (almost) free.
Also, WAFs let you "fast patch" against vulnerabilities. The Log4j example the author gives as a negative is actually a positive. Your vendor can help prevent you from being attacked while you have time to respond. Those rules given in the example are bad, for sure, and probably have false positives - but, a few days of a slightly higher rate of false positives while you patch is probably worth it to most organizations.
Lastly, WAFs let you increase "risk scores" of requests and IPs, which lets you turn up captchas and other roadblocks against malicious IPs. This raises the bar from the floor to somewhere about knee height. Not a lot, but one more thing for an attacker to have to step over.
I do agree, though, that people treat WAFs as a magic solution or make them really heavy. Against application attacks, I view them as a tool of medium importance. Also, there are also better and worse vendors out there. Personally, in a WAF, I view "less as more".
The more rules and complexity, the more problems you're going to have. Adding rules should be temporary, and there's very few reasons to have blocking rulesets for long running issues.
I think ultimately acknowledging that there are no magic solutions, only a variety of options that can each contribute to reducing the probability of something bad happening, is critical to approaching security issues effectively.
My first experience with WAFs was part of a check-the-box security/compliance process, and I thought they were dumb. Easy to work around! Just regexes! With time I've come to appreciate that they're pretty low effort to operate, can be fairly lightweight, and wind up being one more thing for an attacker to deal with, in a world where each thing they have to deal with decreases their chances of success and increases the chance of someone noticing.
If we assume that basically everything can be bypassed by a sufficiently motivated attacker, the best approach is defense-in-depth where there are multiple barriers they need to traverse, and little opportunity to do so "quietly". WAFs can be evaded with clever approaches, sure, but getting to that point means they initially triggered a block, and have to make additional requests to test their evasion payload, each of which increases the signal we have to block more aggressively, trigger an alarm, and get a human involved.
We use AWS WAF for a bunch of this which is necessary because the other AWS components are so dumb. We definitely don't want to overcomplicate our WAF setup. Good enough is good enough.
Completely legitimate requests, like for instance an OpenID redirect from Azure AD (Microsoft Online) login to the application. They then get blocked by the SQL Injection rules in AAG. So user friendly.
At one point we were seriously considering base64 encoding all request bodies from javascript before sending them to the server and decoding them there - just to get around this bullshit.
Such a shame that you're forced to use this. We run Azure Function apps directly to public traffic and the experience is really nice & simple. Not once have I had a "I wonder if a middleman ate my request" experience. OIDC works flawlessly for us.
I would quickly grow to hate my tech stack if I had to cram stuff like AAG into it. Right now, we can spin up a very robust stack with ~3 products. The moment I have to start playing with network policies, I feel like the security level of my solution goes down not up. Manual routing, firewall or certificate management is a canary to me in 2023. I don't want to touch any of that stuff anymore - I'll probably screw it up at some point.
Perhaps you could reframe the AAG as creating an emergent situation that is less secure than the alternative without. It certainly sounds like an honest possibility based upon the workarounds you seem to be entertaining.
Well said. I’d agree that for most scenarios, one never has to touch these. That’s one of the actual selling points of cloud.
Well, there is super duper secure mode in Edge that disables the JIT - so there's that. It also enables CET. https://microsoftedge.github.io/edgevr/posts/Super-Duper-Sec...
The answer is no. The argument about cap1 is also extremely weak. It was a bad incident but it’s a single example of a waf being a vector and most of the damage was caused by IAM misconfiguration.
I would also say that static analysis as a panacea for SQL Injection is laughable. SAST tools have a hard time finding sql injection in code. As they quickly loose track of user controlled data. They almost always create false positives / false negatives when Parameterised queries are used incorrectly. For example when user controlled data gets into the SQL query rather than the parameter of a paremeterised query. And that completely ignores SQL Injection attacks that do not occur within your code directly, but in libraries you are using.
So can IIS request filtering or whatever exists in Nagios. Right on the webserver.
Lot of these issues could be avoided by good API design. But if you need WAF you might not be doing that... So you end up passing tokens as parameters and they end up in logs. Then again if someone has access to them they probably also have other access. But it still is often forgotten aspect.
I’ve seen many use cases where deploying a WAF solution was completely reasonable and the problem couldn’t have been solved in the application code. And let’s not forget often you have to run applications that were written by other companies, and you still need an additional layer of security.
—-
These “stop using X” articles are really pointless (and boring). They also misguide the people who have less experience in a particular field.
PCI-DSS does not mandate the use of a WAF. It is one of two ways you can fulfill requirement 6.5 or 6.6. WAF + OWASP Top Ten ruleset is typically easier to get evidence for your auditor, but you can show that continuous scanning using a DAST scanning engine to meet requirements.
I would have a WAF installed with very few highly tuned rules against mostly SQLi. Why? Because the damage of letting that through and praying that the developer or web-app framework does it right are significant. The rules for SQLi are pretty easy to get right and dropping that traffic before it gets to your web server is a reasonable thing.
I would have a WAF installed with no rules too. It is nice to have something there where you can drop in a Log4J rule and get protection relatively quickly for attacks of that nature. There have been a number of these over the years and a small performance penalty seems worth the big picture safety net.
I am against the pricey models that the cloud vendors push. WAF can get expensive. They typically are bundled with other cloud services, but hey, if you've gotten that far, you are probably outsourcing most things to the cloud provider anyway.
I do not like WAF pragmatically because it lets the developer off the hook in many ways. There is something there doing their work for them and another reason for some developers to not understand or care about the security of their applications. Something else will do it for me whether I know this or not.
Please inspect all my HTTPS traffic, I terminated SSL, so you are free to modify the HTTP, no one will know, we trust you completely!
Why is this a good idea exactly?
I mean, you know TLS will have to be terminated at some point anyways. Might as well rage against NGINX terminating the TLS instead of whatever back end you're using (Node, Django, whatever).
However, some benchmarks from a while ago suggested that Nginx performs worse with ModSecurity than Apache does in a similar configuration: https://blog.litespeedtech.com/2019/12/02/modsecurity-perfor...
Seems like Coraza also has some more recent benchmarks, since from what I can tell they more or less aim to replace ModSecurity in some regards: https://coraza.io/docs/reference/benchmarks/
I wonder whether anyone has undertaken the effort to compare the performance of the self-hosted WAF options in 2023, or at least in the past few years.
Personally, I think the performance tradeoff might sometimes be worth it if security does indeed improve, the Swiss cheese model (defense in depth) and all that: https://en.wikipedia.org/wiki/Swiss_cheese_model
However, for 3rd party code we run/host but don't really own I see value to a WAF. For example, we unfortunately run WordPress, and I don't have time to manually audit all of the stupid plugins people want to be installed beyond a checker for known vulnerabilities, so a WAF is some comfort/protection.
What WAF solution are you using for WordPress?
I don't think the performance concerns here are accurate. I think these days most people are using vendors own cloud infra (Akamai, Cloudflare, F5, Imperva, etc), but even if you are using WAF on-prem, F5 and Imperva sell purpose built hardware that have no problem handling tons of requests. Most WAF's also have weighted signatures these days and won't just fire on ${jndi. "${jndi" might give 5pts, while "org.apache.*" gives another 5, and maybe their threshold is set for 10 for blocking.
I have plenty of issues with WAF's and I would invest a lot more in developer training, but I think they still have their place.
The author complained about the performance cost of WAFs in general, but not all WAFs have be structured like ModSecurity. They could for example be based on something like https://github.com/intel/hyperscan and perf is at a very different level.
[1] https://blog.cloudflare.com/cloudflares-new-waf-compiling-to...
The WAF itself is a complex codebase written in a performance-critical domain, so they're generally implemented in memory-unsafe languages. If the services behind the WAF are implemented at all competently, you're probably increasing the attack surface by more by adding the WAF than you're saving.
If software development was a professional trade group, we could make membership require security training, and industry standards for ensuring security. But that'll never happen. It would mean them giving themselves more work to do, and we all know how lazy devs are.
In this case, a WAF is giving the illusion of safety. So I feel like you're actually making an argument against WAFs.
In my group, I am considered to be the WAF SME. Enough that I wrote a training course to get into ruleset tuning.
What I see a lot is customers who are security-focused demanding "OWASP Top 10" protection and then, somehow, not understanding that it is not 10 rules you enable on the WAF. These are people often with application security and other credentials.
Most people I have seen running WAF's are in "Set it and forget it" mode. Tune the rules until it no longer blocks legitimate traffic and call it a day. I think few really understand what it is, and the why of using them.
Another funny anecdote: I had one of these customers talk about how amazing Akamai WAF was, because it never had false positives. Never? Really? That looks like a red flag to me, but they were not concerned.