I have VMs specifically for running garbage that I find/need to but I don't trust. These have 0 personal information and gets wiped to a clean state afterwards.
And I wonder if it’s good business to be kinda bad for companies that are required to get a pen test or want to put “passed security audit” on their website.
People are paying pentesters because their payer’s policy asks them to. Of course there will be a huge market for someone with an alleged certification to run automated tools, the value is in the box checking and report generation, not the bespoke broken website fixing.
Edit: Followup - here is a primer method. The exact method differs if you are talking with leaders, or middle management, or space occupiers. (Leaders as in intelligent, aggressive forward thinkers). You have to make it personal, and show them how the thing that they care most about (eg: career, personal security, promotion, or liability exposure) is at risk; which now-a-days is a true statement.
Can we collectively refrain from making comments like this on Hacker News—or in any venue, for that matter? Is it that hard to read an assumed “In my experience, it is often the case that…” in front of everyone’s comment?
I was stating that I have great success showing how said items should be prioritized and budgeted, across numerous organization types, sizes, and industries - and making it happen.
Let me demoralise you. There are industries [cough] where (for a good number of jurisdictions) you MUST have the pentests and other audit-like engagements conducted by a specially approved operator. Think of a royal charter.
These operators have no incentive to invest in proper skills. So they don't. They can charge almost anything they like. And sure enough, they do.
In an industry with such massive vendor lock-ins and regulatory capture regimes, organisationally you get to shovel a lot of money for sub-par pentests. The appetite to have yet another one, even from a known-good and reasonably priced provider, is quite hard to come by. After all, these approved providers will REFUSE to even look at a third-party pentest report, let alone let one guide their own testing.
So in aggregate the existence of these approved providers and their level of competence degrades the security posture of entire business domains.
"While the cost of penetration testing can be pretty high (typically between $1,000 and $100,000+),"
I've always been curious about costs for real pentesting. 1-100k is a HUGE range, that can't mean the range in cost for the same project can be that big, can it? What's it cost to hire someone to run a decent set of tests on a Rails app, for instance? Could I really see that big of a range? Would I get way better results for for $100k?
You will find exponentially less vulnerabilities the more money you spend. How much you should spend depends on how big the damage could be in case of a breach.
"A rails app" could be a rails default build, or Github.
You should get significantly better results the more one pays - because of the human element.
For example, $1k gets you a basic Nessus/Nmap (basic vulnerability/mapping) scans, maybe SQL vuln, with automated reporting. More gets you architecture, infrastructure, penetration, white/black scenarios, risk assessment, external organization liability, code repo, data, compliance, and regulatory footing. Those can surpass $100k easily.
So how would someone know they have connected to a honeypot? Presumably detecting this sort of thing is deliberately difficult, otherwise there isn't much point of the honeypot?
Some SSH honeypots accept more than one password for root. So, if 'letmein' and 'changeme' get you a root shell, it's probably a honeypot... but it could be a very broken PAM config as well (I have seen this firsthand).
PAM is one utter freaking nightmare to set up if one wants to stray from the distro defaults. Actually I'm astonished no one has dared to try and develop an alternative... god knows it's about time.
This comes down to the fact that penetration testing has a high false-negative rate. There's no way for a customer to really evaluate what a mostly empty report means, if there actually are no bugs, or the penetration tester missed them/did a poor job.
This isn't really comparing the results to other pentests, so I have a hard time seeing the use of this.
I think almost any pentest will miss some/many security issues.
And these issues look more like CTF style issues than real world security issues.
If you're not thinking about the challenge in the right way, it is probably harder to see the vulnerabilities.
The only thing this article proves yet again is: if you pay peanuts you get monkeys. I mean, 100 dollar is maybe enough for 2 hours of a freelancers time. But I do not think that is enough time to find both issues and scan the server.
I agree with you - yet there is also a philosophy that some organizations take which is to intentionally hire quite subpar 'certified' practitioners who themselves can be duped or walked past issues to provide 'clean reports' to upper management, even if false.
This is not always true. I’ve seen a ton of costly engagements produce super low quality deliverables that yielded no real value except for a checkbox being filled.
I'm trying to imagine being a customer for this kind of service. When would I be satisfied with a pen testing service?
If I'm not familiar with pentesting methodologies, my only metric of satisfaction would be the pen tester's reputation i.e., if a well-reviewed pentester says my site is OK, then maybe my site is safe.
I work as a professional pentester. This question is difficult for both sides.
The business often gets in my way with budgets. I can spend it all on some strange behavior that I detect in the application. I like to spend my time like that. The more experience I get the more time I could spend, even on simple applications. The rabbit hole goes ever deeper.
When am I satisfied that I delivered good work? When I find a number of high impact vulnerabilities. Doesn't mean there aren't still more to be found, like I said you could go on and on, but it does mean that I fixed some bad. Sadly I have a fixed budget, and sometimes I find nothing, those are bad days.
In my experience customers are also satisfied with a report that contains few or no impactful findings. But an empty report is no guarantee that there really isn't anything to be found.
It's difficult to balance the desire to dive deep and to provide broad coverage. I once got bit by that.
I found and clearly documented no less than 3 absolutely critical issues (price adjustment in an e-commerce website). Felt pretty good about that test. Until, in production, someone else found an additional authentication bypass. Oops. I wasn't focused on that due the aforementioned input validation issues and got blinded by trying to find more of those. It's not easy.
The title of the HN submission was changed. It used to be the article's subtitle, "How I Hired 7 Freelancers to Exploit this Weird Vulnerability".
"This one weird X" is a common clickbait pattern, and indeed, in this case, it is misleading: the article's descriptions of the purposely-introduced "weird" vulnerabilities aren't useful, only the freelance test results are.
The problem that strikes me here (as a professional pentester) is that these vulnerabilities are so pathological. There is no realistic series of errors that would lead you to really design an app with hardcoded sqli looking credentials or somehow return a cookie with a malformed header, let alone one that somehow automatically authenticated the user.
I am extremely onboard with the idea that pentesting as a whole has a lot of scanner jockeys (and a lot of my current work is finding people who aren't.) But you're not testing for vulnerabilities that would actually show up in the real world, and it is hurting your analysis.
I suspect these would be realistic vulnerabilities introduced by freelance or inexperienced devs hired by a startup, for example.
They might have hardcoded admin-level passwords for debugging, then forgotten about it.
They might have mis-typed HTTP headers like 'set-cookie', after having written manually a lot of auth & session management that should really be done using well-vetted libraries instead.
Respectfully, you are incorrect. Delegating session and cookie management to the framework or (in php's case) the language is so much simpler that I have seen manual implementations of this behavior only 2-3 times in my career. And the idea that logging in incorrectly would somehow return a malformed, but correct, session-setting header is again, pathological.
Most of the time, these types of profound errors come from simply taking the path of least resistance provided in the framework. Sometimes they do come from a complex homebrewed solution to a simple problem.
Back at my startup days, I once worked with a poor guy who was dynamically generating individual IDs for each element on a page and their corresponding CSS for each page. He was unaware that CSS also had _classes_, which perfectly encapsulated the behavior he was trying to create. His work was complex, and full of errors, but it had a certain logic to it- it was the path of least resistance that he saw available to him.
These don't look like that. They just look like weird errors designed to avoid showing up on, or triggering false positives on, DAST tools. I thinking testing people's skills without those tools is a good goal, i just think the methodology here is wrong.
I can see both of your viewpoints, here. I wouldn't make the blanket statement that OP is incorrect. The fact that you have only seen this a handful of times in your career is not surprising. This sort of silly bullshit is far less common nowadays. However, as OP has stated, these things were somewhat more common a while back. Mind you, much of this very poorly written software is still being used in dusty corners by large companies. You should keep an open mind when testing and not dismiss these things are outright impossible or else you're going to miss a lot of bugs :P
Some of the most interesting issues seem pathological at first blush. Can you really think of no scenario where there would be an sql injection string as a password? Perhaps some try-hard came along before you and attempted sql injection on the user creation form that resulted not in sql injection, but in the password being set to the literal string that the person used as input.
I realize that in this scenario it is literally designed-in, but I understand the point the author is trying to prove. If a "scanner jokey" gets results that tells him or her there is SQL injection, a competent tester will try to verify what their tool is telling them. If the tester is doing that in this case, they'll find other injection strings not working and (hopefully) start looking under the hood to see what is going on and discover this pathological hard-coded pw and be able to tell the client that. Maybe it's the work of a malicious dev?
I agree that a pentester with a lot of experience will have skills that are honed to find common bug patterns, but it's nice to be able to find these seemingly bizarre issues and have an explanation for the client. It shows you really understood the app and what it's doing.
Given this is your area of expertise, I'm genuinely interested in how/why you believe that these vulnerabilities wouldn't show up in the real world. Generally speaking, isn't the software developer community littered with developers who aren't adequately skilled/copy-paste-from-Stack-Overflow, etc?
Now a real interesting article would be what appsec service would have the highest impact for $100? Obviously not pentest. Code review? Architecture analysis? At $100 none of it would be deep of course. I could find a lot more stuff of interest in 2 hours of code review compared to pentesting.
> What is surprising is the number of people purchasing these security services as well as the number of positive reviews. An incomplete security testing methodology results in a false sense of security for the assessed systems.
The same is happening in the "smart contract audit" space. The results of the audit do not matter to the audience that wants to see the checkbox of "audit".
You can currently make a lot of money undercutting other auditors providing these audits.
The person requesting the audit is making multiple orders of magnitude more money by having the audit.
> What is surprising is the number of people purchasing these security services as well as the number of positive reviews.
Because for most people its not about security, its about liability and having a proven attempt at trying to "security" implies in their minds that they are not liable.
Sadly security is a mindset and a process like healthy eating but we socially model it as a eating a single apple which translates to hiring one person for less than $100 and taking what they say as authority.
This is a common problem when hiring supposed experts. Unless the one doing the hiring is reasonably skilled in the area being hired for, it is very difficult to judge the skill and quality of the candidate.
There are unfortunately many small business people who greatly underestimate the skill that goes into anything they don't personally do. Those are the foolish people who shop by price and think they are getting a good deal.
These are also the people who keep Walmart alive. They have limited reasoning ability, and even if they might suspect it's too cheap to be any good, they want it to be good enough and are willing to pretend that it is. These happen to also be the people who easily accept absurd and obviously false statements made by certain political leaders.
I know quite a few of these people, and I have tried countless times to advise them on reasonable choices, but it is like talking to a wall (if a wall could nod and pretend to agree).
I can kind of get behind the malformed header. But I have very hard time thinking up any suite of actions that would lead to a code resulting in leaked obfuscated query with a hard coded password. All while password somehow using a LIKE statement matching a sql injection.
Is there any precedent for this?
Edit: rewrote my sentence because it was barely readable.
Not-invented-here (NIH) coders end up worrying about this kind of situation. People who develop "creative" systems by working ground-up starting with foundational system logic. They create proprietary systems really fast, and those proprietary systems often do some wackadoodle things in the name of internal logic.
As a result, those systems are typically harder to maintain or secure by just anybody, even by some experts who are not from the NIH mindset (here's where the author's approach isn't really fair, IMO). Proprietary systems are are less likely to present the typical bell curve security issues that an automated scanner would pick up.
What they are more vulnerable to is puzzle logic--the "what's going on here, hmm let's have a tinker" approach. Which is exactly what the author presented and it also matches his broader approach of "let's have a tinker...with freelancers".
It's a subjective security vs. objective security approach. Both sides are important. But both sides should know about how the other side works.
> Some of them claim to possess industry-accepted certificates
I'll trust them on that claim. But is my $100 getting me an industry-certified analysis?
I could hire a licensed MD Engineer Pharmacist Cosmetologist Veterinarian to walk my dog for me, but I don't expect them to perform heart surgery for $100
75 comments
[ 0.24 ms ] story [ 145 ms ] threadIn every org I've sat down with the head to explain those risks, I can see how it can be made priority and budget.
Edit: Followup - here is a primer method. The exact method differs if you are talking with leaders, or middle management, or space occupiers. (Leaders as in intelligent, aggressive forward thinkers). You have to make it personal, and show them how the thing that they care most about (eg: career, personal security, promotion, or liability exposure) is at risk; which now-a-days is a true statement.
That is my experience, it may not be others.
These operators have no incentive to invest in proper skills. So they don't. They can charge almost anything they like. And sure enough, they do.
In an industry with such massive vendor lock-ins and regulatory capture regimes, organisationally you get to shovel a lot of money for sub-par pentests. The appetite to have yet another one, even from a known-good and reasonably priced provider, is quite hard to come by. After all, these approved providers will REFUSE to even look at a third-party pentest report, let alone let one guide their own testing.
So in aggregate the existence of these approved providers and their level of competence degrades the security posture of entire business domains.
Macchiavelli would have been proud.
I've always been curious about costs for real pentesting. 1-100k is a HUGE range, that can't mean the range in cost for the same project can be that big, can it? What's it cost to hire someone to run a decent set of tests on a Rails app, for instance? Could I really see that big of a range? Would I get way better results for for $100k?
You should get significantly better results the more one pays - because of the human element.
For example, $1k gets you a basic Nessus/Nmap (basic vulnerability/mapping) scans, maybe SQL vuln, with automated reporting. More gets you architecture, infrastructure, penetration, white/black scenarios, risk assessment, external organization liability, code repo, data, compliance, and regulatory footing. Those can surpass $100k easily.
And these issues look more like CTF style issues than real world security issues. If you're not thinking about the challenge in the right way, it is probably harder to see the vulnerabilities.
If I'm not familiar with pentesting methodologies, my only metric of satisfaction would be the pen tester's reputation i.e., if a well-reviewed pentester says my site is OK, then maybe my site is safe.
The business often gets in my way with budgets. I can spend it all on some strange behavior that I detect in the application. I like to spend my time like that. The more experience I get the more time I could spend, even on simple applications. The rabbit hole goes ever deeper.
When am I satisfied that I delivered good work? When I find a number of high impact vulnerabilities. Doesn't mean there aren't still more to be found, like I said you could go on and on, but it does mean that I fixed some bad. Sadly I have a fixed budget, and sometimes I find nothing, those are bad days.
In my experience customers are also satisfied with a report that contains few or no impactful findings. But an empty report is no guarantee that there really isn't anything to be found.
It's difficult to balance the desire to dive deep and to provide broad coverage. I once got bit by that.
I found and clearly documented no less than 3 absolutely critical issues (price adjustment in an e-commerce website). Felt pretty good about that test. Until, in production, someone else found an additional authentication bypass. Oops. I wasn't focused on that due the aforementioned input validation issues and got blinded by trying to find more of those. It's not easy.
"This one weird X" is a common clickbait pattern, and indeed, in this case, it is misleading: the article's descriptions of the purposely-introduced "weird" vulnerabilities aren't useful, only the freelance test results are.
The results are useless without a control.
They might have hardcoded admin-level passwords for debugging, then forgotten about it.
They might have mis-typed HTTP headers like 'set-cookie', after having written manually a lot of auth & session management that should really be done using well-vetted libraries instead.
Back at my startup days, I once worked with a poor guy who was dynamically generating individual IDs for each element on a page and their corresponding CSS for each page. He was unaware that CSS also had _classes_, which perfectly encapsulated the behavior he was trying to create. His work was complex, and full of errors, but it had a certain logic to it- it was the path of least resistance that he saw available to him.
These don't look like that. They just look like weird errors designed to avoid showing up on, or triggering false positives on, DAST tools. I thinking testing people's skills without those tools is a good goal, i just think the methodology here is wrong.
That looks like what modern bigcorp webapps do, with random looking css class names that change every release.
I realize that in this scenario it is literally designed-in, but I understand the point the author is trying to prove. If a "scanner jokey" gets results that tells him or her there is SQL injection, a competent tester will try to verify what their tool is telling them. If the tester is doing that in this case, they'll find other injection strings not working and (hopefully) start looking under the hood to see what is going on and discover this pathological hard-coded pw and be able to tell the client that. Maybe it's the work of a malicious dev?
I agree that a pentester with a lot of experience will have skills that are honed to find common bug patterns, but it's nice to be able to find these seemingly bizarre issues and have an explanation for the client. It shows you really understood the app and what it's doing.
The same is happening in the "smart contract audit" space. The results of the audit do not matter to the audience that wants to see the checkbox of "audit".
You can currently make a lot of money undercutting other auditors providing these audits.
The person requesting the audit is making multiple orders of magnitude more money by having the audit.
Because for most people its not about security, its about liability and having a proven attempt at trying to "security" implies in their minds that they are not liable.
Sadly security is a mindset and a process like healthy eating but we socially model it as a eating a single apple which translates to hiring one person for less than $100 and taking what they say as authority.
These are also the people who keep Walmart alive. They have limited reasoning ability, and even if they might suspect it's too cheap to be any good, they want it to be good enough and are willing to pretend that it is. These happen to also be the people who easily accept absurd and obviously false statements made by certain political leaders.
I know quite a few of these people, and I have tried countless times to advise them on reasonable choices, but it is like talking to a wall (if a wall could nod and pretend to agree).
I can kind of get behind the malformed header. But I have very hard time thinking up any suite of actions that would lead to a code resulting in leaked obfuscated query with a hard coded password. All while password somehow using a LIKE statement matching a sql injection.
Is there any precedent for this?
Edit: rewrote my sentence because it was barely readable.
As a result, those systems are typically harder to maintain or secure by just anybody, even by some experts who are not from the NIH mindset (here's where the author's approach isn't really fair, IMO). Proprietary systems are are less likely to present the typical bell curve security issues that an automated scanner would pick up.
What they are more vulnerable to is puzzle logic--the "what's going on here, hmm let's have a tinker" approach. Which is exactly what the author presented and it also matches his broader approach of "let's have a tinker...with freelancers".
It's a subjective security vs. objective security approach. Both sides are important. But both sides should know about how the other side works.
I'll trust them on that claim. But is my $100 getting me an industry-certified analysis?
I could hire a licensed MD Engineer Pharmacist Cosmetologist Veterinarian to walk my dog for me, but I don't expect them to perform heart surgery for $100
How in the world could that happen?
Edit: I guess it could happen if a developer decided to debug SQL via http response which seems pretty insane
Broken exception handling, for one.