I think this article makes a lot of good points, but those are overshadowed by the tone, and the seemingly personal grievance regarding security holding up software. In this specific sense, I think the article misses the mark. Many of the criticisms made about cybersecurity are valid, however whether or not they are valid does not really have much to do with whether they should be holding up software more, or less. This primarily seems like a problem of incentive: the security folks are not incentivized to help move the software along. They're incentivized to find issues, and when erring on the side of caution, hold up software deployment. And this holds true even if, as the author claims, the cybersecurity field is full of puffed-up pseudo-intellectuals.
Software developers and project leads are paid to ship product, even if its held together by glue and duct tape.
The problem is they don't have security and risk reduction in their incentives and automation, and they should, call it DevSecOps or whatever. After all, availability is one of the areas of security, but that usually falls on operations.
There's a dedicated department that already does that in most organizations -- risk management.
One could argue that 'cybersecurity' ought to be a component of 'risk management' versus being on its own which only adds to bloated organization structure and increases bureaucratic complexity.
Yeah, and my team and larger cyber org was under risk management, until some new exec decided to shift us under the technology org (a decision I do not agree with due to conflict of interest).
In some orgs, that is the case. In other orgs, risk management might be a functionally absent, with legal teams being reviewers of contracts and abdicating that role to outside counsel.
To be clear I'm not necessarily agreeing with the author's claims that cybersecurity is all puff and BS. I'm just arguing that even if we take the author's point, it feels a bit oblique to his primary grievance.
Software has distinct functional requirements for which you can test in a concrete way, either at a technical level, within a business, or in a competitive way in the marketplace. It's therefore possible to know what's "good" or "not" and in many cases to know who is a "good software engineer" or not.
Cybersecurity isn't like that. It's often hard to tell valuable cybersecurity practices from useless ones, because it's almost impossible to test them, and the practice of defense-in-depth means that useless activities are often piled on top of useful ones.
The target for cybersecurity should be "just good enough", but there's no feedback because "just good enough" has the same outcome as "pointless vendor quackery cargo cult review hell" for the objective function.
"Just good enough" is a constantly moving target. Its a bit like tech debt, what's good enough one year isn't the next, being PCI compliant doesn't keep you from getting customer cc #'s stolen.
Just good enough is also a question of risk appetite. If the business decides it wants to "accept the risk" rather than take whatever measures necessary to remediate a risk, that's up to them.
No; you see, that's the same problem. Software orgs don't have a problem with changing requirements; that's the nature of the beast.
Cybersecurity organizations usually have no idea what "just good enough" is, for the reasons I stated. So you end up with commercially impractical security requirements, and then vigorous sucking of teeth and requests for "compensating controls" (usually: do something equally impractical) and then demands that the business "accept the risk".
I've worked with a bunch of people in "compliance" professions (like line 2 compliance teams in banks, operational risk teams) but this attitude is significantly more common in cybersecurity than in other cases.
Oh, your problem is you don't have anyone doing risk assessments (see Risk Matrix for an example) and quantitative risk analysis and measuring cost to fix vs likelihood of occurrence and cost of an incident against that fix over an extended period. It's a bit like an in house insurance adjustment. Tangentially, a lot of companies decide to buy insurance as a "fix" for a risk. Because the insurance costs less than implementing whatever solution is required.
Just good enough often times is an orgs own policies and standards, and security teams catch when things aren't meeting those standards.
Your complaints are a lack of maturity in your cyber security organizations. There are cases where a business can absolutely decide its "good enough". But that decision is made by people other than those that point out the vulnerability. It's usually done by which ever team handles security remediation or they talk to someone who handles that analysis as part of the remediation process if the fix seems prohibitive. Cost to fix can also go down as new solutions become available or infrastructure changes and a new design makes it easier to build in a fix/control/detection or whatever. So re-evaluating "accepted risks" periodically to see if its something that's easier to implement is also part of it.
Cybersecurity shouldn't be in the business of saying no (unless its something insane like storing passwords in plain text on open shares), we're here to say, "here's some problems we see", here's where we think we can help, everything else is up to the frameworks and policies your company has set to adhere to or someone at a higher level to say cost exceeds risk, we accept the risk until such point cost comes down, etc. Cyber should also be helping to put in systematic controls that make good controls the default, and good practices easy to implement, so that ops and devs don't have to think about it.
I dont think any of that is inherent to cybersecurity, just lots of cybersecurity teams are dysfunctional in their org which leads to bad outcomes like pointless requirements.
In fairness, lots of the quackery cargo cult stuff often comes from outside the security team, and security is just tasked with enforcing it (e.g. it comes from contractual requirements with customers or other compliance stuff)
Sounds like someone is tired of getting dinged for using outdated security practices in their application designs.
I came from the other side of things. I know what its like, but the fact they care more about features and not fixing vulnerabilities or designing applications to have good security is a failing on their end. Shift left, do it early, so we're not finding it right when the product is getting ready to ship when its finally in a form ready to review.
>3. Build standardized patterns.
Yeah, you can have them, but then the various project teams don't follow them or even look at them during the design. Then wonder why its being reported later. Are they kept accountable when they don't follow the prescribed patterns and it causes project delays when its called out? No. It's our fault for road blocking.
>4. Abandon the perimeter model.
Uh yeah, we talk about zero trust but its the app teams saying "its an internal application, not publicly accessible so its not a big deal if there's no encryption or credentials are stored in the clear" in their arguments to why its not an issue or they shouldn't have to fix it.
>5. Advise, don’t dictate.
I can not speak for every place, but we recommend solutions and call them "Recommendations" but if there's another better solution to reducing the risk we're not the SME's in your product, as long as it adequately remediates the issues through some sort of control, detection, or fix, it's not really for us to say how its done.
>6. Ask platform teams to integrate security.
They're not interested, they're interested in copying and pasting code from past applications with the same old/bad practices. They even bring practices into new platforms that have great solutions for managing a risk (such as credential storage) and do it the old way (keep creds in a config file that ships with the artifact).
Cybersecurity isn't special, but its the developers (not all) IME that treat it like it is and not part of their job responsibilities.
Seems like a rage-bait title, but I kind of agree with the general premise that cybersecurity teams shouldn't treat themselves as solving unique problems that other auxiliary teams (like SRE/platform engineering) haven't already come across and/or already solved.
I am of the opinion that this massive push by big organizations (coupled with mandates for C-suite roles like CISO) into building a dedicated army of staffers for "cybersecurity" feels like just another attempt to bloat up the size of an organization and create more 'bullshit' jobs, as David Graeber put it over half a decade ago.
the actual reason for mandating companies to spend on cyber is because they are cutting costs on SRE/Ops by outsourcing all KTLO work to India and other offshore countries. If you ever looked at average S&P500 company IT Budget, there will be nontrivial amount dedicated to WITCH (Wipro Infosys Tata Cognizant HCL and friends) for outsourced KTLO work.
This makes it impossible to do anything meaningful de-novo on a high level, like create a good security architecture as a platform for all dev teams, or adopt a new security platform.
Outsourced companies do only a piece work on a ticket by ticket basis and require very specific instructions upfront.
Mandating companies to keep inhouse cyber staff makes it possible to grow talent inhouse and do high level designs of platforms to keep stuff secure
1. Outsource companies are NOT known to be staffed with cyber pros, you won't be able to get a meaningful return on your money. Good security pros will NEVER work for body shop like Infosys who only focus on minimizing staff costs.
2. Trust issues. Much easier to hire inhouse person whom you know and who shows up in the office (and you can sue him US court in case of malice), than some offshore Dunder Mifflin Corp which can disappear and show up under different name
Legal enforcement of NDAs, non-competes, and being able to chase down some a-hole who steals your intellectual property and sue them. Don't share your secret sauce with people you don't trust, and even you don't fully trust them, you can at least have legal recourse if they sell your access keys.
Compliance issues. Auditors love hearing that your security and auditing team is a revolving door of random Indian guys.
Quality, as you want your Sec Teams to really give a shit, push back on stuff, and not do the absolute minimum to close a ticket. You get what you pay for, and if you want to pay shit you'll get shit security.
Business integration, as ultimately it's about risk and talking to the business as to what they think is important. The distance from Mgmt and Security is often a lot smaller, and they'll have the "keys to the castle".
The interesting take to me is those could all apply to legal or accounting. These fields are also vital to a company above a certain size, yet partial or full outsourcing/contracting to an external cabinet is common.
At the end of the day it's a matter of trust ("you get what you pay for" feels weird to apply to Deloitte for instance. You absolutely get less than what you paid for and they get to pocket the most of it, you just don't care enough about the money to want to handle it yourself)
Anyone in cybersecurity who isn't a fucking moron (of which I freely admit there are many), without a doubt knows the problems they solve are not unique. As someone who has done it for nearly 20 years, me and my colleagues absolutely despise having to repeat the same shit, over and over. I want nothing more than to not be necessary.
I liken my job to being a janitor, and people can't seem to stop from pissing, shitting and trashing everything. It's goddamn 2023 and we still can't get people to always validate input or ensure proper constraints are built in.
Computer Janitor is a more correct description that Security Engineer. Because at the end of the day we are cleaning up and tidying others' mess that they left. Whether it is random software dependencies, or glaring holes in firewall config, or missing OS patches/whatever.
They aren’t bullshit jobs though in the way the author presents the problem.
Honestly in cybersecurity the big hacks that usually go on is the fact that people can get crypto lockers or a whole host of problems that attack humans. The whole argument of the above shouldn’t even be anything about software. The most effective thing to secure networks is to educate your whole staff on when not to click something suspicious so instead of fighting physics we are fighting human psychology.
I could argue the second is the business group overriding security practices because they accept or don’t care about the risk. So then people who were never born when the service was active have to deal with getting a project in with the vendor that doesn’t give a shit about you.
Security usually even is a technical problem it’s human we just like having cool stuff presented at a con because it’s fun.
Except the fallacy with this premise is that the other teams _haven't_ solved the problems, which is why the cybersecurity teams are necessary in the first place. Cybersecurity doesn't solve any problems that are inherent to computing and are unavoidable, cybersecurity solves problems that are created by the mistakes of other teams. It's an unfortunate truth and it's difficult for many people to swallow, but I have found in my career, which is nearly 20 years at this point, that this is true in every case that I have personally experienced. That is to say, if developers created secure software by design, and if infrastructure teams/operations teams handled their assets and processes in a secure manner, then there would be no need for cybersecurity outside of a few specialized roles.
Fully agree with the author, the often quoted CIA (Confidentiality Integrity Availability) actually belongs to the SRE platform's concern.
What I often see however, is SRE teams keep extremely understaffed (because "cost center") and often outsourced to offshore contractor in India - this basically means you can only keep the lights on via SRE team and cannot do anything else.
If you have a competent and fully staffed SRE Platform engineering team - you will NOT need a separate cybersecurity team.
But if company decided to cut costs on SRE folks, they then will have to invest heavily in cybersecurity team. At least because of mandated regulation (FedRAMP SOX HIPAA GDPR compliance disclosure etc) - it is easier to justify cyber budget, than the SRE budget
Plus recent laws mandate having CISO as a high level executive (chief incident scapegoat officer) with huge budget and disclosure of recent hacks etc
I am gonna surprise you, but nobody from cybersecurity is finding vulnerabilities either.
ALL of cybersecurity is just operating and configuring a platform, whether it is Qualys or Snyk or Wiz or something else. "cyber" people are basically TOOL MONKEYS, and if SRE people can handle operating and configuring kubernetes - they sure as hell are capable of operating and configuring Qualys and Snyk and Wyz platforms.
Pentest is the same, I have not met any inhouse dedicated pentest folks, it is always third party contractor hired as needed for a short period of time. All inhouse fulltime cyber engineers are dedicated to blue team operations, rather than red team operations.
As part of the industry, I can tell you I've personally found dozens (hundreds?) of vulnerabilities per year for the more than decade running of my career. The same is true of my collegues.
I have met "cybersecurity teams" who are "tool monkeys" in the way you suggest. Usually these individuals are part of an extended compliance team. They are trying to check compliance boxes by having some level of automated scanning, because their parent organization/team can't/won't staff professionals capable of this or capable of scaling it or focusing attention on where it matters.
I don't want to oust you for where you work or your role within those companies - but really surprised to hear that there's no inhouse pentesting and no in house red teaming. I haven't worked at a company yet (on my fourth) that matches the experience you described.
a lot of your suggested vulnerabilities could be prevented from appearing if company just hired a person who can configure SAST and WAF and etc solutions (solving a class of security problems from showing up on a platform level, not on per each vulnerability instance level).
Why rely on an in-house redteamer who may find one or two vulns per year, or may not.
I think "configuring SAST" goes back to what you had previously said about "tool monkeys". Now there's thousands of potential issues. Many are false positives. Who is going to tune that SAST for accuracy rate? What about vulnerabilities that are framework specific? Will the SAST even allow for this? Who is responsible for triaging all of these alerts? Are they capable of correctly addressing them?
If you just set up SAST and walk away, you end up with more noise than you do signal. And you created the need for security professionals to process the results.
I think the article had better suggestions for scalable security, for example #3 Build standardized patterns and #7 Provide isolation patterns. Better to systematically prevent SQLi with a platform/library than try to detect all variations of SQLi with SAST!
WAF? It's something I would put in front of a product I have confidence in as a defense-in-depth. But would I rely on it as the sole security investment? Absolutely not. The reason is not all vulnerabilities are web-based, and many do not have differentially detectable payloads (missing authz on an API isn't going to get caught by WAF).
Why Red Team? Because the leading way businesses get compromises is with phishing attacks.
A security team needs to ensure application security, network security AND operation security. A SAST or WAF aren't going to do that. Neither with Qualys, etc.
>>Now there's thousands of potential issues. Many are false positives.
Inspecting output/logs of Qualys is no different than inspecting logs of kubernetes (or other SRE platform). and both overlap.
If you have highly skilled SREs - task them with security. If you dont have good SREs, you have to keep IT architects (and call them infosec) who will be able to look at all your IT Zoo across all your on-prem datacenters and cloud accounts and can make a call to do X,Y, and Z to keep company secure.
and who can recover your infra from groun zero in case you got ransomwared
I disagree. I think inspecting the output from Qualys (and other tools, including SAST) are substantially and manifestly different from inspecting Kubernetes logs.
I would worry the argument about "highly skilled SREs" could become a "true Scotsman" argument. If a business has any persons who are skilled enough and plentiful enough to process all of the security output and take action on them, let it be so.
My experience is that in practice, there are not the resources to process all of the output that the tools generate. Do you have experience to the contrary where this has been done at a company scale or is your argument a theoretical one that you believe stands to reason?
If you task your fully staffed SRE division to have some people doing cybersecurity full-time, what's the difference between having that and an actual cyber security team? And if you have that, why wouldn't you want to hire experts in that field instead of cross-training or finding generalists?
hiring security experts is expensive and very few companies are able to afford and retain them.
ask yourself what is cheaper: hire and retain Cloud Operation admins in SRE org, hire and retain Cloud security experts in cybersecurity org -- vs hiring a cloud security guru and task him overseeing with maintaining and security $platform_name ?
very few companies are able to hire and retain SRE-Kubernetes operators and Kubernetes security architects, so it kinda makes sense to merge and hire one good expert
Now you're just haggling over price. Ask yourself, what's more expensive, a good security team, or getting hacked? For LastPass, it's existential.
It just depends on the size of the respective orgs. If engineering is 5 people, a dedicated security person doesn't make sense. At 500 you might be able to get away with one. At 5,000 engineers though, you real do need more than one good security expert.
I'm a full time red team. Depends on your org, but you're not ready for a red team until you get the basics of blue straight.
Most of our blue is threat and anomaly detection. They spend a lot of time in tools, sure but its usually a SIEM like Splunk or similar tools for event correlation and behavior detection like Crowdstrike. Qualys isn't going to find custom written malware using common utility as a C2 channel.
>yeah, but crowdstike or whatever antivirus you use can.
...No, it really can't. Not all of the time. It doesn't understand your business logic behind your architecture, it doesn't know how to separate normal from anomalous behaviour (yes, even if you have XtremeAI or whatever the vendor peddled to you), it cannot correlate alerts and activites in patterns not explicitly programmed and it can't generate alerts with 100% precision. In short, it's not intelligent.
thats not the job of red team, this is blue team work (Detection Engineering or Incident Response or Security Operations whatever they are called) - a SIEM tool monkey basically.
I don't think you know what the point of a red team is, if you did you'd understand how oxymoronic your statement is.
A part time red teamer is just a pen tester. That's not what a red team does, and adequate threat emulation isn't someone who flies in for two weeks does somethings that you can only accomplish in two weeks, then fly out to the next company to do the same things. You actually need to dig deep and research a companies individual weaknesses and that takes time. Threat actors have that kind of time to sit and observe and probe, a part time noisy pen test isn't going to get you more than the basics.
Also a blue team has to be right all the time, the attacker only has to get things right once. That's why you constantly test your assumptions with a red team and give blue someone to "train" with and improve with that isn't an actual threat actor. You don't want the first time you fight to be in the ring, you need to spar.
A lot of pen testing consultancies brand themselves as red teams, but a lot of them are just rebranding pen testing services to the ignorant.
yeah nice word salad, except that it doesnt happen in real world. what you described is probably just one threat intelligence engineer that also does bunch of other stuff like IR
Threat Intel informs the red team, and we'll model our ops or threat emulation exercises off of that information, we also have a few members that came over from threat intel. Last I checked, Threat Intel engineers don't write malware to simulate malware real threat actors that target a companies industry use.
I think you're speaking from your limited experience at your own company. Again, you have to have a mature blue team before you're ready for a red team. Most orgs aren't ready for that, especially if you think Qualys covers your threat models. That's just vulnerability management and might catch some apps/hosts that missed the patch cycle.
If it's word salad, maybe its because you're out of your depth when and ignorant of how these programs are supposed to work.
That was also my short experience in the area, and it was boring AF, almost soulcrushing. Maybe I just got unlucky, but it feels that a lot of security engineering positions are like that.
Would anyone happen to know of what search terms should one use to find open positions that are focused on security tool building rather than deploying a purchased solution?
Tier0: Pure security research shops, like Google Project Zero and alike. YOu can only get this job after doing PhD in Computer Security at top-tier university lab, and have publications on stuff like linux kernel fuzzing, have 10+ linux CVEs under your belt, and listed as contributor to a well-known open-source security tool.
Tier1: top security vendors: crowdstrike zs qualys pan cisco and etc. R&D (or Business Unit) departments responsible for developing security product features and maintaining security content.
Tier2: FAANG and adjacent cloud-native companies, job title "security software engineer" or "security engineer" or "$platform_name engineer" or "cloudsec/appsec engineer"
Tier3: pre-IPO unicorns which are hiring for Product Security and AppSecurity positions
...
Tier10+: all other public companies, regular S&P500 infosec department.
the general rule is the higher the salary the more demanding the job will be and the tougher is competition to get that job
> I am gonna surprise you, but nobody from cybersecurity is finding vulnerabilities either.
My recent interview as a security engineer had distinct sections for software review and for architecture analysis. My last job had me running Burp Suite once a month (because it was fun and I didn't mind doing it), but that was roughly 30 minutes every 4 weeks.
Sounds like you've worked with some quite junior security teams.
interview vs real work is different. FAANG engineers are not flipping a binary tree either, yet this is a common question they ask.
as for Burp Suite - why would a company keep engineer full-time only for him to run burp suite for 30 minutes once a month? Sounds like poor investment on a company's side.
> If you have a competent and fully staffed SRE Platform engineering team - you will NOT need a separate cybersecurity team.
Not all cybersecurity problems are platform problems, or even technical problems. Even if you decide to gather all AppSec/InfraSec engineers inside of SRE/SysAdmin, this creates two problems:
1 - They are now separate from the main cybersecurity team, which is solving other problems and that in turn creates all sorts of issues ranging from lack of coordination to managerial ones.
2 - This creates a HUGE lack of separation of duties (or a conflict of interest). The executor of a task should never be the auditor of a task. You cannot expect the SRE team to go out of their way to keep finding flaws in their own work and exposing them to the board so they can be forced to reprioritize and fix these flaws. The SRE can and should have security built-in as much as possible, but an external team is still required.
SRE is really a more familiar "sysadmin" from good ole days. You know, that bearded IT guru in a basement floor, will curse and shout at you, but will keep your company running when it comes to IT.
if you switch back to sysadmin model - having a single sysadmin do all the SRE and security work, divided per platform, then separate security is indeed feel like extra.
You can obviously hire independent third party security auditors and do pentest, but that thing could be done once a year. Often times your financial auditors (Big 4 accounting firms) will offer IT audit and pentest in a bundle and you can get a good deal.
Cybersecurity is special in a real sense: it's one of the few groups typically charged with enforcing required compliance controls. I was a senior engineer for many years before moving full-time into security, and it's amazing how many of those annoying inconveniences are actually mandated from external customers or frameworks[0]. As an engineer, I don't really care if every PR has an associated Linear/Jira/whatever ticket; sometimes it's enough to say "I'm fixing this thing because it needs to be fixed". As a compliance person, SOC2 says that all change requests must have an associated ticket, and my personal feelings about that aren't part of the equation.[1] Think Security is a pain in the neck? Work in a regulated industry like healthcare or finance where you have lawyers instead of colleagues giving you a list of commandments.
Also, I've never slept so well as I did the month after I was allowed to hire my CISO replacement. It's like looking up and seeing the Milky Way instead of the Sword of Damocles.
[0] Having worked at shops where Security didn't take time to explain those things, I've vowed never to be that person. If I ask you to do a thing, I'll tell you why it's something we have to do. We should all collectively be better about that.
[1] Although there's an analogy between "compliance" and "unit testing". A side benefit of thorough test coverage is that you tend to write better structured, more loosely coupled, easily testable code. That's all good to have even if you never ran a single test. Same with compliance. The ability to audit all the things means you're more likely to design your systems so that they can be inspected, instrumented, and monitored. In the "where's the ticket?" audit case, know what's more annoying than having to match every PR to a specific ticket? Having to individually assess a couple hundred exceptions to very that each of them reflects a reasonable judgment to violate the guidelines.
> As a compliance person, SOC2 says that all change requests must have an associated ticket
Does soc2 actually say this? IIRC their spec docs are proprietary so chumps like myself can’t actually read them. But a (imo great) SOC2 auditor I worked with previously allowed us to treat GitHub PRs as their own sufficient justification. We still used Jira for most, but for minor fixes, quick upgrades, no Jira necessary. A former coworker said they did the same at a previous job. I think, in this particular case, it isn’t SOC2’s fault, it is the fault of auditors who are afraid to not just do what everyone else does.
Sigh. I don't have the docs in front of me, and even if I did, darned if I'm touching those diabolic tomes without a 10-foot pole and a paycheck. But they do say something along the lines that all change requests have to have some kind of a ticket. The good/bad part is that it doesn't specify exactly what that means. If you can find an enlightened auditor who accepts an explanation in the PR, awesome! There's no guarantee that the _next_ auditor will see it the same way. The whole audit process was a back-and-forth with explaining why we think our controls satisfy the request, and the end result was a clean report.
> it isn’t SOC2’s fault, it is the fault of auditors who are afraid to not just do what everyone else does.
There's some liability for them if they're too lax. Suppose a breach happens and a given control, as written in the spec, would have prevented it. It could come back on the auditor who signed off on a clever interpretation of that control.
I haven't worked on anything that had to adhere to that standard, but I also have yet to work somewhere where a "Change Request" was a very specific thing related to workflows that may or may not involve software, i.e. hardware. A pull request rarely had relation to a proper Change Request.
A control that results in a paper trail without making anything materially better, but must still be obeyed to the letter because a certification depends on it? Why, I've never heard of such a thing, I do declare!
It depends what’s in scope and the control being evaluated. For example, if your SDLC/release and infrastructure processes the environment(s) that are built from them are in scope of your SOC2 audit, then a change to the code deployed to the in-scope environment must meet a defined policy for change control. Auditors aren’t eager to accept “there is no SDLC,” or “our change control policy is to deploy to production whenever.”
A lot of SOC2 is hand waived as paperwork, but the point is that an organization (and their independent auditor) can show evidence that they have established certain processes and controls, and that they can demonstrate compliance with their word. (This belies the tension that some use to discredit independent auditors: that the client pays the auditor for their report. What’s less well understood is that the auditor has a much greater vested interest in being honest, and that they’re going to get questioned if a matter ever goes to court. No one client is worth losing the entire firm. There are, of course, bad auditors just as there are bad developers, admins, accountants, etc.)
I spent time working in compliance and led the inaugural SOC2 Type 2 audit at a household name. These days I spend my time primarily on infrastructure.
the SOC requirement is that any change can 1) be looked at before happening, and 2) there is an audit trail. A PR generally works because it's auditable and there may not be an impactful change.
"change request" is often a specific term in things Project Mgmt, SDLC, etc.
"It really doesn’t make sense for cybersecurity to have its own special snowflake process for things. It does not make sense operationally, philosophically, or socially."
In general, I believe it does - cybersecurity is unique in the sense that you're dealing with intelligent adversaries that are actively trying to break your code. It's not like SRE work or normal software engineering. But there's also a lot of snake oil in the industry and most "cybersecurity" professionals don't actually know how to find vulnerabilities.
"Attackers accessing our systems without our consent is one type of failure, but not the only kind. Reliability failures are arguably both more frequent and more damaging when they occur".
This will completely depend on your product, but I've seen this sentiment across industries. "Risk = probability * impact" doesn't work when you throw security into the mix. The author sounds like a very common type of software developer who takes security issues in their code like a personal attack.
I agree with this, finding vulnerabilities is really hard, and there's people coming out of college with no experience anywhere else in the field trying to get high paying jobs looking for vulns in technology they don't understand or have never worked with.
One of the best vulns I've ever remediated was one discovered by an application developer in their own authentication that was prescribed by our platform. He knew it was a problem and knew it needed to be fixed, but knew he needed us to report it before a fire got lit under the appropriate people's butts to make it get fixed.
>"Risk = probability * impact" doesn't work when you throw security into the mix.
In my experience, that's primarily because cybersecurity people know neither the probability nor the impact of the things which they are calling risks, which moreover means they don't know how to order them from a prioritization perspective.
As a cybersecurity person, probability is typically high since so many people release POCs and impact is almost always they steal all our IP and lock our computers, grinding business to a halt. CVSS is kind of helpful for triaging stuff, but if you are even a moderately sized org, it's literally fire drills all the time or you bury your head in the sand.
Business people typically don't wanna hear that and then act all shocked pikachu face when it happens.
Plenty of risks arent vulns in public software. Some are vulns in your own software. Some are just deviations from best practises or lack of system isolation.
Even among public vulns with known PoC, the impact is going to depend on context. Plenty of times the impact is zero based on how software is used.
It was a view in response to the above post, not a generalized view I take on things.
I’m actually a fan of embedding security resources into the eng teams and having those resources fix issues rather than toss shit over the wall. Knowing how software works is really hard without living inside the code base.
Ah, yes, this is the "start from the assumption that everything has failed" trick.
"if attackers can run arbitrary code on our systems then they can exploit vulnerabilities!" or "if attackers can send arbitrary payloads to services which are turned off in the actual image and in any case not accessible without breaching a perimeter firewall and passing through a DMZ then they can exploit vulnerabilities!"
OK, sure, what about "attackers kidnap the families of our AD admin team and force them to do whatever they want"?
Also, "high" meaning "sometimes we just don't care".
No, sorry, I do not care that (e.g.) CVE-2023-4911 has been categorized as "high", I am not in a rush to replace every single Fargate task that contains a vulnerable glibc because it's just not an interesting vulnerability in that context: let's assume you (a) bypass the payload validation on the (private!) API Gateway, (b) construct a payload that allows some kind of RCE in my JVM, (c) somehow escape from the JVM sandbox to run arbitrary native code, (d) successfully implement the attack, (e) get root in the underlying VM ... it's a Firecracker VM that doesn't have ANYTHING ELSE in it.
But, no, it's a "high", I better start cutting change tickets for literally everything in the universe to conduct an incredible load of busy work in order to do a no-op.
Really smart cybersecurity would be (I'm just making this up, I haven't really thought about it) "you know, if we're looking at services contained in a VM security boundary like Fargate, perhaps we don't care about Attack Vector = Local or Physical, or User Interaction = Required?" but what we actually get is "raw CVSS score tells you all you need to know and absolutely dictates urgency and triage priority".
> Ah, yes, this is the "start from the assumption that everything has failed" trick.
I am actually fairly data driven, and it's why things like SBOMs are exciting. Based on my experience, if something is exposed to the world and that thing has a vulnerability... within a period of reasonable time (say within 12 months), it will be exploited.
> No, sorry, I do not care that (e.g.) CVE-2023-4911 has been categorized as "high"
I wasn't actually referring to my "high" being the cvss scores. It was likelihood of exploitation(in my org), not the score. In your example, that would be a local exploit, so wouldn't be a high in my book unless you are shoveling user input to a CLI, which I would hope isn't happening.
Personally unless you know there to be higher risks for certain things, if it ain't 9+ on cvss, it gets a ticket cut to deal with it like any other but at some convenient time in the future.
> but what we actually get is "raw CVSS score tells you all you need to know and absolutely dictates urgency and triage priority"
Your security org sucks and are doing it wrong. I'm genuinely sorry, and I'll say that not all of us are like that :(
> cybersecurity is unique in the sense that you're dealing with intelligent adversaries
I disagree with this on the grounds that normal software engineering is dealing with intelligent adversaries as well. They may not be "adversarial" in the sense they are maliciously trying to _break_ the code, but if it is successful in performing _some_ work they will attempt to use it to perform other work as well, whether it was intended to or not.
But, what the author seems to be saying here is that a "vulnerability" is just a fancy word for a bug that is not inherently more significant than a crash or outage (where the vulnerability may not be available to exploit anyways). The suggest that cybersecurity teams may be more successful by integrating with the pipelines developers and SREs already have for dealing with issues (including critical ones) than interrupting their work and trying to assert how much more important cybersecurity is.
The first side is that ultimately the argument that the author debunks is a strawman argument, and the counter-arguments are equally sophomoric.
The second side is that I agree with the author on (some) of the recommendations and I think its great that the author made recommendations, instead of just pointing out perceived problems.
Cybersecurity isn't special. At least not any more special than any other subfield is in its own way. But the rest of the article doesn't follow from this axiomatic position. In fact, it's irrelevant to the meat of the article (the recommendations), and comes across to me as griping.
I think the article misses some of the realities of the business environment.
Why do (some) security teams (sometimes) try to put in roadblocks to software releases? Usually it is because the the release is going to have a business impact for which: (a) the business has not acknowledged (b) no person is accountable for (c) there are no plans for.
Why do security teams feel they must dictate outcomes rather than recommend? Usually because their leaders (running up the the Board or Executives) are holding the security team accountable for outcomes they have only indirect control over. Another common reason is that the incentive structures of various organizations push for development groups to release as quickly as possible, feature release estimates rarely include security work estimates, and teams/managers feel pressure to release anyway to avoid delaying due to missed work estimation at the project start.
Usually, these kinds of issues represent the failings of other structural aspects of the business.
Ideally security teams wouldn't be needed at all, the same as SREs and many other kinds of roles. Ostensibly developers would be so capable, so knowledgeable, so well rounded, so careful, that they would plan for, mitigate and manage all operational risks on their own.
There's depths to explore on this topic. I'm excited for the article author that they are just starting to get introduced to it. I hope they continue to explore this as they develop more seniority and experience, and share in more detailed public posts as they go along.
I think there is one other unaccounted aspect that the article doesn't acknowledge.
I've never seen a SRE lead go to prison for an outage, but as of late a few CISOs have been charged, convicted, and sentenced to prison for failings on their watch.
Cybersecurity may not be special but the results of a failing there are unique compared to SRE work.
> It really doesn’t make sense for cybersecurity to have its own special snowflake process for things. It does not make sense operationally, philosophically, or socially. It does not make sense to sustain systems resilience. And it even does not make sense for software security.
Sorry, but it actually does make sense in multiple ways.
Firstly, the core purpose of cybersecurity in the C-suite is to ensure the bottom line. System resilience is not the priority, the priority is the stock price. That means that it's way more important to triage an issue, collect evidence, determine impact, and see how many ways you can legally hide evidence or manipulate the situation to deny you ever even had a security incident. The quieter it stays, the smaller the impact to the bottom line. Keeping compromises to a minimum, improving UX, meeting modern best practices, etc are like the farthest things from their minds. And the people who run the company dictate what these teams prioritize. So to "the company", there's a good reason why they focus on their own thing.
Secondly, security is really frickin' hard. Humans are not great at being masters of many things. You need a specialist who understands security well, and has the tools, and training, to do security-specific things. Why do you think "platform teams" (aka the new-new-sysadmins) exist and we don't just have developers build their own platform? Because there's ops crap in there that devs will never know, and shouldn't need to. Humans work more efficiently when they can focus on specific things, keep context switching to a minimum, and maximize their specific knowledge to a small domain. That includes security tools, practices, policies, etc.
Thirdly, because of the competing priorities of security, and the specialization needed to truly do it well, you have to have a "security process" that people who aren't security experts can follow. You also want a "security process" because processes in general are useful tools to keep things working correctly. Even "security people" should be following a "security process". And there needs to be "security people" to come up with the process (ideally in collaboration with the other teams).
What you don't want is bad process, or bad teams, or bad special snowflake solutions, etc. You want it to be supportive of other parts of the business, rather than a hindrance. That's what DevOps is all about - getting people to collaborate together, enabling self-service, increasing efficiency, improving quality, reducing waste, etc, etc, etc. You can do that with security too, and it doesn't mean you have to get rid of your security teams/solutions.
I think the distinction between cybersecurity and sre is that cybersecurity risk is much harder to quantify.
Cybersecurity is full of things that are both very unlikely and very bad if they happen.
What is the risk of some vuln burried deep in the stack that likely nobody will ever notice and even if they did even more unlikely they can further use it as a pivot point? Hard to say, but maybe close to zero unless you are a company that people specificly target. At the same time it can be a business ending risk. With SRE , even in the absolute worst case you can spin up new servers. You can't unleak your customers sensitive data.
I think cybersecurity is hard because we are uniquely bad at evaluating cybersecurity risk which means we are bad at figuring out what appropriate mitgating action is
> I think cybersecurity is hard because we are uniquely bad at evaluating cybersecurity risk
It's not that hard to evaluate. The risk is high! Pretty much anything can be hacked if you try hard enough... What's hard is everything else. It's hard to accept. It's impossible to quantify. It's hard to justify caring about it. It's hard to care. It's hard.
Cool, I agree with the author in concept, but there are so many goddamn engineers that ship incorrect/buggy/insecure software that if it wasn't its own function, nobody would even pretend to care. In my experience, most Project/Product/Program Managers sure as shit aren't going to prioritize it.
Not every org is FAANG. Unfortunately in many (most) organizations, in house developers get caught quite frequently eating the metaphorical security crayons.
They are similar, but different. Like ops is similar to eng, but different.
For one, they can quantify downtime easier. Also, in most orgs, cybersecurity is responsible/accountable for the problems, but only can indirectly influence things.
The goal of cybersecurity should be to align cybersecurity with existing processes, but minimizing friction. What the author experienced seems to be a high friction security program, which is the result of a suboptimal cybersecurity program.
This is unfortunately the reality in many companies because cybersecurity is implemented as a subset of a high level information security framework without qualified people to connect the high and low level requirements.
In the past, security departments were the "firewall gatekepeers", choosing who to allow or deny access. They started to change over time from gatekeepers to support the business but this transition is not complete as we know from experience.
The CISO has also has the dillema of having to support the business but will also be held accountable for any hack, which increases the tension on reducing friction vs increasing friction (and increasing control).
This is not an easy problem to fix, but I particularly think it's very productive to see posts like this so we can bring this topic to light and find the right balance.
And for those interested in learning how to add security to their organizations with minimal friction (security by design, especifically), I'm creating some webinars (free on youtube) here: https://devops.security/webinars.html
91 comments
[ 3.3 ms ] story [ 171 ms ] threadSoftware developers and project leads are paid to ship product, even if its held together by glue and duct tape.
The problem is they don't have security and risk reduction in their incentives and automation, and they should, call it DevSecOps or whatever. After all, availability is one of the areas of security, but that usually falls on operations.
There's a dedicated department that already does that in most organizations -- risk management.
One could argue that 'cybersecurity' ought to be a component of 'risk management' versus being on its own which only adds to bloated organization structure and increases bureaucratic complexity.
... her.
Software has distinct functional requirements for which you can test in a concrete way, either at a technical level, within a business, or in a competitive way in the marketplace. It's therefore possible to know what's "good" or "not" and in many cases to know who is a "good software engineer" or not.
Cybersecurity isn't like that. It's often hard to tell valuable cybersecurity practices from useless ones, because it's almost impossible to test them, and the practice of defense-in-depth means that useless activities are often piled on top of useful ones.
The target for cybersecurity should be "just good enough", but there's no feedback because "just good enough" has the same outcome as "pointless vendor quackery cargo cult review hell" for the objective function.
Just good enough is also a question of risk appetite. If the business decides it wants to "accept the risk" rather than take whatever measures necessary to remediate a risk, that's up to them.
Cybersecurity organizations usually have no idea what "just good enough" is, for the reasons I stated. So you end up with commercially impractical security requirements, and then vigorous sucking of teeth and requests for "compensating controls" (usually: do something equally impractical) and then demands that the business "accept the risk".
I've worked with a bunch of people in "compliance" professions (like line 2 compliance teams in banks, operational risk teams) but this attitude is significantly more common in cybersecurity than in other cases.
Just good enough often times is an orgs own policies and standards, and security teams catch when things aren't meeting those standards.
Your complaints are a lack of maturity in your cyber security organizations. There are cases where a business can absolutely decide its "good enough". But that decision is made by people other than those that point out the vulnerability. It's usually done by which ever team handles security remediation or they talk to someone who handles that analysis as part of the remediation process if the fix seems prohibitive. Cost to fix can also go down as new solutions become available or infrastructure changes and a new design makes it easier to build in a fix/control/detection or whatever. So re-evaluating "accepted risks" periodically to see if its something that's easier to implement is also part of it.
Cybersecurity shouldn't be in the business of saying no (unless its something insane like storing passwords in plain text on open shares), we're here to say, "here's some problems we see", here's where we think we can help, everything else is up to the frameworks and policies your company has set to adhere to or someone at a higher level to say cost exceeds risk, we accept the risk until such point cost comes down, etc. Cyber should also be helping to put in systematic controls that make good controls the default, and good practices easy to implement, so that ops and devs don't have to think about it.
In fairness, lots of the quackery cargo cult stuff often comes from outside the security team, and security is just tasked with enforcing it (e.g. it comes from contractual requirements with customers or other compliance stuff)
I came from the other side of things. I know what its like, but the fact they care more about features and not fixing vulnerabilities or designing applications to have good security is a failing on their end. Shift left, do it early, so we're not finding it right when the product is getting ready to ship when its finally in a form ready to review.
>3. Build standardized patterns.
Yeah, you can have them, but then the various project teams don't follow them or even look at them during the design. Then wonder why its being reported later. Are they kept accountable when they don't follow the prescribed patterns and it causes project delays when its called out? No. It's our fault for road blocking.
>4. Abandon the perimeter model.
Uh yeah, we talk about zero trust but its the app teams saying "its an internal application, not publicly accessible so its not a big deal if there's no encryption or credentials are stored in the clear" in their arguments to why its not an issue or they shouldn't have to fix it.
>5. Advise, don’t dictate.
I can not speak for every place, but we recommend solutions and call them "Recommendations" but if there's another better solution to reducing the risk we're not the SME's in your product, as long as it adequately remediates the issues through some sort of control, detection, or fix, it's not really for us to say how its done.
>6. Ask platform teams to integrate security.
They're not interested, they're interested in copying and pasting code from past applications with the same old/bad practices. They even bring practices into new platforms that have great solutions for managing a risk (such as credential storage) and do it the old way (keep creds in a config file that ships with the artifact).
Cybersecurity isn't special, but its the developers (not all) IME that treat it like it is and not part of their job responsibilities.
I am of the opinion that this massive push by big organizations (coupled with mandates for C-suite roles like CISO) into building a dedicated army of staffers for "cybersecurity" feels like just another attempt to bloat up the size of an organization and create more 'bullshit' jobs, as David Graeber put it over half a decade ago.
This makes it impossible to do anything meaningful de-novo on a high level, like create a good security architecture as a platform for all dev teams, or adopt a new security platform.
Outsourced companies do only a piece work on a ticket by ticket basis and require very specific instructions upfront.
Mandating companies to keep inhouse cyber staff makes it possible to grow talent inhouse and do high level designs of platforms to keep stuff secure
1. Outsource companies are NOT known to be staffed with cyber pros, you won't be able to get a meaningful return on your money. Good security pros will NEVER work for body shop like Infosys who only focus on minimizing staff costs.
2. Trust issues. Much easier to hire inhouse person whom you know and who shows up in the office (and you can sue him US court in case of malice), than some offshore Dunder Mifflin Corp which can disappear and show up under different name
That never stopped any of these companies from doing it IMHO.
but consider them like any retail fastfood place - you will get standardized BigMac, and no ala carte steak
Compliance issues. Auditors love hearing that your security and auditing team is a revolving door of random Indian guys.
Quality, as you want your Sec Teams to really give a shit, push back on stuff, and not do the absolute minimum to close a ticket. You get what you pay for, and if you want to pay shit you'll get shit security.
Business integration, as ultimately it's about risk and talking to the business as to what they think is important. The distance from Mgmt and Security is often a lot smaller, and they'll have the "keys to the castle".
At the end of the day it's a matter of trust ("you get what you pay for" feels weird to apply to Deloitte for instance. You absolutely get less than what you paid for and they get to pocket the most of it, you just don't care enough about the money to want to handle it yourself)
I liken my job to being a janitor, and people can't seem to stop from pissing, shitting and trashing everything. It's goddamn 2023 and we still can't get people to always validate input or ensure proper constraints are built in.
I don't expect magic, but at least cover the absolute basics. Then I might be able to figure out something more interesting or rare.
Or if I get report that something has CVE, just tell me if that is a problem for you or not.
Honestly in cybersecurity the big hacks that usually go on is the fact that people can get crypto lockers or a whole host of problems that attack humans. The whole argument of the above shouldn’t even be anything about software. The most effective thing to secure networks is to educate your whole staff on when not to click something suspicious so instead of fighting physics we are fighting human psychology.
I could argue the second is the business group overriding security practices because they accept or don’t care about the risk. So then people who were never born when the service was active have to deal with getting a project in with the vendor that doesn’t give a shit about you.
Security usually even is a technical problem it’s human we just like having cool stuff presented at a con because it’s fun.
What I often see however, is SRE teams keep extremely understaffed (because "cost center") and often outsourced to offshore contractor in India - this basically means you can only keep the lights on via SRE team and cannot do anything else.
If you have a competent and fully staffed SRE Platform engineering team - you will NOT need a separate cybersecurity team.
But if company decided to cut costs on SRE folks, they then will have to invest heavily in cybersecurity team. At least because of mandated regulation (FedRAMP SOX HIPAA GDPR compliance disclosure etc) - it is easier to justify cyber budget, than the SRE budget
Plus recent laws mandate having CISO as a high level executive (chief incident scapegoat officer) with huge budget and disclosure of recent hacks etc
So the SRE team is supposed to find vulnerabilities in the codebase and pentest the network?
ALL of cybersecurity is just operating and configuring a platform, whether it is Qualys or Snyk or Wiz or something else. "cyber" people are basically TOOL MONKEYS, and if SRE people can handle operating and configuring kubernetes - they sure as hell are capable of operating and configuring Qualys and Snyk and Wyz platforms.
Pentest is the same, I have not met any inhouse dedicated pentest folks, it is always third party contractor hired as needed for a short period of time. All inhouse fulltime cyber engineers are dedicated to blue team operations, rather than red team operations.
I have met "cybersecurity teams" who are "tool monkeys" in the way you suggest. Usually these individuals are part of an extended compliance team. They are trying to check compliance boxes by having some level of automated scanning, because their parent organization/team can't/won't staff professionals capable of this or capable of scaling it or focusing attention on where it matters.
I don't want to oust you for where you work or your role within those companies - but really surprised to hear that there's no inhouse pentesting and no in house red teaming. I haven't worked at a company yet (on my fourth) that matches the experience you described.
Why rely on an in-house redteamer who may find one or two vulns per year, or may not.
I think "configuring SAST" goes back to what you had previously said about "tool monkeys". Now there's thousands of potential issues. Many are false positives. Who is going to tune that SAST for accuracy rate? What about vulnerabilities that are framework specific? Will the SAST even allow for this? Who is responsible for triaging all of these alerts? Are they capable of correctly addressing them?
If you just set up SAST and walk away, you end up with more noise than you do signal. And you created the need for security professionals to process the results.
I think the article had better suggestions for scalable security, for example #3 Build standardized patterns and #7 Provide isolation patterns. Better to systematically prevent SQLi with a platform/library than try to detect all variations of SQLi with SAST!
WAF? It's something I would put in front of a product I have confidence in as a defense-in-depth. But would I rely on it as the sole security investment? Absolutely not. The reason is not all vulnerabilities are web-based, and many do not have differentially detectable payloads (missing authz on an API isn't going to get caught by WAF).
Why Red Team? Because the leading way businesses get compromises is with phishing attacks.
A security team needs to ensure application security, network security AND operation security. A SAST or WAF aren't going to do that. Neither with Qualys, etc.
Inspecting output/logs of Qualys is no different than inspecting logs of kubernetes (or other SRE platform). and both overlap.
If you have highly skilled SREs - task them with security. If you dont have good SREs, you have to keep IT architects (and call them infosec) who will be able to look at all your IT Zoo across all your on-prem datacenters and cloud accounts and can make a call to do X,Y, and Z to keep company secure.
and who can recover your infra from groun zero in case you got ransomwared
I would worry the argument about "highly skilled SREs" could become a "true Scotsman" argument. If a business has any persons who are skilled enough and plentiful enough to process all of the security output and take action on them, let it be so.
My experience is that in practice, there are not the resources to process all of the output that the tools generate. Do you have experience to the contrary where this has been done at a company scale or is your argument a theoretical one that you believe stands to reason?
ask yourself what is cheaper: hire and retain Cloud Operation admins in SRE org, hire and retain Cloud security experts in cybersecurity org -- vs hiring a cloud security guru and task him overseeing with maintaining and security $platform_name ?
very few companies are able to hire and retain SRE-Kubernetes operators and Kubernetes security architects, so it kinda makes sense to merge and hire one good expert
It just depends on the size of the respective orgs. If engineering is 5 people, a dedicated security person doesn't make sense. At 500 you might be able to get away with one. At 5,000 engineers though, you real do need more than one good security expert.
Most of our blue is threat and anomaly detection. They spend a lot of time in tools, sure but its usually a SIEM like Splunk or similar tools for event correlation and behavior detection like Crowdstrike. Qualys isn't going to find custom written malware using common utility as a C2 channel.
yeah, but crowdstike or whatever antivirus you use can. Again, no need for red team, just an operator for Security Tool X/Y/Z
...No, it really can't. Not all of the time. It doesn't understand your business logic behind your architecture, it doesn't know how to separate normal from anomalous behaviour (yes, even if you have XtremeAI or whatever the vendor peddled to you), it cannot correlate alerts and activites in patterns not explicitly programmed and it can't generate alerts with 100% precision. In short, it's not intelligent.
again, no need for a fulltime red teamer
A part time red teamer is just a pen tester. That's not what a red team does, and adequate threat emulation isn't someone who flies in for two weeks does somethings that you can only accomplish in two weeks, then fly out to the next company to do the same things. You actually need to dig deep and research a companies individual weaknesses and that takes time. Threat actors have that kind of time to sit and observe and probe, a part time noisy pen test isn't going to get you more than the basics.
Also a blue team has to be right all the time, the attacker only has to get things right once. That's why you constantly test your assumptions with a red team and give blue someone to "train" with and improve with that isn't an actual threat actor. You don't want the first time you fight to be in the ring, you need to spar.
A lot of pen testing consultancies brand themselves as red teams, but a lot of them are just rebranding pen testing services to the ignorant.
Threat Intel informs the red team, and we'll model our ops or threat emulation exercises off of that information, we also have a few members that came over from threat intel. Last I checked, Threat Intel engineers don't write malware to simulate malware real threat actors that target a companies industry use.
I think you're speaking from your limited experience at your own company. Again, you have to have a mature blue team before you're ready for a red team. Most orgs aren't ready for that, especially if you think Qualys covers your threat models. That's just vulnerability management and might catch some apps/hosts that missed the patch cycle.
If it's word salad, maybe its because you're out of your depth when and ignorant of how these programs are supposed to work.
That was also my short experience in the area, and it was boring AF, almost soulcrushing. Maybe I just got unlucky, but it feels that a lot of security engineering positions are like that.
Would anyone happen to know of what search terms should one use to find open positions that are focused on security tool building rather than deploying a purchased solution?
Tier0: Pure security research shops, like Google Project Zero and alike. YOu can only get this job after doing PhD in Computer Security at top-tier university lab, and have publications on stuff like linux kernel fuzzing, have 10+ linux CVEs under your belt, and listed as contributor to a well-known open-source security tool.
Tier1: top security vendors: crowdstrike zs qualys pan cisco and etc. R&D (or Business Unit) departments responsible for developing security product features and maintaining security content.
Tier2: FAANG and adjacent cloud-native companies, job title "security software engineer" or "security engineer" or "$platform_name engineer" or "cloudsec/appsec engineer"
Tier3: pre-IPO unicorns which are hiring for Product Security and AppSecurity positions
... Tier10+: all other public companies, regular S&P500 infosec department.
the general rule is the higher the salary the more demanding the job will be and the tougher is competition to get that job
My recent interview as a security engineer had distinct sections for software review and for architecture analysis. My last job had me running Burp Suite once a month (because it was fun and I didn't mind doing it), but that was roughly 30 minutes every 4 weeks.
Sounds like you've worked with some quite junior security teams.
as for Burp Suite - why would a company keep engineer full-time only for him to run burp suite for 30 minutes once a month? Sounds like poor investment on a company's side.
...or, and hear me out, maybe some companies are more proactive about their security. Craziness, I know, but maybe!
> why would a company keep engineer full-time only for him to run burp suite for 30 minutes once a month?
Uh, I was doing other things the other 159.5 hours a month.
Well, no. I work in "cybersecurity" and my day job is literally finding vulnerabilities.
Not all cybersecurity problems are platform problems, or even technical problems. Even if you decide to gather all AppSec/InfraSec engineers inside of SRE/SysAdmin, this creates two problems:
1 - They are now separate from the main cybersecurity team, which is solving other problems and that in turn creates all sorts of issues ranging from lack of coordination to managerial ones.
2 - This creates a HUGE lack of separation of duties (or a conflict of interest). The executor of a task should never be the auditor of a task. You cannot expect the SRE team to go out of their way to keep finding flaws in their own work and exposing them to the board so they can be forced to reprioritize and fix these flaws. The SRE can and should have security built-in as much as possible, but an external team is still required.
if you switch back to sysadmin model - having a single sysadmin do all the SRE and security work, divided per platform, then separate security is indeed feel like extra.
You can obviously hire independent third party security auditors and do pentest, but that thing could be done once a year. Often times your financial auditors (Big 4 accounting firms) will offer IT audit and pentest in a bundle and you can get a good deal.
Cybersecurity is special in a real sense: it's one of the few groups typically charged with enforcing required compliance controls. I was a senior engineer for many years before moving full-time into security, and it's amazing how many of those annoying inconveniences are actually mandated from external customers or frameworks[0]. As an engineer, I don't really care if every PR has an associated Linear/Jira/whatever ticket; sometimes it's enough to say "I'm fixing this thing because it needs to be fixed". As a compliance person, SOC2 says that all change requests must have an associated ticket, and my personal feelings about that aren't part of the equation.[1] Think Security is a pain in the neck? Work in a regulated industry like healthcare or finance where you have lawyers instead of colleagues giving you a list of commandments.
Also, I've never slept so well as I did the month after I was allowed to hire my CISO replacement. It's like looking up and seeing the Milky Way instead of the Sword of Damocles.
[0] Having worked at shops where Security didn't take time to explain those things, I've vowed never to be that person. If I ask you to do a thing, I'll tell you why it's something we have to do. We should all collectively be better about that.
[1] Although there's an analogy between "compliance" and "unit testing". A side benefit of thorough test coverage is that you tend to write better structured, more loosely coupled, easily testable code. That's all good to have even if you never ran a single test. Same with compliance. The ability to audit all the things means you're more likely to design your systems so that they can be inspected, instrumented, and monitored. In the "where's the ticket?" audit case, know what's more annoying than having to match every PR to a specific ticket? Having to individually assess a couple hundred exceptions to very that each of them reflects a reasonable judgment to violate the guidelines.
Does soc2 actually say this? IIRC their spec docs are proprietary so chumps like myself can’t actually read them. But a (imo great) SOC2 auditor I worked with previously allowed us to treat GitHub PRs as their own sufficient justification. We still used Jira for most, but for minor fixes, quick upgrades, no Jira necessary. A former coworker said they did the same at a previous job. I think, in this particular case, it isn’t SOC2’s fault, it is the fault of auditors who are afraid to not just do what everyone else does.
> it isn’t SOC2’s fault, it is the fault of auditors who are afraid to not just do what everyone else does.
There's some liability for them if they're too lax. Suppose a breach happens and a given control, as written in the spec, would have prevented it. It could come back on the auditor who signed off on a clever interpretation of that control.
A lot of SOC2 is hand waived as paperwork, but the point is that an organization (and their independent auditor) can show evidence that they have established certain processes and controls, and that they can demonstrate compliance with their word. (This belies the tension that some use to discredit independent auditors: that the client pays the auditor for their report. What’s less well understood is that the auditor has a much greater vested interest in being honest, and that they’re going to get questioned if a matter ever goes to court. No one client is worth losing the entire firm. There are, of course, bad auditors just as there are bad developers, admins, accountants, etc.)
I spent time working in compliance and led the inaugural SOC2 Type 2 audit at a household name. These days I spend my time primarily on infrastructure.
"change request" is often a specific term in things Project Mgmt, SDLC, etc.
In general, I believe it does - cybersecurity is unique in the sense that you're dealing with intelligent adversaries that are actively trying to break your code. It's not like SRE work or normal software engineering. But there's also a lot of snake oil in the industry and most "cybersecurity" professionals don't actually know how to find vulnerabilities.
"Attackers accessing our systems without our consent is one type of failure, but not the only kind. Reliability failures are arguably both more frequent and more damaging when they occur".
This will completely depend on your product, but I've seen this sentiment across industries. "Risk = probability * impact" doesn't work when you throw security into the mix. The author sounds like a very common type of software developer who takes security issues in their code like a personal attack.
One of the best vulns I've ever remediated was one discovered by an application developer in their own authentication that was prescribed by our platform. He knew it was a problem and knew it needed to be fixed, but knew he needed us to report it before a fire got lit under the appropriate people's butts to make it get fixed.
She herself is a cyber security expert, so congratulations on attacking the messenger I guess?
https://www.fastly.com/blog/cve-2023-30534-insecure-deserial...
An unexploitable PHP bug that five people decided to share the credit on.
In my experience, that's primarily because cybersecurity people know neither the probability nor the impact of the things which they are calling risks, which moreover means they don't know how to order them from a prioritization perspective.
Business people typically don't wanna hear that and then act all shocked pikachu face when it happens.
Plenty of risks arent vulns in public software. Some are vulns in your own software. Some are just deviations from best practises or lack of system isolation.
Even among public vulns with known PoC, the impact is going to depend on context. Plenty of times the impact is zero based on how software is used.
I’m actually a fan of embedding security resources into the eng teams and having those resources fix issues rather than toss shit over the wall. Knowing how software works is really hard without living inside the code base.
"if attackers can run arbitrary code on our systems then they can exploit vulnerabilities!" or "if attackers can send arbitrary payloads to services which are turned off in the actual image and in any case not accessible without breaching a perimeter firewall and passing through a DMZ then they can exploit vulnerabilities!"
OK, sure, what about "attackers kidnap the families of our AD admin team and force them to do whatever they want"?
Also, "high" meaning "sometimes we just don't care".
No, sorry, I do not care that (e.g.) CVE-2023-4911 has been categorized as "high", I am not in a rush to replace every single Fargate task that contains a vulnerable glibc because it's just not an interesting vulnerability in that context: let's assume you (a) bypass the payload validation on the (private!) API Gateway, (b) construct a payload that allows some kind of RCE in my JVM, (c) somehow escape from the JVM sandbox to run arbitrary native code, (d) successfully implement the attack, (e) get root in the underlying VM ... it's a Firecracker VM that doesn't have ANYTHING ELSE in it.
But, no, it's a "high", I better start cutting change tickets for literally everything in the universe to conduct an incredible load of busy work in order to do a no-op.
Really smart cybersecurity would be (I'm just making this up, I haven't really thought about it) "you know, if we're looking at services contained in a VM security boundary like Fargate, perhaps we don't care about Attack Vector = Local or Physical, or User Interaction = Required?" but what we actually get is "raw CVSS score tells you all you need to know and absolutely dictates urgency and triage priority".
I am actually fairly data driven, and it's why things like SBOMs are exciting. Based on my experience, if something is exposed to the world and that thing has a vulnerability... within a period of reasonable time (say within 12 months), it will be exploited.
> No, sorry, I do not care that (e.g.) CVE-2023-4911 has been categorized as "high"
I wasn't actually referring to my "high" being the cvss scores. It was likelihood of exploitation(in my org), not the score. In your example, that would be a local exploit, so wouldn't be a high in my book unless you are shoveling user input to a CLI, which I would hope isn't happening.
Personally unless you know there to be higher risks for certain things, if it ain't 9+ on cvss, it gets a ticket cut to deal with it like any other but at some convenient time in the future.
> but what we actually get is "raw CVSS score tells you all you need to know and absolutely dictates urgency and triage priority"
Your security org sucks and are doing it wrong. I'm genuinely sorry, and I'll say that not all of us are like that :(
I disagree with this on the grounds that normal software engineering is dealing with intelligent adversaries as well. They may not be "adversarial" in the sense they are maliciously trying to _break_ the code, but if it is successful in performing _some_ work they will attempt to use it to perform other work as well, whether it was intended to or not.
But, what the author seems to be saying here is that a "vulnerability" is just a fancy word for a bug that is not inherently more significant than a crash or outage (where the vulnerability may not be available to exploit anyways). The suggest that cybersecurity teams may be more successful by integrating with the pipelines developers and SREs already have for dealing with issues (including critical ones) than interrupting their work and trying to assert how much more important cybersecurity is.
The first side is that ultimately the argument that the author debunks is a strawman argument, and the counter-arguments are equally sophomoric.
The second side is that I agree with the author on (some) of the recommendations and I think its great that the author made recommendations, instead of just pointing out perceived problems.
Cybersecurity isn't special. At least not any more special than any other subfield is in its own way. But the rest of the article doesn't follow from this axiomatic position. In fact, it's irrelevant to the meat of the article (the recommendations), and comes across to me as griping.
I think the article misses some of the realities of the business environment.
Why do (some) security teams (sometimes) try to put in roadblocks to software releases? Usually it is because the the release is going to have a business impact for which: (a) the business has not acknowledged (b) no person is accountable for (c) there are no plans for.
Why do security teams feel they must dictate outcomes rather than recommend? Usually because their leaders (running up the the Board or Executives) are holding the security team accountable for outcomes they have only indirect control over. Another common reason is that the incentive structures of various organizations push for development groups to release as quickly as possible, feature release estimates rarely include security work estimates, and teams/managers feel pressure to release anyway to avoid delaying due to missed work estimation at the project start.
Usually, these kinds of issues represent the failings of other structural aspects of the business.
Ideally security teams wouldn't be needed at all, the same as SREs and many other kinds of roles. Ostensibly developers would be so capable, so knowledgeable, so well rounded, so careful, that they would plan for, mitigate and manage all operational risks on their own.
There's depths to explore on this topic. I'm excited for the article author that they are just starting to get introduced to it. I hope they continue to explore this as they develop more seniority and experience, and share in more detailed public posts as they go along.
I've never seen a SRE lead go to prison for an outage, but as of late a few CISOs have been charged, convicted, and sentenced to prison for failings on their watch.
Cybersecurity may not be special but the results of a failing there are unique compared to SRE work.
Sorry, but it actually does make sense in multiple ways.
Firstly, the core purpose of cybersecurity in the C-suite is to ensure the bottom line. System resilience is not the priority, the priority is the stock price. That means that it's way more important to triage an issue, collect evidence, determine impact, and see how many ways you can legally hide evidence or manipulate the situation to deny you ever even had a security incident. The quieter it stays, the smaller the impact to the bottom line. Keeping compromises to a minimum, improving UX, meeting modern best practices, etc are like the farthest things from their minds. And the people who run the company dictate what these teams prioritize. So to "the company", there's a good reason why they focus on their own thing.
Secondly, security is really frickin' hard. Humans are not great at being masters of many things. You need a specialist who understands security well, and has the tools, and training, to do security-specific things. Why do you think "platform teams" (aka the new-new-sysadmins) exist and we don't just have developers build their own platform? Because there's ops crap in there that devs will never know, and shouldn't need to. Humans work more efficiently when they can focus on specific things, keep context switching to a minimum, and maximize their specific knowledge to a small domain. That includes security tools, practices, policies, etc.
Thirdly, because of the competing priorities of security, and the specialization needed to truly do it well, you have to have a "security process" that people who aren't security experts can follow. You also want a "security process" because processes in general are useful tools to keep things working correctly. Even "security people" should be following a "security process". And there needs to be "security people" to come up with the process (ideally in collaboration with the other teams).
What you don't want is bad process, or bad teams, or bad special snowflake solutions, etc. You want it to be supportive of other parts of the business, rather than a hindrance. That's what DevOps is all about - getting people to collaborate together, enabling self-service, increasing efficiency, improving quality, reducing waste, etc, etc, etc. You can do that with security too, and it doesn't mean you have to get rid of your security teams/solutions.
its not. its a process.
It never ends, and you need to work at it constantly. or to put it in common terms :
Its about the journey not the destination.
Cybersecurity is full of things that are both very unlikely and very bad if they happen.
What is the risk of some vuln burried deep in the stack that likely nobody will ever notice and even if they did even more unlikely they can further use it as a pivot point? Hard to say, but maybe close to zero unless you are a company that people specificly target. At the same time it can be a business ending risk. With SRE , even in the absolute worst case you can spin up new servers. You can't unleak your customers sensitive data.
I think cybersecurity is hard because we are uniquely bad at evaluating cybersecurity risk which means we are bad at figuring out what appropriate mitgating action is
It's not that hard to evaluate. The risk is high! Pretty much anything can be hacked if you try hard enough... What's hard is everything else. It's hard to accept. It's impossible to quantify. It's hard to justify caring about it. It's hard to care. It's hard.
> It's hard to accept.
From context you probably mean a different meaning of "accept", but accepting cybersecurity risk is actually super easy ;)
Not every org is FAANG. Unfortunately in many (most) organizations, in house developers get caught quite frequently eating the metaphorical security crayons.
The stuff I have to do to keep people from eating infrastructure crayons...
For one, they can quantify downtime easier. Also, in most orgs, cybersecurity is responsible/accountable for the problems, but only can indirectly influence things.
I don't understand this. The metric for downtime doesn't matter who is measuring it.
> Also, in most orgs, cybersecurity is responsible/accountable for the problems, but only can indirectly influence things.
Right, so very much like platform/infra/devops/SRE. ;-)
If you think cybersec engineers think of themselves as special snowflakes, you should work with robotics engineers :)
This is unfortunately the reality in many companies because cybersecurity is implemented as a subset of a high level information security framework without qualified people to connect the high and low level requirements.
In the past, security departments were the "firewall gatekepeers", choosing who to allow or deny access. They started to change over time from gatekeepers to support the business but this transition is not complete as we know from experience.
The CISO has also has the dillema of having to support the business but will also be held accountable for any hack, which increases the tension on reducing friction vs increasing friction (and increasing control).
This is not an easy problem to fix, but I particularly think it's very productive to see posts like this so we can bring this topic to light and find the right balance.
And for those interested in learning how to add security to their organizations with minimal friction (security by design, especifically), I'm creating some webinars (free on youtube) here: https://devops.security/webinars.html