This content marketing is so good I don’t even know if I’d call it marketing: it is legit educational with a weak (in a good way) “throw some money at us” CTA. Bravo.
I've met a few where it's not explicitly required on download (e.g. GitHub has their SOC2 available for download on the enterprise admin page, but by the time you're using GHEC you've signed a few pieces of paper), but agreed that most companies aren't giving them away for free.
This is a restriction of SOC2 itself. SOC2 produces a "restricted use" report, meant for the client company who purchased it, and for limited access by third parties that do business with the client company.
AIUI, the intent is to ensure that the people reading the SOC2 report don't read into it more than what the auditor intended. This, according to the auditors, is fraught with difficulty, because you need a degree of familiarity with COSO principles and general SOC-ese to understand precisely what the auditor is making strong claims about, and what is _not_ being claimed.
In practice, the way this is accomplished is that the SOC2 report includes standard verbiage saying that the report is only for the client company, and for specific third-parties who must have "sufficient understanding" to parse the report correctly.
The client company then implements some process, with the guideline of "do something that won't piss off your auditor". For small companies, by and large the implementation is "you have to be an existing or prospective customer, email to ask for the report, and sign an NDA." The very act of requesting a SOC2 report implies that you know how to read a SOC2 correctly.
Some larger companies set up portals where you can help yourself to their various reports, sometimes without NDA - but in that case you have to click through a pinky-promise to not redistribute, and the report is watermarked with your identity to deter distribution. A very few publish the report at a hidden URL and hand out the URL to anyone who emails in to ask for it, though I personally think that's walking a bit too close to the edge of what they agreed to with auditors.
Companies with deeper pockets end up paying an auditor extra for a SOC3 report, which is "SOC2, but abridged and with unrestricted distribution rights". I believe the theory (besides giving more money to the auditor) is that the SOC3 removes all the information that might be misinterpreted, boiling it down to barely more than "I, a trustworthy auditor, confirm that the company is doing all the right things." You don't get much detail, but as long as you're willing to transitively trust the auditor (who are themselves scrutinized by their regulatory bodies), that gives you a "compliance is yes" document that you can publish far and wide.
Trust Report seems to be irrelevant to this (from what I can tell from the brochure without being a vanta customer), because it's a way for a company to publish claims about itself. Crucially, nowhere does it say that an independent auditor verified those claims.
SOC2 broadly contains:
- A description of what the company claims to do
- A statement that the description is complete and accurate
- Auditor's testing procedure for verifying the company's claims
- The results of the testing
- The auditor's overall conclusion as to whether the company meets the bar for SOC2.
The Trust Reports contain programmatically-validated information (basically: Vanta's code says the control was in place continuously.)
There's (obviously) pros and cons of trusting a software provider (like Vanta) to validate technical configuration compared to trusting a human auditor to do the same.
Our bet with Trust Reports is that for some cases, having software do the checking and validation continuously is better than having a human auditor do it once a year.
> Everybody would be better off if they stopped believing what they believe about SOC2, and started believing what I believe about SOC2.
Since the author is a member of the set "everybody", we have a paradox. :)
More seriously, it would not be hard to adjust the language just a little bit by saying, e.g. "Most people would be better off...". Alternatively, the author could adopt a common style used in business communication where the author creates a label for the group that would benefit from the SOC2 knowledge. Perhaps call the combination of "cynics", "customers", and "true believers" the "unwise trifecta" or something. (I admit don't have a catchy term in mind yet.)
In all seriousness, I laugh at absurd absolutes. If you show me a sentence with "Most people" and the same sentence with "Everybody", I'm going to smile at the second.
But I don't think that means it's ambiguous. "I literally died" makes me laugh but there's no deception.
If we're being sufficiently pedantic about this, for any person A and belief B, it is perfectly non-paradoxical for A to stop believing B and then start believing B.
If we're being sufficiently pedantic, then after this process person A would be right where they started and would not be "better off", making the original statement false.
Great writeup. Dude is totally right about the importance of compliance and its implications.
There are other ones as well.
* ISO 13485 for medical software
* ISO 2700x for process, security, and change management
* ISO 55000 for asset management
Your eyes might glaze over, but they can really help a startup graduate from "move fast and break things, cowboy shit" to a mature and respectable engineering organization.
Congrats on the SOC2 and finding a good spin on the story. It is also an important number of checkboxes you might need to get those juicy Enterprise and Government deals.
Pedant mode on (this is standards after all) - ISO 13485 is medical devices in general. You'll end up tacking on something like IEC 62304 for medical software and software in medical devices.
I’m underselling SOC2. It assures some things pentests don't:
* consistent policies for who in the company gets access to what
...
That's a start. Of course, having policies raises a new question -- how well are they practiced? To what degree are they:
a. discoverable via internal tools?
b. descriptive and understandable (the policies use terminology that maps to the business processes clearly)?
c. measured?
d. internalized by employees as part of the culture?
e. enforced (as desired)?
f. externally reported (e.g. to downstream customers)?
g. reviewed when appropriate?
h. adjusted or removed as needed?
There are some spectacular failure modes of policies in the real word. Here are five attributes that fit together nicely into a mosaic of dysfunction:
* Everyone has to take a sleep-inducing thirty minute training.
* Policy compliance is "tracked" with a spreadsheet on an ad-hoc basis.
* Everyone in the organization has to "check off" that they comply, starting at the bottom and working up the chain to El Jefe.
* If something hits the fan and you need someone to blame, follow the paper trail and blame the people who incorrectly attested to policy compliance.
* Virtually anyone could fail compliance if you haul out the microscope. This is a feature, not a bug. Now you have a convenient and official way to get rid of people you don't like for arbitrary reasons.
This is stylized, yes, but not too far off the mark at some places.
One of the main points of real audits like SOC2 type II are about validating enforcement of the controls outlined as policies. So the policy that only employees with job role Z can sign onto System Y would be checked by summarizing the login audit logs for Y, and verifying that all the names on the list. Other policies might require verification by reviewing all or some auditor selected subset of occurrences to verify. Some details of some policies cannot be fully verified. As long as that risk is known and documented, it is not necessarily a problem.
SOC2 audit reports provide a high level version of reporting to downstream customers, without necessarily revealing the full details of the policy. (Sufficiently important customers could always insist on seeing the actual policy documents, if the reports don't satisfy them).
But some of your remaining considerations are somewhat outside the scope of SOC2. And they can be tricky problems.
This matches our experience well. Surprisingly intensive in some unexpected ways, and surprisingly silly and incomplete in others. I get that the Fortune 500s want it, but having now done one I don’t quite know why they value it beyond a checklist item in their own compliance.
This is a great way to put it. People here are bashing SOC2 because it doesn't go in-depth enough, it's just checking for the basics, it doesn't actually stop hackers from accessing insecure AWS buckets or ransomware attacks, etc etc, and they're absolutely right.
But it's meant to be a minimum. It verifies that there isn't one copy of the source code on a dev's laptop. It verifies that a dev who gets fired won't be able to log into the production server and delete all data in retaliation. It verifies that an intern isn't able to completely destroy the business by accidentally deleting the production database (because you have routinely tested backups and a documented RTO/RPO, of course). Being able to demonstrate this level of minimum competency is extremely valuable when you're in the B2B world and trying to sell your product to a larger client.
The paperwork is a hassle, but if your company is following best practices for development and operations, there shouldn't be much of a step change in what you're actually doing on a day-to-day basis.
> This is the only issue we ended up having to seriously back-and-forth with our auditors about. We held the line on refusing to do background checks, and ultimately got out of it after tracking down another company our auditors had worked with, finding out what they did instead of background checks, stealing their process, and telling our auditors “do what you did for them”. This worked fine.
What process did that company use instead of background checks, that you ended up doing as well?
Another issue I have here is that I want to be a little bit cagey about what our specific controls are, not because they're sensitive to us, but because there's a limit to how much we're supposed to talk publicly about the specific results of the audit (it's a Type I, people who know SOC2 know that means there are no unhappy surprises in it) --- the audit results are confidential, as a term of our engagement with the auditor.
(This is why there's a SOC3.)
Long story short: it's not complicated, and if you're currently doing a SOC2, like right now (or in the future) and you have reached the point where you're trying to get out of background checking everyone, shoot me a line and I'll tell you what we did and what we said (I may performatively NDA you in the process, because I like our auditors and don't want to irritate them).
Ironically work I dabble in the security automation space and I'd say this is the real "social ill" of all regulatory cultures. It is not the automation that's important is sharing and reusing agreed upon understanding of requirements and best practices (what and why we automate and the real goals not just cargo culting or copying). Most unintentionally hoard and others (auditors, special consultants) intentionally do with the belief this is their market differentiator. This is good but still falls short by not sharing and dropping hints. This is the default I see most of the time.
Most higher level attempts to meaningfully share and reduce toil and wasted effort are not incentivized in risk/governance/oversight culture, so we all get to lose.
In Ireland there is no legal mechanism to do a background check unless you work with children or are in law enforcement. Even collecting and recording public information on individuals can be problematic with the data commission. Employee reference checks are acceptable for auditors in that case.
My company did adopt background checks, as part of our SOC-2 requirements and because my company works with health insurers (which generally impose this requirement via contract, regardless of SOC-2).
Like many people here, I didn't like the requirement. That being said
1) It's possible to configure background checks so you don't receive irrelevant information (e.g., if DUIs aren't relevant, then configure the check so you don't receive information about DUIs). In most cases, you'll just want to receive information about financial and privacy related offenses.
2) What you do with the information is up to you (unless your customers enforce certain actions). In general, the SOC-2 auditors will want to see a plan by which you acknowledge and manage the risk, which doesn't necessarily mean you can't hire the person.
IMHO _recent_ DUIs are more relevant then a lot of "not at all" recent much more serve things.
DUI is a sign of gross recklessness and apathy for the well being of others. Sure I won't blame a young adult for doing this mistake and there are situations where it's understandable (i.e. some kind of emergency making you DUI even through you generally are against it).
But still I would prefer to work with someone who in the youth due to poverty has committed robbery (but not anymore since 20 years), then someone who in their 40th who is frequently driving under influence of alcohol.
Anyway even if I had a company and it for whatever reason would do background checks I wouldn't want to know the outcome as long as whoever is responsible for it following some strict guidelines didn't judge it to be a problem (and no if it's not a car company it wouldn't contain DUI, and generally I don't like background checks).
I just realized that I had forgotten that in the US you often do not have the freedom of not taking the car but e.g. the public transportation. This makes things more complicated. But then doesn't really change how I feel about it.
If it helps, in the US you can get a taxi/uber/lyft to take you home from the bar. Some bars even offer free rides or can help arrange one if you need it. Calling a friend or relative to pick you up is also an option.
It's true that having shit public transportation and everything so far away that you need to drive complicates things, but there are always options. In Japan the public transportation is great, but the trains stop running long before the alcohol stops being served and it's not uncommon for drunk people to wait until morning even if it means waiting/sleeping outside all night. No reason folks here can't do the same.
> If it helps, in the US you can get a taxi/uber/lyft to take you home from the bar. Some bars even offer free rides or can help arrange one if you need it.
I was more thinking about people with an alcohol addiction still getting to/from their job on a daily basis then people going home from partying.
I certainly don't mean to endorse DUIs! And if a company has the viewpoint that a DUI indicates that a person shouldn't be employed in a specific role, then background checks are a good way to achieve that.
My perception is that some people who don't want to do background checks feel that way because they don't want to know embarrassing details about their employees and colleagues that aren't relevant to work. And the good news is that employers can generally set up background check reporting to simply not report issues that employers don't think are relevant. And that makes it easier to offer background checks, and easier to meet SOC-2 audit requirements.
In fact, what I think you'll find in a lot of SOC2 background check regimes is that they're pretty much just automatically filed away without any careful review. As long as you did the check, you'll be fine with the auditors. We could have just did that with our US employees; we were fine, in the audit, with not doing them for people in Europe. But that's stupid, and we're not going to do stupid stuff for SOC2.
Part of taking on SOC 2 is also choosing whether you want your attitude to be "Let's do the minimum to get past the audit" and "Let's take this framework, and figure out where we can learn from it."
The post mentions background checks. On the one hand, I understand there's a real issue with these. On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. Hopefully the things they took from the other mentioned company do increase due diligence in vetting employees who have access to sensitive/regulated information.
Use it as a framework to actually think about BCP, DRP, etc, etc, and it won't be a total waste of time.
Edit:
Also adding I bring up background checks as an example of learning from vetted practices, rather than trying to attack the decisions of fly. I respect this article, especially where it's easy for people on the internet to criticize decisions, when the reality is security is a series of tradeoffs, and to function as a business means having imperfect processes.
Slightly different, but in a past life working in a field the company had extreme access to people's homes, we definitely weeded out some people who should not be given access to a strangers home using background checks.
Again, they're not ideal, and there's a large social concern of a permanent record like that, but you have a duty of care if your customers are trusting you.
I think they're intrusive and mostly unuseful, but I can at least see the rationale behind it at the kinds of companies that hire people en masse, or the kinds of jobs that people get bonded to do.
You are arguing that background checks should be a requirement in order to have access to certain protected resources. While this is a fair argument, the counterargument is that most people in this particular software business shouldn't have access to user data/protected branches anyway. If someone needs elevated access, they would likely already have a significant pedigree at the company, and a background check may not add much value. In reality, most companies don't do background checks for security purposes; they do it to screen out candidates who aren't agreeable people, which raises ethical questions. I don't have an opinion on whether this is fair or unethical, but if security was the sole purpose, it would make more sense to background-check employees as a precondition to privileged access, not candidates as a precondition to employment.
I would think it would be fairly obvious that a candidate could be “bought or bribed,” simply from the fact they’re asking for a job in the first place. They’re willing to exchange their time for money, i.e. “be bought.”
So why do people not commit corporate espionage? Well, it might have more to do with character traits than financial stability. In fact, most spies probably have their life fairly well together, and will have perfect credit. As for any asset they might compromise, what’s the difference between someone with poor credit applying to a job because they need the money, vs. applying to a job because they need more money from your enemy bribing them? I’d argue the difference comes down to character.
So for that reason, I’m skeptical of the effectiveness of a credit report as a proxy for likelihood to commit corporate espionage. A good credit report doesn’t seem to offer meaningful signal in either case of a malicious attacker or a desperate contributor. A bad credit report produces as many false positives as a good one.
This is what Schneier calls a "movie-plot threat". Instead of imagining a complicated narrative that connects a poor credit score with episodes of control fraud, improve internal controls so that individual contributors don't have the ability to steal from customers. This would be safer, and more considerate of your colleagues.
I mean, they're pretty effective for determining whether people have convictions (caveats apply, depending on your jurisdiction) for that kind of stuff, right?
One of the problems with not doing background checks is ex post facto if you do have problems with an employee, and it turns out that you could have discovered them with a background check, then that can figure into your liability.
In some countries (e.g. US) people are convicted all the times for things they haven't done. Or have done but which is in the past and really irrelevant for the job.
On the other time people are not convicted for a lot of kinds of white collar fraud all the time, too.
The main thing that makes me uncomfortable is that they will raise all sorts of unrelated things that can bias you or others in your company against a candidate, or even if you choose to ignore them, be brought up against you if it turns out you hired someone with a drug conviction or something unrelated to their work at your company.
I don't really give a shit if someone was arrested for a drug offense (or many other offenses), and knowing that information brings up all sorts of complications, to the degree that it outweighs the value of knowing relevant stuff (largely because genuinely relevant things from a background check are rare).
What you do is decide what you care about from a conviction perspective (say, crimes of dishonesty, serious violence etc). Then you write up a policy that says these are exclusionary. Then you only let HR run background checks immediately pre-contract, and you don't let anyone outside of the immediate background check team see any of that stuff. They're empowered to give you a yes/no to hire, but that is all.
Hiring managers should not be looking at BGC material.
Are they? How much will that differ between an employee from Peru, one from Canada, and one from Romania? How comparable will the data be, how much data will you even get in the first place?
And what kind of employee will you lose if you set such restrictions?
We definitely don't do the minimum! Our goal was to keep the scope manageable so the Type 2 certification goes smoothly. The side effect of this is that the things we _did_ put into scope are things we expect to do really well.
Preventing access to sensitive data is important. None of the top ten ways we try to solve that include "background checks", though.
I will have more to yell about this in 3 hours because I have a thing I want to yell about this. Free stickers if someone yells it for me before I’m done with this appointment.
You can run a SOC2 compliance program earnestly or as a check-the-box exercise.
If you're running earnestly, I would argue that the hardest thing about a SOC2 is ensuring that you stick to your guns on approaches that work for you and not adding cruft that you don't care about. If you let the latter happen, you will invariably end up a box-checker, and being a box-checker eventually contaminates a robust engineering / security culture.
And it's hard to walk back more restrictive / cumbersome policies; if you delegate your SOC2 to a person who doesn't deeply care, they'll eventually agree to put ClamAV on all the hosts or something (to make the auditors go away) and then you're going to be stuck with that for a while.
(So you need to find someone who has enough business context and good judgement to run the process, which is super painful from an opportunity cost perspective at a startup, and hard to locate at all at a larger company.)
> If you're running earnestly, I would argue that the hardest thing about a SOC2 is ensuring that you stick to your guns on approaches that work for you and not adding cruft that you don't care about. If you let the latter happen, you will invariably end up a box-checker, and being a box-checker eventually contaminates a robust engineering / security culture.
That's spot on, not only for SOC2 but for many, if not most, relevant certifications. The most important part is "not adding cruft". Nothing sucks like being stuck in a ISO9xxx certified process because you over-specified your processes even though you'd get the "ISO9xxx-certified" label for 10% of what you did. Suddenly you cannot react with common sense anymore because doing so would violate contracts you made with exactly those big customers you got the certification for in first place.
> if you delegate your SOC2 to a person who doesn't deeply care, they'll eventually agree to put ClamAV on all the hosts
Bingo. Just spell out the consequence: you no longer can optimize your compute costs by switching to a managed Kubernetes, because there is no ClamAV there.
Glad to see this at the top. While a lot of folks will groan when it comes to the minutia that are necessary to do a compliance audit, there can be real value in them if you take them seriously. I think of it like this:
1. There are a set of things you need to do for "real" security
2. There are a set of boxes you need to check to pass a compliance audit
I think SOC 2 is pretty reasonable in that, if you're taking it with the right mindset, there is a large, large amount of overlap between #1 and #2.
I personally dont consider soc2 or similar certifications a good framework bc of its checklist nature. A lot of those items end up being orthogonal or sometimes even detrimental to actual security.
Using your example of background checks it’s probably more valuable to have proper acls and audit trail internally than doing background checks which is a really low signal compared to the level of hassle
> Using your example of background checks it’s probably more valuable to have proper acls and audit trail internally than doing background checks which is a really low signal compared to the level of hassle
I agree with your idea, but background checks are a poor example. They're negligable cost, always outsourced, and trivial to perform. They're worth doing if only to validate that your candidate said the same things as the background check says (if they say they're not a felon and they are, that's a red flag -- if they admit to it and explain why, you're not being lied to). In contrast, you actually need to spend time working on audit trails and stuff. One is hiring a vendor and checking a box, one is probably engineering work.
It's fine for what it claims to be. It's an actual audit, in the accounting sense, not a detailed investigation of your security engineering practice. People are very hung up on this, and I get the urge to jump into SOC2 conversations to point out that SOC2 isn't a passing grade on security engineering. But your SOC2 auditors are up-front about what they're doing. There are management practices that can be verified by retrospective paperwork audits: from a random sample of the people you off-boarded in the last 12 months, did you reliably terminate access within N hours of severing their employment? SOC2 is fine for that. Do you have a security policy that puts employees on notice of their personal obligations with respect to data security, and did a random sample of your employees sign it? SOC2 FTW.
There's real value in being forced through this stuff, because these kinds of management processes are a real weak point at a lot of shops with otherwise strong security engineering. I'm glad that our policies and processes are clarified, and that there's an external process that keeps us honest and forces us to do the routine scheduled meetings, rather than keeping stuff in our heads. We started doing SOC2 prep work a year ago, and even before the audit, we were better than we were before we started.
But it is what it is. The thing that drives me nuts is when people suggest that good teams will maximize SOC2 so their security engineering can be informed by it. Yikes. No.
What's actual security? Looking for zero days? Malware research? Continuous red team?
I think at the end of the day, SOC 2 aims to instill a basic level of organizational security so the company doesn't shoot itself in the foot. If a company can't genuinely follow a basic set of SOC 2 controls, can I trust them to do actual security?
Also, badly written checklists might be bad, but not all checklist are bad. Pilots use them. Doctors use them. Mechanics use them. In fact, most fields that involve critical life or death operations use them. Why? Because humans have a limited memory and tends to miss critical tasks all the time.
> On the one hand, I understand there's a real issue with these.
My personal hate of these is how much information they require you to hand over to some random organisation.
Some make you take photos or videos of yourself holding your ID, and sure they say that they delete the information, but we all know that rarely happens.
I just don't trust their infosec policies, and it's only a matter of time before one of these companies gets reported as having a public S3 bucket, or employee laptop, or USB drive stuffed with video and ID scans and background check data.
Bingo. Most background check companies seem like literally the scummiest things in the world. I have no problem with a background check, but I don't trust these companies at all.
It would be a different story if people at companies of our ballpark size had stories about how background check results were informative, or mitigated some risk. But nobody has those stories.
It's funny how much fraud could have been prevented with simple background checks however. There's an entire podcast about financial fraud called "Oh My Fraud" about many instances where a simple background check might have prevented hundreds of thousands of dollars of fraud at charities. Ah well.
I understand the negative impact upon recidivism that background checks and stigma may have, but also we have to balance the interest of financial and information controls of organizations performing essential functions in society. Ideally speaking.
I'm not sure I follow what you're trying to say here, so don't take any of this personally, because maybe I'm misreading you.
Telling any startup engineering team that they should maximize their SOC2 Type I audit is borderline malpractice. Every engineering decision you make to support a Type I is something you'll need to live with in your Type II. That might just be an irritating own-goal if it's you doing the Type II, and you have to waste a couple hours on the phone explaining to your auditors why you've decided to remove ClamAV from all your servers. But it's could actually be destructive if it's someone other than you running the Type II, and gets cornered by your original Type I claims into supporting that bad decision.
Telling any startup engineering team that they should build a security practice informed by a SOC2 audit --- that they should take the COSO framework and figure out what they can learn from it --- might also approach malpractice. There are probably reasonable corpsec process controls you could build based on the AICPA Security TSCs. But the TSCs are not based on modern software engineering best practices and they aren't informed by modern software security. They are heavily influenced by the security concerns of medium-to-large sized enterprises with sprawling legacy IT footprints, and you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.
SOC2 is not security. Security is its own thing. Good compliance work is a byproduct of sound security engineering. It does not work the other way around.
Naturally, it's easy and gratifying to point out that a SOC2 report doesn't make a company secure. I don't think we could have come up with a clearer way to have said that, or, for that matter, to explain that we did our best to minimize the impact SOC2 had on our engineering practices.
As for background checks: once again, we can't background check a decent-sized fraction of our team, because they're in jurisdictions that (very reasonably) forbid employers from running intrusive background checks. We considered just background checking the unfortunate US team members we have that can be made subject to them. I had a fairly long conversation in a Slack channel full of secops people from about a dozen security companies, and none of them told me a single story about how background checks (which are, as it turns out, performative, superficial, and error-prone) was a win for them. I did get stories about how they were problematic: for instance, I did not make up the thing about high school transcripts.
So, long story short: because of our workforce, our platform needs to be resilient against bad hires. You don't get that from background checks; you get it from security engineering, tight access control, tightly designed roles informed by those access control decisions, regularly reviewed internal audits, detailed audit logs, and sound hiring practices. SOC2 covers most of this stuff only superficially. Security engineering is the real work, at least for technology startups.
Later
This is way angrier than I want this to come across. I promise, it's not personal, and if I'm caricaturing any point you made, I apologize in advance. I had a 6-hour tattoo session that ended with an hour of ditch work and I am in a fucking _mood_.
Also, since I'm predictably being pulled in the direction of repeatedly dunking on SOC2 in this thread, I want to say very clearly that I had a fantastic experience with our auditors, who were more clueful than any other auditor I've worked with. Our auditors are great. Don't flunk us!
> you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.
Can you talk about that more concretely? I'm sure there are many cases where compliance requires you to consider a risk that's not relevant, but it's hard to imagine that thinking about each risk and taking appropriate action (which, sure, in many cases will just be documenting why it doesn't apply) would damage one's security.
At the end of the day any list of potential risks is going to be imperfect, and some will be more imperfect than others, but I'd still think that the vast majority of the time you'll get a better outcome by engaging seriously with a given list of risk factors than by treating it as an exercise in doing the minimum. If you were trying to do security from the ground up without any regard to compliance you'd ultimately end up doing something quite similar - coming up with a big set of risks and then figuring out what you're doing about each of them - and sure, you could probably start with a better-targetted one than what SOC2 has, but that sounds like a matter of degree rather than being so different that you can't get any shared use out of doing those things together.
I'm sure they'll add their own responses, but I've seen some tremendous issues caused by legacies like this. I spent a solid week including a case opened with Microsoft attempting to determine why Outlooks "report as phishing" button didn't work. That's definitely harming security, people stopped reporting phishing. The root cause was a Windows XP era Internet Explorer hardening policy that served no purpose on a modern desktop.
In 2019, Microsoft removed a recommendation for an old font related security setting[0].
From a management level, a wide variety of modern best practices are already the default from Windows 2019 or so. The cognitive effort of looking at 50 security settings and convincing yourself they are all reasonable and won't break things is substantively better than the 400 or so we used to have. It's one thing to inherit this sort of legacy but it's a much worse thing to be implementing all these policies in a greenfield 2022 environment because they are all on some checklist.
So combine that with “post-facto” code reviews on a weekly cadence; there is potentially 7 day window during which a bad faith employee could act unrestrained?
Certainly this is giving me pause on using your platform for anything other hobby projects
I agree. You're a fintech startup deployed on Fly.io right now?
The best way to get detailed information about how our security practice works at Fly.io is to ask us about it directly. We're trying to be up-front about how weak SOC2, for everything else it might be good for, is with respect to security. Unfortunately, in the process of speaking plainly about SOC2, we have apparently sent the message that we think most of security is performative, which is not remotely true; the point is that we don't think SOC2 is an especially meaningful representation of the work.
Thanks for the detailed response. I definitely don't take it angrily. One additional point of context is I managed a Series B SAAS companies first SOC 2 (and its renewal) in the recent past, so I definitely understand what you're saying, and I think it lines up with the point I was trying to make.
My main point is you can either treat the SOC 2 as an adversary to overcome, or actually try and leverage it to be better. No matter what it's going to suck and be annoying, though Vanta/Drata can help. But one can leverage it to be a better company.
A less controversial example than background checks is infra cost monitoring. Where a lot of SOC 2 is focused on business continuity, in addition to security, one of the things required is that you're actually paying attention to your costs. A lot of cash flushed VC backed startups don't. So, once SOC 2 hits, the company that's treating it adversarially will just rubber stamp some quarterly meeting where they "look" at infra costs. Or, you can actually take that moment to level up the company to have macro review of the cost of goods sold, an ensure the business is on a healthy path.
Again, not a comment on your article, one of the big takeaways for me in running a security program for years was a general anxiety around being transparent on the program externally, because there's a certain type of "security" person who gets off on picking apart policies, without understanding tradeoffs that we were careful to make sure kept the company safe, while letting it function smoothly.
Coming out with an article like this is a great thing to do, where a lot of content out there is just "we got our SOC 2, and now we're prefect."
I won't comment on the background check thing again, not the least because I don't want to argue more for something I don't like, just think may be a needed evil.
Genuine question: is consulting for compliance exclusively the possibility for big four type firms,or can't we do this at (traditional not scale up) start-up style? I'd buy your pre money common stock off your comment alone and the similar applies to numerous others I keep reading here.
A little delayed, but there are plenty of companies doing compliance consulting. Eden Data is a small shop I worked with briefly. If you wanted to talk more, my emails in my bio.
If you're in B2B, plenty of larger companies will disqualify for not having SOC2/ISO27001.
Also, it can help get you out of repeat security assessment questionnaires, so it can actually give you time back, depending on how many of those you have to field.
This. You'll lose more money in lost clients than SOC2 will cost you. It is only really expensive the first time you do it - after that if you just follow your own procedures the annual audits are pretty easy. And yes, being able to just reply to those security questionnaires (do you have armed guards in your data center?) with "see SOC2 report" is gold.
Of course if you are in an industry were clients don't ask for soc2, don't do soc2.
> If you're in B2B, plenty of larger companies will disqualify for not having SOC2/ISO27001.
And it's a good question whether you want such larger clients at all. At one of the previous places where I worked, we used to put deliberately bad answers (the worst that our public version of the security policy would allow, not the actual practices) in security forms in order to get rid of too-demanding clients.
That seems like quite a waste of time. Nobody forces you to take on a customer, so if you don't want them just say no and move on, instead of spending a lot of everyone's time to go through the motions hoping for the deal to break.
A lot of enterprise-scale companies won't even consider your SAAS if you don't have SOC/ISO, but many can certainly make it without those companies as customers.
Having policies, records, procedures and documents for everything might also make due diligence easier in case you want to sell the company at some point. Makes it look a bit less like a messy one man show too.
If you're trying to be a vendor for a medium or larger company, SOC2 is usually one of the bright-line requirements.
... Which is not a good thing, because (as noted already in this thread) SOC2 doesn't actually make you secure. Nor does not having certification make you insecure. But, when used as a shorthand, it leads companies to engaging in compliance theater to get certified, spending a bunch of money without actually making their data noticeably more secure.
aka "make the devops team deal with it even though we're paying double their salary to a 'Security & Compliance Director' who hasn't renewed any of the certs they used to get this job since 1997 and hasn't the foggiest clue how the SIEM works when the auditor shows up"
I'm proud to say that I cost our SRE team less than a day's work from the start of the audit engagement to the end of it. If I'd thought to brag about that in the post, I would have. I cost our bizops person a lot of time though, which I feel bad about.
In my experience, massive reams of screenshots for evidence of process / tooling / etc. are the norm with auditors. I have in the past been asked to provide screenshots of source code even.
Did you get any other message from this post? I'd like to think I just wrote the most SOC2-cynical "we just got SOC2" post that has ever been written, but if you've got a better example, I'd dearly love to read it.
I have seen that SSO requirement creep into more and more audits. It reminds me of the requirement to change passwords every month, which was everywhere but now suddenly is nowhere to be found.
It's weird. It's very convenient for the user, but it's on the opposite side of the convenience/security divide. Hijack one session and re-use it for other purposes. Of course it's preferable to not let systems access credentials but one would have thought that we had yubikeys or cards and whatnot now.
SSO is important because it's the best mechanism to show that employee lifecycle events are being appropriately mapped into access controls.
You only activate the SSO account when the employee onboards. You deactivate the account when the employee off-boards. You can suspend the account if the employee is on an extended leave-of-absence.
You can do this in one place, as single source of truth, rather than having to demonstrate that all the different identity systems in which Bob is represented have deactivated him when you have to let Bob go.
Ideally this happens automatically due to an integration between your HR system and your identity platform.
The alternative is that you have some kind of runbook that tells you exactly who to speak to about removing accounts from what systems (etc) which rapidly becomes unmanageable as scope of applications and number of employees increases.
Even for small firms, you can run into problems when the person you need to let go is the person that knows how to run the process.
You seem to be describing centralized authentication, like the family LDAP/NIS/YP.
That's distinct from SSO. SSO is like Kerberos, where you authenticate with a ticket, that for the lifetime of the ticket will not require the user to authenticate again.
In my experience SSO systems are not a security benefit. They risk worsen the impact of attacks. Even with mutual authentication there are still issues with ticket lifetime, extension and playback. Especially when people do whatever they feel necessary to run cron jobs, CI/CD and other systems. They do bring convenience benefits, which are a good thing in themselves, and may have secondary security benefits, but I can't shake the feeling that the arguments are similar to the password change policies of yesterday.
For some time it felt like a huge push towards things like Yubikeys and other modern smart cards that focus on making authentication painless enough to lessen the need for SSO in modern organizations. I would have expected to see more of that, not less.
FYI, the reason the pw change requirement went away is because NIST published an updated set of guidelines that explicitly disrecommend it: https://pages.nist.gov/800-63-FAQ/#q-b05
On the vendor / policy side, many/most of these questions trickle down from NIST or similar institutional guidance. The auditors pick up on that and on practices from comparable companies they've audited, which can be helpful when your industry is moving towards sanity and painful when there's a meme that makes no sense in your context.
(If you spend significant time dealing with customer compliance issues, I would definitely vote that it's worth being familiar with the relevant subset of NIST pubs.)
> Hijack one session and re-use it for other purposes
If the attacker has the ability to hijack one session they could hijack all the other n sessions from the same place anyways. So while SSO does centralize your identity, it also makes it safer because you can uniformly handle provisioning/deprovisioning, password resets, 2FA and all other policies in one place for your entire organization instead of a hundred separate ones.
Going through SOC2 as well, I have a similar experience :)
> I was surprised by how important this was to our auditors. If they had one clearly articulable concern about what might go wrong with our dev process, it was that some developer on our team might “go rogue” and install malware in our hypervisor build.
> ... our auditors cared a lot about unsupervised PRs hitting production.
It's indeed a growing concern, and (shameless plug) the main reason we founded arnica.io (help organizations automate that process without hurting development velocity)
Have a look at https://www.vanta.com/ when you actually get involved in the SOC2 dance. A couple of years ago I took a startup through the SOC2 and PCI L1 compliance process "manually". At the same time Vanta was kind of starting.
I decided not to use them because I (foolishly in retrospective) wanted to "learn" the SOC2 and PCI cert process by walking through it (kind of how you do derivatives, intergrals, numerical methods in school by hand so that you "grok" them).
Since then, I've heard good things about Vanta from a couple of friend CTOs that adopted them. If I had to go through SOC2, PCI or ISO27001 (I did that in yet a previous startup) I would deffinitely go with them.
The blog post in general was good/informative, but I gotta say, this quote does reduce my confidence in Fly quite a lot:
> To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer.
> This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. But our auditors cared a lot about unsupervised PRs hitting production.
> We asked peers who had done their own SOC2 and stole their answer: post-facto reviews. We do regular reviews on large components, like the Rust fly-proxy that powers our Anycast network and the Go flyd that drives Fly machines. But smaller projects like our private DNS server, and out-of-process changes like urgent bug fixes, can get merged unreviewed (by a subset of authorized developers). We run a Github bot that flags these PRs automatically and hold a weekly meeting where we review the PRs.
Letting code go straight to prod without a review is just IMO a really bad practice. It sounds like they've improved significantly here, but still have plenty of gaps where people can ship to prod with no code review until it's been running in prod for up to a week. This isn't just about stopping bad actors, it's 99% about preventing all sorts of bugs/mistakes/bad ideas from hitting prod. Obviously automated tests are the main line of defence there, but code reviews are very important too. I'm kind of shocked that they want to skip out on this. I personally wouldn't want to rely on a core piece of infrastructure from a team practicing this level of cowboy coding.
In this context, prod means all the things you'd expect, plus: internal admin app, blog, static marketing site, old Rails app no one has touched in 2 years that one customer still needs, bash scripts to diagnose host issues, etc. There's a reasonable scope for "PR reviews are good", and it does not extend across everything SOC2 covers.
That's because SOC2 is only concerned about vectors for exploiting code, and gives very few shits about how well the platform actually works. The policy had to cover the full scope, though.
This is the difference between a "policy" and a "practice". We've long been doing code reviews on critical code, even last year when there were only 7 people here. And we've long had a release process meant to minimize the risk of bugs harming users.
We don't let most code go straight to prod without review:
* Post-facto review is a norm only on a couple oddball projects
* Out-of-process PR merges are a privilege for only a subset of developers
* We do the same in-depth code review most teams I've worked with in the last 10 years do. If you're rolling out a feature, teammates are reviewing your code.
Take a typical, clueful engineering team and SOC2 it sometime, and you'll see the difference between "not allowing cowboy coding culture" and "satisfying the letter of the SOC2 code review controls".
>Bottom-line: SOC2 is a weak positive indicator of security maturity, in the same ballpark of significance as a penetration test report (but less significant than multiple pentest reports).
Having gone through SOC2 a few times, it's more about opening doors to enterprise customers. The audits are very "grey" and subjective to your compliance auditor. Also, you get the freedom to say we're working on this and it allows you pass certain controls.
One final note: watching our CISO go through this I realize it's utterly the most boring, soul crushing job I have ever seen. It's non-stop clerical paperwork that nobody will ever read but everybody demands to cover their ass. Pay your CISO's.
I think one thing that doesn't get covered enough is SOC 2's value in providing additional data for vendor security reviews. That poor CISO that have to work on SOC 2 is probably tasked with reviewing new vendors on a regular basis as well. Sure there are security white papers and pentests (which can come from dubious sources), a SOC 2 report at least serves as a fairly independent assessment of a company's security maturity. Most people don't fully understand the amount of vendors required for a company to operate (take every department you can think of and assume each will have at least 3-5 vendors per quarter).
SOC2 is a deal breaker. It is already a pain to work for large American companies. You are forced to not be honest in the audits. The audits are complete bs by the way. Some external off shored company that wants screenshots of some specifics that they somehow found important. It is more Kafka than actual value. Miss those days, not…
It is better how it works in EU. Track money and stock activity + GDPR. SOC does not prevent Enrons. I mean, even if there was a complete ledger of everything ever done in a company you could still not prevent Enrons. The fraud part should have been detected anyway. And would have in EU, I would say.
Edit: D'oh! I mixed it up with SOX that is mandatory.
When an audit imposes a dumb requirement that nobody will benefit from, and it's easier to change jobs than to do the dumb thing, you have the theoretical option to be honest and do it.
But if it's easier to change jobs than to be honest - is the option to be honest one that will be taken by any rational person?
Actually, the easiest thing is to find a better auditor. A SOC audit isn't like an IRS audit, you actually pay them to come in and audit. Not all are created equal and sometimes you get what you pay for.
Thanks, and right... I don't expect maliciousness on fly's part. But you'd be surprised (or not) how many goog products phone home with anything they can find. In fact it might be called a "business model" of some sort.
Google's SSO can't really phone home. You're either using SAML or OAuth; in either scenario, the information flow is Google --> the app you're SSOing into; name, email, and user group information.
If you're SSOing into, say, AWS, Google doesn't get any access or private info out of AWS in the flow.
Accidentally hit enter the moment before HN went down and didn't get to edit it. :-D
But yes, reading google on your network gave me the heeby-jeebies. That the 'normal' thing is to host them on google is not really pertinent. If I was interested in that, I'd have done it. But I'm looking at other parties on purpose.
ceejayoz seems to think it is not technically an issue, google is not on your network. That sounds plausible to me, assuming it is accurate.
Not OP, but from my experience with SOC2 the auditing and compliance can be a complete joke. There are auditors who're entirely satisfied so long as an engineer dismisses all the dependabot alerts in Github (even if they're all dismissed with the "don't have bandwidth to fix this" option).
> SOC2 is a weak positive indicator of security maturity [quoted from TFA]
> SOC2 is the best-known infosec certification, and the only one routinely demanded by customers.
In the US... and I think that SOC2 is not an "infosec" certification per se but more of a generic policies and procedures certification. When I sucessfully took a startup to SOC2 certification, we were doing PCI DSS Level 1 compliance at the same time. SOC2 was very light on InfoSec per se (it was more "bureaucracy"), whereas PCI leaned heavily into InfoSec itself.
Of course that was for an American company, the rest of the world uses ISO 27001
SOC2 is absolutely an infosec certification; it's just one that's premised around the idea that that the service organization should have its own mechanisms for achieving security goals and then demonstrate compliance with them.
This is different from PCI-DSS which is single-domain-specific and highly prescriptive at a technical level as a result.
The ssh transaction/repl log thing is frightening to me. I get it, but when I've needed to get creds for a user to log into a service, one of my go-tos has been to pull shell history on all the hosts they log into since they've almost definitely typed their password prematurely at some point. So, logging all that sounds like a really fun way of making a password store!
The concern being raised here is that transcript-level SSH audit logs are the equivalent of permanent shell histories for everyone, and they are. But if you're giving team members a reason to ever type a password into an SSH session, you're got a bigger gap to close. We already have to do secrets management at scale, because it's a feature we provide to our customers, and so we already have a process for loading secrets into environments for host work.
I'd be more worried (but not terrified) that the session transcripts can teach an attacker a fair bit about how systems work, should an attacker get access to those. Of course only a small subset of attackers is going to care...
Careful, now. “Getting SOC2-certified” isn't the same as “doing the engineering work to get SOC2-certified”. *Do the engineering now. As early as you can. The work, and particularly its up-front costs, scale with the size of your team.*
Emphasis mine.
I've uttered this phrase or something like it so many times that I want to get this line fire etched onto a large, thick and sturdy block of wood and go back in time to several jobs where I and my cohorts got stuck with shoring up SOC2 tasks and smack my former execs with it.
Half joking but the pain and trauma from past SOC2 audits due to exactly this is real.
252 comments
[ 3.0 ms ] story [ 272 ms ] threadHe definitely knows the audience, but writing something like this is a special skill.
Surely you didn't write all of this just to make me sign an NDA to see your report...
AIUI, the intent is to ensure that the people reading the SOC2 report don't read into it more than what the auditor intended. This, according to the auditors, is fraught with difficulty, because you need a degree of familiarity with COSO principles and general SOC-ese to understand precisely what the auditor is making strong claims about, and what is _not_ being claimed.
In practice, the way this is accomplished is that the SOC2 report includes standard verbiage saying that the report is only for the client company, and for specific third-parties who must have "sufficient understanding" to parse the report correctly.
The client company then implements some process, with the guideline of "do something that won't piss off your auditor". For small companies, by and large the implementation is "you have to be an existing or prospective customer, email to ask for the report, and sign an NDA." The very act of requesting a SOC2 report implies that you know how to read a SOC2 correctly.
Some larger companies set up portals where you can help yourself to their various reports, sometimes without NDA - but in that case you have to click through a pinky-promise to not redistribute, and the report is watermarked with your identity to deter distribution. A very few publish the report at a hidden URL and hand out the URL to anyone who emails in to ask for it, though I personally think that's walking a bit too close to the edge of what they agreed to with auditors.
Companies with deeper pockets end up paying an auditor extra for a SOC3 report, which is "SOC2, but abridged and with unrestricted distribution rights". I believe the theory (besides giving more money to the auditor) is that the SOC3 removes all the information that might be misinterpreted, boiling it down to barely more than "I, a trustworthy auditor, confirm that the company is doing all the right things." You don't get much detail, but as long as you're willing to transitively trust the auditor (who are themselves scrutinized by their regulatory bodies), that gives you a "compliance is yes" document that you can publish far and wide.
If you want an example of what a SOC3 says, Github has one: https://github.githubassets.com/images/modules/site/security...
All it current requires is an agreement and your email address.
The Trust Report doesn’t show all the details unless you configure it to show those details.
SOC2 broadly contains:
Trust Report seems to only cover the first point.The Trust Reports contain programmatically-validated information (basically: Vanta's code says the control was in place continuously.)
There's (obviously) pros and cons of trusting a software provider (like Vanta) to validate technical configuration compared to trusting a human auditor to do the same.
Our bet with Trust Reports is that for some cases, having software do the checking and validation continuously is better than having a human auditor do it once a year.
They usually don't. SOC3s are generally useless.
Since the author is a member of the set "everybody", we have a paradox. :)
More seriously, it would not be hard to adjust the language just a little bit by saying, e.g. "Most people would be better off...". Alternatively, the author could adopt a common style used in business communication where the author creates a label for the group that would benefit from the SOC2 knowledge. Perhaps call the combination of "cynics", "customers", and "true believers" the "unwise trifecta" or something. (I admit don't have a catchy term in mind yet.)
e.g. remember when "synergy" actually meant something?
That said, I'm open to being talked out of this viewpoint. Being cynical makes me unhappy.
In all seriousness, I laugh at absurd absolutes. If you show me a sentence with "Most people" and the same sentence with "Everybody", I'm going to smile at the second.
But I don't think that means it's ambiguous. "I literally died" makes me laugh but there's no deception.
P.S. False dichotomy alert
Yeah no offense but this style of writing makes me go look for something else to read, whereas tptacek's style made me keep reading the (long) post..
There are other ones as well.
Your eyes might glaze over, but they can really help a startup graduate from "move fast and break things, cowboy shit" to a mature and respectable engineering organization.Congrats on the SOC2 and finding a good spin on the story. It is also an important number of checkboxes you might need to get those juicy Enterprise and Government deals.
a. discoverable via internal tools?
b. descriptive and understandable (the policies use terminology that maps to the business processes clearly)?
c. measured?
d. internalized by employees as part of the culture?
e. enforced (as desired)?
f. externally reported (e.g. to downstream customers)?
g. reviewed when appropriate?
h. adjusted or removed as needed?
There are some spectacular failure modes of policies in the real word. Here are five attributes that fit together nicely into a mosaic of dysfunction:
* Everyone has to take a sleep-inducing thirty minute training.
* Policy compliance is "tracked" with a spreadsheet on an ad-hoc basis.
* Everyone in the organization has to "check off" that they comply, starting at the bottom and working up the chain to El Jefe.
* If something hits the fan and you need someone to blame, follow the paper trail and blame the people who incorrectly attested to policy compliance.
* Virtually anyone could fail compliance if you haul out the microscope. This is a feature, not a bug. Now you have a convenient and official way to get rid of people you don't like for arbitrary reasons.
This is stylized, yes, but not too far off the mark at some places.
One of the main points of real audits like SOC2 type II are about validating enforcement of the controls outlined as policies. So the policy that only employees with job role Z can sign onto System Y would be checked by summarizing the login audit logs for Y, and verifying that all the names on the list. Other policies might require verification by reviewing all or some auditor selected subset of occurrences to verify. Some details of some policies cannot be fully verified. As long as that risk is known and documented, it is not necessarily a problem.
SOC2 audit reports provide a high level version of reporting to downstream customers, without necessarily revealing the full details of the policy. (Sufficiently important customers could always insist on seeing the actual policy documents, if the reports don't satisfy them).
But some of your remaining considerations are somewhat outside the scope of SOC2. And they can be tricky problems.
Same reason.
But it's meant to be a minimum. It verifies that there isn't one copy of the source code on a dev's laptop. It verifies that a dev who gets fired won't be able to log into the production server and delete all data in retaliation. It verifies that an intern isn't able to completely destroy the business by accidentally deleting the production database (because you have routinely tested backups and a documented RTO/RPO, of course). Being able to demonstrate this level of minimum competency is extremely valuable when you're in the B2B world and trying to sell your product to a larger client.
The paperwork is a hassle, but if your company is following best practices for development and operations, there shouldn't be much of a step change in what you're actually doing on a day-to-day basis.
What process did that company use instead of background checks, that you ended up doing as well?
In one case, I know a company is using reference checks to comply with the "background check" requirement.
(This is why there's a SOC3.)
Long story short: it's not complicated, and if you're currently doing a SOC2, like right now (or in the future) and you have reached the point where you're trying to get out of background checking everyone, shoot me a line and I'll tell you what we did and what we said (I may performatively NDA you in the process, because I like our auditors and don't want to irritate them).
Most higher level attempts to meaningfully share and reduce toil and wasted effort are not incentivized in risk/governance/oversight culture, so we all get to lose.
Like many people here, I didn't like the requirement. That being said
1) It's possible to configure background checks so you don't receive irrelevant information (e.g., if DUIs aren't relevant, then configure the check so you don't receive information about DUIs). In most cases, you'll just want to receive information about financial and privacy related offenses.
2) What you do with the information is up to you (unless your customers enforce certain actions). In general, the SOC-2 auditors will want to see a plan by which you acknowledge and manage the risk, which doesn't necessarily mean you can't hire the person.
DUI is a sign of gross recklessness and apathy for the well being of others. Sure I won't blame a young adult for doing this mistake and there are situations where it's understandable (i.e. some kind of emergency making you DUI even through you generally are against it).
But still I would prefer to work with someone who in the youth due to poverty has committed robbery (but not anymore since 20 years), then someone who in their 40th who is frequently driving under influence of alcohol.
Anyway even if I had a company and it for whatever reason would do background checks I wouldn't want to know the outcome as long as whoever is responsible for it following some strict guidelines didn't judge it to be a problem (and no if it's not a car company it wouldn't contain DUI, and generally I don't like background checks).
I just realized that I had forgotten that in the US you often do not have the freedom of not taking the car but e.g. the public transportation. This makes things more complicated. But then doesn't really change how I feel about it.
It's true that having shit public transportation and everything so far away that you need to drive complicates things, but there are always options. In Japan the public transportation is great, but the trains stop running long before the alcohol stops being served and it's not uncommon for drunk people to wait until morning even if it means waiting/sleeping outside all night. No reason folks here can't do the same.
I was more thinking about people with an alcohol addiction still getting to/from their job on a daily basis then people going home from partying.
My perception is that some people who don't want to do background checks feel that way because they don't want to know embarrassing details about their employees and colleagues that aren't relevant to work. And the good news is that employers can generally set up background check reporting to simply not report issues that employers don't think are relevant. And that makes it easier to offer background checks, and easier to meet SOC-2 audit requirements.
The post mentions background checks. On the one hand, I understand there's a real issue with these. On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. Hopefully the things they took from the other mentioned company do increase due diligence in vetting employees who have access to sensitive/regulated information.
Use it as a framework to actually think about BCP, DRP, etc, etc, and it won't be a total waste of time.
Edit: Also adding I bring up background checks as an example of learning from vetted practices, rather than trying to attack the decisions of fly. I respect this article, especially where it's easy for people on the internet to criticize decisions, when the reality is security is a series of tradeoffs, and to function as a business means having imperfect processes.
I don’t know that there’s much evidence that background checks work for this, though.
Again, they're not ideal, and there's a large social concern of a permanent record like that, but you have a duty of care if your customers are trusting you.
I'm sure bad credit is the indicator they're looking for, but what does it mean to them?
A new hire makes bad decisions? A new hire could be easily bought or bribed? A new hire is broke and needs a job?
This one. Same reason it's looked at for a security clearance.
So why do people not commit corporate espionage? Well, it might have more to do with character traits than financial stability. In fact, most spies probably have their life fairly well together, and will have perfect credit. As for any asset they might compromise, what’s the difference between someone with poor credit applying to a job because they need the money, vs. applying to a job because they need more money from your enemy bribing them? I’d argue the difference comes down to character.
So for that reason, I’m skeptical of the effectiveness of a credit report as a proxy for likelihood to commit corporate espionage. A good credit report doesn’t seem to offer meaningful signal in either case of a malicious attacker or a desperate contributor. A bad credit report produces as many false positives as a good one.
You're afraid of someone with 100k in gambling debts to the mob.
The bankrupt dude with a felony DWI is a liability for anything involving cash, driving or public contact.
One of the problems with not doing background checks is ex post facto if you do have problems with an employee, and it turns out that you could have discovered them with a background check, then that can figure into your liability.
In some countries (e.g. US) people are convicted all the times for things they haven't done. Or have done but which is in the past and really irrelevant for the job.
On the other time people are not convicted for a lot of kinds of white collar fraud all the time, too.
I don't really give a shit if someone was arrested for a drug offense (or many other offenses), and knowing that information brings up all sorts of complications, to the degree that it outweighs the value of knowing relevant stuff (largely because genuinely relevant things from a background check are rare).
Hiring managers should not be looking at BGC material.
And what kind of employee will you lose if you set such restrictions?
I don’t love background checks but I have seen a situation where a person previously convicted for embezzlement was hired as a controller.
No background check was performed and the person drained the bank account ($XMM) at their new job before being caught when checks started bouncing.
Preventing access to sensitive data is important. None of the top ten ways we try to solve that include "background checks", though.
You can run a SOC2 compliance program earnestly or as a check-the-box exercise.
If you're running earnestly, I would argue that the hardest thing about a SOC2 is ensuring that you stick to your guns on approaches that work for you and not adding cruft that you don't care about. If you let the latter happen, you will invariably end up a box-checker, and being a box-checker eventually contaminates a robust engineering / security culture.
And it's hard to walk back more restrictive / cumbersome policies; if you delegate your SOC2 to a person who doesn't deeply care, they'll eventually agree to put ClamAV on all the hosts or something (to make the auditors go away) and then you're going to be stuck with that for a while.
(So you need to find someone who has enough business context and good judgement to run the process, which is super painful from an opportunity cost perspective at a startup, and hard to locate at all at a larger company.)
That's spot on, not only for SOC2 but for many, if not most, relevant certifications. The most important part is "not adding cruft". Nothing sucks like being stuck in a ISO9xxx certified process because you over-specified your processes even though you'd get the "ISO9xxx-certified" label for 10% of what you did. Suddenly you cannot react with common sense anymore because doing so would violate contracts you made with exactly those big customers you got the certification for in first place.
Bingo. Just spell out the consequence: you no longer can optimize your compute costs by switching to a managed Kubernetes, because there is no ClamAV there.
1. There are a set of things you need to do for "real" security
2. There are a set of boxes you need to check to pass a compliance audit
I think SOC 2 is pretty reasonable in that, if you're taking it with the right mindset, there is a large, large amount of overlap between #1 and #2.
Using your example of background checks it’s probably more valuable to have proper acls and audit trail internally than doing background checks which is a really low signal compared to the level of hassle
I agree with your idea, but background checks are a poor example. They're negligable cost, always outsourced, and trivial to perform. They're worth doing if only to validate that your candidate said the same things as the background check says (if they say they're not a felon and they are, that's a red flag -- if they admit to it and explain why, you're not being lied to). In contrast, you actually need to spend time working on audit trails and stuff. One is hiring a vendor and checking a box, one is probably engineering work.
There's real value in being forced through this stuff, because these kinds of management processes are a real weak point at a lot of shops with otherwise strong security engineering. I'm glad that our policies and processes are clarified, and that there's an external process that keeps us honest and forces us to do the routine scheduled meetings, rather than keeping stuff in our heads. We started doing SOC2 prep work a year ago, and even before the audit, we were better than we were before we started.
But it is what it is. The thing that drives me nuts is when people suggest that good teams will maximize SOC2 so their security engineering can be informed by it. Yikes. No.
I think at the end of the day, SOC 2 aims to instill a basic level of organizational security so the company doesn't shoot itself in the foot. If a company can't genuinely follow a basic set of SOC 2 controls, can I trust them to do actual security?
Also, badly written checklists might be bad, but not all checklist are bad. Pilots use them. Doctors use them. Mechanics use them. In fact, most fields that involve critical life or death operations use them. Why? Because humans have a limited memory and tends to miss critical tasks all the time.
My personal hate of these is how much information they require you to hand over to some random organisation.
Some make you take photos or videos of yourself holding your ID, and sure they say that they delete the information, but we all know that rarely happens.
I just don't trust their infosec policies, and it's only a matter of time before one of these companies gets reported as having a public S3 bucket, or employee laptop, or USB drive stuffed with video and ID scans and background check data.
I understand the negative impact upon recidivism that background checks and stigma may have, but also we have to balance the interest of financial and information controls of organizations performing essential functions in society. Ideally speaking.
Telling any startup engineering team that they should maximize their SOC2 Type I audit is borderline malpractice. Every engineering decision you make to support a Type I is something you'll need to live with in your Type II. That might just be an irritating own-goal if it's you doing the Type II, and you have to waste a couple hours on the phone explaining to your auditors why you've decided to remove ClamAV from all your servers. But it's could actually be destructive if it's someone other than you running the Type II, and gets cornered by your original Type I claims into supporting that bad decision.
Telling any startup engineering team that they should build a security practice informed by a SOC2 audit --- that they should take the COSO framework and figure out what they can learn from it --- might also approach malpractice. There are probably reasonable corpsec process controls you could build based on the AICPA Security TSCs. But the TSCs are not based on modern software engineering best practices and they aren't informed by modern software security. They are heavily influenced by the security concerns of medium-to-large sized enterprises with sprawling legacy IT footprints, and you can easily lose security by rolling out controls that aren't relevant to your environment but require you to add attack surface and operational overhead to mitigate vulnerabilities that you don't have because you don't share printers or run Windows Server 2008.
SOC2 is not security. Security is its own thing. Good compliance work is a byproduct of sound security engineering. It does not work the other way around.
Naturally, it's easy and gratifying to point out that a SOC2 report doesn't make a company secure. I don't think we could have come up with a clearer way to have said that, or, for that matter, to explain that we did our best to minimize the impact SOC2 had on our engineering practices.
As for background checks: once again, we can't background check a decent-sized fraction of our team, because they're in jurisdictions that (very reasonably) forbid employers from running intrusive background checks. We considered just background checking the unfortunate US team members we have that can be made subject to them. I had a fairly long conversation in a Slack channel full of secops people from about a dozen security companies, and none of them told me a single story about how background checks (which are, as it turns out, performative, superficial, and error-prone) was a win for them. I did get stories about how they were problematic: for instance, I did not make up the thing about high school transcripts.
So, long story short: because of our workforce, our platform needs to be resilient against bad hires. You don't get that from background checks; you get it from security engineering, tight access control, tightly designed roles informed by those access control decisions, regularly reviewed internal audits, detailed audit logs, and sound hiring practices. SOC2 covers most of this stuff only superficially. Security engineering is the real work, at least for technology startups.
Later
This is way angrier than I want this to come across. I promise, it's not personal, and if I'm caricaturing any point you made, I apologize in advance. I had a 6-hour tattoo session that ended with an hour of ditch work and I am in a fucking _mood_.
Also, since I'm predictably being pulled in the direction of repeatedly dunking on SOC2 in this thread, I want to say very clearly that I had a fantastic experience with our auditors, who were more clueful than any other auditor I've worked with. Our auditors are great. Don't flunk us!
Can you talk about that more concretely? I'm sure there are many cases where compliance requires you to consider a risk that's not relevant, but it's hard to imagine that thinking about each risk and taking appropriate action (which, sure, in many cases will just be documenting why it doesn't apply) would damage one's security.
At the end of the day any list of potential risks is going to be imperfect, and some will be more imperfect than others, but I'd still think that the vast majority of the time you'll get a better outcome by engaging seriously with a given list of risk factors than by treating it as an exercise in doing the minimum. If you were trying to do security from the ground up without any regard to compliance you'd ultimately end up doing something quite similar - coming up with a big set of risks and then figuring out what you're doing about each of them - and sure, you could probably start with a better-targetted one than what SOC2 has, but that sounds like a matter of degree rather than being so different that you can't get any shared use out of doing those things together.
From a management level, a wide variety of modern best practices are already the default from Windows 2019 or so. The cognitive effort of looking at 50 security settings and convincing yourself they are all reasonable and won't break things is substantively better than the 400 or so we used to have. It's one thing to inherit this sort of legacy but it's a much worse thing to be implementing all these policies in a greenfield 2022 environment because they are all on some checklist.
[0] https://techcommunity.microsoft.com/t5/microsoft-security-ba...
Certainly this is giving me pause on using your platform for anything other hobby projects
Were you already using the platform? What for?
The best way to get detailed information about how our security practice works at Fly.io is to ask us about it directly. We're trying to be up-front about how weak SOC2, for everything else it might be good for, is with respect to security. Unfortunately, in the process of speaking plainly about SOC2, we have apparently sent the message that we think most of security is performative, which is not remotely true; the point is that we don't think SOC2 is an especially meaningful representation of the work.
My main point is you can either treat the SOC 2 as an adversary to overcome, or actually try and leverage it to be better. No matter what it's going to suck and be annoying, though Vanta/Drata can help. But one can leverage it to be a better company.
A less controversial example than background checks is infra cost monitoring. Where a lot of SOC 2 is focused on business continuity, in addition to security, one of the things required is that you're actually paying attention to your costs. A lot of cash flushed VC backed startups don't. So, once SOC 2 hits, the company that's treating it adversarially will just rubber stamp some quarterly meeting where they "look" at infra costs. Or, you can actually take that moment to level up the company to have macro review of the cost of goods sold, an ensure the business is on a healthy path.
Again, not a comment on your article, one of the big takeaways for me in running a security program for years was a general anxiety around being transparent on the program externally, because there's a certain type of "security" person who gets off on picking apart policies, without understanding tradeoffs that we were careful to make sure kept the company safe, while letting it function smoothly.
Coming out with an article like this is a great thing to do, where a lot of content out there is just "we got our SOC 2, and now we're prefect."
I won't comment on the background check thing again, not the least because I don't want to argue more for something I don't like, just think may be a needed evil.
If I don't, then my customers will forgive me in a few weeks and life goes on.
Also, it can help get you out of repeat security assessment questionnaires, so it can actually give you time back, depending on how many of those you have to field.
Of course if you are in an industry were clients don't ask for soc2, don't do soc2.
And it's a good question whether you want such larger clients at all. At one of the previous places where I worked, we used to put deliberately bad answers (the worst that our public version of the security policy would allow, not the actual practices) in security forms in order to get rid of too-demanding clients.
That seems like quite a waste of time. Nobody forces you to take on a customer, so if you don't want them just say no and move on, instead of spending a lot of everyone's time to go through the motions hoping for the deal to break.
... Which is not a good thing, because (as noted already in this thread) SOC2 doesn't actually make you secure. Nor does not having certification make you insecure. But, when used as a shorthand, it leads companies to engaging in compliance theater to get certified, spending a bunch of money without actually making their data noticeably more secure.
If you don't, you are missing out on a lot of customers who would have given you 10000x the one-time cost of SOC2.
aka "make the devops team deal with it even though we're paying double their salary to a 'Security & Compliance Director' who hasn't renewed any of the certs they used to get this job since 1997 and hasn't the foggiest clue how the SIEM works when the auditor shows up"
(I say "prove", as sometimes the question is malformed, and so you're trying to make the best of it as you can.)
Real security starts with a good security culture. Audit and compliance assures audit and compliance passes not a good security culture.
While I respect that the two are independent, being audited doesn’t mean your customers aren’t at risk or you care about them.
It's weird. It's very convenient for the user, but it's on the opposite side of the convenience/security divide. Hijack one session and re-use it for other purposes. Of course it's preferable to not let systems access credentials but one would have thought that we had yubikeys or cards and whatnot now.
You only activate the SSO account when the employee onboards. You deactivate the account when the employee off-boards. You can suspend the account if the employee is on an extended leave-of-absence.
You can do this in one place, as single source of truth, rather than having to demonstrate that all the different identity systems in which Bob is represented have deactivated him when you have to let Bob go.
Ideally this happens automatically due to an integration between your HR system and your identity platform.
The alternative is that you have some kind of runbook that tells you exactly who to speak to about removing accounts from what systems (etc) which rapidly becomes unmanageable as scope of applications and number of employees increases.
Even for small firms, you can run into problems when the person you need to let go is the person that knows how to run the process.
Edit: For your work password manager.
But for certain operations (e.g. privileged access stuff), we do have step-up to a second authentication factor in addition to SSO.
That's distinct from SSO. SSO is like Kerberos, where you authenticate with a ticket, that for the lifetime of the ticket will not require the user to authenticate again.
In my experience SSO systems are not a security benefit. They risk worsen the impact of attacks. Even with mutual authentication there are still issues with ticket lifetime, extension and playback. Especially when people do whatever they feel necessary to run cron jobs, CI/CD and other systems. They do bring convenience benefits, which are a good thing in themselves, and may have secondary security benefits, but I can't shake the feeling that the arguments are similar to the password change policies of yesterday.
For some time it felt like a huge push towards things like Yubikeys and other modern smart cards that focus on making authentication painless enough to lessen the need for SSO in modern organizations. I would have expected to see more of that, not less.
On the vendor / policy side, many/most of these questions trickle down from NIST or similar institutional guidance. The auditors pick up on that and on practices from comparable companies they've audited, which can be helpful when your industry is moving towards sanity and painful when there's a meme that makes no sense in your context.
(If you spend significant time dealing with customer compliance issues, I would definitely vote that it's worth being familiar with the relevant subset of NIST pubs.)
If the attacker has the ability to hijack one session they could hijack all the other n sessions from the same place anyways. So while SSO does centralize your identity, it also makes it safer because you can uniformly handle provisioning/deprovisioning, password resets, 2FA and all other policies in one place for your entire organization instead of a hundred separate ones.
> I was surprised by how important this was to our auditors. If they had one clearly articulable concern about what might go wrong with our dev process, it was that some developer on our team might “go rogue” and install malware in our hypervisor build.
> ... our auditors cared a lot about unsupervised PRs hitting production.
It's indeed a growing concern, and (shameless plug) the main reason we founded arnica.io (help organizations automate that process without hurting development velocity)
I decided not to use them because I (foolishly in retrospective) wanted to "learn" the SOC2 and PCI cert process by walking through it (kind of how you do derivatives, intergrals, numerical methods in school by hand so that you "grok" them).
Since then, I've heard good things about Vanta from a couple of friend CTOs that adopted them. If I had to go through SOC2, PCI or ISO27001 (I did that in yet a previous startup) I would deffinitely go with them.
> To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer.
> This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. But our auditors cared a lot about unsupervised PRs hitting production.
> We asked peers who had done their own SOC2 and stole their answer: post-facto reviews. We do regular reviews on large components, like the Rust fly-proxy that powers our Anycast network and the Go flyd that drives Fly machines. But smaller projects like our private DNS server, and out-of-process changes like urgent bug fixes, can get merged unreviewed (by a subset of authorized developers). We run a Github bot that flags these PRs automatically and hold a weekly meeting where we review the PRs.
Letting code go straight to prod without a review is just IMO a really bad practice. It sounds like they've improved significantly here, but still have plenty of gaps where people can ship to prod with no code review until it's been running in prod for up to a week. This isn't just about stopping bad actors, it's 99% about preventing all sorts of bugs/mistakes/bad ideas from hitting prod. Obviously automated tests are the main line of defence there, but code reviews are very important too. I'm kind of shocked that they want to skip out on this. I personally wouldn't want to rely on a core piece of infrastructure from a team practicing this level of cowboy coding.
That's because SOC2 is only concerned about vectors for exploiting code, and gives very few shits about how well the platform actually works. The policy had to cover the full scope, though.
This is the difference between a "policy" and a "practice". We've long been doing code reviews on critical code, even last year when there were only 7 people here. And we've long had a release process meant to minimize the risk of bugs harming users.
* Post-facto review is a norm only on a couple oddball projects
* Out-of-process PR merges are a privilege for only a subset of developers
* We do the same in-depth code review most teams I've worked with in the last 10 years do. If you're rolling out a feature, teammates are reviewing your code.
Take a typical, clueful engineering team and SOC2 it sometime, and you'll see the difference between "not allowing cowboy coding culture" and "satisfying the letter of the SOC2 code review controls".
Having gone through SOC2 a few times, it's more about opening doors to enterprise customers. The audits are very "grey" and subjective to your compliance auditor. Also, you get the freedom to say we're working on this and it allows you pass certain controls.
One final note: watching our CISO go through this I realize it's utterly the most boring, soul crushing job I have ever seen. It's non-stop clerical paperwork that nobody will ever read but everybody demands to cover their ass. Pay your CISO's.
Maybe in the US. For the rest of the world, ISO27001 is arguably better known.
>Developed by the American Institute of CPAs
I don't know when CPAs became infosec experts.
>Each company designs its own controls to comply with its Trust Services Criteria.
Because it depends on self-assertion, SOC2 is generally a weak organizational certification.
It is better how it works in EU. Track money and stock activity + GDPR. SOC does not prevent Enrons. I mean, even if there was a complete ledger of everything ever done in a company you could still not prevent Enrons. The fraud part should have been detected anyway. And would have in EU, I would say.
Edit: D'oh! I mixed it up with SOX that is mandatory.
SOC does not prevent Enrons; SOX does ... that's a different audit, and a different process, though.
>The fraud part should have been detected anyway. And would have in EU, I would say.
cough ... Wirecard ...
Not sure I agree. The question is, what are you willing to sacrifice in order to be honest.
But if it's easier to change jobs than to be honest - is the option to be honest one that will be taken by any rational person?
Yes. But I don't think this is a good forum for arguing differences in values.
I realize this is for employees and it is hard to escape their gravitational pull, but I hope no customer data is going to Google unless asked for.
If you're SSOing into, say, AWS, Google doesn't get any access or private info out of AWS in the flow.
But yes, reading google on your network gave me the heeby-jeebies. That the 'normal' thing is to host them on google is not really pertinent. If I was interested in that, I'd have done it. But I'm looking at other parties on purpose.
ceejayoz seems to think it is not technically an issue, google is not on your network. That sounds plausible to me, assuming it is accurate.
That seems plausible, but can you flesh out the argument?
> SOC2 is a weak positive indicator of security maturity [quoted from TFA]
I'd argure it's no indicator at all.
In the US... and I think that SOC2 is not an "infosec" certification per se but more of a generic policies and procedures certification. When I sucessfully took a startup to SOC2 certification, we were doing PCI DSS Level 1 compliance at the same time. SOC2 was very light on InfoSec per se (it was more "bureaucracy"), whereas PCI leaned heavily into InfoSec itself.
Of course that was for an American company, the rest of the world uses ISO 27001
This is different from PCI-DSS which is single-domain-specific and highly prescriptive at a technical level as a result.
Anything that you use history for should be set as an alias or shell function or script.
Emphasis mine.
I've uttered this phrase or something like it so many times that I want to get this line fire etched onto a large, thick and sturdy block of wood and go back in time to several jobs where I and my cohorts got stuck with shoring up SOC2 tasks and smack my former execs with it.
Half joking but the pain and trauma from past SOC2 audits due to exactly this is real.