112 comments

[ 3.4 ms ] story [ 176 ms ] thread
> creating an unnecessary risk for Android users in order to score cheap PR points

This seems like an accurate take. Why else would google disclose the bug so quickly?

Because a week had elapsed since the batch became broadly available. Epic is missing a way to push out updates, so they wanted more time.
To reinforce the idea that the only way to safely get apps on Android is through it's store.
Note that is not the same thing as "get safe apps" - just that the app hasn't been tampered with between the publisher and you. There seems to be a lot of spyware on the Play store still (or whatever it's called this week).
Google seems in the right here. The disclosure was in line with their policy and I'm sure it made management feel oh so good to burn Fortnite in public for their decision to forego distribution in the Play Store.

But here's my personal take. As a consumer I want my phones to run secure apps on secure operating systems. This has a cost obviously which is fair to pass on to developers.

It's clear to me, with all the rogue apps and crap in the Play Store that Google is not investing enough in managing app store content.

Fortnite won this battle but in my book Apple will win the war.

You mean google? You're right though, Google does a crap job auditing the playstore. It's ludicrous how they'll allow apps get access to everything on the phone without any kind of serious warning to the user.

My kids install all sorts of crap. I've warned them that all their texts and photos will end up on the internet because of it. Not highly probable, but certainly possible. Makes for a useful double check they're not texting anything silly.

>> It's clear to me, with all the rogue apps and crap in the Play Store that Google is not investing enough in managing app store content. >> >> Fortnite won this battle but in my book Apple will win the war.

> You mean google?

He's saying that Apple, is doing a much better job of "managing app store content" than Google, and so ultimately it doesn't matter if Fortnite or Apple wins this battle: they're fighting over a mound of rubble while Apple builds a castle.

> they're fighting over a mound of rubble while Apple builds a castle

It doesn't matter. Apple doesn't win in the end.

In the other Apple vs mound of rubble fight (Windows), Apple lost.

Android is doing the same thing. Android 4.0 was Windows 3.1 (first Android version to be "modern", IMO), Android 5.0 was Windows 95 (better UX). Android now just needs Windows XP to be stable enough (I'd argue Android 8.0 was that) and Windows 7 to cover the security aspects (most likely wide spread adoption of new Android permissions). But the writing is kind of on the wall, outside the US Android has majority market share and it's only going up.

Would Apple's walled garden make more sense for more vulnerable users?
Part of the point here is that you can run a secure store with quality controlled apps by charging companies to appear in the App Store.

That charge cannot POSSIBLY be 30% of sales. Not least because the cost of checking the apps that go into the App store is not in any way related to the revenue the app generates.

The charges for appearing in the store come from Google (a monopoly in the smartphone market in most places) rent-seeking.

Google has very public disclosure deadlines that they followed in this case. The patch was out, Google told people to update.

Alternatively, Google could of let users continue running unpatched software until the 90 days expired...

(comment deleted)
Aren’t you allowed to use whatever payment service you want inside an Android app, and avoid Google’s in-app payments? Then you avoid the cut to google. Or have I been violating the agreement for years...
You can only do this if the payment is not for digital goods obtained and used in the app. Otherwise, it's a violation of TOS.
Sorry, I'm with Google on this one. Epic chose to go off the reservation (so to speak) and created a giant security hole while doing it. Woops. This part of the article covers this:

> Google's disclosure rules state that it reveals details of bugs to the public 90 days after reporting them to the developers responsible if they have not been tackled, but only waits one week after a patch is made "broadly available".

So Epic made a patch available and Google waited a week. But:

> ... [Time Sweeney] denied suggestions that the tech giant had acted in users' interests by refusing to keep the matter private until mid-November.

The 90 days is for unaddressed bugs.

This is nothing more than an attempt by Epic to fix bad PR from a security vulnerability they introduced (which arguably at this nascent stage might reduce public confidence in their bypassing the Play Store) by trying to deflect it onto Google.

And I get the desire to skip the 30% cut but if you're going to do that you're then responsible for the safety of a person's phone and data. At least be up to the task.

Man, I had to read that headline like 10 times before it made sense. /r/titlegore
Same. Quotation marks are important, people!
Totally, I first that was is that even English.
I also never use "row" to mean a fight, so I read that as "bug row" (like "skid row", or "cannery row") and couldn't figure out what was even going on.
Don't they have an in-app updater? Why does it take more than a week for updates to propagate?
I side on Google on this one.

Want the profits? Then use them to actually make a safe product.

7 days seems awfully short and not in the interest of users.
Seven days after the patch, not after disclosure with Epic.

Epic patched it on August 17th, Google waited 7 more days (as it is their usual practice) and then they went public.

Issue tracker makes more sense than the article IMO: https://issuetracker.google.com/issues/112630336

I guess that usually seven days is more than enough for everyone to update to the latest version available on the Play store, however downloading a giant .apk file within seven days is an inconvenience for most users, which is why Epic requested 90.

Also, Playstore checks for you in the background for updates even if you don't use the app in question, the Fortnite launcher doesn't.
What if I tomorrow figure out a bug in Gmail/adsense and release it public-ally without giving google a chance to fix it.

My logic : Want security, stop recording peoples data. ( The most secure way to store data is not store data in the first place)

Would that be fair to google and the consumers of google.

And I would bet that I would be booked under some "Hacking" related law for doing that.

That would be completely different than publishing a bug a week after it was patched.
I'd argue that Epic is irresponsible for putting users into this position to start with. All Google did was tell users that they need to update ASAP as they're ALREADY in a compromised position.

If there was no update/patch, I might side with Epic, but as soon as the fix was out and everyone needed to update, users needed to know the risks of inaction would inhibit. Not least of all because people may start reverse engineering the patch.

Epic has decided this is revenge for them not using Google Play, but if you look at Google's bug efforts historically, this has always been how they handled these issues. They've been highly consistent about it.

If Epic wants to keep serious security bugs under wraps in the future, maybe they shouldn't rely on unpaid third parties to audit their code.

PS - Google gave them 7 days from patch release, so most auto-updaters likely updated the installer.

They're both in the wrong. Epic for screwing it up and rushing rather than investing in security, and Google for trying to score PR points at the expense of their users. Google is being anti-secure here by not allowing the update to filter through the ecosystem.
> Google for trying to score PR points at the expense of their users.

Except this is how Google has always handled these bugs. The article even links to other examples involving other companies.

> Google is being anti-secure here by not allowing the update to filter through the ecosystem.

Or pro-secure here by telling users to urgently update rather than doing nothing and hoping nobody spots the bug and starts exploiting it before users get lucky.

Well, it's not that there is nothing being done. You're distributing the patch.

You don't have to go yelling about the fact you're distributing a highly important security patch, that only draws the attention of the bad guys.

Wanting to distribute such patches as low profile is a valid choice and is not "doing nothing and waiting to people to exploit it".

If you are a hacker it is not improbable that you are keeping tabs on updates for high profile software like Fortnight. In that case, doing things "low-profike" gives bad actors an edge.
Even if you keep tabs on it, would you inspect every single update that comes out or would you rather inspect the ones labelled "security updates"?

Low-profile means what it says on the tin; make it sound so boring that hackers are less likely to attempt it.

Plus being low profile reduces exposure to people who only look for high profile stuff.

And plus "not improbable" =!= "fact".

(comment deleted)
Google is punishing Epic for communicating. Had Epic silently released the patch and not responded to the thread they could have gotten more time to patch. That doesn’t bode well for future security issues.
If Epic had silently released the patch and not responded to the thread, that would be irresponsible and unethical, be harming the security of their users and be spitting in the face of the responsible disclosure community - that's something that you can do once, demonstrating to the world that they explicitly refuse responsible disclosure and every security researcher should publish Epic's vulnerabilities immediately without notifying Epic first.

And it's not like they would have succeeded - you can't patch something silently, if you know of a vulnerability it's not that hard to verify if it has been fixed in every upcoming version or not. Google would have had noticed that within a day anyway and (with justification) disclosed the vulnerability right then and there, instead of giving them those 7 extra days.

The only other thing I’d do here as Fortnite’s Chief is to broadly make secure install infrastructure available to other indie developers. This would gain some goodwill and could turn into another revenue stream.

+1 points for turning this issue into more publicity.

> make secure install infrastructure available to other indie developers

Epic doesn't have "secure" install infrastructure, that's the whole crux of this issue.

Sure but they are a capable well-capitalized development organization. Such infrastructure is not an impossible technology; numerous open source implementations exist. Therefore they will have such infrastructure sometime in the next 6 months. When they have it, they can offer it as a service to other organizations whose interests are similarly opposed to Google's.
You're absolutely, completely right! Secure infrastructure is possible and numerous implementations exist.

Given how right you are, it's exceptionally odd that a capable well-capitalized development organization would not use such systems and manage such a basic mistake.

"Poor decision by management hoping to save some money" is not a rare event.
For that matter, neither is a poor decision made by engineers who think it's fun to reinvent wheels. Both have happened plenty in history.
As of 7 days ago they do.
According to who or what?

That's definitely not what the bug report and Epic themselves say, they never fixed the execution of unverified code issue, they just moved where the unverified code was stored so that other apps couldn't replace it easily.

They still need to actually verify that what they're executing is what they verified.

> They still need to actually verify that what they're executing is what they verified.

They did. Why would you say otherwise?

If they can't trust the private storage to be private, then you're looking at a compromised phone where they can't trust any mechanism.

> They did. Why would you say otherwise?

Because I read the bug report? Where Epic specifically say they didn't fix the execution, instead they changed the storage context from External to Internal. The main issue is that they verify after download instead of before execution.

You put an exclusive write lock on the file, verify the file, and then execute the file without releasing the lock. It is what every other major installer or updater does. Epic's version instead downloaded the file, verified it, then [Sleep], and installed whatever was at that location. It is insecure.

Internal app-specific storage is private. That's all it needs to do.

I'm not sure holding a lock even works to prevent file renaming on most OSes...

Maybe a lock on the directory.
Interesting! can you execute a file directly from the file descriptor though?
No idea, but it doesn't need to execute it, it needs to trigger an install of it.
Replace execve with fexecve and you can run file descriptors.
Plenty of other vulns can lead to write access in private storage. You don't need a compromised phone for this to still be a security challenge.
If you have that, you probably don't need to wait for an apk install.
What is a bug row? Is that a British thing for bug report?
a "row" is a loud dispute/fight.
"Row" is british english for "argument", so it would be a bug argument (or disagreement).
A row is a fight, though it's pronounced in a way I can't seem to type phonetically. But not like row of corn or row the boat.
> it's pronounced in a way I can't seem to type phonetically.

Rhymes with "how".

Pronounced like in frown. Or like wow or how or the "low" in flower.
Rhymes with bough, not bow.
"bow" is an ambiguous spelling that indicates words with multiple pronunciations.

It can be pronounced like "grow" and like "how" (or "bough".)

Yes, that is correct.

Take a bow with your bow tie on.

Pile up a mow when you mow down your hay.

Have a row when the row is not straight.

Tie up the sow when you sow seeds.

That tow-headed man drives a tow truck.

(For additional fun, pronounce bough, cough, rough, and tough.)

Just to further underline why Google is a dick here: this is the issue tracking the vuln https://issuetracker.google.com/issues/112630336

Timeline:

  8/15/2018: Google reports issue to Epic
  8/15/2018: Epic begins investigating
  8/15/2018: Epic confirms bug and begins working on fix
  8/16/2018: Epic is testing the fix
  8/16/2018: Epic begins deploying fix
  8/16/2018: Epic asks for full 90-day period to deploy and test the fix
  
  8/24/2018: Google finally replies: "now the patched version of Fortnite Installer has been available for 7 days we will proceed to unrestrict this issue in line with Google's standard disclosure practices"
They literally waited until the last minute to go, no, sorry, we're not waiting, it's full-disclosure time, :trollface:

Epic noted in the article that the game won't auto-update until the user runs it. So they clearly would have needed to inform users that didn't play regularly to get the update. However, Epic could have notified its own users when they were ready, rather than Google outing them by default.

If Google had replied, like, 7 days earlier, and said "sorry we're not waiting", maybe Epic could have made different decisions, knowing they wouldn't get the time they requested. It's clear that Epic worked their asses off to churn out a patch. They could have announced within that time if this was properly coordinated.

Whether this was just Google being indifferent to a company's reputation, or an active attempt to penalize Epic for not forking over 30% of their revenue, is left up to the reader. But Google acting in the interest of users is not a defense here, because Google just ignored Epic and then fulldisc'd.

> So they clearly would have needed to inform users that didn't play regularly to get the update.

The game came out on August 9th. Knowing how the release was big news, that the game itself is gigantic in size, and only runs on certain higher end smartphones, I would argue that there are close to no Android users that would not enter the game within those seven days.

No, they didn't wait until the last minute. Read their description, it is the last two lines, they did warn Epic.

Here, I'll quote it for you:

> [NOTE: This bug is subject to a 90-day disclosure deadline. After 90 days elapse or a patch has been made broadly available, the bug report - including any comments and attachments - will become visible to the public.]

Google waited a full week after the patch has been made available then they disclosed it.

"After 90 days elapse or a patch has been made broadly available"

This incentivizes companies not to announce their patches, not to coordinate with Google, and to only update in secret, because as soon as they tell Google they have a patch, the clock runs down in 7 days, rather than 90. This is a dangerous policy.

No, it isn't.

Anyone can break apart the patch and figure out what was changed, meaning they can figure out the security issue and take advantage of it. At this point, both Epic and Google are at fault for not disclosing the details and warning people to update right away.

There is usually a big difference time and yield in creating exploits from random binary updates and doing the same from disclosures. The amount of people who dissect, or even can dissect, updates are far less than everyone who follows disclosures. So I don't think that argument makes much sense in this scenario.
Normally I agree but this is Fortnite, massive profits all the time. Just a few hours worth of exploit can gain some a chunk of cash.
Epic is at fault, maybe, for not e-mailing Google every single day asking that they coordinate the announcement better. Google is at fault for not responding for 7 days and then disclosing, and is at fault for having a policy that makes announcing patches dangerous.

If you think people should be warned right away, Epic should have the right to notify its own users, and Google should have the basic decency to reply to a request to coordinate.

Nope, Google is not obligated to reply at all nor do they have any obligation to hold on these announcements. There are no faults for Google here. Their 90-day policy is just a courtesy call that comes with an exception, the moment a patch is made available, the disclosure will be made public. Google is not being paid nor asked by Epic to find reports, Epic has no rights to anything.

Google found the bug, they have the right to disclose it however they want. There are no legal standards anywhere that requires anyone to do anything with bug reports.

Epic doesn't have any rights here, it sucks but they're not entitled to anything. I hate this because I've experienced this from Google before (at work), I would love to have that right to hold up to 90 days before we can notify people but until there are legal standards, there is no right.

> Google is not obligated to reply at all

They have an ethical obligation. Security disclosures have major ethical concerns. If you're acting unethically, you are a dick.

Sure, Epic doesn't have any rights here. But we don't need a law to know people [and corporations] should not be dicks.

Google could have been more ethical by replying to the vendor after making their initial communication. They chose not to.

> They have an ethical obligation. Security disclosures have major ethical concerns. If you're acting unethically, you are a dick.

You are correct in that Google has an ethical obligation; and they and met it by protecting all Android users. I would say that Google making this announcement lets all Android users know that its not safe to just randomly install apps on your phone. Yes, they allow it, but at the same time the public is genuinely ignorant of any type of security practices.

> Google is not obligated to reply at all nor do they have any obligation to hold on these announcements. There are no faults for Google here.

This is a non-sequitur. You are equivocating between legal obligation and ethical obligation.

Their ethical obligation would also be to err towards full disclosure, not away from it.

Google's original view of the situation was that installers like this make devices measurably less secure (to which Epic's response was pretty meh), and the installer being exploitable to install pretty much anything only proved that point and then some.

...and frankly, it would not matter who wrote the installer if it got used for a mass infection and attack. Google would still get the bad press and a generous helping of blame because of the trashy way so many "tech reporters" behave to get those mouse clicks.

Google independently discovered the issue, Google can therefore independently verify if a patch has fixed it with or without Epic's information.
No, look at "broadly available" part of the sentence you quoted.

It would be reasonable to tell Google that they have a patch, show them the patch, and tell that's going to be released on third thursday of September or whatever, and the disclosure should wait until that - that's compatible with that policy. Showing the patch to Google and coordinating with them doesn't start the clock, the clock starts when you start sending the patch to random users.

It's not acceptable to release a patch to the general public and then delay the disclosure for weeks. Once Epic discloses the vulnerability to the bad guys (by making the patch broadly available), the good guys i.e. the general public needs to know the details ASAP.

The 90 day buffer is the time for analyzing the vulnerability and developing (and testing) a fix for it - but not for deploying the fix; deployment must be much, much faster than that, preferably on a single day.

That doesn't really change the incentive though.

I guarantee Fortnite wishes they had silently released the patch and hope Google didn't know.

I'm guessing they might in the future.

This seems to be why Epic asked for Google to extend the period to 90 days, which didn't seem too unreasonable. Ignoring this request only to reply just before disclosing the vulnerability seems rather careless.
The policy prescribes no more than 90 days between informing about the vulnerability and releasing the patch and/or information, however, it prescribes 0 days between releasing the patch and disclosure.

If Epic needed more time to prepare for deploying the patch, then they should have released the patch later, coordinating with Google - they certainly had time left for that, up to these 90 days. But once Epic started the clock by making the patch widely available, it is unreasonable to wait anything comparable to 90 days. Google should have disclosed the vulnerability on the same day as the patch, since the patch also 'discloses' the vulnerability to those who would use it. Delaying it for a day or two could be mostly safe; 7 days is pushing it, anything more than that is not a good idea - by that time the bad guys know the vulnerability anyway, and the delay is just harming everyone else.

I still see no reason this couldn't have been communicated to Epic, nor why Google felt the need to take it upon themselves to decide. Whatever happens now can no longer be considered to be Epic's fault, as other than existence of the bug itself their response to the report has been close to perfect and Google overruled them on anything else.
Google did communicate this, in the initial post where the bug was filed and disclosed to epic. (I work for Google).
Announcing something doesn't make it ethical.
this is so often the case with comments about Google here on HN; someone vilifies them, then another commenter explains what really went down. The people at Google are not evil or malicious, lol
No, Epic is at fault for having a basic bug that is incredibly easy to exploit.

At what point did people who browse a site called "Hacker News" start believing in Security through Obscurity?

It's not named for the security variety of hacker.
Right about the time "hacker" started to mean "VC funded startup person".
(comment deleted)
I don't blame Epic at all for avoiding the Play store. I find it patently unacceptable that two large tech companies now act as gatekeepers for software releases. Sure, it provides extra security, but at a heavy cost. Between Facebook, Google, and Apple the vast majority of the public's digital speech (software and written language) are filtered through large corporations with immense power and few regulations. This feels very, very scary to me.

It looks like the Fortnite APK is 1.88GB. I don't have familiarity with the Android platform, can you update via patches (over a full binary download)? If updates are anywhere near that size, a 7 day disclosure after patch is patently irresponsible (play store or not). This smells like a cheap PR play by Google.

Play Store does incremental updates on the background.
The Fortnite Installer APK, the part with the security flaw, seems to be about 4MB?

The Fornite APK itself is also <100MB.

I don't blame Epic at all for avoiding the Play store. I find it patently unacceptable that two large tech companies now act as gatekeepers for software releases.

More like 5 - Nintendo, Microsoft, and Sony all control what software goes on their platform either digitally through their console app stores or physically since all media has to have keys from the console makers to run.

Sure, it provides extra security, but at a heavy cost.

Seeing how the last 30+ years of personal computer use has played out, the vast majoriyy of users should probably be using locked down devices.

And Epic provided the perfect example of why app developers should be forced into a strict sandbox. I need my phone to always work it’s much easier for me (but not most people) to recover from a virus/malware/ransomware on my computer than a hypothetical one on my phone.

Why isn't Epic at fault for dishing out insecurity on a platform that contain's a user's everything? If instead it Google wasn't in the picture at all, but an unknown hacker who disclosed vulnerability, would it still make sense to divert blame from Epic? I personally would still blame Epic.

Similarly, I also wouldn't care if a random hacker or reputable firm disclosed Target's vulnerabilities when they put millions of customers at risk. I'd blame Target by a big margin.

I blame them. This is the world we live in. Google was nice enough not to lock down their ecosystem (like Apple), so in a way they're being sensitive to your concerns. Epic should not be punishing Google for being permissive. This way of distribution should be the exceptional case; for when Google is being unreasonable (eg. censorship), or for more advanced situations. This isn't the ideal, general purpose channel for releasing software. That's what the app store is for. Epic is putting their users at risk to save some bucks.

On a side note, I don't love Google's rules as I understand them. They seem to punish prompt action and honest communication in some cases (eg. this one). I think you should just get the 90 days if they're willing to allow it. Of course, this is always their call.

EDIT: Other users are saying the short deadline is because the public will be made aware of an exploit, due to existence of a patch. I did not think of that initially, but it does make sense

> to save some bucks.

Epic is also saving their users some bucks. I'm not so sure that users hate the tradeoff.

Hey, you're probably right. I'm not so sure the user should be making that decision though. Users would probably do away with strong passwords and automatic updates too, if they had the option. It's not their call, and shouldn't be.
> Sure, it provides extra security, but at a heavy cost.

There is nothing stopping Google and Apple charging a fixed fee for the cost of verifying the quality of submissions to the App store. The 30% is just pure rent-seeking, and frankly now it's monopolistic on Google's side.

It's ironic that Epic is going to accuse anyone of being irresponsible. I got an alert from haveibeenpwned a while back showing that my login credentials from Epic were part of a leak, including my password in plaintext. It wasn't a password that shows up in public password dictionaries.
How do you know it wasn't your computer that got compromised?
It's not too surprising that Google's definition of "patched" is "rolled out to a small percentage of users" since that's essentially how it's worked on Android for years.
Just want to make this clear, this is the standard disclosure policy of Google:

> [NOTE: This bug is subject to a 90-day disclosure deadline. After 90 days elapse or a patch has been made broadly available, the bug report - including any comments and attachments - will become visible to the public.]

Note the "or a patch has been made broadly available". 90 days policy almost never apply when there is a patch released.

It does not make sense to keep the vuln report hidden when a patch comes out, the act of the patch itself reveals the security issue for anyone who takes the time to check what it does.

This is exactly why epic didn't and shouldn't get the full 90 days -- users that haven't yet updated are more insecure with the patch out than before.

The only thing Google should have done better is make it clear to epic that they will lift it in 7 days, not the 90 that they asked for. Epic could have known by looking at other incidents, but based on their request they obviously did not and Google should have corrected them.

Google has effectively mandated everyone implement force-push software updating for all software everywhere, under their 7-day "this is a nice piece of software you've got here, I'd hate to see something bad happen to it" disclosure policy.

Google is throwing their weight around to dictate their philosophy on software to the entire industry. It's just good for us that it's debatable whether this particular edict is in everyone's interest, not just google's. Not all that reassuring about Alphabet in general, tho.

Epic is happy to pay Apple a 30% tax, but does not want to do the same for Android? And when Google exposes vulnerabilities in their approach (which was totally predictable), they cry foul?

If anything, Google should just make sideloading of apps illegal and let companies like Epic face the legal risks if they try to bypass security for profit motives.

Don't you need to jailbreak to install a 3rd party app on apple?
Either that or a Mac and Xcode. It’s practically impossible for most people.
They're probably not "happy" to pay Apple 30%, it's just they have no alternative on that platform, whereas on Android sideloading is actually a possibility.
Didn't Epic take this action (releasing without the protections of the play store) directly saying FU to Google? Isn't an act like that going to receive a response like Google did? I'm not questioning the morality of either side. Each action correlates exactly with the response.
hmm admitting that a non GooglePLay store android app has a bug...that is an essential duty to ALL ANdroid users whether they use Google pLay Sotre to get apps or not.

The dick is Epic not Google