Ask HN: How does Apple achieve both secrecy and quality for a release?
I know there are several counterexamples, but in general I'm impressed that Apple manages to keep new products/features under wraps until a major event, and then the product is in stores a couple weeks later with pretty high quality (user experience, reliability, minimal major bugs, etc.) Can anyone who has first-hand knowledge shed some light on how this is achieved with what I assume is quite limited user testing to contain leaks. Even working on products with multiple public betas and customer feedback sessions, it's hard to hit a high quality bar with a product release.
92 comments
[ 5.5 ms ] story [ 151 ms ] thread- Apple has practically unlimited money
- The company can afford the best of the planet's talent
- The companies business is ethical and therefore the top layer of talent hesitant to join ad-based companies pick Apple naturally
- The company is highly design driven, which means it's the perfectionists making most decisions (roughly speaking)
All this seems to have a cultural impact that optimizes for quality without much a/b testing.
They also make it challenging to switch teams internally, you basically compete for the same role an external candidate would, and you interview as if you were an external candidate.
All of that makes it very easy to get burnt out and so either you are someone bought into the "cult" of Apple and couldn't see yourself working anywhere else, or you peace-out once the rose-colored glasses come off.
Highly ethical ? They're China's lapdogs (Google left China). Also they're the ones who don't want to pay employees for the time they spend being searched, nor acknowledge when their platform or their customers have a virus.
Design-driven ? They're the ones with keyboards with a 30% failure rate and $500 repair cost. (They're PR and financials driven).
They certainly have unlimited money, and have shown they're not afraid to sue, and bully, or reward compliance.
That said, of the companies who market their ethics, Apple seems to actually _do_ more than most.
One of my previous employers worked with Apple a couple of years ago to ensure the stability and security of Apple Arcade, and every employee, even the janitor, needed to sign an extensive NDA to ensure the secrecy of the project.
We kept “the secret” for six (6) months or so, and at one point we almost lose the contract because someone mentioned the project’s name during a phone call, at home, while a family member was listening, and that person told someone else, and that someone apparently talked about it somewhere and Apple found out.
Fortunately, the CEO managed to, somehow, keep the contract.
Even inside Apple, they have to sign NDAs when you need to talk with a different team about a new product, project, feature, etc. I asked about it during recent interviews (January 2022) to join a team that is supposed to have massive cross-team work with the entire engineering organization.
https://www.computerworld.com/article/2468894/cops-raid-home...
- Secrecy does actually get in the way. It's not great when you aren't aware of large features or products that will impact your work.
- Apple hires people who have great product design instincts, and even end-users know Apple's foundational design principles. However, lack of feedback from end-users does occasionally bite them (see: aborted Safari redesign)
- Apple takes QA very seriously and invests in high quality QA engineers. QA is not just "a step in the process" or a marginalized outsourced group.
- It's a culture of accountability, not committees. Every product and feature has a DRI (Directly Responsible Individual). When something isn't working out, the first question is "who's the DRI on this?"
- Most of the organization is highly siloed by function. This allows units to focus entirely on their objectives, which can be productive. But it makes cross-functional collaboration rare. In my opinion, this is why certain aspects of Apple's ecosystem feel disjointed, missing, or don't hang together holistically as well as they could.
Wow. Every company in SF Bay area I worked at stressed that they don't look for who's to blame, but on how to solve the issue.
It doesn't have to have anything to do with blame.
That said, from what I've heard, culture at Apple varies heavily by group and manager. There are plenty of horror stories!
It’s less of making scapegoat to blame, and rather a person who has been empowered to call the shots on their product, and entrusted with knowing their product or domain inside -out like the back of their hand. If something goes wrong, you ask for the RE/DRI so you can get the person who is supposed to have all the answers, well, answer.
From the article-- Of course, when things do go wrong, it's also the DRI who (usually) takes the fall as was the case when Scott Forestall, then iOS senior vice president, was forced to resign after he "refused to sign the letter apologizing" for Apple's infamously error-laden Maps app redesign in 2011.
https://about.gitlab.com/handbook/people-group/directly-resp...
https://www.youtube.com/watch?v=tVq1wgIN62E
This seems to be in contrast to the tenet of blameless postmortems [1] adopted at Google et al. Does this culture lead to blaming said responsible individual, or is the feedback seen as constructive?
[1] https://sre.google/sre-book/postmortem-culture/
While teams may vary, in my experience postmortems seek to understand where in the process a breakdown occurred, rather than fault individuals.
An SRE may be blameless for a bug but they may also be the DRI for getting an incident resolved.
You usually don’t get that type of appointment unless leadership is confident you’re the right person to be successful at it anyway.
They also are very hesitant to allow researchers to publish their breakthroughs which makes recruitment very hard. Although this is changing
https://machinelearning.apple.com/
Just yesterday, though, I said "Hey, Siri, navigate home" and that literally got into a failed state. The first time I did it it told me I didn't have a "Home" contact, so I made it. Didn't work when I was in the car yesterday, but it did just now. So, I agree, there are issues with Siri.
The biggest “WTF” moment was when I was using Siri in Russian, and told it to “lower the volume” (as I’d done hundreds of times in English). It lowered Siri’s volume, but nothing else (and told me as much).
Never did figure out how to adjust my music volume with Siri in Russian.
Just compare the quality of products and failure rate of the competition...
Ex: https://news.ycombinator.com/item?id=31069434
But some of those buzzwords are perfectly appropriate for companies operating at a smaller scale. Apple has the ability to spend a huge amount of time and effort up-front to get the product right before it's ever announced. They're perfectly happy to spend 3 years on a product no one has seen.
Smaller companies [believe they] have to generate value faster, so they push out MVPs with the hope they can find product-market fit and iterate their way to success. For better or worse (often worse), that kind of post-launch iteration is just not in Apple's DNA. Look how little the Apple Watch software has changed in 8 years.
I don't think everyone should spend as long "perfecting" a product as Apple does, but I do think most companies could save a huge amount of time and money by doing more up-front research and design, to make sure they're building the right thing.
Proof: The majority of today's winning or great products were started by technical people. Yet the industry feels the need to systematically discover "the next Steve Jobs" with MBAs, PMs, methodologies and buzzwords to justify the success of the next innovations...
Not denying its usefulness, some of these might help but seems like too much buzz
Willing to bet this is worth 80% of the value to the task, especially if your QA testers can write enough code to have good instincts about why something might break & control for a test environment / create novel test situations.
Hardly.
It is number six on this list: https://en.wikipedia.org/wiki/List_of_largest_companies_by_r...
And does not even appear on this list: https://en.wikipedia.org/wiki/List_of_largest_employers
> they are the most prestigious place to work
Eh, that doesn't seem to be a universally accepted point of view.
A pointless metric given that over the last 5+ years, Google/Apple/Amazon have had and lost that spot back and forth. It's a vanity metric and in terms of the actual business, is an abstraction.
"they are the most prestigious place to work."
By what criteria? If you are talking about by patents and research filed, it would be IBM (which is also the longest running firm at over a century), if its hardware one might be more impressed by SpaceX, if....etc. "Prestige" is another vanity abstraction, and isn't even prone to a metric.
"Their market cap is on it's way to 10 Trillion. There's very little incentive for people to jeopardize this job and give up their stock in the company."
If that were as true as you state, people wouldn't leave for startups at the rate that they do. Startups wouldn't be as common in general if these big 'prestige' firms were such a thing, but the valley only in the last dozen years or so has fallen to that as 'common wisdom' when the actual habits are a lot more mundane -- NDAs and it's a small valley so people who can't keep a secret don't get hired elsewhere. That was the case in the 90s when the big players were HP, IBM, MSFT, Sun, SGI,and folks bounced from firm to firm, it was the case a decade later when folks started moving into what became the dotcom wave........
In addition to the NDAs, Apple tracks employee data very closely (e.g. iCloud) to mitigate leaks.
The major downside to this approach is the sample size cannot be “internet scale” and it will be biased towards what Apple thinks it knows. That’s why things like Apple Maps and Siri had such poor roll-outs.
Intriguing! Could you elaborate more on this?
> Since they want to keep new hardware support a secret, they work on it in parallel on another branch / whatever. So all the betas only include a subset of the changes going into the real release. And then when the -RC drops, they merge major changes, especially around boot/HW.
I don't wish to comment on the "and quality" part in your question though.
The long answer is I don't think either of these statements are necessarily true? Apple products usually leak before their official launch. And as for quality while generally ok... Apple has had some poor launches including: Apple Maps, Safari on Windows, butterfly keyboard mechanisms. And other issues are user unfriendly product design like the mouse that has as charging port on the bottom rendering it unusable. And still more quality issues stem from lack of transparency like Battery Gate.
I personally think Apple has a strong fanbase that overlooks these issues and is quick to forget
Apple has a strong core of two users, in my opinion
- people who couldn't give a shit less about the actual software, the UX, or whatever else it provides, they just know that it can take photos, browse the internet, and that Apple products are associated to "premium" and they want to be premium. This trickles down to the younger generation through their favorite movie stars, singers, tiktok influencers all use Iphones, so they want to use Iphones. The "rich" kids at school have Iphones and if they see you have the green bubble instead of the blue Imessage bubble in group chat, you'll get made fun of. (I just had this exact scenario happen with one of my friend's daughter, who took my old Iphone 8+ off my hands to replace their Galaxy)
- users who have grown to realize all consumer tech is essentially trash, and recognize the marginal utility that Apple products uniquely offer: namely the fact that their products have resaleability, have a pretty well connected ecosystem, and at the end of the day, a physical store near you that you can take your product to.
I am in the latter boat, having used PCs my entire life, modding custom setups, rolling my own linux distros, installing the latest de-googled android OS, logitech, the 40 brands of headphones and headsets, and all of that fun (miserable) stuff. You could tell me right now the latest X1 carbon is 994234% faster than my M1 mac, with a graphics card capable of running the latest AAA games, and I wouldn't give a shit. I just don't give a shit anymore. I want something that will work 99% of the time, and let me do my job wherever. Apple products have done that for me much more often than when I relied on whatever other amalgamations of Dells/Samsungs/Lennovos/Blackberrys/etc, whether its getting frustrated with wifi connectivity, not having my passwords or photos sync across devices, or fumbling with bluetooth headsets and devices from god knows how many vendors and companies. Give me an Iphone, a Mac, some Airpods and I will deal with whatever minor inconveniences like having to charge my mouse upside down.
>You could tell me right now the latest X1 carbon is 994234% faster than my M1 mac, with a graphics card capable of running the latest AAA games, and I wouldn't give a shit.
You could have your cake and eat it too as I hear the M1s are pretty competitive with most PC laptops.
I think a lot of people write off the (G)UI/UX of macOS as toy-like but one thing I've also learned is I like looking at pretty things. Sue me. I like my apps to look nice, I like my UI to look nice, and I like when it all feels connected/consistent. Of course there are outliers but far fewer than when I look over the wall at linux/windows. It really comes down to (for me) "where do you want to spend your time?". For myself I want to spend it producing and doing the things I love, not endlessly configuring or fighting my OS, macOS lets me do that. It's not perfect and I won't pretend it is, but it's leaps and bounds ahead of the competition when it comes to things that matter to me.
One quick anecdote, when my sisters went to college my dad got them Macs (after being a windows-only household growing up and him only starting to using Apple products after I did) and he still tells me how happy he was with that decision. He will say "Do you know how many times I got called to do tech support? Zero, not once". He and I both had the "tech support" role in our friends/family groups so we both knew what a windows computer would have resulted in, we had lived it for other relatives in and out of college, heck I rebuilt a ton of friends computers when I was in college due to virus/malware/slow/etc. The people that would buy a new $400-$500 dollar bargain bin laptop every 1-2 years in college when their previous one fell apart yet would make fun of the cost of Apple computers drove me up a wall seeing how I had the same laptop all 4 years and it cost ~$1K (and it felt great to carry and looked good to boot).
Do I believe that you can game better on windows or customize linux to be your personal nirvana? Absolutely, but neither of those things matter to me, I just want to be able to sit down and produce, not be digging in config files to figure out why my graphics card isn't working.
This is a great point. A couple of my mates have had issues with their laptops and had to talk to the manufacturer, send it across the world at their own cost and wait forever to get it fixed. I've had plenty of issues with my macbook pro and it's not covered by warranty anymore, but the people in the store are happy to fix it for free out of goodwill and the repair is always fast.
That was a feature according to Apple not a bug.
If Apple had a serious hard-on for getting rid of cables as is claimed from everyone's desk they would have done the same thing to those devices. It's much more likely they just wanted to recycle the old Magic Mouse chassis, parts and tooling and this was the only way.
Also control. Here is a story I heard from an iPad launch partner, who got an iPad before it was announced:
The device arrived in a large box with little plungers sticking out to press the buttons and the screen. You couldn't actually see the device inside, so there was no way to know what buttons were on it, what its dimensions or weight were, etc. Furthermore the box had to be kept in a locked room with strict access controls. Anyone who had access to the room had to be under draconian NDA that included severe personal penalties for leaking anything (like enough to financially ruin you). Also everything was watermarked so you couldn't release photos without them knowing exactly where it came from.
The box was taken back and replaced with a standard consumer model after the launch. Most people in the company weren't even aware they were a launch partner until after the announcement.
They haven't had a leak that big ever since, but occasionally things have gotten leaked (specs, components being used, etc) from China who does most of the manufacturing.
Controlling the narrative _requires_ secrecy during development. Every person who joins Apple today is well aware of the culture and how it has contributed to Apple's success. They have every incentive to maintain that secretive culture to continue the success.