That is true, but I am pretty sure this is misattributed. At the time we had two major issues which slowed down loading these sorts of dylibs.
One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur).
The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it).
IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.
"Botched introduction, really? Perhaps Apple could have been a bit more upfront about the state of the tooling (which, of course, they never will be) but I think they did a rather good job considering that they moved development out from "total secrecy" to "community-involved" within two years, all the while shipping new features and improvements to the experience at a fairly decent pace.
That’s the problem. Apple knows by now that their developer community laps up absolutely everything they put out and regularly gets burned. (I’m old enough to have rewritten code for Cocoa Bindings in 2004. It’s the next big thing! And garbage collection in 2008. It will fix everything! And GCD... and...)
For such an enormous change as Swift, they should have switched off the reality distortion field, and gone with a more sober, more academic approach.
This page overstates the issues. Almost all the initial GCD design is still good, we just ended up with a lot more smaller low-core and non-SMP processors than expected.
Also most of the performance issues are with excess parallelism (runnable threads), not concurrency (hiding IO latencies).
Aside from ObjC garbage collection, those things are all still good. Bindings enabled you to create things like complex editable tabular UI without writing any code. And the recommendations about how to use GCD efficiently have changed over the years, but it's still the foundational solution to concurrency on Darwin platforms (including being the basis for the new async/await runtime for Swift).
Yes totally botched. We know because have can compare to Kotlin, a project which started around the same time. Google's migration to Kotlin has been progressing very well with no disaster stories like this one. Kotlin was engineered very carefully, developers love it, it scales well and importantly it can be incrementally added to existing codebases because interop is nearly perfect. So the risk to adopt it is far lower. Sounds like Uber got burned with Swift and now aren't really adopting Kotlin which is understandable but IMO a mistake.
I’m afraid it could easily be the biggest mistake made by Apple over the last decade. The bloatedness and slow build time just aren't justified by the shiny, convenient language features.
Note that some of the dropping platforms advice doesn’t really help here, or only helps coincidentally, because the App Store serves you a device-specific app binary.
Yep, it was to prevent you from accidentally blowing through your cellular data cap. They no longer prevent the download, just warn you that the app is over 200MB.
The biggest engineering disaster Uber had was when it intentionally disabled the automatic safety brakes on their 'self driving' car and killed a person.
I don't know why you're getting downvoted. Making a crappy, slow app is nothing compared to conciously deciding to switch off the built-in autonomous breaking system in lieu of your own(which then doesn't work and kills a person). That's an engineering disaster, not a taxi going to the wrong city block a few times.
Considering that the engineer describes his story as "the biggest engineering disaster I’ve ever had the misfortune of being involved in," I don't think it is relevant to his story that the self-driving pedestrian collision occurred. He isn't claiming--and even from the thread title, I wouldn't assume he is claiming--that this is the biggest engineering disaster ever, or even the biggest engineering disaster Uber faced.
I was working at Uber ATG when Elaine Herzberg was killed. They kept the details secret from the other employees, citing fears of "too many cooks". I recall being offered grief counseling.
Uber disabled Volvo's automatic emergency braking system, which probably would have saved Elaine, to develop their system on public streets. I left the company and took a pay cut to work on ethical robotics instead.
Whoa. They disabled Volvo safety system. That’s nuts. My Toyota 2016 Rav has saved me a couple of times with the auto brakes and loud beeps. It was totally worth the 2k extra for the 7 radars and 6 cameras.
I don’t care about L4 self driving. I really care about a car that is resilient to human mistakes and fatigue. Uber failed at that.
Android was a much easier story, Still had some binary size and method count issues but we were able to overcome them thanks to another brilliant engineer we had in the team in SF who worked on Android.
Computing is always susceptible to shiny object syndrome. Just because it’s new doesn’t make it better. I wonder if things would be different with a more nuanced approach to Swift adoption, knowing what they knew of its limitations already, and being realistic that the language is not a panacea (none are!)
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.
> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
This would scare me too.
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
This isn't sunk cost; the exec chain just has different incentives. They're hired for outcomes. It's their job to make it happen, not make it make sense.
> A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks.
When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.
Uber also has a habit of solving problems by hiring a manager externally and building a brand new team of all new hires for them.
There's some upsides to that, but basically even though you're a W-2 employee, you're effectively a consultant in that arrangement. Your job is not secure.
> > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
> This would scare me too.
For what it's worth a few (10ish) years ago we did this with the J9 compiler at IBM (technically just the backed that was shared between Java, C++, etc). Same idea with simulated annealing. It did end up finding some really strange bugs, but they were bugs that were present regardless, we just hadn't found the right test case for it. We did end up getting some performance out of it, but not nearly as much as you'd expect. It mostly boiled down to running a few extra constant folding passes.
It sounds almost like it is a kind of fuzz-testing then; I'm interested in what percentage of problems uncovered were problems in the spec or undefined behavior, versus problems introduced by the simulated annealing itself.
Yep, and it's why Twitter as a blogging platform will never take off. I don't understand why people do that. Just find some place to blog and post a link to that in your twitter feed. Gah! It's so frustrating when people abuse a platform like that...
I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it.
Problem here with Swift came down to
- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)
- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)
Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.
Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
Appreciate your insight, and I'm sorry to hear about the AMS team.
The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".
Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.
Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
I was not part of the initial analysis team so cannot really comment if objc was more suitable as it would require to look at objc at that time and do tests etc but I suspect we would have had a more smoother experience. Though the the code at the time was bad and indeed something needed to happen to ensure future scalability of the teams and product/code, but I think there were more sane paths we could have taken. It would have cost the company more time/money and at that time the company was battling Grab, GoJek, didi, Lyft etc so we did not have time to lose. Lots of factors to take in to consideration. No real easy answer
I havent dabbled in mobile developer platform tooling for the last few years now so I cannot comment on the iOS side, all i remember is that back then it was a shitshow :) Mobile is still playing catch up to how backend infra and tooling operates, I suspect its better but probably still got a long way to go.
We could of made it work. I think the one big benefit of the rewrite was how the architecture and nullable types basically gave us one more 9 of crash free rate, which we saw after the release with swift. With ObjC I could forsee how it would of crashed a bit more, because nullables aren't hard enforced in ObjC, but I could also see how we could of stepped around that by strong guarantees in the model & network service generation code.
A complete rewrite of a large project is risky anyway. Doing so in a language in which literally no other project of that size exists yet strikes me as insane.
I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language innovations and better familiarity with the idioms have helped a lot with this.
One thing I was surprised not to see was a mention of their Swift 2 to 3 migration, which was terrible in my experience working on apps that were nowhere near the complexity of Uber.
Apple should have never pushed Swift as hard as it did at the time. Between the constant API changes and issues like requiring dynamic framework linkage and copying the whole runtime into every app, it was absolutely not production ready.
It was a lot of work, but we had enough staff to make it someones job for half a year or a quarter, depending on how bad the migration is. With big codebases there are other issues that pop up with every new major Xcode release that we have to migrate, mitigate or fix. For example we are still at xcode 11 because of various bugs in xcode 12. Doing the migration was annoying, but it was not much compared to all the other issues, because the fixes were tractable.
The Mac CI situation has gotten much worse this year, but Swift has gotten much better. Writing a large app in Swift is still a major undertaking, but there are people doing it these days.
As someone who was there, a rewrite was necessary, and with hindsight we should of done it with objective-c. The original obj-c app was built with under 10 iOS engineers in mind, and now Uber had 100s of mobile engineers work on one app in some form or another.
We didn't do a simple build scalability test until we were well into the project. If we did it would of revealed swift's build problems to us. Our swift experience is what slowed down our kotlin migration significantly. Today Uber android is still a majority Java app.
Even very recently the experience of working in swift compared to the old obj-c code base isn't as good. To this day there are debugger issues, xcode responsiveness issues and a slower build time.
Hiring on the other hand would be harder now, because it's getting hard to find people who know Objective-C and it's surface ugliness scares new people away. Swift is a decent language if you don't have to have a very large code base, so most people rightfully so don't have that much Objective-C, and if you were to work in Obj-C, you might as well go work at Facebook then.
You can enjoy coffee without grinding your own beans.
I personally find lower level programing to be frustrating and difficult. I've built an amazing career with JavaScript and C# primarily. My hobbyist projects use C# , JS and Flutter.
I can't wrap my head around things like pointers or memory management
post-ARC, Objective-C and Swift aren't very different in terms of memory management. retain and release are handled for you, and you only need to worry about retain cycles. structs in Swift change things a bit, but not that much
With swift & obj-c your coding against the same library APIs, so in many ways it's a very similar level of programming. In many ways swift is far more complicated than Obj-C. With Obj-C iOS apps your not really writing that much C vs. the pseudo-smalltalk attachment to it.
I was an iOS engineer at Airbnb when we were battling binary size increases caused by Swift. I rewrote the watch app twice; once to migrate to watchOS 2 and once to convert it all to ObjC (which saved ~9mb). While Swift does have its problems (and had a lot more back when we worked on this), there are many factors to why the Uber app (and Airbnb) is getting so large.
Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.
Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.
While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!
Parts of the binary on the App Store are encrypted so we can only get limited information from it, but the whole app bundle is available and the rest of the files are not encrypted. We’re building a startup to help monitor, analyze and reduce app size and I was able to quickly grab these statistics from the tool we’re working on, here’s a link to a quick demo: https://www.youtube.com/watch?v=3x-hLxxUNm8
Looking at the bundle now is not going to give you insight into the problems in the Uber app 4 years ago. Also you’re looking at the uncompressed install size, which was not the problem (over the air compressed download size was the limiting factor). Strings files are extremely compressible, encrypted binaries are not (Apple has done some work in subsequent years to make them more compressible but back then they were basically uncompressible).
Absolutely right! I left out some details to try to keep the post concise. 4 years ago at Airbnb we had the watch app problem, but now a Swift watch app wouldn’t be an issue since the latest OSs don’t require embedded Swift standard libraries. The tweet that started this thread cited the 300+mb size of Uber’s app, which looks like the current install size (not thinned), that’s why I used it as the benchmark in my explanation.
Lately Apple has been featuring size more prominently on the App Store, and it appears to be displaying the install size. This is also what users see when they look at which apps are taking up storage on their device in Settings. I think it’s important to reduce both, so users see a good number and downloads are fast/efficient. It is a very welcome change that binary encryption doesn’t negate all the gains from compression anymore. I had noticed it in the numbers but looked around for an official statement and didn’t find any, if you happen to know where this was announced I’d appreciate a link!
Yes, installed size is also an issue. And on iOS tens of thousands of small files will waste a lot of space. One may say that compared to photos and videos that's small potatoes but I disagree. Efficient use of resources, memory, bandwidth, battery, etc. matters.
I was the person that actually managed the binary size task force and proved that it was a business problem in the first place. Not all management didn’t care about AMS :)
The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.
Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here.
Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
Management had very different incentives while engineers had different motives. The two were not aligned, and TK culture was still very prominent back then (work hard, dont go home early, we are going to be a 1 trillion dollar company etc etc) and Thuan definitely helped him spread that culture. So definitely not great days but I think the engineers took away a lot from their time there. Lots of lessons learned
Yeah, I imagine this pressure from the top had something to do with the "90 hour weeks" cited in the thread...I've never worked in an org as large as Uber, but personally I have made some very bad engineering choices when in crunch mode, especially on the doubling-down-on-bad-decisions front.
When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
When you write code like shit, the language doesn't matter. What Uber did was write a monstrosity with too many engineers and too many features without considering that was beyond their ability to deliver reliably. The product decision making was also clearly deficient and didn't help.
Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
It can be quality code written by good engineers, but there are issues you won't see until you deal with big numbers. The problem here is that Swift seemed to be a cool language that it might work for a "Hello world" app. You won't notice any problem until the thing grows up to a certain point.
For example, I work in embedded and I am proposed with a new "X brand" Chinese Modem to integrate in our devices. I don't recommend it but since it costs less I'm pushed to evaluate it. I am handed 3 samples and they all work fine, so management convinced themselves that Brand X from China is the way to go.
Once in production the modems started to fail at a 10% rate (10 devices over 100 won't work). And that is a problem you only see only when manufacturing a certain quantity of devices.
Now there are delays, an ongoing battle with the manufacturer of modems and a unhappy customer (that's why I didn't recommended it).
We started with Swift 2.X as well but it became Swift 3 before we shipped anything; Swift 5.X now is much more mature and we have no issues. Trying to do too much with a new language plus all the other things was asking for trouble.
Great article, thank you for sharing this. Makes a good case that the right thing to do, from a business sense, is usually not to do the full rewrite. Rather, you should do slow, un-sexy work of refactoring and cleaning up while adding new and maintaining current features.
He's right on, but maybe people don't actually know what to do on those cases
A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.
Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first
Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.
Couple things always to mind whenever this article is referenced.
Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it?
Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).
I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?
I don't think you can consider Windows NT to be a rewrite of DOS. They were different products entirely, and existed in the marketplace concurrently for upwards of a decade.
Microsoft took their entire userbase and successfully shifted them from DOS (and then Windows on DOS) to NT. They did it in a piecemeal fashion with e.g. Windows 95 and WIN32.
It's a completely different system, that happens to support the same API/ABI, backed by long experience at Microsoft at making things backwards compatible (it's probably the biggest source of Microsoft's success).
Microsoft also put ~7 years into preparing the transition piece by piece, with probably the biggest impact point being Windows 2000 that provided significant host of features that compelled big (and loud in complaints) clients over, pulling a lot of software vendors along.
Mozilla eventually beat IE only because Microsoft disbanded the IE team. The years of product stagnation due to the Mozilla/Firefox rewrite didn't matter in the end because AOL acted as a sugar daddy for a while and then IE was also not moving forward. The Spolski advice is for the more typical situation where your competitors are advancing.
The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point.
Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"
Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
Yeah it’s kind of bizarre that after writing everything leading up to that, the author would still frame the ObjC folks in that way. Like, this or that eng. was so brilliant with their hack, but all those obj-c eng. who advocated boring but stable solutions all along were just curmudgeons with no helpful advice.
The problem from the author's perspective is that the Obj-C "curmudgeons" were proving the right solution to save the app (drop Swift and cut losses + go back to a functioning language) but not the careers of the people who had gotten them into this terrible situation.
This perspective is reflexive. The "ObjC curmudgeons" didn't feel like the adventures that the "Swift zealots" were generating helped their careers in any way.
We won't know because all we have is one glib comment from one person, but I took the comment on face value. I've definitely met the class of developer who is mainly interested in pointing out problems and a stick in the mud, and isnt interested in working together on solutions.
Wasn't really clear what benefit the rewrite offered anyway other than a new UI, which brought it's own problems and couldn't that have just been written in Obj-C?
In my experience, the problem with scenarios like fight the ObjC folks in the story is that the "curmudgeons" don't see the need for anything to change. Their arguments are not "We should use ObjC and do X, Y, and Z to make sure the problems are fixed," but rather usually turn out to be something like "We should use ObjC (and throw out the rewrite entirely; everything was fine before)." Which at best only results in kicking the problem down the street a ways and at worst is not possible.
The emphasis could be on their behavior of 'complaining without providing much in the way of solutions'. Certainly, though obvious now, they could have empirically presented at least one legitimate case as to why Obj-C was a better choice (app size).
Indeed the ObjC people were right (I was one of them). But there is nothing more useless than standing in the corner and saying “I told you so”. That’s the curmudgeon.
It was pretty obvious when Swift 1.0 launched that it should’ve been called Swift 1.0 beta 1. Not a knock against anyone who worked on the project, other than senior management at Apple.
Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.
I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
Large Swift apps are easy to find; companies would invariably use it as a marketing point on their engineering blog. Even if not, people open up apps all the time, any large app that uses Swift would be found and discussed.
My point was that there was no need to ask Chris Lattner back in 2016: you could just look around the landscape and see that there were not many, as they would be easy to see if they did exist.
As I mentioned, there is no need for a company to publicly post details of their products for people to figure out what languages they were written in.
Uber app contains very similar functionality, namely real-time map view that needs to be able to render any part of the globe (or at least serviceable area), a location search engine, the ability to render drivers around you in real-time.
Just like Google Maps lets you select point A-B directions across cars, walking, public transit, Uber gives you multiple products to select (UberX, UberPOOL), and renders the path on the app accordingly.
The Uber app also does a lot of extra computation at the app side to solve some of the business problems that the OP mentioned in the thread. For example (from the Twitter thread):
> Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience.
Uber app also tries to resolve and render points-of-interest, so that you don't have to search for just an address as your destination, you can also search for "Dolores Park". That not only powers the search on the backend, it's also rendered on the frontend so that when the map tile for Dolores Park is rendered, the app knows to drop a tree-shaped icon on top of it labeled "Dolores Park". Uber has been using their own maps for a lot of this, and has an entire Maps org that does in-house map display, routing, and search.
And that's just the before-trip functionality. There's extra stuff that happens during the trip, including showing your current location, being able to share your trip with friends, real-time 911 calling for safety, etc etc.
And this is all just the similarities with Google Maps. Uber does extra stuff on top of that, including:
Onboarding flows for signup/signin, including the 3rd party identity services.
Payments: the app needs to support payment linking flows so that you can add and verify your credit card. Sometimes your credit card declines, and there's a whole arrears flow that needs to be built out to support that. And credit cards aren't the only way to pay for trips in Uber, you also have proprietary systems like Venmo, and PayPal that you can link. And on top of that, there are myriad such proprietary wallets all over the world that Uber has supported since at least 2015, including PayTM (in India) and Airtel Money (India and Africa), just to name a couple examples. And in most of the world outside of the US, you can pay for Uber trips with paper cash, and that's a whole flow on the frontend during onboarding / arrears settlement. This needs to all be packed into the same binary because you don't download a different version of the app when you go to a different country.
They also bundle UberEats into the app for some people, and there are probably hundreds of other random features that are prototyped and hidden behind feature flags, or variable experiences that are selectively rolled out in certain geographies. They all need to be packaged into the same binary for everyone.
Throw in other random product features like their Spotify integration, and it all adds up. Uber isn't just a dumb CRUD app.
You can also see the same in other comparable apps, Lyft's iOS app is 290 MB, Ola is 205.3 MB, Grab is 289.6 MB, GoJek is 236.2 MB, Yandex Go is 145.4 MB, etc etc.
EDIT: there's an even better written summary somewhere up-thread of the sheer amount of business complexity that needs to be baked into the app -> https://news.ycombinator.com/item?id=25376346
> So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.
This is a good point, but it's important to be clear about it. There is no universally superior language. There are a number of competing factors which means that optimizing for one sacrifices others. So instead of a single point in space that maximizes all of them, there is instead a volume bounded by a curve and going farther in one direction requires lower values on other axes.
But, there is a lot of space nowhere near that trade-off optimizing boundary. You can absolutely make languages worse in ways that are strictly negative for all users. For historical reasons, all languages contain some amount of that dead weight. So the absense of a perfect language does not imply that all existing languages are right at the edge of the optimizing boundary.
I think not clearly understanding the tradeoffs was a huge factor here, but I would go one step deeper into that decision-making process.
One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and took a cursory look at build times for a small pilot, but failed to look deep enough to find some of the crippling shortcomings.
Related to the first point, a lot of the time with a new technology, the show stopper bugs/shortcomings/issues haven't been discovered yet. If you're choosing a battle-tested technology, most of the nasty edge cases and limitations have been discovered at some point. But with something new, there's a lot more uncertainty with what you'll run into once you put it into production.
That risk of surprises is higher with some new tools and technologies than others. But it's a big wildcard when you're betting your billion dollar business on it. Maybe that bet pays off, but you need to do some sober analysis on that risk of unknown unknowns and have some idea of how and when you'd make the decision to cut your losses.
Rewriting apps (esp. as large as Uber) is essentially your plan meeting the enemy. There's many unknown-unknowns simply because technology is complex, and although it's pretty much unprovable, the brisk pace of technology also means you'll have more unknown-unknowns. Only a big industry shift in priorities will change that now.
It hints at a rule-of-thumb I use: try to wait making an important technical choice until you can explain a pro & con of each option (or, similarly, explain when you'd use each one instead of the other).
One company I worked for a few years ago, I joined right as a just such a ground-up rewrite of their core system was coming into place. Everyone in orientation was telling me "You'll be spending most of your working life dealing with <new project codename>" but they were actually training me on the existing system. About a month out of orientation there is a company wide. They canceled <new project codename> because it wasn't meeting it's technical specs.
It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
I think it was clear early on that Swift is the future of iOS development. But if you’ve been around long enough, you’ll have observed that it takes about 10 years for a new language and associated tooling to reach “maturity”. What “maturity” means is debatable, but one aspect is that the ecosystem grows to handle the extreme edge cases like very large scale development. And boy was Uber pushing the edges of what was possible on iOS at the time.
So by my way of thinking, they moved at least a couple years too early. (The Swift project started internally in 2010, first public release in 2014, Uber’s rewrite started in 2016.)
It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that.
I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?
I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same. People saw the issues that were placed on a national stage, then looked inwards and saw similar issues much closer to home, and these things started coming out as people were no longer OK with keeping quiet about it.
For the other half of your comment: I personally think many companies have issues focusing on non-technical problems. From the thread, the issues were clearly that they were writing far too much code, it seems like they were not testing enough, they lacked a person who would tell them how their changes would be perceived by the public. This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others. And you can see the solutions: while highly technical and "awesome", they are obvious band-aids. It's just that it's much easier to slap on technical band-aids than fix the root, non-technical problem.
> I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same.
> I personally think many companies have issues focusing on non-technical problems...This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others.
I agree with all that. To me, this is about taking responsibility. Our industry has shown a pattern of thinking poorly about non-technical problems while rolling out changes that affect millions of people. That's part of why so many people are angry at the tech industry -- and for good reason.
The thread is thoughtful in a lot of ways, but I get a whiff of not-taking-responsibility through some of it. The first example struck me for whatever reason. I get that people in 2016 didn't necessarily know what was going on or how bad it was. But I would expect that in 2020, we'd look back at 2016 as the time when serious issues beneath the surface were about to explode as a result of choices made at Uber, not some golden time before external forces made life hard for Uber. I don't mean to be too hard on the author or even Uber (and after all, this was a casual tweet thread). I think it's a widespread challenge for our culture in the tech industry.
Yes, I think that many engineers can often have a hard time understanding that problems that occur at their workplace don't exist because the media changed attitudes but because it shone a light at internal issues. I remember talking to someone at Facebook privacy and they were somewhat indignant that the media was making them out for being such a privacy nightmare–ignoring the fact, of course, that the tide was turning against surveillance and data gathering in general and Facebook was just one of the most prominent examples of what everyone hated ;)
> There’s a video somewhere in one of our talks of an Uber engineer typing a single line statement in Xcode and then waiting 45 seconds for the letter to appear in the editor slowly, one-by-one.
I'm not sure if you are familiar with Xcode, but this indicates an off-the-charts LOC count. Something isn't adding up -- the user-visible features and screens in the app don't necessitate a codebase this large.
My guess would be the intersection of several compounding factors:
- a product team endlessly pushing special-case features which aren't core to the user experience (the twitter thread talks about the rate of new code being added as if it were a foregone conclusion the app would continue to grow without bound),
- an A/B testing framework which leads to the deployed codesize being much larger than what the typical user actually interacts with (worse if they are lax about culling the vestigial A's and B's),
- and reaching a "thermal runaway" point of too many devs x too many lines of code, where code reuse stops happening. Once you reach the point where it takes less time for a dev to write a given feature in a from-scratch fashion rather than doing some codebase archaeology to find existing code structures which can be reused, you've reached thermal runaway. The Facebook iOS app revelation of "18,000 classes" indicates they were likely suffering from this as well https://news.ycombinator.com/item?id=10066338
Was Swift up to the task? Perhaps not. But does a ride-hailing app really necessitate this level of codebase complexity? Uber's iOS app is now over 300MB...
It’s funny you quote that specific line because that one line stood out to me too. I agree that something extremely fishy was going on that couldn’t have been solely explained by “too many linked libraries”.
However, choosing a more mature technology like Java or C# or even Golang would have certainly alleviated a great deal of stress from their process.
341 comments
[ 2.2 ms ] story [ 338 ms ] threadI really loved it and continue to from v3 onwards though.
One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur).
The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it).
IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.
That’s the problem. Apple knows by now that their developer community laps up absolutely everything they put out and regularly gets burned. (I’m old enough to have rewritten code for Cocoa Bindings in 2004. It’s the next big thing! And garbage collection in 2008. It will fix everything! And GCD... and...)
For such an enormous change as Swift, they should have switched off the reality distortion field, and gone with a more sober, more academic approach.
Here's one account: https://tclementdev.com/posts/what_went_wrong_with_the_libdi...
Also most of the performance issues are with excess parallelism (runnable threads), not concurrency (hiding IO latencies).
https://www.wandera.com/ios-13-app-download-cap-removed/
Uber disabled Volvo's automatic emergency braking system, which probably would have saved Elaine, to develop their system on public streets. I left the company and took a pay cut to work on ethical robotics instead.
https://www.washingtonpost.com/local/trafficandcommuting/ped...
I don’t care about L4 self driving. I really care about a car that is resilient to human mistakes and fatigue. Uber failed at that.
On recent Android devices, the APK wil be recompiled to machine code on the device after a small number of runs (to gather code statistics).
So you are comparing an apple and a robot.
(disclaimer: this is me)
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.
> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
This would scare me too.
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
> A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks.
When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.
> This would scare me too.
FWIW, I made it clear that this was also a great way to break the app :)
There's some upsides to that, but basically even though you're a W-2 employee, you're effectively a consultant in that arrangement. Your job is not secure.
This is a damn cool use of simulated annealing.
> This would scare me too.
For what it's worth a few (10ish) years ago we did this with the J9 compiler at IBM (technically just the backed that was shared between Java, C++, etc). Same idea with simulated annealing. It did end up finding some really strange bugs, but they were bugs that were present regardless, we just hadn't found the right test case for it. We did end up getting some performance out of it, but not nearly as much as you'd expect. It mostly boiled down to running a few extra constant folding passes.
I just supposed that this was a story about Uber Elevate ElectricVTOL (eCRM) design.[0,1]
[0] https://twitter.com/TheWoracle/status/995107989886353408
[1] https://github.com/OpenVSP/OpenVSP/tree/main/examples/vspaer...
Problem here with Swift came down to
- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)
- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)
Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.
Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".
Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.
Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
I havent dabbled in mobile developer platform tooling for the last few years now so I cannot comment on the iOS side, all i remember is that back then it was a shitshow :) Mobile is still playing catch up to how backend infra and tooling operates, I suspect its better but probably still got a long way to go.
I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language innovations and better familiarity with the idioms have helped a lot with this.
Apple should have never pushed Swift as hard as it did at the time. Between the constant API changes and issues like requiring dynamic framework linkage and copying the whole runtime into every app, it was absolutely not production ready.
We didn't do a simple build scalability test until we were well into the project. If we did it would of revealed swift's build problems to us. Our swift experience is what slowed down our kotlin migration significantly. Today Uber android is still a majority Java app.
Even very recently the experience of working in swift compared to the old obj-c code base isn't as good. To this day there are debugger issues, xcode responsiveness issues and a slower build time.
Hiring on the other hand would be harder now, because it's getting hard to find people who know Objective-C and it's surface ugliness scares new people away. Swift is a decent language if you don't have to have a very large code base, so most people rightfully so don't have that much Objective-C, and if you were to work in Obj-C, you might as well go work at Facebook then.
I hate objective C with a passion. It's like some archaic language of a bygone era forced to run on a cell phone
I personally find lower level programing to be frustrating and difficult. I've built an amazing career with JavaScript and C# primarily. My hobbyist projects use C# , JS and Flutter.
I can't wrap my head around things like pointers or memory management
With swift & obj-c your coding against the same library APIs, so in many ways it's a very similar level of programming. In many ways swift is far more complicated than Obj-C. With Obj-C iOS apps your not really writing that much C vs. the pseudo-smalltalk attachment to it.
Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.
Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.
While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!
Lately Apple has been featuring size more prominently on the App Store, and it appears to be displaying the install size. This is also what users see when they look at which apps are taking up storage on their device in Settings. I think it’s important to reduce both, so users see a good number and downloads are fast/efficient. It is a very welcome change that binary encryption doesn’t negate all the gains from compression anymore. I had noticed it in the numbers but looked around for an official statement and didn’t find any, if you happen to know where this was announced I’d appreciate a link!
The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.
Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.
Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
For example, I work in embedded and I am proposed with a new "X brand" Chinese Modem to integrate in our devices. I don't recommend it but since it costs less I'm pushed to evaluate it. I am handed 3 samples and they all work fine, so management convinced themselves that Brand X from China is the way to go.
Once in production the modems started to fail at a 10% rate (10 devices over 100 won't work). And that is a problem you only see only when manufacturing a certain quantity of devices.
Now there are delays, an ongoing battle with the manufacturer of modems and a unhappy customer (that's why I didn't recommended it).
A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.
Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first
Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.
Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it?
Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).
I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?
What else you would call it?
It's a completely different system, that happens to support the same API/ABI, backed by long experience at Microsoft at making things backwards compatible (it's probably the biggest source of Microsoft's success).
Microsoft also put ~7 years into preparing the transition piece by piece, with probably the biggest impact point being Windows 2000 that provided significant host of features that compelled big (and loud in complaints) clients over, pulling a lot of software vendors along.
Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"
Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
Maybe as a junior engineer, or an intermediate one. But not beyond that. Seniority isn't just being right more often.
Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.
I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
Sure, if they exist. I'm not talking about the world as it is today in December 2020 with Swift 5.3. I'm talking about 2016.
Just like Google Maps lets you select point A-B directions across cars, walking, public transit, Uber gives you multiple products to select (UberX, UberPOOL), and renders the path on the app accordingly.
The Uber app also does a lot of extra computation at the app side to solve some of the business problems that the OP mentioned in the thread. For example (from the Twitter thread):
> Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience.
Uber's approach to solving this involves some novel computation that adds to the binary size -> https://eng.uber.com/rethinking-gps/
Uber app also tries to resolve and render points-of-interest, so that you don't have to search for just an address as your destination, you can also search for "Dolores Park". That not only powers the search on the backend, it's also rendered on the frontend so that when the map tile for Dolores Park is rendered, the app knows to drop a tree-shaped icon on top of it labeled "Dolores Park". Uber has been using their own maps for a lot of this, and has an entire Maps org that does in-house map display, routing, and search.
And that's just the before-trip functionality. There's extra stuff that happens during the trip, including showing your current location, being able to share your trip with friends, real-time 911 calling for safety, etc etc.
And this is all just the similarities with Google Maps. Uber does extra stuff on top of that, including:
Onboarding flows for signup/signin, including the 3rd party identity services.
Payments: the app needs to support payment linking flows so that you can add and verify your credit card. Sometimes your credit card declines, and there's a whole arrears flow that needs to be built out to support that. And credit cards aren't the only way to pay for trips in Uber, you also have proprietary systems like Venmo, and PayPal that you can link. And on top of that, there are myriad such proprietary wallets all over the world that Uber has supported since at least 2015, including PayTM (in India) and Airtel Money (India and Africa), just to name a couple examples. And in most of the world outside of the US, you can pay for Uber trips with paper cash, and that's a whole flow on the frontend during onboarding / arrears settlement. This needs to all be packed into the same binary because you don't download a different version of the app when you go to a different country.
They also bundle UberEats into the app for some people, and there are probably hundreds of other random features that are prototyped and hidden behind feature flags, or variable experiences that are selectively rolled out in certain geographies. They all need to be packaged into the same binary for everyone.
Throw in other random product features like their Spotify integration, and it all adds up. Uber isn't just a dumb CRUD app.
You can also see the same in other comparable apps, Lyft's iOS app is 290 MB, Ola is 205.3 MB, Grab is 289.6 MB, GoJek is 236.2 MB, Yandex Go is 145.4 MB, etc etc.
EDIT: there's an even better written summary somewhere up-thread of the sheer amount of business complexity that needs to be baked into the app -> https://news.ycombinator.com/item?id=25376346
I really like this point.
But, there is a lot of space nowhere near that trade-off optimizing boundary. You can absolutely make languages worse in ways that are strictly negative for all users. For historical reasons, all languages contain some amount of that dead weight. So the absense of a perfect language does not imply that all existing languages are right at the edge of the optimizing boundary.
One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and took a cursory look at build times for a small pilot, but failed to look deep enough to find some of the crippling shortcomings.
Related to the first point, a lot of the time with a new technology, the show stopper bugs/shortcomings/issues haven't been discovered yet. If you're choosing a battle-tested technology, most of the nasty edge cases and limitations have been discovered at some point. But with something new, there's a lot more uncertainty with what you'll run into once you put it into production.
That risk of surprises is higher with some new tools and technologies than others. But it's a big wildcard when you're betting your billion dollar business on it. Maybe that bet pays off, but you need to do some sober analysis on that risk of unknown unknowns and have some idea of how and when you'd make the decision to cut your losses.
It hints at a rule-of-thumb I use: try to wait making an important technical choice until you can explain a pro & con of each option (or, similarly, explain when you'd use each one instead of the other).
It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
Was the move to Swift inevitable and they started too early or should they have stayed with - and still be using - Objective-C?
So by my way of thinking, they moved at least a couple years too early. (The Swift project started internally in 2010, first public release in 2014, Uber’s rewrite started in 2016.)
It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that.
I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?
For the other half of your comment: I personally think many companies have issues focusing on non-technical problems. From the thread, the issues were clearly that they were writing far too much code, it seems like they were not testing enough, they lacked a person who would tell them how their changes would be perceived by the public. This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others. And you can see the solutions: while highly technical and "awesome", they are obvious band-aids. It's just that it's much easier to slap on technical band-aids than fix the root, non-technical problem.
> I personally think many companies have issues focusing on non-technical problems...This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others.
I agree with all that. To me, this is about taking responsibility. Our industry has shown a pattern of thinking poorly about non-technical problems while rolling out changes that affect millions of people. That's part of why so many people are angry at the tech industry -- and for good reason.
The thread is thoughtful in a lot of ways, but I get a whiff of not-taking-responsibility through some of it. The first example struck me for whatever reason. I get that people in 2016 didn't necessarily know what was going on or how bad it was. But I would expect that in 2020, we'd look back at 2016 as the time when serious issues beneath the surface were about to explode as a result of choices made at Uber, not some golden time before external forces made life hard for Uber. I don't mean to be too hard on the author or even Uber (and after all, this was a casual tweet thread). I think it's a widespread challenge for our culture in the tech industry.
I'm not sure if you are familiar with Xcode, but this indicates an off-the-charts LOC count. Something isn't adding up -- the user-visible features and screens in the app don't necessitate a codebase this large.
My guess would be the intersection of several compounding factors:
- a product team endlessly pushing special-case features which aren't core to the user experience (the twitter thread talks about the rate of new code being added as if it were a foregone conclusion the app would continue to grow without bound),
- an A/B testing framework which leads to the deployed codesize being much larger than what the typical user actually interacts with (worse if they are lax about culling the vestigial A's and B's),
- and reaching a "thermal runaway" point of too many devs x too many lines of code, where code reuse stops happening. Once you reach the point where it takes less time for a dev to write a given feature in a from-scratch fashion rather than doing some codebase archaeology to find existing code structures which can be reused, you've reached thermal runaway. The Facebook iOS app revelation of "18,000 classes" indicates they were likely suffering from this as well https://news.ycombinator.com/item?id=10066338
Was Swift up to the task? Perhaps not. But does a ride-hailing app really necessitate this level of codebase complexity? Uber's iOS app is now over 300MB...
However, choosing a more mature technology like Java or C# or even Golang would have certainly alleviated a great deal of stress from their process.