Was just kidding, but in all seriousness, the Mini has slightly better performance and cooling (not by some huge margin), and it's easier to connect accessories. But yeah, having a battery might be good for some use cases.
Basically ran a macro to adjust how far restaurants would deliver by time of day / day of week.
The goal was to balance supply and demand of restaurants and orders.
Eats product didn't have any demand shifting levers at that time yet restaurants were getting blown up with demand at peak times (ex. top pizza restos at 7pm on a Friday).
The product did however allow ops teams to manually set a delivery radius for a given restaurant.
Clever ops teams scripted shrinking / expanding this delivery radius to encourage demand at non-peak times and reduce it during period where restaurants were overloaded and likely to ignore / cancel orders.
This laptop ran scripts for all cities in Canada to adjust delivery radii for each restaurant until a few different levers were built into product.
Ian (who is a _super nice_ guy) owned this process which kept things working smoothly while Eats was experiencing explosive growth.
Another fun fact - Eats Marketplace, which is the current Eats you see today, was built and launched in Toronto - SF engineers / product / design supported by Toronto ops teams. Quickly eclipsed the Instant Eats product which was the 3rd (and most successful to date) try by Uber at food delivery.
lmao... figured it was radius reduction or somesuch. i ended up working on Winterfell (the service that managed automated radius reduction) at some point during my time there.
This was obviously temporary but is actually cool for showing how powerful modern machines actually are. People routinely underestimate the load that can be handled by one box properly configured and running efficient code.
I’ve found that modern developers almost always overestimate the amount of physical hardware needed to run things. I’m not sure why.
I was once interviewed by Lyft and I told them that a 64 GB Mac Book Pro is sufficient to run the core business.
The interviewer didn't believe me.
I did the calculations in front of him.
This was the time, I just learned Rust.
I did the full analysis and showed him that QPS is great.
Of course, my design didn't include 500 microservices, 50 deployments of Hadoop, Kafka, Elastic, Mongo DB, and 500 slightly different dashboards that different teams in Lyft probably use. Most modern tooling in ZIRP phenomenen.
How do you define "core business"? Any laptop os would crumple purely by running out of resource handles on the connections, even without doing work. I've run into file descriptor / handle size limits on small clusters before, I can't imagine trying to deal with 1m+ concurrent tcp connections.
MacOs is a terrible server OS, unless you want to YOLO lack of synflood protections. That said, if 64GB macbookpro means 64GB of ram, that's enough for a lot more than 1M file descriptors. I'd still tend to pick other hardware; I don't see the sense in buying Apple hardware to run a different OS; especially as they move farther away from mainstream hardware.
The Mac Studio running Linux seems like it'd actually make a pretty decent server. Not the most cost-effective option but not terrible when you also factor in power consumption.
Apple should really license the M architecture for servers. They could even have someone else manufacture and deal with delivering it. They're leaving free money on the table. I don't think it would cannibalize their Mac sales at all.
"core bussiness" is finding and dispatcching drivers including payments. Everything in-app.
> I can't imagine trying to deal with 1m+ concurrent tcp connections.
2 million connections on a BSD mchine is doable though not trivial and that too even in 2012.
https://blog.whatsapp.com/1-million-is-so-2011
You'd think that with all the advances in hardware, computers would get faster. No, this modern computer takes 20x longer to open the calculator then windows NT
You should probably also keep your very expensive and sensitive space equipment at a safe distance from any microwave too, what I was trying to say I guess
Ouch. German ATC software in the 90s seems to have been a special place. (My favorite war story is "Hey, we have somebody 10 miles out from FRA airport, low on fuel, ATC software is down, can we escalate" - yes, yes we can.)
I mean if the options you’re facing are Uber Eats Canada running because of something happening on a laptop or Uber Eats Canada not running, it’s not a hard choice to make.
Given that Uber Eats is losing money on every order the evaluation might actually be that high because, not despite of the laptop given that it can thankfully only do so many orders at any given time
Legality aside I’d love how many people you could serve with $2000 worth of hardware (used or new) and a 1gbit up/down home internet connection With max p99 latency of 500ms.
It is strange that they didn’t at least use a Mac mini, but I assume they already had the hardware.
The problem with a laptop is terrible cooling that you can't sustain high performance. They say new Macbooks are better in that aspect but didn't have a chance to try one.
I ran one of the popular web sites in Turkey on a AMD Duron 600Mhz 512MB RAM server until 2003. It used to serve 70 million pages per month on a Windows Server when it was retired. Because the site is a social platform, most of the content had to be dynamically generated too. I believe the PC had cost me around $500 when I bought it in 2000 or so, obviously a lower spec than even a Raspberry Pi4 today.
Exactly. Developers seems to be hell bent on making software as slow as possible and/or maximising the cost of running servers. I bet that 99% of businesses out there could comfortable run on a single server with lots of RAM. With another servers on warm stand-by.
It's more fun to try and figure out where the bottleneck would be.
Back in the early 00s I hosted websites on my local off the shelf Dell desktop on a cable internet connection.
Cloud computing wasn't really a thing and I didn't know better / have much better options (eventually I did migrate to some webhost bundled with the DNS registar).
For the longest time network bandwidth was my limiting factor. I remember finally upgrading my connection, NIC, etc. to the point where it was actually harddrive write speed that was the bottleneck.
Probably depends on your application but harddrive I/O does seem like a likely limiting factor to me today.
Hardware wise, kinda depending on the characteristics of the app. Network wise, I've seen this tried. Commercial internet, even if it is symmetrical 1Gibps has nearly no SLA tied to it and is subject to traffic shaping. Not only could you be turned off at the companies whim if you began to saturate the points up to your demarcation point, but they could also traffic shape you for a time (or indefinitely). This all largely has to do with how internet telephone companies and cable companies categorize and pool traffic in their regions up to backbones.
Also save for the fact that commodity hardware lacks a lot of the hardware safeties a server does. For instance, your m.2 under sustained load would not perform at the same rate that an enterprise server hard drive can; we're talking on an order of 180MiB/s vs 550MiB/s per drive. Once the m.2's cache is exhausted it will be comparable to a spinning disk in performance.
I’m assuming it’s a service, so I’m curious how many people the Mac is responsible for serving. Of course more likely to be innocuous and maybe not in the serving path at all
There's a difference between "hosting all of Eats Canada" and "hosting some component or service until we can fix prod." I think it's more likely similar to the latter
This tracks. At a former employer, one of their acquired startups (that I joined on to help guide technical direction for) ran their entire order flow by calling a url every minute via a cron task from the primary database server.
At one point, the newer devops organization migrated to a bigger, beefier MySQL (this is when fast NVMe was first available in specialized EC2 instances).
Nobody remembered to migrate the cron script at that moment, so for about an hour I ran all of customer order flow with a bash script curling a public facing (yikes) url to initialize those jobs.
Probably was running a script that ssh'ed into production boxes and restart some service at some specified interval or cleared some log files to stop the drive from filling up.
“until we can fix prod”, but then other stuff comes up, and just look at the backlog! And, I mean, it works pretty well for now…and before you know it, it’s been there forever
(Desktop app, but...) whenever I or another developer would say "it works fine on my computer", my boss would sarcastically reply "so, we should put your computer on the customer's site then?"
It’s funny how people assume things. It seems odd that the assumption would be that this laptop runs production. I don’t think anyone who builds things with tech would assume this.
I think the origin of that ping is the "ping" sound that various instant messengers made/make on receiving a message, not some nerd reference to internet protocols.
Ping me when you have a chance to use that comeback, I'm curious to know how it goes.
Jokes aside, I use it when I'm chatting (IM) with other net-ops people, it just basically means "get ahold of me however you think is best based on the urgency".
Blackberry Messenger had a ping built in. Nothing since has had an equivalent feature. It was absolutely amazing for getting someones attention urgently....
even for 2015 (when this was likely taken) Uber Eats would most certainly not have run from a single laptop. Core services like dispatch, matching, pricing etc in any case weren't geography specific. If i had to guess, this was some Ops person running a local driver incentive, payout query + email workflow from their laptop.
OP — I’m not that privy to the details of what in the world was running on that MacBook Air, but I’m 100% sure it wasn’t “the production servers for Uber eats Canada.” It was some kind of hack doing some kind of thing on a recurring basis to make sure the service continued. These things were often hacked together by engineers, and sometimes ops people, in the middle of outages, as a stop gap.
yep definitely pulled this move before. Probably on a laptop because it is the only thing that has permissions to do whatever insanely janky thing it’s doing.
Not that I condone this sort of thing, but folks who do need to do it should know about caffeine [0] which allows a Mac to disable it's lid switch and keep running with the lid closed.
I’ve worked for big companies in the past where mission critical cron jobs were run on a Linux vm running off an old little desktop tucked away on a benchtop in the corner that no one ever noticed. Done to bypass various network controls to allow us to fix problems as they popped up. Absolute cowboy stuff, but effective.
One place I worked at had some data science stuff that pulled data from prod on a cron and re-uploaded the results every day, on some core2duo in the back of the office. AFAIK it was still being used in 2017 and might still be lol. It was a pretty powerful machine because someone got it to see how much money he could get the company to spend.
I’m not surprised. Amazon payment service’s reconciliation and settlement system was run off of a developer desktop.
My memory is hazy as it was 18 years ago but I think it was due to some IP restrictions by banks and Amazon didn’t have tech to migrate IPs across machines.
I totally believe the job was on a desktop, but I’ll call bullshit on the IP reason. The desktops don’t have direct external IPs allocated, so there’s a NAT with standard DHCP. There was no IP migration issues to deal with on the internal network.
103 comments
[ 3.1 ms ] story [ 84.3 ms ] threadSeemed to work ok, honestly. Our DB boxes, which do much more heavy lifting are not now (and weren't then) Mini's, or anything close, though, heh.
The goal was to balance supply and demand of restaurants and orders.
Eats product didn't have any demand shifting levers at that time yet restaurants were getting blown up with demand at peak times (ex. top pizza restos at 7pm on a Friday).
The product did however allow ops teams to manually set a delivery radius for a given restaurant.
Clever ops teams scripted shrinking / expanding this delivery radius to encourage demand at non-peak times and reduce it during period where restaurants were overloaded and likely to ignore / cancel orders.
This laptop ran scripts for all cities in Canada to adjust delivery radii for each restaurant until a few different levers were built into product.
Ian (who is a _super nice_ guy) owned this process which kept things working smoothly while Eats was experiencing explosive growth.
Another fun fact - Eats Marketplace, which is the current Eats you see today, was built and launched in Toronto - SF engineers / product / design supported by Toronto ops teams. Quickly eclipsed the Instant Eats product which was the 3rd (and most successful to date) try by Uber at food delivery.
I’ve found that modern developers almost always overestimate the amount of physical hardware needed to run things. I’m not sure why.
The interviewer didn't believe me.
I did the calculations in front of him.
This was the time, I just learned Rust.
I did the full analysis and showed him that QPS is great.
Of course, my design didn't include 500 microservices, 50 deployments of Hadoop, Kafka, Elastic, Mongo DB, and 500 slightly different dashboards that different teams in Lyft probably use. Most modern tooling in ZIRP phenomenen.
Someone's Hadoop experience - https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Apple should really license the M architecture for servers. They could even have someone else manufacture and deal with delivering it. They're leaving free money on the table. I don't think it would cannibalize their Mac sales at all.
> I can't imagine trying to deal with 1m+ concurrent tcp connections. 2 million connections on a BSD mchine is doable though not trivial and that too even in 2012. https://blog.whatsapp.com/1-million-is-so-2011
Microwave means 0.3 to 300 GHz.
You should probably also keep your very expensive and sensitive space equipment at a safe distance from any microwave too, what I was trying to say I guess
Hello, this is the German Coast Guard.
We are sinking! We're sinking!
What are you 'sinking' about?
So no surprise here really.
^[[^[[^[[^[[:wq^[[^[[::wq!^[[:wq!
Fix the underlying problem later — or don’t.
It is strange that they didn’t at least use a Mac mini, but I assume they already had the hardware.
Back in the early 00s I hosted websites on my local off the shelf Dell desktop on a cable internet connection.
Cloud computing wasn't really a thing and I didn't know better / have much better options (eventually I did migrate to some webhost bundled with the DNS registar).
For the longest time network bandwidth was my limiting factor. I remember finally upgrading my connection, NIC, etc. to the point where it was actually harddrive write speed that was the bottleneck.
Probably depends on your application but harddrive I/O does seem like a likely limiting factor to me today.
Also save for the fact that commodity hardware lacks a lot of the hardware safeties a server does. For instance, your m.2 under sustained load would not perform at the same rate that an enterprise server hard drive can; we're talking on an order of 180MiB/s vs 550MiB/s per drive. Once the m.2's cache is exhausted it will be comparable to a spinning disk in performance.
At one point, the newer devops organization migrated to a bigger, beefier MySQL (this is when fast NVMe was first available in specialized EC2 instances).
Nobody remembered to migrate the cron script at that moment, so for about an hour I ran all of customer order flow with a bash script curling a public facing (yikes) url to initialize those jobs.
Sadly I've never had the chance to ask "Ah, are you ICMP compliant?" in response to a "ping me!"...
Jokes aside, I use it when I'm chatting (IM) with other net-ops people, it just basically means "get ahold of me however you think is best based on the urgency".
And for annoying friends in their meetings.
Incidentally, the command to unlock the system keychain (eg in a cronjob) is:
That doesn't seem to be widely known, and is very useful for things like build jobs. ;)[0] https://www.macupdate.com/app/mac/24120/caffeine
[1] Author link (includes link to source): https://www.intelliscapesolutions.com/apps/caffeine
[2] Original author link (stopped updating before OSX Mojave): https://www.lightheadsw.com/caffeine/
E.g. you can prevent sleeping until another process exists, or initiate sleep or shutdown.
My memory is hazy as it was 18 years ago but I think it was due to some IP restrictions by banks and Amazon didn’t have tech to migrate IPs across machines.