I personally have never seen this complaint leveled against the mighty Goog, nor really any company that's cash positive. When I see people giggle at bloat, it's more often than not twitter and still more often than not, it's not intrinsically because big company = waste, but because company that should be small is made to be big either by VCs or Wall Street, and then can't turn a profit anymore.
No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business.
Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process).
Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.
Right, I feel like the author has a good angle with which to tackle some of this criticism, but choosing Google as the example company is really unpersuasive for me. I'd agree that Twitter would be a better example (or Uber, 2,000 engineers - doubled in size over the last year)
Joking aside I enjoyed that video (https://www.youtube.com/watch?v=kb-m2fasdDY - for those who didn't see it), the presenter was pretty honest and way more open than I expected. I'm just so curious how those services break down - if it's genuinely on the "lstrip as a service" level or whatever, whether that's "1,000 services" figure was actually a relatively simple API with 10 versions at various levels of deprecation, whether he meant it was 1,000 different service endpoints (250 services each with POST, GET, PUT, DELETE) or what. I want to know more basically :)
I am usually not someone who falls into the "what do they need so many people for, it should be simple"-trap but I was still surprised to learn that Lyft has 400 engineers and Twilio has around 300 too. I am note sure they would be that big if they had to finance everything out of their profits, but i guess it's a good way to maximize progress, on the other hand they also had a lot of challenges with growing dev teams.
Things like Dropbox are striking. I guess it was mostly written originally by Drew Houston over a year or so and was cool.
Then 2011, 50 million users 70 employees. Seems fair enough.
Now about 1500 employees. It worked fine and scaled fine with 70 employees so you wonder if hiring the other 1430 was good business. I guess if VCs are throwing money at you you may as well spend it on something. Though that's probably what Yahoo was thinking when it went to 11,700 employees and that didn't play out so well.
Sales persons are necessary when continuing growth requires efforts that do not scale as effectively. Marketing and advertising campaigns work sufficiently well for b2c with 1-2 people but you hit a wall if you want to selling to, for example, the Pentagon because the way those with the deepest pockets buy is nothing like how most efficient individuals and organizations buy. Groupon's massive needs for salespeople is an example of how even lower margin b2b even needs sales due to how scaling out b2b is so much less cost-effective. But for investors, landing a huge contract is not as much about the revenue as much as an indicator of stability in terms of both revenue and market signal (no Fortune 100 will buy some random fly-by-night vendor's stuff without code escrow even if it's the best thing ever because their sourcing overhead is so poor and big companies are immensely risk-averse by design).
They definitely have sales roles in more recent years and have a sales ops team [1][2][3]. The Federal channel manager description even acknowledges that there's unique challenges in selling to the government that necessitates someone that knows the different contracting vehicles. What they've done with little turn to sales culture is pretty admirable but at the same time only possible with organic, engineer-to-engineer driven growth Github and Atlassian suites.
Salesforce grew somewhat similarly via rogue accounts on expense accounts, but to actually clinch the biggest growth there was no way to avoid salespeople to get over $500MM in revenue probably.
I think those people miss the point that not all engineers are working on the core products.
For example, as one grows they need to have a dedicated HR teams. These teams will demand for a HR software to make their life easier.
If the software is an on premise solution then you require engineers to manage that software.
Most HR systems require a RDBMS backend, so there is an additional need for a DBA.
As this adds to the company, there is a need for a hosted Identity Management solution, which again requires dedicated engineers...so and so forth.
While there are many reasons for a company to be loosing money and most of them are related to building a sustainable business, from an engineering point of view it has a lot to do with hiring. Companies with lot of cash want to hire talented engineers and there is nothing wrong with that when getting the company off the ground. But as they grow the talented pool they can hire from gets smaller. So either they start overpaying for a great talent or offering above market rates for a mediocre talent. Its mostly the latter people who offer negligible intrinsic value while getting boat loads of money.
I'm definitely not claiming to know what's good/bad for a company I've never worked for, I'm saying the perception from outside is that companies hit these sizes, hire massive amounts of people to do something, and then the products almost universally become worse.
What features has Twitter shipped? Moments??? Pfft.
On the contrary I'd argue they've hardly done anything new, and that is why they are struggling. Think of all the things Facebook has done and abandoned in the same time.
But I like Twitter - I just wished they fixed some pretty obvious issues. I mean... Tweetstorms are like a request for a feature right there, and Twitter's response is... silence.
As a counter point, on every product I've worked on, on every time we've wanted to make a 'big leap' we've need to get a small expert team to go make it happen.
And yet sometimes a small team really could do it. I spent years complaining about MSN/AOL/Skype etc. and talking about how a small team could build a better messenger in a week, but we didn't really believe it. Then Slack actually took that possibility seriously, and look at the results.
Slack also jsut recently hit 3 million users. 100 million people were using AOL Messenger at it's peak, and 12 million were using ICQ when AOL bought Mirabillas.
Yes, but that's something a lot of people are arguing here and I agree. The "I could build in a week" is ridiculous, but "I could what this 1000 engineers company do that with a small team of 10, in a lean way in 18 months" is very frequent in the market.
MSN and AOL come from the 90s, and Skype is from 2003. Slack is from 2013 - the web has more than doubled in age for Slack compared to the others. Computers are ridiculously faster, browsers are more mature, language libraries are more complete.
Slack also took much longer than a week, and to be honest, the actual chat part of slack is pretty meh. Where Slack unbelievably shines is in the buttery-smooth onboarding process and the management around the chat accounts.
Yup. Have never gotten the whole "Airbnb/Uber/<startup>" is just a crud app line of thought.
If they're crud apps then everything else in between is just glue programming.
The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own.
Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...
These companies are not "tech" companies in my book any more than the Sears catalog was a "tech" innovation.
They are normal business that happen to be enabled by technology, but the technological challenges are not new or to be frank uncommonly challenging.
Because we're I assume mostly coders, we think they are differentiated by tech, almost everything else (marketing, sales, customer service) is likely more important.
Bad technology could certainly kill a platform, but excellent tech won't save it, and a mediocre implementation is good enough.
Take a look at Uber's career's page. Even broken down to USA Engineering on the Engineering team hires, there are dozens and dozens of open positions. It's safe to say Uber thinks it's a > 20 person job.
I agree with you and parent that it is (and should) be a > 20 person job for its scale, but the career page is probably not the best example.
Many of those positions are perpetually open or rotated periodically (see: "You're leaking trade secrets" by Michael Schrenk) while the actual position was given to someone internal or someone connected to someone else internal to the firm.
Most of these pages are used for resume harvesting so that [if the firm has resources] an HR or technical recruiter can periodically evaluate the rest of the field. The firms holding out for the "rock star" often are more aggressive in recruiting them (whether at Job fairs or through LinkedIn headhunting).
They have an Android app, an iOS app, driver management system, fleet management system, ride tracking system, ride scheduling, navigation system (not sure if this is 3rd party or in-house), payment processing, profile system for both riders and drivers, price calculation system, and probably a bunch of other stuff in the background.
All this must work correctly and reliably in the 400+ cities they're operating in. To me at least, 20 developers is not enough to develop and maintain that many interconnected systems. I would say 100s at the very least.
Are you serious? With many 9s of uptime, low latency requirements and hundreds of millions of dollars in revenue on the line? Uber employs over 2k developers right now.
Did you read the article? In it, the author discusses the engineering issues that arise from trying to deal tackling problems in different languages. Now, granted, Uber in this particular thread of discussion does not have to deal with tokenization issues of Japanese/Chinese character search strings but I think it is naive to not think that the nuances of individual communities of 400+ cities intra-country, let alone dealing with separate nations, will add significant engineering overhead... (even if the relationship is not linear)
If you think scaling a service like Uber is not challenging, you're vastly underestimating how large they are.
Uber especially has requirements that make it tough to scale correctly. You need to track each Uber session in near real-time, right? That means you simply cannot afford to drop or interrupt those sessions. It's a completely different problem than scaling Instagram or WhatsApp.
Hmmm... nothing like near real-time. Challenging near real-time is sub-nanosecond.
Uber real-time requirement of what < 30s? (for a viable product).
No that doesn't seem like a challenging problem. It seems like a problem that would require almost no throught for a MVP, and then incremental challenges as the problem scales.
It seems like a problem that is so comparatively easy compared to what computers are capable of that I wouldn't even class it as "difficult".
But what's telling about your answer is that your thinking about this technical challenge, this feature as being of primary importance.
Do you think if Uber was 10 times cheaper than a regular taxi real-time tracking would be as significant?
Uber's funding which has enabled them to subsidize their expansion (and rides), advertising and promotion and possibly that they plugged in to a easily accessed communications channel (mobile apps). Were likely all more important than the quality of their implementation.
Probably less than 30s between pings during a ride, but I haven't tested it tbh. I usually use the word real-time when talking about systems that operate in the ns time range, and near real-time when talking about software. Sorry for the confusion.
What I was trying to emphasize is that Uber needs to keep track of these sessions and make sure they're not interrupted, or else they lose customers.
Yes, I know building something that fits these requirements and works most of the time is trivial. But we're talking about a system that must basically work as close to always as possible. More importantly, it must do this at scale, maintaining and keeping track of perhaps millions of sessions at once.
In a nutshell, an Uber MVP is probably not difficult, but the transition from MVP to real-world product is non-trivial in my opinion.
I'm actually planning on setting up an Uber-like service for my home country, which is why I've been giving these issues some thought.
Even 1M simultaneous users reporting every ~30 seconds could average down to ~30k messages per second. And they'd be extremely trivial to segment between servers. You're right, not a weekend hack, but a few weeks of a small team's time isn't out of the question. The technical side is not the challenge here though really - the business side is.
Perhaps you're right when it comes to scaling, but you're assuming that the system was designed correctly in the first place, which would be impressive imo. There is also a lot of failure handling and recovery that needs to be done gracefully, which I think would be complicated for a session-based system.
Again, I'm no expert, just making educated guesses!
I'm not sure why you need near real-time single session for the tracking. A ping from the app every couple seconds for the available driver screen. Then for the route tracking and billing have the same pings include the GPS track that the phone is keeping track of. If the servers lose track of any of the cars the data would be stale until it comes back into connection.
Also Uber is pretty conveniently partitionable. There's a little bit of overlap in some places but that's mainly for the available drivers screen.
I agree with you completely. I think a lot more effort is required once you are running the system at scale. Since I don't have experience with running Uber, I'm simply making an educated guess as to what kind of complexity is involved.
In addition the whole "we code for a better world" is just HR marketing to attract engineers. Sure there are companies that have innovative areas and oppurtunities, but many engineers are just code monkeys fullfilling daily duty without glory.
I read an article the other day that claimed Uber was beating Lyft because they had a strategy of opening a physical office in each city, having an office for drivers to sign up, and a human being who tailored things for that location. Their rival on the other hand assumed that some programmers in SF could do everything neccessary from a distance. Not sure how true it is, but it seemed an interesting theory.
Well, that's the thing - they are just a CRUD app in a lot of ways. Even at "uber scale" it really isn't that bad, it'd be a few incremental challenges but nothing a small team couldn't handle from a technical perspective. The catch though is that on top of that CRUD app is a business which involves some serious legal and marketing divisions to make that app work.
The tech may be simple, but the business side of it certainly isn't. The "I could hack that up in a weekend" attitude is often true - but it only covers the bare basics of the software it doesn't even attempt to address the business, which if often where the real struggle is these days. Especially if your background is development. I think many engineers still say this knowingly though, just to (perhaps incorrectly) scoff at the value of good business people.
But, adding the wireless connections and the servers and the technical staff to code and maintain, would save the driver the labor of pressing the button. That's what technology is for.
KISS applies here. By keeping the 'driver presses a button' interaction, the complexity is reduced by at least a factor of 5, and the TCO ends up being much less.
Sometimes, a doorbell just needs to be a doorbell.
Sure but if I add all the above complications then I get automated bus tracking system where everyone on their phone can see where the bus is and when it's arriving, I can track and automatically detect issues with traffic, with the bus, with the driver.
Sure you can do the simple announce but the even easier one is to just have the driver talk over a mic, he needs one anyway. There I one upp'd you on KISS. But there is a reason not to KISS.
How often do you want the driver to press buttons? And how do you mark the intermediate positions in between bus stops? Because on any remotely complicated high congestion route, you will not get reasonable estimates if you only update at bus stops.
I am so glad you said this. With the boom of the tech industry, it feels like a lot of stuff is getting over engineered these days. Instead of products that focus on something that does it really well, it does its job mostly alright with half-broken bells and whistles.
Except asking a bus driver to keep track of button pushing is distracting and could endanger the lives of the passengers or people outside. The stop needs to be announced 30-60 seconds before they arrive at the stop so then is driver is still actively driving. This is something the drivers union would be all over, it's not as straightforward as you assume.
If there's a long gap between the stops, people may forget what's the next stop. Announcing at the opening may be too late or the stop has to be long enough. Even for slow passengers in a crowded bus. Announcing a minute before the stop is a good heads-up that bus is approaching the stop and people should prepare.
A skilled engineer weighs the costs and benefits of a system before architecting it.
Is it more economical to pay for constant cellular connectivity across 1,000 buses, support IT staff for when the system fails or needs upgrades, etc. or to have the guy you're already paying $15/hr press a button after each stop?
At least on most UK buses, the driver already needs to tell the computer where on the route the bus is so that it can issue the correct tickets to customers. This has been the case for decades.
Or... microphone and speakers for the driver to announce the stop. If you'd like visual indicator too (accessibility FTW!) plenty of low tech and/or cheap solutions that would meet the requirements.
Ha, indeed. Just thought of another - even lower tech - a printed list of all stops, visible to passengers on the bus, then clear signage for each stop. If the bus stops every stop by default, then this is simpler, if not then the passenger needs to be able to see this on the approach, with another time to signal that they want to get off. Can this get even lower tech?
Really? Either it's a location 'Piccadilly Circus' or its a place type 'Airport' - any of those can easily be learnt by the driver. Or writen on a printed sheet that they can easily see. Don't jump to making it more complex than it needs to be.
I see you have never tried it in practice. Train conductors in Austria/Czechia/Germany – which tend to be more trained and less busy than bus drivers – tend to stumble around for half a minute for each unintelligibly badly pronounced sentence. Even the pre-recorded messages in buses are of so bad quality that tourists prefer asking other passengers rather than trying to figure them out.
On the other hand, the overthinking in question (which is probably true for bare-bones next stop indication) provides the base for my personal bus bug: know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.
They actually do this as well (and many have live updating signs), so the buses do have GPS linked to a server of some kind. It's normally absolutely fine, but on the couple of occasions for me where it broke it seems that the server doesn't conduct the bus in any way and kind of hopes the bus is on the right route.
> They actually do this as well (and many have live updating signs)
I don't know about London, I've seen some systems where the stops had "live-updating signs" but only provided the time of the next official passage. And while I wasn't clear, my bug is not just stop signage but phone access as well (that's most useful when e.g. it's raining cats and dogs and you want to avoid waiting in the rain at uncovered bus stops)
The TFL system will remove a bus if it doesn't get an update often enough that includes actual progress on the route. This is in fact the opposite problem of what you mention, in that they don't do a very good job of predicting based on past performance. E.g. I used to take a specific bus to pick my son up from nursery that always would "disappear" for a certain number of minutes at a certain part of the journey because a very short stretch of the route was always extremely congested at that time of day.
It's a tricky one, because they have big concerns about over-promising (e.g. the bus could be stuck behind an accident instead, in which case guessing that it's 5 minutes away because of past behaviour will really annoy people) so in some situations they err on the side of caution.
But they are unable to get rid of all of the corner cases. E.g. a bus that moves rapidly for a short stretch, then gets "stuck", will often appear, disappear, appear, disappear, with much less progress in time estimates than elapsed time should indicate.
The thing is, the team working on it has a huge amount of gathered info on how to give these predictions in a way that tries to minimise surprise and avoid making promises that cause problems for people. E.g. often it's better to take a bus off the list so people who are in a hurry will grab a taxi, rather than "promise" the bus will be there in just another few minutes. They still get it wrong too often, but I really don't envy them trying to get the right balance between promising too much and too little.
This is available for public buses in NYC. The city added GPS to their buses and then the ability to query distance from current stop via SMS.
Then they opened their APIs, and now I have a half dozen apps to choose from that will show the approximate time until the next bus from a particular line will reach any given stop.
(Not always 100% accurate, but useful enough to know when to wait indoors for 5 min if it's raining, or pick another line/choose to walk because of delays.)
>know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.
We had that, through GPS + GPRS (or SMS?), back in 2001-2 in a small city I was living in Europe. Complete with full (flash based) website map of where each bus is.
I used it to get out of the tech park/institute I worked at and into the bus stop outside at the very last minute (since bus timetables where completely unreliable).
GPS is insufficient to give accurate enough estimates these days. It may or may not work depend on your local routes and stops, but it's too imprecise for a lot of locations and traffic conditions to give decent precision. It's good as a "first approximation" and if you don't have anything better. Most systems today end up integrating data from multiple different sensors to improve on gps.
Thats clever. Most public bus systems I know in the US still see frequent cash payments onboard (the elderly, mostly). Is this not the case in Sydney, or is the system cashless?
Over the past few years they've rolled out a new electronic ticketing system which is used exclusively.
On some bus routes (and/or outside of peek times) you can pay a cash fare on the buss, but even that is still a single-use paper RFID card that should be tapped on and off. These sales are in the extreme minority though - the elderly require an electronic card to get their discount IIRC.
This is the only thing I want. Displaying a bus timetable can be done just fine with laminated paper, and I'm not convinced that the ability to update (with cancellations, route changes, etc) justifies a tech solution.
Displaying the actual bus status, though, is huge progress.
Halifax, NS, was doing that in the mid-'80s without the benefit of GPS or handy mobile apps. (Every bus stop/route had a phone extension, and major stops/terminals had talk boxes.) Granted, it wasn't a huge system, but it was hitting rocks with sticks and hollering over tin cans and string compared to what's available as a starting point today, and it worked. The idea that it should be a hard problem today, given GPS, vastly improved wireless communications and eleventy-seven different ways of figuring out where a user is (if entering a stop code seems like it might be too much work) is just plain silly. Or we've significantly redefined what "hard" means.
Or expectations have increased, both in terms of accuracy and in keeping costs low.
Consider that e.g. in London the old route indicators on the buses used odometers. Which worked for telling you were along the route a bus was. Most of the time. But didn't tell you how long it takes until the bus reaches the next stop other than on average, as it would tell you nothing about traffic etc.
Today London buses don't use just GPS to replace it either, because that's nowhere near good enough. See the "Tracking" section here [1].
When I said "worked", I meant worked, not approximated working or came lose enough not to have caused too many upsets. If expectations have grown to be higher than plus-ten-second accuracy (early wasn't tolerated), then we collectively need some sort of counselling/therapy.
You don't get that kind of accuracy in any system that has to deal with congested roads, or any dense major city. E.g. there's plenty of bus-stops near me where stops on the same route involves going around corners and traffic machines and stopping close enough that GPS would be insufficient to be guaranteed to tell the stops apart if you don't get a good enough fix on additional satellites quickly enough.
Even with all the extra data, and a lot number crunching, in London it's a challenge to get it to withing a few minutes during rush hour.
So, sure, if you're dealing with simple, small systems with little congestion, you can simplify.
German bus drivers have it as well. The routes are lists of stop names and their change is triggered by opening the doors. Can be adjusted by the driver if stops are passed, etc.
Runs on some windows if I remember correctly and routes are probably hardcoded along with the data for the segment display outside above the windscreen.
what about the bus stop broadcasting a signal with the id, name etc? the system is activated by the proximity..so you can have smthing like approaching station A, leaving station A.
it is scalable because the bus-stop transmitters are added , moved, removed as needed.
The announcing the next stop could be achieved with a dual radio setup, one high powered and low powered. The high powered could be used to detect an upcoming stop and the low powered could be used to detect proximity to the stop.
(Lets ignore the security issues that have been shown with amplifying the low powered radio in cars for a second) It's a method used in keyless entry in cars. A higher powered radio is used for button press on the cars key so you can unlock the car/open the boot/sound the horn to locate the car at a distance but keyless unlock should only work at a much closer distance.
Not saying its the correct way to do it (my local bus network doesn't do this) but stop proximity can be worked into their idea.
My local bus network use M2M Sim's running on the Vodafone network but its basically a virtual private mobile network running on top of Vodafone's network with no connection to the internet.
These connections are in both the busses and the stops. The reason is so the stops can provide real time updates to people at the stops too. If a bus breaks down its no longer tracked on the real time updates so you are not effing and blinding because a the stop said there is a bus due in 5 mins and no bus comes. And if a bus is removed from service unexpected then the information on why can be feed to people waiting for the bus. People are less likely to be pissed that a bus didn't come when expected if they are given a valid reason.
Personally I was hoping they used some form of radio link I could of sniffed with a SDR so I could put a live updating sign in my house. I just scrape the data from their live feed they use to update the tracker on their mobile app/website instead.
Broadcasting out a signal from the stops could need a transmitter that may need to be licensed as the signal would be to be picked up quite a distance from the stop. Also not all the stops in my area are these "real time updating" stops as they are normally put into covered stops (EDIT: Bus Shelters... The correct term wouldn't come to mind while I was typing this.) and some of our stops are just poles in the ground with a sign on them. The busses tracking their own position wouldn't need these stops upgrading to give this information to people on board the bus as the covered stops can be quite a distance between each other and removes any issues if a stop get damaged and stops broadcasting data (say a car crashes into it).
The busses tracking themselves also have benefits that don't visibly benefit passengers. The services back office can monitor busses and spot problems on the network as they happen. Unexpected heavy traffic on a route may trigger the company to skip or stagger a bus (one of my local routes in advertised as being "every 5-10 mins". The real time traffic condition information could also be sold on to other companies (not sure on the value of this data esp for busses that run once an hour or even longer).
I was surprised my local network didn't do something similar as it seemed like a logical choice. But after some research I found nothing about them using RDS to do this.
I would love to be able to talk to a tech at the travel network about this. My own research came to the suspicion that because where I am in the country they would of had to strike up deals with multiple transmission sites to get the coverage needed.
What special announces? Either the bus driver does it or it's one that's used frequently (like 'Please move down inside the bus'), in which case there is a button for it.
Same in Munich. I think those work offline, also the driver can pick the next stop.
I guess it is easier in the long run to have all routes mapped and instead of GPS you use something like doors open / doors close as a trigger to jump to the next entry.
I work part time at the company that manages the new style signs for bus stations (EDIT: in London, to be clear), so we've gotten to see a bit of the size of the operation TFL has to deal with gathering and processing the bus information. He might have some details wrong, but he's certainly not gotten the complexity level wrong.
One of my "fights" at work is prototypes. A prototype can be build very quickly (cheaply) and can exhibit many problems that may be overseen by the "heavy design process". A prototype is a marvelous way to get feedback from users very early in the projects.
The only downside to prototypes is that then the managers don't understand why the actual project takes so much longer and end up asking to just go with the prototype for production.
The GPS and mobile data are overthinking it. The routes are known so simply trigger on door open/close (with some logic for denounce/etc) and cycle through.
For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.
And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.
Except that quickly becomes inaccurate if the bus driver decides to re-open the doors for an incoming passenger, or due to traffic conditions, opens the bus doors further from the stop than the beacon reaches. Or skips a stop due to road conditions.
Consider a bus stuck in traffic. It is not unusual, at least in London, for a driver to open/close the door multiple times between stops when traffic moves slowly enough, often with longer intervals in between than there usually would be between stops.
> Set a minimum time until next stop can be announced
This feels like a dangerous hack that will come back to bite you in any number of unexpected scenarios that probably occur frequently during an average bus driver's day.
For an robust system I would use multiple data streams including bust status data/beacons/GPS. I've seen busses with offset stations.
3G is necessary for live ETA, if you want to get fancy. All these complications, in my view, are worth it if you want a system that works 24/7 and serves thousands of people daily. The HW and SW dev costs are acceptable, but it does need a decent team, depending on what the starting point is and the ability of the team members.
I'm confused, are you suggesting there are places where bus routes change hundreds of times a day?
I can't imagine many reasons a route would have to change on short notice (accident blocking a road?), and in those situations the announcements not quite being right would not be the end of the world.
Buses are often assigned to a route and then leave that route unexpectedly in the middle of the day to service another route (accident, other bus broke down, traffic, subway shutdown etc) , often not starting at the beginning of that route.
Oh right, that makes more sense. I guess you could just keep a record of all the different routes in every bus though? And skip to the right spot manually as needed.
This has all kind of usability issues (now you're expecting the driver to pay attention to and correct mistakes), while providing less accurate data (buss pulls away from stop, gets stuck in traffic; estimates at next stop are based on the bus having pulled away and average times; customers gets angry) and less ability to do traffic management (e.g. spacing out buses by slowing down later buses slightly) without having drivers guessing and reporting data back.
> For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.
That's not remotely accurate for the reason above.
> And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.
It can. E.g. the old London system used odometers. But that was sufficient only for showing "next stop ..." and for reporting very rough estimates. Today it uses odometers, gps, rate gyros and turn rate sensors to be able to more accurately position the buses along a route, and they still regularly "give up" and take a bus off the schedule when traffic conditions means the uncertainty is too high.
Generallly my approach if engineers disagree significantly on a timeline is to multiply the worst case estimate by a large factor, on the assumption that odds are good it means neither one has a clue what the actual complexity is if they don't manage to get to rough agreement...
Far more useful is the system which tells you, at the bus stop, what the ETA of the next bus(es) is. And that really does need the whole GPS+data system, along with physically robust signs. But on the other hand, that can still be done by a small company - I know there's one here in Edinburgh but I can neither remember their name nor find them in a search.
You can do quite a lot with a small local company. It's global reach that's expensive, as soon as you start needing any kind of high-touch sales.
>
Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.
The question is what is more work: Using the existing data that is stored in some obscure format for which you first have to write an importer/converter or just recreating the data. Add the fact that the existing data might need some cleanup/additions (e.g. times on timetable are only stored with minute precision and you want second precision for more accuracy).
You generally want to make the announcement before the bus stops so that people can be ready to get off when the bus stops. Also buses don't always stop at every station on it's route so have to deal with what happens when stop is skipped.
You can certainly do it with an RPi - although you might have to keep rebooting it every hour or so. :)
GPS and 3G/4G are trivial add-ons. There are stacks for mobile data.
RFID doesn't have the reliable range, so that won't work as a solution.
It's not really a hard problem for a competent small engineering team with a mix of embedded hardware and software skills. It's not quite a trivial problem, because there are some interesting edge cases, mostly when a bus is diverted. But there is absolutely zero rocket science required.
You could hack something unreliable together in a weekend, but it would take up to six months to get all the bugs out, do the production engineering for a bullet-proof solution, standardise the installation process (if retrofitted), and so on.
Double that if management is poor. :)
Those last parts often get forgotten in software. Hacking something together that kind of works if you don't look at it with a visible frown is hobby coding, not engineering. Engineering is the boring work that happens after that, to build something with a quantifiable many-9s uptime that meets or exceeds a standard spec with comprehensive test cases.
Usually the announcement is not "we have stopped at Foo and Main". The announcement is "Next stop Foo and Main" while you're on the way to it, so you have time to sling your bag back over your shoulder, get your coat back on, and make your way towards the door. Depending on the density of stops along the route this can come anywhere from 1-7 blocks ahead of the stop.
Conductors aren't a solution to a problem in themselves, they're a means to solve a problem. So what's the actual problem? WHY do they want conductors back?
>- Hardware needs to be resistant to harsh diesel engine vibrations
That's a non-issue.
>- 3G/4G connectivity, need a mobile data contract with local ISP
This is a duh!
>- The announcement should be bi-lingual for Airport busses
It can always not be. How would that be worse than NOT having the announcement at all? Besides what would the other language be? French, German? That would still be useless to 90% of foreign visitors...
Check out APRS[0]. Solves most of the above already by centralizing data-collection. Would just need a beacon on every vehicle. No need for internet connectivity. Receiving stations can be scaled up as required. Logging of course not an issue as can be logged on-board and/or externally.
The on-board display can overlay a map and call the next stop based on proximity. Not a weekend project, I'll admit, but many existing moving parts available off the shelf.
You are massively overthinking the problem. The very first system of this type I saw didnt even have a microcontroller. Just an eprom for driving LED array, bunch of TTL logic and two inputs. One linked with 'open doors' switch, the other going to undo switch. And it worked fine for couple of years in a >million city.
No idea, but it would be a fairly reasonable way to test a prime-finding algorithm - or to replace a prime-finding algorithm with something less complicated.
Because the alternative: generating the primes at compilation time with a template functional expression would be too slow, and quite possibly would explode the compiler's memory limits. (No Turing Machine here), and doing it at run-time or deliverying it as a "resource file" is obviously out of the question.
To me this is a form of the Dunning-Kruger effect[0], where only somewhat informed people estimate the cost of a "0.1" (or POC) release without considering the famous "unknown unknowns" which may involve scaling, billing, 3rd party integrations, etc.
I find I've had to bar myself from thinking about many of these factors, after I realised how many projects I stop myself from pursuing because I know about the hard bits, and realise how many of my past projects I'd never have taken a shot at if I knew what was coming up and was thinking like that.
It's great to know, but often it's not so great if it leads to inaction...
E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea.
But it's hard. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.
The sad truth is that in the majority of cases the quality or time taken over the implementation isn't particularly important other factors matter much more:
* building something people actually want.
* advertising
* engaging with users
* aggressively promoting your product.
* customer service
It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.
Yes and if you look at the footnote where the arithmetic is laid out, it seems quite confused. It conflates querying with indexing and also flatly assumes that each machine will index 5M pages, which seems low to me.
> In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed.
> Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244.
Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.
Yeah, but asking readers to resize windows, resize the font, and tweak the CSS to get decent line heights is a bit of cheek when it's trivially easy to implement at least minimally pleasant-to-read typography and layout.
It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.
To me this post addresses the same naivety that David Graeber demonstrates when he claims that everyone not involved in hammering out a sheet of steel to make a horseshoe or pulling a plough is engaged in a bullshit job[1].
I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months.
Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.' I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman. The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.
People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.
But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.
> Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'
I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward not making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its supposed to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it does deliver the value it is supposed to.)
There's no particular reason to think the people working in the position are in the best position to evaluate value delivery (or that even the people in the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)
There is a great incentive to obfuscate when your "value delivery" is skimming (mutual funds that charge 1.25% management fees for underperforming the S&P), privatization fraud (health insurance in the US, government contractors, and pretty much everything related to real estate since "home appreciation" is now supposed to pay for people's retirements instead of pensions), taking advantage of stupid people (lottery, penny auctions, telemarketing, almost all advertising), education fraud (textbook industry, "student loans now, regret later" private colleges and universities - which since many student loans are federally guaranteed is also a form of privatization fraud), or "intellectual property" racketeering.
I think that a lot of this is because things that seem quite pointless on an individual level are actually very important at scale. When I started at Google my job was pushing pixels around on the search results page, and when I left my job was leading a team pushing the pixels around (in the middle I got to do actual cool stuff). On an individual level, this is pointless and boring. On a corporate level, some of those pixels were worth literally $1B+. I would love to have a billion dollar business, but I can't get there from scratch by pushing pixels around on a page.
Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'
Yeah I know that's his point and that's why I describe his point of view as being naive.
It may be true that for some percentage of workers at any given time their role is experimental and may turn out to be unimportant or a failed attempt at change, and it may be the case that it take a while for these sorts of inefficiencies to be rooted out or resolved because the cost and time associated with doing so make it a low priority but that is something that people who are in those positions would not necessarily be aware of.
This is the same as when people say things like "I don't know why we are so precious with our kids these days. I never wore a bike helmet and I survived!".
Without the proper perspective people frequently form wildly inaccurate opinions.
I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman.
If all I have to do in order to make it a straw man is replace the word iPhone with horseshoe he's doing a pretty good job of constructing his own straw man.
The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.
That story is completely ridiculous.
People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours.
Then businesses have an opportunity to compete for talent by offering better working conditions. If they can do so profitably then they win.
Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.
This is more a result of the spoils of productivity increases not being distributed evenly.
By having massive unemployment and under employment then expecting the remaining workers to work harder for less pay on average while more and more income is paid towards servicing private debt the finance, banking and real estate sectors have made out like bandits.
But that's a different subject entirely from automation.
I don't think so. It has a lot to do with automation because that is where the productivity increases have come from, but rather than the federal government guaranteeing full employment and setting a floor price for labour and a minimum set of conditions below which we consider employment to be exploitation, which would not only maintain demand in the economy but also reduce the inequality with which the spoils of those productivity increases are distributed, they have left people to fend for themselves and the economy has languished as a result.
Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.
Maybe in 1,000 years but I disagree that automation will have the type of impact people (particularly in the UBI advocacy camp) claim it will.
Weird example, where it's assumed that a single machine can only hold 50GB of document data, which results in the assumption that you'd need 200k machines to store 1T documents...
I feel if you have something better to offer, say a better way to look at information, not just a better page rank algo, but a different way to look at information that is scattered across, one that enables a better insight into existing web pages and less overhead of constant 'searching' by typing a query, something that automatically helps to "see" relevant information with less effort, then it can be an alternative to what a Google search does today and can position itself as a competitor in the long run even if it does not exist today.
I have worked with one-man companies that did this and often they are shoddy products. Healthcare just seems to suck at software.
Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this.
There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".
Besides underestimating the scope of the problem, one logical fallacy that programmers fall for is believing that a single developer should be able to juggle lots of tasks easily. If a small team can create a SaaS application, why do large companies need so many engineers for so many specific things?
I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.
Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature-rich as the actual thing, and besides, nobody will be using it.
Everything seems easier when you're not doing it yourself. It's only a minor variation on the grass being greener. Everything gets way more complicated when you're doing it at scale, with high reliability, for real users, with some hope of making money. Practically nobody adds complexity for its own sake. Usually, if you think something is much more complicated than it needs to be, it's because you don't have any idea what the real requirements and constraints are.
The article raises some good points but there is still a dichotomy here to me. Compare a successful startup with 2 founders with an established company with 2000 employees in the same business. The scale of the difference in employees vs the difference in the product is often shocking. BigCo product might be better, but that much better?
Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.
Those 1998 extra staff don't have to make the product 1000x or even 2x better (however you might quantify that). They just have to make it sufficiently better to maintain or gain market share.
442 comments
[ 5.9 ms ] story [ 286 ms ] threadA nigger does not know what the word "random" means.
No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business.
Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process).
Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.
<snark/>
Then 2011, 50 million users 70 employees. Seems fair enough.
Now about 1500 employees. It worked fine and scaled fine with 70 employees so you wonder if hiring the other 1430 was good business. I guess if VCs are throwing money at you you may as well spend it on something. Though that's probably what Yahoo was thinking when it went to 11,700 employees and that didn't play out so well.
Salesforce grew somewhat similarly via rogue accounts on expense accounts, but to actually clinch the biggest growth there was no way to avoid salespeople to get over $500MM in revenue probably.
[1] https://www.smartrecruiters.com/Atlassian/99086565 [2] https://www.smartrecruiters.com/Atlassian/95395260 [3] https://www.smartrecruiters.com/Atlassian/93068238
For example, as one grows they need to have a dedicated HR teams. These teams will demand for a HR software to make their life easier. If the software is an on premise solution then you require engineers to manage that software. Most HR systems require a RDBMS backend, so there is an additional need for a DBA. As this adds to the company, there is a need for a hosted Identity Management solution, which again requires dedicated engineers...so and so forth.
While there are many reasons for a company to be loosing money and most of them are related to building a sustainable business, from an engineering point of view it has a lot to do with hiring. Companies with lot of cash want to hire talented engineers and there is nothing wrong with that when getting the company off the ground. But as they grow the talented pool they can hire from gets smaller. So either they start overpaying for a great talent or offering above market rates for a mediocre talent. Its mostly the latter people who offer negligible intrinsic value while getting boat loads of money.
Maybe it's just big-company-itis.
On the contrary I'd argue they've hardly done anything new, and that is why they are struggling. Think of all the things Facebook has done and abandoned in the same time.
But I like Twitter - I just wished they fixed some pretty obvious issues. I mean... Tweetstorms are like a request for a feature right there, and Twitter's response is... silence.
Something about 80%/20% rule...
Slack also took much longer than a week, and to be honest, the actual chat part of slack is pretty meh. Where Slack unbelievably shines is in the buttery-smooth onboarding process and the management around the chat accounts.
If they're crud apps then everything else in between is just glue programming.
The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own.
Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...
They are normal business that happen to be enabled by technology, but the technological challenges are not new or to be frank uncommonly challenging.
Because we're I assume mostly coders, we think they are differentiated by tech, almost everything else (marketing, sales, customer service) is likely more important.
Bad technology could certainly kill a platform, but excellent tech won't save it, and a mediocre implementation is good enough.
https://www.youtube.com/watch?v=kb-m2fasdDY
https://www.uber.com/careers/list/?city=all&country=united-s...
Many of those positions are perpetually open or rotated periodically (see: "You're leaking trade secrets" by Michael Schrenk) while the actual position was given to someone internal or someone connected to someone else internal to the firm.
Most of these pages are used for resume harvesting so that [if the firm has resources] an HR or technical recruiter can periodically evaluate the rest of the field. The firms holding out for the "rock star" often are more aggressive in recruiting them (whether at Job fairs or through LinkedIn headhunting).
All this must work correctly and reliably in the 400+ cities they're operating in. To me at least, 20 developers is not enough to develop and maintain that many interconnected systems. I would say 100s at the very least.
Uber especially has requirements that make it tough to scale correctly. You need to track each Uber session in near real-time, right? That means you simply cannot afford to drop or interrupt those sessions. It's a completely different problem than scaling Instagram or WhatsApp.
Uber real-time requirement of what < 30s? (for a viable product).
No that doesn't seem like a challenging problem. It seems like a problem that would require almost no throught for a MVP, and then incremental challenges as the problem scales.
It seems like a problem that is so comparatively easy compared to what computers are capable of that I wouldn't even class it as "difficult".
But what's telling about your answer is that your thinking about this technical challenge, this feature as being of primary importance.
Do you think if Uber was 10 times cheaper than a regular taxi real-time tracking would be as significant?
Uber's funding which has enabled them to subsidize their expansion (and rides), advertising and promotion and possibly that they plugged in to a easily accessed communications channel (mobile apps). Were likely all more important than the quality of their implementation.
What I was trying to emphasize is that Uber needs to keep track of these sessions and make sure they're not interrupted, or else they lose customers.
Yes, I know building something that fits these requirements and works most of the time is trivial. But we're talking about a system that must basically work as close to always as possible. More importantly, it must do this at scale, maintaining and keeping track of perhaps millions of sessions at once.
In a nutshell, an Uber MVP is probably not difficult, but the transition from MVP to real-world product is non-trivial in my opinion.
I'm actually planning on setting up an Uber-like service for my home country, which is why I've been giving these issues some thought.
Again, I'm no expert, just making educated guesses!
Also Uber is pretty conveniently partitionable. There's a little bit of overlap in some places but that's mainly for the available drivers screen.
The tech may be simple, but the business side of it certainly isn't. The "I could hack that up in a weekend" attitude is often true - but it only covers the bare basics of the software it doesn't even attempt to address the business, which if often where the real struggle is these days. Especially if your background is development. I think many engineers still say this knowingly though, just to (perhaps incorrectly) scoff at the value of good business people.
Think a bit further:
- Hardware needs to be resistant to harsh diesel engine vibrations
- 3G/4G connectivity
- Need a mobile data contract with local ISP
- Software needs to be able to handle network disconnections
- GPS needs to be able to pin-point at which stop the bus is currently stopped, even with bad GPS coverage in larger cities
- If the bus skips a stop because of a detour, the software should be able to detect it and announce the next stop
- The announcement should be bi-lingual for Airport busses
- The announcement should work for people with hearing aids
- Server needs to know all bus-stops
- Server needs to know different stop types, such as bus terminals, intersection stops, regular stops, hand-over stops, virtual stops
- Server needs to be able both work of a yearly bus schedule and real-time update
- Server needs to output in an understandable JSON/XML because the government subsidies demand an open-data format
- Server needs to publish data to an open-data server because of the subsidies
- Because the bus-company is sponsored by European Union money the control interface should be translated to German/French/English/Spanish
And now this weekend project takes a team of 10 engineers working for a year.
I think it just counts the number of stops and has some input from the driver
Sometimes, a doorbell just needs to be a doorbell.
Sure you can do the simple announce but the even easier one is to just have the driver talk over a mic, he needs one anyway. There I one upp'd you on KISS. But there is a reason not to KISS.
Why? You can announce the next stop a few seconds after departing from the current stop, and then again at the opening of the doors.
Is it more economical to pay for constant cellular connectivity across 1,000 buses, support IT staff for when the system fails or needs upgrades, etc. or to have the guy you're already paying $15/hr press a button after each stop?
Breaks down quickly if you need bi- or tri-lingual announcements, which is the norm in many popular tourist destinations.
Aside: I'm impressed with the amount of bike-shedding for this one. Though I'm part of the problem, too :-)
That's all speculation though.
I don't know about London, I've seen some systems where the stops had "live-updating signs" but only provided the time of the next official passage. And while I wasn't clear, my bug is not just stop signage but phone access as well (that's most useful when e.g. it's raining cats and dogs and you want to avoid waiting in the rain at uncovered bus stops)
Apps like 'Transit' also (accurately) show the current GPS location of the bus as it approaches, and it updated every minute (I believe)
http://reisapi.ruter.no/Help/Api/GET-StopVisit-GetDepartures...
>Returns a List of StopVisits (departures) from a Stop. If no time parameter is supplied, departures in realtime will be returned.
It's a tricky one, because they have big concerns about over-promising (e.g. the bus could be stuck behind an accident instead, in which case guessing that it's 5 minutes away because of past behaviour will really annoy people) so in some situations they err on the side of caution.
But they are unable to get rid of all of the corner cases. E.g. a bus that moves rapidly for a short stretch, then gets "stuck", will often appear, disappear, appear, disappear, with much less progress in time estimates than elapsed time should indicate.
The thing is, the team working on it has a huge amount of gathered info on how to give these predictions in a way that tries to minimise surprise and avoid making promises that cause problems for people. E.g. often it's better to take a bus off the list so people who are in a hurry will grab a taxi, rather than "promise" the bus will be there in just another few minutes. They still get it wrong too often, but I really don't envy them trying to get the right balance between promising too much and too little.
Then they opened their APIs, and now I have a half dozen apps to choose from that will show the approximate time until the next bus from a particular line will reach any given stop.
(Not always 100% accurate, but useful enough to know when to wait indoors for 5 min if it's raining, or pick another line/choose to walk because of delays.)
We had that, through GPS + GPRS (or SMS?), back in 2001-2 in a small city I was living in Europe. Complete with full (flash based) website map of where each bus is.
I used it to get out of the tech park/institute I worked at and into the bus stop outside at the very last minute (since bus timetables where completely unreliable).
https://i.imgur.com/LjV7MtB.jpg
On some bus routes (and/or outside of peek times) you can pay a cash fare on the buss, but even that is still a single-use paper RFID card that should be tapped on and off. These sales are in the extreme minority though - the elderly require an electronic card to get their discount IIRC.
Displaying the actual bus status, though, is huge progress.
Consider that e.g. in London the old route indicators on the buses used odometers. Which worked for telling you were along the route a bus was. Most of the time. But didn't tell you how long it takes until the bus reaches the next stop other than on average, as it would tell you nothing about traffic etc.
Today London buses don't use just GPS to replace it either, because that's nowhere near good enough. See the "Tracking" section here [1].
https://en.wikipedia.org/wiki/IBus_(London)
Even with all the extra data, and a lot number crunching, in London it's a challenge to get it to withing a few minutes during rush hour.
So, sure, if you're dealing with simple, small systems with little congestion, you can simplify.
it is scalable because the bus-stop transmitters are added , moved, removed as needed.
updating the data can also be done remotely
Also, there are things you want to do that this doesn't cover. For example, announcing the next stop when you pull away from the last stop.
(Lets ignore the security issues that have been shown with amplifying the low powered radio in cars for a second) It's a method used in keyless entry in cars. A higher powered radio is used for button press on the cars key so you can unlock the car/open the boot/sound the horn to locate the car at a distance but keyless unlock should only work at a much closer distance.
Not saying its the correct way to do it (my local bus network doesn't do this) but stop proximity can be worked into their idea.
These connections are in both the busses and the stops. The reason is so the stops can provide real time updates to people at the stops too. If a bus breaks down its no longer tracked on the real time updates so you are not effing and blinding because a the stop said there is a bus due in 5 mins and no bus comes. And if a bus is removed from service unexpected then the information on why can be feed to people waiting for the bus. People are less likely to be pissed that a bus didn't come when expected if they are given a valid reason.
Personally I was hoping they used some form of radio link I could of sniffed with a SDR so I could put a live updating sign in my house. I just scrape the data from their live feed they use to update the tracker on their mobile app/website instead.
Broadcasting out a signal from the stops could need a transmitter that may need to be licensed as the signal would be to be picked up quite a distance from the stop. Also not all the stops in my area are these "real time updating" stops as they are normally put into covered stops (EDIT: Bus Shelters... The correct term wouldn't come to mind while I was typing this.) and some of our stops are just poles in the ground with a sign on them. The busses tracking their own position wouldn't need these stops upgrading to give this information to people on board the bus as the covered stops can be quite a distance between each other and removes any issues if a stop get damaged and stops broadcasting data (say a car crashes into it).
The busses tracking themselves also have benefits that don't visibly benefit passengers. The services back office can monitor busses and spot problems on the network as they happen. Unexpected heavy traffic on a route may trigger the company to skip or stagger a bus (one of my local routes in advertised as being "every 5-10 mins". The real time traffic condition information could also be sold on to other companies (not sure on the value of this data esp for busses that run once an hour or even longer).
http://www.windytan.com/2013/11/decoding-radio-controlled-bu...
I would love to be able to talk to a tech at the travel network about this. My own research came to the suspicion that because where I am in the country they would of had to strike up deals with multiple transmission sites to get the coverage needed.
I guess it is easier in the long run to have all routes mapped and instead of GPS you use something like doors open / doors close as a trigger to jump to the next entry.
I've never had this problem.
For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.
And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.
Notice how we're now adding more complexity now anyway though...
This feels like a dangerous hack that will come back to bite you in any number of unexpected scenarios that probably occur frequently during an average bus driver's day.
3G is necessary for live ETA, if you want to get fancy. All these complications, in my view, are worth it if you want a system that works 24/7 and serves thousands of people daily. The HW and SW dev costs are acceptable, but it does need a decent team, depending on what the starting point is and the ability of the team members.
I can't imagine many reasons a route would have to change on short notice (accident blocking a road?), and in those situations the announcements not quite being right would not be the end of the world.
Yeah, and now we need to install and manage thousands of beacons in bus stops in pretty hospitable conditions.
Of course, GPS isnt perfect either. It tends to fail in CBD areas.
> For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.
That's not remotely accurate for the reason above.
> And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.
It can. E.g. the old London system used odometers. But that was sufficient only for showing "next stop ..." and for reporting very rough estimates. Today it uses odometers, gps, rate gyros and turn rate sensors to be able to more accurately position the buses along a route, and they still regularly "give up" and take a bus off the schedule when traffic conditions means the uncertainty is too high.
More like 2 guys working for a month.
(I calculated the geometric mean of the numbers both of you have given, in case you're wondering ;) )
I'd say something like 3 people for 6 months to 1 year (not all 3 being needed 100% of the time).
You can do quite a lot with a small local company. It's global reach that's expensive, as soon as you start needing any kind of high-touch sales.
Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.
Anyway I haven't been in a bus that doesn't have such a system for at least for a few years (in western Europe and China).
The question is what is more work: Using the existing data that is stored in some obscure format for which you first have to write an importer/converter or just recreating the data. Add the fact that the existing data might need some cleanup/additions (e.g. times on timetable are only stored with minute precision and you want second precision for more accuracy).
GPS and 3G/4G are trivial add-ons. There are stacks for mobile data.
RFID doesn't have the reliable range, so that won't work as a solution.
It's not really a hard problem for a competent small engineering team with a mix of embedded hardware and software skills. It's not quite a trivial problem, because there are some interesting edge cases, mostly when a bus is diverted. But there is absolutely zero rocket science required.
You could hack something unreliable together in a weekend, but it would take up to six months to get all the bugs out, do the production engineering for a bullet-proof solution, standardise the installation process (if retrofitted), and so on.
Double that if management is poor. :)
Those last parts often get forgotten in software. Hacking something together that kind of works if you don't look at it with a visible frown is hobby coding, not engineering. Engineering is the boring work that happens after that, to build something with a quantifiable many-9s uptime that meets or exceeds a standard spec with comprehensive test cases.
Source: I don't drive,and ride the bus a lot.
That's just faster horses though.
Conductors aren't a solution to a problem in themselves, they're a means to solve a problem. So what's the actual problem? WHY do they want conductors back?
That's a non-issue.
>- 3G/4G connectivity, need a mobile data contract with local ISP
This is a duh!
>- The announcement should be bi-lingual for Airport busses
It can always not be. How would that be worse than NOT having the announcement at all? Besides what would the other language be? French, German? That would still be useless to 90% of foreign visitors...
I guess the names of the stops are untranslatable anyway :)
The on-board display can overlay a map and call the next stop based on proximity. Not a weekend project, I'll admit, but many existing moving parts available off the shelf.
Hams play with this stuff... a lot.
[0] https://en.wikipedia.org/wiki/Automatic_Packet_Reporting_Sys...
Some sort of near-field beacon on the stops might be more durable than relying on GPS, but it won't help with missed stops.
Maybe something like that ?
[0]: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
It's great to know, but often it's not so great if it leads to inaction...
E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea.
But it's hard. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.
* building something people actually want. * advertising * engaging with users * aggressively promoting your product. * customer service
It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.
> In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed.
> Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244.
Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.
This is unreadable as is.
It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.
I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months.
[1] http://strikemag.org/bullshit-jobs/
People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.
But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.
I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward not making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its supposed to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it does deliver the value it is supposed to.)
There's no particular reason to think the people working in the position are in the best position to evaluate value delivery (or that even the people in the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)
Yeah I know that's his point and that's why I describe his point of view as being naive.
It may be true that for some percentage of workers at any given time their role is experimental and may turn out to be unimportant or a failed attempt at change, and it may be the case that it take a while for these sorts of inefficiencies to be rooted out or resolved because the cost and time associated with doing so make it a low priority but that is something that people who are in those positions would not necessarily be aware of.
This is the same as when people say things like "I don't know why we are so precious with our kids these days. I never wore a bike helmet and I survived!".
Without the proper perspective people frequently form wildly inaccurate opinions.
I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman.
If all I have to do in order to make it a straw man is replace the word iPhone with horseshoe he's doing a pretty good job of constructing his own straw man.
The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.
That story is completely ridiculous.
People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours.
Then businesses have an opportunity to compete for talent by offering better working conditions. If they can do so profitably then they win.
Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.
This is more a result of the spoils of productivity increases not being distributed evenly.
By having massive unemployment and under employment then expecting the remaining workers to work harder for less pay on average while more and more income is paid towards servicing private debt the finance, banking and real estate sectors have made out like bandits.
But that's a different subject entirely from automation.
I don't think so. It has a lot to do with automation because that is where the productivity increases have come from, but rather than the federal government guaranteeing full employment and setting a floor price for labour and a minimum set of conditions below which we consider employment to be exploitation, which would not only maintain demand in the economy but also reduce the inequality with which the spoils of those productivity increases are distributed, they have left people to fend for themselves and the economy has languished as a result.
Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.
Maybe in 1,000 years but I disagree that automation will have the type of impact people (particularly in the UBI advocacy camp) claim it will.
Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this.
There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".
In the early years of Twitter, it would have been relatively easy to clone Twitter, but you won't have had the ecosystem.
Poster example: Google+
I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.
Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature-rich as the actual thing, and besides, nobody will be using it.
Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.