I think the Medium blog was set up for posts intended for a more tech-savvy audience than the broader audience that visits MBTA.com to check schedules, plan trips, view alerts, etc. Maybe at some point it will be merged in.
The website was relaunched ~year ago as an Elixir / Phoenix application and there are still quite a few features and content being added.
This is awesome! I fully believe that government procurement needs serious reform, which this article and effort is clearly attempting to address. So, props to David Block-Schachter (the CTO).
There are far too many large, bloated consultancies that specialize not in delivering quality products and services, but rather in "surviving" the government procurement process.
Props as well to the Swiftly team - I had a change to take a peak at some of the APIs they expose to their customers and it's quite valuable. In particular, they roll up stop-pair segment performance on routes by time of day, which allows someone to query for bus performance by discrete route-schedule-segments.
Gathering this type of data is quite labor intensive and a significant technical lift (I was once part of a project doing this with GTFS-RT data from the NYC MTA). This type of information, and the broader ecosystem of performance related API services they provide to their users (based off the limited amount I have seen), can enable operators to extract highly articulated performance statistics about their fleet, on their own.
Not really. Just tracking the location in an app isn't going to tell you anything about traffic conditions. Seeing that the bus is only a mile from your house might not be that useful if one day it's in gridlocked traffic and the next it's nothing but open road and green lights.
I use the MBTA buses sometimes to get from my house to the nearest train stop after which I make a transfer to another bus. A real time map of the bus locations would be 100x more valuable than just a prediction.
Same with the train - I use a website that shows the exact locations of all the MBTA trains, and I am able to accurately predict how long the train will sit outside of the Alewife station (which is infuriating BTW) based on the number of trains currently at the platform. A prediction would not be able to take into account this domain knowledge and would be completely worthless.
King County Metro (Seattle) had a system like this back around 2003. Interestingly enough they did not have GPS on the buses, they'd simply instrumented the trip meter which was plenty good enough since buses were (usually) running along fixed routes. The real time location of the bus was displayed in a fairly crummy Java applet, but at least for the route I live on it worked great. I just waited for a bus to show up on the map about a mile from my apartment and that gave me plenty of time to walk to the nearest the bus stop.
I have found One Bus Away to be completely inferior, to the point that I usually don't bother to check it.
I think you may be presuming that every UI that consumes this data is able to display a map showing the bus’s location. We do make real-time bus lat/lng available through our API and lots of clients use that data. But there are other applications where a time prediction is still helpful. For example in many of our stations that serve bus routes, we don’t have the infrastructure (yet) to display a live map but we do have digital signs that tell when the next bus is expected to arrive.
A map isn't necessary to show a location. The digital sign could just display the name of the last intersection the bus was at. (I'm not convinced that's any better than a reasonably good prediction though.)
- Time. Time has the advantage of brevity ("15 min" takes up a grand total of five characters) and indisputable meaning. It's also the hardest to predict, and people get pretty upset about incorrect predictions.
- Intersection. On a signboard, intersection names can get quite long. You would also need familiarity with where exactly the intersection is located, how the bus gets from there to your stop, etc. Boston is famously non-gridded, and has no real consistent rhyme or reason to its street layout, so that would be very confusing.
- Miles/stops away. New York's BusTime uses this. This is a pretty indisputable metric, and if you have GPS or a working odometer is trivial to confirm. But this runs into the same problem as intersections; you need context. The bus is two stops away, but how long does it take to make those two stops? A mile passing through Times Square is very different from a mile through the leafy, quiet suburbs of Queens.
You can pick between meaning and certainty, but it's very hard to have both.
This is more a business issue than a tech issue. There are two dominant RTI systems providers in England (where my bus knowledge comes from), and only a few more in the world. If they start providing an open data feed of real time locations, then they are giving a leg up to competitors who would potentially be able to leapfrog their predictions, as well as undercut them on price. This would threaten their ticket machine business (the ticket machines incorporate the GPS tracker - something like 98% of buses in England have GPS). Having spoken to all of the RTI suppliers during my work for DfT in the UK government, I am convinced that they will not release these data unless legislation forces them to.
And that is basically what is happening early next year.
Just curious if you're finding it challenging to hire for Erlang? Have you generally hired people with some experience, or do you typically expect new devs to pick it up upon hiring?
I also work in the customer tech department for the MBTA. We use Elixir and all of our web interfaces use Phoenix on top of that. We don’t require anyone to have experience with elixir or erlang; most of our developers have learned on the job. We look for solid general programming skills and a willingness to learn. We’ve had a lot of success with that strategy so far!
In theory, one should be able to use information about the stop lights to improve the prediction.
Anyway, I really would like to see a histogram of the predicted distribution of arrival times. Sometimes I 1-sigma need to make the bus and sometimes I 3-sigma need to make the bus...
Even better, give buses priority at stop lights when they are behind schedule, or intentionally delay them when they are ahead of schedule. This is being done in a few countries and it seems to work quite well. Why predict when you can control it?
Awesome. I hope SFMTA can learn from this. They are taking proposals to replace Nextbus currently [1]. When I last measured Nextbus reliability, the first prediction for a bus was weakly correlated with arrival, and the second and third predictions were totally free of information. One of the main problems around here seems to be buses with the route and destination sign set to the wrong thing (e.g. the bus says it is headed to downtown, it's actually going the other way) which apparently throws a huge wrench in Nextbus.
I just skimmed the article, so apologies if this was covered.
The article suggests people will use these predictions to spend more time at home before leaving their bus stop. If that is the case, what happens if the prediction says their bus will be, say, 10 minutes later, but then that bus catches some lucky breaks with lights, heavy traffic unexpectedly clears up, etc., and the bus makes up 5 minutes of that?
People who relied on the prediction could miss their bus.
Is there a mechanism to address this?
The only one that I can think of offhand that would cover almost all cases would be if the predicted arrival times are also sent to the buses, and if the buses arrive ahead of the prediction they wait until the predicted time before leaving.
You wouldn't want to do that for all predictions, though. Maybe only predictions made within 10 minutes of the original scheduled arrival time are binding, or maybe only predictions made when the bus is within three stops of the stop in questions, or something like that.
It explicitly talks about this in the article where he talks about how they try to predict the bus will arrive earlier rather than later. In the 12-30 minute bucket, "12% of weekday predictions were up to 4 minutes early, 71% were up to 6 minutes late, and 17% were outside both of those windows".
So they are weighting it to under-predict lateness, which will reduce the chances someone misses the bus by arriving at the stop at the predicted time, it seems.
As a bus rider who uses apps to track expected arrival times (not to be confused with scheduled arrival times), here are some thoughts:
Timing when you leave the house/office to catch the bus matters a lot more when weather is miserable and/or when buses arrive infrequently than when they arrive frequently and the weather is nice.
The bus I take most often is scheduled to run every 3 minutes on average during my morning commute. That means that even with severe bunching I rarely have to wait more than 10 minutes. Unless it's under 10F outside or is raining buckets, I just leave the house when I'm ready to leave the house, and use the bus predictions mostly as a way to manage frustration levels.
It takes me about 90 seconds to get from home to the bus stop if I make the light, BUT there's a really long traffic light in between. If the bus is listed as 4 minutes away, I have a 50% chance of catching it (depending on whether it makes the previous light). Less than that and there's a 90% chance I'll miss it.
Also, the arrival predictions seem to be based on distance and stop count rather than typical local traffic conditions. It also doesn't take into account times when the bus is empty and skips most stops or is full and not only stops everywhere but boarding and debarking take forever. Which means that while it's eerily accurate for certain stops and times of day, at others I can count on the bus to arrive in half the time predicted and at yet others in twice the time. Unless there's a crash or construction along the route, in which case all bets are off.
During my evening commute, the buses are scheduled much further apart (and are a much longer walk from the office). A 10 or 15 min scheduled headway means leaving whenever you're ready works well if the buses are reasonably spaced (regardless of whether they're actually on schedule). But bunching sometimes causes a 20-25 minute wait (and occasionally more). What I find the bus arrival predictions best for is to be informed on those occasions when the bus is half an hour away and your best bet is to say "screw it" and walk or take a cab.
On another route I frequent, I get on at the first stop on the line. The next bus is always parked around the corner (unless it's super late completing the incoming leg of the route). The app I use lists it as due in 1 or 0 minutes. The actual arrival time will in fact be the scheduled time.
What it comes down to is mostly about knowing your route and adjusting your perception of the predictions to take that into account.
It's working pretty well, the key thing is the point 4 in the article - the timetables show "in X min" if they know how long it will take for the bus to get here, or just "at XX:XX" if they don't know for some reason (transmitter or bus broke, whatever) so they just show the scheduled arrival.
So you can be pretty sure that if it says it's coming in 4 minutes you won't be waiting 30 minutes.
Clearly marking how sure you are of your prediction is IMHO more important than making very good predictions. I don't care if I have to wait 3 minutes or 6 minutes, I do care if I skip another bus waiting for a more convenient one in 3 minutes that never arrives.
Seattle has GPS on most busses that integrates into apps like Transit. Transit is also crowdsourced if people leave it running when they're on the bus. Some bus stops also have screen timetables that update live information regarding arrival time.
I know I have to leave my condo when Transit says 3 minutes and it has the "this bus is reporting" logo.
Having developed similar systems in the past, I would like to advice against arbitrary boundaries on accuracy.
We used a couple of metrics to assess the quality of the predictions. The first one is sMAPE (Scaled Mean Absolute Precision Error), which tells us quickly where and how our predictions fail. We plot this with the precision on the y-axis and the minutes till arrival on x. Also plotted is bias, which is important, especially since you want to be slightly biased towards being too early. Similar axes.
Other metrics we used are MAE (Mean Absolute Error), RSE (Root Square Error) and 2D kernel density plots.
In the end, for a contract, I would take into account the density of use of these predictions. It's nice that you can predict perfectly during the middle of the night, but if that line is not used, it is next to useless. So something like sMAPE * passengers or sth like that.
Also, even though I cannot relay this one-to-one to the passengers, a confidence interval on predictions is gold.
32 comments
[ 119 ms ] story [ 1972 ms ] threadThere are pages on the main site about this project, for example: https://www.mbta.com/projects/better-bus-project
I think the Medium blog was set up for posts intended for a more tech-savvy audience than the broader audience that visits MBTA.com to check schedules, plan trips, view alerts, etc. Maybe at some point it will be merged in.
The website was relaunched ~year ago as an Elixir / Phoenix application and there are still quite a few features and content being added.
There are far too many large, bloated consultancies that specialize not in delivering quality products and services, but rather in "surviving" the government procurement process.
Props as well to the Swiftly team - I had a change to take a peak at some of the APIs they expose to their customers and it's quite valuable. In particular, they roll up stop-pair segment performance on routes by time of day, which allows someone to query for bus performance by discrete route-schedule-segments.
Gathering this type of data is quite labor intensive and a significant technical lift (I was once part of a project doing this with GTFS-RT data from the NYC MTA). This type of information, and the broader ecosystem of performance related API services they provide to their users (based off the limited amount I have seen), can enable operators to extract highly articulated performance statistics about their fleet, on their own.
Why are we "predicting" at all?
Why doesn't every bus have a tracker that tells you exactly where it is at all times?
This is 2018. GPS with refinement isn't rocket science anymore.
Tell me where the bus is and I'll do my own prediction thanks.
The problem is that they should have spent 0% of their time on prediction and 100% of their time on actual tracking.
Once you have tracking, prediction pretty much comes along for the ride.
Same with the train - I use a website that shows the exact locations of all the MBTA trains, and I am able to accurately predict how long the train will sit outside of the Alewife station (which is infuriating BTW) based on the number of trains currently at the platform. A prediction would not be able to take into account this domain knowledge and would be completely worthless.
I have found One Bus Away to be completely inferior, to the point that I usually don't bother to check it.
- Time. Time has the advantage of brevity ("15 min" takes up a grand total of five characters) and indisputable meaning. It's also the hardest to predict, and people get pretty upset about incorrect predictions.
- Intersection. On a signboard, intersection names can get quite long. You would also need familiarity with where exactly the intersection is located, how the bus gets from there to your stop, etc. Boston is famously non-gridded, and has no real consistent rhyme or reason to its street layout, so that would be very confusing.
- Miles/stops away. New York's BusTime uses this. This is a pretty indisputable metric, and if you have GPS or a working odometer is trivial to confirm. But this runs into the same problem as intersections; you need context. The bus is two stops away, but how long does it take to make those two stops? A mile passing through Times Square is very different from a mile through the leafy, quiet suburbs of Queens.
You can pick between meaning and certainty, but it's very hard to have both.
And that is basically what is happening early next year.
There is a post on the first of every month called "Who is Hiring." You might do better planning to participate in that.
Most recent:
https://news.ycombinator.com/item?id=18113144
Anyway, I really would like to see a histogram of the predicted distribution of arrival times. Sometimes I 1-sigma need to make the bus and sometimes I 3-sigma need to make the bus...
1: https://www.sfchronicle.com/bayarea/article/Muni-looks-to-re...
The article suggests people will use these predictions to spend more time at home before leaving their bus stop. If that is the case, what happens if the prediction says their bus will be, say, 10 minutes later, but then that bus catches some lucky breaks with lights, heavy traffic unexpectedly clears up, etc., and the bus makes up 5 minutes of that?
People who relied on the prediction could miss their bus.
Is there a mechanism to address this?
The only one that I can think of offhand that would cover almost all cases would be if the predicted arrival times are also sent to the buses, and if the buses arrive ahead of the prediction they wait until the predicted time before leaving.
You wouldn't want to do that for all predictions, though. Maybe only predictions made within 10 minutes of the original scheduled arrival time are binding, or maybe only predictions made when the bus is within three stops of the stop in questions, or something like that.
Timing when you leave the house/office to catch the bus matters a lot more when weather is miserable and/or when buses arrive infrequently than when they arrive frequently and the weather is nice.
The bus I take most often is scheduled to run every 3 minutes on average during my morning commute. That means that even with severe bunching I rarely have to wait more than 10 minutes. Unless it's under 10F outside or is raining buckets, I just leave the house when I'm ready to leave the house, and use the bus predictions mostly as a way to manage frustration levels.
It takes me about 90 seconds to get from home to the bus stop if I make the light, BUT there's a really long traffic light in between. If the bus is listed as 4 minutes away, I have a 50% chance of catching it (depending on whether it makes the previous light). Less than that and there's a 90% chance I'll miss it.
Also, the arrival predictions seem to be based on distance and stop count rather than typical local traffic conditions. It also doesn't take into account times when the bus is empty and skips most stops or is full and not only stops everywhere but boarding and debarking take forever. Which means that while it's eerily accurate for certain stops and times of day, at others I can count on the bus to arrive in half the time predicted and at yet others in twice the time. Unless there's a crash or construction along the route, in which case all bets are off.
During my evening commute, the buses are scheduled much further apart (and are a much longer walk from the office). A 10 or 15 min scheduled headway means leaving whenever you're ready works well if the buses are reasonably spaced (regardless of whether they're actually on schedule). But bunching sometimes causes a 20-25 minute wait (and occasionally more). What I find the bus arrival predictions best for is to be informed on those occasions when the bus is half an hour away and your best bet is to say "screw it" and walk or take a cab.
On another route I frequent, I get on at the first stop on the line. The next bus is always parked around the corner (unless it's super late completing the incoming leg of the route). The app I use lists it as due in 1 or 0 minutes. The actual arrival time will in fact be the scheduled time.
What it comes down to is mostly about knowing your route and adjusting your perception of the predictions to take that into account.
It's working pretty well, the key thing is the point 4 in the article - the timetables show "in X min" if they know how long it will take for the bus to get here, or just "at XX:XX" if they don't know for some reason (transmitter or bus broke, whatever) so they just show the scheduled arrival.
So you can be pretty sure that if it says it's coming in 4 minutes you won't be waiting 30 minutes.
Clearly marking how sure you are of your prediction is IMHO more important than making very good predictions. I don't care if I have to wait 3 minutes or 6 minutes, I do care if I skip another bus waiting for a more convenient one in 3 minutes that never arrives.
I know I have to leave my condo when Transit says 3 minutes and it has the "this bus is reporting" logo.
We used a couple of metrics to assess the quality of the predictions. The first one is sMAPE (Scaled Mean Absolute Precision Error), which tells us quickly where and how our predictions fail. We plot this with the precision on the y-axis and the minutes till arrival on x. Also plotted is bias, which is important, especially since you want to be slightly biased towards being too early. Similar axes.
Other metrics we used are MAE (Mean Absolute Error), RSE (Root Square Error) and 2D kernel density plots.
In the end, for a contract, I would take into account the density of use of these predictions. It's nice that you can predict perfectly during the middle of the night, but if that line is not used, it is next to useless. So something like sMAPE * passengers or sth like that.
Also, even though I cannot relay this one-to-one to the passengers, a confidence interval on predictions is gold.