67% for right time (i.e. <1min late) arrival seems to be pretty average. I’d be more interested in the % of arrival at destination within 5mins as that’s the standard used in many countries and thus easier to compare different systems.
In many parts of the GB network, once you get more than a minute or two late you end up with delays propagating throughout the network, because enough is timetabled with lines near capacity.
You get up to about ~82% within 3 minutes and ~88% within 5 minutes. Beyond that you see little gains.
I can tell you from experience the trains in the UK are expensive and miserable.
The common theory for why is because of the mess made of privatising British Rail in the 1990s. The government did it in a way which managed to create monopolies while at the same time splitting out responsibility between different companies in such a way that millions are drained away on legal costs while the companies argue with each other about who is responsible for each problem.
It's a well worn topic. Had they not achieved the weird corporate separation, such that track, train, station building and ticket can all be separate companies putting on a new service, or adjusting timetabling and even putting up new posters could have been achievable things. As is they spend months, and millions going nowhere. [Edit: Not forgetting that station hotels, ferries and hospitality were separated out to yet more companies. Just because.]
Had they just gone back to the big 4 of 1921-1947, or created a "medium 8" with extra rights to run charters on the tracks we might have rail that works by now.
Then there's the 50+ year story of line and infrastructure upgrades... Each time we update, we seem to update to the previous generation's capability.
Britain runs 20% of Europe's rail journeys. It has the least accidents and deaths of passengers and staff. It is considered the most improved rail network year on year since 1997. It has some of the lowest fares if you book far in advance (and count the cost of subsidisation for foreign fares).
How many decades did it take to electrify Europe's busiest line, the West Coast Main Line? Did it become modern? No. It used last generation signalling such that when we had 125mph HST they were limited to 80 and 100, when we bought back BR's APT as 150mph Pendolino they were limited to 1980s HST speeds as the tracks were electrified and upgraded to 125mph. In 2010.
The "negotiations" between train operators wanting better rolling stock, railtrack (government), station upgrades and the inconvenient length of franchises made the whole process suitable for a Laurel and Hardy or Buster Keaton movie.
Little wonder they're most improved, they gave themselves so much scope...
The Office of Rail and Road publish annual safety statistics for UK rail. Here is a link to the 2019 edition. The page linked states: "2018-19 is the twelfth year in a row with no passenger fatalities as a result of a train accident."
South coast was still third rail, last I checked. Has Wales been upgraded since I left? Not that it matters if the power supply affects the number of crossings and those crossings are supposed to be the cause of accidents, new crossings will be rarely built.
No idea if these statistics reflect reality or are just cherry picked, but one of the problems that England has is attempting to cram a everyone into the only city attracts top businesses and talent and then ship them out again. Daily.
Germany for example has multiple cities with over 1 in million population that are attractive for business and distribute resources and infrastructure. England has London with around ~7 million in population then Birmingham at 0.9.
"It is considered the most improved rail network year on year since 1997"
Improved in what way? Punctuality? Frequency of services? Affordability of tickets? Refurbishment of stations? New rolling stock? It's a claim that doesn't match the experiences of many passengers. What is the source?
"It has some of the lowest fares if you book far in advance"
Even if this is true, it's little consolation for the vast majority of passengers who need to commute daily. The reality is that, for most passengers, rail in the UK is expensive.
Here is a monthly season ticket comparison from 2017: UK vs Continental Europe:
- UK: Luton to London St. Pancras (35 miles) | Monthly season ticket cost: £387
- UK: Liverpool Lime Street to Manchester Piccadilly (32 miles) | Monthly season ticket cost: £292
- Germany: Dusseldorf to Cologne (28 miles) | Monthly season ticket cost: £85
- France: Mantes-la-Jolie to Paris (34 miles) | Monthly season ticket cost: £61
- Italy: Anzione to Rome (31 miles) | Monthly season ticket cost: £61
- Spain: Aranjuez to Madrid (31 miles) | Monthly season ticket cost: £75
We should have plenty of tests of that theory in the upcoming years, since the EU decided that this weird seperation is such a good idea they're going to mandate all members do it. If I remember rightly, currently track and train have to be run by totally independent management structures (though as a sop to Germany and co, both can be part of nominally the same umbrella organisation), other companies have to have the right to run passenger services, and in a few years public passenger services will have to be put up for competitive tender UK-style. I'm not sure off-hand what they're doing with station ownership.
This was something of an issue in the recent UK election. Obviously, EU membership was a major issue, and one of the main parties also wanted to renationize the railways... which is technically allowed by EU rules, it's just that there's very little point since we wouldn't be able to undo the weird structuring created by privatization.
That seems... well, beyond idiotic. Have to confess I've not been paying much attention to EU plans for continental railways, I just visit Netherlands and France from time to time and remind myself we could have done so much better. Netherlands has proper integration, France actually stumped up for a proper modernisation and high speed rail. UK had BR's (actually very good) plan that no one paid for...
With the number of failed and failing franchises, there's a lot to be said for a new approach, perhaps via a period nationalised to draw a line and start again. It seems patently clear that the post-Beeching nationalised years, decades as under-funded political football, and UK's privatisation were all absolute disasters for passengers... Seems like the EU should look, shake their head, and mutter "definitely not".
If it were me, I'd go back to the pre-war big 4 era, where there were a dozen or two smaller companies running parallel competition on some routes, or running goods and charters on big 4 lines. I probably wouldn't reinvent the pre-1921 situation that gave Manchester four stations all essentially overlapping - Central, Victoria, Oxford Road and Exchange. Pre-big 4 company history detangles that! Manchester Piccadilly was lucky in having less parallel competition.
What we've lost, and totally forgotten, is just how much those pre-war railway companies did to feed the railways. Hotel stations that were not the manky unloved budget rooms of the seventies through today, but often a premium offering. The railways bought into early bus, ferry and tram services to develop those, and ensure services fed the stations. Integrated transport and travel! Compatible timetabling! Now I have to go to the Netherlands to experience that.
It's been ongoing for twenty-five years at this point. The initial goal was to improve access especially, and entered into force in 1992. Essentially, to prevent the historic case of (e.g.) a nationalised railway forbidding anyone else operating on their infrastructure (so if you want to run a freight service and the nationalised company don't care? trucks it is!). This also vastly simplified cross-border services as it's then possible to apply for track-access on a non-discriminatory basis, which completely changed the face of the European rail freight market.
Note that as of 1 Jan 2019, international and domestic freight and passenger markets are, at a national level, all competitive. The world hasn't fallen apart in most of the EU. I think it's clear plenty did look at the British example and clearly decided to avoid following the British example in many ways.
Also I think it's worthwhile pointing out the UK's privatisation has been in some ways a success: subsidy now is much lower per-passenger (and lowering this has been a constant goal across all governments since privatisation), and the financial risk of the success is not borne by the government (and as the NAO concluded, no franchises making loses and/or failing would imply everyone is underbidding and the government is losing money as a result)
To me, it's not clear that a return to a nationalised operator would help things in GB; many of the problems come from political imperatives (c.f. "decades as under-funded political football"), and the railway is far more micromanaged by the government today than it ever was under British Rail. By and large the franchising bodies (mostly the DfT) are simply getting the services they ask their contractor for, and passenger should be more directing their annoyance at them; we also have a network that has seen massive growth in passenger numbers and number of services and comparatively little money spent on capacity enhancements to avoid a network run at capacity where a single short-term failure can propagate through large parts of the system. The change that needs to happen is political, and whether ultimate delivery is contracted out is mostly irrelevant.
OK, that puts a rather different complexion on it.
> To me, it's not clear that a return to a nationalised operator would help things in GB; ... the railway is far more micromanaged by the government today than it ever was under British Rail
True, whilst I understand the reasoning of Labour's suggestion to re-nationalise, at best we would get a less effective repeat of 1947. 1947-1960 (before Beeching) worked OK in good part because BR still had the shape of the big 4 years. Even if Labour planned, executed and funded perfectly we would, at some point, get a Tory government, who would be inclined to under-fund it. Off we go again, still with infrastructure 50 years behind.
I mention nationalisation more as a reason to bring it all back into one whole first -- to split it up more sensibly at the next attempt. Divvying up various bits of all the various companies differently seems challenging. :)
The central micromanagement speaks to other UK post-war problems, which also affect rail. We've become one of the most centralised countries there is, whilst talking of smaller government and deregulation. Local funding has been steadily gutted since the end of the general rates. Local authorities used to be able to make far wider reaching decisions, and fund them. They could start, fund or develop local services, such as bus, power or train, and often create a municipal corporation to run it -- hands off.
That's the other appeal of returning to the pre-war organisation, it would distance the government from the day to day, and local choices, especially in Scotland, the North etc, might be possible once again. We could add capacity, and undo some of the worst of Beeching.
Now if only we could only undo some of the UK's excess centralisation as well...
I'm not going to claim there aren't ways nationalisation could play out well (and with a time machine I certainly wouldn't sell BR off), just… I'm dubious with the politically situation in this country that it would end well. Labour's railway plans essentially view nationalisation as the solution to all problems (though it might make people blame those actually making the decisions!), and it would likely take longer than one election cycle for people to realise mere nationalising hasn't made trains run on time, fares go down, etc.
Regional bodies having more power and influence (and money, probably with tax raising powers to actually control their supply!) would be a fantastic thing, but I think would be needed before nationalisation would significantly change the British rail scene at this point.
No, I wouldn't have sold off in the first place either. You;re entirely right on the politics though. Post war had the post war consensus, and until the demise of Heath, One Nation Conservatism. They hadn't discovered globalism and neoliberalism, would still see benefit in putting Britain first -- it was the Tories who nationalised Rolls Royce, who decided not to undo the NHS in 51, etc. When only one side thinks nationalised should exist at all, well, it gets doomed to fail. Water and a few other natural monopolies might survive as I suspect few on the right still claim those as successes.
You might be right about regionalism being needed first, I'm not sure. Seems like we should try and regain both, and need both, so Liverpool and Manchester can work together and get a new railway, or compete and brief against each other and end up with the Manchester Ship Canal. Whitehall's only involvement in those essentially limited to saying "OK". The Whitehall conceived, planned and delayed "northern powerhouse", that has both Liverpool and Manchester mayors tearing their hair out at how little input their cities get just marks it as the usual, never fully delivered, centralised horseshit for the regions...
I commute on British trains every day and have done for years.
They’re not perfect but I am rarely delayed by more than a minute or two. Agree that a massive issue is the price, my train is close to £30 a day for a 50 minute ride into the city and it goes up by 3-5% every year.
The only proper delays I see are caused by unusual weather, as it’s the UK - flooding or snow.
Can I hazard a guess you are doing so in the South East? Try the trains the rest (most) of us experience.
Northern Rail, Transpennine, West Coast, ScotRail, East Coast (there already) etc that have or are about to be given to "operator of last resort", i.e. renationalised -- under a Tory government --for having failed badly.
I commuted in the South East (particularly South East London which means SouthEastern trains) and it was a miracle if I was only delayed by 1 minute on a 12-20 minute journey. Coming to down the Charlton line into London Bridge almost always involved a 5-10 minute delay at (the crossover junction just outside the station that I forget the name of).
I commute there now, the past weeks have been a nightmare of cancelled trains 2 minutes before they're due while I stand at an open air station freezing my nuts off or getting rained on. If we had a little more notice id stay home where it's warm for an extra 15 minutes. The delay you speak of is still there, and it's now worsened by the fact that the new Thameslink trains are given priority to pass as we sit and wait.
Don't know that line, but the approach to London Bridge is overall far less congested after the improvement work over the last few years, despite far more frequent Thameslink trains.
It has follow on effects far down some lines as well, as problems at London Bridge would often cause delays far south of Croydon and up again towards Victoria (because Croydon becomes a chokepoint) for example.
ScotRail train carriages are mostly vastly improved over what they used to be (they don't smell nearly as much of stale piss as they used to...), but still a million miles to the comparative luxury you'd find in Japan.
I want to commute on the line you do then. Thameslink trains during rush hour are constantly delayed or cancelled, so often you have to just get an earlier train to avoid being late.
Over the summer there were at least 4 occasions where the whole line was suspended. They did get unlucky with some power failures if I remember. But twice the line was suspended through London due to heat effecting overhead wires.
Once could be just about forgiven (it gets hot in summer who knew that), but to have it happen again a few weeks later cannot. Sort the issue that suspends your whole network after the first occurrence! If it can't be fixed with a reasonable amount of time to schedule emergency works the system is not fit for purpose.
Once could be just about forgiven (it gets hot in summer who knew that), but to have it happen again a few weeks later cannot.
Well, the thing is: we weren't able to change the laws of physics in those few weeks. Metal still expands, so the wires still droop in the heat. The system is designed for normal conditions; sometimes it doesn't work in abnormal conditions. They could probably do better, but it takes time, effort, and money which are all in limited supply.
I think the point is: if it happens multiple times per year, it’s something that should’ve been designed for, and “abnormal conditions” isn’t an OK excuse.
You're lucky. I mainly drive to work, but every time I have to take public transport I am reminded why I drive.
Earlier this year I had to take public transport for a few weeks whilst getting my power steering repaired and on one occasion my first train was delayed for reasons that were never explained. When I got to my interchange the next train was delayed due to signal failure. And when I finally got my train the driver announced that due to the delay they were passing straight through two stations, one of which was mine, so I had to get off earlier and wait for yet another train.
In the end it took me about 3hrs to complete a journey I can do in 30 mins by car.
There's huge variation, though. London to Oxford? A delight. London to Norwich? Survival horror. And I hear that up North it's positively Lovecraftian.
Yes, Northern R'Lyeh closed many of their routes recently, which caused unknowable horror to descend over Innsmouth (among others on local stopping lines).
I know that HN generally frowns on "me too" or other Reddit-like comments or replies, but I feel a deep (ones) need to reply to this.
Your comment made me very happy, I laughed and snorted, and I'm glad I wasn't drinking my coffee at the time I read it. Thank you for writing it, you are awesome.
Now I'm thinking of a Cult of Cthulhu who design railway depots as an homage to their unknowably evil master, with junctions and lines as a representation of its tentacles and body.
If you like that, Good Omens (the series at least; not read the book) has a bit where it is suggested the M25 is effectively the shape of a demonic curse, designed to cause misery to everyone traveling on it.
I bet a big part of it in Japan is the sheer number of trains running drags down that average. If you count the Yamanote, with peak frequency of trains every 2 or 3 minutes, you’re gonna get a lot of stuff on time.
Think about how rare power outages are nowadays. When infrastructure becomes important enough it becomes run pretty smoothly.
Rail !== tube. You’d be comparing Tokyo’s Yamanote to London’s Circle. I used both to commute at one point or another and found them equally robust. OTOH rail in the UK and Japan are very different stories indeed.
I think it would depend on the stats used. Yamanote is still JR. The Joban line goes all the way to Sendai, if you really want it to.
So if you got lateness stats on trains, it would likely take all the JR stuff and mix stuff up
I agree with your sentiment, my comment is more about my theory that stats would not make the distinction (though it would likely exclude things like Tokyo Metro or the monorails).
Nice to see a post mortem made public. Also fairly well written for laymen while still retaining detail. Most rail infrastructure operators from other countries don’t make this sort of info public (apart from PR type messages which typically don’t explain the issue at at all and are more focused on deflecting blame etc). Kudos to Network Rail.
Cynical me: The UK railway system has plenty of practice of delays and cancellations, it seems they have learned to at least provide some detail in "lay" form.
Practical me: Having that information allows better decisions to be made. Rather than simply wasting my evening at a station other arrangements can be made, possibly having a meal in a local bar/restaurant rather than shivering on a station.
One of the things I've really liked about my local commuter rail (Virginia Railway Express [0]) is the level of detail and communication they have with the community/riders. Their RSS feed typically has a post if there's any delay over 10 minutes explaining the delay and how long it is expected to be. They'll also post when the GPS tracking for a train isn't working but is still on schedule. The after-incident reports are also really nice; this one [1] resulted in delays of over an hour and a half because of track slippage leaving a station. Since then VRE has been letting us (riders) know if they're reversing the engine placement to mitigate possible slippage, so we're able to see that they're actually implementing the changes they propose. Other disruptions are also communicated fairly clearly and promptly [2]. As a side note, I'm pleasantly surprised by the transparency with historical service levels [3] and passenger utilization [4].
Saying all this though, the level of service is low when compared to other countries' commuter rail systems. One of the challenges VRE works with is that it hasn't owned the rail lines it's running on; they're owned by a freight railroad and passenger rail is typically run at a lower priority than freight. There has been some recent movement though by the state [5] to significantly improve passenger rail transit, especially in the northern part of the state (almost certainly related to Amazon HQ2).
This kind of jingoistic comment demonstrates how a tactical post with some technical information, can massively skew judgement. According to Network Rail, the routes in question are not only one of the most congested in the country, but have been mired with problems of under-investment and lack of desire to ease the bottleneck. Some of the routes connect salubrious stock-broker belts with the City of London, in addition to Brighton & Hove, East Sussex. Other routes traversing East Croydon, acting as a hub, also serve the mainline stations of London Victoria and London Bridge to (London) Gatwick ─ the second busiest airport in the country.
I wonder how many people will be forgiving or cheering for Network Rail and handing out accolades for their neatly constructed blogs, when they miss their flight or arrive to work late 'due to signalling problems'.
> We are investigating whether fitting Uninterruptable Power Supplies to each signalling location would have prevented this failure, although it is also possible they would have had to shut down to protect themselves too, as happened at another location during the same event.
This specific remark sounds rather amateurish. This is part of the reason UPSes were invented. A Double Conversion Online UPS would have been able to handle this situation just fine.
I worked for an ISP that was deploying switches nationwide... We got a lot of space in BT Telephone Exchanges and thought nothing more of it.
I have never seen such dirty and bad power! Literally once a week we had to go to exchanges to reset fuses that had tripped.
We then started deploying (rather entry level) UPS to all locations and the problem just disappeared completely... we then invested in more high end UPS systems, but, yeah - dirty power is a big problem and UPS help.
I'm an IT and networking guy... This is one thing that got on my nerves completely - most equipment comes as AC or DC, and I'm never sure what to order, so, I usually go for the AC one as it's the safe choice
Or by development under time, like Forsmark in Sweden where an adapted UPS surge protection almost caused a meltdown. When a surge happened 12 security systems was disabled. From what I heard the diesel didn't because it was searching for incoming voltage it could adapt to.
> De flesta av felen berodde på att det batterisäkrade nätet hade överspänningsskydd, som skyddade komponenten i stället för funktionen. [0]
I don’t mind delays very much, even long ones, as long as there is information readily available about why the delay happens and how long it is likely to last. I have very positive experience with UK railways in this regard; this post is a prime example.
Contrast with PKP (Polish rail), where it’s not uncommon for a train to stand in the middle of nowhere for half an hour, without so much as a single word of explanation from staff. It’s changing for the better, but slowly.
Yes I thought power surge happens in milliseconds to a second range, may be few seconds at worst, first time I ever read a double digit seconds power surge.
Luckily the system shutter itself down as intended.
OK, that's what should happen for an overvoltage. The control systems tripped offline to protect themselves, all signals went to red, and all trains stopped.
Electric railways have multiple different power systems - traction power, signaling power, and utility power, at least. The possibility of a short between systems means a conservative shutdown-on-overvoltage circuit breaker setup is appropriate.
The need to manually turn this system back on is non optimal. A control loop powered by a battery backup should have been able to reset the system once the voltage came back down.
This is a railroad signaling system. It's designed to fail safe, and that's not a buzzword. Any broken wire, go to red. Any broken relay, go to red. Any rail break, go to red. Loss of power, go to red.
If you're getting power surges in signaling power, it's time to shut down, go to red, and not reset until someone has tested the system.
The backup system can be programmed to do the same testing a human can to bootstrap the system. Another benefit is that they can simulate a surge in the system regularly and watch it recover itself versus a human running a checklist once every 10 years.
Rail equipment is all ancient and highly redundant for both uptime and safety.
Yet the actual system reliability and safety doesn't seem great. Signal outages happen nearly every day in the UK, and delay tens of thousands of passengers. Safety failures in signalling systems kill on average a few people per year.
How about a modern system, built with JavaScript-de-jure, running over 4G and WiFi, consumer GPS, "Download train-control from the app store", running on tablets or the train drivers BYOD phone, etc. How would it work out for safety/reliability?
I suspect counterintuitively it would work out much better. Simple cheap portable devices means a bunch of spares can be around always. "Phone battery dead? No worries, there's a tablet I can control the train from too". No WiFi? It also works with 4G or Bluetooth. Server down? It has a Peer connection backup. Hackers broke in? The system has safety logic built in to every device, and uses a consensus system so a few rogue systems can't crash trains. Power supply broken? We have 24 hours of battery, longer if I turn the brightness down.
Not sure about daily signal failures, even when I commuted daily it wasn't near that bad. The idea of powering such a safety oriented system with JavaScript quite literally terrifies me, I'd expect it to be about 30 minutes before two trains collided.
Aside: I think you might mean "Javascript-du-jour"? "De jure" (pronounced roughly "day jooray") means basically "legally speaking, but probably not practically speaking", the dual of "de facto". "Du jour" (pronounced roughly "doo zhoor") means "of the day", "the new thing", "the current fad".
I'm sure this is sarcasm but I'll play along: can't wait for the day the country's rail infrastructure stops working because someone pulled a package from NPM.
The article doesn't mention a date. Presumably the "Wednesday" referred to is 18 Dec 2019.
On a previous occasion when rail equipment failed because of a fluctuation in supply voltage the electricity company was not at fault: the voltage had remained within specified limits. I find it a bit suspicious that the article doesn't mention what the change in voltage was this time, and what variation the specification allows. The penultimate paragraph is perhaps an implicit admission that the electricity supply was within its specification.
I wonder if any of this is indirectly related to the official change in supply voltage, from 240 V to 230 V, in 1995. Nah, probably not; they can't have equipment that old, can they?
Did you get the voltages backwards? If the supply voltage was lowered wouldn't it mean that older equipment designed for 240V would have an even bigger safety margin when it comes to over voltage?
It could be related to the official change in supply voltage, though not for the reason you're thinking. Although Britain's mains supply officially changed to 230V, in reality it's the same 240V as before and there's a wider tolerance band for voltages above the nominal 230V than below to make this work. So if Network Rail installed equipment that expected actual continental-style 230V, that could cause problems.
It wouldn't be the first time that rail companies not paying attention to actual grid specs caused transport chaos this year. Back in August, a power failure caused load shedding that dropped a million houses off the grid but preserved power to the rail network as intended. However, one train company had bought Siemens trains that couldn't cope and immediately shut down when they saw a dip in frequency of the level required to cause automatic load shedding, and required an engineer visit to restart. That one set of trains effectively rendered all the measures taken to ensure the rail network retained power almost useless, as they turned into immobile blocks all over the network.
> rail companies not paying attention to actual grid specs
I would give the rail companies a little slack. They did specify trains that should have handled the frequency change. But Siemens didn’t build the trains to spec.
The shutdown was unexpected and entirely down to Siemens not building their power supplies the specs set out by the rail company.
This isn't accurate. Siemens delivered trains that could tolerate the power fluctuations at the time they were specified.
The problem was that the national grid then changed (loosened) the specification of what an acceptable fluctuation was. The trains could not tolerate this new specification.
What we can blame Siemens for is not providing an easier and quicker way to get the electrical systems back on their feet after a problem, instead relying on a specialist to physically visit a stranded train. I think they said they'd fix that one...
That's not true - the frequency spec hasn't changed in any way relevant to what happened. (I think there was some changes to rate-of-change-of-frequency disconnect specs for generators and possibly medium-term frequency accuracy, but neither of those things mattered here.) This kind of near-simultaneous failure of two large generators, tripping the low frequency demand disconnect, was always specified to cause a dip in frequency of a level those Siemens trains couldn't cope with. There was actually a very similar failure back in 2008. The only reason that didn't cause such chaos is because those trains hadn't been ordered yet.
One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers.
Alternatively, a good enough automatic transfer switch should transfer the power supply rather than tripping if the primary supply goes out of acceptable parameters.
(This goes to one of basics of high availability: failures, when they happen, should be short.)
Most power systems are built with a mix or clever automated systems that can self recover, along with backup dumb systems that can’t.
This is done to ensure that if your smart stuff fails, you don’t end up seriously damaging something.
Unfortunately the dumb stuff pretty much always has to be reset manually. Usually because the dumb stuff tripping means you’ve encountered a scenario that no one anticipated, you really want someone to double check that nothing got damaged before the safety tripped.
It’s a little like a database starting up and discovering that part of its Write-Ahead-Log is corrupt. You really want a human to go in there and see what happened, rather than just ignoring the damage, plowing on and loosing data.
> One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers.
No remote control of them, had to get people to each breaker. (This was mentioned on Twitter closer to the time by NR.)
By reading the postmortem I got the feeling that it focuses primarily on providing a timeline of historical events without being future-looking.
I did not find any questions that would trigger a path of improvement for similar future events:
- Why did the recovery take so long (up until Thursday morning)?
- If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event?
- Why is an on-site technician needed in order to re-activate the equipment?
- Could we build a system that enables the reactivation remotely?
There might be very well-rooted answers to the questions above (based on how real world electrical supplies work), but if the questions aren't asked, we won't end up exploring better solutions for the answers.
Why did the recovery take so long (up until Thursday morning)?
Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day.
If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event?
Why is an on-site technician needed in order to re-activate the equipment?
Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge.
Could we build a system that enables the reactivation remotely?
Probably, but that strikes me as dangerous.
The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.
> Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day.
And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
For those unaware, Croydon is basically a massive chokepoint, with trains coming in from London Bridge and London Victoria to the North and large part of the South coast to the South.
If trains start piling up near either Victoria (and Clapham Junction) or London Bridge, it quickly causes trains to back up all the way to East Croydon, a 15 minute train journey away.
Once it does, it blocks trains towards the other line, as East Croydon only has 6 platforms and no tracks bypassing it.
Once that happens, trains can start backing up tens of miles further South, and will be out of place when things are up and running again.
Victoria is a terminus, and is at times busy enough that trains gets 'stacked' two to a platform on the platforms serving Croydon, so turning trains back quickly becomes a fun exercise... (London Bridge also has lots of platforms that terminates, but seems to have fewer platform capacity issues after the overhaul)
So even if the trains that are in position get moving again pretty quickly, untangling that and getting platform space for all of those trains in short order can compound short delays very substantially.
> how we can improve their resilience so we can stop this from happening in future
I'm not so sure about that.
Just a few months ago a temporary power cut in London left some modern Siemens trains stuck for hours because they had shutdown do to irregular power and had to be manually rebooted.
Apparently nobody thought to investigate if the signaling system is also vulnerable to irregular power.
This PM exuberates professionalism with a clear and concise message! This is an excellent learning experience, for bystanders, travelers, and employees. Seems to me that this was handled in a good way.
The issue was highly unusual and difficult to anticipate but now you know. Good luck mitigating this issue preparing for the future. You will now be better as a result of this failure.
In my opinion, the problem is in poor logical thinking in the design of the supply switchover systems. The article indicates there are multiple power feeds and power supplies (presumably to provide low voltage equipment power).
For a single power feed all of the following results in the loss of the output, (1) no input voltage, (2) input over voltage (under voltage also???). However, in the article it implies that switching to one of the other redundant power feeds or power supplies only happens for (1).
If the redundancy system had considered either (1) or (2) as a fault and to "find another supply, then shutdown this supply" then complete shutoff would have been avoided.
Also, since the active power supply shut down and switchover also didn't happen, there must not be any monitoring of the output of the individual supplies to trigger a switchover. To me, this suggests there is also another single point of failure in the current design.
I guess there is a market for smarter power failover systems.
89 comments
[ 5.0 ms ] story [ 166 ms ] threadHow does it fare against other EU countries? What is so different abount Japan trains, that they are just 0.7 minutes late on average?
[1] https://dataportal.orr.gov.uk/media/1630/passenger-performan...
You get up to about ~82% within 3 minutes and ~88% within 5 minutes. Beyond that you see little gains.
The common theory for why is because of the mess made of privatising British Rail in the 1990s. The government did it in a way which managed to create monopolies while at the same time splitting out responsibility between different companies in such a way that millions are drained away on legal costs while the companies argue with each other about who is responsible for each problem.
There are many articles analysing the problems you can find online, here's one that turned up at the top of search: https://www.theguardian.com/uk-news/2018/jan/07/railways-pri... Edit: Better article: https://www.citylab.com/transportation/2012/09/why-britains-...
Had they just gone back to the big 4 of 1921-1947, or created a "medium 8" with extra rights to run charters on the tracks we might have rail that works by now.
Then there's the 50+ year story of line and infrastructure upgrades... Each time we update, we seem to update to the previous generation's capability.
The "negotiations" between train operators wanting better rolling stock, railtrack (government), station upgrades and the inconvenient length of franchises made the whole process suitable for a Laurel and Hardy or Buster Keaton movie.
Little wonder they're most improved, they gave themselves so much scope...
https://dataportal.orr.gov.uk/statistics/health-and-safety/r...
The European Union Agency for Railways (ERAIL) collects statistics for EU countries:
https://erail.era.europa.eu/safety-indicators.aspx
I think this is largely only an effect of a lot of rails being third rail and as a result there are fewer crossings.
We have been phasing out level crossings wherever possible though.
Germany for example has multiple cities with over 1 in million population that are attractive for business and distribute resources and infrastructure. England has London with around ~7 million in population then Birmingham at 0.9.
Improved in what way? Punctuality? Frequency of services? Affordability of tickets? Refurbishment of stations? New rolling stock? It's a claim that doesn't match the experiences of many passengers. What is the source?
"It has some of the lowest fares if you book far in advance"
Even if this is true, it's little consolation for the vast majority of passengers who need to commute daily. The reality is that, for most passengers, rail in the UK is expensive.
Here is a monthly season ticket comparison from 2017: UK vs Continental Europe:
- UK: Luton to London St. Pancras (35 miles) | Monthly season ticket cost: £387
- UK: Liverpool Lime Street to Manchester Piccadilly (32 miles) | Monthly season ticket cost: £292
- Germany: Dusseldorf to Cologne (28 miles) | Monthly season ticket cost: £85
- France: Mantes-la-Jolie to Paris (34 miles) | Monthly season ticket cost: £61
- Italy: Anzione to Rome (31 miles) | Monthly season ticket cost: £61
- Spain: Aranjuez to Madrid (31 miles) | Monthly season ticket cost: £75
Source: https://www.tuc.org.uk/industrial-issues/transport-policy/uk...
Dublin/Pleasanton to San Francisco - Embarcadero (35 miles): $6.80 each way, which works out to ~£204 per month for 40 trips.
This was something of an issue in the recent UK election. Obviously, EU membership was a major issue, and one of the main parties also wanted to renationize the railways... which is technically allowed by EU rules, it's just that there's very little point since we wouldn't be able to undo the weird structuring created by privatization.
With the number of failed and failing franchises, there's a lot to be said for a new approach, perhaps via a period nationalised to draw a line and start again. It seems patently clear that the post-Beeching nationalised years, decades as under-funded political football, and UK's privatisation were all absolute disasters for passengers... Seems like the EU should look, shake their head, and mutter "definitely not".
If it were me, I'd go back to the pre-war big 4 era, where there were a dozen or two smaller companies running parallel competition on some routes, or running goods and charters on big 4 lines. I probably wouldn't reinvent the pre-1921 situation that gave Manchester four stations all essentially overlapping - Central, Victoria, Oxford Road and Exchange. Pre-big 4 company history detangles that! Manchester Piccadilly was lucky in having less parallel competition.
What we've lost, and totally forgotten, is just how much those pre-war railway companies did to feed the railways. Hotel stations that were not the manky unloved budget rooms of the seventies through today, but often a premium offering. The railways bought into early bus, ferry and tram services to develop those, and ensure services fed the stations. Integrated transport and travel! Compatible timetabling! Now I have to go to the Netherlands to experience that.
It's been ongoing for twenty-five years at this point. The initial goal was to improve access especially, and entered into force in 1992. Essentially, to prevent the historic case of (e.g.) a nationalised railway forbidding anyone else operating on their infrastructure (so if you want to run a freight service and the nationalised company don't care? trucks it is!). This also vastly simplified cross-border services as it's then possible to apply for track-access on a non-discriminatory basis, which completely changed the face of the European rail freight market.
Note that as of 1 Jan 2019, international and domestic freight and passenger markets are, at a national level, all competitive. The world hasn't fallen apart in most of the EU. I think it's clear plenty did look at the British example and clearly decided to avoid following the British example in many ways.
Also I think it's worthwhile pointing out the UK's privatisation has been in some ways a success: subsidy now is much lower per-passenger (and lowering this has been a constant goal across all governments since privatisation), and the financial risk of the success is not borne by the government (and as the NAO concluded, no franchises making loses and/or failing would imply everyone is underbidding and the government is losing money as a result)
To me, it's not clear that a return to a nationalised operator would help things in GB; many of the problems come from political imperatives (c.f. "decades as under-funded political football"), and the railway is far more micromanaged by the government today than it ever was under British Rail. By and large the franchising bodies (mostly the DfT) are simply getting the services they ask their contractor for, and passenger should be more directing their annoyance at them; we also have a network that has seen massive growth in passenger numbers and number of services and comparatively little money spent on capacity enhancements to avoid a network run at capacity where a single short-term failure can propagate through large parts of the system. The change that needs to happen is political, and whether ultimate delivery is contracted out is mostly irrelevant.
> To me, it's not clear that a return to a nationalised operator would help things in GB; ... the railway is far more micromanaged by the government today than it ever was under British Rail
True, whilst I understand the reasoning of Labour's suggestion to re-nationalise, at best we would get a less effective repeat of 1947. 1947-1960 (before Beeching) worked OK in good part because BR still had the shape of the big 4 years. Even if Labour planned, executed and funded perfectly we would, at some point, get a Tory government, who would be inclined to under-fund it. Off we go again, still with infrastructure 50 years behind.
I mention nationalisation more as a reason to bring it all back into one whole first -- to split it up more sensibly at the next attempt. Divvying up various bits of all the various companies differently seems challenging. :)
The central micromanagement speaks to other UK post-war problems, which also affect rail. We've become one of the most centralised countries there is, whilst talking of smaller government and deregulation. Local funding has been steadily gutted since the end of the general rates. Local authorities used to be able to make far wider reaching decisions, and fund them. They could start, fund or develop local services, such as bus, power or train, and often create a municipal corporation to run it -- hands off.
That's the other appeal of returning to the pre-war organisation, it would distance the government from the day to day, and local choices, especially in Scotland, the North etc, might be possible once again. We could add capacity, and undo some of the worst of Beeching.
Now if only we could only undo some of the UK's excess centralisation as well...
Regional bodies having more power and influence (and money, probably with tax raising powers to actually control their supply!) would be a fantastic thing, but I think would be needed before nationalisation would significantly change the British rail scene at this point.
You might be right about regionalism being needed first, I'm not sure. Seems like we should try and regain both, and need both, so Liverpool and Manchester can work together and get a new railway, or compete and brief against each other and end up with the Manchester Ship Canal. Whitehall's only involvement in those essentially limited to saying "OK". The Whitehall conceived, planned and delayed "northern powerhouse", that has both Liverpool and Manchester mayors tearing their hair out at how little input their cities get just marks it as the usual, never fully delivered, centralised horseshit for the regions...
They’re not perfect but I am rarely delayed by more than a minute or two. Agree that a massive issue is the price, my train is close to £30 a day for a 50 minute ride into the city and it goes up by 3-5% every year.
The only proper delays I see are caused by unusual weather, as it’s the UK - flooding or snow.
Northern Rail, Transpennine, West Coast, ScotRail, East Coast (there already) etc that have or are about to be given to "operator of last resort", i.e. renationalised -- under a Tory government --for having failed badly.
It has follow on effects far down some lines as well, as problems at London Bridge would often cause delays far south of Croydon and up again towards Victoria (because Croydon becomes a chokepoint) for example.
But as elsewhere in the UK, expensive .
Over the summer there were at least 4 occasions where the whole line was suspended. They did get unlucky with some power failures if I remember. But twice the line was suspended through London due to heat effecting overhead wires.
Once could be just about forgiven (it gets hot in summer who knew that), but to have it happen again a few weeks later cannot. Sort the issue that suspends your whole network after the first occurrence! If it can't be fixed with a reasonable amount of time to schedule emergency works the system is not fit for purpose.
Well, the thing is: we weren't able to change the laws of physics in those few weeks. Metal still expands, so the wires still droop in the heat. The system is designed for normal conditions; sometimes it doesn't work in abnormal conditions. They could probably do better, but it takes time, effort, and money which are all in limited supply.
Earlier this year I had to take public transport for a few weeks whilst getting my power steering repaired and on one occasion my first train was delayed for reasons that were never explained. When I got to my interchange the next train was delayed due to signal failure. And when I finally got my train the driver announced that due to the delay they were passing straight through two stations, one of which was mine, so I had to get off earlier and wait for yet another train.
In the end it took me about 3hrs to complete a journey I can do in 30 mins by car.
Your comment made me very happy, I laughed and snorted, and I'm glad I wasn't drinking my coffee at the time I read it. Thank you for writing it, you are awesome.
Now I'm thinking of a Cult of Cthulhu who design railway depots as an homage to their unknowably evil master, with junctions and lines as a representation of its tentacles and body.
http://www.supernaturalwiki.com/Devil%27s_Gate_in_Wyoming
Think about how rare power outages are nowadays. When infrastructure becomes important enough it becomes run pretty smoothly.
So if you got lateness stats on trains, it would likely take all the JR stuff and mix stuff up
I agree with your sentiment, my comment is more about my theory that stats would not make the distinction (though it would likely exclude things like Tokyo Metro or the monorails).
Practical me: Having that information allows better decisions to be made. Rather than simply wasting my evening at a station other arrangements can be made, possibly having a meal in a local bar/restaurant rather than shivering on a station.
Saying all this though, the level of service is low when compared to other countries' commuter rail systems. One of the challenges VRE works with is that it hasn't owned the rail lines it's running on; they're owned by a freight railroad and passenger rail is typically run at a lower priority than freight. There has been some recent movement though by the state [5] to significantly improve passenger rail transit, especially in the northern part of the state (almost certainly related to Amazon HQ2).
0: https://www.vre.org/
1: https://content.govdelivery.com/accounts/VAVRE/bulletins/26a...
2: https://content.govdelivery.com/accounts/VAVRE/bulletins/26f...
3: https://www.vre.org/service/daily-performance/
4: https://www.vre.org/service/rider/train-utilization-trends/ (not actually trends)
5: https://www.governor.virginia.gov/newsroom/all-releases/2019...
This kind of jingoistic comment demonstrates how a tactical post with some technical information, can massively skew judgement. According to Network Rail, the routes in question are not only one of the most congested in the country, but have been mired with problems of under-investment and lack of desire to ease the bottleneck. Some of the routes connect salubrious stock-broker belts with the City of London, in addition to Brighton & Hove, East Sussex. Other routes traversing East Croydon, acting as a hub, also serve the mainline stations of London Victoria and London Bridge to (London) Gatwick ─ the second busiest airport in the country.
I wonder how many people will be forgiving or cheering for Network Rail and handing out accolades for their neatly constructed blogs, when they miss their flight or arrive to work late 'due to signalling problems'.
Part 1:https://www.networkrail.co.uk/news/strong-support-for-propos...
Part 2:https://www.networkrail.co.uk/running-the-railway/our-routes...
This specific remark sounds rather amateurish. This is part of the reason UPSes were invented. A Double Conversion Online UPS would have been able to handle this situation just fine.
I have never seen such dirty and bad power! Literally once a week we had to go to exchanges to reset fuses that had tripped.
We then started deploying (rather entry level) UPS to all locations and the problem just disappeared completely... we then invested in more high end UPS systems, but, yeah - dirty power is a big problem and UPS help.
> De flesta av felen berodde på att det batterisäkrade nätet hade överspänningsskydd, som skyddade komponenten i stället för funktionen. [0]
Also mentioned in english [1]
[0] https://www.nyteknik.se/energi/hardsmaltan-var-nara-2006-642...
[1]: http://www.castor.de/english/2006/0820.html
Contrast with PKP (Polish rail), where it’s not uncommon for a train to stand in the middle of nowhere for half an hour, without so much as a single word of explanation from staff. It’s changing for the better, but slowly.
Luckily the system shutter itself down as intended.
Electric railways have multiple different power systems - traction power, signaling power, and utility power, at least. The possibility of a short between systems means a conservative shutdown-on-overvoltage circuit breaker setup is appropriate.
If you're getting power surges in signaling power, it's time to shut down, go to red, and not reset until someone has tested the system.
Yet the actual system reliability and safety doesn't seem great. Signal outages happen nearly every day in the UK, and delay tens of thousands of passengers. Safety failures in signalling systems kill on average a few people per year.
How about a modern system, built with JavaScript-de-jure, running over 4G and WiFi, consumer GPS, "Download train-control from the app store", running on tablets or the train drivers BYOD phone, etc. How would it work out for safety/reliability?
I suspect counterintuitively it would work out much better. Simple cheap portable devices means a bunch of spares can be around always. "Phone battery dead? No worries, there's a tablet I can control the train from too". No WiFi? It also works with 4G or Bluetooth. Server down? It has a Peer connection backup. Hackers broke in? The system has safety logic built in to every device, and uses a consensus system so a few rogue systems can't crash trains. Power supply broken? We have 24 hours of battery, longer if I turn the brightness down.
No thank you.
See also
https://news.ycombinator.com/item?id=12071240
On a previous occasion when rail equipment failed because of a fluctuation in supply voltage the electricity company was not at fault: the voltage had remained within specified limits. I find it a bit suspicious that the article doesn't mention what the change in voltage was this time, and what variation the specification allows. The penultimate paragraph is perhaps an implicit admission that the electricity supply was within its specification.
I wonder if any of this is indirectly related to the official change in supply voltage, from 240 V to 230 V, in 1995. Nah, probably not; they can't have equipment that old, can they?
It wouldn't be the first time that rail companies not paying attention to actual grid specs caused transport chaos this year. Back in August, a power failure caused load shedding that dropped a million houses off the grid but preserved power to the rail network as intended. However, one train company had bought Siemens trains that couldn't cope and immediately shut down when they saw a dip in frequency of the level required to cause automatic load shedding, and required an engineer visit to restart. That one set of trains effectively rendered all the measures taken to ensure the rail network retained power almost useless, as they turned into immobile blocks all over the network.
I would give the rail companies a little slack. They did specify trains that should have handled the frequency change. But Siemens didn’t build the trains to spec.
The shutdown was unexpected and entirely down to Siemens not building their power supplies the specs set out by the rail company.
The problem was that the national grid then changed (loosened) the specification of what an acceptable fluctuation was. The trains could not tolerate this new specification.
What we can blame Siemens for is not providing an easier and quicker way to get the electrical systems back on their feet after a problem, instead relying on a specialist to physically visit a stranded train. I think they said they'd fix that one...
Alternatively, a good enough automatic transfer switch should transfer the power supply rather than tripping if the primary supply goes out of acceptable parameters.
(This goes to one of basics of high availability: failures, when they happen, should be short.)
This is done to ensure that if your smart stuff fails, you don’t end up seriously damaging something.
Unfortunately the dumb stuff pretty much always has to be reset manually. Usually because the dumb stuff tripping means you’ve encountered a scenario that no one anticipated, you really want someone to double check that nothing got damaged before the safety tripped.
It’s a little like a database starting up and discovering that part of its Write-Ahead-Log is corrupt. You really want a human to go in there and see what happened, rather than just ignoring the damage, plowing on and loosing data.
No remote control of them, had to get people to each breaker. (This was mentioned on Twitter closer to the time by NR.)
I did not find any questions that would trigger a path of improvement for similar future events:
- Why did the recovery take so long (up until Thursday morning)?
- If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event?
- Why is an on-site technician needed in order to re-activate the equipment?
- Could we build a system that enables the reactivation remotely?
There might be very well-rooted answers to the questions above (based on how real world electrical supplies work), but if the questions aren't asked, we won't end up exploring better solutions for the answers.
Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day.
If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event?
Why is an on-site technician needed in order to re-activate the equipment?
Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge.
Could we build a system that enables the reactivation remotely?
Probably, but that strikes me as dangerous.
The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.
And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
If trains start piling up near either Victoria (and Clapham Junction) or London Bridge, it quickly causes trains to back up all the way to East Croydon, a 15 minute train journey away.
Once it does, it blocks trains towards the other line, as East Croydon only has 6 platforms and no tracks bypassing it.
Once that happens, trains can start backing up tens of miles further South, and will be out of place when things are up and running again.
Victoria is a terminus, and is at times busy enough that trains gets 'stacked' two to a platform on the platforms serving Croydon, so turning trains back quickly becomes a fun exercise... (London Bridge also has lots of platforms that terminates, but seems to have fewer platform capacity issues after the overhaul)
So even if the trains that are in position get moving again pretty quickly, untangling that and getting platform space for all of those trains in short order can compound short delays very substantially.
I'm not so sure about that.
Just a few months ago a temporary power cut in London left some modern Siemens trains stuck for hours because they had shutdown do to irregular power and had to be manually rebooted.
Apparently nobody thought to investigate if the signaling system is also vulnerable to irregular power.
This PM exuberates professionalism with a clear and concise message! This is an excellent learning experience, for bystanders, travelers, and employees. Seems to me that this was handled in a good way.
The issue was highly unusual and difficult to anticipate but now you know. Good luck mitigating this issue preparing for the future. You will now be better as a result of this failure.
For a single power feed all of the following results in the loss of the output, (1) no input voltage, (2) input over voltage (under voltage also???). However, in the article it implies that switching to one of the other redundant power feeds or power supplies only happens for (1).
If the redundancy system had considered either (1) or (2) as a fault and to "find another supply, then shutdown this supply" then complete shutoff would have been avoided.
Also, since the active power supply shut down and switchover also didn't happen, there must not be any monitoring of the output of the individual supplies to trigger a switchover. To me, this suggests there is also another single point of failure in the current design.
I guess there is a market for smarter power failover systems.