82 comments

[ 2.5 ms ] story [ 137 ms ] thread
Completely shambolic schoolboy errors!
[flagged]
Competence and professionalism.
Nothing much to do with government: organisations of all types regularly screw up in similar ways. Probably some over-promoted growling manager insisting "Do it now!" without listening to the "but ... but ... but" caveats. But maybe also an undereducated and oblivious techie.
Because private companies like Microsoft, British Airways, T-Mobile, Equifax never make amateur mistakes in cyber security.
Hey, only one of those is a true tech company. The others are failing legacy comoanies, former government owned ones, that just don't get tech / software. And Ms is, well, MS. /s
I thought the project was implemented by private agencies? Article says MCI and Caracal.

Isn't that supposed to be how the government gets the best results with the greatest efficiency under neoliberal capitalism? Subcontract the private sector to do it. They have the expertise to know what they're doing and avoid obvious errors, which would not be the case if the government did it themselves?

Even the best way of doing something is not a guarantee that it will always go perfectly. Usually the government mismanages a project like this and the contractor can do little aside from implement whatever the client asks for. Maybe it would have gone better if the government had built performance standards into the contract so that the contractor would have to pay a penalty if the site wasn’t reliable or available to the public, or if the entrant’s personal information was leaked.
Large, state-backed, entities that predate the internet seem to get away with stuff of this kind all the time.

The other day my mother tried to buy a train ticket. The payment went through, but something went wrong on the site and the ticket was not issued.

If this were just some e-commerce site, the payment provider would have had their head on a spike.

In this case she had to go through the usual return process.

Zerforschung (research to the point of destruction) is a perfect name for a site with this post.
The sad thing is I don't see the public sector getting any better at this anytime soon.
Don't see the private sector doing so either. It's a scramble every year for Glastonbury festival tickets.
Also sad: the (terrible) response times and efforts detailed here exceed what many organizations, private and public, have shown in their situations.
Worse, there's laws in place (in the name of "cost savings") that need changed before any policy improvement could be made. A group can't just decide "it would be better if we didn't use the lowest bidder." There's legal repercussions and losers can sue (leading to more expense than if they just went with them in the first place). It's truly terrible.
They don't just accept the lowest bid. It's the lowest bid that complies with the requirements. You can tighten up the requirements and conditions.
You can also consider the demonstrated qualifications/competency of the bidder. I could submit a low bid for a project and if I have no history of ever completing similar projects my bid can be rejected on that basis.

However (at least in the few governemnt bids I've been involved in) if the low bidder does not get the award, they can challenge the award and often do. Then the government has to defend their decision and give the reasons the low bid was disqualified.

> I could submit a low bid for a project and if I have no history of ever completing similar projects my bid can be rejected on that basis.

Good luck with that. It is prejudiced discrimination because you assume that if the bidder never did a contract like what you're looking for, they will not be able to do it.

It would be just like you don't want to hire a junior because they never worked before :)

(comment deleted)
I can tell you for the procurement that I had anything to do with at a UK school, though for projects above a certain cost floor three or more quotes were required, I advised NOT taking the cheapest nor most expensive bids unless there was a specific good reason...
It's supposed to limit corruption by making it harder to give contracts as political favors
I know there's good intentions, but the system has been gamed/exploited for a long time with no real way to defend thanks to the legal implications.
You scored the mystery ride
Stuff like this is why I prefer to take a bus in Germany.

Trains are overbooked with free tickets and promotions (free pass for entire summer for 50 euro). While underlying infrastructure is not ready for such load. It leads to delays and mistakes. Plus railway stations in Germany look like homeless shelters!

On other side Germany has excellent motorway network. Flixbus is very cheap, quite comfortable, goes all the way to airport, and always on time!

Let's see. Cologne to Berlin takes ~4:40 hours by train. Flixbus takes 9-10 hours, not counting the time it takes to get to their departure station which would involve a train journey as it's not actually in the city centre.

Flixbus is 50€ cheaper when traveling that route tomorrow but that's about all it has going for it.

1 hour by plane (+ time hanging around the airport, but train/bus has the same issue there)
Train and bus normally have that “10-20 minutes ahead” planning to be at the station.

Planes? At least an hour, and if you cut into that, and the queues or security theatre more mind boggling than normal, you’ve missed your flights.

Eurostar is similar to airports, so I’m glowering at them too!

Even if you aren't at the airport that early, it still takes you an hour to get from the airport to the city centre in Berlin, and about half an hour to get to the airport from Cologne's city centre. That's by train, by car it takes even longer.
It depends if you're going from "centre to centre" or "somewhere near Cologne to somewhere near Berlin".
Sure but assuming you're traveling from centre to centre, which is where population densities are highest, is a sound assumption. Otherwise you can always find spots where getting to the airport, train station, flixbus stop or whatever takes extremely long with one mode of transport over the other.

Doesn't distract from the point that long distances busses are very much not an alternative to rail (or planes for that matter) unless price is the deciding factor. And even the latter is questionable in many cases thanks to the 49€ ticket.

Also no time at all when arriving. Getting out of a train and the station rarely takes more than five minutes; usually less.
The 50eur pass doesn't include the trains you'd use for the trips you'd use Flixbus for.
Just wanted to mention the photos of the model trains used to show the progression of events was charming.
> This project was implemented by the same agencies - MCI together with Caracal.

I suspect that this is the root cause of this and for many other systems failing. When a project is created by the lowest bidder, as a one time effort with fluffy requirements why would they invest in proper architecture, planning or testing? Why would they invest in securing resources when they are paid anyway?

This is where quite a lot of people would insert a rant about "state capacity": the ability of the state to actually do things it wants and intends to do. Which requires people to do those things, trained with appropriate skills.

The peak of "state capacity" was undoubtedly WW2, when governments bypassed market mechanisms and became command economies. Out of necessity - war is the one venture in which failed state capacity can end the state itself, and the personal privileges of those running it and the elite around them.

It's not a coincidence that the centralized socialist institutions of the UK, the NHS and state education, date from that period. Heck, the state commissioned the invention and building of cutting-edge computer technology! But since that no longer matters, there's little to no will to build state capacity in computing.

It’s not WW2 mobilisation but I’ve always thought the UK’s Government Digital Service is a wonderful example of what government can achieve in tech:

https://www.gov.uk/government/organisations/government-digit...

As I understand it they’re effectively a central dev shop for other government agencies. It’s worth their time investing in good practises because they’re going to use them over and over again. And from the user perspective you get a very consistent, reliable set of tools for interacting with government. A win win in my book.

This is something we desperately need more of in other countries. We've found something like a dozen breaches of similar severity in the last 6 years and they all came from systems developed through public tenders by companies that either aggressively under-priced or used other (legal or illegal) dirty tactics to win them. The very few things that were developed in-house have proven to be far more reliable and secure, not to mention developing them was far cheaper and the UX is better and more consistent between them.
As a software "consultant" (meaning i do what everyone else does but get paid more) the consultancy business is fucking bullshit. Just padding resumes and counting billable hours, nobody gives a fuck about quality or maintainability or anything. Just money.
GDS did a great job building gov.uk, but everythig else they touched was an abject disaster. From the Diabetes project at the NHS, the fiasco that was the Office of the Public Guardian, the even bigger fiasco that happened at Border Force, the NHS, DWP...

Sure, when they were building web sites they delivered stellar stuff. Agile, break things and all that. But when you had real complexity they just... couldn't...

The Government Gateway is a prime example - single citizen login for ALL government services. It ran well, super robust and mature enough to have ironed out virtually all issues.

Then GDS decided that because the Government Gateway was based on a Microsoft stack, it needed to be re-done. The tech lead didn't understand the concept of Identity Federation, let alone SAML tokens, and that you just! can't! do secure code using agile (2-week sprint no good for meaningful security testing...).

I spent two long years at GDS banging my head against a wall. And then I left. And unsurprisingly the Microsoft-based Government gateway was never replaced, still going strong.

It's a shame that (from my perspective anyway) a lot of that state capacity seems to have degassed for the NHS. Right now it's impossible to get a doctor's appointment where I live. If you call any local surgery within a split second of 8:00am then you have a very low chance of getting an appointment, every time I've tried the line's busy or I'm number 60 in the queue and after a 40 minute wait all the appointments are gone. They are also fully booked for advance appointments every day (they limit the time in advance you can book, probably to prevent having e.g. a 2 year wait time on appointments, because it 'looks better' if nobody can go than if there's literally a 2 year advance appointment wait).

The only way to see a doctor is to go to A&E, or convince NHS 111 to give you an "emergency appointment" of some kind. All of this drains emergency resources and are not an option due to the time investment for average people with precarious employment.

Unfortunately many people won't believe these facts, because depending on which area you live in there's always plentiful appointments, in advance or on the day! Where I last lived, getting an appointment was easy, the difference in outcomes based on how wealthy or urban your area is leaves a bad taste in my mouth.

Briefly, we had a new surgery open that offered more appointments, at more convenient times of day, and we could actually see the doctor. All the other GP's started losing their patients to them. Then within a few months they were shut down by the local NHS trust, under multiple investigations (one of these investigations was regarding an offensive Facebook post by the surgery's chief, I kid you not). They then later reopened with normal appointment times and no free spaces, like all the other surgeries.

I agree with what you say re the command economy of WW2 allowing the creation of the NHS. But it was not this country that founded the NHS: it is some ancient, lost nation that seems utterly alien to me today. I don't believe we could achieve even a small version of what WW2 Britain did anymore, if our survival depended on it. I cite the UK's response to COVID19 as evidence.

Same as in the US -- right wing politicians who believe the state is bad at doing things remove the funding that allowed the state to do it. Then when that means those government provided services become worse, they use that to show that the state is bad at doing those things, and therefore even less funding should go there, et cetera.
This mirrors my exact experience with the NHS – the futile 8am phone calls, to relying on NHS 111 for any hope at getting medical attention.

Coming from Australia and previously NZ, the healthcare system here seems barbaric.

New Zealand isn’t much better - last time I tried to book a doctor, they said they had n appointment free at the end of the month.

Meanwhile, if you actually need to go to the doctor for anything urgent, it is often best to go direct to A&E.

> "I cite the UK's response to COVID19 as evidence."

Here's some things I would love to see an alternate-history version of:

1) Vitamin D has some involvement in the immune system. The US Department of Health[1] says "Your immune system needs vitamin D to fight off invading bacteria and viruses.". Harvard School of Publich Health says[2] "laboratory studies show that vitamin D can reduce cancer cell growth, help control infections and reduce inflammation", "a large meta-analysis of individual participant data indicated that daily or weekly vitamin D supplementation lowers risk of acute respiratory infections"

2) The UK NHS page on Vitamin D does not mention immune function at all[3] but does strongly imply that everyone in the UK is deficient during winter when it recommends "since it's difficult for people to get enough vitamin D from food alone, everyone (including pregnant and breastfeeding women) should consider taking a daily supplement containing 10 micrograms of vitamin D during the autumn and winter."

3) The Harvard page linked earlier says a randomized controlled trial with 340 Japanese school children given either Vitamin D or a placebo, the Vitamin D group had 40% fewer flu infections in winter.

What do we know about COVID? It's an infection, it's expected to be more prominent in winter, some of the knock-on effects are to do with inflammation of tissues all around the body - lung, heart, brain, nerves.

So, would anything have played out differently if during the early days of no vaccines and no effective treatment, the NHS had leaned hard into Vitamin D testing and supplementation? Anyone presenting to a doctor or hospital or care home of any kind for any medical problem gets a routine blood test for VitD levels as well, any blood tests happening for anything also test for VitD levels, high risk people picked out specifically and called for testing, generic supplments freely available from GPs and pharmacies even without prescription using the NHS's large scale buying and negotiating power, anyone found deficient given a strong dose or large injection to start with, public relations push for the public to supplement or get checked, kept up all through the year leading into the first winter. Would it have made a difference to the ease of it spreading, to the amount of dead people, to the amount of hospitalized people, to the amount of long-term complications, would it have flattened the curve, helped the NHS, reduced or eliminated the lockdowns?

I am indoors most of the time, but I eat a lot of the recommended vitamin D foods - dairy, eggs, red meat, sardine, mackerel - and still had 'severely deficient' levels the first time I paid for my own test out of my own curiosity[4], and then 'insufficient' the next time.

That seems like the kind of thing a "national health service" would be well placed and incentivised to do, whereas a for-profit expensive-pills-and-surgery-and-insurance-profit "service" isn't.

[1] https://ods.od.nih.gov/factsheets/VitaminD-Consumer/

[2] https://www.hsph.harvard.edu/nutritionsource/vitamin-d/ (click to expand the 'immune function' section)

[3] https://www.nhs.uk/conditions/vitamins-and-minerals/vitamin-...

[4] (by a UK NHS lab, one which doesn't sell supplements so it's not incentivised to report misleadingly low figures)

Governments do in-house development just as well as any company with a large development department.

Governments also do development off-shoring just as well as any company with a large off-shored project.

The kind of project people are talking about here never works. It doesn't matter who is doing it.

It's a fallacy to believe that all projects are just sold to the lowest bidder.

There are probably a dozen reasons why something like this might have occurred, and not giving the vendors a free pass, but assuming that a more expensive vendor would do a better job with security and reviews is just as likely to be a mistaken belief.

If a project is too expensive for a client to do well, they should not be doing that work in the first place.

Its actually probably more of a situation that a client cant discern quality. Its impossible to tell if the most expensive or least expensive if the best option. How does a non technical/semi technical actually grade this stuff appropriately?
This is a problem traditionally solved by the professional engineering licensing system. Most engineering curriculum in the USA involve an Engineering Ethics course that goes over such issues.

We're quite far from implementing such a system for software "engineers".

Just hire pentesters along with the development team. Make sure they are not affiliates. You can put in contract that as long as security issues are present they need to fix them before getting paid, which seems like a reasonable expectation. Even the best make mistakes, so let's at least leave those which are not trivial.

We don't trust building ethics, independent inspector comes and checks if everything is as it should be before it can be used by the public.

This can't happen more often than not, because the client has no idea what a pentest is. They don't even know what a is XSS means, or even API. These things are negotiated by people that only can see the frontend, and if it looks great, _snappy_, _flashy_, with random animations and following current trends it's OK. The contractors know and can easily detect this, so they focus on frontend and don't waste their time in behind-the-scenes polish. You don't polish the security or find a costly query that could bring the site to their knees, but you add a scroll-spy that brings some images from nowhere.
Investors have no idea what thickness the wall should be in their building either.

Clients not being experts at the job they are getting somebody else to do is not a new pattern. So while some trust is required, it's best if you can get somebody else to verify.

I've seen a few smart clients over the years which when faced with some excuses from a software house hired another one to give them opinion about the codebase and capabilities. It seems pretty intuitive. It seems like a money well spent.

> Clients not being experts at the job they are getting somebody else to do is not a new pattern

In fact, a whole lot of the time, the entire reason someone hires a professional is precisely because they themselves aren't experts.

> solved by the professional engineering licensing system

Good narrative: but certification and guilds do not solve the problem.

No, but actual liability does.
If that were true, then there would be zero disasters in high liability countries. liability doesn’t prevent disasters from happening.

For example: UK Grenfall towers. https://www.bbc.com/news/uk-61724373

It is a wrong to assume that only engineers can cause deadly mistakes. Also we have penal liabilities that don’t need licensing:

   The [UK] Health and Safety at Work Act 1974 is designed to stop employers putting the public at risk, not just employees in the workplace. Individuals can be prosecuted and a serious breach could attract a two-year prison sentence. 
And sometimes some pretty big exceptions:

  the [UK] government can't be prosecuted for corporate manslaughter

Locally to me in Christchurch, New Zealand, there have been no prosecution for the CTV tower collapse: https://www.nzherald.co.nz/nz/fatal-ctv-building-collapse-po... https://www.nzherald.co.nz/business/govt-considering-introdu... The second link is interesting because it looks at changing the law to add liability (not engineers licensing changes )

In both cases, there are multiple layers of failure, and many causes could be asssigned. Especially the CTV building with inspections before collapse and warnings from people working there ignored.

Hire independent consultants with software experience to work on an hourly basis to write portions of the RFP and evaluate the responses. There is a niche industry of experts who help buyers with this stuff. Of course, if the customer is totally ignorant about software then it can be difficult to know which consultants to trust but generally they can ask around industry circles and check references.
You are right of course, I was just theatrically Exaggerating.

If we assume there is no corruption involved, then lack of competence from the project management side can fail such a project. For this example it could be failing to mention or think about the extra load on the first hours in the SOA

If my company is anything like the others, its very rarely the lowest bidder who wins at all.

Usually there is some sort of RFI process, where they ask a few companies 'hey can you build this for us? What are the types of services you would propose'. The list of companies here is already more or less pre-existing partnerships, or ex colleagues or...

(it mostly always contains Microsoft, and your boss is ex Accenture, so it involves Accenture, and for good measure to seem like they are open to other options they invite Deloitte and some other players as well, sometimes even IBM has joined the club again)

Then they decide who they deem thrustworthy, and you end up with Microsoft and (insert boss previous employer). So not only do you not get the lowest bidder, you can some veeerrryyy generic company that doesn't care and just sends juniors to solve it. This process is called the RFP. And it typically is far from neutral

Sounds spot on to me. Just let a bunch of unsupervised kids loose to screw shit up as much as they like, then after they've done that for a couple years you call them seniors and charge double for their time.
Yeah, there is probably some minority/equity politics that is giving the contracts to totally incompetent people that have the right gender claims or have the right skin colour.
The way that government procurement often works, it isn't the cheap contractors you end up with, either. Not many companies have the resources or willingness to stick through the long time it often takes to go through the bidding process, nor all of the pre-qualification requirements.

The whole process is long and drawn out with all sorts of checks and balances built in to it, based on previous learnings from previous contracts that have failed in various ways (particularly if it has embarrassed an elected figure). No doubt on the back of this failure, there will be more conditions added to the bidding process, making it even harder to find a vendor.

Usually by the time the entire process is done, there's not many vendors left and in my experience they're usually not the ones you'd actually want to do the work if you had a choice, just often ones that'll at least get you something.

And sometimes the technology becomes close to not supported and you end up creating a new project with 5 year old technology because of the drawn out bidding process.
As this is a government contract, and there are strict public transparency rules on government contracts, I went digging.

Here's the call for tenders: https://etendering.ted.europa.eu/cft/cft-display.html?cftId=...

And here's the award: https://ted.europa.eu/udl?uri=TED:NOTICE:120998-2022:TEXT:EN...

Some interesting things:

1. This is a broad framework contract for marketing, for the eye-watering amount of 300 million euro. The title is "Belgium-Brussels: Framework Service Contract for the Organisation of Large-scale Travels of Participants in the Context of Erasmus+/DiscoverEU"

The reason they do these kinds of framework contracts is because the legally required tendering procedures surrounding government contracts are so onerous that it's better to do a broad contract once and bundle a lot of projects inside of them, than to have one contract per project.

2. The executing party (Caracal) is nowhere to be seen, instead the contract is awarded to EURail and MCI.

EURail is the intermediary I suspect, responsible for navigating the wild world of government contracting, and MCI is a marketing agency. They have no doubt built up years of expertise in how to successfully navigate these kinds of tendering procedures, and they probably are not the lowest bidder. Caracal is no doubt subcontracted by MCI, but as MCI is a private company we cannot see how much they were paid or how they were selected. So much for transparency.

In my own experience in government contracting, price is a factor but usually not the largest factor. There's a large set of requirements (which you can read through if you follow the first link), and the ability to prove that you will be able to meet them is mostly what determines who wins the contract. However, because it is so difficult to know how to do that, only a few parties will have submitted a tender, and the best of a poor batch may still not be very good.

Personally I think this kind of public procurement legislation is well intended but ultimately flawed. It does not result in lower costs, faster turnaround, better transparency, or overall better government. I'm in favor of transparency rules, but they need to be a lot more thorough and they need to cover subcontracting as well. I'm against public tendering legislation, as I think it prevents the government from being efficient.

(By the way, how awful is that public tendering website? It's like a flashback to 2003. No doubt built under one of those big framework contracts.)

The linked tender is mostly not about marketing. They tendered basically a pretty large travel agency. Eurail is surely not just a intermediary here. They are in the business of selling train passes and operate lots of the services tendered for decades (e.g. an online train booking platform)
Made me think about this podcast I listened to the other day: https://www.nytimes.com/2023/06/06/opinion/ezra-klein-podcas...

In it Jennifer Pahlka, a high ranking US government official who worked on heathcare.gov and other digital government projects, talks about her book that is about why most of these projects go as poorly as they do. Quite illuminating...

Lots of people complaining about state-run projects or suppliers who do stuff on the cheap but I think the simple fact is that in most people's minds, buying a "IT system" is like buying a car except that the car is built from scratch each time even though the customer wants off-the-shelf prices.

How many applications do we create that all do exactly the same thing? Payments, customer details, tasks, shopping baskets, items for sale etc. and how many times have we rebuilt all of that from the ground up with all the risks? Even if we know what we are doing, it is easy enough to forget something, for someone who didn't know what they were doing to build part of it, to cost enormous money to plumb together a tonne of bespoke parts.

I think the solution is 1) We need much better regulation of who has the relevant skills to do work to the required standard, we still allow untrained and unqualified people to build banking apps etc. 2) We need to create something that allows us to possibly certify implementations of standard functionality so they can be used to create standard applications, just like Peugeot might buy engines from Toyota that they know already work.

We talk about freedom of thought and creativity but the price of reliable and trustworthy software is probably only going to come by establishing a much higher level of quality - hopefully minus some of the BS you get with some accreditations.

It's a lot like building houses. Lots of manual processing and making the standard formula fit the specific application.
It appears as though you are merely rehashing the realm of SAAS and, to compound matters, the labyrinthine government contracting procedure.

The market already boasts software solutions that are more or less ready-made, precisely catering to your described needs, particularly concerning areas like payments.

However, governmental entities abstain from employing such software, as their provider selection process deliberately embraces a convoluted nature to sidestep any hint of impropriety.

Thus, the government contracting industry flourishes—a cohort proficient in maneuvering through the intricate channels of governmental procurement. Most private enterprises that excel in providing top-tier services opt out of engaging in this government contracting labyrinth because it's not worth the headache. It involves an assortment of antiquated procedures and certifications that the private sector seldom finds worthwhile to partake in, as they exclusively pertain to the realm of government contracting and are often accompanied by a disheartening degree of bureaucratic rigmarole.

Deciphering a pathway towards resolving this predicament would transcend the mere realm of overhauling regulations; rather, it necessitates the overhaul of modern bureaucracy and solving the arduous struggle government faces to keep pace with fast-evolving fields like technology.

Basically: Good luck with that!

It's a good thing the people writing software for the railway interlockings, unlike these guys, are held to SIL4 standards.
How were they able to generate / obtain the `apiKey` shown in the technical details in part 6?
Without looking closer I just assumed it was trivially extracted from the frontend.
You're right, it's the Supabase's anonymous API key they send with each anonymous request.
The age limits on the free tickets stand out to me. I guess it's all ROI forecasting. Either older folks are assumed to have exposure to the target country already, can afford the travel, or aren't worth it?
Although the faults are massive, the time to fix was relatively short, scroll to the bottom and look at the timeline... this is not a fault sitting in the open for months after reporting.
You're lucky that you got in touch with someone who understood the report and didn't refer you to the polizei, like happened in Hungary a few years ago when a 17 year old kid figured out he could change the price of a ticket in his browser dev tools.