119 comments

[ 2.8 ms ] story [ 174 ms ] thread
George Soros is pissed off
(comment deleted)
If someone with mod powers sees this (or OP), the title has a typo in it. It's intended to be "says CEO whose app roiled Iowa", not "rolled Iowa."
To “roll” is to mug or to rob someone. Seems fitting?
Agreed, I thought it was an intentional backhanded comment to the CEO.
A funny typo and one I wouldn't mind leaving too much, but I fixed it. :)
I've never heard that, but i've heard of being picked up by police being referred to as being rolled.
Seriously curious, there were only 1,765 precincts.

Couldn't they just create a Google Forms link share it to the field people sending the results back? It would be only 1765 lines in a Google Sheets doc, and would be practically free. (Not to mention more secure than a custom implementation by semi-incapable development contractors like in this case.)

Where is the fun in that? Gotta throw an elastic kafka beanstalk at it behind a series of microservices that serve the frontend as multiple single page apps composed into one umbrella via iframes.
If you don’t at least have a kubernetes cluster, you’re doing it wrong.
> Where is the fun in that? Gotta throw an elastic kafka beanstalk at it behind a series of microservices that serve the frontend as multiple single page apps composed into one umbrella via iframes.

Had they done any of that, it might even have worked. I don't think their stack is as exciting as that.

They legitimately wanted more security than that, with two-factor and other similar systems to prevent hostile actors inserting bad data.
Google probably does 2 factor better than $smallapp. I'd probably take more than 1 form per report though (I think that falls under "similar systems" stuff you mentioned).
Although a little kludgey, it does cross of the boxes:

- Only allow one entry per GSuite ID in Google Forms

- Enforce yubikey 2FA on GSuite.

- Issue each captain a Pixel 1 with Google MDM for remote wiping and app deployment.

- Turn off all other apps (Docs, Fi, Photos, etc...) in GSuite.

Wouldn’t all of this be more elegant using Graphql?
Are you serious or making a joke? Not saying it can't be serious. But it might be a joke. I can't tell anymore.
If that’s not a joke I’m quitting this industry and becoming a carpenter
It sounds like they made their account the day of the vote, how does 2 factor prevent someone from registering an account in your place?
Well you did see people just Tweeting the results from their precinct in all of this which managed to produce a good picture of what was going on. So you don't even need a collaborative document!
Of course, those tweeted images included PIN numbers which might have been useful to someone wanting to gum up the API with requests.
It would be actually free in addition to being practically free, and it would come with multi-factor authentication, authorization, and abuse protection that is undoubtedly in another league compared to what this company barfed out.
Both state parties used apps developed by Microsoft for the 2016 caucus. The Sanders campaign complained vociferously about it and the move away from MS was one of a number of "reforms" intended to placate those concerns.

https://thehill.com/policy/technology/267302-sanders-campaig...

While Sanders raised complaints about both the opaque results and Microsoft's role, other candidates also raised complaints about the opaque results; the move to allay all those complaints wasn't “move away from Microsoft” (a move to a historically Clinton linked and currently Buttigeig linked firm wouldn't allay Sanders bias complaints, anyway, on it own), but the transparency requirements imposed by the DNC that required.

Now, this necessitated a new app, since the Microsoft one provided previously didn't address the new reporting requirements. But it was the reporting requirements, not moving from MS to a firm that had even clearer reason to suspect anti-Sanders bias, that was intended to negate the complaints of Sanders and other candidates about the 2016 cycle.

This really gets me - you have to go out of your way to build a solution that won't work for 1,700 users[1]. I have seen systems that would collapse under less users but they are few... and all of them were train wrecks.

1. Ignoring like... stupid high information transmission systems where clever solutions may be required to concurrently stream data to even dozens of customers - the data here was measurable in bytes, _maybe_ kilobytes.

It doesn't appear that data transmission rates or quantity of users had anything to do with issues with the app. It was a bug in the code that was probably not apparent until being used in a live environment.
Last time I tried, a few years ago admittedly, Google Docs didn't perform well when you have large numbers of people trying to access a single document. It obviously scales great for total number of users, but I don't think having everybody hit one document is a mainstream use case that's necessarily well-supported.
There are google forms that you can setup that populate a spreadsheet. People don't need access to see the spreadsheet. One problem is that you have to authenticate it is the legit precinct captain, presumably they have some pin/password that they'll enter that is unique to their precinct. You can't prevent people from entering the wrong stuff, you'll have to cull the spreadsheet, but still, it seems pretty damn manageable.
^ this.

P.S. I work at Google and internally sheets is used thoroughly to survey large numbers of employees and it can view large sheets for a lot of people in "read-only" mode pretty well.

That's exactly what I did when I had performance problems. I also had some processing that happened whenever someone submitted a form. I found that I hit a performance limit when many people tried to submit the form in a short period of time.

I eventually found that if I changed how the spreadsheet was structured, a lot of problems were drastically reduced. I forget the details, but I think I switched to having one sheet that just collects form submissions and does nothing else, then another sheet that uses Javascript to pull data off the first and do the computations. That way processing was moved offline from form submission.

But the general point is, you can't just write a spreadsheet, collect thousands of form submissions, and have no issues. You can't just assume "Google scales everything". N people interacting with N (or 2N or 3N if you share with a few) spreadsheets is an embarrassingly parallel problem. N people interacting with 1 spreadsheet is not.

Or 'shard' it into 40 sheets of 45 rows each.

Deputize 40 administrators, give them each the 45 phone numbers for their districts. At the appointed time, have the administrators direct-dial each of the 45 numbers and quickly note the results. Combine the results from the 40 as they come in.

Secure and effective. It'd probably take 3 hours to wrap it all up.

How would the salespeople make their money then?
You could have sent someone with a sealed letter by taxi from every precinct, and it would have still been cheaper.
Back-of-the-envelope math going backwards from the $60,000 the Iowa Democratic Party paid for this app suggest that Shadow put in less than a handful of person-months of work.

(Presume $100,000 base salary, 1.4x overhead for a salaried employee, $0 profit, all employees working on the project are developers, and no code reuse with any other project. Some of these assumptions are clearly false, but I think it gives a reasonable guess at an upper bound).

However, seeing as the developers they hired didn't bother changing the default manifest and forgot to prefix a URL with HTTPS, I hope they Shadow at least wasn't paying them that much?

I really don't see a lot of accountability in any of his statements. It just seems like "well this thing went wrong, somehow, and its nobodies fault and hey nothing was lost". I find that to be absolutely abhorrant. Think about if you're one of their junior developers - you'd probably be blaming yourself for whatever bug that happened to get in. That the CEO wouldn't take any responsibility is a complete lack of leadership.

To those devs, jr or otherwise - every step that was supposed to protect against a failure in the app seems to have failed here, and you should not feel bad. Mistakes happen and should be expected, but the damage that they are allowed to wreck is the blame of poor management, not you.

Every single executive in this little software company used to work at Clinton's 2016 campaign [1] so they have political connections.

Not to mention, political parties aren't transparently run in many cases like we've seen in 2016 Democratic National Committee email leak. Most likely, these people are feeding off of their political connections and harvesting donation money from not just the party, but also the candidates [2].

This is also why I'm no longer donating to a political party directly, or a candidate who would in turn would do the same. It seems like the only independent candidate running under democratic party is keeping his campaign money in-house.

[1] https://www.crn.com/slide-shows/applications-os/who-is-shado...

[2] https://twitter.com/BustinTrudeau/status/1224697566182498306

These developers also developed Buttigieg's campaign software and Buttigieg donated to help develop the caucus app.
No COI to see here, folks.

Why isn't the DNC harder on crap like this?

The national party does not operate state elections and caucuses.
>Why isn't the DNC harder on crap like this?

Well, for starters, because the DNC doesn't run the Iowa caucus.

The DNC doesn't directly run the caucus, but the DNC (or maybe, more precisely, the democratic party) chooses to accept the results of the Iowa caucus and subsidizes (or entirely funds) the running of the caucus.

Additionally, the Iowa dem caucus (and primaries) aren't a governmental thing, the political parties aren't government organizations - they're private organizations that are just _all about the government and politics_ as such the Iowa caucus is really weirdly in a grey zone where there is even less necessary oversight than the administration of the general election in Iowa will have - but sort of more oversight since the DNC could just reject the results outright... Buuuut things like the voting rights act have forced some regulations into primary operation - it's all quite confusing...

The TL;DR though is that the DNC doesn't technically run the Iowa caucus but is more than capable of making the Iowa dem caucus committee feel a whole lot of pain and hold them accountable.

The DNC is composed of Clinton loyalists who want to stop Bernie Sanders from winning the nomination. John freaking Podesta was named to the Rules Committee for the national convention in Milwaukee.
This is a political response but I think it's important to highlight that the DNC seems quite enthused about Buttigieg, lowering the likelihood that they'll pursue anything about a connection between his campaign and Shadow when compared to more fringe candidates like Warren, Steyer, Yang or Sanders.
I believe that that is false: https://www.politifact.com/article/2020/feb/04/what-we-know-...

It looks like multiple campaigns (including the one the most to lose and the one with most to gain from a confusing Iowa) paid Shadow for services - Buttigieg’s campaign was not an investor, and does not appear to have anything to do with the development of the app. Do you have any sources for your claim?

> Some claims, such as that the Iowa caucus app was funded by Buttigieg, mischaracterize what we know.

> Buttigieg’s campaign wasn’t involved in the app’s development.

> Buttigieg’s campaign was not an investor, and does not appear to have anything to do with the development of the app. Do you have any sources for your claim?

This was not my claim at all

When you said "and Buttigieg donated to help develop the caucus app" was that anything to do with the development of the app or not?
It was not an investment and Buttigieg did not commission the app but they did help to pay for it by donating money to the Iowa dem group that was had the app developed last year
"We need more time testing this app, sir" - Dev

"No" - Manager

"But it could..." - Dev

"I don't care." - Manager

"Okay :(" - Dev

We'll test in production.
I doubt anyone will publish a good post mortem. I agree somebody is responsible for such a high profile failure, but I’m sure that the wrong person will be chosen as the scapegoat.

The app itself doesn’t sound like it needed to be that complex, and by all accounts, it wasn’t. But it feels like the company believed that the secret to a successful rollout was to ask the developer beforehand “are you sure there won’t be any problems?”

Every time I’ve been involved in releasing a web service, the step before “tell the world that the software is live” has been “make one or more complete end-to-end requests to make sure everything is running correctly.” The company clearly skipped that step: they announced the project was live without first making a single end-to-end request. We know that because the problems people reported and the problems that the company has described would have made it impossible for any upload to succeed.

i love a smug, passive-aggressive apology. just OWN up to it, acknowledge the implications of the issue you caused and provide an actual apology. unreal.

“I’m really disappointed that some of our technology created an issue that made the caucus difficult,” said Gerard Niemira, chief executive of political technology company Shadow Inc., in his first interview after the caucus. “We feel really terrible about that.”

You can totally visualize them deeply sniffing their own farts between the lines here.
Well, take a look at whose campaign they worked for prior to this.
Sounds like it’s all the interns fault.

Wonder how much tech this person understands

> just OWN up to it

That’s how you end up getting sued.

Its not like they can avoid being sued by not doing this. It just changes the cost point to be sued successfully, and actually denying impact and consequence probably increases their burden. Owning up, being sued but showing contrition has an upside.
Basically someone didn’t do the load testing before prod release.
People getting trained on the app last week reported manhy issues with it but the reply was 'don't worry you can always phone in the results too'
The company leadership basically didn’t take the Iowa caucus seriously. It’s obvious to me they wanted to bang out a quick frontend mobile app with a skeleton backend (Skeleton frontend for that matter) as a proof of concept to say ‘hey we did it! Now give us all the other primary contracts’, at which point perhaps they’d invest more heavily on a more comprehensive tech team/platform.

They did this with a minimal viable product approach for what should have been a critical production ready/tested app.

Wasn’t even a prod release. They used a Test Flight like distribution platform to bypass the app stores because the app wasn’t ready in time to go through App Store approval processes.
That’s what I thought originally, but there really shouldn’t have been much load (ignoring the load of downloading the app to begin with). There were fewer than 2000 precincts, and each precinct had to upload around 100 bytes of data. That shouldn’t have been hard.

One news story I read quoted an error message about a “missing protocol.” I believe that the “coding error” was really a configuration error: somebody forgot to include https:// in the web service URL. The app recorded the data, but couldn’t upload it.

Load testing would have found this, but so would a single integration test/smoke test.

My thinking is they simply didn’t do any dry runs with possible simulated data (bad data, malformed data, partial data), they didn’t consider all the ways things can go wrong in real time. This is less about load testing but more about exception handling/fallbacks.
A big part of the issue was that the app needed to be downloaded first and many of the precincts were in rural areas of Iowa with particularly spotty cell service.
There was no load to test, even the most inefficient code running a web server off a mobile phone could have handled this load.
From the article here - https://www.bostonherald.com/2020/02/04/companies-behind-iow...

Shadow Inc. — the tech company behind the app — launched in 2019 with backing from political nonprofit ACRONYM, according to a statement from the nonprofit.

ACRONYM founder and CEO Tara McGowan, a Newport, R.I., native, said on Twitter that Shadow is “an independent company ACRONYM invested in.

McGowan, who lists herself as a former journalist, married then-Hillary Clinton campaign staffer and current Buttigeig strategist Michael Halle in 2015, according to a Providence Journal marriage announcement and Halle’s Twitter account.

Not at all suspicious !

It's not really. That's how most software projects happen. "We need XYZ app/site/solution? Hm...I worked with these guys back in 2016 and they did good work. I'll make an introduction...."
But that first step, "we need xyz"... they didn't.
Also "they did good work" can't possibly be true, since Clinton 2016 was a bad campaign, with bad data, that lost.
Your logic is there can be no individual person who did good work on the campaign, if the overall campaign was bad?
We're talking about the person who _led_ the software and data efforts for Clinton in 2016. Clinton's data was bad and misleading and stale, which contributed to the candidate taking victory tours a week before losing four states the campaign thought they would easily win. Yes, I think every high-ranking official of that campaign should be unemployable in the same field.
Or their data was good and the election was hacked... but keep on executing people without trial.
This greatly oversimplifies a complex system that relies heavily on the honesty and behavior of thousands and thousands of humans. The fact that Hillary Clinton "lost" or even that the data was bad has very little to do with the quality of the application that was developed. Even if the wrong thing was built, it can still be built well.

I'm not a developer at all, but even I recognize that the world is full of crappy products that no one needs or that dont actually solve a problem but nonetheless were engineered and built exactly as the original scope dictated. Doing good work doesnt always mean that the work is any good.

> The problem was caused by a bug in the code that transmits results data into the state party’s data warehouse.

The amount of data we're talking about here would fit comfortably in an Excel spreadsheet. Why on Earth was there a data warehouse involved at all?

They could given restricted access to a shared Google Sheet at this scale.
From what I read on various articles linked in this thread and others, the votes were being checked against the warehouse (who knows where the warehouse is getting data, if that’s the one true source, why even have this app). It failed to check against it properly (they were getting back an error). I guess election night was the first time they tested this validation phase , as they were not able to account for it and the app simply failed.
CEO claims the "The app was sound and good" but there was a "transmission bug" in the code...

Other way to say it, if you don't count the mistakes - then the app is excellent.

(comment deleted)
These guys deserve all the credit they get...
“Other than that, how was the play, Mrs. Lincoln?”
> the app was sound and good, the problem was a transmission code ...

This guy doesn’t know what he’s talking about is he? Is this same person who wondered why he could copy the google frontend and it would look literally the same? What use is a browser with faulty internet ‘transmission code’

Yeah, like what does that mean? Their post requests had bad data from the frontend? How were they validating what they were sending?
Made by shadow, inc. No joke
Election software made by Shadow, inc. No joke
Does anyone have the source code or packages for the apps? I think the HN community could have a field day with this!
A typical non-apology. Making it sound like there were just minor subtle errors that they could never have foreseen and were caught off-guard by.

They were deploying an app to a very inexperienced user group, via developer certificate enterprise sandbox because they couldn't get the app done in time for App Store review and approval. They didn't do the requisite amount of live user testing and were sending out instructions up to 3pm on the day of voting.

They had a chance to evaluate whether they should take on this job and do it right or not. They could've advised and warned the IDP that they were going to get in trouble.

But they thought they could handle it based on their past inconsequential apps. And they wanted the business and publicity of doing this job. They were wrong, and now playing it off like it was some minor mistake.

It wasn't. Just a typical story of overly confident people getting in over their heads, and now passing it off as no big deal when in fact it is a big deal. Maybe if such behavior was fined or penalized there would be a lesson learned...

This guy is quoted in Vice magazine throwing shade on Vertica, a database written by a Turing Award recipient and used in many demanding roles. Then he hired some Starbucks prep cooks (literally, check LinkedIn) with no education and no industry experience and created a disaster. These events should replace whatever is currently in the literature as an example of the consequences of the Dunning-Kruger effect.
I really don't think its okay to be criticizing the job history of people. Everyone did something before they were a developer. You used to shit yourself daily, but we don't bring it up during code review. So i don't see how a previous position that lead towards being a developer is grounds for derogatory commentary.
You can teach anybody to do anything. It isn't the fault of the employees at this company that they ended up here. The failure in this case is the CEO's. It was their hubris to think they could found a company with no skilled labor whatsoever and produce a better solution than the legendary giants of our field.
It's Ok to have new / inexperienced devs on the team. In fact it's good for the industry to bring them in.

The problems come about when there are _nothing but_ inexperienced devs on the team.

"... that we got caught manipulating elections for our buddies."
> Niemira also confirmed Acronym’s role as majority investor in Shadow but declined to name the company’s other investors. He also declined to name the members of Shadow’s board of directors.

Hey guys, you done messed up - no more lack of disclosure. Your company will probably be dissolved due to this hilariously well publicized failure.

> The company’s mission is to help advance Democratic causes and candidates, and its employees were excited to have an important role in Iowa’s historic presidential caucus.

That sounds noble and all, but also pretty silly - how was your solution ever going to help "advance Democratic causes and candidates" purely by serving as a drop-box for everyone to upload voter tallies to. The employees at your company can feel excited to be working to support the DNC - but it's really a stretch to claim that your vote tallying solution was advancing Democratic causes. It's like claiming, as a car manufacturer, that you're excited to help advance the cause of addressing back problems because some of the people who might buy your car end up driving to staples and buying a new office chair.

With all the furor around election tampering, why aren't there regulations around vote counting hardware/software?

Is the CMM still a thing?

https://en.wikipedia.org/wiki/Capability_Maturity_Model

"The CMM was originally intended as a tool to evaluate the ability of government contractors to perform a contracted software project."

For something as important as elections, developers (organizations) should be required to adhere to CMM 5 and use a very tight waterfall + TDD practices. Further, the product and testing requirements should be public. Not this shadow development crap.

As far as I know there is no legal basis for a vote having to happen at all, let alone be on the up-and-up for things like party primaries.
the dnc and associated state organizations are not government organizations and there are no rules or regulations on whatever processes they use to select a candidate for the presidential election. they can literally do whatever the hell they want.

for that matter there is no (federal level) law or regulation requiring a public vote for the president at all. the constitution establishes the electoral college but the methods by which electors are chosen and vote are left to the state themselves.

What a den of incompetence, thanks DNC. Trying to rig Iowa and failed so badly is laughable.
Most comments here seem to be about "what Shadow should have done", making it obvious, given the distribution of HN users, that Shadow really should have used a different systems approach. But I'd like to focus on a different aspect: they used the systems approach that they did (given budget and deadline), and the outcome was poor. Is it difficult, using their approach, to accomplish what they accomplished? Seems like a single function call is the likely culprit. What exactly caused this... A race condition, database sharding? What's the cause of this issue, at a technical level?
I remember hearing on NPR how annoyed people were with the DNC’s lack of transparency on this app, and their reasoning was “we don’t want to let the Russian hackers know how we’re doing this”.

The more this saga continues the more the democrat gang seems like a troop of out of touch hippie professors recruiting freshman to do their “technology” work.

Or, few rungs up the ladder, it's a convenient way to get in Sanders' way while claiming incompetence!
All you need to know is that they could have done this in any number of ways and an app was definitely the most complicated way. It is fishy.
First line in their job application:

"Shadow is a technology company dedicated to building power within the progressive movement."

Can some things remain non political these days? What a weird sentiment.

Are we moving to a future where I need to check the political leanings of software providers before they can agree to do business with me? I'm looking for a professional to do a job, you can keep your politics at home...

They aren’t progressive. So caring about the statement is pointless. North Korea’s official name doesn’t mean shit.
They were formed specifically to aid the Democratic party in the election. I think there's a lot of former Hillary campaign people on the team.

Most tech companies try to hold the appearance of being neutral.