WooHoo!! They hosted their website on AWS. And totally ignored privacy sensitive nature of census. Also deployed Google Analytics. And as with anything these days, added in a healthy dose of Trump. Neat work, guys.
BTW, this looks like neat program name, "Bernd is studying Creative Industries and Information Technology".
AWS is certainly capable of supporting confidential information.
Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not.
Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
I believe AWS and Azure can support government workloads. For example, Azure is part of IRAP which means the Australian Signals Directorate has signed off on them for certain types of non-classified workloads.
> Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies.
> ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
It's more that if you don't have exclusive access to the hardware then you have no security. Fine for most things, but storing an entire population's details not so much.
We've managed to get confidential data stored in Azure for a large financial organisation, all compliances they required passed.
A server is a server right?
Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese.
We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
I know this is a popular way to look at things around here, but no. It's not about cloud / no cloud. People with sensitive data welcome VMs and cloud deployment - as long as the cloud is in a very separate datacentre with only local people handling it.
2. an audited linking environment, involving staff activity being logged, monitored and, if
inappropriate activity is found, investigated. Any misuse would result in immediate
termination of access for the staff member, with further sanctions imposed if necessary;
3. ABS staff and in-posted officers sign legally binding Undertakings of Fidelity and Secrecy to
ensure they are aware of their obligation to protect confidential information, and the
consequences of disclosure (which include criminal penalties);
4. enforcement of the clear desks and clear screen policy;
5. access on a ‘need to know’ basis;
How do you enforce those rules on AWS / Azure / whatever provider's staff? How do you make sure only people with appropriate clearance access the servers?
7. Vulnerability Assessments are carried out on all new IT Systems by specialised staff in IT
Security trained in the field of Ethical Hacking;
How do you get those providers to agree to internal pentest exercises?
At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack.
> Older people are suspicious of cloud. The younger generation are more savvy.
So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)
That's great, but you are trusting AWS. If someone has access to the server hardware thay can completely own you and all your TLS data. End to end encryption is one way, but my understanding is that achieving this in browser with javascript isn't a good idea.
The younger generation are more gullible, lacking awareness of data security, oblivious to data sovereignty, and easily distracted by squirrels.
An Australian census is never going to take place in hardware controlled by a foreign national on foreign soil, especially not in a country where cloud servers have been seized in the past for simply having the possibility of containing data associated with a person associated with a crime.
One thing you will learn as you gain real world experience is that there is no such thing as "too paranoid" when it comes to IT projects handling sensitive data.
But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.
You beat me to it by a minute. Yes. It would take them considerably more to add data verification, audit trail, full logging, etc. It would also kill part of that performance. And all of that needs to be done allowing for the extra data storage requirements. No AWS for you.
I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.
The requirements of the census should be obvious to anyone using it. Not the requirements of some suits sitting around blowing money because they can't think of scalable ways to host a web form. The system has one main purpose and that is data collection. A concise list of requirements could be:
- Scalable to the entire population of Australia (not less than 1 million submissions per hour)
- Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services)
- Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet)
- Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)]
These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h.
Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
You think IT guys in ABS do not know about AWS? It is about what the management thinks. And management needs code.gov like initiative desperately. Otherwise we will keep bleeding money to IBM time and time again.
A benchmark doesn't mean anything unless it's benchmarked with all features and requirements accounted for.
I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form.
Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts).
Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.
> The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Sure, they made a nice little prototype -- or MVP, if you will. I'll certainly give them credit for that.
But did they really do it "better and cheaper ... in 24ish hours" when they didn't come anywhere near meeting all of the requirements of the project (most of which probably aren't even public/published)?
That's the trivial part. You seem to imply ABS spent 5 years doing just the web form and that's a false premise just there. Once they match the privacy requirements you can read about here:
While I agree with this sentiment if they are doing all of the post processing and data operations in band and not off in separate queues then the architecture was wrong to begin with.
Collect data => add to queue => process later.
It doesn't matter how long that process later step takes as people only care about being able to submit their census and their government not looking like a laughing stock.
This exercise is about performance not things like security clearances.
If the exercise was about performance, they could just point at google forms. They didn't. You said:
> The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Neither did ABS spend 5 years on the webform, nor the guys implemented everything that IBM and ABS did in 24h. They did something completely different. These guys got to skip all of the complexity of the solution and did the easy part in a weekend. You're also presenting the easy parts only. Even "add to queue => process later" is easy only if you want to show notification on a small site. Add all the assurances you need for the census and it's a multi-day task on its own.
Basically we're reliving the times of "why is Twitter going down all the time, I implemented Twitter in RoR in a weekend, look"
Thank you. I'm surprised where this discussion has headed.
Of course the kids didn't rebuild the ABS census system in 50-some hours, and I don't think the sensationalism in this article is meant to be taken so literally. The purpose of the hackathon project and of the article is more likely to help the average person understand that the ABS could have built this in a fraction of the time, at a fraction of the cost, and given the technically trivial task at hand, it shouldn't have failed.
Also, IBM built the original eCensus application in 2005 based on its own AIX operating system and WebSphere application. ABS tried firing up its cloud servers but failed miserably, and hence they had no option but to rely on IBM again, because of previous contract and bureaucracy. Australia needs code.gov.
Yeah this is what I don't get. Yes they solved the scale issue by not having anywhere near as many questions, using plain html, drastically reduced javascript complexity, but they failed on the issue that dominated the headlines for weeks prior which was data security. Build it again using software that can be hosted in a government datacentre.
Pretty much any half decent software developer could have done what they did over a weekend. This is incredibly clickbaity and is definitely not the sort of calibre of article I'd expect to see on HN.
It's obvious there's a lot more to a Government project than stands out. A project I was involved with had burnt more than 54 hours just in meetings and project management before anyone wrote a line of code.
But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal.
They've said a lot to the average non-tech about what you can achieve.
A bit of context for foreign readers. Every 5 years a mandatory national census is undertaken by Australia's premier data gathering agency, the Australian Bureau of Statistics (ABS). For over 100 years the census has been completed on paper forms delivered and collected by government employees.
Political interference by prior governments who cut funding and left the department without a boss for 12 months. Change of leadership, no re-funding and the newly appointed ABS leader over promises savings while implementing a new online census without a backup plan. At the same time (this is the real story) decides to collect names and addresses of every person completing the census to link these previously anonymous data gathering.
New code, untested systems and an expected audience of 15 million people logging on, most likely after dinner around 1930 EST. The system experiences load, the site admins panic, make mistakes then pull the server off-line. At the same time politicians and other stake holders deride ordinary Austrians who question the name-gathering issue. As the service goes off-line and people try to enter the census, the site is down, the PM reports on twitter everything is good just before 1930. The ABS continues to auto-tweet the service is working. Australian gold medal swimmer at the Olympics derides convicted Chinese swimmer of drug use. Australian politicians blame Chinese for hacking. [0]
72 hours later the service was still down. The census requires around 95% completion to be useful. 10M AUD is spent and we have around a 50% completion. The Australian security services investigated the alleged hack and found the administration and contractor (IBM) to be at most fault.
An all round cluster-f*ck.
I saw this claim yesterday (https://twitter.com/peterrenshaw/status/764991292828815360) and had a quick chat to Austin (SWE) and Adam (Organiser/AI & Neural networks) asking for them to write a summary of what they did addressing both non-technical for the benefit of very non-technical press and political wonks as well as a technical article that explains the assumptions made and addressing how the ABS may have technical reasons for not using third-party computing services. As far as I can tell this article is the general non-technical article by Adam.
Good summary. The only thing I wanted to add is that the site was load tested at 1 million submissions per hour, which if obviously not enough for the evening peak time on census night.
"the site was load tested at 1 million submissions per hour, which if obviously not enough for the evening peak time on census night."
Yep. There were some other technical/spec insights. Like the router failure [0] and the most important bit the use of really poor techniques [1],[2] to ID sensitive data and the potential sale of it. [3]
Nice demonstration of smart system design, but the first item on the Australian government's specification would be the use of a private Australia-based system. The bureaucrats who wrote the spec will have limited understanding of data security, and will believe building their own local system costing an order of magnitude more than a cloud-based system must be an order of magnitude more secure.
On a much smaller scale, we provide a school interview booking system used by thousands of Australian schools, built on Appengine. Sadly Queensland's state government has woken up to the fact that the data (names, email addresses and booking times - not hugely sensitive information) is stored overseas, and we're having to jump through all sort of hoops to avoid a blanket ban.
Basically if your customers are not in the US, you'll be well aware of this issue.
33 comments
[ 3.2 ms ] story [ 71.8 ms ] threadBTW, this looks like neat program name, "Bernd is studying Creative Industries and Information Technology".
I'd think I'd have read about that somewhere because it seems like a fairly serious limitation.
Are two Junior engineers capable of producing an actually scalable, secure, fault tolerant system with adequate consistency, integrity, availability? Probably not. In a couple of days? Almost certainly not.
Could it have been done in less time and for less cost than a government did it for? Almost certainly yes.
https://www.microsoft.com/en-us/TrustCenter/Compliance/CCSL
> Microsoft Azure and Microsoft Office 365 were among the first cloud services to achieve this certification for the storage and processing of unclassified (DLM) data. The certification recognizes the successful completion, review, and acceptance of a comprehensive assessment undertaken by an Information Security Registered Assessor. This certification can be leveraged by all Australian and New Zealand government agencies.
> ...certification provides assurance to public sector customers in government and their partners that Microsoft has appropriate and effective security controls in place for the processing, storage, and transmission of unclassified sensitive data (the majority of government, healthcare, and education data in Australia).
A server is a server right?
Problem could be cultural and generational. Some old school types like the idea of a server without redundancies and fire prevention and fault tolerance and high security clearance in a data center. They want it to be blinking lights at them in a cupboard on premises. They feel it is safe due to physical proximity, even though their firewalls are like cottage cheese.
We're noticing that issue with health data for medical practises. Older people are suspicious of cloud. The younger generation are more savvy.
Look at the actual rules and what risks they want to protect from: http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3...
Here are some of them:
How do you enforce those rules on AWS / Azure / whatever provider's staff? How do you make sure only people with appropriate clearance access the servers? How do you get those providers to agree to internal pentest exercises?At that point, it's just easier to stuff some more servers in the government datacentre - completely isolated from other projects. And whether it's cloud or not is a completely separate thing - it may as well be on OpenStack.
> Older people are suspicious of cloud. The younger generation are more savvy.
So no, this is complete bullshit. (and so is putting this as us-vs-them - I'm part of a fairly young generation in the CS, working on cloud infrastructure, and I still would not want to see census data processed anywhere outside of "no phones in the building" government environment)
An Australian census is never going to take place in hardware controlled by a foreign national on foreign soil, especially not in a country where cloud servers have been seized in the past for simply having the possibility of containing data associated with a person associated with a crime.
One thing you will learn as you gain real world experience is that there is no such thing as "too paranoid" when it comes to IT projects handling sensitive data.
It's not a yes/no question. It depends on the information: https://aws.amazon.com/compliance/
But as far as census goes, no. I don't expect any country to be stupid enough to trust a foreign third-party with census information. Even in the US, AWS has separate zones for federal applications. This includes people who specifically can touch the data involved.
I'm not saying their overall architecture wasn't better than what the official census had, but the solution is not comparable without knowing all the requirements.
- Scalable to the entire population of Australia (not less than 1 million submissions per hour)
- Significant attempts to combat expected DDoS attacks (the original census site decided not to take the offered help from upstream DDoS protection services)
- Data Security needs to be paramount (seems like the one thing they did right as nothing has been leaked yet)
- Just because it is a government contract doesn't mean the cost is irrelevant. [$9.8m + (testing costs exceeding .5m and .5m of OFFICE plants)]
These students achieved this in a caffeine filled weekend hackathon for around $500 and it achieves at least my first point as they benchmarked it at 4mil req/h. Bantering on about how it isn't an adequate solution just isn't productive here. The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Credit where it is due guys.
I have all sympathy for the argument that you could do things better and cheaper than the government and IBM, but while banter isn't productive, neither is a benchmark of a bare hosted web form.
Edit: and as another poster below me pointed out, lower downs in the ABS are well aware of AWS. Inability to get new tools and methods in was one of the reasons I left (also the philosophy that tech pay basically tops out at the APS six level, and they don't want tech experts at EL1 / EL2 levels, they want "managers". Also, the government has been pretty explicit that they don't value or respect the ABS: efficiency dividends, job cuts, left without a head for a long time, moving jobs too Geelong!?!, and low levels of pay for technical experts).
Edit 2: also I just want to stare that I in no way think AWS is necessarily the right platform on which to be conducting a national census.
Sure, they made a nice little prototype -- or MVP, if you will. I'll certainly give them credit for that.
But did they really do it "better and cheaper ... in 24ish hours" when they didn't come anywhere near meeting all of the requirements of the project (most of which probably aren't even public/published)?
That's the trivial part. You seem to imply ABS spent 5 years doing just the web form and that's a false premise just there. Once they match the privacy requirements you can read about here:
http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3...
Let me know how much time it took them (including getting security clearance to do it, all the post-processing, hardware management, etc.)
This exercise is about performance not things like security clearances.
> The purpose was to show that it can be done better and cheaper than what IBM and the Australian Government came up with in FIVE years vs 2 students in 24ish hours.
Neither did ABS spend 5 years on the webform, nor the guys implemented everything that IBM and ABS did in 24h. They did something completely different. These guys got to skip all of the complexity of the solution and did the easy part in a weekend. You're also presenting the easy parts only. Even "add to queue => process later" is easy only if you want to show notification on a small site. Add all the assurances you need for the census and it's a multi-day task on its own.
Basically we're reliving the times of "why is Twitter going down all the time, I implemented Twitter in RoR in a weekend, look"
Of course the kids didn't rebuild the ABS census system in 50-some hours, and I don't think the sensationalism in this article is meant to be taken so literally. The purpose of the hackathon project and of the article is more likely to help the average person understand that the ABS could have built this in a fraction of the time, at a fraction of the cost, and given the technically trivial task at hand, it shouldn't have failed.
http://www.itnews.com.au/news/ibm-wins-96m-to-host-ecensus-i...
http://www.abs.gov.au/websitedbs/d3310114.nsf/4a256353001af3...
Karl Stefanovic on HN, really? ~ https://news.ycombinator.com/submitted?id=codeka
But you can extrapolate a lot of the build into a more "Government approved" environment, and you still come in demonstrating that a modern type of environment can be done a lot cheaper than census, which was their goal.
They've said a lot to the average non-tech about what you can achieve.
Uhm... so an unbound AWS bill... that doesn't sound "fine" to me.
Political interference by prior governments who cut funding and left the department without a boss for 12 months. Change of leadership, no re-funding and the newly appointed ABS leader over promises savings while implementing a new online census without a backup plan. At the same time (this is the real story) decides to collect names and addresses of every person completing the census to link these previously anonymous data gathering.
New code, untested systems and an expected audience of 15 million people logging on, most likely after dinner around 1930 EST. The system experiences load, the site admins panic, make mistakes then pull the server off-line. At the same time politicians and other stake holders deride ordinary Austrians who question the name-gathering issue. As the service goes off-line and people try to enter the census, the site is down, the PM reports on twitter everything is good just before 1930. The ABS continues to auto-tweet the service is working. Australian gold medal swimmer at the Olympics derides convicted Chinese swimmer of drug use. Australian politicians blame Chinese for hacking. [0]
72 hours later the service was still down. The census requires around 95% completion to be useful. 10M AUD is spent and we have around a 50% completion. The Australian security services investigated the alleged hack and found the administration and contractor (IBM) to be at most fault.
An all round cluster-f*ck.
I saw this claim yesterday (https://twitter.com/peterrenshaw/status/764991292828815360) and had a quick chat to Austin (SWE) and Adam (Organiser/AI & Neural networks) asking for them to write a summary of what they did addressing both non-technical for the benefit of very non-technical press and political wonks as well as a technical article that explains the assumptions made and addressing how the ABS may have technical reasons for not using third-party computing services. As far as I can tell this article is the general non-technical article by Adam.
[0] check my twitter feed for examples of articles explaining the buildup: https://twitter.com/peterrenshaw
Yep. There were some other technical/spec insights. Like the router failure [0] and the most important bit the use of really poor techniques [1],[2] to ID sensitive data and the potential sale of it. [3]
[0] https://twitter.com/riskybusiness/status/763583981749100545
[1] https://twitter.com/peterrenshaw/status/763266008383578112
[2] "using publicly available data @TurnbullMalcolm yr #SLK581 is URBCO241019542" ~ https://twitter.com/peterrenshaw/status/763244867564601344
[3] http://www.smh.com.au/federal-politics/political-opinion/cen...
On a much smaller scale, we provide a school interview booking system used by thousands of Australian schools, built on Appengine. Sadly Queensland's state government has woken up to the fact that the data (names, email addresses and booking times - not hugely sensitive information) is stored overseas, and we're having to jump through all sort of hoops to avoid a blanket ban.
Basically if your customers are not in the US, you'll be well aware of this issue.