From personal experience - no it won't. I have made mistakes with overriding warnings - when under pressure, hyper focused and already in the mental mode of - of course I am sure I want to do it - your slow brain is just shut down so you are moving a bit like on muscle memory.
From the many depressing years I've done UX work I can promise you no amount of changes they make will protect users from their own mistakes. It is far better for your own sanity to accept people make mistakes. This is what development and staging environments are for.
Can you change the title of this submission to match the title of the post please? Normally people change titles when the original is sensationalised however the opposite is true in this case.
Sounds like they only lost $661k worth, which is peanuts by comparison to most other crypto hacks/losses, probably didn't destroy the entire company since they're compensating their users, and not really worthy of the sensational title.
The currency was USDC, which is a stablecoin pegged at $1 by Circle (www.circle.com) who are generally held to be reputable, so it very much was real money.
It sounds like a bunch of people are going to pay for the cost to users of their own mistake out of their own pocket, so if they follow through with that we should probably applaud the move?
If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?
For all its flaws (and there are countless) the one thing about the show cryptocurrency space that stands out to me as a programmer is that programming errors can be suddenly very costly (granted, in this case it was more of a DevOps blunder).
Being able to very easily put a price tag on sloppy programming is intriguing to me.
When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
This is missing the point to a degree that makes me think you're being intentionally obtuse, but maybe you're just ignorant so I'll bite. Banking computer errors can easily be rectified by humans, banks are regulated, your funds are at least partially guaranteed by the government (depending on where you live). The degree to which you're trusting computer programs with your finances is orders of magnitude less than with cryptocurrency where it's possible to lose any amount $ of asset value in an instant with absolutely no means of recourse.
Your bank analogy is silly and nowhere near analogous.
>CitiBank can't get the money back they accidentally transferred to another company.
Given it was an accidental early repayment of a loan, this isn't quite the slam dunk you think it is. If they had paid a company they didn't owe money to, they could get the money back through the courts.
The reason was not because of a repayment but because you wouldn't expect a respectable bank like CitiBank to do such a mistake.
> “To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion, would have been borderline irrational,” he wrote.
> I and OP are saying, the future will look more like crypto looks today. Not a bright future.
I don’t agree. I trust that loopholes like that will be slowly rectified with legislation if not present today.
In fact, it’s likely that crypto will (problematically?) be heading the same way. There was a recent case of a crypto buff who found a bug in some project and made off with a few $Million and I think the courts said he could be arrested and expected to return it, just as if he made off with cash. Importantly, they basically said “blockchain isn’t the source of truth to the courts” which was the guys defense. IMO a bright future for people, but not for a crypto venture.
> he could be arrested and expected to return it, just as if he made off with cash.
You have cash in your house and someone breaks in to steal everything. The insurance will _maybe_ cover your loss given that you secured it with basic security. _Maybe_ the police will investigate and arrest the burglar.
But even in this situation, you were better off putting your savings in any bank account where any fraudulent transaction can be reverted with a button.
Now, you have your crypto wallet. It gets emptied by some random bot. Well, you are as fucked than with your cash, except that nobody will cover your loss and nobody will investigate your case since the burglar is probably from another country.
By cash I meant fiat. But yea crypto adds a lot of risk to storing your monies.
You can insure the crypto you have. It’s probs expensive to the point of being not worth it.
People investigate crypto hacks though. And if the perpetrators are in a jurisdiction that you have some legal availability to you can totally use legal means. Basically any western nation will allow such a suit.
> When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program?
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
I don't disagree with that there have always been niche areas where there is a high cost to programming errors (like space or medical).
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
At the first proper job I had we would measure our service outages in dollars per second. It was always in the thousands, and we were just a rather ordinary financial OLTP service.
First to market is only part of the problem. Another self inflicted problem is that execution of those "smart" "contracts" cost tokens (so essentially money). This incentivizes developers to write the shortest possible code, without any "fluff" like tests or additional checks or more verbose style. And the immutability problem also increases severity of the problem - you either deploy immutable "contract" signifying that it should be respected because it can't be changed later for malicious purpose, or you deploy modifiable "contract" and that is not good for the reputation of the company.
Immutable code is a poor replacement for trust in a company.
It shifts the trust from the company as it is now to the same company (and their capabilities) sometimes in the past. Preferably the company is operated completely anonymously.
It seems like the crypto community would do anything to avoid the legal/regulatory system and it's established processes of operation requirements, insurance and liabilities.
Doing that they regularly fail at performing the most basics of basic financial duties like not getting hacked and not throwing the keys to the kingdom.
I've yet to hear of a bank loosing funds and getting away with telling "tough luck" to their customers, but we've witnessed many crypto "banks" doing just that.
That also involved failing to have a viable rollback plan. It affected so many people so badly that questions were asked in Parliament; TSB were down for a week.
Now with cryptocurrency we've disintermediated the bank to produce a much better solution: your contract can be down forever and there's no Parliament to ask questions in.
The level of self-perceived vs. actual competence in the crypto space never ceases to amaze.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
When I first found about cryptocurrencies it struck me as it was the first time in the history where data on my computer were directly worth something. Which suddenly made security worth something.
This sort of thing is not exclusive to Solana. Ethereum will also join Solana and be another highly centralized PoS network, so I expect more of the same centralization points to apply to Ethereum and many other chains.
As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
Why is this called DeFi? If a dev can nuke an entire app (intentionally or accidentally) it's not decentralized.
Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
Multiple DeFi platforms exist. Some are mostly decentralized, others only use DeFi as a marketing buzzword. Some of these are actually centralized, but it doesn't mean all of them are.
In agreement with GP, Solana and other very centralized networks should have to earn their way into the term DeFi. It's hard to quantify but a network that can be "paused" by few devs and relying on the existence of a registered company clearly don't fit this term.
Yeah. I don't get why bother at all with the blockchain when a simple dB would be even superior, then they can prototype away and migrate to decentralized later, if anyone even wants that. Decentralization shares the "weakest link" property with cyber security: you don't get the benefits of you only do half of it.
Lessoned we learned harshly
EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
I mean that won't work, because --preserve-root is on by default. In the same way, eventually crypto may acquire the same checks and balances that other industries have, but it'll take a lot of costly mistakes to get to that point.
That's my point. This is the early days for crypto too. I'm not a fan of crypto, but it's clearly going to learn from these mistakes and develop the same kind of safety guarantees over time.
I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
Commands like that really need a confirmation prompt and a command line switch to override like --do-as-i-say (long form only). Good example of developers being put in an end user's shoes, I hope they learn from that mistake and update their programs.
I don't know about you folks but I can do stupid stuff also with confirmation. Maybe splitting the damaging actions in some substeps, so instead of a big bang you have a slow death?
You are right, something like closing a program shouldn't be valid immediately, the program should be frozen for a day with the ability to cancel the closing at any time until the day is over.
Also, for me when I make big mistakes like this I think I’m in a different environment or something. It’s not that I don’t realise what command I’m running, it’s that I miss the context.
So I would just confirm but not realise what I was confirming.
An example of such a command came up on Linus Tech Tips (LTT), from apt (IIRC) on Pop_OS!. Basically (paraphrasing) "this will destroy your system and render it useless; type 'I'm an idiot and wish to proceed'". Our eponymous youtuber went right on, typed the phrase and blitzed the system, and then moaned about how the system shouldn't have let him break it ... I suspect such prompts only work for cautious people, who possibly don't need the prompts as they'll actually read the scary warnings.
In his defense, it was literally the first package he tried to install, which was in the official repositories but horribly broken, so he had no real way to tell whether this was the normal flow and linux just made it really irritating to install packages.
That guy is a savvy YouTuber, man. He did that because the resulting content has high memetic replication properties.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
It's a solid practice in UX design. Physically having to type or copy/paste it in really highlights the action for the end user. There's only so much you can do to stop people setting their own house on fire but something like that puts the onus on them and fairly places the blame where it belongs.
The problems start when you know you want to light a house on fire, but you pick the wrong house.
It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.
Some actions have to be irreversible by design (think: emptying the trash to free up space on your drive, or deleting sensible user data). At some point, someone has to greenlight that action, and the best you can do is trying to ensure the user is aware of what they're about to do – and you have to trust that they're using their brain for once. You know, that thing in your head which distinguishes you from that thing sitting in the metal box under your desk... If it were possible to automate that decision in a flawless, 100% safe and correct way, there'd be no need for a human to press the button.
Life definitely has one-way gates, but tech can avoid them a lot more than we do today. Soft deletes and backups give a time-bound undo period to user actions, and so freeing up space on my drive isn't irreversible.
Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.
UX is not about putting the blame on someone, but to prevent errors. For this reason confirmations are not really the best practice. Sure, they reduce the number of mistakes by some margin, but they never get it 100%.
The better alternative will be instead of asking for confirmation to ask to explain the intent by picking one of the options or writing it down. Ideally, an alternative must be suggested. This will avoid automatic reaction, because there's no one clear path to the goal.
Command line switches like that won't, IMO and IME, make developers think "wow, I'm so glad this exists to guard against footguns". Rather they'll just set an alias to get around it. Likewise confirmation prompts albeit with a little more resitance. Once a repeat user is frustrated enough they'll bust out `expect` or similar, angry at the original command for mollycoddling them because they know what they're doing and don't make daft mistakes, damn it.
If you alias a —do-as-i-say switch to a command without wrapping into a script to make sanity checks of your own, IMO that’s negligence. The name of the flag should make you think thrice about invoking it without safety checks.
That's a technical solution to a psychological and organizational problem: when there should be no CLI in production, people still use it and get used to various errors and confirmations so much, that they ignore the signs of a catastrophe. Technical solutions never work in such cases.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
Normally such deployment mechanism is one standard command that picks up a tested code, enriches it with environment configuration and rolls it out. This command rarely changes and should be the same as deployment to pre-production or staging. This command is usually executed from a CI tool via a button click and requires only user credentials as an input.
Any changes to production must be tested before release. They did not do it.
The money within the contract wasn't the problem, that was transferred out as part of closing the program.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
This is why coins/tokens/contracts shouldn't have the code be updatable when they contain money...
If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
I once deleted like, maybe a few petabytes of data valuable enough to store on FusionIO cards with a bad config change. I think it destroyed $50-100MM once the dust settled in all the various ways that it cost money were added up.
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone!
Thanks that's interesting! Definitely not outraged, was just poking a bit for fun. 50MM is an insane amount of money/value that would bankrupt most companies and have a noticable impact on many industries. Funny that for Facebook it was probably just a small hiccup on the ad revenue gravy train.
Bernoulli distributions are everywhere, and as a result binary classifiers are everywhere. COVID tests (any binary medical diagnostic), fraud/no-fraud credit-card processing decisions, loan decisions, spam filtering, “toxic” tweet management and other sentiment analysis, it just goes on forever.
All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click prediction R&D. Maybe those benefits outweighs the costs, maybe they don’t.
We have GPUs because shoot-em-up games were “the big thing” in the 90s. Net win? Personal preference. But entertainment if one form or another often pushes advances in computing. The BOPR stuff from the linked paper originated on XBox Live matchmaking.
It’s perfectly fair to be like: I don’t like the ad-funded internet. There are tons of plausible reasons why someone wouldn’t like the ad-funded Internet, but most reasonable objections to it are of the form: “my internet experience is worse than it would be without ad funding”, past that you rapidly wander into “other people’s preferences are inferior to mine”.
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
If MM were million-million, that would mean OP destroyed $50-100 trillion dollars. About the GDP of the entire world. So it probably doesn't mean million-million ;)
This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value.
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
It's not exactly unique to crypto... in adtech, a configuration mistake on a web server can cost 6-7 figures as ads are being paid for, but take people nowhere.
Sure, but the company managing the ad network can choose to be gentle, and refund the money. With these sorts of crypto mistakes, the money is just gone, because humans are -- as a desired feature of the system -- unable to correct these sorts of mistakes.
indeed! the fintech world is already full of these footguns, but HN is very anti-crypto, so everything wrong that happens there is proof the whole industry is doomed
That's your opinion. Who do you think loses out in a situation like that? The ad network is getting paid regardless, the adtech company (DSP, etc) will pretend nothing is wrong unless their client has hard proof and the client (probably a company you buy things every day from) was simply trying to promote a new product and will assume performance was just off.
Ultimately it is their customer that pays for all this by continuing to buy their products and use ad-powered platforms.
> Yeah I can already imagine in other places, just ask Santander Bank. [0]
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
Sure - but a lot of systems have a safety net of test environments and change control, and a safety net as they can revert changes, and a safety net as they can restore backups, and a safety net as they can ask counterparties nicely to help undo things, and a safety net through the legal system, and a safety net of insurance.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
But we moved away from moving currency in sinkable boats hundreds of years ago. Error in plain old finance today can often be fixed. Do we need to regress back to antiquity?
This is a deeply stupid analogy, because as everyone is pointing out .. why would you do that? Why would you choose to build a system in which this is a risk, when it's not actually imposed by the laws of nature?
This is kind of a nitpick but it actually wasn't even radiography, it wasn't for imaging, it was for radiation therapy - blasting cancer with radiation to kill it. So it was even more dangerous than it might sound :( definitely something that should not have safety controlled by software.
There was also a horrible disaster with someone taking the radioactive material from a machine like that in an abandoned hospital and messing around with it at home. People died in both incidents.
it's not the dev's mistake, it's the whole unnecessary complicated and bullshit of this solana network and most other blockchain networks that are put up by incompetent developers.
> Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet.
Code executions reflect intentions, transactions are final. As promised on a blockchain. Let's see if it stays that way, wouldn't be the first time if not.
As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
> Strictly execute *peer-surveillance approach*that requires at least 3 peers to engage in the deployment process
> - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
> - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.
I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief.
I wish them swift recovery from their predicament.
157 comments
[ 2.8 ms ] story [ 215 ms ] threadThe program's designers should likely have chosen the verb "shutdown", instead of "close".
solana program lose-all-money
If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.
Otherwise why the edited title that explicitly points this out?
What was lost was not money, but electronic conkers of which a small number people paid real money for.
If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?
https://twitter.com/OptifiLabs/status/1564367455220219904
Being able to very easily put a price tag on sloppy programming is intriguing to me.
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
Where do you think your income and bank balances are tracked and stored? On pieces of paper?
Your bank analogy is silly and nowhere near analogous.
https://www.bloomberg.com/news/articles/2022-08-15/citi-sues...
You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.
Given it was an accidental early repayment of a loan, this isn't quite the slam dunk you think it is. If they had paid a company they didn't owe money to, they could get the money back through the courts.
Crypto is meant to evade those courts.
> “To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion, would have been borderline irrational,” he wrote.
https://www.nytimes.com/2021/02/16/business/citibank-revlon-...
That's as close to "code is law" and other nonsense of the crypto libertarians as you can get.
> Recipients of cash wired in error are typically required to return it.
I don’t agree. I trust that loopholes like that will be slowly rectified with legislation if not present today.
In fact, it’s likely that crypto will (problematically?) be heading the same way. There was a recent case of a crypto buff who found a bug in some project and made off with a few $Million and I think the courts said he could be arrested and expected to return it, just as if he made off with cash. Importantly, they basically said “blockchain isn’t the source of truth to the courts” which was the guys defense. IMO a bright future for people, but not for a crypto venture.
You have cash in your house and someone breaks in to steal everything. The insurance will _maybe_ cover your loss given that you secured it with basic security. _Maybe_ the police will investigate and arrest the burglar.
But even in this situation, you were better off putting your savings in any bank account where any fraudulent transaction can be reverted with a button.
Now, you have your crypto wallet. It gets emptied by some random bot. Well, you are as fucked than with your cash, except that nobody will cover your loss and nobody will investigate your case since the burglar is probably from another country.
You can insure the crypto you have. It’s probs expensive to the point of being not worth it.
People investigate crypto hacks though. And if the perpetrators are in a jurisdiction that you have some legal availability to you can totally use legal means. Basically any western nation will allow such a suit.
https://www.coindesk.com/policy/2021/12/22/teenage-suspect-i...
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
It seems like the crypto community would do anything to avoid the legal/regulatory system and it's established processes of operation requirements, insurance and liabilities.
Doing that they regularly fail at performing the most basics of basic financial duties like not getting hacked and not throwing the keys to the kingdom.
I've yet to hear of a bank loosing funds and getting away with telling "tough luck" to their customers, but we've witnessed many crypto "banks" doing just that.
That also involved failing to have a viable rollback plan. It affected so many people so badly that questions were asked in Parliament; TSB were down for a week.
Now with cryptocurrency we've disintermediated the bank to produce a much better solution: your contract can be down forever and there's no Parliament to ask questions in.
You're welcome.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
Whoops, I lost everything! Computers are useless.
In the crypto world all changes are permanent and immutable.
There is no recourse.
That’s the difference.
I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
Doesn't even look like the Solana cli supports an HSM for this usecase.
So I would just confirm but not realise what I was confirming.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.
Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.
The better alternative will be instead of asking for confirmation to ask to explain the intent by picking one of the options or writing it down. Ideally, an alternative must be suggested. This will avoid automatic reaction, because there's no one clear path to the goal.
People would tape one button on, then wonder why it would chop a finger sometimes
Yes it is annoying. Yes, there may be better choices. But it is there for a reason
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
It seems like this was in the "deploy to production" stage, and there has to be some mechanism for doing that.
(It also seems unnecessarily complicated; I'd appreciate a plaintext explainer of what actually went wrong)
Any changes to production must be tested before release. They did not do it.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-face...
If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer.
The average employee works at a larger than average company. Similar for the average customer or average investor.
That's just because big companies have more employees / customers / investors to contribute to that kind of average.
All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click prediction R&D. Maybe those benefits outweighs the costs, maybe they don’t.
We have GPUs because shoot-em-up games were “the big thing” in the 90s. Net win? Personal preference. But entertainment if one form or another often pushes advances in computing. The BOPR stuff from the linked paper originated on XBox Live matchmaking.
It’s perfectly fair to be like: I don’t like the ad-funded internet. There are tons of plausible reasons why someone wouldn’t like the ad-funded Internet, but most reasonable objections to it are of the form: “my internet experience is worse than it would be without ad funding”, past that you rapidly wander into “other people’s preferences are inferior to mine”.
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
Edit: thanks for replies! :)
See "How big is a billion?" from Numberphile:
* https://www.youtube.com/watch?v=C-52AI_ojyQ
* https://en.wikipedia.org/wiki/Long_and_short_scales
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
see: Knight Capital
Ultimately it is their customer that pays for all this by continuing to buy their products and use ad-powered platforms.
This is hardly exclusive to crypto.
> Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
Yeah I can already imagine in other places, just ask Santander Bank. [0]
[0] https://www.theguardian.com/business/2021/dec/30/santander-b...
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
That's reality.
Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
Are you saying that it is impossible to get valuables out of a boat that sunk?
"Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas"
https://www.livescience.com/bahamas-shipwreck-gold-jewels
If you'd prefer you can throw your gold bullion into a volcano, or perhaps mix it in with radioactive fuel.
But there were a few instances in WW2 where large amounts of value had to be transferred by boat, and in at least one case sunk by enemy action. HMS Edinburgh: https://www.warhistoryonline.com/instant-articles/hms-edinbu...
[0] https://en.wikipedia.org/wiki/Therac-25
There was also a horrible disaster with someone taking the radioactive material from a machine like that in an abandoned hospital and messing around with it at home. People died in both incidents.
As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
> - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
> - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.
I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief.
I wish them swift recovery from their predicament.